Copy batches into db #50
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Kudos, SonarCloud Quality Gate passed! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses postgres copy to batch insert values into temp tables
Temp tables now have the following name pattern: given source system ID TEST/ABC-123-XYZ, the table names will be:
abc_123_xyz_dwc_identification, abc_123_xyz_gbif_multimedia, etc. We also replace "/" and "." characters with "_".
The above characters are illegal in postgres table names, and the sql statement couldn't recognize capitalized table names (would get error messages like "the table ABC_123_XYZ_dwc_identification does not exist"). Not 100% sure how these tables were able to have these names, probably some fancy jooq abstraction we can't use with postgres copy.
On an exception, we rollback all the temp tables and log the sql error.
Jira