Bmenasha/cai fix issue 900 #905
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bmenasha
force-pushed
the
bmenasha/cai-fix-issue-900
branch
from
2c7055f
to
dca5945
Compare
- Fix GoogleCloudPlatform#900 by not overriding timestamp's type from `timestamp` to string as it's used as a table partition. - Performance improvements: - Don't perform a deep copy of the destination field within _merge_fields, it's not necessary as it's always a new object and Cloud Profiler says it was resource intensive. - Don't deep copy the 'data' field within ProduceResourceJson by removing it prior to the deep copy and adding it back to the element after. - Fix a bug where enforce_schema wouldn't work because it yieled the original elements, it needs to yeild a new element key pair. - Fix a bug when sharding asset types were we would delete the table while loading because not all shards were processed before deleting and loading occured. To fix this we now group by table name after WriteToGCS rather then shard key. Then assign back the sharded prior to loading.
bmenasha
force-pushed
the
bmenasha/cai-fix-issue-900
branch
from
97c3209
to
b3f0905
Compare
|
can someone review this? thanks |
|
can someone review this? |
|
There is another fix I wish to commit for issue #910 but they depend on this being merged. |
agold-rh
approved these changes
Jan 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Don't overide the
timestampfield's type.Closes asset inventory dataflow jobs failing with "The field specified for time partitioning can only be of type TIMESTAMP, DATE or DATETIME. The type found is: STRING." and "A hot key was detected in step" #900 by not overriding timestamp's type from
timestampto string as it'sused as a table partition.
Performance improvements:
Don't perform a deep copy of the destination field within _merge_fields,
it's not necessary as it's always a new object and Cloud Profiler says it
was resource intensive.
Don't deep copy the 'data' field within ProduceResourceJson by removing it
prior to the deep copy and adding it back to the element after.
Fix a bug where enforce_schema wouldn't work because it yieled the original
elements, it needs to yeild a new element key pair.
Fix a bug when sharding asset types were we would delete the table while
loading because not all shards were processed before deleting and loading
occured. To fix this we now group by table name after WriteToGCS rather then
shard key. Then assign back the sharded prior to loading.