logs warning if deduplication state is large #1877
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs canceled.
|
willi-mueller
force-pushed
the
feat/1131-warn-large-deduplication-state
branch
from
cf80f0d
to
aeca74a
Compare
willi-mueller
changed the title
willi-mueller
changed the title
rudolfix
requested changes
Oct 8, 2024
burnash
force-pushed
the
feat/1131-warn-large-deduplication-state
branch
from
4e59fb7
to
dd66d42
Compare
burnash
added a commit
that referenced
this pull request
Nov 10, 2024
rudolfix
requested changes
Nov 12, 2024
dlt/extract/incremental/__init__.py
Outdated
| @@ -118,6 +118,8 @@ class Incremental(ItemTransform[TDataItem], BaseConfiguration, Generic[TCursorVa | |||
| EMPTY: ClassVar["Incremental[Any]"] = None | |||
| placement_affinity: ClassVar[float] = 1 # stick to end | |||
|
|
|||
| DEFAULT_DUPLICATE_CURSOR_WARNING_THRESHOLD: ClassVar[int] = 200 | |||
There was a problem hiding this comment.
Incremental is also a dataclass. you should declare uplicate_cursor_warning_threshold under lag and this is a good place to attach default value so DEFAULT_DUPLICATE_CURSOR_WARNING_THRESHOLD is not longer needed.
tbh. my preference would be to not create a new field at all and just keep DEFAULT_DUPLICATE_CURSOR_WARNING_THRESHOLD if someone want to change that, one can do it for all instances by changing the classvar. which is IMO pretty cool. WDYT?
There was a problem hiding this comment.
Good point, changed in ca3471c
…e, update the warning message, change the test to check for single warning
burnash
force-pushed
the
feat/1131-warn-large-deduplication-state
branch
from
dd66d42
to
ca3471c
Compare
rudolfix
approved these changes
Nov 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
As a first step towards resolving this issue: warns if the deduplication state grows over a large count of hashes of the primary key.
Related Issues
#1131