You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
We do not annotate table schemas with Incremental settings if present. This annotations may be useful downstream for systems that will automatically generate pipelines that incrementally append or merge data and are without access to original Python code.
Tasks
standardize column annotation for Incremental. We should be able to recreate Incremental instance from those settings. Look what we already do when defining incremental for rest_api - we may reuse the same TypedDict.
When computing schema, create a proper annotation on the table. we allow only one incremental, so we need to update the code that diffs and merge tables so only one incremental is allowed.
For incremental that do not refer to a single field (simple json path) we obviously do not generate annotations.
now it is trivial to add incremental to dlt.resource decorator. accept both incremental instance and new typed dict. there's existing code that will use such incremental to override the one defined in resource function arguments.
Tests
on top of usual tests, test also the updated decorator:
do the value in decorator override the default in function signature
is it passed to an argument with Incremental type but without value?
what if there's no incremental argument?
The text was updated successfully, but these errors were encountered:
Hi, interesting, I have one question, it's somewhat common to use multiple columns (like an or clause) to load data from different incremental values, imagine a MongoDB source that have a created date and updated date but the updated date only exists when the record is updated the first time after creation, is there any way to solve this officially with dlt?
Background
We do not annotate table schemas with Incremental settings if present. This annotations may be useful downstream for systems that will automatically generate pipelines that incrementally append or merge data and are without access to original Python code.
Tasks
rest_api
- we may reuse the same TypedDict.incremental
todlt.resource
decorator. accept both incremental instance and new typed dict. there's existing code that will use such incremental to override the one defined in resource function arguments.Tests
The text was updated successfully, but these errors were encountered: