"Microbatch" incremental strategy #6194
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
github-actions
bot
added
content
mirnawong1
reviewed
Oct 1, 2024
mirnawong1
reviewed
Oct 1, 2024
mirnawong1
reviewed
Oct 1, 2024
mirnawong1
reviewed
Oct 1, 2024
mirnawong1
reviewed
Oct 1, 2024
mirnawong1
reviewed
Oct 1, 2024
|
|
||
| Incremental models in dbt are a [materialization](/docs/build/materializations) designed to efficiently update your data warehouse tables by only transforming and loading _new or changed data_ since the last run. Instead of reprocessing an entire dataset every time, incremental models append, update, or replace rows in the existing table with the new data just processed. This can significantly reduce the time and resources required for your data transformations. | ||
|
|
||
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. |
There was a problem hiding this comment.
Suggested change
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. | |
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. Instead of processing all of your data at once. Since each "batch" is based on a time period, like a single day, it makes it much faster and more efficient to update large datasets, especially when you're working with data that changes over time (like new records being added daily). |
There was a problem hiding this comment.
thanks for the suggestion! I will add more here
mirnawong1
reviewed
Oct 1, 2024
|
|
||
| During standard incremental runs, dbt will process new batches and any according to the configured `lookback` (with one query per batch) | ||
|
|
||
| <Lightbox src="/img/docs/building-a-dbt-project/microbatch/microbatch_lookback.png" title="Configure a lookback to reprocess additional batches during standard incremental runs"/> |
graciegoheen
reviewed
Oct 1, 2024
|
|
||
| Incremental models in dbt are a [materialization](/docs/build/materializations) designed to efficiently update your data warehouse tables by only transforming and loading _new or changed data_ since the last run. Instead of reprocessing an entire dataset every time, incremental models append, update, or replace rows in the existing table with the new data just processed. This can significantly reduce the time and resources required for your data transformations. | ||
|
|
||
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. |
There was a problem hiding this comment.
Suggested change
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. | |
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` columns and `batch_size` you configure. |
|
|
||
| <Lightbox src="/img/docs/building-a-dbt-project/microbatch/event_time.png" title="The event_time column configures the real-world time of this record"/> | ||
|
|
||
| - `batch_size` (string, optional) - The granularity of your batches. The default is `day`, and currently that is the only granularity supported. |
There was a problem hiding this comment.
Thoughts on making this a table instead of a bulleted list?
|
|
||
| - `batch_size` (string, optional) - The granularity of your batches. The default is `day`, and currently that is the only granularity supported. | ||
| - `lookback` (integer, optional) - Process X batches prior to the latest bookmark, in order to capture late-arriving records. The default value is `0`. | ||
| - `begin` (date, optional) - The "beginning of time" for your data. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches. (It's a leap year!) |
There was a problem hiding this comment.
I don't think this is optional
|
|
||
| <Lightbox src="/img/docs/building-a-dbt-project/microbatch/microbatch_lookback.png" title="Configure a lookback to reprocess additional batches during standard incremental runs"/> | ||
|
|
||
| If there’s an upstream model that configures `event_time`, but you *don’t* want the reference to it to be filtered, you can specify `ref('upstream_model').render()` to opt-out of auto-filtering. |
There was a problem hiding this comment.
Thoughts on nesting this under the sentence "dbt will automatically filter upstream inputs (source or ref) that define event_time, based on the lookback and batch_size configs for this model."
There was a problem hiding this comment.
I tried both above / below. I'd rather stick this below the image, because it feels like an exception rather than the rule — but I could be wrong! If we expect people to need to do this very often, that's one more vote in favor of a more intuitive (less-ugly) syntax
|
|
||
| ### Available configs | ||
|
|
||
| - `event_time` - The column indicating "at what time did the row occur" (for both your microbatch model and its direct parents) |
There was a problem hiding this comment.
maybe a callout that event_time and begin need to be in UTC? (also true of the CLI args --event-time-start/end
There was a problem hiding this comment.
or maybe having the "timezone"s section is better!
| select * from {{ ref('stg_events') }} | ||
| where my_time_field >= '2024-10-01 00:00:00' | ||
| and my_time_field < '2024-10-02 00:00:00' | ||
| ) # this ref will be auto-filtered |
There was a problem hiding this comment.
Suggested change
| ) # this ref will be auto-filtered | |
| ) |
|
|
||
| Microbatch incremental models make it possible to process transformations on large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or during manual backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` column you configure. | ||
|
|
||
| Where other incremental strategies operate only on "old" and "new" data, microbatch models treat each "batch" of data as a unit that can be built or replaced on its own. Each batch is independent and <Term id="idempotent" />. This is a powerful abstraction that makes it possible for dbt to run batches separately — in the future, concurrently — and to retry them independently. |
There was a problem hiding this comment.
I know we have an example in the "## How does microbatch compare to other incremental strategies?" section - but maybe we should have one here as well?
There was a problem hiding this comment.
good all! adding an example
|
|
||
| dbt has changed the default materialization for incremental table merges from `temporary table` to `view`. For more information about this change and instructions for setting the configuration to a temp table, please read about [Snowflake temporary tables](/reference/resource-configs/snowflake-configs#temporary-tables). | ||
| The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in dbt v1.6. |
There was a problem hiding this comment.
Do we need this? Or can we add it to the table somehow?
There was a problem hiding this comment.
I'm just going to cut it because older versions are EOL anyway
graciegoheen
reviewed
Oct 1, 2024
|
|
||
| ## How does `microbatch` compare to other incremental strategies? | ||
|
|
||
| Most incremental models rely on the end user (you) to explicitly tell dbt what "new" means, in the context of each model, by writing a filter in an `is_incremental()` block. You are responsibly for crafting this SQL in a way that queries `this` to check when the most recent record was last loaded, with an optional look-back window for late-arriving records. Other incremental strategies will control _how_ the data is being added into the table — whether append-only `insert`, `delete` + `insert`, `merge`, `insert overwrite`, etc — but they all have this in common. |
There was a problem hiding this comment.
"Other incremental strategies will control how the data is being added into the table"
maybe add a callout of how microbatch choose which of these "how"s is best?
|
Currently
with more adapters to come! |
|
Currently, as microbatch is still in "beta", this functionality is still gated behind an env var (will be swapped to behavior change flag ahead of the final 1.9 release) - so you need to set |
mirnawong1
reviewed
Oct 1, 2024
…bt.com into jerco/microbatch-docs
…o/microbatch-docs
graciegoheen
reviewed
Oct 2, 2024
Co-authored-by: Grace Goheen <[email protected]>
mirnawong1
reviewed
Oct 3, 2024
| | Config | Type | Description | Default | | ||
| |----------|------|---------------|---------| | ||
| | `event_time` | Column | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | | ||
| | `begin` | Date | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | |
There was a problem hiding this comment.
assuming begin is required , right?
mirnawong1
approved these changes
Oct 3, 2024
There was a problem hiding this comment.
this looks great, i'm so excited by this feature! i've fixed up the table as it was rendering weird and made some other small tweaks.
i have two questions:
- is the
beginconfig optional or required? - should we add the microbatch strategy on this 'built in strategies' section or is it not necessary bc it's still beta?
dbeatty10
reviewed
Oct 3, 2024
| @@ -28,7 +29,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo | |||
| - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This will also be released in dbt Core 1.9. | |||
| - **Enhancement**: In dbt Cloud versionless, dbt infers a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](/docs/deploy/run-visibility#compare-tab) feature. This will also be released in dbt Core 1.9. | |||
| - **New**: In dbt Cloud Versionless, the `snapshot_meta_column_names` config allows for customizing the snapshot metadata columns. This feature allows an organization to align these automatically-generated column names with their conventions, and will be included in the upcoming dbt Core 1.9 release. | |||
| - **Enhancement**: dbt Cloud versionless began inferring a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](/docs/deploy/run-visibility#compare-tab) feature. This will also be released in dbt Core 1.9. | |||
|
|
|||
There was a problem hiding this comment.
Fixing this in a separate PR (#6236) because I think we actually want to remove the other one. This is because this line is followed by content that will be orphaned otherwise:
Read about the order dbt infers columns can be used as primary key of a model.
dbeatty10
approved these changes
Oct 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #6136
What are you changing in this pull request and why?
microbatchincremental strategy (v1.9+)microbatchto table of incremental strategiesChecklist
website/sidebars.js