Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Time Aware Freshness Checks #10963

Open
3 tasks done
LewendbtLabs opened this issue Oct 31, 2024 · 0 comments
Open
3 tasks done

[Feature] Time Aware Freshness Checks #10963

LewendbtLabs opened this issue Oct 31, 2024 · 0 comments
Labels
enhancement New feature or request freshness related to the dbt source freshness command triage

Comments

@LewendbtLabs
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I and several others (see bug) have encountered a common need among our clients/customers: the ability to adjust freshness checks based on the day of the week or external factors like holidays.

Typically, customers prefer longer freshness checks on weekends or holidays due to reduced or absent data updates. Additionally, the time of day can significantly impact freshness checks for frequently run jobs.

We propose a feature that allows for more dynamic and flexible configuration of freshness checks within dbt. This would enable users to:

Specify different freshness thresholds based on the day of the week.
Adjust freshness checks based on external date sources (e.g., a holiday calendar).
Potentially incorporate time-of-day considerations into freshness checks.
This enhancement would provide greater control and accuracy in managing data freshness, particularly for those dealing with irregular data update schedules or time-sensitive data operations.

Describe alternatives you've considered

The alternatives considered are in the the bug post above. The issue with the current solutions is that they are impacted by partial parsing or a convenient solution is not possible for holidays.

Who will this benefit?

Pacific Life is a customer that is actively trying to change their freshness checks based on day of the week and holidays. Most banks or business that are no open on certain days will not receive data on the days that they are closed.

Are you interested in contributing this feature?

probably not in my wheelhouse

Anything else?

nothing else

@LewendbtLabs LewendbtLabs added enhancement New feature or request triage labels Oct 31, 2024
@LewendbtLabs LewendbtLabs changed the title [Feature] <title>Time Aware Freshness Checks [Feature] Time Aware Freshness Checks Oct 31, 2024
@dbeatty10 dbeatty10 added the freshness related to the dbt source freshness command label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request freshness related to the dbt source freshness command triage
Projects
None yet
Development

No branches or pull requests

2 participants