Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support loading default_args from shared defaults.yml #330

Merged
merged 14 commits into from
Jan 3, 2025
Merged

Conversation

pankajastro
Copy link
Collaborator

@pankajastro pankajastro commented Jan 2, 2025

closes: #297

Currently, DAG Factory provides two places to configure default_args:

  1. At the top of the YML file.
  2. In the DAG configuration YML.

The second option overrides the first one.

Post this PR, the user can also keep the default_args in the defaults.yml file. The configuration from defaults.yml will be applied to all DAG Factory DAG

Sample defaults.yml

default_args:
  start_date: "2025-01-01"
  owner: "global_owner"
  depends_on_past: true

The precedence for default_args will be as follows, after this implementation:

  1. At DAG configuration YML
  2. At the top of the YML file
  3. defaults.yml

i.e At DAG configuration YML will take precedence.

@codecov-commenter
Copy link

codecov-commenter commented Jan 3, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.65%. Comparing base (38555b1) to head (9c7ac52).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #330      +/-   ##
==========================================
+ Coverage   93.57%   93.65%   +0.08%     
==========================================
  Files          10       10              
  Lines         778      788      +10     
==========================================
+ Hits          728      738      +10     
  Misses         50       50              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pankajastro pankajastro marked this pull request as ready for review January 3, 2025 10:57
@pankajastro pankajastro requested a review from a team as a code owner January 3, 2025 10:57
@tatiana tatiana added this to the DAG Factory 0.22.0 milestone Jan 3, 2025
dagfactory/dagfactory.py Outdated Show resolved Hide resolved
dagfactory/dagfactory.py Outdated Show resolved Hide resolved
dagfactory/dagfactory.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks a lot, @pankajastro!

@pankajastro pankajastro merged commit fd685b2 into main Jan 3, 2025
67 checks passed
@pankajastro pankajastro deleted the default_args branch January 3, 2025 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Shared defaults for load_yaml_dags
3 participants