[CT-699] [Bug] Source freshness enabled config doesn't read variables

[fivetran-joemarkiewicz]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When leveraging the sources config to disable sources within a package, I am noticing that the use of variables with the intent to be overwritten does not work. See the example below:

In my root dbt_project.yml

Expected Behavior

The default variables is required otherwise I get the following error.

18:20:35  Encountered an error:
Compilation Error
  Could not render {{ var('jira_using_versions') }}: Required var 'jira_using_versions' not found in config:
  Vars supplied to <Configuration> = {}

Therefore, I set the default value to true, but found that this default value is the only value that is used. I would expect dbt would use the variable I declare in the vars section. This may not be that big of a deal of individual dbt projects. However, this is pretty crucial for dbt packages to leverage various variables and allow users to override them. It seems users cannot override these source variables however.

Steps To Reproduce

• Override a variable in your dbt_project.yml
• Use that variable to disable a source via the sources config.
  • You will need to set a default value, but use a different value than the one you set in the vars config.
• Run dbt source freshness and see the variable you declared is not used.

Relevant log output

See above.

Environment

- OS: MacOS M1
- Python: 3.8.9
- dbt: 1.1.0

What database are you using dbt with?

bigquery

Additional Context

Similar to what I said above. You could just hard code the values for the sources and see it works fine. However, for packages we leverage the variables so users can change the behavior of the package. I would expect using the same variable to disable features in other parts of the package would also be able to disable the source itself.

[gshank]

Are you trying to override a source that is provided by a different package (i.e. not your root project)? Because in order for source configs to override a source from a different package, you need to set the attribute: "overrides: <package_name>". https://docs.getdbt.com/reference/resource-properties/overrides

[fivetran-joemarkiewicz]

Correct, I am trying to disable a single source table from the dbt_jira_source package. I didn't think leveraging the sources config meant I was overriding the source entirely. The interesting thing is if I change my above dbt_project.yml to be a hard coded false instead of the variable, it does disable the version source table.

[jtcohen6]

@fivetran-joemarkiewicz The issue here is that vars defined in dbt_project.yml aren't available for setting configs that are also defined in dbt_project.yml, since the former and the latter are parsed at the same time. This has definitely come up before: #4873, #4938

If you're actually defining the sources in the package, you can use those variables in the sources' .yml definitions directly:

# models/src_jira.yml
version: 2
sources:
  - name: jira 
    tables:
      - name: sprint
        description: Table of all sprints.
        config:
          enabled: "{{ var('jira_using_sprints', true) }}"

If you're trying (from a separate root project) to disable a single source table from an installed package, your better bet is to use a simple boolean false.

[fivetran-joemarkiewicz]

@jtcohen6 this is exactly what I was looking for! I had attempted the below and noticed it wasn't working. It looks like I just needed the config. Thanks for sharing!

# models/src_jira.yml
version: 2
sources:
  - name: jira 
    tables:
      - name: sprint
        description: Table of all sprints.
        enabled: "{{ var('jira_using_sprints', true) }}"

Is this behavior only available in dbt-core >= v1.1.0?

Also, thank you for sharing the #4938 discussion! That is incredibly interesting and I look forward to seeing the vibe evolution of vars 😄

[jtcohen6]

@fivetran-joemarkiewicz Glad to hear that's working! And correct, this is v1.1.0+ only

[fivetran-joemarkiewicz]

Sounds great! This completely answers my original question. Thanks @gshank and @jtcohen6 for all the help 🎉