[CT-170] [Bug] Changes to a nested macro are not recompiled

[tjmw]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In my model I have a call to a custom macro nested within a call to set_sql_header:

{% call set_sql_header(config) %}
    {{ my_custom_macro() }}
{%- endcall %}

custom_macro is defined in another file. When I make changes to my_custom_macro they are not picked up when I re-compile my model - i.e. when I inspect the compiled SQL I see the output of the macro in its old state.

If, instead of using set_sql_header, I inline its functionality into my model directly, the issue goes away. Changes to the custom macro are picked up when I recompile my model:

{% set header_sql %}
    {{ my_custom_macro() }}
{% endset %}

{{ config.set('sql_header', header_sql) }}

I'm not sure if relevant, but I noticed that in target/manifest.json when I use set_sql_header then my_custom_macro does not appear in the list of macros that my model depends on (i.e. under depends_on.macros). When I switch to using config.set('sql_header', header_sql) directly then my_custom_macro does appear in that list of macros.

Expected Behavior

When I have a model with a call to a custom macro nested within a call to set_sql_header:

{% call set_sql_header(config) %}
    {{ my_custom_macro() }}
{%- endcall %}

and I make changes to my_custom_macro, I'd expect those changes to be reflected when I re-compile my model.

Steps To Reproduce

1. Create a model my_model and a custom macro my_custom_macro in a separate file.
2. Within my_model, nest a call to my_custom_macro within set_sql_header:

{% call set_sql_header(config) %}
    {{ my_custom_macro() }}
{%- endcall %}

3. Compile my_model:

$ dbt compile --models my_model

4. Inspect compiled SQL for my_model
5. Make a change to my_custom_macro
6. Re-compile my_model
7. Inspect compiled SQL for my_model and see it has not changed

Relevant log output

No response

Environment

- OS: MacOS 10.14.6
- Python: 3.9.7
- dbt: 1.0.1

What database are you using dbt with?

bigquery

Additional Context

No response

[gshank]

We statically extract the macro calls to fill in the depends_on. I suspect that the code that does the extraction does not handle this case. I will investigate .

[gshank]

It looks like the 'set_sql_header' is probably not actually executed at parse time, which would be the way that that nested macro would be found and added to the macro depends_on. @jtcohen6 Could you advise?

[jtcohen6]

@gshank I agree with your appraisal. The way set_sql_header is implemented behind the scenes is pretty strange:

dbt-core/core/dbt/include/global_project/macros/materializations/configs.sql

Lines 1 to 3 in c112050

	{% macro set_sql_header(config) -%}
	{{ config.set('sql_header', caller()) }}
	{%- endmacro %}

I think, because my_custom_macro() is not actually passed as an argument, but evaluated as part of the caller(), the dependency on my_custom_macro() isn't being captured at parse time.

Separate from this particular issue, but — macros within sql_header are already a less-than-ideal combination, since the macro's returned value (rendered at parse time) is stored in the node's config, not re-rendered at execution time, and lacks the (ugly) "double curly" workaround that pre/post hooks can offer to force re-rendering: #2793, #3264. We're overdue for updating docs to advise against it / detail specific known limitations.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.