[CT-2286] [Bug] return behaves in unexpected ways when used within a call tag / block

[darist]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When I try to use return() from within a call block, I get an error.

Example:

# macros/something.sql:
{% macro callable() %}
  {% do print(caller()) %}
{% endmacro %}

# models/something.sql
{% call callable() %}
     {% do return([1, 2, 3]) %}
{% endcall %}

dbt compile

Observed: TypeError: sequence item 9: expected str instance, list found

Expected Behavior

I expect the value passed to return() to be returned by the call to caller() . In this particular case, I expect it to print [1, 2, 3]

Steps To Reproduce

# macros/something.sql:
{% macro callable() %}
  {% do print(caller()) %}
{% endmacro %}

# models/something.sql
{% call callable() %}
     {% do return([1, 2, 3]) %}
{% endcall %}

dbt compile

Relevant log output

No response

Environment

- OS: Linux
- Python: Python 3.10.9
- dbt:
dbt --version
Core:
  - installed: 1.4.4
  - latest:    1.4.4 - Up to date!

Plugins:
  - bigquery: 1.4.1 - Up to date!
  - postgres: 1.4.4 - Up to date!

Which database adapter are you using with dbt?

postgres, bigquery

Additional Context

This part in the code looks suspicious :)
https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/exceptions.py#L18

class MacroReturn(builtins.BaseException):
    """
    Hack of all hacks
    This is not actually an exception.
    It's how we return a value from a macro.
    """

[dbeatty10]

Thanks for reaching out @darist !

You found a true (not so hidden) gem in our codebase with the MacroReturn class 💎

Could you tell me more about what you are trying to do? I can't really tell, so I rewrote it a bit to try to do some similar things.

Could you try this and see how it works for you?

# macros/something.sql:
{% macro callable() %}
    {% do return([1, 2, 3]) %}
{% endmacro %}

# models/something.sql
-- print the result of the callable() macro
{{ callable() }}

{% set result = callable() %}

-- print the result the return value of the callable() macro
{{ result }}

-- iterate through the results and print each item
{% for i in result %}
    {{ i }}
{% endfor %}

[darist]

Thanks for taking a look, @dbeatty10 !

Could you tell me more about what you are trying to do?

Sure! I'm trying to create mocks for unit testing macros: EqualExperts/dbt-unit-testing#123. One of the ideas is to create mock implementations on-the-fly using the {% call %} tag. What I reported above was a contrived, minimal example that reproduces the issue.

... However, now I'm even more confused. I was able to return() from some {% call %} bodies, but not others... take a look at the 2 places where I mention this issue (7144) in the macro_test.sql file in that PR.

Any idea what's causing this behavior? possible workarounds?

Thanks!

[dbeatty10]

Sure thing @darist 👍

I took a look -- that's some impressive dark magic you are working on 🧙

I think return() will only work with macros and not within {% call %} bodies. For the case where it appeared to work, my guess is that it didn't actually reach the return() -- maybe mock_fn is always defined?

Anyways, I don't think is a bug -- we expect macros to be able to return values, but we don't expect the same for call blocks.

I don't have any great suggestions for workarounds, but have faith that you'll figure out something clever now that you know how it's expected to behave. 🧠

I'm going to close this as "not planned" since we don't plan on adding this ability to call blocks.

[darist]

oh no! I'm having to do a lot of business logic in jinja, which is pretty painful. Being able to define anonymous macros "on the fly" and associate them with the data they manipulate (so we can program like we do with structs or objects) would be helpful. Would you be open to contributions to enable support of return from call blocks?

Actually, I'd prefer to be able to implement macros in Python. Jinja is not really designed as a programming language, but I'm using (abusing?) it that way because dbt locks us out of the Python side of the jinja context. Are there any plans or discussions around supporting Python implementations of macros?

Cheers!