Jinja undefined errors when extracting openlineage events #612
Comments
|
Hi @ivanstillfront , thanks for reporting this issue. Could you confirm that running this model using plain dbt works as expected? The library Cosmos is using to build Dataset URIs ( astronomer-cosmos/cosmos/operators/local.py Lines 230 to 241 in c82e6bc Is there any chance you could share a reproducible dbt project for us to test this out - it would save us time reproducing the issue - and would help us finding a fix quicker. |
|
@tatiana I get a different error, but it happens at the same spot when initialising
{{ config(materialized = 'table', tags=['test']) }}
select
try_to_timestamp('{{ run_started_at }}') as run_started_at
name: 'dbt_imo'
version: '1.0.0'
config-version: 2
profile: 'dbt_imo'
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
models:
dbt_imo:
materialized: incremental
incremental_strategy: delete+insert
test:
schema: testMy DAG code is very simple: with DAG(...):
profile_mapping = (
get_automatic_profile_mapping(conn_id="snowflake"),
)
profile_config = ProfileConfig(
profile_name="dbt_imo",
target_name="prod" if is_prod() else "dev",
profile_mapping=profile_mapping[0],
)
project_config = ProjectConfig(
dbt_project_path=dbt_root,
)
render_config = RenderConfig(
test_behavior=TestBehavior.NONE,
select=["config.tags:test"],
emit_datasets=False,
)
dbt_models = DbtTaskGroup(
project_config=project_config,
profile_config=profile_config,
render_config=render_config,
execution_config=ExecutionConfig(
dbt_executable_path=str(
Path(airflow_home)
/ "dbt_venv"
/ "bin"
/ "dbt"
),
),
) |
|
@tatiana is there anything else I can do to help figure out what this is about? |
|
@ivanstillfront The issue If you update this dependency, things should work: |
Thank you @tatiana , that did resolve the |
|
@ivanstillfront It seems Cosmos could pass variables/envvars to Could you confirm what If we merge this PR: |
|
@tatiana |
|
@ivanstillfront I just logged a ticket in the Openlineage project: In the meantime, we'll merge the PR I mentioned and release it as part of 1.2.1 - so you are no longer blocked. |
) Before this change, Cosmos failed to run if there is an issue in the openlineage-common parsing of the dbt project due to a jinja2 exception, which did not happen when running the dbt command by itself: #612 (comment) ``` File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 268, in calculate_openlineage_events_completes openlineage_processor = DbtLocalArtifactProcessor( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 57, in __init__ dbt_project = self.load_yaml_with_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 157, in load_yaml_with_jinja return self.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 190, in render_values_jinja return environment.from_string(value).render() # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "<template>", line 1, in top-level template code jinja2.exceptions.UndefinedError: 'run_started_at' is undefined ``` Closes: #612 Relates to: OpenLineage/OpenLineage#2212
) Before this change, Cosmos failed to run if there is an issue in the openlineage-common parsing of the dbt project due to a jinja2 exception, which did not happen when running the dbt command by itself: #612 (comment) ``` File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 268, in calculate_openlineage_events_completes openlineage_processor = DbtLocalArtifactProcessor( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 57, in __init__ dbt_project = self.load_yaml_with_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 157, in load_yaml_with_jinja return self.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja parsed_dict[key] = cls.render_values_jinja( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 190, in render_values_jinja return environment.from_string(value).render() # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "<template>", line 1, in top-level template code jinja2.exceptions.UndefinedError: 'run_started_at' is undefined ``` Closes: #612 Relates to: OpenLineage/OpenLineage#2212 (cherry picked from commit 4c65f25)
|
@ivanstillfront The release 1.2.1 was just released: |
|
Same as In dbt profiles.yml, I set env_var and use `as_native'. dbt seems normal, but is getting `jinja2.exceptions.TemplateAssertionError', causing the DAG task to fail. Task log: |
|
@sylashsu thank you very much for reporting this issue! If you're not yet using this yet, could you please try: We can probably improve this by implementing this ticket: #595 |
|
@tatiana Thanks for your answer. I used the My |
|
@sylashsu, thanks for getting back so quickly! The openlineage library will improve to avoid this type of issue: In the meantime, I'll aim to release early next week to ensure Cosmos users are not blocked. |
tatiana
added
priority:high
|
@ivanstillfront we'll be soon releasing a possible fix to most cases with failing renders: OpenLineage/OpenLineage#2221 However, I still don't recognize where OL integration would try to read value with Anyway, I'll be grateful if you could please check if the issue is solved in your case. Thanks for the feedback! |
|
@JDarDagran thank you very much for promptly working on this! This version was just released: |
@JDarDagran I haven't tried yet, but you are correct that |
@tatiana |
All my cosmos-generated tasks are failing after the dbt run because of Jinja "undefined" errors during lineage processing. I have lineage disabled with
OPENLINEAGE_DISABLED=Trueenv var.I am using Astro runtime 9.1.0 with dbt in a virtualenv as recommended. This is my Dockerfile:
My model is very simple:
{{ config(materialized = 'table') }} select try_to_timestamp('{{ run_started_at }}') as run_started_atEverything works fine until the Task starts extracting openlineage data.
Here is the task log:
What am I doing wrong?
The text was updated successfully, but these errors were encountered: