[Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests #11110

michael-domotz · 2024-12-09T16:31:08Z

Is this a new bug in dbt-core?

I believe this is a new bug in dbt-core
I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When running dbt snapshot followed by dbt test (or vice versa), dbt creates both a file and a directory with the same name under the target/run/ path, leading to a “Not a directory” error during the dbt test step. This results in all tests failing.

Expected Behavior

dbt snapshot and dbt test should not create conflicting file and directory names. Both commands should execute successfully without path conflicts, creating a directory containing the executed code.

Steps To Reproduce

In this environment:

OS: macOS
dbt version: 1.9.0-rc2
Adapter: Snowflake

With this folder structure:

/dbt/
   └── snapshots/
       └── _snapshots.yml/

where _snapshots.yml file is as follows:

version: 2

snapshots:
  - name: snapshot_name1
    relation: source('source1', 'table1')
    config:
      alias: registry_user
      unique_key: id
      strategy: check
      check_cols:
        - col1
        - col2
        - ....
    data_tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
            - id
            - _dbt_valid_from

Run the following commands:

dbt snapshot 
dbt test

Relevant log output

16:13:21  Unhandled error while executing target/run/project/snapshots/_snapshots.yml/dbt_utils_unique_combination_o_e57a0aaea3c559916f4dd3135f401dde.sql
[Errno 20] Not a directory: '/path/to/target/run/target/run/project/snapshots/_snapshots.yml/dbt_utils_unique_combination_o_e57a0aaea3c559916f4dd3135f401dde.sql'

Environment

- OS: macOS
- Python: 3.11.9
- dbt: 1.9.0-rc2

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

The text was updated successfully, but these errors were encountered:

dbeatty10 · 2024-12-09T17:55:15Z

Thanks for reporting this @michael-domotz !

I was able to reproduce the issue you reported using dbt=1.9.0-rc2. See "reprex" below for details.

Note: I got this error if I just did dbt build (which runs snapshots + tests in DAG order).

Here's where it happens in the source code:

dbt-core/core/dbt/contracts/graph/nodes.py

Line 277 in 03fdb4c

write_file(full_path, compiled_code)
https://github.com/dbt-labs/dbt-common/blob/965ad815f0dd546678d2595a3010d81f344f8b73/dbt_common/clients/system.py#L268-L269

Reprex

Create these files:

models/my_model.sql

select 1 as id

models/_snapshots.yml

snapshots:
  - name: my_snapshot
    relation: ref('my_model')
    config:
      unique_key: id
      strategy: check
      check_cols: all
    columns:
      - name: id
        data_tests:
          - not_null

Assuming the table associated with my_snapshot doesn't already exist...

Run these commands:

dbt build

Get this output:

$ dbt build    
17:47:43  Running with dbt=1.9.0-rc2
17:47:44  Registered adapter: postgres=1.9.0-rc1
17:47:44  Found 1 model, 1 snapshot, 1 test, 429 macros
17:47:44  
17:47:44  Concurrency: 5 threads (target='postgres')
17:47:44  
17:47:44  1 of 3 START sql view model dbt_dbeatty.my_model ............................... [RUN]
17:47:44  1 of 3 OK created sql view model dbt_dbeatty.my_model .......................... [CREATE VIEW in 0.15s]
17:47:44  2 of 3 START snapshot dbt_dbeatty.my_snapshot .................................. [RUN]
17:47:45  Unhandled error while executing target/run/my_project/models/_snapshots.yml
[Errno 21] Is a directory: '/target/run/my_project/models/_snapshots.yml'
17:47:45  2 of 3 ERROR snapshotting dbt_dbeatty.my_snapshot .............................. [ERROR in 0.27s]
17:47:45  3 of 3 SKIP test not_null_my_snapshot_id ....................................... [SKIP]
17:47:45  
17:47:45  Finished running 1 snapshot, 1 test, 1 view model in 0 hours 0 minutes and 0.75 seconds (0.75s).
17:47:45  
17:47:45  Completed with 1 error, 0 partial successes, and 0 warnings:
17:47:45  
17:47:45    [Errno 21] Is a directory: '/target/run/my_project/models/_snapshots.yml'
17:47:45  
17:47:45  Done. PASS=1 WARN=0 ERROR=1 SKIP=1 TOTAL=3

ttusing · 2024-12-13T01:43:02Z

I ran into a similar error when attempting to use YML-only config snapshots, though it seems maybe not the exact same?

It seems that I cannot put a YML snapshot in a sub-directory, or I get an error like below. I tried removing all tests from the snapshot or just running dbt snapshot, neither worked, so think it is unrelated to tests. Moved it to my base snapshots directory AND removed all tests, and it works fine.

dbt snapshot -s mysnapshot --debug
...
01:33:15  SQL status: SUCCESS 0 in 0.125 seconds
01:33:15  Writing runtime sql for node "snapshot.myproject.mysnapshot"
01:33:15  Unhandled error while executing target/run/snapshots/mysnapshot.yml
[Errno 21] Is a directory: '/myproject/target/run/snapshots/mysnapshot.yml'
01:33:15  Traceback (most recent call last):
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/task/base.py", line 361, in safe_run
    result = self.compile_and_execute(manifest, ctx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/task/base.py", line 308, in compile_and_execute
    result = self.run(ctx.node, manifest)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/task/base.py", line 408, in run
    return self.execute(compiled_node, manifest)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/task/run.py", line 332, in execute
    return self._execute_model(hook_ctx, context_config, model, context, materialization_macro)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/task/run.py", line 292, in _execute_model
    result = MacroGenerator(
             ^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 323, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 29, in macro
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 323, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 223, in macro
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/clients/jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt_common/clients/jinja.py", line 323, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 48, in macro
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/context/providers.py", line 1020, in write
    self.model.write_node(self.config.project_root, self.model.build_path, payload)
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt/contracts/graph/nodes.py", line 277, in write_node
    write_file(full_path, compiled_code)
  File "/[replaced venv path]/lib/python3.11/site-packages/dbt_common/clients/system.py", line 269, in write_file
    with open(path, "w", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/myproject/target/run/snapshots/mysnapshot.yml'

01:33:15  Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'df9517ba-3735-4b2d-9420-bedb51176df7', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x13efccbd0>]}
01:33:15  An error was encountered while trying to send an event
01:33:15  1 of 1 ERROR snapshotting snapshots.mysnapshot ............ [ERROR in 4.57s]

michael-domotz added bug Something isn't working triage labels Dec 9, 2024

dbeatty10 added snapshots Issues related to dbt's snapshot functionality dbt tests Issues related to built-in dbt testing functionality pre-release Bug not yet in a stable release labels Dec 9, 2024

dbeatty10 removed the triage label Dec 9, 2024

dbeatty10 changed the title ~~[Bug] dbt snapshot and dbt test create conflicting file and directory with the same name, causing “Not a directory” errors~~ [Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests Dec 9, 2024

graciegoheen added backport 1.9.latest and removed pre-release Bug not yet in a stable release labels Dec 11, 2024

gshank self-assigned this Dec 16, 2024

gshank mentioned this issue Dec 16, 2024

Fix yaml snapshot specification with data tests #11156

Merged

5 tasks

gshank closed this as completed in #11156 Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests #11110

[Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests #11110

michael-domotz commented Dec 9, 2024 •

edited by dbeatty10

Loading

dbeatty10 commented Dec 9, 2024

ttusing commented Dec 13, 2024

[Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests #11110

[Bug] Conflicting file and directory names cause “Not a directory” errors when a YAML snapshot has data tests #11110

Comments

michael-domotz commented Dec 9, 2024 • edited by dbeatty10 Loading

Is this a new bug in dbt-core?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Environment

Which database adapter are you using with dbt?

Additional Context

dbeatty10 commented Dec 9, 2024

Reprex

ttusing commented Dec 13, 2024

michael-domotz commented Dec 9, 2024 •

edited by dbeatty10

Loading