Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stage all external sources related to a source_name broken #327

Closed
1 of 3 tasks
bullamiders opened this issue Nov 29, 2024 · 8 comments · Fixed by #330
Closed
1 of 3 tasks

stage all external sources related to a source_name broken #327

bullamiders opened this issue Nov 29, 2024 · 8 comments · Fixed by #330
Labels
bug Something isn't working triage

Comments

@bullamiders
Copy link

Describe the bug

In Version 0.10.0 it is not possible anymore to stage all external sources related to a source_name.

In my understanding:

  1. node.name references a table in a source while node.source_name references the whole source
  2. a single table of a source is handled here
  3. a whole source is handled here

The change introduced in the issue #309 and the related commit are probably the source, because the code part that handles a whole source is restricted to a single table now.

Steps to reproduce

# stage all <source_name> external sources
$ dbt run-operation stage_external_sources --args "select: <source_name>"

Expected results

1 of X START external source <schema>.<table>
....

Actual results

No external sources selected

Screenshots and log output

System information

The contents of your packages.yml file:

packages:
  - package: dbt-labs/dbt_utils
    version: 1.3.0
  - package: dbt-labs/codegen
    version: 0.12.1
  - package: dbt-labs/dbt_external_tables
    version: 0.10.0

Which database are you using dbt with?

  • redshift
  • snowflake
  • other (specify: ____________)

I only tested Redshift, but since the change was in macros/common/stage_external_sources.sql it probably affects all databases.

The output of dbt --version:

Core:
  - installed: 1.8.8
  - latest:    1.8.9 - Update available!

Plugins:
  - postgres: 1.8.2 - Up to date!
  - redshift: 1.8.1 - Up to date!

The operating system you're using:
Windows 11 Enterprise

The output of python --version:
Python 3.10.15

Additional context

@bullamiders bullamiders added bug Something isn't working triage labels Nov 29, 2024
@kyleburke-meq
Copy link
Contributor

can confirm I am having the same issue on 0.10.0

@mikkosulonen
Copy link

Also happening on Snowflake:

packages:
  - package: dbt-labs/dbt_external_tables
    version: 0.10.0
  - package: dbt-labs/codegen
    version: 0.13.1
  - package: dbt-labs/dbt_utils
    version: 1.3.0

dbt Cloud latest

@stegus64
Copy link

Same issue on databricks

@dataders
Copy link
Collaborator

yikes -- sorry everyone! my instinct is that #312's one-line change is to blame.

@stegus64 @mikkosulonen @kyleburke-meq @bullamiders could one of you please try adding the stage_external_sources() macro to your local project's macros/ dir with line 20 reverted to how it was in 0.9.x

this

-- this
{% if src == node.source_name %}
-- not this
{% if src == node.name %}

as soon as someone can validate that this resolves the issue, I can yank the release and cut a new version. cheers!

@bullamiders
Copy link
Author

I'm on vacation therefore I cannot valdiate it right now, but this was the only change to the macro in release 0.10.0.

@stegus64
Copy link

I can verify that undoing this change fixes the issue.

@laurianneol
Copy link

This will resolve the issue. I had the same problem hier.

@dataders
Copy link
Collaborator

FYI I've reverted the PR and released the change at 0.10.1. Thanks for your patience let me know if you still see the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
6 participants