[Bug] Partial parsing occasionally causes jobs to error out

[sophiad96]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When a customer has “enable partial parsing between deployment runs” enabled in their account settings for their jobs, some jobs will error out with a message that points to a node that had changes made to it recently. However, the job should not be erroring out and the issue is rectified by disabling partial parsing and running the job again (which results in a successful run).

In this specific instance, two jobs in the customer's project errored out, stating that two macros named “dropDataset” were found. Zendesk ticket: https://dbtcloud.zendesk.com/agent/tickets/61977

Similar issues have been reported in another ticket where the removal of a test config from a model caused the job to begin erroring out. Disabling partial parsing resolved the issue: https://dbtcloud.zendesk.com/agent/tickets/60694

Expected Behavior

Because there was only one macro by this name in their project, these jobs should have both succeeded without the user needing to manually disable partial parsing for jobs.

Steps To Reproduce

Have not been able to reproduce yet.

Relevant log output

No response

Environment

- OS: NA
- Python: NA
- dbt: 1.6

Which database adapter are you using with dbt?

No response

Additional Context

Example run: https://dbtlabsmt.datadoghq.com/logs?query=service%3Ascheduler-run%20%40extra.run_id%3A254481367%20&cols=host%2Cservice&fromUser=true&index=%2A&messageDisplay=inline&refresh_mode=paused&stream_sort=desc&viz=stream&from_ts=1708498959395&to_ts=1708531778000&live=false

Compilation error for above run: https://dbtlabsmt.datadoghq.com/logs?query=service%3Ascheduler-run%20%40extra.run_id%3A254481367%20&cols=host%2Cservice&event=AgAAAY3L9FnDppjrPwAAAAAAAAAYAAAAAEFZM0w5R2lBQUFBbUc3MGdPOGxGbndBQwAAACQAAAAAMDE4ZGNiZjUtZGI5OC00MzJkLThiOWUtZDU5MmY1MzY5M2U1&fromUser=true&index=%2A&messageDisplay=inline&refresh_mode=paused&stream_sort=desc&viz=stream&from_ts=1708498959395&to_ts=1708531778000&live=false

Other occurrences: https://dbtlabsmt.datadoghq.com/logs?query=service%3Ascheduler-run%20%22dbt%20found%20two%20macros%22%20&cols=host%2Cservice&fromUser=true&index=%2A&messageDisplay=inline&refresh_mode=sliding&stream_sort=desc&viz=stream&from_ts=1707236315915&to_ts=1708532315915&live=true

[sophiad96]

#4233

[dbeatty10]

Thanks for reporting this @sophiad96 !

The crucial thing will be to come up with a reproducible example so we'll know if it's fixed or not.

Here's the steps that were reported in #8775:

1. Create macros/macros.sql file with 2 macros inside
2. Create macros/macros.yml with documentation for these 2 macros
3. Run dbt compile
4. Change something in macros/macros.yml
5. Run dbt compile again

Do you think something similar would explain what the customer experienced?

If so, could you try tweaking the steps above and see if you can reproduce the error?

[sophiad96]

@dbeatty10 Sorry, I should have provided more clarity here! I think it's important to note that, while in this case the error message that dbt returned did point to the macro, the other instance where I've seen this happen (disabling partial parsing resulted in a successful job run) did not return an error message pertaining to a macro, but rather a message about an alias that had been added to a model config months beforehand. So I feel like while related, this is a different issue. I'm still waiting on confirmation from the customer, but I'm fairly certain they never had two macros by the same name in their project.

[dbeatty10]

@sophiad96 The following reproduction case looks like it gives the same error message that you saw with the customer.

Could you check if they use a YAML file to document their macros? And check if it was changed recently prior to them receiving the error message?

Reprex

Clear out the target directory and make sure there is a macros directory:

rm -rf target
mkdir -p macros

Create a Jinja-SQL file and a YAML file in your macros directory:

cat << EOF > macros/macros.sql
{% macro blue() %}
    {{ log("My favorite color is: blue", true) }}
{% endmacro %}

{% macro green(column_name, scale=2) %}
    {{ log("My favorite color is: green", true) }}
{% endmacro %}
EOF

cat << EOF > macros/macros.yml
macros:
  - name: blue
    description: My favorite color is blue
  - name: green
    description: My favorite color is green
EOF

Parse the project (which should succeed):

dbt parse

Make a change to the YAML file:

cat << EOF > macros/macros.yml
macros:
  - name: blue
    description: My favorite color is blue
  - name: green
    description: My favorite color is (not actually) green
EOF

Try parsing again (which fails, unfortunately):

dbt parse

Output:

19:04:39  Encountered an error:
Compilation Error
  dbt found two macros named "blue" in the project "my_project".
   To fix this error, rename or remove one of the following macros:
      - macros/macros.sql
      - macros/macros.sql

But disabling partial parsing works:

dbt parse --no-partial-parse

[sophiad96]

@dbeatty10 sorry for missing this!! It does look like all of their macros (they have a lot) are documented in YAML. The macro in the example (dropDatasets.sql) was changed right before they experienced this. I've reached out to the customer for details on what changes were made specifically and will follow up when I hear back.

[dbeatty10]

No prob @sophiad96 !

If it ends up being the same thing, then we'll close this particular report as a duplicate of #4233 and #8775.

[sophiad96]

@dbeatty10 unfortunately the customer doe not remember what changes were made to the macro beforehand.

[dbeatty10]

Thanks for trying @sophiad96 !

From the information we have, this looks the same as #4233 (and #8775). So I'm going to close this one as a duplicate.

If we end up determining that it is unique, we'll re-open this one (or open a fresh one).