-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Normalize is_iceberg Field for Consistency in Snowflake with QUOTED_IDENTIFIERS_IGNORE_CASE #1229
Normalize is_iceberg Field for Consistency in Snowflake with QUOTED_IDENTIFIERS_IGNORE_CASE #1229
Conversation
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One sanity check, other wise it looks good. The failing CI is due to an expired token that we need to update.
I made a big chance to test things properly.
@VersusFacit, there are a few other scenarios we need to consider as well, like this one: dbt-labs/dbt-core#4422 |
Use a session-only configuration.
Columns that unquoted are capitalized regardless of the quoting ignore flag. So we let it be uppercase and then normalize it down to lowercase in Python.
@@ -147,7 +149,7 @@ | |||
{# -- Gated for performance reason. If you don't want Iceberg, you shouldn't pay the | |||
-- latency penalty. #} | |||
{% if adapter.behavior.enable_iceberg_materializations.no_warn %} | |||
select all_objects.*, is_iceberg as "is_iceberg" | |||
select all_objects.*, is_iceberg |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested this by manually place QUOTED_IDENTIFIERS_IGNORE_CASE = true
and QUOTED_IDENTIFIERS_IGNORE_CASE = false
and then verifying it didn't matter. This is a more predictable way to handle metadata query column names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two comments suggesting docstring updates, but these are non-blocking.
resolves #1227
Problem
In Snowflake, the
QUOTED_IDENTIFIERS_IGNORE_CASE
setting influences column names created by users, such asis_iceberg
. However, this setting does not affect the column values returned by Snowflake’sSHOW OBJECTS
query, creating inconsistency in how theis_iceberg
field is referenced across different parts of the code.When
QUOTED_IDENTIFIERS_IGNORE_CASE
is enabled, theis_iceberg
column may appear in uppercase (IS_ICEBERG
) in our homemadeSHOW OBJECTS
query, causing potential misalignment with the system table's always lowercase column names. This mismatch requires normalization to consistently refer tois_iceberg
as a lowercase field name across Snowflake queries and Python processing, especially when working with adapters that handle Iceberg materializations.Solution
Take 2: Removing quoting from the problem entirely and prove by hand that quoting in this metadata query doesn't matter one way or another.
I did so by hand and verified the test fails before the two fix lines are inserted.
just for fun, I did this with tdd: reproduce the failure/error, add the fix, go green!~~
Checklist