@francescomucio Thanks for opening! Context from Slack thread here.

It looks like bigint and long are aliases for one another in Databricks SQL: https://docs.databricks.com/sql/language-manual/data-types/bigint-type.html

There's a specific fix we can make here: Add a mapping between those types within SparkColumn, for use in the translate_type method here:

On other databases, we use that translate_type method to handle type aliasing. Check out the default implementation in dbt-core here.

the fix

I think this might be as simple as:

• removing the translate_type override in dbt-spark, to use the default implementation instead
• reimplement the type label mapping, to support just long → bigint:

class SparkColumn(dbtClassMixin, Column):
    TYPE_LABELS: ClassVar[Dict[str, str]] = {
        "LONG": "BIGINT",
    }

Plus a test case, for exactly the experience you're having. We have existing integration tests for on_schema_change functionality in tests/integration/incremental_on_schema_change.

Is that a fix you'd be interested in contributing?

larger context

In v1.2, we are looking to move away from using show table extended for top-of-run caching, and use show tables + show views instead (much much faster: #342). That would then mean we always use describe table to power adapter.get_columns_in_relation, with consistent results.

I can contribute with a quick fix, let me try :)

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

Please reopen @jtcohen6 @leahwicz

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.