Skip to content

Commit

Permalink
[Bug] Fix incremental python tables - dbt can't find temporary table …
Browse files Browse the repository at this point in the history
…transaction logs (#1042)

* unskip test
* align test pyodbc version with build deps; add pyspark from build deps
* revert change to exclude database and schema from temp relations
* retain existing behavior for unit tests by vendoring temp relation into the incremental materialization
* changelog entry
  • Loading branch information
mikealfare authored May 14, 2024
1 parent 94af50e commit 629ea94
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 9 deletions.
7 changes: 7 additions & 0 deletions .changes/unreleased/Fixes-20240513-160121.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Fixes
body: Fix incremental python models error where Databricks could not find the temp
table transaction logs
time: 2024-05-13T16:01:21.255833-04:00
custom:
Author: mikealfare
Issue: "1033"
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
{%- set incremental_predicates = config.get('predicates', none) or config.get('incremental_predicates', none) -%}
{%- set target_relation = this -%}
{%- set existing_relation = load_relation(this) -%}
{%- set tmp_relation = make_temp_relation(this) -%}
{% set tmp_relation = this.incorporate(path = {"identifier": this.identifier ~ '__dbt_tmp'}) -%}

{#-- for SQL model we will create temp view that doesn't have database and schema --#}
{%- if language == 'sql'-%}
Expand Down
3 changes: 2 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
pyhive[hive_pure_sasl]~=0.7.0
requests>=2.28.1

pyodbc~=5.0.1 --no-binary pyodbc
pyodbc~=4.0.39 --no-binary pyodbc
sqlparams>=3.0.0
thrift>=0.13.0
pyspark>=3.0.0,<4.0.0
sqlparse>=0.4.2 # not directly required, pinned by Snyk to avoid a vulnerability

types-PyYAML
Expand Down
7 changes: 0 additions & 7 deletions tests/functional/adapter/test_python_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,6 @@ class TestPythonIncrementalModelSpark(BasePythonIncrementalTests):
def project_config_update(self):
return {}

@pytest.mark.skip(
"Databricks can't find the transaction log"
"See https://github.com/dbt-labs/dbt-spark/issues/1033"
)
def test_incremental(self, project):
super().test_incremental(project)


models__simple_python_model = """
import pandas
Expand Down

0 comments on commit 629ea94

Please sign in to comment.