Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAP-870] [Bug] Adapter seems to malformat kill queries when using --fail-fast dbt-redshift 1.5.8 #593

Closed
2 tasks done
lsabreu96 opened this issue Sep 1, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@lsabreu96
Copy link

lsabreu96 commented Sep 1, 2023

Is this a new bug in dbt-redshift?

  • I believe this is a new bug in dbt-redshift
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Whenever running dbt tests or run statements (didn't check for other commands, but given the adapter I believe to be the same issue) with the --fail-fast flag, dbt seems to try to kill any early failing tests with pg_termiante_backend method, but it's malformating the query using [], as can be seen in the logs.
According to Redshift docs, pg_terminate_backend expects a integer.

Expected Behavior

No errors when running in --fail-fast

Steps To Reproduce

--> Sample model
teste.sql

{{
    config(
        materialized='table',
    )
}}


SELECT 1 as primary_key
      ,'foo' as column_1

UNION ALL

SELECT 1 as primary_key
      ,'foo' as column_1

UNION ALL

SELECT NULL as primary_key
      ,'bar' as column_1    

--> sample yaml
teste.yml


models:
    - name: teste
      columns:
          - name: primary_key
            tests:
                - not_null
                - unique

--> Execute
dbt test --select teste --fail-fast

Relevant log output

�[0m12:09:22.164850 [debug] [Thread-1  ]: Using redshift connection "test.loggi_warehouse.not_null_teste_primary_key.b3f47b1cc4"
�[0m12:09:22.178688 [debug] [Thread-1  ]: On test.loggi_warehouse.not_null_teste_primary_key.b3f47b1cc4: select
      count(*) as failures,
      count(*) != 0 as should_warn,
      count(*) != 0 as should_error
    from (
      
    
    



select primary_key
from "prod"."dbt_labreu"."teste"
where primary_key is null



      
    ) dbt_internal_test
�[0m12:09:23.895653 [debug] [Thread-1  ]: SQL status: SUCCESS in 2.0 seconds
�[0m12:09:23.902557 [debug] [Thread-1  ]: On test.loggi_warehouse.not_null_teste_primary_key.b3f47b1cc4: ROLLBACK
�[0m12:09:24.023291 [debug] [Thread-2  ]: SQL status: SUCCESS in 2.0 seconds
�[0m12:09:24.033371 [debug] [Thread-2  ]: On test.loggi_warehouse.unique_teste_primary_key.3d2e9e109c: ROLLBACK
�[0m12:09:24.458887 [debug] [Thread-1  ]: On test.loggi_warehouse.not_null_teste_primary_key.b3f47b1cc4: Close
�[0m12:09:24.463183 [error] [Thread-1  ]: 1 of 2 FAIL 1 not_null_teste_primary_key ....................................... [�[31mFAIL 1�[0m in 4.86s]
�[0m12:09:24.466465 [debug] [Thread-1  ]: Finished running node test.loggi_warehouse.not_null_teste_primary_key.b3f47b1cc4
�[0m12:09:24.475994 [debug] [MainThread]: Using redshift connection "master"
�[0m12:09:24.479689 [debug] [MainThread]: On master: BEGIN
�[0m12:09:24.483133 [debug] [MainThread]: Opening a new connection, currently in state closed
�[0m12:09:24.487899 [debug] [MainThread]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
�[0m12:09:24.491494 [debug] [MainThread]: Redshift adapter: Connecting to redshift with username/password based auth...
�[0m12:09:24.588848 [debug] [Thread-2  ]: On test.loggi_warehouse.unique_teste_primary_key.3d2e9e109c: Close
�[0m12:09:25.954990 [debug] [MainThread]: SQL status: SUCCESS in 1.0 seconds
�[0m12:09:25.956538 [debug] [MainThread]: Using redshift connection "master"
�[0m12:09:25.957992 [debug] [MainThread]: On master: select pg_backend_pid()
�[0m12:09:26.414642 [debug] [MainThread]: SQL status: SUCCESS in 0.0 seconds
�[0m12:09:26.416567 [debug] [MainThread]: Using redshift connection "master"
�[0m12:09:26.417985 [debug] [MainThread]: On master: select pg_terminate_backend([1073957580])

Environment

- OS:Ubuntu 20.04
- Python:3.8.10
- dbt-core:1.5.4
- dbt-redshift:1.5.8

Additional Context

No response

@lsabreu96 lsabreu96 added bug Something isn't working triage labels Sep 1, 2023
@github-actions github-actions bot changed the title [Bug] Adapter seems to malformat kill queries when using --fail-fast dbt-redshift 1.5.8 [ADAP-870] [Bug] Adapter seems to malformat kill queries when using --fail-fast dbt-redshift 1.5.8 Sep 1, 2023
@lsabreu96
Copy link
Author

If I've read the code right, the problem seems to be coming from this step.
I wasn't able to find which lib is being used to connect to the database, assuming it's psycopg2, the fetchone() method being used returns a tuple, so maybe checking if there are results and then electing the first element of the list would suffice

@dbeatty10
Copy link
Contributor

Thanks for raising this issue @lsabreu96 ! And thank you for researching and identifying the source of issue as well 🧠

This sounds the same as #553 to me, so I'm going to close this one as a duplicate.

(Good news: #553 may be fixed by #568)

@dbeatty10 dbeatty10 closed this as not planned Won't fix, can't repro, duplicate, stale Sep 5, 2023
@dbeatty10 dbeatty10 added duplicate and removed triage labels Sep 5, 2023
@lsabreu96
Copy link
Author

Hi @dbeatty10 , glad to hear
has anyone reported connection instability when moving to this new library ?
We're experiencing connection reset by peer and I was wondering if some internal process is killing its own session

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants