Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading empty Arrow table with delta table format and replace write disposition does not empty table #2092

Open
jorritsandbrink opened this issue Nov 24, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@jorritsandbrink
Copy link
Collaborator

jorritsandbrink commented Nov 24, 2024

dlt version

1.4.0

Describe the problem

Loading an empty Arrow table with delta table format and replace write disposition does not empty the table. More context here.

Expected behavior

delta table is empty after load.

Steps to reproduce

  1. find tests/load/pipeline/test_filesystem_pipeline.py::test_table_format_schema_evolution[replace-filesystem-parquet-delta-no-staging-_storage] test
  2. remove
        if destination_config.table_format == "delta":
            # TODO: fix https://github.com/dlt-hub/dlt/issues/2092 and remove this if-clause
            expected_num_rows = 2
  1. run the test

Operating system

Linux

Runtime environment

Local

Python version

3.8

dlt data source

No response

dlt destination

Filesystem & buckets

Other deployment details

No response

Additional information

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Todo
Development

No branches or pull requests

1 participant