Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Progress log percentage doubled by merge jobs in sql_database #2064

Open
FridayPush opened this issue Nov 14, 2024 · 0 comments
Open

Progress log percentage doubled by merge jobs in sql_database #2064

FridayPush opened this issue Nov 14, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@FridayPush
Copy link

dlt version

dlt 1.3.0

Describe the problem

During a load operation the progress bar generated by logging grows larger than 100%. For example

-------------------- Load sql_database in 1731600286.779139 --------------------
Jobs: 8/18 (44.4%) | Time: 47.06s | Rate: 0.17/s

-------------------- Load sql_database in 1731600286.779139 --------------------
Jobs: 17/18 (94.4%) | Time: 68.30s | Rate: 0.25/s

-------------------- Load sql_database in 1731600286.779139 --------------------
Jobs: 24/18 (133.3%) | Time: 94.54s | Rate: 0.25/s

Expected behavior

The final part of the log above has 24/18 however that should be 12/18`.

Steps to reproduce

This was a use of sql_database to a SQL Server database with a destination of Redshift. Sample code below:

destination = dlt.destinations.redshift(
            staging_dataset_name_layout="z_staging",
            credentials=connection_string,
        )
pipeline = dlt.pipeline(pipeline_name="my_sync", destination=destination, dataset_name=dest_schema, progress=dlt.progress.log(
            log_period=20, log_level=logging.INFO, dump_system_stats=False
        ))

source_1 = sql_database(credentials, reflection_level="full_with_precision")
info = pipeline.run(source_1, write_disposition="merge")

Operating system

Linux, macOS

Runtime environment

Local

Python version

3.11

dlt data source

sql_database

dlt destination

Amazon Redshift

Other deployment details

No response

Additional information

No response

@rudolfix rudolfix added the bug Something isn't working label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: In Progress
Development

No branches or pull requests

3 participants