Skip to content

Commit

Permalink
feat: reduce chunksize to decode
Browse files Browse the repository at this point in the history
  • Loading branch information
folhesgabriel committed Sep 11, 2024
1 parent 5ce6b75 commit cbc58d6
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pipelines/utils/crawler_datasus/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ def is_empty(lista):


@task
def read_dbf_save_parquet_chunks(file_list: list, table_id: str, dataset_id:str= "br_ms_sia", chunk_size : int = 300000) -> str:
def read_dbf_save_parquet_chunks(file_list: list, table_id: str, dataset_id:str= "br_ms_sia", chunk_size : int = 100000) -> str:
"""
Convert dbc to parquet
"""
Expand Down

0 comments on commit cbc58d6

Please sign in to comment.