Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] br_me_cnpj #905

Merged
merged 5 commits into from
Dec 5, 2024
Merged

[Refactor] br_me_cnpj #905

merged 5 commits into from
Dec 5, 2024

Conversation

folhesgabriel
Copy link
Contributor

@folhesgabriel folhesgabriel commented Dec 3, 2024

Descrição do PR:

  • Melhora logs do crawler assíncrono e modifica parâmetros de download das funções async

Detalhes Técnicos:

  • Melhoria de logs para (1) monitorar o processo de donwload paralelo dos chunks de um mesmo arquivo e (2) esclarecer o que o código executado está fazendo.
    image

  • Reduzir tamanho de cada chunk (de 20mb para 15mb) e aumentar o número de chunks baixados por vez (de 5 para 15)

Teste e Validações:

  • Relate os testes e validações relacionado aos dados/script:
    • Testado localmente
    • Testado na Cloud

Copy link

github-actions bot commented Dec 3, 2024

Coverage

Tests Skipped Failures Errors Time
8 7 💤 0 ❌ 0 🔥 13.467s ⏱️

@folhesgabriel folhesgabriel requested a review from a team December 3, 2024 16:07
@folhesgabriel folhesgabriel marked this pull request as draft December 3, 2024 16:07
@folhesgabriel folhesgabriel self-assigned this Dec 3, 2024
@folhesgabriel folhesgabriel linked an issue Dec 3, 2024 that may be closed by this pull request
2 tasks
@folhesgabriel folhesgabriel changed the title refactor: improve logs, increase parallel downloads, reduce chunk size [Refactor] br_me_cnpj Dec 3, 2024
@folhesgabriel folhesgabriel marked this pull request as ready for review December 4, 2024 18:31
@folhesgabriel folhesgabriel merged commit 61c0812 into main Dec 5, 2024
7 checks passed
@folhesgabriel folhesgabriel deleted the staging/fix_cnpj_crawler branch December 5, 2024 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bugfix] br_me_cnpj
2 participants