Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] add not_null_multiple_columns test #418

Merged
merged 21 commits into from
Feb 15, 2024
Merged

[feat] add not_null_multiple_columns test #418

merged 21 commits into from
Feb 15, 2024

Conversation

arthurfg
Copy link
Collaborator

@arthurfg arthurfg commented Feb 8, 2024

basedosdados/pipelines#649
basedosdados/queries-basedosdados#425

Teste que roda para todas as colunas de um determinado modelo e verifica quantas colunas tem uma porcentagem de nulos maior que x%.

Exemplo de uso:

version: 2

models:
  - name: br_ibge_pevs__producao_extracao_vegetal
    description: Quantidade produzida e valor da produção na extração vegetal, por tipo de produto extrativo
    tests:
      - not_null_proportion_multiple_columns:
          threshold: 0.99

Exemplo do output:

Captura de Tela 2024-02-15 às 16 57 25

TODO:

  • Adicionar log's indicando para qual coluna o teste não passou, aparentemente fazer isso não é tão trivial como pensei que fosse.
  • Testar o output apresentado no elementary-data e ver se encaixa com o que a gente pensou dos testes.

@arthurfg arthurfg self-assigned this Feb 8, 2024
@laura-l-amaral laura-l-amaral changed the title [feat[: add not_null_multiple_columns test [feat] add not_null_multiple_columns test Feb 9, 2024
@arthurfg arthurfg merged commit 0def7a2 into main Feb 15, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants