Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dq analytics level #622

Merged
merged 38 commits into from
Oct 21, 2024
Merged

Dq analytics level #622

merged 38 commits into from
Oct 21, 2024

Conversation

bradmontierth
Copy link
Contributor

Describe your changes

Please include a summary of any changes.
adding analytics data quality checks

How has this been tested?

Please describe the tests you ran to verify your changes. Provide instructions or code to reproduce output.
ran on real data and viewed output of union tables
ci testing

Reviewer focus

Please summarize the specific items you’d like the reviewer(s) to look into.

Checklist before requesting a review

  • I have added at least one Github label to this PR (bug, enhancement, breaking change,...)
  • My code follows style guidelines
  • (New models) YAML files are categorized by sub folder and models listed in alphabetical order
  • (New models) I have added a config to each new model to enable it for claims and/or clinical data
  • (New models) I have added the variable tuva_last_run to the final output
  • (Optional) I have recorded a Loom to explain this PR

Package release checklist

  • I have updated dbt docs
  • I have updated the version number in the dbt_project.yml

(Optional) Gif of how this PR makes you feel

Loom link

tom-tuva and others added 30 commits October 14, 2024 15:59
@sarah-tuva sarah-tuva merged commit 264512d into main Oct 21, 2024
10 checks passed
@sarah-tuva sarah-tuva deleted the dq_analytics_level branch October 21, 2024 18:39
bradmontierth added a commit that referenced this pull request Oct 22, 2024
commit 4ec9e4b
Merge: b56bef4 055f119
Author: bradmontierth <[email protected]>
Date:   Tue Oct 22 09:12:50 2024 -0600

    Merge pull request #626 from tuva-health/gitignore_updates

    updating gitignore for .envrc

commit 055f119
Author: Tom Sherwood <[email protected]>
Date:   Tue Oct 22 09:33:56 2024 -0400

    updating gitignore for .envrc

commit b56bef4
Merge: 560bee9 6c7eb36
Author: bradmontierth <[email protected]>
Date:   Mon Oct 21 15:45:52 2024 -0600

    Merge pull request #625 from tuva-health/patch-0.12.1

    Patch v0.12.1

commit 6c7eb36
Author: sarah-tuva <[email protected]>
Date:   Mon Oct 21 14:57:14 2024 -0600

    update dbt docs

commit 560bee9
Author: Forrest Beckwith <[email protected]>
Date:   Mon Oct 21 16:16:16 2024 -0400

    fixed schema typo (#624)

    fix schema typo

    fixed normalize__unmapped_observation schema suffix

commit 264512d
Author: bradmontierth <[email protected]>
Date:   Mon Oct 21 12:39:06 2024 -0600

    Dq analytics level (#622)

    * Chronic condition prevalence check

    * Switching total_patients to use member_months

    * Updating final sql to get reference values

    * adding readmission checks

    * creating 2 union output tables for analytics checks

    * Chronic condition check updates

    * adding cms hcc checks

    * Finishing chronic condition checks

    * adding concept and measure to dq tables

    * Encounters start

    * adding inpatient checks

    * Updated encounters

    * adding inpatient drg

    * encounter pkpy and paid per trend

    * adding ed classification

    * service categories pmpm and trend

    * updates to chronic conditions and reference data

    * aligning column names between models for union

    * fix ci testing syntax issues

    * fixing syntax issues

    * fixing data type for bigquery

    * fixing chronic conditions and adding financial pmpm

    * fixing union all

    * fixing data type for big query

    * moving seed to s3

    ---------

    Co-authored-by: Tom Sherwood <[email protected]>

commit 34fc393
Author: thutuva <[email protected]>
Date:   Thu Oct 17 13:25:06 2024 -0400

    Ci outside contributor fixes (#621)

    * added step to check out pr code to run through CI

    * troubleshooting

    * reverting ref syntax

    * edited BQ

    * save spot

    * added macro for limit 0 to fix fabric bug; added prov attribution testing;

    * added macro for limit 0 to fix fabric bug; added prov attribution testing;

    * added missing comma

    * fixed dependencies

    * added checkout step to BQ

    * upped threads in profiles.yml

    * updated job flow/dependencies

    * updated job flow/dependencies

    * set default in integration_tests dbt_project.yml; added testing of provider_attribution_enabled var only

    * fixed dupe job name

    * fixed job name

    * edited vars being executed in each job

commit 11b92b8
Author: thutuva <[email protected]>
Date:   Tue Oct 15 16:01:43 2024 -0400

    Refactor CI testing (#615)

    * removed old CI workflows; testing snowflake workflow

    * resolving conflicts

    * workflow edit

    * testing

    * edited Slack message

    * edited Slack message

    * updated all workflows to: latest dbt, post slack message, add fabric

    * fixed snowflake to run serial; code to parse BQ json creds

    * fixed snowflake to run serial; code to parse BQ json creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * edit BQ creds

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * commented out debugging steps; uncommented dbt build

    * testing

    * testing

    * testing

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * archived old workflows; updated outside contributor with fabric; new workflow to consolidate old ones

    * archived old workflows; updated outside contributor with fabric; new workflow to consolidate old ones

    * fixed formatting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * troubleshooting

    * uncommented out build for production

    * fixing data type bug

commit faa38f9
Author: bradmontierth <[email protected]>
Date:   Tue Oct 15 13:32:53 2024 -0600

    Align data quality column and table names with docs (#616)

    * align column and table names with docs

    * fix redshift syntax

    * update redshift syntax

commit 0f498b3
Merge: 9196539 e44412c
Author: eldon-tuva <[email protected]>
Date:   Fri Oct 11 18:21:07 2024 -0600

    Merge pull request #614 from tuva-health/patch_0.12.0

    branch for new release

commit e44412c
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 11 17:56:00 2024 -0600

    branch for new release

commit 9196539
Merge: c81b536 7b61736
Author: eldon-tuva <[email protected]>
Date:   Fri Oct 11 16:52:48 2024 -0600

    Merge pull request #607 from tuva-health/provider_attribution

    provider attribution input layer table added

commit 7b61736
Merge: 408446a c81b536
Author: eldon-tuva <[email protected]>
Date:   Fri Oct 11 16:27:54 2024 -0600

    Merge branch 'main' into provider_attribution

commit c81b536
Author: bradmontierth <[email protected]>
Date:   Fri Oct 11 15:10:32 2024 -0600

    Data Quality Checks (#613)

    * Adding initial eligibility dq checks

    * adding pharmacy and medical claim checks

    * added demographics query and edited model name

    * added demographics query and edited model name

    * renaming models per naming convention

    * edited models to count distinct eligibility spans; updated YAML; added new models

    * edited model name and table to match design; added data loss query

    * edited data_loss to add counts for eligibility spans and member months

    * moving dqi to own folder and data quality formatting

    * revert dbt_project back to default

    * changing tag for new models

    * removed data loss model

    * changed try_cast

    * fixed getdate() syntax

    * fix syntax on inst header fields

    * adding pharmacy and medical claim checks

    * renaming models per naming convention

    * moving dqi to own folder and data quality formatting

    * revert dbt_project back to default

    * changing tag for new models

    * fix syntax on inst header fields

    * fix big query data type error and add eligibility to data loss

    * moved dq audit script to final folder; edited getdate syntax; update YAML to include new models

    * moved dq audit script to final folder; edited getdate syntax; update YAML to include new models

    * fix syntax issue on data loss check

    * adding tables to test summary table

    * fixing redshift syntax

    * added syntax for redshift

    * testing redshift fix

    * fixing syntax for fabric

    ---------

    Co-authored-by: Tom Sherwood <[email protected]>
    Co-authored-by: thutuva <[email protected]>
    Co-authored-by: tuvaforrest <[email protected]>

commit b8c8119
Author: bradmontierth <[email protected]>
Date:   Fri Oct 11 06:34:29 2024 -0600

    Fix missing clinical paid amounts  (#610)

    * fix encounter flag bug

    * fix clinical encounter paid amount bug

    * change rehab to rehabilitation

commit 864ea4e
Author: BijayaShres <[email protected]>
Date:   Fri Oct 11 01:16:11 2024 +0545

    Quality Measure: Medication Adherence for Hypertension (ADH-RAS) (#597)

    * feat: performance period  implementation

    * chore: denom case1 implementation

    * feat: denominator implemented

    * feat: adh-ras exclusion model implementation

    * fix: performance period reviews and comments fixed

    * fix: denominator comments applied

    * fix: minor issues denominator solved

    * fix: adhras denominator fixed

    * fix: denominator fixed

    * fix:denominator death patient data fixed

    * chore: final and yml modified

    * feat: adh_ras features added

    * Update models/quality_measures/intermediate/adh-ras_medication_adherence_for_hypertension/quality_measures__int_adhras_long.sql

    Co-authored-by: Utsav Paudel <[email protected]>

    * Update models/quality_measures/intermediate/adh-ras_medication_adherence_for_hypertension/quality_measures__int_adhras_numerator.sql

    Co-authored-by: Utsav Paudel <[email protected]>

    * fix: minor issues fixed

    * fix: cte name fixed in denominator

    * fix: cte with row_number updated in denominator

    * fix: numerator row number naming fixed

    * fix: bigquery typecast bug

    * fix: denominator cross join result fix

    * fix: naming in denominator and pdc logic in numerator updated

    * fix: treatment_period cte name changed in denominator

    * fix: redshift supportable feature added

    ---------

    Co-authored-by: Utsav Paudel <[email protected]>

commit 0b471b0
Merge: 22b7a7d f9e8959
Author: bradmontierth <[email protected]>
Date:   Thu Oct 10 13:11:22 2024 -0600

    Merge pull request #605 from tuva-health/data_quality_feature_604

    added new data quality models

commit f9e8959
Author: thutuva <[email protected]>
Date:   Thu Oct 10 09:13:02 2024 -0400

    Update dbt_project.yml

commit 408446a
Author: Coco Zuloaga <[email protected]>
Date:   Wed Oct 9 17:24:56 2024 -0600

    Update dbt_project.yml

commit 3c1ca54
Author: Coco Zuloaga <[email protected]>
Date:   Wed Oct 9 13:56:23 2024 -0600

    Update _integration_tests__models.yml

commit 2ab99dd
Author: thutuva <[email protected]>
Date:   Wed Oct 9 15:52:41 2024 -0400

    fixed alias

commit 83da5ee
Author: thutuva <[email protected]>
Date:   Wed Oct 9 15:30:06 2024 -0400

    added alias to fix fabric error; changed column name to result_count; update YAML with descriptions and reorganized

commit 321c687
Author: thutuva <[email protected]>
Date:   Wed Oct 9 14:33:32 2024 -0400

    fix ref syntax

commit c565edd
Merge: dcf756c 3c8f5e1
Author: thutuva <[email protected]>
Date:   Wed Oct 9 14:28:39 2024 -0400

    Merge remote-tracking branch 'origin/data_quality_feature_604' into data_quality_feature_604

commit dcf756c
Merge: e1d8c43 22b7a7d
Author: thutuva <[email protected]>
Date:   Wed Oct 9 14:28:23 2024 -0400

    Merge branch 'main' of https://github.com/tuva-health/tuva into data_quality_feature_604

commit a7ce2ae
Merge: 809facf 22b7a7d
Author: tuvaforrest <[email protected]>
Date:   Wed Oct 9 11:57:24 2024 -0400

    Merge branch 'main' of https://github.com/tuva-health/tuva into provider_attribution

commit 809facf
Author: tuvaforrest <[email protected]>
Date:   Wed Oct 9 11:55:18 2024 -0400

    changed seed to point to versioned synthetic version

commit 3c8f5e1
Merge: e1d8c43 22b7a7d
Author: thutuva <[email protected]>
Date:   Wed Oct 9 11:50:36 2024 -0400

    Merge branch 'main' into data_quality_feature_604

commit e1d8c43
Author: thutuva <[email protected]>
Date:   Wed Oct 9 11:45:00 2024 -0400

    updated queries to pull from input_layer

commit 22b7a7d
Author: Utsav Paudel <[email protected]>
Date:   Wed Oct 9 00:59:53 2024 +0545

    New Quality Measure: Medication Adherence for Diabetes (#603)

    * feat: add days_supply field in pharmacy_claims staging

    * feat: performance period model

    * fix: days_supply in pharmacy claim for fabric

    * feat: qm adh diabetes intermediate models

    * fix: numerator cte reference

    * fix: pdc rounding bug fix

    * fix: cte names change

    * feat: final tables update

    * docs: adh_diabetes measure documentation

    * fix: aesthetic fixes

    * fix: rolling sum redshift syntax

commit f0eb954
Author: thutuva <[email protected]>
Date:   Mon Oct 7 11:11:36 2024 -0400

    syntax changes for BQ

commit 287d0cd
Author: tuvaforrest <[email protected]>
Date:   Mon Oct 7 09:36:55 2024 -0400

    removed bad config

commit 7bf4190
Author: thutuva <[email protected]>
Date:   Mon Oct 7 09:32:38 2024 -0400

    reverting profile

commit 3bd8aa2
Merge: f198a6f f854740
Author: thutuva <[email protected]>
Date:   Mon Oct 7 09:32:07 2024 -0400

    Merge branch 'main' of https://github.com/tuva-health/tuva into data_quality_feature_604

commit f198a6f
Author: thutuva <[email protected]>
Date:   Mon Oct 7 09:30:22 2024 -0400

    syntax changes for fabric

commit 7a03a73
Author: thutuva <[email protected]>
Date:   Mon Oct 7 09:30:21 2024 -0400

    syntax changes for fabric

commit e1389cd
Author: Forrest Beckwith <[email protected]>
Date:   Fri Oct 4 17:08:08 2024 -0400

    removed provider attribution ~ file

commit 08ceb11
Author: tuvaforrest <[email protected]>
Date:   Fri Oct 4 16:58:48 2024 -0400

    added seed yml, changed default for provider_attribution_enabled in integration_tests

commit 142b306
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 14:19:25 2024 -0600

    fix provider attribution in ci testing

commit 1ff52b4
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 14:12:45 2024 -0600

    fix redshift yml

commit b16d726
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 14:08:11 2024 -0600

    changes to medical claim in ci testing

commit 57423cb
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 14:04:37 2024 -0600

    fix ci testing model

commit 91a9cbf
Merge: 32a8883 87773df
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:58:43 2024 -0600

    Merge branch 'provider_attribution' of https://github.com/tuva-health/tuva into provider_attribution

commit 32a8883
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:55:31 2024 -0600

    add ci testing model

commit 87773df
Merge: aee4e26 f854740
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:48:36 2024 -0600

    Merge branch 'main' into provider_attribution

commit aee4e26
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:40:03 2024 -0600

    remove unnecessary files

commit f01a97b
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:39:30 2024 -0600

    changes to pmpm prep

commit a5b962b
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 13:16:44 2024 -0600

    added columns to financial_pmpm__pmpm_prep

commit 18351d0
Author: Coco Zuloaga <[email protected]>
Date:   Fri Oct 4 12:33:26 2024 -0600

    provider attribution input layer table added

commit f854740
Author: tom-tuva <[email protected]>
Date:   Fri Oct 4 13:32:27 2024 -0400

    Doc blocks fix (#606)

    * Moving column_descriptions.md to its own folder.

    * Updating docs for doc blocks.

commit 1a04b3a
Merge: 8d9e63b fa273ca
Author: thutuva <[email protected]>
Date:   Fri Oct 4 11:03:28 2024 -0400

    Merge branch 'main' into data_quality_feature_604

commit 8d9e63b
Author: thutuva <[email protected]>
Date:   Fri Oct 4 10:34:42 2024 -0400

    edited query for BQ

commit fa273ca
Author: bradmontierth <[email protected]>
Date:   Fri Oct 4 08:17:03 2024 -0600

    adding service category 3 to medical_claim (#602)

commit 1cf562a
Author: thutuva <[email protected]>
Date:   Fri Oct 4 08:34:59 2024 -0400

    added tuva_last_run

commit 7ba7bea
Author: thutuva <[email protected]>
Date:   Fri Oct 4 08:30:15 2024 -0400

    added new data quality models

commit e27e9da
Author: Addy Naik <[email protected]>
Date:   Thu Oct 3 15:35:45 2024 -0400

    fix schemas template for claims preprocessing (#598)

    fix schemas

commit 72d8db0
Merge: 6253700 ffbf02e
Author: eldon-tuva <[email protected]>
Date:   Wed Oct 2 15:31:12 2024 -0600

    Merge pull request #599 from tuva-health/doc_blocks

    Doc blocks

commit ffbf02e
Merge: 9207317 6253700
Author: tom-tuva <[email protected]>
Date:   Wed Oct 2 16:23:11 2024 -0400

    Merge branch 'main' into doc_blocks

commit 6253700
Author: eldon-tuva <[email protected]>
Date:   Wed Oct 2 13:46:42 2024 -0600

    Update access key (#600)

commit 9207317
Merge: ebb4522 d85099e
Author: Tom Sherwood <[email protected]>
Date:   Wed Oct 2 12:44:08 2024 -0400

    Merge branch 'doc_blocks' of https://github.com/tuva-health/tuva into doc_blocks

commit ebb4522
Author: Tom Sherwood <[email protected]>
Date:   Wed Oct 2 12:43:50 2024 -0400

    Removing unique tests on encounter_id.

commit d85099e
Merge: e22ee63 c48bad4
Author: tom-tuva <[email protected]>
Date:   Wed Oct 2 12:00:56 2024 -0400

    Merge branch 'main' into doc_blocks

commit e22ee63
Author: Tom Sherwood <[email protected]>
Date:   Wed Oct 2 11:34:41 2024 -0400

    Adding remaining doc block column definitions

commit c48bad4
Author: mgoldhirsh <[email protected]>
Date:   Wed Oct 2 11:16:36 2024 -0400

    noticed a bug with institutional procedures, this fixes it (#596)

commit caf82b1
Author: Tom Sherwood <[email protected]>
Date:   Wed Oct 2 10:02:26 2024 -0400

    Adding doc blocks for column definitions in encounter models

commit e4908b7
Author: Addy Naik <[email protected]>
Date:   Tue Oct 1 13:06:29 2024 -0400

    fix the space in the generated schema name template (#594)

    Update pharmacy_models.yml

commit 2563687
Author: tom-tuva <[email protected]>
Date:   Fri Sep 27 18:49:01 2024 -0400

    Release for version 0.11.0 (#595)

commit 9873562
Merge: af81e97 3a52e89
Author: Coco Zuloaga <[email protected]>
Date:   Tue Sep 24 15:56:42 2024 -0600

    Merge pull request #592 from tuva-health/add_schema_extract_script

    added helper scripts folder

commit af81e97
Author: bradmontierth <[email protected]>
Date:   Tue Sep 24 14:27:29 2024 -0600

    Encounters and Service Categories (#585)

    Updating service categories and adding new encounter types.

commit 3a52e89
Author: tuvaforrest <[email protected]>
Date:   Tue Sep 24 16:15:09 2024 -0400

    created helper scripts folder, added script to extract tuva project schema and readme.

commit 85f9fd2
Merge: 4e055eb 2a5dd60
Author: Coco Zuloaga <[email protected]>
Date:   Thu Sep 19 10:21:14 2024 -0600

    Merge pull request #590 from tuva-health/patch_0.10.8

    patch 0.10.8

commit 2a5dd60
Author: thutuva <[email protected]>
Date:   Thu Sep 19 10:39:56 2024 -0400

    updated docs

commit a9b8873
Author: thutuva <[email protected]>
Date:   Thu Sep 19 09:38:59 2024 -0400

    updated dbt_project.yml

commit 4e055eb
Merge: b2fb6f3 0c93b70
Author: Coco Zuloaga <[email protected]>
Date:   Wed Sep 18 12:09:24 2024 -0600

    Merge pull request #589 from tuva-health/ed_encounter_age_bug_fix

    removed column admit_age

commit 0c93b70
Author: thutuva <[email protected]>
Date:   Wed Sep 18 13:32:01 2024 -0400

    removed column admit_age

commit b2fb6f3
Author: Utsav Paudel <[email protected]>
Date:   Tue Sep 17 03:18:07 2024 +0545

    New Quality Measure NQF0420: Pain Assessment and Follow-Up (#586)

    * feat: denominator raw implementation

    * feat: exclusions and numerator raw implementation

    * docs: nqf0420 qm models documentation

    * feat: nqf0420 final tables

    * fix: tuva_last_run added in numerator

    * fix: POS 02 logic implementation

commit 0825261
Merge: 6db8904 8442d07
Author: Coco Zuloaga <[email protected]>
Date:   Tue Sep 10 14:49:07 2024 -0600

    Merge pull request #583 from tuva-health/cocozuloaga-integration_tests_input_layer_schema

    Update _integration_tests__models.yml to use input_layer schema

commit 8442d07
Author: Coco Zuloaga <[email protected]>
Date:   Fri Sep 6 18:49:53 2024 -0600

    Update _integration_tests__models.yml to use input_layer schema

    Replace _tuva_claims and _tuva_clinical schemas with an input_layer schema for input layer tables inside integration_tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants