[IBCDPE-983] Updates GX functionality to surface warnings #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
BryanFauble
reviewed
Dec 6, 2024
BryanFauble
reviewed
Dec 6, 2024
BryanFauble
approved these changes
Dec 6, 2024
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
For the purposes of this project, "warnings" refer to messages raised following GX data validation informing us about fields which have some values that fail an expectation, but which still meet the threshold set by the GX
mostlyparameter.This PR adds logic and does some refactoring in the
GreatExpectationsRunnerclass and elsewhere to surface these warnings along with any failures. An example of what this looks like in the GX report table on Synapse can bee seen here. In this example, I have also intentionally added a badgenes_biodomainsfile so we can see what it looks like when there are failures and warnings. This second example shows what it looks like with warnings only, which is what we expect now usingmostlyin thegene_infoandnetworkdatasets.Notes:
mostlyparameter was added to three expectations as requested here, but I was unsure of what to set the value to (since they were all passing at 100%, I started withmostly=0.95). Please let me know if this should change.mostlyparameter and its usage in this package inCONTRIBUTING.mdsynapseclientversion up to 4.4.1. This is the last stable version that supports Python 3.8, so until we deprecate support for 3.8 inagora-data-toolswe can no longer upgradesynapseclientversions.