[Feature Clarification] Storage of Test Metrics in External Databases (Postgres/Delta Tables) in Great Expectations v1.0.6 #10589
Unanswered
MidnightRambo
asked this question in
Support
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Background:
We have recently upgraded from Great Expectations version 0.18 to 1.0.6, marking a transition from a pre-release to a stable version. In version 0.18, it was possible to create a metrics store, allowing us to save metrics and test results in a Postgres SQL server. This functionality was integral to our workflow, enabling persistent storage and easy access to test outcomes.
Issue:
Upon upgrading to version 1.0.6, we’ve encountered challenges related to the metrics store:
Documentation Redirect:
The official Great Expectations documentation, specifically the “Configure a metrics store” manual, redirects to the introduction page for version 1.0.6, making it unclear whether the metrics store feature is still supported or if its configuration has changed.
Unclear Feature Availability:
Given the lack of accessible documentation for the metrics store in the new version, we are uncertain whether:
In Discussion #10376, a user inquired about writing checkpoint results to Delta Tables using PySpark on Databricks. This raises the question of whether Great Expectations v1.0.6 supports storing results in table formats like Delta or Postgres.
Request for Clarification:
Is the metrics store feature still available in Great Expectations v1.0.6?
If so, could you provide updated documentation or guidance on how to configure it?
Can test metrics and results be stored externally in databases such as Postgres or Delta Tables in the latest version?
If external storage is supported, what are the recommended methods or best practices to implement this?
If the metrics store has been deprecated, are there plans to provide alternative solutions or migrate existing users to new storage mechanisms?
Impact:
The ability to persist test metrics externally is crucial for our data analysis and reporting workflows. Clarification on this matter will help us decide whether to proceed with the upgrade or revert to the previous stable version until a suitable solution is available.
Additional Information:
Current Version: Great Expectations v1.0.6
Previous Version: Great Expectations v0.18
Environment: Running on Databricks using PySpark
Related Discussion:
#10376
Beta Was this translation helpful? Give feedback.
All reactions