Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

row ids not displayed in data docs for Spark, SQLAlchemy if specifying unexpected_index_column_names in result_format #10427

Open
NathanJM opened this issue Sep 19, 2024 · 1 comment

Comments

@NathanJM
Copy link

Describe the bug
If running a validation using Spark or SQLAlchemy and specifying values for unexpected_index_column_names, the row ids are not displayed in the data docs.

To Reproduce
Run any expectation suite containing a rule, such as expect_column_value_lengths_to_be, and define a column list for unexpected_index_column_names within the result format object when invoking a checkpoint. This bug is present on all versions of GX.

Expected behavior
The column name/values of the failed rows should be rendered

Environment (please complete the following information):

  • Operating System: [e.g. Linux, MacOS, Windows] All
  • Great Expectations Version: [e.g. 0.13.2] 0.18.x and 1.0.x, probably others
  • Data Source: [e.g. Pandas, Snowflake] SparkDF
  • Cloud environment: [e.g. Airflow, AWS, Azure, Databricks, GCP]

Additional context
the root of the issue appears to be
great_expectations/render/util.py: _convert_unexpected_indices_to_df(), line 422:
set(first_unexpected_index.keys()).difference(set(unexpected_index_column_names))

this ALWAYS evaluates to the empty list, [], causing an exception to be raised later in that same function on line 442 when that empty list is passed to the .groupby() method, aborting the rendering of the failed row indices.

@adeola-ak
Copy link
Contributor

adeola-ak commented Oct 23, 2024

hey do you mind to show me a basic screenshot of what you are seeing in data docs and a screenshot of what you are expecting to see (assuming you see different, expected behavior with different expectations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

No branches or pull requests

2 participants