Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cascading and spillover of exception_traceback info all over other expectations #10713

Open
indresh-singh opened this issue Nov 29, 2024 · 0 comments

Comments

@indresh-singh
Copy link

indresh-singh commented Nov 29, 2024

Bug: If one expectation fails (e.g., due to a data type mismatch), the failure prevents the rest of the expectations from being executed successfully. All the expectations in my Great Expectations results object show the same error traceback, which actually corresponds to only one expectation (expect_column_values_to_match_strftime_format)

To Reproduce
Here's the Expectation_suite:

{
"name": "my_expectation_suite",
"id": "ac5bc98b-a950-4111-8fb2-c0e76d8eaf06",
"expectations": [
{
"type": "expect_column_values_to_not_be_null",
"kwargs": {
"catch_exceptions": false,
"column": "sku_id"
},
"meta": {},
"id": "39ce50e4-57d2-4791-909d-c43a0d1d0417"
},
{
"type": "expect_column_values_to_not_be_null",
"kwargs": {
"catch_exceptions": false,
"column": "location_id"
},
"meta": {},
"id": "ded50f35-018e-4e7a-a2f3-b27adc64ba2f"
},
{
"type": "expect_column_values_to_match_regex",
"kwargs": {
"catch_exceptions": false,
"column": "forecast_date",
"regex": "@gmail.com"
},
"meta": {},
"id": "a36daa82-8637-45e0-b3c5-9bb8fefedbab"
},
{
"type": "expect_column_values_to_match_strftime_format",
"kwargs": {
"catch_exceptions": false,
"column": "snapshotdate",
"strftime_format": "%Y-%m-%d"
},
"meta": {},
"id": "58df67c0-77bf-4e2e-bda3-0e07f622baf6"
}
],
"meta": {
"great_expectations_version": "1.2.4"
},
"notes": null
}

Here's the output received (notice how exception_traceback is the same for every expectation, ideally this exception_traceback should be only contained for expectation - expect_column_values_to_match_strftime_format

{
"success": false,
"results": [
{
"success": false,
"expectation_config": {
"type": "expect_column_values_to_not_be_null",
"kwargs": {
"catch_exceptions": false,
"column": "sku_id",
"batch_id": "bronze_wc_forecast-my_dataframe_data_asset"
},
"meta": {},
"id": "39ce50e4-57d2-4791-909d-c43a0d1d0417"
},
"result": {},
"meta": {},
"exception_info": {
"('column_values.nonnull.unexpected_count', 'e8bf7e0f55c1614aa69feae117c3a61e', ())": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
}
}
},
{
"success": false,
"expectation_config": {
"type": "expect_column_values_to_not_be_null",
"kwargs": {
"catch_exceptions": false,
"column": "location_id",
"batch_id": "bronze_wc_forecast-my_dataframe_data_asset"
},
"meta": {},
"id": "ded50f35-018e-4e7a-a2f3-b27adc64ba2f"
},
"result": {},
"meta": {},
"exception_info": {
"('column_values.nonnull.unexpected_count', '5830051c38d53711d10f1b4d04301826', ())": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
}
}
},
{
"success": false,
"expectation_config": {
"type": "expect_column_values_to_match_regex",
"kwargs": {
"catch_exceptions": false,
"column": "forecast_date",
"regex": "@gmail.com",
"batch_id": "bronze_wc_forecast-my_dataframe_data_asset"
},
"meta": {},
"id": "a36daa82-8637-45e0-b3c5-9bb8fefedbab"
},
"result": {},
"meta": {},
"exception_info": {
"('column_values.match_regex.unexpected_count', 'eb97dc1c342c4a0c0b4028e3b0a435d9', '[email protected]')": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
},
"('column_values.nonnull.unexpected_count', 'eb97dc1c342c4a0c0b4028e3b0a435d9', ())": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
}
}
},
{
"success": false,
"expectation_config": {
"type": "expect_column_values_to_match_strftime_format",
"kwargs": {
"catch_exceptions": false,
"column": "snapshotdate",
"strftime_format": "%Y-%m-%d",
"batch_id": "bronze_wc_forecast-my_dataframe_data_asset"
},
"meta": {},
"id": "58df67c0-77bf-4e2e-bda3-0e07f622baf6"
},
"result": {},
"meta": {},
"exception_info": {
"('column_values.match_strftime_format.unexpected_values', 'e15033a11125fa0bba03afdbfd94e870', '71ebb60c1d1cd456f6ac2098748e502f')": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 532, in _process_direct_and_bundled_metric_computation_configurations\n metric_computation_configuration.metric_fn( # type: ignore[misc] # F not callable\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/map_metric_provider/column_map_condition_auxilliary_methods.py", line 309, in _spark_column_map_condition_values\n return [row[column_name] for row in query.collect()]\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 537, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
},
"('column_values.nonnull.unexpected_count', 'e15033a11125fa0bba03afdbfd94e870', ())": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
},
"('column_values.match_strftime_format.unexpected_count', 'e15033a11125fa0bba03afdbfd94e870', 'strftime_format=%Y-%m-%d')": {
"exception_traceback": "Traceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 545, in _process_direct_and_bundled_metric_computation_configurations\n self.resolve_metric_bundle(metric_fn_bundle=metric_fn_bundle_configurations)\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/sparkdf_execution_engine.py", line 904, in resolve_metric_bundle\n res = df.agg(*aggregate["column_aggregates"]).collect()\n File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1218, in collect\n sock_info = self._jdf.collectToPython()\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/py4j/java_gateway.py", line 1322, in call\n return_value = get_return_value(\n File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco\n raise converted from None\npyspark.errors.exceptions.captured.PythonException: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/validator/validation_graph.py", line 276, in _resolve\n self._execution_engine.resolve_metrics(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 279, in resolve_metrics\n return self._process_direct_and_bundled_metric_computation_configurations(\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/execution_engine/execution_engine.py", line 549, in _process_direct_and_bundled_metric_computation_configurations\n raise gx_exceptions.MetricResolutionError(\ngreat_expectations.exceptions.exceptions.MetricResolutionError: \n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n\n",
"exception_message": "\n An exception was thrown from the Python worker. Please see the stack trace below.\nTraceback (most recent call last):\n File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.10/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_strftime_format.py", line 55, in is_parseable_by_format\n raise TypeError( # noqa: TRY003\nTypeError: Values passed to expect_column_values_to_match_strftime_format must be of type string.\nIf you want to validate a column of dates or timestamps, please call the expectation before converting from string format.\n",
"raised_exception": true
}
}
}
],
"suite_name": "my_expectation_suite",
"suite_parameters": {},
"statistics": {
"evaluated_expectations": 4,
"successful_expectations": 0,
"unsuccessful_expectations": 4,
"success_percent": 0.0
},
"meta": {
"great_expectations_version": "1.2.4",
"batch_spec": {
"batch_data": "SparkDataFrame"
},
"batch_markers": {
"ge_load_time": "20241129T085618.749891Z"
},
"active_batch_definition": {
"datasource_name": "bronze_wc_forecast",
"data_connector_name": "fluent",
"data_asset_name": "my_dataframe_data_asset",
"batch_identifiers": {
"dataframe": ""
}
},
"validation_id": "287d27f5-2581-4471-a0f8-541013889773",
"checkpoint_id": null,
"batch_parameters": {
"dataframe": ""
}
},
"id": null
}

Expected behavior
Exception_traceback with the errors should be contained only to the erroneous expectation: column_values_match_strftime_format instead of spilling over to exception_traceback of all other expectations as well.

Environment:

  • Operating System: MacOS
  • Great Expectations Version: 1.2.4
  • Data Source: Dataframe
  • Cloud environment: MS Fabric
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: To Do
Development

No branches or pull requests

1 participant