Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INFO: _get_default_value called with key "table", but it is not a known field in logs #10553

Open
VolkovGeoPhy opened this issue Oct 23, 2024 · 6 comments

Comments

@VolkovGeoPhy
Copy link

Describe the bug
For instance I use this type of expectations. (or many other types)
class ExpectColumnValuesToMatchRegex(ColumnMapExpectation):
`class ColumnMapExpectation(BatchExpectation, ABC):
"""Base class for ColumnMapExpectations.

ColumnMapExpectations are evaluated for a column and ask a yes/no question about every row in the column.
Based on the result, they then calculate the percentage of rows that gave a positive answer.
If the percentage is high enough, the Expectation considers that data valid.

ColumnMapExpectations must implement a `_validate(...)` method containing logic
for determining whether the Expectation is successfully validated.

ColumnMapExpectations may optionally provide implementations of `validate_configuration`,
which should raise an error if the configuration will not be usable for the Expectation. By default,
the `validate_configuration` method will return an error if `column` is missing from the configuration.

Raises:
    InvalidExpectationConfigurationError: If `column` is missing from configuration.
Args:
    domain_keys (tuple): A tuple of the keys used to determine the domain of the
        expectation.
    success_keys (tuple): A tuple of the keys used to determine the success of
        the expectation.
"""  # noqa: E501

column: StrictStr = Field(min_length=1, description=COLUMN_DESCRIPTION)
mostly: Mostly = 1.0  # type: ignore[assignment] # TODO: Fix in CORE-412

catch_exceptions: bool = True

map_metric: ClassVar[Optional[str]] = None
domain_keys: ClassVar[Tuple[str, ...]] = (
    "batch_id",
    "table",
    "column",
    "row_condition",
    "condition_parser",
)`

To Reproduce
Add expectation ExpectColumnValuesToMatchRegex

Expected behavior
Empty logs

Environment (please complete the following information):

  • Operating System: MacOS (Docker)
  • Great Expectations Version: 1.1.3
  • Data Source: Redshift sqlalchemy

Additional context
Add any other context about the problem here.

@adeola-ak
Copy link
Contributor

Hi there! Redshift isn’t a fully supported or tested data source in Great Expectations. This means that while it can work with SQLAlchemy, certain features or expectations, like ColumnMapExpectation, might run into issues if Redshift handles SQL slightly differently than other databases.

We generate SQL queries to validate data against your expectations. If Redshift doesn’t fully support the type of SQL query being generated, that can cause the validation to fail. Redshift may not support certain functions or data operations that the expectation relies on.

I’d recommend reviewing the SQL queries generated by the expectation to ensure they’re compatible with Redshift’s SQL dialect. You can enable SQL query logging in SQLAlchemy to see what’s being executed and compare it with Redshift’s SQL requirements. If there’s a mismatch, it may be necessary to adjust the_validate() method in your custom expectation to work better with Redshift.

Unfortunately, as Redshift isn't supported, I will be closing this issue. But I do encourage you to call on our community for assistance if the above tip I shared doesn't get you in the right direction. Thank you for reaching out!

@github-project-automation github-project-automation bot moved this from To Do to Completed in GX Core Issues Board Oct 24, 2024
@VolkovGeoPhy
Copy link
Author

@adeola-ak nice but it doesn't relate to Redshift dialect.
https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1729850880800489?thread_ts=1729691473.828959&cid=CUTCNHN82
It just inconsistency in your code. Check the classes that I meantioned in the description!

@VolkovGeoPhy
Copy link
Author

You have one class that inherits another. And in your child class you use validation that fails because of the structure of the parent class

@VolkovGeoPhy
Copy link
Author

@adeola-ak the same with 1.2.2 version

@adeola-ak adeola-ak reopened this Nov 14, 2024
@adeola-ak
Copy link
Contributor

The way this issue was written is not easy to follow. @VolkovGeoPhy can you please refine so I can take another look at what is going on?

Are you saying that anytime you use this expectation ExpectColumnValuesToMatchRegex(ColumnMapExpectation) you are met with the following error?:

Raises: InvalidExpectationConfigurationError: If column is missing from configuration. Args: domain_keys (tuple): A tuple of the keys used to determine the domain of the expectation. success_keys (tuple): A tuple of the keys used to determine the success of the expectation.

You seem to have included code, docs, and errors all in one place. I will be able to escalate this when the issue description, steps to reproduce, and exception are clearer to follow.

@philsv
Copy link

philsv commented Nov 26, 2024

I am having the same issue with the logs, tried to suppress the logs with the standard lib logging but wasn't possible for me, any way to remove these logs?

Also there is nothing mentioned in the docs for a config_provider, what does it mean?

DataFrameAsset.dict() - missing `config_provider`, skipping config substitution
...
_get_default_value called with key "table", but it is not a known field

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

3 participants