Splitter methods

[Sernight]

Hi,
I've been trying to use SimpleSqlalchemyDatasource for a while now and I'm already having problems at the very beginning in introspection.

    module_name: great_expectations.datasource
    class_name: SimpleSqlalchemyDatasource
    credentials: ${clickhouse_db_credentials}
    introspection:
      hourly:
        introspection_directives:
            include_views: false
        skip_inapplicable_tables: true
        include_schema_name: true
        included_tables:  # only include tables in this list when inferring data asset_names
            - invopt.log_adv_all  # format: schema_name.table_name
        splitter_method: split_on_date_parts
        splitter_kwargs:
            column_name: DateTime
            date_parts:
              - year
              - month
              - day
              - hour

My DateTime column is of type DateTime (Clickhouse).

I've tried to use split_on_converted_datetime, but I get an error:

Cannot initialize datasource clickhousev2: Cannot initialize datasource clickhousev2, error: Splitter method "split_on_converted_datetime" is not supported for "clickhouse" SQL dialect.

Using split_on_date_parts I get error:
Cannot initialize datasource clickhousev2: Cannot initialize datasource clickhousev2, error: number of values in row (1) differ from number of column processors (5)

I have no idea how to solve this, I give up looking for any information on this subject. Checking the Internet, documentation, I cannot even find information about custom splitter method.

[talagluck]

Hi @Sernight - thanks for raising. There are two separate issues here - first is that GX does not offer full support for Clickhouse and is not at all tested on Clickhouse. It's totally possible that none of the splitter methods work for it - the only reason you get a better error message for split_on_converted_datetime is because that one should only work properly for sqlite.

The second thing is that you shouldn't need to use SimpleSqlAlchemyDatasource anymore - we've done the work so that regular Datasource should work with SqlAlchemy, which should actually be more straightforward to connect. That said, we've just done a lot of work on the fluent method of connecting to data, which should be even easier, and you may want to try that out. Again, you may have some luck with Clickhouse since we use a generic SQL Alchemy connection, but it is not at all tested, so we can't vouch for the results.