Data Assistants Feedback

[kyleaton]

Please use this venue to ask questions and give feedback on your experience using Data Assistants.

[shpolina]

I wanted to try the new Data Assistant and running into issues with timestamp columns. In my Redshift table, I have columns of type timestamp without timezone and the DataAssistant fails with the following error.

     59 min_value: MetricValue = np.amin(a=metric_values_converted)
     60 max_value: MetricValue = np.amax(a=metric_values_converted)
---> 61 return build_numeric_range_estimation_result(
     62     metric_values=metric_values_converted,
     63     min_value=min_value,
     64     max_value=max_value,
     65 )

File ~\.virtualenvs\G-1n0w2gCu\lib\site-packages\great_expectations\rule_based_profiler\helpers\util.py:842, in build_numeric_range_estimation_result(metric_values, min_value, max_value)
    840     histogram = np.histogram(a=metric_values_converted, bins=NUM_HISTOGRAM_BINS)
    841     # Use "UTC" TimeZone normalization in "bin_edges" when "metric_values" consists of "datetime.datetime" objects.
--> 842     bin_edges = convert_ndarray_float_to_datetime_dtype(data=histogram[1])
    843 else:
    844     histogram = np.histogram(a=metric_values, bins=NUM_HISTOGRAM_BINS)

File ~\.virtualenvs\G-1n0w2gCu\lib\site-packages\great_expectations\util.py:1558, in convert_ndarray_float_to_datetime_dtype(data)
   1552 """
   1553 Convert all elements of 1-D "np.ndarray" argument from "float" type to "datetime.datetime" type objects.
   1554 
   1555 Note: Converts to "naive" "datetime.datetime" values (assumes "UTC" TimeZone based floating point timestamps).
   1556 """
   1557 value: Any
-> 1558 return np.asarray([datetime.datetime.utcfromtimestamp(value) for value in data])

File ~\.virtualenvs\G-1n0w2gCu\lib\site-packages\great_expectations\util.py:1558, in <listcomp>(.0)
   1552 """
   1553 Convert all elements of 1-D "np.ndarray" argument from "float" type to "datetime.datetime" type objects.
   1554 
   1555 Note: Converts to "naive" "datetime.datetime" values (assumes "UTC" TimeZone based floating point timestamps).
   1556 """
   1557 value: Any
-> 1558 return np.asarray([datetime.datetime.utcfromtimestamp(value) for value in data])

OSError: [Errno 22] Invalid argument```

[talagluck]

Hi @shpolina! I believe a bug around this issue was resolved recently. Can you please confirm whether this is still an issue for you? Thanks!

[jmcorreia]

I would like to share my feedback, that the Data Assistants are looking good, but it would be really nice if we could make this part of the data_docs, so that teams can check in a single place both their validations/expectations and also profiling of the data. Similar to what the BasicDatasetProfiler was doing.

Btw, the plots are not showing up on databricks, so having this integrated in the data_docs would probably make it more generally supported.

[jaymilagroso-globe]

Any update on this?

[jaymilagroso-globe]

Can data assistant results be included in the generated data docs?