Check for zeros in the data

[peterdudfield]

Detailed Description

It would be great to have a check in place that checks for zeros. A large amount of these is normally an error

Context

• good to catch data problems early and fail hard

Possible Implementation

• add a a check, if number os zeros >20% of the data, error
• @devsjc can you suggest a place in the code for this check?

[devsjc]

Yes, I would add it to the filter here:

nwp-consumer/src/nwp_consumer/internal/service/consumer.py

Lines 410 to 424 in 372359c

	def _dataQualityFilter(ds: xr.Dataset) -> bool:
	"""Filter out data that is not of sufficient quality."""
	if ds == xr.Dataset():
	return False
	# Carry out a basic data quality check
	for data_var in ds.data_vars:
	if ds[f"{data_var}"].isnull().any():
	log.warn(
	event=f"Dataset has NaNs in variable {data_var}",
	initTime=str(ds.coords["init_time"].values[0])[:16],
	variable=data_var,
	)
	return True

[GAMinsect]

Hi, I'm someone who is just starting in the field of open source development.
I would like to know if i can contribute to this, since it' a good first issue.
Thank you in advance.

[peterdudfield]

Hi, I'm someone who is just starting in the field of open source development. I would like to know if i can contribute to this, since it' a good first issue. Thank you in advance.

You can definately contirbute. Are you familiar with python and xarray?

[GAMinsect]

I'm familiar with python and common libraries like pandas, but not with xarray

[peterdudfield]

thanks good @GAMinsect, you migh need to learn a bit of xarray. Your welcome to give it ago.

[GAMinsect]

After 2 weeks of working on it in my free time, here's my implementation.
I formatted the code following your coding style.

def _dataQualityFilter(ds: xr.Dataset) -> bool:
    """Filter out data that is not of sufficient quality."""
    if ds == xr.Dataset():
        return False

    zeroCount = 0
    elementCount = 0
    # Carry out a basic data quality check
    for data_var in ds.data_vars:
        if ds[f"{data_var}"].isnull().any():
            log.warn(
                event=f"Dataset has NaNs in variable {data_var}",
                initTime=str(ds.coords["init_time"].values[0])[:16],
                variable=data_var,
            )

        data = ds[data_var].data
        elementCount += data.size
        zeroCount += (data == 0).sum()

    if zeroCount / elementCount > 0.2:
        raise ValueError("In your dataset more than 20% of your data are 0's")

    return True
 

[GAMinsect]

@peterdudfield is the code fine?

[peterdudfield]

ill let @devsjc review if thats ok.