Handle Missing Data

[sharmaashish]

We need to setup our filters and visualizations, so that we can exclude missing/invalid data

[birm]

I'm not sure how we'd mark missing data, but I'm sure there's a standard. My thought is to use dataDescription.json to signify whether to enforce datatype and how to handle missing data.
Currently, for example, "datatype":"int" seems to allow decimal numbers.

[sharmaashish]

So here's what happens. Let's say we have an attribute called age. It's a numeric attribute. But missing data is labeled as NA or null or has no value. So we start treating it as a string. That is a mistake. It should be treated as a number. Open to suggestions

[birm]

It seems like this should be done in dataDescription.json, I was envisioning something like:

...{
    "attributeName": "Fare",
    "attributeType": [
        "filtering",
        "visual"
    ],
    "dataProvider": "",
    "datatype": "integer",
    **"enforce": {"present"|"integer"|"numeric"|{some regex}}**
}...

where I'm adding the enforce field.
A solution like this seems necessary, because there may be cases where what is invalid is different or very specific.
I'm not sure how people will think of missing/invalid data here, though. Maybe it makes more sense to describe what missing data looks like instead of present data?

[birm]

From the scrum, it seems as though we need a bit of a more nuanced approach than filtering out data that doesn't meet a specified pattern.
So, it sounds like this issues now has two parts, marking data as missing/invalid, and rendering that data. In my "missing data" branch, I'm working on this strategy, starting with marking the data.

[birm]

As not to flood this issue with comments, I'm making a document: here

[birm]

Need to have a distinction between missing and Not Applicable.