RFE: improve support for upper and lower limits

[mcdittmar]

Presentations by Vandana Desai

• Nov. 2020 Interop: Implementation of the IVOA Spectral Data Model at IPAC” assessment of V1.1 Recommendation
• May 2021 Interop: "IVOA Spectral Models and Access in the Era of Big Data” enhancement request to support IPAC use cases

DM group meeting:

• DM Meeting 20220622: Notes
  • decision to add Upper/lower limits to the Accuracy class

[mcdittmar]

Interop DAL/DM Session - May 27, 2021

• Alberto Micol: You proposed an upper limit and a lower limit column, but I do not understand how that will work with the FITS binary table which requires every spectrum or SED to be serialised in one record using arrays: one cell for the wavelength array, one cell for the flux array, etc. While I see the usefulness of an "order" column I cannot figure out how you'd serialise the upper/lower limits into columns. Could you explain that?

• G.D-F. In general it is meaningful to quote both a central value with (possibly asymmetric) error bars and statistical upper (and sometimes lower) limits. For values that are statistically consistent with zero, at some significance threshold, one tends to prefer to quote a limit for some purposes. But the original measurement is still relevant statistically, particularly for use in combining with other measurements.

• In current IRSA datasets we usually only have either a central value with uncertainties or a limit, but this is likely to change in the future. For now, we use the "limit" columns to allow displaying spectral and/or SED data points for which only a limit is quoted in the source. Firefly displays them distinctively, as Vandana showed. Distinguishing them from proper measurements allows clients to avoid incorrectly statistically combining the two.

• A.M.: It is the serialisation part that I do not fully understand: would be the upper limit a column containing one array of booleans that tells if the related flux point is a real measurement or an upper limit?

• In the VOTable serialization we are literally using a different column, so that we have the ability to quote both a measurement and a limit, as I noted above. We have not confronted the FITS serialization of this. Obviously it is not efficient for bulk spectral data to have multiple columns - this is why many existing datasets (e.g., WISE photometry) overload limits onto the flux column, with some "magic number" flags to indicate the meaning of the value found in the column. As Vandana said, we are now moving on to the bulk-spectral-data problem and will be looking at this.