Dataset sap #14

Bonnarel · 2022-04-22T21:12:27Z

Major change to change the scope from images/cubes to all datasets. Solves issue #10

wrkflow

remove blank lines

jd-au

Great to see the expansion of SIA to further data types.

jd-au · 2022-04-23T00:15:11Z

SIA.tex

-    \item Visibility data and event lists are not considered as valid DPTYPES in this version due to complexity of data access functionalities for those dataset types.  
-\end{itemize}
+DsSAP specifies standardID values for each capability, as defined by VODataService \citep{std:VODS11}. DsSAP services may be registered in an IVOA Registry using the SimpleDALRegExt \citep{std:DALREGEXT} extension schema.
+\subsection{Changes from SIA-2.0i to DsSAP}


Is the i on SIA-2.0i intended?

jd-au · 2022-04-23T00:16:22Z

SIA.tex


+Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade.  Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes.
+DsSAP is an extension of SIA2 to other dataproduct types. It can be seen as a server side  parameter based proxy to an ObsTAP service.
+IA-2.0 differs from legacy SIA-1.0 in the following aspects:


Is there a list of differences missing here?

No. Have to remove line 59

jd-au · 2022-04-23T00:17:51Z

SIA.tex


+Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade.  Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes.


long lines like this should be broken up to assist readability of the code. This applies to quite a bit of the new content.

jd-au · 2022-04-23T00:18:22Z

SIA.tex


+Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade.  Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes.


The std:SSA and std:SIA2 citations are unresolved and come up as ?

jd-au · 2022-04-23T00:22:36Z

SIA.tex

 \label{sec:async}
-In many cases, datacubes are too large to download and process locally, so the client must be able to perform remote operations. Data discovery could be performed using any discovery protocol (SIA, TAP with ObsCore, etc.).  The client must be able to easily figure out if a low level access service is available for a discovered dataset. This could be using a URL provided in the response or by calling an associated DataLink service. Access operations include basic filtering (cut out a subsection of the data), transformations, or other pixel-level operations or even analysis. With current  version of the AccessData specification, we will only cover extracting a simple subset of an image or datacube. 
+In many cases, datasets are too large to download and process locally, so the client must be able to perform remote operations. Data discovery could be performed using any discovery protocol (DsSAP, SIA, TAP with ObsCore, etc.).  The client must be able to easily figure out if a low level access service is available for a discovered dataset. This could be using a URL provided in the response or by calling an associated DataLink service. Access operations include basic filtering (cut out a subsection of the data), transformations, or other pixel-level operations or even analysis. With version 1.0 of the SODA 1.0  specification, we will only cover extracting a simple subset of an image or datacube. 


The 1.0 in SODA 1.0 is repeated and should this be replaced with a citation of SODA 1.0?

OK. Remove second 1.0

jd-au · 2022-04-23T00:33:58Z

ivoatexmeta.tex

@@ -0,0 +1,6 @@
+% GENERATED FILE -- edit this in the Makefile


Should this file be included in the PR?

jd-au · 2022-04-23T00:35:00Z

.github/workflows/main.yml

@@ -0,0 +1,60 @@
+name: Update PDF Preview


How does this relate to the existing preview.yml build script?

jd-au · 2022-04-23T00:38:20Z

SIA.tex

@@ -472,7 +443,9 @@ \subsubsection{INSTRUMENT}
 The INSTRUMENT parameter is a string-valued parameter that specifies the name of the instrument with which the data was acquired. The value is compared with the instrument\_name from the ObsCore data model.

 \subsubsection{DPTYPE}
-The DPTYPE parameter is a string-valued parameter that specifies the type of data. The value is compared with the dataproduct\_type from the ObsCore data model. For the SIA \{query\} resource, the only values that should be returned for dataproduct\_type are \textit{image} and \textit{cube}, so this parameter can be only really be used to select one of these.
+The DPTYPE parameter is a string-valued parameter that specifies the type of data. The value is compared with the dataproduct\_type from the ObsCore data model.  In contrast to SIA2.0 which allows only dataproduct\_types  \textit{image} and \textit{cube}, all the other dataproduct\_types are allowed, in such a way that we can constrain the service to retrieve visibility data, timeseries and event lists. Using DPTYPE with the "spectrum" value can be considered as an upgrade of the SSA protocol.


Should we include catalogs here? CASDA has downloadable catalogs included in obscore but the dataproduct_type for those must be null according to obscore, which hinders discoverability.

Tricky point. Please open a new issue for that

Bonnarel · 2022-04-23T17:36:10Z

Le 23/04/2022 à 02:39, James Dempsey a écrit :

***@***.**** commented on this pull request. Great to see the expansion of SIA to further data types.

Thanks for reviewing this. I'll answer and fix the bugs ASAP !

…

------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > -\end{itemize} -While some of the more advanced capabilities for dynamic access to multi-dimensional images are still under development, the initial specification provides the capability to discover and download multi-dimensional image datasets via a service interface consistent with current VO standards - -\subsection{Current and Future support} - These are the list of changes which will be considered in a later version of this protocol (SIA-2.1): -\begin{itemize} - \item The protocol should support a detailed {metadata} capability. In the current design, this capability will take a single parameter with an identifier (typically a publisher dataset identifier from a data discovery response) and return a document with detailed metadata about the dataset. The primary usage is to drill-down to the detailed metadata after discovery - \item The standardID for the {metadata} capability will probably be : ivo://ivoa.net/std/SIA\#metadata-2.1 - \item The parameter language should be extended to support case-insensitivity, wildcards, and pattern recognition. - \item The DALI UPLOAD facility will be defined in order to allow queries based on values stored in a file. - \item The protocol will also support an option for virtual data discovery. With this option, controlled by usage of an optional parameter, the query can force the discovery of virtual datasets with best matching to the input parameter constraints - \item COLLECTION and FACILITY currently provide query parameters provide selection on service defined set of strings. It would be good to define standardized lists of those at IVOA level for next version. - \item Visibility data and event lists are not considered as valid DPTYPES in this version due to complexity of data access functionalities for those dataset types. -\end{itemize} +DsSAP specifies standardID values for each capability, as defined by VODataService \citep{std:VODS11}. DsSAP services may be registered in an IVOA Registry using the SimpleDALRegExt \citep{std:DALREGEXT} extension schema. +\subsection{Changes from SIA-2.0i to DsSAP} Is the i on SIA-2.0i intended? ------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > +Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade. Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes. +DsSAP is an extension of SIA2 to other dataproduct types. It can be seen as a server side parameter based proxy to an ObsTAP service. +IA-2.0 differs from legacy SIA-1.0 in the following aspects: Is there a list of differences missing here? ------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > +Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade. Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes. long lines like this should be broken up to assist readability of the code. This applies to quite a bit of the new content. ------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > +Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade. Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes. The |std:SSA| and |std:SIA2| citations are unresolved and come up as |?| ------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > \label{sec:async} -In many cases, datacubes are too large to download and process locally, so the client must be able to perform remote operations. Data discovery could be performed using any discovery protocol (SIA, TAP with ObsCore, etc.). The client must be able to easily figure out if a low level access service is available for a discovered dataset. This could be using a URL provided in the response or by calling an associated DataLink service. Access operations include basic filtering (cut out a subsection of the data), transformations, or other pixel-level operations or even analysis. With current version of the AccessData specification, we will only cover extracting a simple subset of an image or datacube. +In many cases, datasets are too large to download and process locally, so the client must be able to perform remote operations. Data discovery could be performed using any discovery protocol (DsSAP, SIA, TAP with ObsCore, etc.). The client must be able to easily figure out if a low level access service is available for a discovered dataset. This could be using a URL provided in the response or by calling an associated DataLink service. Access operations include basic filtering (cut out a subsection of the data), transformations, or other pixel-level operations or even analysis. With version 1.0 of the SODA 1.0 specification, we will only cover extracting a simple subset of an image or datacube. The 1.0 in SODA 1.0 is repeated and should this be replaced with a citation of SODA 1.0? ------------------------------------------------------------------------ In ivoatexmeta.tex <#14 (comment)>: > @@ -0,0 +1,6 @@ +% GENERATED FILE -- edit this in the Makefile Should this file be included in the PR? ------------------------------------------------------------------------ In .github/workflows/main.yml <#14 (comment)>: > @@ -0,0 +1,60 @@ +name: Update PDF Preview How does this relate to the existing |preview.yml| build script? ------------------------------------------------------------------------ In SIA.tex <#14 (comment)>: > @@ -472,7 +443,9 @@ \subsubsection{INSTRUMENT} The INSTRUMENT parameter is a string-valued parameter that specifies the name of the instrument with which the data was acquired. The value is compared with the instrument\_name from the ObsCore data model. \subsubsection{DPTYPE} -The DPTYPE parameter is a string-valued parameter that specifies the type of data. The value is compared with the dataproduct\_type from the ObsCore data model. For the SIA \{query\} resource, the only values that should be returned for dataproduct\_type are \textit{image} and \textit{cube}, so this parameter can be only really be used to select one of these. +The DPTYPE parameter is a string-valued parameter that specifies the type of data. The value is compared with the dataproduct\_type from the ObsCore data model. In contrast to SIA2.0 which allows only dataproduct\_types \textit{image} and \textit{cube}, all the other dataproduct\_types are allowed, in such a way that we can constrain the service to retrieve visibility data, timeseries and event lists. Using DPTYPE with the "spectrum" value can be considered as an upgrade of the SSA protocol. Should we include catalogs here? CASDA has downloadable catalogs included in obscore but the dataproduct_type for those must be null according to obscore, which hinders discoverability. — Reply to this email directly, view it on GitHub <#14 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMP5LTDTPZ43OPC2P7ZU7ELVGNBD7ANCNFSM5UDNEZEA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

…asetSAP

Bonnarel · 2023-05-07T06:58:05Z

Changes will no more be provided by bunches of little PRs

Bonnarel and others added 7 commits June 3, 2021 14:53

Create main.yml

de71124

wrkflow

Update SIA.tex

80abde2

remove blank lines

Merge branch 'ivoa-std:master' into master

2624b6b

Update SIA.tex

19b46ec

Update SIA.tex

02306c5

Update SIA.tex

d35c979

create DsSAP draft

13da57b

jd-au reviewed Apr 23, 2022

View reviewed changes

Bonnarel and others added 3 commits April 25, 2022 16:42

Delete main.yml

6746682

Fixes on DSSAP version 1.0

7a7de92

Merge branch 'DatasetSAP' of https://github.com/Bonnarel/SIA into Dat…

80b2377

…asetSAP

Bonnarel closed this May 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataset sap #14

Dataset sap #14

Bonnarel commented Apr 22, 2022

jd-au left a comment

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

jd-au Apr 23, 2022

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

jd-au Apr 23, 2022

jd-au Apr 23, 2022

Bonnarel Apr 25, 2022

Bonnarel commented Apr 23, 2022 via email

Bonnarel commented May 7, 2023


		Virtual Observatory access to astronomical images has been available via the SIA-1.0 protocol for over a decade. Many such services have been implemented since 2002, and SIA-1.0 \citep{std:SIAP} was formally standardized as an IVOA Recommendation in 2009. SSA \citep{std:SSA} played a similar role for spectra and also used specific metadata in the query response. SIA-2.0 \citep{std:SIA2} was multi-dimensional and fully integrated with the modern VO architecture and related standards, but restricted to images and data cubes.

		@@ -0,0 +1,6 @@
		% GENERATED FILE -- edit this in the Makefile

Dataset sap #14

Dataset sap #14

Conversation

Bonnarel commented Apr 22, 2022

jd-au left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bonnarel commented Apr 23, 2022 via email

Bonnarel commented May 7, 2023