Switch to vo-models for nearly all UWS responses

Finish the conversion to using vo-models to generate nearly all
UWS responses. Templating remains only to generate the error
VOTable document.

Due to limitations in pydantic-xml plus the complexity of the
type system, clients will have to provide an additional argument
to the UWS configuration specifying the qualified type of the
`vo_models.uws.JobSummary` class used to represent jobs. Clients
will also have to implement a new `to_xml_model` method on their
`ParametersModel` class.

changelog.d/20241203_171205_rra_DM_47790.md

@@ -0,0 +1,11 @@
+### Backwards-incompatible changes
+
+- UWS clients must now pass an additional `job_summary_type` argument to `UWSAppSettings.build_uws_config` and implement `to_xml_model` in their implementation of `ParametersModel`, returning a subclass of the vo-models `Parameters` class.
+
+### Bug fixes
+
+- Append a colon after the error code when reporting UWS errors.
+
+### Other changes
+
+- Render all UWS XML output except for error VOTables using vo-models rather than hand-written XML templates.

docs/user-guide/uws/create-a-service.rst

@@ -57,6 +57,7 @@ For now, you can just insert placeholder values.
        def uws_config(self) -> UWSConfig:
            return self.build_uws_config(
                async_post_route=UWSRoute(...),
+               job_summary_type=...,
                parameters_type=...,
                sync_get_route=UWSRoute(...),
                sync_post_route=UWSRoute(...),

docs/user-guide/uws/define-models.rst

@@ -4,13 +4,26 @@
 Define job parameter models
 ###########################
 
-UWS models all parameters as simple lists of key/value pairs with string values.
-However, for internal purposes, most applications will want a more sophisticated parameter model than that, with better input validation.
-The frontend should parse and validate the input parameters so that it can fail quickly if they are invalid, rather than creating a job, dispatching it, and only then having it fail due to invalid parameters.
-
-UWS applications therefore must define two models for input parameters, both Pydantic models.
-The first is the model of parameters as provided by users, and is used to validate the input parameters.
-The second is the model that will be passed to the backend worker.
+A UWS job is defined by its input parameters.
+Unfortunately, due to issues with the IVOA UWS standard and the need for separation between the API and backend processing, the input parameters for a job have to be defined in five different ways.
+
+#. A Pydantic API model representing the validated input parameters for a job.
+   This is the canonical input form and corresponds to a native JSON API.
+#. The parameters sent to the backend worker.
+   Often, this may be the same as the API model, but best practice is to define two separate models.
+   This allows the two models to change independently, permitting changes to the backend without changing the API or changes to the API without changing the backend code.
+#. An XML representation of the input parameters.
+   This is essentially a list of key/value pairs wrapped in a child class of `~vo_models.uws.models.Parameters` and is used for XML serialization and deserialization for the IVOA UWS protocol.
+   This separate model is required because the IVOA UWS standard requires a very simplistic XML serialization of job parameters that flattens any complex structure into strings, and thus is not suitable for use as the general API model for many applications.
+#. The input parameters for job creation via ``POST``, since the IVOA UWS standard requires support for job creation via form ``POST``.
+#. The input parameters for job creation via ``GET``, used for sync jobs.
+   Supporting this is optional.
+
+In some cases (jobs whose parameters are all simple strings or numbers), the same model can be used for 1 and 4 by specifying it as a form parameter model.
+Unfortunately, the same model cannot be used for 1 and 3 even for simple applications because the XML model requires additional structure that obscures the parameters and should not be included in the JSON API model.
+
+Therefore, in the most general case, UWS applications must define three models for input parameters: the API model of parameters as provided by users via a JSON API, the model passed to the backend worker, and an XML model that flattens all parameters to strings.
+The input parameters for job creation via ``POST`` and ``GET`` are discussed in :doc:`define-inputs`.
 
 .. _uws-worker-model:
 
@@ -58,12 +71,41 @@ Astropy types do not serialize to JSON by default, so you will need to add seria
 
 If you do this, consider adding a test case for your application that serializes your worker model to JSON, deserializes it back from JSON, and verifies that the resulting object matches the original object.
 
+.. _uws-xml-model:
+
+XML parameter model
+===================
+
+The XML parameter model must be a subclass of `~vo_models.uws.models.Parameters`.
+Each parameter must be either a `~vo_models.uws.models.Parameter` or a ``MultiValuedParameter`` (for the case where the parameter can be specified more than once for simple list support).
+
+This effectively requires serialization of all parameter values to strings, since the value attribute of a `~vo_models.uws.models.Parameter` only accepts simple strings to follow the IVOA UWS standard.
+
+Here is a simple example for the same cutout service:
+
+.. code-block:: python
+
+   from pydantic import Field
+   from vo_models.uws import MultiValuedParameter, Parameter, Parameters
+
+
+   class CutoutXmlParameters(Parameters):
+       id: MultiValuedParameter = Field([])
+       circle: MultiValuedParameter = Field([])
+
+This class should not do any input validation other than validation of the permitted parameter IDs.
+Input validation will be done by the input parameter model.
+
+Single-valued parameters can use the syntax shown in `the vo-models documentation <https://vo-models.readthedocs.io/latest/pages/protocols/uws.html#parameters>`__ to define the parameter ID if it differs from the attribute name.
+Optional multi-valued parameters, such as the above, have to use attribute names that match the XML parameter ID and the ``Field([])`` syntax to define the default to be an empty list, or you will get typing errors.
+
 Input parameter model
 =====================
 
 Every UWS application must define a Pydantic model for its input parameters.
 This model must inherit from `ParametersModel`.
-In addition to defining the parameter model, it must provide two methods: a class method named ``from_job_parameters`` that takes as input the list of `UWSJobParameter` objects and returns an instance of the model, and an instance method named ``to_worker_parameters`` that converts the model to the one that will be passed to the backend worker (see :ref:`uws-worker-model`).
+
+In addition to defining the parameter model, it must provide three methods: a class method named ``from_job_parameters`` that takes as input the list of `UWSJobParameter` objects and returns an instance of the model, an instance method named ``to_worker_parameters`` that converts the model to the one that will be passed to the backend worker (see :ref:`uws-worker-model`), and an instance method named ``to_xml_model`` that converts the model to the XML model (see :ref:`uws-xml-model`).
 
 Often, the worker parameter model will look very similar to the input parameter model.
 They are still kept separate, since the input parameter model defines the API and the worker model defines the interface to the backend.
@@ -78,6 +120,7 @@ Here is an example of a simple model for a cutout service:
 
    from pydantic import Field
    from safir.uws import ParameterParseError, ParametersModel, UWSJobParameter
+   from vo_models.uws import Parameter
 
    from .domain.cutout import Point, WorkerCircleStencil, WorkerCutout
 
@@ -88,8 +131,11 @@ Here is an example of a simple model for a cutout service:
            ra, dec, radius = (float(p) for p in params.split())
            return cls(center=Point(ra=ra, dec=dec), radius=radius)
 
+       def to_string(self) -> str:
+           return f"{c.center.ra!s} {c.center.dec!s} {c.radius!s}"
 
-   class CutoutParameters(ParametersModel[WorkerCutout]):
+
+   class CutoutParameters(ParametersModel[WorkerCutout, CutoutXmlParameters]):
        ids: list[str] = Field(..., title="Dataset IDs")
        stencils: list[CircleStencil] = Field(..., title="Cutout stencils")
 
@@ -111,19 +157,33 @@ Here is an example of a simple model for a cutout service:
        def to_worker_parameters(self) -> WorkerCutout:
            return WorkerCutout(dataset_ids=self.ids, stencils=self.stencils)
 
+       def to_xml_model(self) -> CutoutXmlParameters:
+           ids = [Parameter(id="id", value=i) for i in self.ids]
+           circles = []
+           for circle in self.stencils:
+               circles.append(Parameter(id="circle", value=circle.to_string()))
+           return CutoutXmlParameters(id=ids, circle=circles)
+
 Notice that the input parameter model reuses some models from the worker (``Point`` and ``WorkerCircleStencil``), but adds a new class method to the latter via inheritance.
 It also uses a different parameter for the dataset IDs (``ids`` instead of ``dataset_ids``), which is a trivial example of the sort of divergence one might see between input models and backend worker models.
+``CutoutXmlParameters`` is defined in :ref:`uws-xml-model`.
 
 The input models are also responsible for input parsing and validation (note the ``from_job_parameters`` and ``from_string`` methods) and converting to the worker model.
 The worker model should be in a separate file and kept as simple as possible, since it has to be imported by the backend worker, which may not have the dependencies installed to be able to import other frontend code.
 
+The XML model must use simple key/value pairs of strings to satisfy the UWS XML API, so ``to_xml_model`` may need to do some conversion from the model back to a string representation of the parameters.
+
 Update the application configuration
 ====================================
 
 Now that you've defined the parameters model, you can update :file:`config.py` to pass that model to `UWSAppSettings.build_uws_config`, as mentioned in :ref:`uws-config`.
+
 Set the ``parameters_type`` argument to the class name of the parameters model.
 In the example above, that would be ``CutoutParameters``.
 
+Set the ``job_summary_type`` argument to ``JobSummary[XmlModel]`` where ``XmlModel`` is whatever the class name of your XML parameter model is.
+In the example above, that would be ``JobSummary[CutoutXmlParameters]``.
+
 Next steps
 ==========
 

safir/src/safir/testing/uws.py

@@ -9,14 +9,39 @@
 
 import structlog
 from sqlalchemy.ext.asyncio import AsyncEngine
+from vo_models.uws import JobSummary
 
 from safir.arq import JobMetadata, JobResult, MockArqQueue
 from safir.database import create_async_session, create_database_engine
 from safir.uws import UWSConfig, UWSJob, UWSJobResult
 from safir.uws._service import JobService
 from safir.uws._storage import JobStore
 
-__all__ = ["MockUWSJobRunner"]
+__all__ = ["MockUWSJobRunner", "assert_job_summary_equal"]
+
+
+def assert_job_summary_equal(
+    job_summary_type: type[JobSummary], seen: str, expected: str
+) -> None:
+    """Assert that two job XML documents are equal.
+
+    The comparison is done by converting both to a
+    `~vo_models.uws.models.JobSummary` object qualified with the parameter
+    type used for tests.
+
+    Parameters
+    ----------
+    job_summary_type
+        Type of XML job summary specialized for the XML model used for job
+        parameters. For example, ``JobSummary[SomeXmlParameters]``.
+    seen
+        XML returned by the application under test.
+    expected
+        Expected XML.
+    """
+    seen_model = job_summary_type.from_xml(seen)
+    expected_model = job_summary_type.from_xml(expected)
+    assert seen_model.model_dump() == expected_model.model_dump()
 
 
 class MockUWSJobRunner:

safir/src/safir/uws/__init__.py

@@ -1,7 +1,7 @@
 """Support library for writing UWS-enabled services."""
 
 from ._app import UWSApplication
-from ._config import ParametersModel, UWSAppSettings, UWSConfig, UWSRoute
+from ._config import UWSAppSettings, UWSConfig, UWSRoute
 from ._exceptions import (
     DatabaseSchemaError,
     MultiValuedParameterError,
@@ -12,6 +12,7 @@
 )
 from ._models import (
     ErrorCode,
+    ParametersModel,
     UWSJob,
     UWSJobError,
     UWSJobParameter,

safir/src/safir/uws/_config.py

@@ -2,16 +2,16 @@
 
 from __future__ import annotations
 
-from abc import ABC, abstractmethod
 from collections.abc import Callable, Coroutine
 from dataclasses import dataclass
 from datetime import datetime, timedelta
-from typing import Generic, Self, TypeAlias, TypeVar
+from typing import TypeAlias
 
 from arq.connections import RedisSettings
-from pydantic import BaseModel, Field, SecretStr
+from pydantic import Field, SecretStr
 from pydantic_core import Url
 from pydantic_settings import BaseSettings
+from vo_models.uws import JobSummary
 
 from safir.arq import ArqMode, build_arq_redis_settings
 from safir.pydantic import (
@@ -21,7 +21,7 @@
     SecondsTimedelta,
 )
 
-from ._models import UWSJob, UWSJobParameter
+from ._models import ParametersModel, UWSJob, UWSJobParameter
 
 DestructionValidator: TypeAlias = Callable[[datetime, UWSJob], datetime]
 """Type for a validator for a new destruction time."""
@@ -31,12 +31,7 @@
 ]
 """Type for a validator for a new execution duration."""
 
-T = TypeVar("T", bound=BaseModel)
-"""Generic type for the worker parameters."""
-
 __all__ = [
-    "ParametersModel",
-    "T",
     "UWSAppSettings",
     "UWSConfig",
     "UWSRoute",
@@ -57,39 +52,6 @@ class UWSRoute:
     """Description string for API documentation."""
 
 
-class ParametersModel(BaseModel, ABC, Generic[T]):
-    """Defines the interface for a model suitable for job parameters."""
-
-    @classmethod
-    @abstractmethod
-    def from_job_parameters(cls, params: list[UWSJobParameter]) -> Self:
-        """Validate generic UWS parameters and convert to the internal model.
-
-        Parameters
-        ----------
-        params
-            Generic input job parameters.
-
-        Returns
-        -------
-        ParametersModel
-            Parsed cutout parameters specific to service.
-
-        Raises
-        ------
-        safir.uws.MultiValuedParameterError
-            Raised if multiple parameters are provided but not supported.
-        safir.uws.ParameterError
-            Raised if one of the parameters could not be parsed.
-        pydantic.ValidationError
-            Raised if the parameters do not validate.
-        """
-
-    @abstractmethod
-    def to_worker_parameters(self) -> T:
-        """Convert to the domain model used by the backend worker."""
-
-
 @dataclass
 class UWSConfig:
     """Configuration for the UWS service.
@@ -123,6 +85,15 @@ class encapsulates the configuration of the UWS component that may vary by
     aborted.
     """
 
+    job_summary_type: type[JobSummary]
+    """Type representing the parameter-qualified job summary type.
+
+    Must be set to `~vo_models.uws.JobSummary` qualified with the appropriate
+    subclass of `~vo_models.uws.Parameters`. This is necessary to work around
+    limitations in pydantic-xml, which require the types to be known at class
+    instantiation time.
+    """
+
     lifetime: timedelta
     """The lifetime of jobs.
 
@@ -291,6 +262,7 @@ def build_uws_config(
         self,
         *,
         async_post_route: UWSRoute,
+        job_summary_type: type[JobSummary],
         parameters_type: type[ParametersModel],
         slack_webhook: SecretStr | None = None,
         sync_get_route: UWSRoute | None = None,
@@ -308,18 +280,18 @@ def build_uws_config(
         configuration. Its parameters are the additional settings accepted by
         the UWS library that are not part of the ``UWSAppSettings`` model.
 
-        Returns
-        -------
-        UWSConfig
-            UWS configuration.
-
         Parameters
         ----------
         async_post_route
             Route configuration for job parameters for an async job via
             POST. The FastAPI dependency included in this object should expect
             POST parameters and return a list of `~safir.uws.UWSJobParameter`
             objects representing the job parameters.
+        job_summary_type
+            Type representing the XML job summary type, qualified with an
+            appropriate subclass of `~vo_models.uws.models.Parameters`. That
+            subclass should be the same as that returned by the
+            ``to_xml_model`` method of ``parameters_type``.
         parameters_type
             Type representing the job parameters. This will be used to
             validate parameters and to parse them before passing them to the
@@ -355,6 +327,11 @@ def build_uws_config(
         worker
             Name of the backend worker to call to execute a job.
 
+        Returns
+        -------
+        UWSConfig
+            UWS configuration.
+
         Examples
         --------
         Normally, this method is used from a property method that returns the
@@ -386,6 +363,7 @@ def uws_config(self) -> UWSConfig:
             arq_mode=self.arq_mode,
             arq_redis_settings=self.arq_redis_settings,
             execution_duration=self.timeout,
+            job_summary_type=job_summary_type,
             lifetime=self.lifetime,
             parameters_type=parameters_type,
             signing_service_account=self.service_account,