DM-45281: Add UWS support library #281
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Import the UWS implementation from vo-cutouts. Provide the support code for the backend worker as safir.arq.uws, since it needs to be installable separately and dependeded on safir.arq anyway. Provide the rest as safir.uws, hiding as many of the implementation details as possible (which requires importing internal modules in the test suite, since the tests need to poke at the internals). The UWS code depends on vo-models, which currently still uses Pydantic v1 features, so the deprecation warning for the legacy class configuration syntax has to be suppressed for now until vo-models is fixed. Add a new uws dependency group that should be used when using the uws code. This duplicates some dependencies from arq and db, since that code is used internally.
To allow the UWS support code to be reused by UWS applications, move the mock job runner to a new safir.testing.uws module. Add some additional support to the UWSApplication object so that test suites don't need access to private members of the safir.uws module.
Rather than documenting specific middleware that UWS applications should add, add a method to `UWSApplication` to add all necessary middleware.
UWS applications have a lot of common settings. Rather than making application authors cut and paste those into each application, provide a base settings class that defines them and a helper function for constructing the UWS configuration.
Add a section to the user guide documenting how to use the UWS library. Add anchors to some other documentation pages to make linking easier.
stvoutsin
requested changes
Jul 29, 2024
Now that Safir has switched to nox, remove the top-level tox.ini, which is no longer used.
Each Safir submodule has to be installed with -e separately or uv caches the second and subsequent modules and doesn't install them editable. Fix the noxfile to do this installation correctly.
Previously, the UWS dependency had to be initialized first before the arq queue could be overridden by the test suite. This means that Redis has to be running, since the application will enable UWS first with a Redis queue (which immediately tries to connect) and only later will switch to a mock queue. Modify the UWS dependency so that the arq queue can be overridden before initialize and the overridden queue will be used instead, which should allow UWS applications to be tested without a local Redis.
Rather than doing all the timeout logic directly, use asyncio.timeout when waiting for UWS jobs. Fix various incorrect parameters in docstrings in the new UWS library code. Co-authored-by: Stelios Voutsinas <[email protected]>
|
Thanks for the review! I've also committed all the changes to docstrings and the comment fix. |
The WorkerSettings dataclass in safir.arq.uws is generally useful for any service using arq, not just UWS services. Lift it into safir.arq and update the documentation to mention it. Also update the documentation to reflect the fact that the settings can be either a class or an object, and that the arq command-line tool wants the fully-qualified name of the WorkerSettings object or class, not just the namespace that contains it. (Both might work, but the arq documentation now says to do the former.)
It uses structlog internally for logging from UWS workers.
Note that if one creates the application from a template and selects the UWS flavor, most of the code structure will already be set up for you. Adjust the captions of some code blocks to reflect that code structure.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
safir.uws, which provides a general framework for writing UWS applications with backend workers that use arq. The backend worker function can be any sync Python code and the backend image can be different from the frontend image.Add
safir.arq.uws, which contains only the code required by the backend worker with a minimum of dependencies so that it can be installed on backend workers where more complex dependencies from Safir may conflict.Tested by converting vo-cutouts to this version of the library and checked that it still passed its test suite.