You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The CLI currently contains a lot of code that could be useful for interacting with Cowait from other Python applications. We'd like to extract this into an API client that can be used on its own. Among other things, this is required to implement #287
For the API client to be truly useful it should be able to interact with RemoteTask references similar to what is possible when creating subtasks. A key observation is that all the information available to parent tasks through the websocket connection is also available through the structured log output. This means that we can subscribe to a task through Docker or Kubernetes by attaching to the container logs and parsing them. While its not a perfect solution, it provides a means to retrieve status updates, errors and return values from tasks. RPC calls can not be supported this way.
RemoteTask can then properly implement __await__() and become API-compatible with Task.spawn(). For use cases where asyncio is not desired, a RemoteTask.wait() method could be implemented. It should block until the task completes and either return the result or throw an error.
Ideally, attaching and following container logs should be done in an asynchronous fashion so that multiple tasks could run in parallel. Docker/Kubernetes SDKs provide limited options for asyncio, which means we can either implement our own websocket logs functionality for them - or use additional dependencies. aiodocker seems like a good option for Docker, kubernetes_asyncio could be an option for kubernetes. Both of them have relatively few dependencies (most of which are already dependencies of Cowait), but would still bloat the library considerably.
Other considerations:
Should the client respect cowait.yml configuration by default? Is it the clients' responsibility to load configuration?
The text was updated successfully, but these errors were encountered:
The CLI currently contains a lot of code that could be useful for interacting with Cowait from other Python applications. We'd like to extract this into an API client that can be used on its own. Among other things, this is required to implement #287
For the API client to be truly useful it should be able to interact with
RemoteTask
references similar to what is possible when creating subtasks. A key observation is that all the information available to parent tasks through the websocket connection is also available through the structured log output. This means that we can subscribe to a task through Docker or Kubernetes by attaching to the container logs and parsing them. While its not a perfect solution, it provides a means to retrieve status updates, errors and return values from tasks. RPC calls can not be supported this way.RemoteTask
can then properly implement__await__()
and become API-compatible withTask.spawn()
. For use cases whereasyncio
is not desired, aRemoteTask.wait()
method could be implemented. It should block until the task completes and either return the result or throw an error.Ideally, attaching and following container logs should be done in an asynchronous fashion so that multiple tasks could run in parallel. Docker/Kubernetes SDKs provide limited options for
asyncio
, which means we can either implement our own websocket logs functionality for them - or use additional dependencies.aiodocker
seems like a good option for Docker,kubernetes_asyncio
could be an option for kubernetes. Both of them have relatively few dependencies (most of which are already dependencies of Cowait), but would still bloat the library considerably.Other considerations:
cowait.yml
configuration by default? Is it the clients' responsibility to load configuration?The text was updated successfully, but these errors were encountered: