Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for structured log info #216

Draft
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

uniqueg
Copy link
Contributor

@uniqueg uniqueg commented Apr 29, 2024

Implementation notes:

  • Adds a structured_logs array property to the Log schema that allows implementers to provide well-defined structure log information for run (via RunLog.run_log) and task logs (via RunLog.task_logs[]).
  • The structured_logs property uses a custom schema of type StructuredLog that defines a model or schema (log_type) of a given version (log_type_version) that describe the shape of the log contents (logs). Optional properties comprise pointers to additional information, in particular a URL to the specification of the model or schema itself (spec_url) and a URL to human-readable implementation notes (doc_url) describing how to consume the log contents.
  • Valid values for log_type are defined in the enum LogType and their mapping to the log contents in logs should be sufficiently well described in the enum description such that server-side implementers can reasonably be expected to provide the logs in a uniform way, while client-side implementers can reasonably be expected to consume them as long as they provide support for the indicated log_type_version.
  • The enum LogType is nullable. Specifying null can be used to provide information on structured log formats that are not (yet) explicitly supported by WES. In such cases, it is strongly advised that at least one of spec_url or doc_url are provided to allow clients to interpret the log concents.

Open questions:

  • Given that the proposed mechanism may also be relevant for TES, the StructureLog and LogType schemas could also be defined in a separate repository where schemas are kept that can be reused across multiple APIs (perhaps governed by the Cloud WS or DaMaSC?).
  • While the logs property was defined in a deliberately promiscuous way (it can take a string, an object or an array of arbitrary values), it is not clear if the proposed schema is sufficient for supporting a wide range of external log format models and schemas.
  • The proposed code adds the Workflow Run RO-Crate profile collection as known/defined structured log formats. This is motivated by an important use case for the ELIXIR Cloud & AAI Driver Project. Moreover, mapping the Workflow Run RO-Crate profile collection should be easy, given that RO-Crates are JSON-LD documents that should be relatively trivial to return as a single string item. However, if adding a set of concrete log formats goes too far at this point, these can also be added at a later point.
  • OpenAPI 3.1 provides a way to annotate individual enum values via oneOf and const (see here). Adopting this would greatly improve upon the maintainability of the LogType enum. However, it would require to upgrade WES to OpenAPI 3.1.

Resolves #215

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for external logging schemas
1 participant