Skip to content

Latest commit

 

History

History
280 lines (213 loc) · 9.7 KB

README.md

File metadata and controls

280 lines (213 loc) · 9.7 KB
Raw

cwl-WES

License CI Website

Synopsis

Microservice implementing the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification for the execution of workflows written in the Common Workflow Language (CWL).

cwl-WES is a core service of the ELIXIR Cloud & AAI project.

Description

cwl-WES (formerly: WES-ELIXIR) is a Flask/Gunicorn application that makes use of Connexion to implement the GA4GH WES OpenAPI specification. It enables clients/users to execute CWL workflows in the cloud via a GA4GH Task Execution Service (TES)-compatible execution backend (e.g., TESK or Funnel). Workflows can be sent for execution, previous runs can be listed, and the status and run information of individual runs can be queried. The service leverages cwl-tes to interpret CWL workflows, break them down into individual tasks and emit GA4GH TES-compatible HTTP requests to a configured TES instance. Access to endpoints can be configured to require JSON Web Token-based access tokens, such as those issued by ELIXIR AAI. Run information is stored in a MongoDB database.

Note that development is currently in beta stage. Check the website badge at the top of this document for a publicly available test deployment. Further test deployments can be found at the ELIXIR Cloud & AAI's resource listings.

cwl-WES is developed and maintained by the ELIXIR Cloud & AAI project, a multinational effort aimed at establishing and implementing FAIR research in the Life Sciences.

Installation

Kubernetes

See separate instructions available here.

docker-compose

Requirements

Ensure you have the following software installed:

  • Docker (18.06.1-ce, build e68fc7a)
  • docker-compose (1.22.0, build f46880fe)
  • Git (1.8.3.1)

These are the versions used for development/testing. Other versions may or may not work. Please let us know if you encounter any issues with newer versions than the listed ones.

Instructions

Set up environment

Create data directory and required subdiretories:

mkdir -p data/cwl_wes/db data/cwl_wes/output data/cwl_wes/tmp

Clone repository:

git clone https://github.com/elixir-cloud-aai/cwl-WES.git app

Traverse to app directory:

cd app
Optional: Edit/override app config

  • Via the app configuration file

    vi cwl_wes/config/app_config.yaml

  • Via environment variables

    A few configuration settings can be overridden by environment variables:

    export <ENV_VAR_NAME>=<VALUE>

    List of the available environment variables:

    Variable Description
    MONGO_HOST MongoDB host endpoint
    MONGO_PORT MongoDB service port
    MONGO_DBNAME MongoDB database name
    MONGO_USERNAME MongoDB client username
    MONGO_PASSWORD MongoDB client password
    RABBIT_HOST RabbitMQ host endpoint
    RABBIT_PORT RabbitMQ service port
Build & deploy

Build and run services in detached/daemonized mode:

docker-compose up -d --build
Copy FTP credentials

Create a .netrc file with credentials for accessing an FTP server:

cat << EOF > .netrc
machine <HOST>
login <USERNAME>
password <PASSWORD>
EOF

Don't forget to replace <HOST>, <USERNAME> and <PASSWORD> with real values.

If you do not know what to put here, creating an empty file .netrc with, e.g., touch .netrc will be fine for testing purposes.

Copy the file into the running worker container(s):

for cont in $(docker ps --all | grep cwl-wes_wes-worker | cut -f1 -d" "); do
    docker cp .netrc "${cont}:/tmp/user"
done
Use service

Visit Swagger UI:

firefox http://localhost:7777/ga4gh/wes/v1/ui

Example values to start a simple CWL test workflow via the POST /runs endpoint:

workflow_params: {"input":{"class":"File","path":"ftp://ftp-private.ebi.ac.uk/upload/foivos/test.txt"}}`
workflow_type: CWL
workflow_type_version`: v1.0
workflow_url: https://github.com/uniqueg/cwl-example-workflows/blob/master/hashsplitter-workflow.cwl

Leave the rest of the values empty and hit the Try it out! button.

You can also use the service through curl. For example, to send a request to the GET /runs endpoint:

curl -X GET \
    --header 'Accept: application/json' \
    'http://localhost:7777/ga4gh/wes/v1/runs'
Authorization

All endpoints except for GET /service-info can be configured to require the presence of a valid JWT Bearer token in a request's header. This can be enabled by setting the following paraneter in the app configuration to True:

# Security settings
security:
    authorization_required: True

To send authorized requests, you must be in possession of a valid JWT Bearer token. The app was developed against JWTs issues by ELIXIR AAI, although JWTs issued by other identity providers may work.

If you want to utilize actual compute resources in any of the ELIXIR test deployments, you must be a member of a specific ELIXIR user group. You can apply here to be added to that group (ELIXIR AAI only!). When applying, please include a brief description of why you would like to make use of our test deployments and please note that they are for fair use only (please don't try to run your production workflows on them or we will need to re-think our strategy of offering free resources for testing/developing).

To access protected endpoints via curl, you only need to add an Authorization header to your request, followed by the Bearer prefix and your JWT token, like so:

curl -X GET \
    --header 'Accept: application/json' \
    --header 'Authorization: Bearer <YOUR_TOKEN>' \
    'http://localhost:7777/ga4gh/wes/v1/runs'

If you are making use of the Swagger UI, you will need to click on the Authorize button (which appears on the right-hand side of the top bar upon enabling the authorization_required option) of the Swagger UI website and enter your JWT token in the api_key field, preceded by the Bearer prefix:

api_key

Contributing

This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please refer to our organization's contributing guidelines if you are interested to contribute. Please mind the code of conduct for all interactions with the community.

Versioning

Development of the app is currently still in beta stage, and current versions are for internal use only. We are aiming to have a fully spec-compliant version of the app available soon. The plan is to then adopt a semantic versioning scheme in which we would shadow WES spec versioning, with added date stamp patches for any patch-level changes to our service.

License

This project is covered by the Apache License 2.0 also shipped with this repository.

Contact

The project is a collaborative effort under the umbrella of ELIXIR Cloud & AAI. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.