toil with CWL on LSF status

[mr-c]

• Must keep --workdir on a non shared filesystem like /tmp until Torque and workDir fixes (resolves #1462, #1628) DataBiosphere/toil#1573 is merged (Might be better from a performance perspective anyway)
• Make sure to specify --retries 1 or higher so that killed job get retried with at least the default memory (from --defaultMemory 10Gi or similar) automatically Nope, hand specify minimum memory and update those as jobs fail.
• Speaking of memory, add ResourceRequirements with fixed or dynamic ramMin to all tools.
• test specifying ResourceRequirements at the Workflow and WorkflowStep levels
• toil is experiencing a serialization bug, so don't use format with multiple options for inputs (for now) deserialization of CWL jobs is incorrect for Process.format field DataBiosphere/toil#1692
• --preserve-environment takes a space separated list of environment variables to preserve, not a comma separated list as the docs previously reported Match doc to reality for --preserve-environment DataBiosphere/toil#1689
• Use the TOIL_LSF_ARGS to specify the queue in your runscript: export TOIL_LSF_ARGS="-q production-rh7" Add TOIL_LSF_ARGS support (resolves #1639) DataBiosphere/toil#1640
• There's an error in enumerating jobs in Toil 3.7.0, fix is at Don't error if job hasn't been submitted yet (resolves #1666) DataBiosphere/toil#1690
• Toil doesn't have an override for cwltool's strict filename check, so be sure to strip out offending characters such as :, example at 767cc8f cwl: expose cwltool's --relax-path-checks option DataBiosphere/toil#1782
• like most cwltool based CWL executors, you'll be happier if you set a dedicated output directory via --outdir
• the CWL output object is written to stdout, so redirect that to a file for posterity (example: cwltoil […] | tee output)
• --restart is handy for resuming a previous run, but (due to the lack of cache support while using the LSF batch system) any changes to the CWL descriptions will require a clean start
• apparently Toil will "make up" resource requirements on its own (randomly?) for tools without those annotations, so better be safe least cat get assigned 16 cores and 100GiB of memory :-)
• Toil runs testing using many batch systems (SLURM, Yarn, parasol, mesos, spark, GridEngine), but not LSF -- need to add setup instructions for spinning up a LSF cluster to https://github.com/BD2KGenomics/cgcloud/blob/master/jenkins/src/cgcloud/jenkins/toil_jenkins_slave.py
• Review Globus toolkit's LSF code for inspiration: https://github.com/globus/globus-toolkit/blob/globus_6_branch/gram/jobmanager/lrms/lsf/source/lsf.pm
• how to capture timestamps? they are output to stderr, but not in the on disk log
• how to capture output from LSF?
• Is it possible to run the housekeeping jobs on the launcher node and not via cluster submission? (CWLWorkflow, ResolveIndirect, CWLGather, CWLScatter, etc.. ) run CWL housekeeping jobs on the executor, don't send them to the batch scheduler DataBiosphere/toil#1783
• Restore usage of InitialWorkDirRequirment and confirm
• write up the above lessons learned
• we don't request space in /tmp even though Toil does write there
• Migrate Toil's LSF code to use AbstractGridEngineBatchSystem LSF to use AbstractGridEngineBatchSystem DataBiosphere/toil#2043

Current working branch will the bulk of the above fixed merged: https://github.com/mr-c/toil/tree/issues/1666-fail-not-on-unsubmitted-jobs Latest Toil release has all of the above mentioned fixes merged

[mr-c]

Note: In cwltoil, sub-workflows must fully complete before any of their outputs are available for use by any other step/job. For example, the go_summary in the functional analysis (IPS) workflow isn't subject to further processing, but its production holds up the availability of the functional_annotations for futher processing by the parent workflow.

[mr-c]

To run the CWL conformance tests using cwltoil on LSF

virtualenv env
source env/bin/activate
pip install -U pip
pip install -U setuptools wheel
pip install .[cwl]
git clone https://github.com/common-workflow-language/common-workflow-language.git
cd common-workflow-language
pip install cwltest
TMP=$PWD ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem LSF --logDebug --logFile ${PWD}/log --disableCaching --user-space-docker-cmd=udocker" -j8"

(edited to use " double quote instead of single with EXTRA)
(edited to set TMP to a path on the shared filesystem, needed for cwltest)

[hmenager]

Note for @mr-c : here's what I got from toil[cwl] running a workflow on a single machine, at some point I hear complaints about disk usage, although I never specified any requirements on that:

ripley 2017-06-08 18:50:26,180 Thread-82 WARNING toil.statsAndLogging: Got message from job at time 06-08-2017 18:50:26: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'file:///home/hmenager/ReproHackathon/reprohackathon1/cwl/tools/fastq-dump.cwl' fastq-dump 8/A/job6HSIGE used 128.93% (2.6 GB [2768723968B] used, 2.0 GB [2147483648B] requested) at the end of its run.

The tool itself is defined there:
https://github.com/IFB-ElixirFr/ReproHackathon/blob/cwl/reprohackathon1/cwl/tools/fastq-dump.cwl

[mr-c]

Priorities:

• correct unit detection LSF clusters have varying resource units -- toil needs to detect this and adapt accordingly DataBiosphere/toil#1691, see automatically use the correct LSF units (resolves #1691) DataBiosphere/toil#1762
• move LSF to leverage the abstractGridEngineBatchSystem
• fix dynamic resource requirements: CWL: Deriving resource requirements from inputs fails DataBiosphere/toil#1647