Expressions in Janis

[illusional]

This PR addresses and closes:

• (PRIMARY) closes Expressions in Janis janis#8
• closes cwl/wdl basename methods janis#17
• closes question: add arguments to workflow step inputs janis#19

Most workflow specifications allow you to transform some value:

• CWL supports JavaScript in certain places
• WDL supports a Standard Library

We want to support at least some of this functionality. CWL affords us much more flexibility as we could generate javascript functions, WDL is a little stricter in which we would really only implement the standard library for now.

One generic way might be to create a "JavaScript tool", and then we could share generated javascript expressions between CWL and WDL. This would allow us to very flexibly go outside the standard library functions, but adds complexity and potential costs to perform this computation (depending on how it's executed).

Requirements

• Transform an input, or a derived
• Be moderately type-safe

Considerations

• We must be considerate of the automatic type coercions that a library might do:
  • JavaScript
• [WDL] (https://github.com/openwdl/wdl/blob/master/versions/development/SPEC.md#type-coercion)

As part of this type system, we might reconsider adding a check between the selector and the ToolOutput type (to make sure they're valid).

Guide

Janis now supports an Operator to transform some value from a Selector, or the result from another Operator. You could see an operator as a function, but that might include adding, subtracting, and so on.

Where possible, we've tried to override the inbuilt python functions. You can see how we could prefix the value from the input node w.input_value with "prefix".

w.step("stepId", MyTool(changed_input="prefix" + w.input_value))

Note, this doesn't right now change the value of the input node, it builds up a set of operations (AddOperator("prefix", InputSelector("input_value"))), which then get turned into either javascript or WDL functions.

Applying the representation function of a single operator (operator.__repr__()) will give you pseudocode of the operation you're going to implement.

WIP

This PR is work in progress, there is moderate support for WDL currently, and the type system is lacking.

[illusional]

This is looking pretty good. I've got some basic docs up:Janis: Expressions.

Because we've introduced the sep wdl operator, I'm waiting on the following PRs to merge:

• Clarify the expression placeholder and string interpolation behaviour openwdl/wdl#229
• Remove string interpolator options from development spec + add sep function chanzuckerberg/miniwdl#381 (merged)
• Add sep function for joining array of strings broadinstitute/cromwell#5494
• Cleanup command line builder python-wdlgen#5

[illusional]

More thoughts, I've yet to categorise these yet:

• On resources (memory, cpu, timelimit)
• On output_tag / output_format [SHEP-56]
• Convert CWL to use cwlutil (based on Test add cwl-utils implementation #21)

[illusional]

Cromwell PR has been merged 🎉 !!

This PR has been folded into #38, so see that for more information.