From 1d2309b812bc70c6347183472bec86bb26e4ea9f Mon Sep 17 00:00:00 2001 From: jdidion Date: Mon, 24 Jun 2024 18:01:29 -0700 Subject: [PATCH 01/25] update versions --- README.md | 2 +- SPEC.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9be70103..b018b0fe 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ The WDL *language* has a two-number version (e.g., `1.2`). An increase in the minor (second) version number (e.g., `1.1` to `1.2`) indicates the addition of, or non-breaking changes to, the language or standard library functions. An increase in the major (first) version number (e.g., `1.0` to `2.0`) indicates that breaking changes have been made. -The WDL *specification* has a three-number version (e.g., `1.2.0`). +The WDL *specification* has a three-number version (e.g., `1.2.1`). The specification version tracks the language version, but there may also be patch releases (indicated by a change to the patch, or third, version number) that include fixes for typos, additional examples, or non-breaking clarifications of ambiguous language. ## Language Specifications diff --git a/SPEC.md b/SPEC.md index dfc4c5a4..edca69c5 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,11 +1,12 @@ # Workflow Description Language (WDL) -This is version 1.2.0 of the Workflow Description Language (WDL) specification. It describes WDL `version 1.2`. It introduces a number of new features (denoted by the ✨ symbol) and clarifications to the [1.1.*](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md) version of the specification. It also deprecates several aspects of the 1.0 and 1.1 specifications that will be removed in the [next major WDL version](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md) (denoted by the 🗑 symbol). +This is version 1.2.1 of the Workflow Description Language (WDL) specification. It describes WDL `version 1.2`. It introduces a number of new features (denoted by the ✨ symbol) and clarifications to the [1.1.*](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md) version of the specification. It also deprecates several aspects of the 1.0 and 1.1 specifications that will be removed in the [next major WDL version](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md) (denoted by the 🗑 symbol). ## Revisions Revisions to this specification are made periodically in order to correct errors, clarify language, or add additional examples. Revisions are released as "patches" to the specification, i.e., the third number in the specification version is incremented. No functionality is added or removed after the initial revision of the specification is ratified. +* [1.2.1](): * [1.2.0](https://github.com/openwdl/wdl/tree/release-1.2.0/SPEC.md): 2024-05-24 ## Table of Contents From d7a71564052c2c29eae4a9a84a9d9e6de6736fcc Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 17:51:58 -0700 Subject: [PATCH 02/25] fix #661 --- CHANGELOG.md | 7 ++++++- SPEC.md | 5 +++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a68714c6..f10052b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,11 @@ Keep the changelog pleasant to read in the text editor: + Properly indent blocks. --> +version 1.2.1 +--------------------------- + +* Fix issues with examples (#653, #654, ). Thanks to @stxue1! + version 1.2.0 --------------------------- @@ -35,7 +40,7 @@ version 1.2.0 + Added `disks` and `gpu` reserved hints for requesting specific resources. + Added `contains_key` function to standard library. [PR 603](https://github.com/openwdl/wdl/pull/603) - +**** + Added exponentiation operator (`**`). + Added `find`, and `matches` functions. diff --git a/SPEC.md b/SPEC.md index edca69c5..9b4a0236 100644 --- a/SPEC.md +++ b/SPEC.md @@ -443,7 +443,7 @@ There is no special syntax for multi-line comments - simply use a `#` at the sta # This comment will not be included within the command command <<< # This comment WILL be included within the command after it has been parsed - cat ~{number * 2} + echo ~{number * 2} >>> output { @@ -1408,7 +1408,7 @@ Example output: ```json { - "test_struct.person": { + "test_struct.john": { "name": "John", "account": { "account_number": "123456", @@ -5040,6 +5040,7 @@ task test_gpu { } requirements { + container: "archlinux:latest" gpu: true } } From 100e82fcdb2e4c481ef6ac8dd232f071e05e9a81 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 17:54:29 -0700 Subject: [PATCH 03/25] update changelog; --- CHANGELOG.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f10052b7..ed103a95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ Keep the changelog pleasant to read in the text editor: version 1.2.1 --------------------------- -* Fix issues with examples (#653, #654, ). Thanks to @stxue1! +* Include fixes to examples introduced in v1.1.3 version 1.2.0 --------------------------- @@ -89,6 +89,11 @@ version 1.2.0 + Clarified that accessing a non-existent member of an object, struct, or call is an error. +version 1.1.3 +--------------------------- + +* Fix issues with examples (#653, #654, #661). Thanks to @stxue1! + version 1.1.2 --------------------------- From edd1d9bbdbc72929d9bd075cb96c8ba0665772c0 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 17:56:58 -0700 Subject: [PATCH 04/25] update changelog, fix #662 --- CHANGELOG.md | 2 +- SPEC.md | 26 ++++++++++++++------------ 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed103a95..85380cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662). Thanks to @stxue1! version 1.1.2 --------------------------- diff --git a/SPEC.md b/SPEC.md index 9b4a0236..f0672d01 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1750,18 +1750,20 @@ workflow map_to_struct { String b = "key" String c = "lookup" - # What are the keys to this Struct? - Words literal_syntax = Words { - a: 10, - b: 11, - c: 12 - } - - # What are the keys to this Struct? - Words map_coercion = { - a: 10, - b: 11, - c: 12 + output { + # What are the keys to this Struct? + Words literal_syntax = Words { + a: 10, + b: 11, + c: 12 + } + + # What are the keys to this Struct? + Words map_coercion = { + a: 10, + b: 11, + c: 12 + } } } ``` From efd4165a0ba6328100b7844eb9f2f885210c4748 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 17:58:03 -0700 Subject: [PATCH 05/25] update changelog, fix #663 --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index f0672d01..53953611 100644 --- a/SPEC.md +++ b/SPEC.md @@ -4394,7 +4394,7 @@ Example output: ```json { - "python_strip": ["A", "B", "C"] + "python_strip.lines": ["A", "B", "C"] } ```

From 4361edcd133555e14b79aab12cc55150d5fcc25c Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 17:58:21 -0700 Subject: [PATCH 06/25] update changelog, fix #663 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85380cb0..df408063 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663). Thanks to @stxue1! version 1.1.2 --------------------------- From 7b9e5fef8c265007f9362df3758cc2bb907e41d2 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:04:08 -0700 Subject: [PATCH 07/25] update changelog, fix #664 --- CHANGELOG.md | 2 +- SPEC.md | 6 ++---- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df408063..c1f666bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664). Thanks to @stxue1! version 1.1.2 --------------------------- diff --git a/SPEC.md b/SPEC.md index 53953611..e6b04caa 100644 --- a/SPEC.md +++ b/SPEC.md @@ -6836,7 +6836,6 @@ workflow allow_nested { input { Int int_val String msg1 - String msg2 Array[Int] my_ints File ref_file } @@ -6852,7 +6851,7 @@ workflow allow_nested { call lib.repeat as repeat2 { # Note: the default value of `0` for the `i` input causes the task to fail - opt_string = msg2 + i = 2 } scatter (i in my_ints) { @@ -6876,10 +6875,9 @@ Example input: { "allow_nested.int_val": 3, "allow_nested.msg1": "hello", - "allow_nested.msg2": "goodbye", "allow_nested.my_ints": [1, 2, 3], "allow_nested.ref_file": "hello.txt", - "allow_nested.repeat2.i": 2 + "allow_nested.repeat2.opt_string": "goodbye" } ``` From 273e7cc50d1d7575a2bc88e3f544e3a6306efd6b Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:06:14 -0700 Subject: [PATCH 08/25] update changelog, fix #665 --- CHANGELOG.md | 2 +- SPEC.md | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1f666bb..fdd0ac2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663, #664). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664, #665). Thanks to @stxue1! version 1.1.2 --------------------------- diff --git a/SPEC.md b/SPEC.md index e6b04caa..0d560d2c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -6850,7 +6850,6 @@ workflow allow_nested { } call lib.repeat as repeat2 { - # Note: the default value of `0` for the `i` input causes the task to fail i = 2 } @@ -7249,12 +7248,12 @@ workflow if_else { # the body *is not* evaluated since 'b' is false if (is_morning) { - call greet as morning { time = "morning" } + call greet as morning { input: time = "morning" } } # the body *is* evaluated since !b is true if (!is_morning) { - call greet as afternoon { time = "afternoon" } + call greet as afternoon { input: time = "afternoon" } } output { @@ -7299,7 +7298,7 @@ workflow nested_if { if (morning) { if (friendly) { - call if_else.greet { time = "morning" } + call if_else.greet { input: time = "morning" } } } From 7a509ae5db1f74c0c260e3985edf6e78b2638609 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:07:20 -0700 Subject: [PATCH 09/25] update changelog, fix #666 --- SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 0d560d2c..66ed9a54 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9613,7 +9613,7 @@ workflow test_prefix { Array[Int] env2 = [1, 2, 3] output { - Array[String] env_prefixed = prefix("-e ", env1) + Array[String] env1_prefixed = prefix("-e ", env1) Array[String] env2_prefixed = prefix("-f ", env2) } } @@ -11822,7 +11822,7 @@ Example output: ```json { - "serialize_array_delim.strings": [ + "serialize_array_delim.heads": [ "hello world", "hello world", "hi_world" From be15cf26ae315575a678099248bdb5caca025c49 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:07:47 -0700 Subject: [PATCH 10/25] update changelog, fix #665 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fdd0ac2a..c1f666bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663, #664, #665). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664). Thanks to @stxue1! version 1.1.2 --------------------------- From e2418eb07018953aa03b9729740feac7155c35e0 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:08:04 -0700 Subject: [PATCH 11/25] update changelog, fix #666 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1f666bb..34c65b9a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663, #664). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666). Thanks to @stxue1! version 1.1.2 --------------------------- From a2f6b57aeb11e06b3821c332f11e8628e77793f0 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 25 Jun 2024 18:12:46 -0700 Subject: [PATCH 12/25] update changelog, fix #668 --- CHANGELOG.md | 2 +- SPEC.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34c65b9a..d80d2180 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666,, #668). Thanks to @stxue1! version 1.1.2 --------------------------- diff --git a/SPEC.md b/SPEC.md index 66ed9a54..525abb4e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -4363,12 +4363,12 @@ task python_strip { } command<<< - python <>> output { @@ -4404,10 +4404,10 @@ Given an `infile` value of `/path/to/file`, the execution engine will produce th ```sh python < Date: Wed, 26 Jun 2024 13:23:53 -0700 Subject: [PATCH 13/25] fix #667 --- SPEC.md | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/SPEC.md b/SPEC.md index 525abb4e..693316de 100644 --- a/SPEC.md +++ b/SPEC.md @@ -849,23 +849,37 @@ Single- and double-quotes do not need to be escaped within a multi-line string. A `File` or `Directory` declaration may have have a string value indicating a relative or absolute path on the local file system. -Within a WDL file, literal values for files may only be (relative or absolute) paths that are local to the execution environment. If the specified path does not exist, it is an error unless the declaration is optional. - ```wdl task literals_paths { input { - # If the user does not overide the value of `f1`, and /foo/bar.txt - # does not exist, it is an error. File f1 = "/foo/bar.txt" - - # If the user does not override the value of `f2` and /foo/bar.txt - # does not exist, then `f2` is set to `None`. - File? f2 = "/foo/bar.txt" + File? f2 } + + command <<< + # If the user does not overide the value of `f1`, and /foo/bar.txt + # does not exist, an error will occur when the file is accessed here. + cat "~{f1}" + + # If the user does not specify the value of `f2` it's value is `None`, + # which results in the empty-string when interpolated. `-f ""` is + # always false. + if [ -f "~{f2}" ]; then + echo "~{f2}" + fi + >>> } ``` -An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path. +Within a WDL file, literal values for files and directories may only be (relative or absolute) paths that are local to the execution environment. + +A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional. + +* To read from a path, the file/directory must exist and be accessible for reading (i.e., be assigned the appropriate permissions). +* To write to a file, the path's parent directory must be accessible for writing. +* To write to a directory, it must exist and be accessible for writing. + +An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path. Remote files must be treated as read-only. A remote file is only required to be vaild at the time that the execution engine needs to localize it. #### Optional Types and None From 3da88257bf44529e9c9641026288497fb1f74a4d Mon Sep 17 00:00:00 2001 From: jdidion Date: Wed, 26 Jun 2024 13:25:25 -0700 Subject: [PATCH 14/25] update changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d80d2180..594980df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,7 +92,7 @@ version 1.2.0 version 1.1.3 --------------------------- -* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666,, #668). Thanks to @stxue1! +* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666, #667, #668). Thanks to @stxue1! version 1.1.2 --------------------------- From 5bd0b0f25e311a64cfc993ed00adcb93ee62c942 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 23 Jul 2024 18:08:30 -0700 Subject: [PATCH 15/25] 672: clarify that mount points must be empty, writable, and ephemeral --- CHANGELOG.md | 2 ++ SPEC.md | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 594980df..89109d7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,8 @@ Keep the changelog pleasant to read in the text editor: version 1.2.1 --------------------------- +* Clarify that `disks` mount points ephemeral and should not already exist + * Include fixes to examples introduced in v1.1.3 version 1.2.0 diff --git a/SPEC.md b/SPEC.md index 693316de..d093b8b9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -5099,9 +5099,11 @@ Test config: * `Array[String]` - An array of disk specifications. * Default value: `1 GiB` -The `disks` attribute provides a way to request one or more persistent volumes, each of which has a minimum size and is mounted at a specific location. When the `disks` attribute is provided, the execution engine must guarantee the requested resources are available or immediately fail the task prior to instantiating the command. +The `disks` attribute provides a way to request one or more persistent volumes, each of which has a minimum size and is mounted at a specific location with both read and write permissions. When the `disks` attribute is provided, the execution engine must guarantee the requested resources are available or immediately fail the task prior to instantiating the command. -If a mount point is specified, then it must be an absolute path to a location in the host environment. If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task. +If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task. + +If a mount point is specified, then it must be an absolute path to a location in the host environment (i.e., within the container). The specified path either must not already exist in the host environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes. The execution engine is free to provision any class(es) of persistent volume it has available (e.g., SSD or HDD). The [`disks` hint](#-disks) hint can be used to request specific attributes for the provisioned disks. From ea1cd0a64e46371eadb61b0423b2b542bc3a1919 Mon Sep 17 00:00:00 2001 From: jdidion Date: Tue, 23 Jul 2024 18:16:37 -0700 Subject: [PATCH 16/25] 671: fix python json serialization of string list --- SPEC.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index d093b8b9..0a43fe93 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9178,9 +9178,10 @@ task write_json { command <<< python <>> From b92757327eeed1c1804f8a58db5938e720fa7cd0 Mon Sep 17 00:00:00 2001 From: jdidion Date: Wed, 24 Jul 2024 12:27:21 -0700 Subject: [PATCH 17/25] migrate changes from #674 --- SPEC.md | 149 +++++++++++++++++++++++++++++++------------------------- 1 file changed, 82 insertions(+), 67 deletions(-) diff --git a/SPEC.md b/SPEC.md index 0a43fe93..94f6850e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -391,7 +391,7 @@ WDL also provides features for implementing more complex workflows. For example, ```json { - "hello.all_matches": [["hi_world"], ["hi_pal"]] + "hello_parallel.all_matches": [["hi_world"], ["hi_pal"]] } ```

@@ -930,7 +930,8 @@ An optional declaration has a default initialization of `None`, which indicates "optionals.test_defined": false, "optionals.test_defined2": true, "optionals.test_is_none": true, - "optionals.test_not_none": false + "optionals.test_not_none": false, + "optionals.test_non_equal": true } ```

@@ -1042,7 +1043,7 @@ task sum { } command <<< - printf ~{sep(" ", ints)} | awk '{tot=0; for(i=1;i<=NF;i++) tot+=$i; print tot}' + printf "~{sep(" ", ints)}" | awk '{tot=0; for(i=1;i<=NF;i++) tot+=$i; print tot}' >>> output { @@ -1107,8 +1108,8 @@ Example output: { "non_empty_optional.nonempty1": [0.0], "non_empty_optional.nonempty2": [null, 1], - "non_empty_optional.nonempty3": [], - "non_empty_optional.nonempty4": [0.0] + "non_empty_optional.nonempty3": null, + "non_empty_optional.nonempty4": [0] } ```

@@ -1974,7 +1975,7 @@ task count_lines { } command <<< - wc -l ~{write_lines(array)} + wc -l < ~{write_lines(array)} >>> output { @@ -2466,7 +2467,7 @@ version 1.2 struct Experiment { String id Array[String] variables - Map[String, Float] data + Map[String, String] data } workflow nested_access { @@ -2499,7 +2500,7 @@ Example input: "variables": ["name", "height"], "data": { "name": "Pinky", - "height": 7 + "height": "7" } }, { @@ -2507,7 +2508,7 @@ Example input: "variables": ["name", "weight"], "data": { "name": "Porky", - "weight": 1000 + "weight": "1000" } } ] @@ -2676,7 +2677,7 @@ Example input: { "placeholders.start": "h", "placeholders.end": "o", - "placeholders.input": "hello" + "placeholders.instr": "hello" } ``` @@ -2684,7 +2685,8 @@ Example output: ```json { - "placeholders.cmd": "grep 'h...o' hello" + "placeholders.cmd": "grep 'h...o' hello", + "placehoders.s": "4" } ```

@@ -2805,12 +2807,15 @@ Example: placeholder_coercion.wdl version 1.2 workflow placeholder_coercion { - File x = "/hij" + input { + File x + } + String x_as_str = x Int? i = None output { Boolean is_true1 = "~{"abc"}" == "abc" - Boolean is_true2 = "~{x}" == "/hij" + Boolean is_true2 = "~{x}" == x_as_str Boolean is_true3 = "~{5}" == "5" Boolean is_true4 = "~{3.141}" == "3.141000" Boolean is_true5 = "~{3.141 * 1E-10}" == "0.000000" @@ -2824,7 +2829,9 @@ workflow placeholder_coercion { Example input: ```json -{} +{ + "placeholder_coercion.x": "hello.txt" +} ``` Example output: @@ -2973,7 +2980,7 @@ Example output: ```json { - "flags.num_matches": 2 + "flags.num_matches": "2" } ```

@@ -3321,7 +3328,7 @@ Example input: ```json { - "person_struct.person": { + "greet_person.person": { "name": { "first": "Richard", "last": "Rich" @@ -3342,7 +3349,7 @@ Example output: ```json { - "person_struct.message": "Hello Richard! You have 1 test result(s) available.\nPlease transfer USD 500 to continue" + "greet_person.message": "Hello Richard! You have 1 test result(s) available.\nPlease transfer USD 500 to continue" } ``` @@ -3482,6 +3489,7 @@ task calculate_bill { workflow import_structs { input { + File infile Person doctor = Person { age: 10, name: Name { @@ -3506,12 +3514,12 @@ workflow import_structs { period: "hourly" }, assay_data: { - "glucose": "hello.txt" + "glucose": infile } } } - call person_struct.greet_person { + call person_struct_task.greet_person { person = patient } @@ -3529,14 +3537,16 @@ workflow import_structs { Example input: ```json -{} +{ + "import_structs.infile": "hello.txt" +} ``` Example output: ```json { - "import_structs.bill": 175000 + "import_structs.bill": 175000.0 } ```

@@ -4249,7 +4259,7 @@ task test_placeholders { # The `read_lines` function reads the lines from a file into an # array. The `sep` function concatenates the lines with a space # (" ") delimiter. The resulting string is then printed to stdout. - printf ~{sep(" ", read_lines(infile))} + printf "~{sep(" ", read_lines(infile))}" >>> output { @@ -4461,8 +4471,7 @@ Example input: ```json { - "outputs.t": 5, - "outputs.write_outstr": false + "outputs.t": 5 } ``` @@ -4557,7 +4566,7 @@ task glob { } command <<< - for i in 1..~{num_files}; do + for i in {1..~{num_files}}; do printf ${i} > file_${i}.txt done >>> @@ -4612,7 +4621,7 @@ task relative_and_absolute { >>> output { - File something = read_string("my/path/to/something.txt") + String something = read_string("my/path/to/something.txt") File bashrc = "/root/.bashrc" } @@ -4663,7 +4672,7 @@ task optional_output { command <<< printf "1" > example1.txt - if ~{make_example2}; do + if ~{make_example2}; then printf "2" > example2.txt fi >>> @@ -4691,6 +4700,8 @@ Example output: ```json { "optional_output.example2": null, + "optional_output.example1": "example1.txt", + "optional_output.file_array": ["example1.txt", null], "optional_output.file_array_len": 1 } ``` @@ -4789,7 +4800,7 @@ task dynamic_container { >>> output { - String is_true = ubuntu_version == read_string(stdout()) + Boolean is_true = ubuntu_version == read_string(stdout()) } requirements { @@ -5389,7 +5400,7 @@ task test_hints { } command <<< - wc -l ~{foo} + wc -l < ~{foo} >>> output { @@ -5710,11 +5721,11 @@ task ex_paramter_meta { } command <<< - wc ~{if lines_only then '-l' else ''} ~{infile} + wc ~{if lines_only then '-l' else ''} < ~{infile} >>> output { - String result = stdout() + Int result = read_int(stdout()) } requirements { @@ -5737,7 +5748,7 @@ Example output: ```json { - "ex_paramter_meta.result": "3" + "ex_paramter_meta.result": 2 } ```

@@ -5848,7 +5859,7 @@ version 1.2 task hisat2 { input { - File index + File index_tar_gz String sra_acc Int? max_reads Int threads = 8 @@ -5856,15 +5867,15 @@ task hisat2 { Float disk_size_gb = 100 } - String index_id = basename(index, ".tar.gz") + String index_id = basename(index_tar_gz, ".tar.gz") command <<< mkdir index - tar -C index -xzf ~{index} + tar -C index -xzf ~{index_tar_gz} hisat2 \ -p ~{threads} \ - ~{if defined(max_reads) then "-u ~{select_first([max_reads])}" else ""} \ - -x index/~{index_id} \ + ~{if defined(max_reads) then "-u ~{round(select_first([max_reads]))}" else ""} \ + -x index/grch38/genome \ --sra-acc ~{sra_acc} > ~{sra_acc}.sam >>> @@ -5884,7 +5895,7 @@ task hisat2 { } parameter_meta { - index: "Gzipped tar file with HISAT2 index files" + index_tar_gz: "Gzipped tar file with HISAT2 index files" sra_acc: "SRA accession number or reads to align" } } @@ -6216,7 +6227,7 @@ task echo { } command <<< - printf ~{msg} + printf '~{msg}\n' >>> output { @@ -6284,7 +6295,7 @@ task foobar { } command <<< - wc -l ~{infile} + wc -l < ~{infile} >>> output { @@ -6326,7 +6337,7 @@ Example output: ```json { - "other.results": 3 + "other.results": 2 } ```

@@ -6556,8 +6567,8 @@ task repeat { echo "i must be >= 1" exit 1 fi - for i in 1..~{i}; do - printf ~{select_first([opt_string, "default"])} + for i in {1..~{i}}; do + printf '~{select_first([opt_string, "default"])}\n' done >>> @@ -7061,7 +7072,7 @@ workflow nested_scatter { Array[String] salutations = ["Hello", "Goodbye"] } - Array[String] honorifics = ["Wizard", "Mr."] + Array[String] honorifics = ["Mr.", "Wizard"] # the zip() function creates an array of pairs Array[Pair[String, String]] name_pairs = zip(first_names, last_names) @@ -7141,7 +7152,8 @@ Example output: ["Hello Mr. Merry, how are you?", "Hello Mr. Merry Brandybuck, how are you?"], ["Goodbye Mr. Merry, how are you?", "Goodbye Mr. Merry Brandybuck, how are you?"] ] - ] + ], + "nested_scatter.used_honorifics": ["Mr.;", "Wizard", "Mr."] } ```

@@ -7228,7 +7240,8 @@ Example output: ```json { "test_conditional.result_array": [4, 6, 8, 10], - "test_conditional.maybe_result2": [0, 4, 6, 8, 10] + "test_conditional.maybe_result2": [0, 4, 6, 8, 10], + "test_conditional.j_out": 2 } ```

@@ -7419,7 +7432,7 @@ Example output: ```json { - "test_floor.all_true": true + "test_floor.all_true": [true, true] } ```

@@ -7474,7 +7487,7 @@ Example output: ```json { - "test_ceil.all_true": true + "test_ceil.all_true": [true, true] } ```

@@ -7529,7 +7542,7 @@ Example output: ```json { - "test_round.all_true": true + "test_round.all_true": [true, true] } ```

@@ -7632,8 +7645,8 @@ workflow test_max { output { # these two expressions are equivalent - Float min1 = if value1 > value2 then value1 else value2 - Float min2 = max(value1, value2) + Float max1 = if value1 > value2 then value1 else value2 + Float max2 = max(value1, value2) } } ``` @@ -7652,8 +7665,8 @@ Example output: ```json { - "test_max.min1": 1.0, - "test_max.min2": 1.0 + "test_max.max1": 2.0, + "test_max.max2": 2.0 } ```

@@ -7813,7 +7826,7 @@ workflow test_sub { String chocoearly = sub(chocolike, "late", "early") # I like chocoearly when\nit's early String chocolate = sub(chocolike, "late$", "early") # I like chocolate when\nit's early String chocoearlylate = sub(chocolike, "[^ ]late", "early") # I like chocearly when\nit's late - String choco4 = sub(chocolike, " [:alpha:]{4} ", " 4444 ") # I 4444 chocolate 4444\nit's late + String choco4 = sub(chocolike, " [:alpha:]{4} ", " 4444 ") # I 4444 chocolate when\nit's late String no_newline = sub(chocolike, "\\n", " ") # "I like chocolate when it's late" } } @@ -8087,7 +8100,7 @@ task gen_files { } command <<< - for i in 1..~{num_files}; do + for i in {1..~{num_files}}; do printf ${i} > a_file_${i}.txt done mkdir a_dir @@ -8241,7 +8254,7 @@ task echo_stdout { command <<< printf "hello world" >>> output { - File message = read_string(stdout()) + String message = read_string(stdout()) } } ``` @@ -8286,7 +8299,7 @@ task echo_stderr { command <<< >&2 printf "hello world" >>> output { - File message = read_string(stderr()) + String message = read_string(stderr()) } } ``` @@ -8919,8 +8932,8 @@ version 1.2 task read_map { command <<< - printf "key1\tvalue1\n" >> map_file - printf "key2\tvalue2\n" >> map_file + printf "key1\tvalue1\n" + printf "key2\tvalue2\n" >>> output { @@ -9718,7 +9731,7 @@ workflow test_suffix { Array[Int] env2 = [1, 2, 3] output { - Array[String] env1_suffix = suffix(".txt ", env1) + Array[String] env1_suffix = suffix(".txt", env1) Array[String] env2_suffix = suffix(".0", env2) } } @@ -9968,7 +9981,7 @@ task double { command <<< >>> output { - Int d = n * n + Int d = 2 * n } } @@ -9993,7 +10006,7 @@ Example input: ```json { - "test_range.n": 5 + "test_range.i": 5 } ``` @@ -10035,7 +10048,9 @@ workflow test_transpose { Array[Array[Int]] expected_output_array = [[0, 3], [1, 4], [2, 5]] output { - Boolean is_true = transpose(input_array) == expected_output_array + Array[Array[Int]] out = transpose(input_array) + Array[Array[Int]] expected = expected_output_array + Boolean is_true = out == expected } } ``` @@ -12179,7 +12194,7 @@ task grep2 { opts=( ~{sep(" ", quote(opts_and_values.left))} ) values=( ~{sep(" ", quote(opts_and_values.right))} ) command="grep" - for i in 1..~{n}; do + for i in {0..~{n-1}}; do command="$command ${opts[i]}"="${values[i]}" done $command ~{pattern} ~{infile} @@ -12262,7 +12277,7 @@ task serde_map_tsv { >>> output { - Map[String, String] new_items = read_map("lines") + Map[String, String] new_items = read_map(stdout()) } } ``` @@ -12333,7 +12348,7 @@ version 1.2 task serde_map_json { input { - Map[String, Float] read_quality_scores + Map[String, Int] read_quality_scores } command <<< @@ -12349,7 +12364,7 @@ task serde_map_json { >>> output { - Map[String, Float] ascii_values = read_json(stdout()) + Map[String, Int] ascii_values = read_json(stdout()) } requirements { From 91f2788e9d045334a8441602dcfc1418dd4293fc Mon Sep 17 00:00:00 2001 From: jdidion Date: Wed, 24 Jul 2024 12:33:52 -0700 Subject: [PATCH 18/25] fix --- SPEC.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/SPEC.md b/SPEC.md index 94f6850e..6e6b9de6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1590,14 +1590,17 @@ Example: string_to_file.wdl version 1.2 workflow string_to_file { - String path1 = "/path/to/file" - File path2 = "/path/to/file" + input { + File infile + } + + String path1 = "~{infile}" # valid - String coerces unambiguously to File - File path3 = path1 + File path2 = path1 output { - Boolean paths_equal = path2 == path3 + Boolean paths_equal = path1 == path2 } } ``` @@ -1606,7 +1609,9 @@ workflow string_to_file { Example input: ```json -{} +{ + "string_to_file.infile": "hello.txt" +} ``` Example output: @@ -2282,8 +2287,8 @@ Example output: { "array_map_equality.is_true1": true, "array_map_equality.is_true2": true, - "array_map_equality.is_false1": true, - "array_map_equality.is_false2": true + "array_map_equality.is_false1": false, + "array_map_equality.is_false2": false } ```

From 830d074dbef2aaaf071849441a3118e8a8484f50 Mon Sep 17 00:00:00 2001 From: jdidion Date: Wed, 24 Jul 2024 19:27:30 -0700 Subject: [PATCH 19/25] disallow relative path literals --- SPEC.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 6e6b9de6..72a4ff43 100644 --- a/SPEC.md +++ b/SPEC.md @@ -871,7 +871,7 @@ task literals_paths { } ``` -Within a WDL file, literal values for files and directories may only be (relative or absolute) paths that are local to the execution environment. +Within a WDL file, literal values for files and directories may only be paths that are local to the execution environment. A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional. @@ -879,6 +879,8 @@ A path is only required to be valid if and when it is accessed. A path assigned * To write to a file, the path's parent directory must be accessible for writing. * To write to a directory, it must exist and be accessible for writing. +Path literals should be absolute paths, since the execution engine is free to set the working directory of a task execution, and thus relative paths may not exist at runtime. 🗑 Use of relative path literals when defining input and private declarations is currently allowed but is deprecated and will be disallowed in WDL 2.0. + An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path. Remote files must be treated as read-only. A remote file is only required to be vaild at the time that the execution engine needs to localize it. #### Optional Types and None From a38d1610d52f42ab9e341ff83a355893b641aaba Mon Sep 17 00:00:00 2001 From: jdidion Date: Wed, 24 Jul 2024 19:28:14 -0700 Subject: [PATCH 20/25] update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 89109d7f..b75f4357 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,8 @@ version 1.2.1 * Clarify that `disks` mount points ephemeral and should not already exist +* Deprecate the use of relative path literals in input and private variable declarations. + * Include fixes to examples introduced in v1.1.3 version 1.2.0 From fcd5751322444928538798d0065e1575b1b804cf Mon Sep 17 00:00:00 2001 From: Mark Woon Date: Thu, 25 Jul 2024 11:29:45 -0700 Subject: [PATCH 21/25] Fix typo --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index dfc4c5a4..b2f64393 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3267,7 +3267,7 @@ struct Person { description: "Encapsulates data about a person" } - paramter_meta { + parameter_meta { name: "The person's name" age: "The person's age" income: "How much the person makes (optional)" From 5f650f4ebfa02cdf03135d0cb076750f80fff1d6 Mon Sep 17 00:00:00 2001 From: jdidion Date: Thu, 25 Jul 2024 17:51:09 -0700 Subject: [PATCH 22/25] use index_id rather than hard-coded path --- SPEC.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/SPEC.md b/SPEC.md index 76a1f096..45109ecb 100644 --- a/SPEC.md +++ b/SPEC.md @@ -5877,12 +5877,12 @@ task hisat2 { String index_id = basename(index_tar_gz, ".tar.gz") command <<< - mkdir index - tar -C index -xzf ~{index_tar_gz} + mkdir "~{index_id}" + tar -C "~{index_id}" --strip-components 2 -xzf "~{index_tar_gz}" hisat2 \ -p ~{threads} \ - ~{if defined(max_reads) then "-u ~{round(select_first([max_reads]))}" else ""} \ - -x index/grch38/genome \ + ~{if defined(max_reads) then "-u ~{select_first([max_reads])}" else ""} \ + -x "~{index_id}" \ --sra-acc ~{sra_acc} > ~{sra_acc}.sam >>> From d954a11f04b98f5fb070d472392c5bb3c2fedbbf Mon Sep 17 00:00:00 2001 From: jdidion Date: Thu, 25 Jul 2024 17:54:25 -0700 Subject: [PATCH 23/25] clarify --- SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 45109ecb..7ba0be9c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -871,7 +871,7 @@ task literals_paths { } ``` -Within a WDL file, literal values for files and directories may only be paths that are local to the execution environment. +Within a WDL file, the execution engine is only required to support literal values for files and directories that are paths local to the execution environment. A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional. From b9943180be9f07f7b0d00702e12c6e299e91df56 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 26 Jul 2024 09:06:09 -0700 Subject: [PATCH 24/25] fix test, make use of "host" vs "execution environment" consistent --- SPEC.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 7ba0be9c..4daf91ae 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3417,7 +3417,7 @@ A document is imported using it's [URI](https://en.wikipedia.org/wiki/Uniform_Re * `https://` * 🗑 `file://` - Using the `file://` protocol for local imports can be problematic. Its use is deprecated and will be removed in WDL 2.0. -In the event that there is no protocol specified, the import is resolved **relative to the location of the current document**. In the primary WDL document, a protocol-less import is relative to the host file system. If a protocol-less import starts with `/` it is interpreted as relative to the root of the host in the resolved URI. +In the event that there is no protocol specified, the import is resolved **relative to the location of the current document**. In the primary WDL document, a protocol-less import is relative to the folder that contains the primary WDL file. If a protocol-less import starts with `/` it is interpreted as relative to the root of the file system that contains the primary WDL file. Some examples of correct import resolution: @@ -5121,7 +5121,7 @@ The `disks` attribute provides a way to request one or more persistent volumes, If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task. -If a mount point is specified, then it must be an absolute path to a location in the host environment (i.e., within the container). The specified path either must not already exist in the host environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes. +If a mount point is specified, then it must be an absolute path to a location in the execution environment (i.e., within the container). The specified path either must not already exist in the execution environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes. The execution engine is free to provision any class(es) of persistent volume it has available (e.g., SSD or HDD). The [`disks` hint](#-disks) hint can be used to request specific attributes for the provisioned disks. @@ -7994,7 +7994,7 @@ File join_paths(File, Array[String]+) File join_paths(Array[String]+) ``` -Joins together two or more paths into an absolute path in the host filesystem. +Joins together two or more paths into an absolute path in the execution environment's filesystem. There are three variants of this function: @@ -10073,6 +10073,8 @@ Example output: ```json { + "test_transpose.out": [[0, 3], [1, 4], [2, 5]], + "test_transpose.expected": [[0, 3], [1, 4], [2, 5]], "test_transpose.is_true": true } ``` From 865b913eb5579160dc9003c2a093594861acdcb2 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 26 Jul 2024 09:41:33 -0700 Subject: [PATCH 25/25] suggest values for file/directory inputs and outputs in stadnard JSON formats --- SPEC.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/SPEC.md b/SPEC.md index 4daf91ae..9f58abe2 100644 --- a/SPEC.md +++ b/SPEC.md @@ -197,9 +197,11 @@ Revisions to this specification are made periodically in order to correct errors - [`length`](#length) - [Input and Output Formats](#input-and-output-formats) - [JSON Input Format](#json-input-format) + - [File/Directory Inputs](#filedirectory-inputs) - [Optional Inputs](#optional-inputs) - [Specifying / Overriding Requirements and Hints](#specifying--overriding-requirements-and-hints) - [JSON Output Format](#json-output-format) + - [File/Directory Outputs](#filedirectory-outputs) - [Extended File/Directory Input/Output Format](#extended-filedirectory-inputoutput-format) - [JSON Serialization of WDL Types](#json-serialization-of-wdl-types) - [Primitive Types](#primitive-types-1) @@ -11299,6 +11301,12 @@ Here is an example JSON input file for a workflow `wf`: WDL implementations are only required to support workflow execution, and not necessarily task execution, so a JSON input format for tasks is not specified. However, it is strongly suggested that if an implementation does support task execution, that it also supports this JSON input format for tasks. It is left to the discretion of the WDL implementation whether it is required to prefix the task input with the task name, i.e., `mytask.infile` vs. `infile`. +### File/Directory Inputs + +It is up to the execution engine to resolve input files and directories and stage them into the execution environment. The execution engine is free to specify the values that are allowed for `File` and `Directory` parameters, but at a minimum it is required to support POSIX absolute file paths (e.g., `/path/to/file`). + +It is strongly recommended that input files and directories be specified as absolute paths to local files or as URLs. If relative paths are allowed, then it is suggested that they be resolved relative to the directory that contains the input JSON file (if a file is provided) or to the working directory in which the workflow is initially launched. + ### Optional Inputs If a workflow has an optional input, its value may or may not be specified in the JSON input. It is also valid to explicitly set the value of an optional input to be undefined using JSON `null`. @@ -11400,6 +11408,12 @@ The output JSON will look like: It is recommended (but not required) that JSON outputs be "pretty printed" to be more human-readable. +### File/Directory Outputs + +It is up to the execution engine to provide workflow `File` and `Directory` outputs to the user that persist following a successful execution of the workflow. The execution engine is free to specify the values that are allowed for `File` and `Directory` parameters, but at a minimum it is required to support POSIX absolute file paths (e.g., `/path/to/file`). + +It is strongly recommended that output files and directories be specified as absolute paths to local files or as URLs. If relative paths are allowed, then it is suggested that they be resolved relative to the directory that contains the output JSON file (if a file is written) or to a single common directory containing all the workflow outputs. + ## Extended File/Directory Input/Output Format There is no guarantee that executing a workflow multiple times with the same input file or directory URIs will result in the same outputs. For example the contents of a file may change between one execution and the next, or a file may be added to or removed from a directory.