From 1d2309b812bc70c6347183472bec86bb26e4ea9f Mon Sep 17 00:00:00 2001
From: jdidion
>>
output {
@@ -4404,10 +4404,10 @@ Given an `infile` value of `/path/to/file`, the execution engine will produce th
```sh
python <
Date: Wed, 26 Jun 2024 13:23:53 -0700
Subject: [PATCH 13/25] fix #667
---
SPEC.md | 32 +++++++++++++++++++++++---------
1 file changed, 23 insertions(+), 9 deletions(-)
diff --git a/SPEC.md b/SPEC.md
index 525abb4e..693316de 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -849,23 +849,37 @@ Single- and double-quotes do not need to be escaped within a multi-line string.
A `File` or `Directory` declaration may have have a string value indicating a relative or absolute path on the local file system.
-Within a WDL file, literal values for files may only be (relative or absolute) paths that are local to the execution environment. If the specified path does not exist, it is an error unless the declaration is optional.
-
```wdl
task literals_paths {
input {
- # If the user does not overide the value of `f1`, and /foo/bar.txt
- # does not exist, it is an error.
File f1 = "/foo/bar.txt"
-
- # If the user does not override the value of `f2` and /foo/bar.txt
- # does not exist, then `f2` is set to `None`.
- File? f2 = "/foo/bar.txt"
+ File? f2
}
+
+ command <<<
+ # If the user does not overide the value of `f1`, and /foo/bar.txt
+ # does not exist, an error will occur when the file is accessed here.
+ cat "~{f1}"
+
+ # If the user does not specify the value of `f2` it's value is `None`,
+ # which results in the empty-string when interpolated. `-f ""` is
+ # always false.
+ if [ -f "~{f2}" ]; then
+ echo "~{f2}"
+ fi
+ >>>
}
```
-An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path.
+Within a WDL file, literal values for files and directories may only be (relative or absolute) paths that are local to the execution environment.
+
+A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional.
+
+* To read from a path, the file/directory must exist and be accessible for reading (i.e., be assigned the appropriate permissions).
+* To write to a file, the path's parent directory must be accessible for writing.
+* To write to a directory, it must exist and be accessible for writing.
+
+An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path. Remote files must be treated as read-only. A remote file is only required to be vaild at the time that the execution engine needs to localize it.
#### Optional Types and None
From 3da88257bf44529e9c9641026288497fb1f74a4d Mon Sep 17 00:00:00 2001
From: jdidion
Date: Wed, 26 Jun 2024 13:25:25 -0700
Subject: [PATCH 14/25] update changelog
---
CHANGELOG.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d80d2180..594980df 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -92,7 +92,7 @@ version 1.2.0
version 1.1.3
---------------------------
-* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666,, #668). Thanks to @stxue1!
+* Fix issues with examples (#653, #654, #661, #662, #663, #664, #666, #667, #668). Thanks to @stxue1!
version 1.1.2
---------------------------
From 5bd0b0f25e311a64cfc993ed00adcb93ee62c942 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Tue, 23 Jul 2024 18:08:30 -0700
Subject: [PATCH 15/25] 672: clarify that mount points must be empty, writable,
and ephemeral
---
CHANGELOG.md | 2 ++
SPEC.md | 6 ++++--
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 594980df..89109d7f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -20,6 +20,8 @@ Keep the changelog pleasant to read in the text editor:
version 1.2.1
---------------------------
+* Clarify that `disks` mount points ephemeral and should not already exist
+
* Include fixes to examples introduced in v1.1.3
version 1.2.0
diff --git a/SPEC.md b/SPEC.md
index 693316de..d093b8b9 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -5099,9 +5099,11 @@ Test config:
* `Array[String]` - An array of disk specifications.
* Default value: `1 GiB`
-The `disks` attribute provides a way to request one or more persistent volumes, each of which has a minimum size and is mounted at a specific location. When the `disks` attribute is provided, the execution engine must guarantee the requested resources are available or immediately fail the task prior to instantiating the command.
+The `disks` attribute provides a way to request one or more persistent volumes, each of which has a minimum size and is mounted at a specific location with both read and write permissions. When the `disks` attribute is provided, the execution engine must guarantee the requested resources are available or immediately fail the task prior to instantiating the command.
-If a mount point is specified, then it must be an absolute path to a location in the host environment. If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task.
+If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task.
+
+If a mount point is specified, then it must be an absolute path to a location in the host environment (i.e., within the container). The specified path either must not already exist in the host environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes.
The execution engine is free to provision any class(es) of persistent volume it has available (e.g., SSD or HDD). The [`disks` hint](#-disks) hint can be used to request specific attributes for the provisioned disks.
From ea1cd0a64e46371eadb61b0423b2b542bc3a1919 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Tue, 23 Jul 2024 18:16:37 -0700
Subject: [PATCH 16/25] 671: fix python json serialization of string list
---
SPEC.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/SPEC.md b/SPEC.md
index d093b8b9..0a43fe93 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -9178,9 +9178,10 @@ task write_json {
command <<<
python <>>
From b92757327eeed1c1804f8a58db5938e720fa7cd0 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Wed, 24 Jul 2024 12:27:21 -0700
Subject: [PATCH 17/25] migrate changes from #674
---
SPEC.md | 149 +++++++++++++++++++++++++++++++-------------------------
1 file changed, 82 insertions(+), 67 deletions(-)
diff --git a/SPEC.md b/SPEC.md
index 0a43fe93..94f6850e 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -391,7 +391,7 @@ WDL also provides features for implementing more complex workflows. For example,
```json
{
- "hello.all_matches": [["hi_world"], ["hi_pal"]]
+ "hello_parallel.all_matches": [["hi_world"], ["hi_pal"]]
}
```
@@ -930,7 +930,8 @@ An optional declaration has a default initialization of `None`, which indicates
"optionals.test_defined": false,
"optionals.test_defined2": true,
"optionals.test_is_none": true,
- "optionals.test_not_none": false
+ "optionals.test_not_none": false,
+ "optionals.test_non_equal": true
}
```
@@ -1042,7 +1043,7 @@ task sum {
}
command <<<
- printf ~{sep(" ", ints)} | awk '{tot=0; for(i=1;i<=NF;i++) tot+=$i; print tot}'
+ printf "~{sep(" ", ints)}" | awk '{tot=0; for(i=1;i<=NF;i++) tot+=$i; print tot}'
>>>
output {
@@ -1107,8 +1108,8 @@ Example output:
{
"non_empty_optional.nonempty1": [0.0],
"non_empty_optional.nonempty2": [null, 1],
- "non_empty_optional.nonempty3": [],
- "non_empty_optional.nonempty4": [0.0]
+ "non_empty_optional.nonempty3": null,
+ "non_empty_optional.nonempty4": [0]
}
```
@@ -1974,7 +1975,7 @@ task count_lines {
}
command <<<
- wc -l ~{write_lines(array)}
+ wc -l < ~{write_lines(array)}
>>>
output {
@@ -2466,7 +2467,7 @@ version 1.2
struct Experiment {
String id
Array[String] variables
- Map[String, Float] data
+ Map[String, String] data
}
workflow nested_access {
@@ -2499,7 +2500,7 @@ Example input:
"variables": ["name", "height"],
"data": {
"name": "Pinky",
- "height": 7
+ "height": "7"
}
},
{
@@ -2507,7 +2508,7 @@ Example input:
"variables": ["name", "weight"],
"data": {
"name": "Porky",
- "weight": 1000
+ "weight": "1000"
}
}
]
@@ -2676,7 +2677,7 @@ Example input:
{
"placeholders.start": "h",
"placeholders.end": "o",
- "placeholders.input": "hello"
+ "placeholders.instr": "hello"
}
```
@@ -2684,7 +2685,8 @@ Example output:
```json
{
- "placeholders.cmd": "grep 'h...o' hello"
+ "placeholders.cmd": "grep 'h...o' hello",
+ "placehoders.s": "4"
}
```
@@ -2805,12 +2807,15 @@ Example: placeholder_coercion.wdl
version 1.2
workflow placeholder_coercion {
- File x = "/hij"
+ input {
+ File x
+ }
+ String x_as_str = x
Int? i = None
output {
Boolean is_true1 = "~{"abc"}" == "abc"
- Boolean is_true2 = "~{x}" == "/hij"
+ Boolean is_true2 = "~{x}" == x_as_str
Boolean is_true3 = "~{5}" == "5"
Boolean is_true4 = "~{3.141}" == "3.141000"
Boolean is_true5 = "~{3.141 * 1E-10}" == "0.000000"
@@ -2824,7 +2829,9 @@ workflow placeholder_coercion {
Example input:
```json
-{}
+{
+ "placeholder_coercion.x": "hello.txt"
+}
```
Example output:
@@ -2973,7 +2980,7 @@ Example output:
```json
{
- "flags.num_matches": 2
+ "flags.num_matches": "2"
}
```
@@ -3321,7 +3328,7 @@ Example input:
```json
{
- "person_struct.person": {
+ "greet_person.person": {
"name": {
"first": "Richard",
"last": "Rich"
@@ -3342,7 +3349,7 @@ Example output:
```json
{
- "person_struct.message": "Hello Richard! You have 1 test result(s) available.\nPlease transfer USD 500 to continue"
+ "greet_person.message": "Hello Richard! You have 1 test result(s) available.\nPlease transfer USD 500 to continue"
}
```
@@ -3482,6 +3489,7 @@ task calculate_bill {
workflow import_structs {
input {
+ File infile
Person doctor = Person {
age: 10,
name: Name {
@@ -3506,12 +3514,12 @@ workflow import_structs {
period: "hourly"
},
assay_data: {
- "glucose": "hello.txt"
+ "glucose": infile
}
}
}
- call person_struct.greet_person {
+ call person_struct_task.greet_person {
person = patient
}
@@ -3529,14 +3537,16 @@ workflow import_structs {
Example input:
```json
-{}
+{
+ "import_structs.infile": "hello.txt"
+}
```
Example output:
```json
{
- "import_structs.bill": 175000
+ "import_structs.bill": 175000.0
}
```
@@ -4249,7 +4259,7 @@ task test_placeholders {
# The `read_lines` function reads the lines from a file into an
# array. The `sep` function concatenates the lines with a space
# (" ") delimiter. The resulting string is then printed to stdout.
- printf ~{sep(" ", read_lines(infile))}
+ printf "~{sep(" ", read_lines(infile))}"
>>>
output {
@@ -4461,8 +4471,7 @@ Example input:
```json
{
- "outputs.t": 5,
- "outputs.write_outstr": false
+ "outputs.t": 5
}
```
@@ -4557,7 +4566,7 @@ task glob {
}
command <<<
- for i in 1..~{num_files}; do
+ for i in {1..~{num_files}}; do
printf ${i} > file_${i}.txt
done
>>>
@@ -4612,7 +4621,7 @@ task relative_and_absolute {
>>>
output {
- File something = read_string("my/path/to/something.txt")
+ String something = read_string("my/path/to/something.txt")
File bashrc = "/root/.bashrc"
}
@@ -4663,7 +4672,7 @@ task optional_output {
command <<<
printf "1" > example1.txt
- if ~{make_example2}; do
+ if ~{make_example2}; then
printf "2" > example2.txt
fi
>>>
@@ -4691,6 +4700,8 @@ Example output:
```json
{
"optional_output.example2": null,
+ "optional_output.example1": "example1.txt",
+ "optional_output.file_array": ["example1.txt", null],
"optional_output.file_array_len": 1
}
```
@@ -4789,7 +4800,7 @@ task dynamic_container {
>>>
output {
- String is_true = ubuntu_version == read_string(stdout())
+ Boolean is_true = ubuntu_version == read_string(stdout())
}
requirements {
@@ -5389,7 +5400,7 @@ task test_hints {
}
command <<<
- wc -l ~{foo}
+ wc -l < ~{foo}
>>>
output {
@@ -5710,11 +5721,11 @@ task ex_paramter_meta {
}
command <<<
- wc ~{if lines_only then '-l' else ''} ~{infile}
+ wc ~{if lines_only then '-l' else ''} < ~{infile}
>>>
output {
- String result = stdout()
+ Int result = read_int(stdout())
}
requirements {
@@ -5737,7 +5748,7 @@ Example output:
```json
{
- "ex_paramter_meta.result": "3"
+ "ex_paramter_meta.result": 2
}
```
@@ -5848,7 +5859,7 @@ version 1.2
task hisat2 {
input {
- File index
+ File index_tar_gz
String sra_acc
Int? max_reads
Int threads = 8
@@ -5856,15 +5867,15 @@ task hisat2 {
Float disk_size_gb = 100
}
- String index_id = basename(index, ".tar.gz")
+ String index_id = basename(index_tar_gz, ".tar.gz")
command <<<
mkdir index
- tar -C index -xzf ~{index}
+ tar -C index -xzf ~{index_tar_gz}
hisat2 \
-p ~{threads} \
- ~{if defined(max_reads) then "-u ~{select_first([max_reads])}" else ""} \
- -x index/~{index_id} \
+ ~{if defined(max_reads) then "-u ~{round(select_first([max_reads]))}" else ""} \
+ -x index/grch38/genome \
--sra-acc ~{sra_acc} > ~{sra_acc}.sam
>>>
@@ -5884,7 +5895,7 @@ task hisat2 {
}
parameter_meta {
- index: "Gzipped tar file with HISAT2 index files"
+ index_tar_gz: "Gzipped tar file with HISAT2 index files"
sra_acc: "SRA accession number or reads to align"
}
}
@@ -6216,7 +6227,7 @@ task echo {
}
command <<<
- printf ~{msg}
+ printf '~{msg}\n'
>>>
output {
@@ -6284,7 +6295,7 @@ task foobar {
}
command <<<
- wc -l ~{infile}
+ wc -l < ~{infile}
>>>
output {
@@ -6326,7 +6337,7 @@ Example output:
```json
{
- "other.results": 3
+ "other.results": 2
}
```
@@ -6556,8 +6567,8 @@ task repeat {
echo "i must be >= 1"
exit 1
fi
- for i in 1..~{i}; do
- printf ~{select_first([opt_string, "default"])}
+ for i in {1..~{i}}; do
+ printf '~{select_first([opt_string, "default"])}\n'
done
>>>
@@ -7061,7 +7072,7 @@ workflow nested_scatter {
Array[String] salutations = ["Hello", "Goodbye"]
}
- Array[String] honorifics = ["Wizard", "Mr."]
+ Array[String] honorifics = ["Mr.", "Wizard"]
# the zip() function creates an array of pairs
Array[Pair[String, String]] name_pairs = zip(first_names, last_names)
@@ -7141,7 +7152,8 @@ Example output:
["Hello Mr. Merry, how are you?", "Hello Mr. Merry Brandybuck, how are you?"],
["Goodbye Mr. Merry, how are you?", "Goodbye Mr. Merry Brandybuck, how are you?"]
]
- ]
+ ],
+ "nested_scatter.used_honorifics": ["Mr.;", "Wizard", "Mr."]
}
```
@@ -7228,7 +7240,8 @@ Example output:
```json
{
"test_conditional.result_array": [4, 6, 8, 10],
- "test_conditional.maybe_result2": [0, 4, 6, 8, 10]
+ "test_conditional.maybe_result2": [0, 4, 6, 8, 10],
+ "test_conditional.j_out": 2
}
```
@@ -7419,7 +7432,7 @@ Example output:
```json
{
- "test_floor.all_true": true
+ "test_floor.all_true": [true, true]
}
```
@@ -7474,7 +7487,7 @@ Example output:
```json
{
- "test_ceil.all_true": true
+ "test_ceil.all_true": [true, true]
}
```
@@ -7529,7 +7542,7 @@ Example output:
```json
{
- "test_round.all_true": true
+ "test_round.all_true": [true, true]
}
```
@@ -7632,8 +7645,8 @@ workflow test_max {
output {
# these two expressions are equivalent
- Float min1 = if value1 > value2 then value1 else value2
- Float min2 = max(value1, value2)
+ Float max1 = if value1 > value2 then value1 else value2
+ Float max2 = max(value1, value2)
}
}
```
@@ -7652,8 +7665,8 @@ Example output:
```json
{
- "test_max.min1": 1.0,
- "test_max.min2": 1.0
+ "test_max.max1": 2.0,
+ "test_max.max2": 2.0
}
```
@@ -7813,7 +7826,7 @@ workflow test_sub {
String chocoearly = sub(chocolike, "late", "early") # I like chocoearly when\nit's early
String chocolate = sub(chocolike, "late$", "early") # I like chocolate when\nit's early
String chocoearlylate = sub(chocolike, "[^ ]late", "early") # I like chocearly when\nit's late
- String choco4 = sub(chocolike, " [:alpha:]{4} ", " 4444 ") # I 4444 chocolate 4444\nit's late
+ String choco4 = sub(chocolike, " [:alpha:]{4} ", " 4444 ") # I 4444 chocolate when\nit's late
String no_newline = sub(chocolike, "\\n", " ") # "I like chocolate when it's late"
}
}
@@ -8087,7 +8100,7 @@ task gen_files {
}
command <<<
- for i in 1..~{num_files}; do
+ for i in {1..~{num_files}}; do
printf ${i} > a_file_${i}.txt
done
mkdir a_dir
@@ -8241,7 +8254,7 @@ task echo_stdout {
command <<< printf "hello world" >>>
output {
- File message = read_string(stdout())
+ String message = read_string(stdout())
}
}
```
@@ -8286,7 +8299,7 @@ task echo_stderr {
command <<< >&2 printf "hello world" >>>
output {
- File message = read_string(stderr())
+ String message = read_string(stderr())
}
}
```
@@ -8919,8 +8932,8 @@ version 1.2
task read_map {
command <<<
- printf "key1\tvalue1\n" >> map_file
- printf "key2\tvalue2\n" >> map_file
+ printf "key1\tvalue1\n"
+ printf "key2\tvalue2\n"
>>>
output {
@@ -9718,7 +9731,7 @@ workflow test_suffix {
Array[Int] env2 = [1, 2, 3]
output {
- Array[String] env1_suffix = suffix(".txt ", env1)
+ Array[String] env1_suffix = suffix(".txt", env1)
Array[String] env2_suffix = suffix(".0", env2)
}
}
@@ -9968,7 +9981,7 @@ task double {
command <<< >>>
output {
- Int d = n * n
+ Int d = 2 * n
}
}
@@ -9993,7 +10006,7 @@ Example input:
```json
{
- "test_range.n": 5
+ "test_range.i": 5
}
```
@@ -10035,7 +10048,9 @@ workflow test_transpose {
Array[Array[Int]] expected_output_array = [[0, 3], [1, 4], [2, 5]]
output {
- Boolean is_true = transpose(input_array) == expected_output_array
+ Array[Array[Int]] out = transpose(input_array)
+ Array[Array[Int]] expected = expected_output_array
+ Boolean is_true = out == expected
}
}
```
@@ -12179,7 +12194,7 @@ task grep2 {
opts=( ~{sep(" ", quote(opts_and_values.left))} )
values=( ~{sep(" ", quote(opts_and_values.right))} )
command="grep"
- for i in 1..~{n}; do
+ for i in {0..~{n-1}}; do
command="$command ${opts[i]}"="${values[i]}"
done
$command ~{pattern} ~{infile}
@@ -12262,7 +12277,7 @@ task serde_map_tsv {
>>>
output {
- Map[String, String] new_items = read_map("lines")
+ Map[String, String] new_items = read_map(stdout())
}
}
```
@@ -12333,7 +12348,7 @@ version 1.2
task serde_map_json {
input {
- Map[String, Float] read_quality_scores
+ Map[String, Int] read_quality_scores
}
command <<<
@@ -12349,7 +12364,7 @@ task serde_map_json {
>>>
output {
- Map[String, Float] ascii_values = read_json(stdout())
+ Map[String, Int] ascii_values = read_json(stdout())
}
requirements {
From 91f2788e9d045334a8441602dcfc1418dd4293fc Mon Sep 17 00:00:00 2001
From: jdidion
Date: Wed, 24 Jul 2024 12:33:52 -0700
Subject: [PATCH 18/25] fix
---
SPEC.md | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/SPEC.md b/SPEC.md
index 94f6850e..6e6b9de6 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -1590,14 +1590,17 @@ Example: string_to_file.wdl
version 1.2
workflow string_to_file {
- String path1 = "/path/to/file"
- File path2 = "/path/to/file"
+ input {
+ File infile
+ }
+
+ String path1 = "~{infile}"
# valid - String coerces unambiguously to File
- File path3 = path1
+ File path2 = path1
output {
- Boolean paths_equal = path2 == path3
+ Boolean paths_equal = path1 == path2
}
}
```
@@ -1606,7 +1609,9 @@ workflow string_to_file {
Example input:
```json
-{}
+{
+ "string_to_file.infile": "hello.txt"
+}
```
Example output:
@@ -2282,8 +2287,8 @@ Example output:
{
"array_map_equality.is_true1": true,
"array_map_equality.is_true2": true,
- "array_map_equality.is_false1": true,
- "array_map_equality.is_false2": true
+ "array_map_equality.is_false1": false,
+ "array_map_equality.is_false2": false
}
```
From 830d074dbef2aaaf071849441a3118e8a8484f50 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Wed, 24 Jul 2024 19:27:30 -0700
Subject: [PATCH 19/25] disallow relative path literals
---
SPEC.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/SPEC.md b/SPEC.md
index 6e6b9de6..72a4ff43 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -871,7 +871,7 @@ task literals_paths {
}
```
-Within a WDL file, literal values for files and directories may only be (relative or absolute) paths that are local to the execution environment.
+Within a WDL file, literal values for files and directories may only be paths that are local to the execution environment.
A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional.
@@ -879,6 +879,8 @@ A path is only required to be valid if and when it is accessed. A path assigned
* To write to a file, the path's parent directory must be accessible for writing.
* To write to a directory, it must exist and be accessible for writing.
+Path literals should be absolute paths, since the execution engine is free to set the working directory of a task execution, and thus relative paths may not exist at runtime. 🗑 Use of relative path literals when defining input and private declarations is currently allowed but is deprecated and will be disallowed in WDL 2.0.
+
An execution engine may support [other ways](#input-and-output-formats) to specify `File` and `Directory` inputs (e.g., as URIs), but prior to task execution it must [localize inputs](#task-input-localization) so that the runtime value of a `File`/`Directory` variable is a local path. Remote files must be treated as read-only. A remote file is only required to be vaild at the time that the execution engine needs to localize it.
#### Optional Types and None
From a38d1610d52f42ab9e341ff83a355893b641aaba Mon Sep 17 00:00:00 2001
From: jdidion
Date: Wed, 24 Jul 2024 19:28:14 -0700
Subject: [PATCH 20/25] update changelog
---
CHANGELOG.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 89109d7f..b75f4357 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,8 @@ version 1.2.1
* Clarify that `disks` mount points ephemeral and should not already exist
+* Deprecate the use of relative path literals in input and private variable declarations.
+
* Include fixes to examples introduced in v1.1.3
version 1.2.0
From fcd5751322444928538798d0065e1575b1b804cf Mon Sep 17 00:00:00 2001
From: Mark Woon
Date: Thu, 25 Jul 2024 11:29:45 -0700
Subject: [PATCH 21/25] Fix typo
---
SPEC.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/SPEC.md b/SPEC.md
index dfc4c5a4..b2f64393 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -3267,7 +3267,7 @@ struct Person {
description: "Encapsulates data about a person"
}
- paramter_meta {
+ parameter_meta {
name: "The person's name"
age: "The person's age"
income: "How much the person makes (optional)"
From 5f650f4ebfa02cdf03135d0cb076750f80fff1d6 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Thu, 25 Jul 2024 17:51:09 -0700
Subject: [PATCH 22/25] use index_id rather than hard-coded path
---
SPEC.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/SPEC.md b/SPEC.md
index 76a1f096..45109ecb 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -5877,12 +5877,12 @@ task hisat2 {
String index_id = basename(index_tar_gz, ".tar.gz")
command <<<
- mkdir index
- tar -C index -xzf ~{index_tar_gz}
+ mkdir "~{index_id}"
+ tar -C "~{index_id}" --strip-components 2 -xzf "~{index_tar_gz}"
hisat2 \
-p ~{threads} \
- ~{if defined(max_reads) then "-u ~{round(select_first([max_reads]))}" else ""} \
- -x index/grch38/genome \
+ ~{if defined(max_reads) then "-u ~{select_first([max_reads])}" else ""} \
+ -x "~{index_id}" \
--sra-acc ~{sra_acc} > ~{sra_acc}.sam
>>>
From d954a11f04b98f5fb070d472392c5bb3c2fedbbf Mon Sep 17 00:00:00 2001
From: jdidion
Date: Thu, 25 Jul 2024 17:54:25 -0700
Subject: [PATCH 23/25] clarify
---
SPEC.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/SPEC.md b/SPEC.md
index 45109ecb..7ba0be9c 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -871,7 +871,7 @@ task literals_paths {
}
```
-Within a WDL file, literal values for files and directories may only be paths that are local to the execution environment.
+Within a WDL file, the execution engine is only required to support literal values for files and directories that are paths local to the execution environment.
A path is only required to be valid if and when it is accessed. A path assigned to an input or private declaration is only accessed if it is referred to in the `command` or `output` sections. A path assigned to an output declaration must be valid unless the declaration is optional.
From b9943180be9f07f7b0d00702e12c6e299e91df56 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Fri, 26 Jul 2024 09:06:09 -0700
Subject: [PATCH 24/25] fix test, make use of "host" vs "execution environment"
consistent
---
SPEC.md | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/SPEC.md b/SPEC.md
index 7ba0be9c..4daf91ae 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -3417,7 +3417,7 @@ A document is imported using it's [URI](https://en.wikipedia.org/wiki/Uniform_Re
* `https://`
* 🗑 `file://` - Using the `file://` protocol for local imports can be problematic. Its use is deprecated and will be removed in WDL 2.0.
-In the event that there is no protocol specified, the import is resolved **relative to the location of the current document**. In the primary WDL document, a protocol-less import is relative to the host file system. If a protocol-less import starts with `/` it is interpreted as relative to the root of the host in the resolved URI.
+In the event that there is no protocol specified, the import is resolved **relative to the location of the current document**. In the primary WDL document, a protocol-less import is relative to the folder that contains the primary WDL file. If a protocol-less import starts with `/` it is interpreted as relative to the root of the file system that contains the primary WDL file.
Some examples of correct import resolution:
@@ -5121,7 +5121,7 @@ The `disks` attribute provides a way to request one or more persistent volumes,
If the mount point is omitted, it is assumed to be a persistent volume mounted at the root of the execution directory within a task.
-If a mount point is specified, then it must be an absolute path to a location in the host environment (i.e., within the container). The specified path either must not already exist in the host environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes.
+If a mount point is specified, then it must be an absolute path to a location in the execution environment (i.e., within the container). The specified path either must not already exist in the execution environment, or it must be empty and have at least the requested amount of space available. The mount point should be assumed to be ephemeral, i.e., it will be deleted after the task completes.
The execution engine is free to provision any class(es) of persistent volume it has available (e.g., SSD or HDD). The [`disks` hint](#-disks) hint can be used to request specific attributes for the provisioned disks.
@@ -7994,7 +7994,7 @@ File join_paths(File, Array[String]+)
File join_paths(Array[String]+)
```
-Joins together two or more paths into an absolute path in the host filesystem.
+Joins together two or more paths into an absolute path in the execution environment's filesystem.
There are three variants of this function:
@@ -10073,6 +10073,8 @@ Example output:
```json
{
+ "test_transpose.out": [[0, 3], [1, 4], [2, 5]],
+ "test_transpose.expected": [[0, 3], [1, 4], [2, 5]],
"test_transpose.is_true": true
}
```
From 865b913eb5579160dc9003c2a093594861acdcb2 Mon Sep 17 00:00:00 2001
From: jdidion
Date: Fri, 26 Jul 2024 09:41:33 -0700
Subject: [PATCH 25/25] suggest values for file/directory inputs and outputs in
stadnard JSON formats
---
SPEC.md | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/SPEC.md b/SPEC.md
index 4daf91ae..9f58abe2 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -197,9 +197,11 @@ Revisions to this specification are made periodically in order to correct errors
- [`length`](#length)
- [Input and Output Formats](#input-and-output-formats)
- [JSON Input Format](#json-input-format)
+ - [File/Directory Inputs](#filedirectory-inputs)
- [Optional Inputs](#optional-inputs)
- [Specifying / Overriding Requirements and Hints](#specifying--overriding-requirements-and-hints)
- [JSON Output Format](#json-output-format)
+ - [File/Directory Outputs](#filedirectory-outputs)
- [Extended File/Directory Input/Output Format](#extended-filedirectory-inputoutput-format)
- [JSON Serialization of WDL Types](#json-serialization-of-wdl-types)
- [Primitive Types](#primitive-types-1)
@@ -11299,6 +11301,12 @@ Here is an example JSON input file for a workflow `wf`:
WDL implementations are only required to support workflow execution, and not necessarily task execution, so a JSON input format for tasks is not specified. However, it is strongly suggested that if an implementation does support task execution, that it also supports this JSON input format for tasks. It is left to the discretion of the WDL implementation whether it is required to prefix the task input with the task name, i.e., `mytask.infile` vs. `infile`.
+### File/Directory Inputs
+
+It is up to the execution engine to resolve input files and directories and stage them into the execution environment. The execution engine is free to specify the values that are allowed for `File` and `Directory` parameters, but at a minimum it is required to support POSIX absolute file paths (e.g., `/path/to/file`).
+
+It is strongly recommended that input files and directories be specified as absolute paths to local files or as URLs. If relative paths are allowed, then it is suggested that they be resolved relative to the directory that contains the input JSON file (if a file is provided) or to the working directory in which the workflow is initially launched.
+
### Optional Inputs
If a workflow has an optional input, its value may or may not be specified in the JSON input. It is also valid to explicitly set the value of an optional input to be undefined using JSON `null`.
@@ -11400,6 +11408,12 @@ The output JSON will look like:
It is recommended (but not required) that JSON outputs be "pretty printed" to be more human-readable.
+### File/Directory Outputs
+
+It is up to the execution engine to provide workflow `File` and `Directory` outputs to the user that persist following a successful execution of the workflow. The execution engine is free to specify the values that are allowed for `File` and `Directory` parameters, but at a minimum it is required to support POSIX absolute file paths (e.g., `/path/to/file`).
+
+It is strongly recommended that output files and directories be specified as absolute paths to local files or as URLs. If relative paths are allowed, then it is suggested that they be resolved relative to the directory that contains the output JSON file (if a file is written) or to a single common directory containing all the workflow outputs.
+
## Extended File/Directory Input/Output Format
There is no guarantee that executing a workflow multiple times with the same input file or directory URIs will result in the same outputs. For example the contents of a file may change between one execution and the next, or a file may be added to or removed from a directory.