equivalent of for x,y in zip(a,b)

[o-smirnov]

From @landmanbester:

Is there some kind of zip feature for the loop? I mean if I want to scatter over observations but I need to pass in multiple
observation specific variables (eg. output directories and reference antennas)

In lieu of a better solution, this will do the trick:

recipe:
  assign_based_on:
    obs:
      a1: 
        ms: ms1.ms
        dir.out: dir1
      a2: 
        ms: ms2.ms
        dir.out: dir2
      ...
  for_loop:
    var: obs
    over: [a1, a2, a3]

[o-smirnov]

Open to more elegant suggestions... the need for a zip-like thingy had occurred to me before, but I'm not settled on whether a more explicit solution is needed or what it should look like.

[o-smirnov]

multiple observation specific variables

Actually for your particular use case the above construct is perfect. I used it in the PARROT recipes: https://github.com/ratt-ru/parrot-stew-recipes/blob/parrot1/parrot-observation-sets.yml#L15

It's perfect because then you have a single-observation recipe you can run standalone (with obs=foo from the command line), and then a separate "outer loop" recipe a-la https://github.com/ratt-ru/parrot-stew-recipes/blob/parrot1/jove-pol-loop.yml.

@Athanaseus has used something similar in his MGPLS recipes, maybe he can link those in here.

[landmanbester]

This is exactly what I need, thanks

[landmanbester]

I made a toy recipe to try out the above but I can't seem to get it working. I'm pretty sure there is a bug in the way assign_based_on is treated in a for loop (see below). Here is the recipe

cabs:
  print_inputs:
    flavour: python-code
    command: |
      print(ms)
      print(out)

    inputs:
      ms:
        dtype: str
        required: true
      out:
        dtype: str
        required: true

init_loop:
  info: xds initialisation loop

  assign_based_on:
    obs:
      F001:
        ms: test1
        out: out1

      F002:
        ms: test2
        out: out2

  assign:
    obs: F001

  for_loop:
    var: obs
    over: [F001, F002]

  inputs:
    input1:
      dtype: int
      default: 1
    input2:
      dtype: int
      default: 2

  steps:
    test:
      cab: print_inputs
      params:
        ms: =recipe.obs.ms
        out: =recipe.obs.out

This is supposed to loop over obs with obs taking values F001 and F002 each of which should have ms and out attributes. Some things to note:

• if the assign statement is missing from the recipe level stimela falls over with

init_loop.assign_based_on: 'obs' is not a known variable

I think this is a bug, the assign_based_on happens before the loop is triggered so obs does not exist yet.

• Running it as is results in the following

stimela run test.yaml init_loop
2024-01-26 12:58:07 STIMELA INFO: starting
2024-01-26 12:58:07 STIMELA INFO: added include paths: /home/landman/software/stimecipes
2024-01-26 12:58:07 STIMELA INFO: loaded full configuration from cache
2024-01-26 12:58:07 STIMELA INFO: saving config dependencies to ./stimela.config.deps
2024-01-26 12:58:07 STIMELA INFO: loading recipe/config test.yaml
2024-01-26 12:58:07 STIMELA INFO: test.yaml contains the following recipe sections: 'init_loop'
2024-01-26 12:58:07 STIMELA INFO: selected recipe is 'init_loop'
──────────────────────────────────────────────────── prevalidation ─────────────────────────────────────────────────────
2024-01-26 12:58:07 STIMELA INFO: pre-validating the recipe
2024-01-26 12:58:08 STIMELA INFO: recipe logs will be saved under .
2024-01-26 12:58:08 STIMELA INFO: saved recipe dependencies to ./stimela.recipe.deps
────────────────────────────────────────────────────── init_loop ───────────────────────────────────────────────────────
2024-01-26 12:58:08 STIMELA.init_loop INFO: ### validated inputs
2024-01-26 12:58:08 STIMELA.init_loop INFO: recipe 'init_loop':
2024-01-26 12:58:08 STIMELA.init_loop INFO:   input1 = 1
2024-01-26 12:58:08 STIMELA.init_loop INFO:   input2 = 2
2024-01-26 12:58:08 STIMELA.init_loop INFO: running recipe 'init_loop'
2024-01-26 12:58:08 STIMELA.init_loop INFO: for loop iteration 0: obs = F001
for loop iteration 0: obs = F001
2024-01-26 12:58:08 STIMELA.init_loop INFO: processing step 'test'
──────────────────────────────────────────────────── init_loop.test ────────────────────────────────────────────────────
2024-01-26 12:58:08 STIMELA.init_loop.test INFO: ### validated inputs
2024-01-26 12:58:08 STIMELA.init_loop.test INFO: cab print(ms):
2024-01-26 12:58:08 STIMELA.init_loop.test INFO:   ms = Unresolved(recipe.obs.ms)
2024-01-26 12:58:08 STIMELA.init_loop.test INFO:   out = Unresolved(recipe.obs.out)
2024-01-26 12:58:08 STIMELA.init_loop.test ERROR: step 'test': invalid inputs: 'ms', 'out'
────────────────────────────────────────────────── profiling results ───────────────────────────────────────────────────

                                                  averages & total I/O
                 ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━┓
                 ┃                ┃  time hms ┃ CPU % ┃ Mem GB ┃ Load ┃ R GB/s ┃ W GB/s ┃ R GB ┃ W GB ┃
                 ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━┩
                 │ init_loop      │ 0:00:00.0 │ 13.53 │   3.00 │ 0.25 │   0.00 │   0.00 │ 0.00 │ 0.00 │
                 │ init_loop.1/2  │ 0:00:00.0 │ 13.90 │   3.00 │ 0.25 │   0.00 │   0.00 │ 0.00 │ 0.00 │
                 │ init_loop.test │ 0:00:00.0 │ 14.30 │   3.00 │ 0.25 │   0.00 │   0.00 │ 0.00 │ 0.00 │
                 └────────────────┴───────────┴───────┴────────┴──────┴────────┴────────┴──────┴──────┘

                                                  peaks
                 ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┓
                 ┃                ┃  time hms ┃ CPU % ┃ Mem GB ┃ Load ┃ R GB/s ┃ W GB/s ┃
                 ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━┩
                 │ init_loop      │ 0:00:00.0 │ 16.70 │   3.00 │ 0.25 │   0.00 │   0.00 │
                 │ init_loop.1/2  │ 0:00:00.0 │ 16.70 │   3.00 │ 0.25 │   0.00 │   0.00 │
                 │ init_loop.test │ 0:00:00.0 │ 14.30 │   3.00 │ 0.25 │   0.00 │   0.00 │
                 └────────────────┴───────────┴───────┴────────┴──────┴────────┴────────┘

2024-01-26 12:58:08 STIMELA.init_loop INFO: saved full profiling stats to ./stimela.stats.full
2024-01-26 12:58:08 STIMELA.init_loop INFO: saved summary to ./stimela.stats.summary.txt
2024-01-26 12:58:08 STIMELA ERROR: run failed after 0:00:00
──────────────────────────────────────────── detailed error report follows ─────────────────────────────────────────────
        ⚠ run failed after 0:00:00
        └── step 'init_loop.test' has failed, aborting the recipe
            └── step 'test': invalid inputs: 'ms', 'out'
2024-01-26 12:58:08 STIMELA.init_loop INFO: last log directory was .

So obs doesn't seem to have the necessary attributes.

• If I remove the .ms and .out from obs in the params section it runs through printing

### full command line is python -u -c import sys, json
_params = json.loads(sys.argv[1])
ms = _params["ms"]
out = _params["out"]
print(ms)
print(out)
 {"ms": "F002", "out": "F002"}
# F002
# F002

for each iteration. So it seems obs is getting parsed as string and not as intended by assign_based_on. Any ideas how to get this working @o-smirnov ?

[o-smirnov]

This:

        ms: =recipe.obs.ms
        out: =recipe.obs.out

should just be

        ms: =recipe.ms
        out: =recipe.out

[landmanbester]

Awesome, that works. Thanks!