python-flavour cabs

[o-smirnov]

@caracal-pipeline/stimelators, another example of the new "flavour" features. Here's how to embed python in a cab.

The old pattern of command: (module)function is replaced by

  convolve_image:
    flavour: python
    command: image_utils.convolve_image
    inputs:
      image:
        dtype: File
        required: true
      size_arcsec:
        dtype: float
      size_pix:
        dtype: float
    outputs:
      outimage:
        dtype: File

The inputs and outputs describe the function signature, as before.

Here's a more advanced example, the python-code flavour treats the command as an embedded piece of Python code. Things to note here:

• the command: | thingy is a standard YaML syntax for embedding a multi-line string

• the example here shows how to embed an on-the-fly cab definition into a step. Basically, the cab field of a step can be a string (in which case it's looked up in global cabs), or it can be a mapping, which then should be a cab definition. The former case is useful when you want to reuse a cab in multiple places, the latter case is useful if you're just doing a quick one-off operation.

• inputs (and named file outputs) of a python-code flavour cab are (by default) passed to the code as local variables with the same names (except "-" is replaced by "_"). Outputs of such a cab are picked up from named variables after the code is executed (see the ms-info cab below for an example of such an output).

• not strictly related to Python, but note how the save-ms-info step saves stuff into a yml file, and is conditionally skipped if the file already exists. The ms-info step then reads this file. This seems to be a recurring pattern in some of your recipes (thinking of solarkat), so I thought I'd illustrate it here.

  steps:
    save-ms-info:
      cab:
        flavour: python-code
        name: save-ms-info
        command: |
          from omegaconf import OmegaConf
          import casacore.tables
          tab = casacore.tables.table(ms)
          info = OmegaConf.create()
          info.num_intervals = len(set(tab.getcol("TIME")))
          info.intervals = sorted(map(float, set(tab.getcol("INTERVAL"))))
          print(f"MS has {info.num_intervals} timeslots of interval {info.intervals}")
          info.total = [i*info.num_intervals/3600. for i in info.intervals]
          print(f"Total synthesis time is {info.total}h")
          OmegaConf.save(info, ms_info_file)
        inputs:
          ms: 
            dtype: MS
        outputs:
          ms-info-file:
            dtype: File
      params:
        ms: =recipe.ms
        ms-info-file: '{recipe.ms}.yml'
      skip: =EXISTS(current.ms-info-file)

    ms-info:
      cab:
        flavour: python-code
        command: |
          from omegaconf import OmegaConf
          info = OmegaConf.load(ms_info_file)
          num_intervals = info.num_intervals
        inputs:
          ms-info-file: 
            dtype: File
        outputs:
          num_intervals:
            dtype: int
      params:
        ms-info-file: =steps.save-ms-info.ms-info-file