Check whether SOF files are consistent with DRLD

[hugobuddel]

Check whether the names and contents of the SOF files match the DRLD. Also fixes many inconsistencies found with the DRLD.

@JenniferKarr could you please verify these changes?

@sesquideus can you ensure that the recipes will work with these new SOF files?

I also renamed the files such that now the start of the name is the actual name of the recipe. If there is any specifier, like that the recipe is ran for a specific band, or for standard or science observations, then that is indicated after the recipe name. Like:

metis_det_dark.ifu.sof
metis_det_dark.lm.sof
metis_det_dark.n.sof

or

metis_ifu_telluric.sci.sof
metis_ifu_telluric.std.sof

@JenniferKarr are these names OK?

The script still complains about these things:

Some tags are not in the DRLD:
- metis_ifu_adi_cgrph.sof
  - IFU_APP_SCI_THROUGHPUT
- metis_lm_img_background.sof
  - LM_SKY_BASIC_REDUCED

Some tags in SOF files are not actually input to the recipe:
- metis_ifu_adi_cgrph.sof
  - IFU_APP_SCI_THROUGHPUT
- metis_N_lss_sci.sof
  - STD_TRANSMISSION
- metis_lm_img_background.sof
  - LM_SKY_BASIC_REDUCED

But these might actually be errors in the DRLD. What do you think @JenniferKarr ?

[hugobuddel]

I'm merging this, because it is already leading to problems to have this open. @JenniferKarr could you please still review this retroactively? I'll also merge AstarVienna/METIS_Pipeline_Test_Data#4 to update the test data

[JenniferKarr]

The new naming system looks fine.

I took a look at the DRLD, and:

IFU_cgrph_SCI_THROUGHPUT is listed as input for the recipe metis_ifu_adi_cgrph

STD_TRANSMISSION is listed as optional input for metis_LM_lss_std but not metis_N_lss_std, so that is either a difference in the recipes or something that needs to be updated in the DRLD, I'm not sure which.

The sky frames are a case of needing to update the DRLD.

[hugobuddel]

IFU_cgrph_SCI_THROUGHPUT is listed as input for the recipe metis_ifu_adi_cgrph

I thought that cgrph excludes the APP. As for example, we have both LM_APP_SCI_THROUGHPUT and LM_cgrph_SCI_THROUGHPUT defined. So I expect that metis_ifu_adi_cgrph would accept any IFU_cgrph_SCI_THROUGHPUT except IFU_APP_SCI_THROUGHPUT.

But I don't know what values cgrph actually can have, and maybe I misunderstand things completely.

It doesn't really matter for this release, as we currently have no way to run these recipes anyway because the recipes earlier in the pipeline do not exist.

[gotten]

The focal plane coronagraphs are RAVC, CVC and the backup CLC.
The pupil plane coronagraphs are APP and the backup SPP (shaped pupil plate).

The APP doesn't work at N band and the RAVC is not a good match for N band because of its reduced throughput compared to the CVC. The improved contrast with RAVC is not needed due to the thermal background. The CVC, CLC and SPP also work in N band. The SPP will not be ideal in N band due to reduced strehl but is inherently achromatic.

The focal plane coronagraphic modes can probably be reduced by the exact same recipe.
For the APP we have a different recipe than the focal plane coronagraphs because you get 3 PSFs (two of which have 1 dark hole each), which need a different way of preprocessing and there is only a small region with good correction. The SPP only has 1 PSF but 2 dark holes and is probably similar enough that it can use the same recipe.

So in the implementation phase we probably only need two main recipes for each subinstrument; focal-plane vs pupil-plane optimized.

[hugobuddel]

Thanks for the explanation @gotten.

I don't have a good mental picture of the ADI pipeline so I keep forgetting how things tie together, like which coronagraphs are in the focal plane and which in the pupil plane.

From your explanation, I gather that the LM_APP_SCI_THROUGHPUT is indeed not necessary for the metis_ifu_adi_cgrph recipe. But for now this does not really matter. We'll resolve this eventually. I mainly noted it here because it is (seemingly at least) an inconsistency with the DRLD.

[gotten]

It might be we concluded at some point that because of the IFUs limited FOV only one APP PSF is visible. Which might mean that the same recipe can be used for all coronagraphs. The APP throughput would only apply to the APP, not the focal plane ones. I expect we need to revise the DRLD in the implementation phase anyway.