Skip to content

Commit

Permalink
Added SOF files
Browse files Browse the repository at this point in the history
  - templates in sofTemplates/
  - versions to use in sofFiles/
Added prototypes for external calibrations; generated as part of script.

A few minor fixes
  • Loading branch information
JenniferKarr committed Oct 23, 2024
1 parent 98c4dc4 commit a0dac18
Show file tree
Hide file tree
Showing 121 changed files with 829 additions and 672 deletions.
55 changes: 43 additions & 12 deletions ESO/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@ resultant files **DO NOT** contain instrumentally or scientifically
accurate data, and under no circumstances should be used to evaluate
potential performance of the METIS instrument.</span>



# METIS Simulations

This respository contains scripts which can be used as a wrapper for ScopeSim to generate a set of simulated METIS data for pipeline development.
Expand Down Expand Up @@ -50,12 +48,12 @@ To run the default set of FITS files, as described in [Data Product Summary](#da


```
> ./run_recipes.py --doCalib=5
> ./run_recipes.py --doCalib=1 --sequence=1
> md5sum -c checksums.dat | grep -v OK
```

This will run the script, automatically determining the necessary flats and darks and running them at the end of the sequence; the number indicates
how many of each type to generate.
how many of each type to generate. The sequence takes the observation time of the first entry in the YAML file and increments from there.

## Command Line Options

Expand Down Expand Up @@ -93,7 +91,7 @@ writes the output to myDirectory.
runs the set of simulations starting the sequence at the given date and automatically incrementing the time stamp through the sequence.

```
> ./run_recipes.py --sequence 1
> ./run_recipes.py --sequence 2
```

does the same but takes the dateobs from the first item in the sequence.
Expand All @@ -108,6 +106,13 @@ Runs the script without executing the simulations to check input values.
> ./run_recipes.py --small
```

```
> ./run_recipes.py --doCalib=1 --dumpCalib=calib.yaml
```

runs the script with automatically generated calibration files, and writes the calibration files to a
recipe YAML file.

runs the script with 32x32 pixel outputs, for testing purposes.


Expand Down Expand Up @@ -151,19 +156,31 @@ set DIT/NDIT/Filters for reasonable flux levels, and to refine the
choice of science targets and standard stars for more accurate (and
science-case appropriate) choices. We have generated data and
calibrations for one set of filters for each mode as a base set; this
can easily be extended to multiple sets as needed. For images where
the data will be averaged, two files have been generated with
different time of observation and the same settings.
can easily be extended to multiple sets as needed. One of each type of
image has been generated; this is sufficient for pipeline skeleton development.

The coronagraph, pupil imaging and chopper home images currently
contain placeholder data as these are not modes sximulated by
ScopeSim; by the next release these will be updated to include input
simulated images.

This release does not include external calibration files (such as source catalogues).
This release contains placeholder files for the internal pipeline
representation of the of the external calibration files (such as
source catalogues). The EDPS relevant header keywords are set, and
there is a first draft of the internal format.

FITS keywords needed by the recipes themselves (but not by the EDPS skeleton) may not
be complete.

## SOF Files

A set of SOF files to match the simulated data is provided in sofFiles/. There is one SOF file for
each recipe, with the exception of

- SOF files for both lamp and twilight flats
- recipes for both sci/std processing in the metis_det_img_basic_reduce recipes

due to recipes that handle more than one type of input data that is used as input for another recipe.

# Output FITS Files

Expand Down Expand Up @@ -202,7 +219,23 @@ The set of simulations is as follows
- pupil images
- dark frames to match the exposure times for the above

The list of RAW files was compiled from the DRLD recipe listings
## External Calibration Files

- PINHOLE_TABLE
- LASER_TAB
- LSF_KERNEL
- LM_LSS_WAVE_GUESS
- N_LSS_WAVE_GUESS
- N_LSS_DIST_SOL
- LM_DIST_SOL
- ATM_LINE_CAT
- AO_PSF_MODEL
- N_SYNTH_TRANS
- LM_SYNTH_TRANS
- FLUXSTD_CATALOG
- REF_STD_CAT_star1

The list of RAW files and external calibration files was compiled from the DRLD recipe listings
in Chapter 6, specifically the "Input Data" entry. FITS keywords and file types were
cross-checked against

Expand All @@ -212,13 +245,11 @@ cross-checked against
- Matched keywords (needed for the EDPS skeleton) given in Table 6.
- The DPR.CATG, TECH, TYPE etc. given in Table 20.


# Generating Custom Simulations

If you want to run the scripts for your own models, there are two files you will need to edit, in addition to the
command line options given above.


## YAML file

This file consists of a sequence of templates in the form
Expand Down
58 changes: 58 additions & 0 deletions ESO/grabHeaders.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
from astropy.io import fits
import numpy as np
import pandas as pd
from importlib import reload
import glob
import os

fNames = glob.glob("/Users/karr/software/METIS_Simulations/ESO/output/METIS*.fits")
fNames.sort()

outFile = open("summary.csv","w")

line = "File\tDIT\tNDIT\tTech\tCATG\tTYPE\tDRS.SLIT\tDRS.FILTER\tDRS.IFU\tDRS.MASK\tINS."
print(line,file=outFile)

for fName in fNames:
print(fName)
hdul = fits.open(fName)
dit = hdul[0].header['HIERARCH ESO DET DIT']
ndit = hdul[0].header['HIERARCH ESO DET NDIT']

tech = hdul[0].header['HIERARCH ESO DPR TECH']
catg = hdul[0].header['HIERARCH ESO DPR CATG']
tipe = hdul[0].header['HIERARCH ESO DPR TYPE']

slit = hdul[0].header['*DRS SLIT*']
filt = hdul[0].header['*DRS FILTER*']
ifu = hdul[0].header['*DRS IFU*']
mask = hdul[0].header['*DRS MASK*']

ins = hdul[0].header['*INS OPTI*']

fShort = os.path.basename(fName)
line = ""
line = f'{fShort}\t{dit}\t{ndit}\t{tech}\t{catg}\t{tipe}\t'

for elem in slit:
line = f'{line}{hdul[0].header[elem]}'
line=line+"\t"
for elem in filt:
line = f'{line}{hdul[0].header[elem]}'
line=line+"\t"
for elem in ifu:
line = f'{line}{hdul[0].header[elem]}'
line=line+"\t"
for elem in mask:
line = f'{line}{hdul[0].header[elem]}'
line=line+"\t"



for elem in ins:
line = f'{line}{elem}={hdul[0].header[elem]},'
line=f'{line[:-1]}\t'

print(line,file=outFile)
hdul.close()
outFile.close()
28 changes: 14 additions & 14 deletions ESO/makeCalibPrototypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def generateStaticCalibs(outputDir):


hdul = fits.HDUList([primaryhdu, hdu])
hdul.writeto(f"{outputDir}/METIS.REF_STD_CAT_{starName}.fits",overwrite=True)
hdul.writeto(f"{outputDir}/REF_STD_CAT_{starName}.fits",overwrite=True)

#################### FLUXSTD_CATALOG ###################

Expand All @@ -89,7 +89,7 @@ def generateStaticCalibs(outputDir):
hdu.header['TTYPE4'] = "flux"

hdul = fits.HDUList([primaryhdu, hdu])
hdul.writeto(f"{outputDir}/METIS.FLUXSTD_CATALOG.fits",overwrite=True)
hdul.writeto(f"{outputDir}/FLUXSTD_CATALOG.fits",overwrite=True)


#################### LM_SYNTH_TRANS ###################
Expand All @@ -110,7 +110,7 @@ def generateStaticCalibs(outputDir):


hdul = fits.HDUList([primaryhdu, hdu])
hdul.writeto(f"{outputDir}/METIS.LM_SYNTH_TRANS.fits",overwrite=True)
hdul.writeto(f"{outputDir}/LM_SYNTH_TRANS.fits",overwrite=True)

#################### N_SYNTH_TRANS ###################

Expand All @@ -130,7 +130,7 @@ def generateStaticCalibs(outputDir):


hdul = fits.HDUList([primaryhdu, hdu])
hdul.writeto(f"{outputDir}/METIS.N_SYNTH_TRANS.fits",overwrite=True)
hdul.writeto(f"{outputDir}/N_SYNTH_TRANS.fits",overwrite=True)

#################### AO_PSF_MODEL ###################

Expand All @@ -151,7 +151,7 @@ def generateStaticCalibs(outputDir):
hdu.header['CRPIX']=14.5

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.AO_PSF_MODEL.fits",overwrite=True)
hdul.writeto(f"{outputDir}/AO_PSF_MODEL.fits",overwrite=True)


#################### ATM_LINE_CAT ###################
Expand Down Expand Up @@ -183,7 +183,7 @@ def generateStaticCalibs(outputDir):
hdu.header[f'TUNIT{ii+1}']=tt

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.ATM_LINE_CAT.fits",overwrite=True)
hdul.writeto(f"{outputDir}/ATM_LINE_CAT.fits",overwrite=True)



Expand Down Expand Up @@ -240,7 +240,7 @@ def generateStaticCalibs(outputDir):
hdu.header[f'TUNIT{i+4}']="ppmv"

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.ATM_LINE_CAT.fits",overwrite=True)
hdul.writeto(f"{outputDir}/ATM_LINE_CAT.fits",overwrite=True)


#################### LM_LSS_DIST_SOL ###################
Expand Down Expand Up @@ -269,7 +269,7 @@ def generateStaticCalibs(outputDir):
primaryhdu = fits.PrimaryHDU()
primaryhdu.header['HIERARCH ESO PRO CATG'] = "LM_LSS_DIST_SOL"
hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.LM_DIST_SOL.fits",overwrite=True)
hdul.writeto(f"{outputDir}/LM_DIST_SOL.fits",overwrite=True)

#################### N_LSS_DIST_SOL ###################

Expand Down Expand Up @@ -299,7 +299,7 @@ def generateStaticCalibs(outputDir):
primaryhdu = fits.PrimaryHDU()
primaryhdu.header['HIERARCH ESO PRO CATG'] = "N_LSS_DIST_SOL"
hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.N_LSS_DIST_SOL.fits",overwrite=True)
hdul.writeto(f"{outputDir}/N_LSS_DIST_SOL.fits",overwrite=True)


#################### LM_LSS_WAVE_GUESS ###################
Expand Down Expand Up @@ -327,7 +327,7 @@ def generateStaticCalibs(outputDir):
primaryhdu = fits.PrimaryHDU()
primaryhdu.header['HIERARCH ESO PRO CATG'] = "N_LSS_WAVE_GUESS"
hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.N_LSS_WAVE_GUESS.fits",overwrite=True)
hdul.writeto(f"{outputDir}/N_LSS_WAVE_GUESS.fits",overwrite=True)

#################### N_LSS_WAVE_GUESS ###################

Expand All @@ -350,7 +350,7 @@ def generateStaticCalibs(outputDir):
primaryhdu = fits.PrimaryHDU()
primaryhdu.header['HIERARCH ESO PRO CATG'] = "LM_LSS_WAVE_GUESS"
hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.LM_LSS_WAVE_GUESS.fits",overwrite=True)
hdul.writeto(f"{outputDir}/LM_LSS_WAVE_GUESS.fits",overwrite=True)

#################### LSF_KERNEL ###################

Expand All @@ -370,7 +370,7 @@ def generateStaticCalibs(outputDir):
hdu.header[f'TTYPE2']=""

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.LSF_KERNEL.fits",overwrite=True)
hdul.writeto(f"{outputDir}/LSF_KERNEL.fits",overwrite=True)


#################### LASER_TAB ###################
Expand All @@ -387,7 +387,7 @@ def generateStaticCalibs(outputDir):
hdu.header[f'TTYPE1']="Hz"

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.LASER_TAB.fits",overwrite=True)
hdul.writeto(f"{outputDir}/LASER_TAB.fits",overwrite=True)


#################### PINHOLE_TABLE ###################
Expand All @@ -407,7 +407,7 @@ def generateStaticCalibs(outputDir):
hdu.header[f'TTYPE2']="mm"

hdul = fits.HDUList([primaryhdu,hdu])
hdul.writeto(f"{outputDir}/METIS.PINHOLE_TABLE.fits",overwrite=True)
hdul.writeto(f"{outputDir}/PINHOLE_TABLE.fits",overwrite=True)


if __name__ == "__main__":
Expand Down
Loading

0 comments on commit a0dac18

Please sign in to comment.