AAAREADME

Copyright (c) 2020, California Institute of Technology.  All rights reserved.


        KCWI Data Extraction and Reduction Pipeline (KDERP) README


Bug fix for Second minor release; Version: 1.2.1 (REL) Date: 2020/02/19

NOTE: If using KOA archive, please select filenames assigned at telescope.
      This pipeline will not work with KOA-assigned filenames.

**CONTENTS**

0. QUICK START
1. INSTALLATTION
2. RUNNING THE PIPELINE
3. ANCILLARY UTILITIES
4. USER CONTRIBUTED SOFTWARE
5. NEW FEATURES
6. TROUBLESHOOTING


0. QUICK START:

Follow these instructions and you'll be up and running quickly.  For details
see the full installation instructions below.

Substitute the real version number for <ver> and real paths for
'/Path/to/package' and '/Path/to/tarball'.  Trailing slashes are
significant in step e.  NOTE: It is very important to put the kderp
package path at the head of the list of paths in the IDL_PATH
environment variable.

a. mkdir /Path/to/package
b. cd /Path/to/package
c. tar -xvzf /Path/to/tarball/kderp-<ver>.tar.gz
d. setenv IDL_PATH "+/Path/to/package/kderp-<ver>:<IDL_DEFAULT>"
e. edit /Path/to/package/kderp-<ver>/startup.pro
	point !KCWI_DATA to /Path/to/package/kderp-<ver>/data/
f. setenv IDL_STARTUP /Path/to/package/kderp-<ver>/startup.pro


1. INSTALLATION:

Installation requires four steps.  In all steps below replace the string
<ver> with the DRP version string, e.g., "1.0.1" (not including the
quotes).


A. Unpack the tarball:

All of the code is written in IDL.  As such, you can integrate the package
within a directory structure that is already in your IDL_PATH, or it can be
installed in a stand-alone mode.  Decide where you would like to put the
package, cd into that directory and issue this command:

> tar -xvzf /Path/to/tarball/kderp-<ver>.tar.gz

Specifying the real path, of course.  The tarball will extract into the
directory 'kderp-<ver>'.


B. Adjust IDL_PATH environment variable:

If you decide to make it stand-alone, then you must set your IDL_PATH
environment variable to point to the top-level directory, something like
this:

> setenv IDL_PATH "+/Path/to/package/kderp-<ver>:<IDL_DEFAULT>"

The "+" at the beginning ensures that IDL searches the entire directory
tree below the top level for all the code.  

**** NOTE ****: The item <IDL_DEFAULT> should be entered as is, not
replaced with another path.  It tells IDL to include all the standard
packages.  If this is missing, bad things will happen!

**** NOTE ****: It is very important that you put the kderp package
directory at the head of the list of paths in the IDL_PATH environment
variable.  IDL has a flat name-space and this avoids collisions with
routines with the same names in other packages.


C. Point !KCWI_DATA to correct directory:

The file 'startup.pro' at the top level of the package contains the
definition of an IDL system variable called !KCWI_DATA.  This must point to
the correct directory in order for the pipeline to function.  It should
point to the subdirectory 'data' within the package and should look like
this:

defsysv,'!KCWI_DATA','/Path/to/package/kderp-<ver>/data/'

Be sure to include the trailing '/'.  

The other lines in the file should be left alone.


D. Adjust the IDL_STARTUP environment variable or file:

The file 'startup.pro' must be either pointed to by the environment
variable IDL_STARTUP, or it must be edited into your existing IDL startup
file.  If you are running the pipeline stand-alone then do this:

> setenv IDL_STARTUP /Path/to/package/kderp-<ver>/startup.pro

If you are integrating the pipeline with other IDL software, then you must
add the system variable definitions from 'startup.pro' into the file that
your current IDL_STARTUP environment variable points to.

Once you have verified that all the paths are correct, you may wish to add
the environment variable definitions for IDL_PATH and IDL_STARTUP to your
shell startup script so you don't have to enter them each time you run the
pipeline.


2. RUNNING THE PIPELINE:

All commands are issued at the IDL prompt.  It is probably simplest to cd
into the directory with the raw data.  No raw data files are overwritten.
Output files all have altered names to avoid clobbering raw data.

The pipeline has 8 stages:

Stage 1 takes raw images and performs a basic CCD reduction to produce bias
and overscan subtracted, gain-corrected, trimmed and cosmic ray removed
images.  For nod-and-shuffle observations, the sky will also be subtracted.
The major output is an 'intensity' image in units of electrons.  For object
images, a variance image and a mask image are also produced.  NOTE: the
mask values are their meanings are tabulated in the DATA_PRODUCTS file.

Stage 2 is a dark and scattered light subtraction step.  Even if you have
no dark frames, this step should be run to remove scattered light.  For
nod-and-shuffle images scattered light has already been subtracted so this
step will automatically be skipped.

Stage 3 uses a corresponding 'cbars' and an 'arc' image to define the
geometric transformations required to map each pixel in the 2d image into
slice, postion, and wavelength.  These maps are used in subsequent stages.

Stage 4 is a flat field and illumination correction step.  This will use
the continuum flat images to determine the pixel-to-pixel variation in the
ccd response along with the slice response function and the relative slice
response function.

Stage 5 is an optional sky subtraction step.  This uses the maps generated
in stage 3 to collect all the spectra in a given frame and reject object
flux to determine the best sky spectrum.  This is then fit with a bspline
algorithm to generate a model of the sky for each pixel in the 2d image.
This 'noise-free' sky is then subtracted from the 2d object frame.  This
step should be skipped if the sky is not well sampled in the frame or is
mixed with a large fraction of object light (e.g., objects that fill the
field-of-view of the slicer).

Stage 6 is the cube generation step.  It applies the transformations solved
for in stage 3 to the object intensity, variance and mask images output
from any of the previous stages.

Stage 7 performs a differential atmospheric refraction correction based on
the airmass of the observation, the orientation of the slicer during the
observation, and the wavelengths within the observation.  The data cube is
padded with zeros to avoid spurious data caused by wrap-around from the
shifting process.

Stage 8 uses a standard star observation to generate an inverse sensitivity
curve which is applied to the corresponding observations to flux calibrate
them.  This defaults to be run interactively (display=2), so be aware of
this if you are running the pipeline in batch mode.  It is not recommended,
but you disable interactive mode by setting display to 1.

Each step is controlled by the master 'proc' file which defaults to
kcwi.proc, and a corresponding 'ppar' file which defaults to kcwi.ppar.
These are generated by running the KCWI_PREP program with the appropriate
options for your needs.  Run KCWI_PREP with the /help keyword set and it
will print all the available options for you.

The proc file allows you to see the automatic associations that are made and 
to adjust which images are processed.  It also allows you to set pipeline
parameters on an image-by-image basis.  The ppar file allows you to adjust
the default or global pipeline parameters.

STAGE 0 - preparation:

Here you choose several global options for processing.  To see the
parameters and keywords that are available, issue the command with the
/help keyword set:

IDL> kcwi_prep,/help

These are the parameters:

RawDir		- Input directory, location of raw images, defaults to current.
ReducedDir	- Output directory, defaults to ./redux/
DatDir		- Data directory, defaults to IDL system variable !KCWI_DATA

These are the keywords:

/BIASSKIP1	- skip the first bias in a group
/SKIPOSCANSUB	- skip the overscan subtraction (only if bias subtracted)
/NOCRREJECT	- turn off CR rejection (good for quick look)
/NONASSUB	- turn off nod-and-shuffle subtraction (why?)
/SAVEINTIMS	- save intermediate images (takes extra disk space)
/INCLUDETEST	- includes images of type 'test' (focus images, etc.)
/EXTERNAL_FLAT	- set to give priority to external (dome/twilight) flats (why?)
/CLOBBER	- set to overwrite output files (default: it won't)
/NOCLEANCOEFFS	- turn on to skip cleaning the wavelength coeffs of errant bars
/WAVEITER	- use a wave solution method that starts with a central fit 
			and iteratively expands the wavelength range (why?)
/HELP		- print the command usage info

ALTCALDIR	- set to an alternate source directory for calibrations
VERBOSE		- verbosity level for extra diagnostic output (1 is recommended)
DISPLAY		- display level for extra diagnostic plots:
			0 - display no plots
			1 - display some plots, non-interactive
			2 - display more plots, interactive
			3 - display even more plots, interactive
		NOTE: These can be overridden for each stage on the IDL
		command line by using verbose and display keywords in the
		stage invocation (see below).
SAVEPLOTS	- save level for plots:
			1 - default to save basic set of plots
			2 - save extra diagnostic plots
			3 - save maximum number of diagnostic plots
		NOTE: This should only be used to help diagnose a
		problem and should be set on an image-by-image basis
		in the kcwi.proc file (see below).  Setting this to
		2 or 3 at the global level will generate a large number
		of plots.

MINGROUPBIAS	- minimum required number of bias frames to combine
			this defaults to 5
MINGROUPDARK	- minimum required number of dark frames to combine
			this defaults to 3
MINOSCANPIX	- minimum number of overscan pixels in a given row
			this defaults to 70
FROOT		- string overriding default file root ('kcwi')
FDIGITS		- integer overriding the default image number digits (5)
FIRST		- imgnum of first image to process (1)

TAPERFRAC	- taper fraction for cross-correlation (0.2)
PKDEL		- arc line matching threshold in fraction of resolution (0.75)

Once you have decided on the options just execute it. For example:

IDL> kcwi_prep,/verbose,display=2

This will run very quickly depending on how many image headers need to be
ingested.  It will prompt you to create the output directory if it doesn't
exist.  Exit IDL and look in the output directory (redux is the default):

> cd redux
> ls

You'll see all the files that specify master calibration file generation
(specific ppar files), as well as a log file.

Take a look at the kcwi.proc file.  This shows the automatically generated
linking between the master calibrations and the other images.  Header lines
are designated by starting with a hash ('#') character.  After the header
there is one record per processed image followed by a set of keyword-value
pairs (separated by an equals sign) that control the association of each
object image with the corresponding calibration object.  The header gives
the legend for the columns in the image record.  This image record
specifically gives the image number and all the configuration data relevant
to the calibration process.  The processing keywords that follow are
derived from the master ppar file which defaults to 'kcwi.ppar' in the
output directory.  If one wants to change these associations, one simply
edits the keyword values.  One may also add new keyword-value pairs, as
long as the keywords exist in the ppar file.  These keyword-value pairs
override the default values in the ppar file for the given image being
processed.

An example of this is the SAVEPLOTS keyword.  If you add the text

saveplots=2

to a single image's keyword-value pairs list, it will only save the
diagnostic plots for that particular image.  This is the safest way to use
the SAVEPLOTS keyword as a global setting will generate a large number of
diagnostic plots.

The proc file is read in by all subsequent stages so you can control which
images are processed by commenting out any you don't want run through the
pipeline.  Do this by inserting the '#' character at the beginning of the
line.  This will ensure that the line gets skipped by all subsequent
processing.  NB: be careful of the flow-down effect of not processing any
of the calibration images like arcs that need to be processed for
subsequent calibration.  You can also change which master file or other
standard image is applied to each object image by editing this file.  Do
this with caution as incorrect associations can result in pipeline failures
or unpredictable results.

You can also edit the kcwi.ppar file and change some of the default
processing options.  You can turn on or off verbose output for example.
The most likely change you'll want to make is adjusting the DISPLAY
parameter (see the description above).  It's a good idea to run the
pipeline first with DISPLAY set to 2 or more so you can see the details of
what is going on.  After that, if you have a lot of images to process,
you'll want to make the pipeline less interactive by setting DISPLAY to 1
or less.  If you set the keywords properly on the command line, it is
unlikely you will need to edit this file.


STAGE 1 - basic image reduction:

Once you are happy with the files, go back to the raw directory and start
IDL again:

> cd ..
> idl

By default each stage looks for the kcwi.proc and kcwi.ppar files in
'./redux/', but you can specify them on the command line.

IDL> kcwi_stage1, './kcwi.proc', './kcwi.ppar'

NB: Unless there is a good reason, it is not recommended that you specify
these filenames on the command line.  In general, you will want to edit the
files in the redux directory and just issue the stage commands with no
parameters.

Now you just run stage1:

IDL> kcwi_stage1

You can override the verbosity and display levels set in the kcwi.ppar file
(as set by running kcwi_prep.pro) for this (and any subsequent stage) on the
IDL command line with keywords, so you can also type:

IDL> kcwi_stage1,verbose=2,display=3

This will take longer to run, especially if you kept cosmic ray removal
'on'.  On a macbook pro with a 2.2 GHz Intel Core i7 processor and 8 GB of
RAM, it takes roughly 30s just to do the cosmic ray removal on one 2x2
binned image.  Typical total pipeline batch run times for a night's worth
of images can vary from 20 to 60 minutes.

Exit IDL again and examine the log file in the output directory.

> more redux/kcwi_stage1.log

This keeps track of the parameters that were used to derive the intensity
images.  It logs the number of cosmic rays removed, for example.

The output images for this stage all have the same prefix as the input raw
image, but different suffixes that indicate what kind of image they are:
'_int.fits' for the intensity images, '_var.fits' for the variance images,
and '_msk.fits' for the mask images (see DATA_PRODUCTS file for mask values
and their meanings).  If you had nod-and-shuffle observations, then there
will also be '_obj.fits' and '_sky.fits' images containing just the object
panel and just the sky panel respectively of the nod-and-shuffle segments.


STAGE 2 - dark/scattered light subtraction:

The input and output directories default to './redux'.  You can run the
command from the raw directory and it will find the files you already
processed in stage1.  You can control which master dark is used and which
observations are dark subtracted using the 'kcwi.proc' file in the output
directory specified when you ran KCWI_PREP.

This stage is run as follows:

IDL> kcwi_stage2dark

The log for this stage is in the output directory and is called
kcwi_stag2dark.log.  Here the outputs will be the same as for the stage1
outputs, but will have a 'd' appended as such: '_intd.fits', '_vard.fits',
and '_mskd.fits'.

NB: This step was previously skipped if dark frames were not taken.  Now,
we subtract scattered light in this stage, so it should always be run
unless there is some compelling scientific reason not to subtract the
scattered light.


STAGE 3 - geometric solution and data cube generation:

This stage is run as follows:

IDL> kcwi_stage3geom

The first step will be to trace the bars in the 'cbars' image.  If you set
DISPLAY=2, you will see a plot for the middle row of each bar.  You can
enter 'q' at the prompt to turn off plotting (there are 120 bars).  The
program will then proceed without plotting until the next step.  You will
next be shown a plot of the control points for the entire image.  Just hit
return to proceed (assuming everything looks OK).

Next the DRP will extract the arc spectrum along each bar.  It will
cross-correlate each spectrum with the reference bar (defaults to center
bar of slice 11, bar 57) and display a plot (if DISPLAY=2, otherwise
nothing).  Here again, you can enter 'q' to skip the plots and proceed.

Each bar spectrum is then cross-correlated with either a ThAr or an FeAr
atlas spectrum using a central window about equal to the nod-and-shuffle
mask (central third).  This provides a preliminary wavelength solution that
is accurate over the central third of the wavelength range for each bar.
This is sufficient for nod-and-shuffle observations.  For full-frame
observations a second step is performed which uses the full range of
wavelengths in the observation.  This step finds good, isolated lines in
the atlas spectrum and then uses the preliminary solution to find them in
the observed spectrum.  A new solution is generated using this wider data
set.  If you kept the coefficient cleaning 'on' (CLEANCOEFFS=1), the DRP
will print a list of coefficients that have been fixed and, if you have
DISPLAY set to 2 or more, a plot of each slice showing the original and
fixed coefficient.  Cleaning is done after the initial solution is found
and after each iteration for the full-frame observations.

If you are reducing full-frame observations and you set DISPLAY=2, you will
next see a plot of the residuals between the arc bar spectrum and the atlas
spectrum.  Acceptable residual values depend on the grating.  For the BH
gratings, you should expect residuals better than 0.1 Angstroms.  For BM
this goes up to about 0.2 Angstroms, and for the BL grating the residuals
can be as large as 0.6 Angstroms.  Acceptable results can be obtained with
even larger residuals.  Next, you will have to hit enter to display the
next plot or enter 'q' to skip the plotting of the remaining bars.

Once the wavelengths are solved, a diagnostic plot will be displayed (if
DISPLAY is >= 1) and the geometry values are calculated and applied to the
relevant images.

Regardless of if you are reducing nod-and-shuffle or full-frame
observations a diagnostic plot set is output in the redux directory that
will allow you to assess the quality of wavelength solution for each bar.
This file starts with 'wave_' and the remainder of the filename
incorporates the bar spectrum image number and the arc spectrum image
number, e.g.:  'wave_cb1234_arc1232.ps'.  This will contain plots of the
coefficients as a function of bar number and a set of plots of each bar
compared to the atlas spectrum.  Problems can be identified in the
coeficient versus bar plots and then verified by looking at the specific
bars.  The specific bar plots will show problems if there is an obvious
discrepancy between the atlas spectrum and the bar arc spectrum.

If you have trouble finding a good solution for a particular bar or set of
bars, you can tweak some of the wavelength solution parameters in the
kcwi.ppar file and re-run stage3.  You can also edit the kcwi.proc file
and add a specific adjustment using a keyword/value pair that relates to
the wavelength solution for a specific data set.  Increasing the taper
fraction (TAPERFRAC) parameter helps with edge effects produced by strong
arc lines near the edges of the wavelength window.  The line-matching
parameter (PKDEL) will automatically be adjusted upwards from its default
of 0.75 in 0.25 increments until at least five peaks are matched.  You may
want to start out with a larger fraction of the resolution.  You must
delete the the old geometry files from stage3 or they will just be
re-used.  You can delete the old data-cube files or just set the CLOBBER
parameter to 1 to allow new data-cubes to be generated.  You may want to
move or rename the diagnostic 'wave_*.ps' files to compare with the newest
run to see if your tweaking has improved things or not.  If they are not
moved or renamed, they will just be overwritten.

The outputs from stage3 are the wave plot files, geometry fits files, the
kcwi_stage3geom.log file and three map image files: a slice number map, a
position (along the slice) map, and a wavelength map.  The maps and
geometry fits files are named after the 'cbars' image that was used to
trace the bars and will end in '_slicemap.fits', '_posmap.fits',
'_wavemap.fits', and '_geom.fits'.  If you re-run the stage3 pipeline
without deleting these, they are just re-used.

If you want to save any of the intermediate plots, edit the kcwi.proc file
and add 'saveplots=2' or 'saveplots=3' to the list of keyword-value pairs
for the cbars and arc images that are used to generate the geometry.  This
will save many of the plots listed above.


STAGE 4 - flat field and illumination correction:

You can control which master flat is used and which observations are flat
corrected using the 'kcwi.proc' file in the output directory specified when
you ran KCWI_PREP.

This step is run as follows:

IDL> kcwi_stage4flat

The log for this stage is in the output directory and is called
kcwi_stag4flat.log.  Here the outputs will be the same as for the stage2
outputs, but will have an 'f' appended as such: '_intf.fits', '_varf.fits',
and '_mskf.fits'.

As a diagnostic, you can display the master flat images with ds9 to see how
well the CCD pixel-to-pixel variations and the illumination variations were
fit.  If you find a problem, you can edit the kcwi.proc file and use a
different master flat.  Just be careful to use a flat image with the same
calibration setting as your object image.


STAGE 5 - sky modeling and subtraction:

This stage uses the geometry maps output from stage3 to assemble all of the
image spectra in order to derive an accurate estimate of the sky spectrum.
For each wavelength, it collects all image pixels at that wavelength and
then does a 1-sigma clip to remove light from non-sky photons.  The
resulting sky observation is fit with bsplines to generate the best
estimate of the sky spectrum with as little noise as possible.  Here, the
outputs will have a 'k' appended: '_intk.fits, '_vark.fits', '_mskk.fits'.
The model sky image is also output with the suffix '_sky.fits'.

The sky model relies on having enough sky pixels to allow the 1-sigma
rejection to remove object pixels.  If there are object features that skew
the sky estimate (as evidenced by over-subtraction in the output
'_intk.fits' image), then you will have to use a sky mask to avoid those
regions.  This stage automatically searches for a sky mask file, which is
named with the suffix '_smsk.txt' and uses it, if it is found, to exclude
regions specified in the sky mask file from the sky fit.  The sky mask file
is a simple text file that specifies regions to exclude, one per line, with
the following format:

column 1 - starting slice number,
column 2 - ending slice number,
column 3 - starting position,
column 4 - ending position,
column 5 - starting wavelength,
column 6 - ending wavelength.

These values can be derived by loading the object image along with the
corresponding map images into ds9.  Keep adjusting this sky mask file until
you no longer see oversubtraction near object spectral features in the
output image.  If an object fills the IFU with object light and there are
no sky pixels, then a separate sky image must be taken and used as the
input sky master as specified in the kcwi.proc file.

Object pixels can also be masked for sky determination using a fits image.
The mask image should be a simple byte image with the value 1 for pixels that
should be masked and 0 everywhere else.  It should have the suffix '_smsk.fits'
and have the same prefix and dimensions as the corresponding '_intf.fits'
image.  This mask image can be generated from a standard ds9 region file that
delineates the regions of object flux in the image using a new python
utility in the 'devel' subdirectory of the pipeline release.  For example:

# python ~/kderp/devel/kcwi_masksky_ds9.py  kb180101_00101_intf.fits ds9.reg

will generate a mask image, kb180101_00101_smsk.fits, which the pipeline
will automatically detect and use during stage 5.  See this routine in the
devel directory for details about python library dependencies.

This stage is run as follows:

IDL> kcwi_stage5sky

In this case, the cleaned pixels are plotted as a function of wavelength
and the model is shown as an orange line.  If you set the display keyword
to 2 or higher, it will show all the pixels and the un-rejected pixels as
blue points, with the orange model fit.

The outputs are the same as the stage4 outputs, but will have a 'k'
appended as such: '_intk.fits', '_vark.fits', '_mskk.fits'.  In addition,
each sky model is output with the suffix '_sky.fits'.  It is a good idea to
load each set of '_intf.fits', '_intk.fits', and '_sky.fits' images into
ds9 and blink them to determine how well the sky subtraction has done.


STAGE 6 - cube generation:

This step is run as follows:

IDL> kcwi_stage6cube

The outputs from stage6 are the data cubes and the kcwi_stage6cube.log
file.  The data cubes are an intensity cube '_icube.fits', a variance cube
'_vcube.fits' and a mask cube '_mcube.fits'.  If you had nod-and-shuffle
observations, there will also be an object cube '_ocube.fits' and a sky
cube '_scube.fits'.  If you performed sky subtraction, this will also
result in a sky cube.  Images that were not taken with nod-and-shuffle mode
will also have a '_ocube.fits' image cube output derived from the
non-sky-subtracted, but flat-fielded '_intf.fits' image.  These images are
useful to verify that spectral features are not the results of sky
residuals or in the case when sky subtraction is not wanted.

Some new outputs have been added for more detailed diagnosing of the
wavelength solutions.  Most important for this is the generation of a 2D
version of the data cube of arc observations.  These can be displayed in
ds9 and the arc lines examined across the slices to see if there are any
deviations from straightness across the image.  These are written into the
output directory and have names like *_icube_2d.fits.


STAGE 7 - differential atmospheric refraction correction:

This stage requires no calibration files and is performed purely on the
basis of each observation's properties.  The relevant properties are the
temperature, the relative humidity, the airmass, the IFU orientation with
respect to the parallactic angle, and the wavelengths of each spatial image
within the cube.  All of these parameters are derived from the FITS image
header keywords.  The output cubes will be padded in the spatial dimensions
a standard amount that depends on the grating used.

This step is run as follows:

IDL> kcwi_stage7dar

The log for this stage is in the output directory and is called
kcwi_stage7dar.log.  Here the outputs will be the DAR corrected data cubes,
'_icubed.fits', '_vcubed.fits', and '_mcubed.fits' (and '_scubed.fits' and
'_ocubed.fits').  As a diagnostic, you can display the uncorrected
'_icuber.fits' file in DS9 and scan in wavelength and observe the motion of
a stellar continuum source due to the effects of DAR.  Displaying the
'_icubed.fits' version and performing the same scan should exhibit no
motion.


STAGE 8 - standard star calibration:

The kcwi_prep.pro run will attempt to associate any standard observations
you have with the appropriate object observations by comparing the object
name with the list of standard star files in the directory pointed to by
!KCWI_DATA+'stds/'.  This comparision is NOT case-sensitive.

You can control which standard star images are used and which
observations are standard star corrected using the 'kcwi.proc' file in the
output directory specified when you ran KCWI_PREP.  Currently the standard
star library is somewhat limited.  A Keck-formatted star list that includes
all the standard stars can be found in the data/stds directory.  It is
called 'kderp_stds_starlist.txt' and should be given to the observatory
assistant prior to your run.  It is recommended that you use standards that
have HST observations (STIS, NICMOS, etc.) since these have wavelength
sampling closer to KCWI.

This stage runs interactively by default to allow the user to specify the
wavelength range for fitting the calibration and to tweak the spectral
regions that are used for the fit (to avoid sharp absorption features, for
example).  If you do not want to run it interactively, you should set
display to 1 or 0 on the command line.

This step is run as follows:

IDL> kcwi_stage8std

This will display a plot of the observed standard star spectrum with the
'good' spectral region bounded by green vertical lines.  You will be
prompted to specify the wavelength range for the fitting.  If the shape
looks good beyond the 'good' range, you might want to include these data
in the fit to avoid edge problems within the good range.  You should always
include the wavelength range between the two green vertical lines.

Next, you will be presented with a plot of the inverse sensitivity with
some automatic masking of the Balmer lines indicated by vertical dashed
blue lines and 'X' points plotted on the data.  Another plot in a window
below will show the resulting calibrated standard spectrum and a residual
plot, so you an gauge the quality of the fit.  You will be prompted to
enter a single letter action code that have the following meanings:

r - restore deleted points,
d - delete points,
+ - increase polynomial fit order by one
- - decrease polynomial fit order by one
f - re-fit the data
q - exit the fitting process and use the last fit.

This allows you to tweak the fit until you get the lowest residuals.

If you have run the observation through stage5sky and performed sky
subtraction, there will be no further sky subtraction done here.  However,
if you have not run stage5sky, a sky subtraction will be performed.  In
that case, if you have the DISPLAY parameter in kcwi.ppar set to 3 or
greater, you will see plots of the sky spectra used to sky-subtract the
standard star observations.  

The final fit inverse sensitivity curve will be applied to each observation
associated with the standard.  The inverse sensitivity curve will
be output using the same image number as the standard star observation but
with a '_invsens.fits' suffix.  The object files that have had the inverse
sensitivity applied will also be output with the following suffixes:
'_icubes.fits' for intensity, '_vcubes.fits' for variance, and
'_mcubes.fits' for the mask data cube (and also the '_ocubes.fits' and
'_scubes.fits' files).

In addition, diagnostic plots showing the derived effective area, the
derived inverse sensitivity, and the observerd versus reference flux and
residual spectrum are saved.  Look for '*.png' files that start with the
name of the standard star being used.

You can generate postscript versions of these plots using the routine
'kcwi_test_std.pro' which requires only the standard star observation image
number as input.  To create the postscript output of the plots, just add
the keyword '/ps' to the command line call.


3. ANCILLARY UTILITIES:

Here are some useful utilities that might be of use to observers:

kcwi_masksky_ds9.py:

This is a python utility that converts ds9 region files into a mask image
that will be used in stage5 (sky subtraction).  It requires astropy and
pyregions libraries to be installed.  This item can be found in the 'devel'
subdirectory in the DRP release directory.  Details are found within the
file.


KCWI_READ_CFGS, and KCWI_PRINT_CFGS:

KCWI_READ_CFGS is a function that returns the KCWI_CFG struct for each
image in a directory so you can examine the configurations of your images
with KCWI_PRINT_CFGS.  You can specify input file specs to limit your
search:

> cd 130405
> idl
IDL> kcfg = kcwi_read_cfgs('./',filespec='kb170517_001??.fit*')
IDL> kcwi_print_cfgs,kcfg,/header

This will print a list of settings for the set of images matching the file
spec 'kb170517_001??.fit*' (with a header to identify the columns).  If
you've done stage1 already you can get the results as follows:

> cd 170517/redux
> idl
IDL> kcfg = kcwi_read_cfgs(/stage1)
IDL> kcwi_print_cfgs,kcfg,/header

This will read the configurations of all the *_int.fits images and print
them out.  There is a similar keyword for stage2 output:

IDL> kcfg = kcwi_read_cfgs(/stage2)
IDL> kcwi_print_cfgs,kcfg,/header


4. USER CONTRIBUTED SOFTWARE

In this section we provide links to user contributed software.  These
routines carry out their own testing and reliability independent from the
KCWI DRP, and we cannot guarantee the quality of the outputs.  However,
many of these routines are very useful and add value to the output from the
standard pipeline.

https://github.com/dbosul/CWITools

CWITools are a set of python tools for post-processing of KCWI (and PCWI)
data written by Donal O'Sullivan.  These tools include stacking and
continuum subtraction.


5. NEW FEATURES:

BIASSKIP1 pipeline switch

If the CCD controller has been power cycled less than two hours before taking
biases, there will be an exponential decay of dark current that will be at
it's highest level for the first bias.  Set this parameter when running
kcwi_prep to skip this first bias.  Better yet, wait at least three hours
after power cycling before taking biases or any science data.

NB: changing binning will cycle the power on the controller.

SKIPOSCANSUB pipeline switch

In rare cases where there is extra flux in the overscan (again close to a
power cycle event), one may want to skip using the overscan for bias correction.
This is only implemented if the image has been bias subtracted.

KNOTSPP pipeline parameter

In order to give the user more control over sky subtraction, we have added
a new pipeline parameter that specifies the number of knots per Y pixel
for the sky fitting with b-splines.  Currently this defaults to a value of
1.25, but if the fitting is not tracking the sky well, this can be doubled
to 2.50.  After running kcwi_prep.pro, the default value for this parameter
will be set in the redux/kcwi.ppar file.  If you only want to change it for
specific images, you can use the redux/kcwi.proc file and add an entry:

knotspp=2.5

under the specific image entry in the proc file.  The value of 2.5 is just
an example.  Feel free to experiment until you get the best sky
subtraction.

SAVEPLOTS keyword

You can now use the saveplots keyword either on the command line when
running kcwi_prep.pro (not recommended), or in the kcwi.proc file on an
image-by-image basis (recommended).  The pipeline will generate a basic set
of diagnostic plots in the redux directory if this keyword is not used.
However, if you wish to save and examine other diagnostic plots that are
shown on the screen when display values of 2 or 3 are used, you can use the
saveplots keyword to indicate the level of plots to save beyond the default
level.  Adding 'saveplots=3' to the keyword-value pairs for a given image
will generate the maximum number of output '*.png' plots.  Setting
saveplots to 2 will save a intermediate number of '*.png' plots and is the
recommended first step when trying to diagnose a problem.

New Flat Fielding and Sky Subtraction

We have implemented the use of bspline fitting to determine the
flat-fielding and illumination corrections using the internal continuum
flats.  These flats are combined, corrected for internal vignetting,
and then used to characterize the illumination variation across slices and
from slice to slice.  The bspline method has also been found to do a very
good job of characterizing the sky spectrum when combined with a suitable
rejection threshold for object flux.  This method produces a sky model
image that has much lower noise than a sky observation.  We are in the
process of developing tools that allow regions where object flux dominates
over sky flux to be masked out in the sky modeling.

These corrections are done using the 2D images that are mapped to
wavelength, slice position and slice number.  Since this process requires
the cube geometry maps, but are applied to the 2D images, we now solve the
geometry in stage 3, but don't generate cubes until stage 6.  Now, stage 4
of the pipeline calculates and applies the flat-field and illumination
correction, and stage 5 calculates and subtracts the sky model.  Stage 5 is
optional.

Alternate Calibration Directory

In anticipation of having a library of calibrations, I have implemented the
ALTCALDIR keyword for the kcwi_prep stage.  This allows you to specify
another directory as a source for calibrations.  The kcwi_prep stage will
attempt to find local calibrations first, but if none are found, it will
attempt to find them in the directory specified by ALTCALDIR.

New Cosmic Ray Removal Method

The ability to account for an object PSF has been added to the routine
kcwi_la_cosmic.pro.  This was motivated by observing that parts of night
sky lines can sometimes be flagged as cosmic rays and removed from the
stage1 intensity image.  We have added a PSF model option called 'gaussy'
which accounts for real objects that have a Gaussian profile in the y
direction, like night sky lines.  In most cases, the night sky lines are
not important, but in the case where they serve as a reference or some
other purpos, we recommed using this model to minimize their removal.  This
model can be invoked using the kcwi.ppar file by setting the parameter
CRPSFMOD to 'gaussy' (it defaults to '').  Other relevant parameters for
this method are CRPSFFWHM, which specifies the FWHM of the Gaussian model of
real objects (defaults to 2.5px), and CRPSFSIZE, which specifies the overal
size of the model PSF (defaults to 7px).  Be aware that real cosmic rays
that have a horizontal trace will be harder to remove using this method.

Flatten Cube to 2-D image

The new routine, kcwi_flatten_cube.pro, takes each slice and packs them
into a 2-D image with a WCS that preserves the wavelength solution on the y
axis as well as the slice coordinate (0-24) on the x axis.  This image is
automatically generated by the cube generation segment of the pipeline
(kcwi_stage6cube), but kcwi_flat_cube can be run on any cube.  This
provides a great diagnostic for the geometry fit as problems will distort
the arc lines away from appearing straight.

Direct Mode Reductions

These are mostly transparent to the user with the exception of the relative
response stage.  For direct mode observations, the user should use the
routine kcwi_stage6drr.pro.  There is no profile or standard star step for
direct mode observations as of now.

Standard Star Associations

The prep routine, kcwi_prep.pro now attempts an automatic association
between standard star image numbers and target image numbers.  This is done
in a very simple-minded fashion by comparing the object name with the list
of standard star calibration files in !KCWI_DATA+'stds/'.  The comparison
is case-insensitive and the list of standard is fairly small.  The airmass
correction is now included in the calibration using the Palomar Observatory
coefficients.


6. TROUBLESHOOTING:

1. No mouse reaction during kcwi_stage4geom:
	If the mouse seems to not be read by the program, check the X windows
	configuration and either set 'Focus Follows Mouse' or 'Click-through
	Inactive Windows'.

2. Missing variable or routine:
	If the pipeline stops and complains about an unknown variable or a
	missing routine, it is most likely due to an error in the setup of
	the IDL_PATH environment variable.  Please review the instructions
	above carefully and these problems persist, please contact Don
	Neill: neill@srl.caltech.edu with the details of the error message.

3. Bad wavelength solution:
	As stated above, two parameters are available to tweak the
	wavelength solution: TAPERFRAC, and PKDEL.  The recommended
	first tweak is to increase TAPERFRAC from 0.2 to 0.5.  There may 
	also be good results from tweaking PKDEL, but we recommend very small
	adjustments.