Merge pull request #83 from Keck-DataReductionPipelines/release

Release v4.1

.gitignore

@@ -1,9 +1,53 @@
+# Compiled files
+*.py[co]
+*.a
 *.o
-*.so*
-.cache/*
+*.so
+__pycache__
+*.0
+
+# Other generated files
+*/version.py
+*/cython_version.py
+htmlcov
+.coverage
+.cache
+
+# Sphinx
+_build
+docs/api
+
+# Packages/installer info
+*.egg
+*.egg-info
+dist
+build
+eggs
+parts
+bin
+var
+sdist
+develop-eggs
+.installed.cfg
+distribute-*.tar.gz
+
+# Other
+.*.swp
+*~
+.settings
+.project
+.pydevproject
+
+# Mac OSX
+.DS_Store
+
+#coverage related ignores
+
+htmlcov
+.coverage
 
 # This directory usually contains recmats.
 calib
 
 # This is the default queue directory.
-drf_queue
+drf_queue

INSTALLPROBLEMS.md

@@ -34,3 +34,26 @@ If you need to explicitly tell your compilier to target a specific architecutre,
 - For 32 bit systems, either ``-m32`` or ``-arch i386``
 - For 64 bit systems, ``-arch x86_64``
 
+## Error Starting IDL scripts
+
+If you get an error that says something like:
+
+```
+    xterm: Can’t execvp idl: No such file or directory
+```
+
+Then it is likely one of two issues.
+
+The first is that the IDL executable is not in your PATH. Something like the following should be in your path:
+
+```
+    /Applications/exelis/idl85/bin/
+```
+
+You can check what is in your PATH by entering the following command:
+
+```
+    echo $PATH
+```
+
+If the IDL executable is in your PATH and you are running this on a MAC then there may be a compatibility issue between your versions of XQuarts, IDL, and ENVI. This problem and the solution to it is documented on the Harris Geospatial website [here.] (http://www.harrisgeospatial.com/Home/NewsUpdates/TabId/170/ArtMID/735/ArticleID/14944/XQuartz-2710-is-Not-Compatible-with-ENVI-531-and-IDL-851.aspx)

README.md

@@ -6,6 +6,25 @@
 * [Running the Pipeline](#running-the-pipeline)
 * [Testing the Pipeline](#testing-the-pipeline)
 
+## Release Notes for v4.1.0
+**2017-11-07**
+Major Updates
+- Includes a new wavelength solution for data after May 2017. A shift in the wavelength solution (on average about 2.8 Angstroms offset) in May 2017 required a re-derivation of the solution. The new solution has an average offset between the observed and vacuum wavelength of OH lines of 0.07 +- 0.06 Angstroms in Kn3 35 mas.
+- A preliminary bad pixel mask is available for data taken after 2016 (new spectrograph detector). The mask was computed from a series of darks. There is both a bad pixel mask of hot pixels (pixels with permanently elevated value) as well as a dead pixel mask (pixels with permanently low values). This mask meant to be used as extension 2 in the raw fits files. Currently, the mask is not automatically applied by Keck. To apply it, use the following command in the raw spectra directory once the pipeline is installed:
+```
+apply_mask.py *.fits
+```
+NOTE: this requires python installed with ``numpy`` and ``astropy`` packages. Tests show that using the bad pixel mask improves the SNR by about 10%.
+
+- A new keyword is available in the Scaled Sky Subtraction module called ``scale_fitted_lines_only``. To turn on the new behavior, the keyword should be set to YES  and the ``Scale_K_Continuum`` should be set to NO:
+```
+scale_fitted_lines_only='YES'
+Scale_K_Continuum='NO'
+```
+ This keyword will only scale only OH lines, not the rest of the spectrum as well. This setting greatly improves sky subtraction for the case where the science target fills the lenslets and there are no true sky locations. It may also help in other cases. Users are encouraged to try this option if they see large residuals in sky subtraction, or if the residual continuum is problematic.
+
+- The cosmic ray module is now automatically turned off for all data with the new detector (see reasoning below). Cosmic rays represent about 1% of the bad pixels in a typical 15 minute exposure -- the majority are static bad pixels that should now be accounted for by the bad pixel mask.
+
 ## Release Notes for v4.0.0
 **2017-01-23**
 
@@ -26,7 +45,7 @@ Minor Updates:
 ## Important Runtime Notes
 
 **Cosmic Ray Module**
-- We do not recommend running the cosmic ray module on data in 2016 or later, as it will introduce significant artifacts and reduce overall signal to noise. This is due to the fact that unresolved emission lines such as OH lines are now sharper on the detector. The cosmic ray module will tend to interpret this as a cosmic ray. To remove cosmic rays, we recommend combining data with the mosaic module with the MEANCLIP keyword if there are sufficient number of frames. 
+- We do not recommend running the cosmic ray module on data in 2016 or later, as it will introduce significant artifacts and reduce overall signal to noise. This is due to the fact that unresolved emission lines such as OH lines are now sharper on the detector. The cosmic ray module will tend to interpret this as a cosmic ray. To remove cosmic rays, we recommend combining data with the mosaic module with the MEANCLIP keyword if there are sufficient number of frames.
 - The cosmic ray module may lead to artifacts in data before 2016 as well, but at a lesser level. We recommend checking on individual science cases.
 - The pipeline team is investigating different cosmic ray modules for
 a future release.
@@ -37,7 +56,7 @@ a future release.
 
 **Current Important OSIRIS Issues**
 
-- For certain cases, there are flux mis-assignment: [Issue 20](https://github.com/Keck-DataReductionPipelines/OsirisDRP/issues/20), [wiki link](https://github.com/Keck-DataReductionPipelines/OsirisDRP/wiki/Tests:-Quantified-Flux-Mis-assignment)
+- For certain cases, there are flux artifacts: [Issue 20](https://github.com/Keck-DataReductionPipelines/OsirisDRP/issues/20), [wiki link](https://github.com/Keck-DataReductionPipelines/OsirisDRP/wiki/Tests:-Quantified-Flux-Mis-assignment)
 - Spatial rippling is seen in the integrate flux of sky lines spatially across the field: [Issue 21](https://github.com/Keck-DataReductionPipelines/OsirisDRP/issues/21)
 - [2016-09-07 OSIRIS Hackathon report](https://drive.google.com/open?id=0B_YkzZoUSrX-YnpCRjVZRkRPWnM) on these and other issues from the most recent OSIRIS Hackathon
 
@@ -48,32 +67,17 @@ To install and run the OSIRIS DRP, you will need the following:
 
 - A working C compiler (e.g. ``gcc``)
 - A copy of the compiled library cfitsio
-- A working installation of IDL (the IDL binary directory should be in your ``PATH`` environment variable)
+- A working installation of IDL 7 or IDL 8 (the IDL binary directory should be in your ``PATH`` environment variable)
 - Python dependencies (optional, for testing): pytest, astropy
 
-### Options to install the pipeline
-
-- [Using a python script](#installing-with-a-python-script) -- this is the simplest method for the most recent release
-- [Manually install from github repository](#installing-from-source) -- useful if you want to use the latest pipeline development version
-
-### Installing with a python script
-
-- Download the installation script [install_osiris_drs.py](https://github.com/Keck-DataReductionPipelines/OsirisDRP/blob/release/scripts/install_osiris_drs.py)
-- Run the script
-```
-chmod +x install_osiris_drs.py
-./install_osiris_drs.py
-```
-- For more information and script options:
-```
-./install_osiris_drs.py --help
-```
 
 ### Installing from source
 
-Either clone or download the source from github (the [``develop``](https://github.com/Keck-DataReductionPipelines/OsirisDRP/tree/develop) branch has the latest development)
+Either clone or download the source from github, choose either the master branch or the develop branch.
+ - the [``master``](https://github.com/Keck-DataReductionPipelines/OsirisDRP) branch as the latest official release.
+ - the [``develop``](https://github.com/Keck-DataReductionPipelines/OsirisDRP/tree/develop) branch has the latest development
 
-Set up the following environment variables (optional). The defaults should work for installations of IDL on Mac OS X and CFITSIO installed using [MacPorts][]:
+Set up the following environment variables to compile the code (you can remove these variables after compiling). The defaults should work for installations of IDL on Mac OS X and CFITSIO installed using [MacPorts][]:
 
 - ``IDL_INCLUDE``: The IDL include directory. If you don't set ``IDL_INCLUDE``, it defaults to ``IDL_INCLUDE=/Applications/exelis/idl/external/include``
 - ``CFITSIOLIBDIR``: The directory containing your installation of CFITSIO. If you don't set ``CFITSIOLIBDIR``, it will default to ``CFITSIOLIBDIR=/opt/local/lib``, which is correct for [MacPorts][].
@@ -88,7 +92,7 @@ You should see that the pipeline has been built correctly. Be sure you are using
 
 [MacPorts]: https://www.macports.org
 
-### OSIRIS DRP Runtime Environment
+### Setup OSIRIS DRP Runtime Environment
 
 The OSIRIS DRP requires various environment variables to find and run
 the pipeline. Instructions are below for bash (should work for other
@@ -101,60 +105,23 @@ your OSIRIS pipeline environment.
 
 #### Environment Setup in Bash
 
-To setup the OSIRIS environment, source the file
-``scripts/osirisSetup.sh``, then run ``osirisSetup`` with the root
-directory of your OSIRIS DRF installation. If your OSIRIS pipeline is
-installed in ``/usr/local/osiris/drs/``, then you would do:
-
-```
-$ source scripts/osirisSetup.sh
-To use the OSIRIS DRP, run osirisSetup /path/to/my/drp
-
-$ osirisSetup /my/path/to/osiris/drp/
-Setting OSIRIS_ROOT=/my/path/to/osiris/drp/
-Adding /my/path/to/osiris/drp/scripts to your path.
-Successfully setup OSIRIS DRP environment.
-The DRP is in /my/path/to/osiris/drp/
-```
-
-You can change all of the relevant OSIRIS variables later by running
-``osirisSetup`` again. ``osirisSetup`` will add
-``$OSIRIS_ROOT/scripts`` to your environment's PATH variable by
-default. To skip this step, run ``osirisSetup`` with ``-n``:
-
-```
-$ osirisSetup -n /my/path/to/osiris/drp/
-Setting OSIRIS_ROOT=/my/path/to/osiris/drp/
-Successfully setup OSIRIS DRP environment.
-The DRP is in /my/path/to/osiris/drp/
-```
-
-You can add these lines to your ``.bashrc`` file or other startup profile if you want to set up the osiris environment variables for all of your shell sessions. Add lines like this to your profile:
+You can add these lines to your ``.bashrc`` file or other startup profile if you want to set up the osiris environment variables for all of your shell sessions. Add these lines to your profile:
 
 ```
 OSIRIS_VERBOSE=0
 source /my/path/to/osiris/drp/scripts/osirisSetup.sh
 osirisSetup /my/path/to/osiris/drp
 ```
 
-
-#### Environment Setup in CSH
-
-To setup the OSIRIS environment, set the environment variable
-``OSIRIS_ROOT`` to the root directory for the OSIRIS data reduction
-pipeline. Then source the file ``scripts/osirisSetup.csh``.
+Remember if your IDL binary is not in your path, you should also add it to your ``.bashrc`` file, for example:
 
 ```
-$ setenv OSIRIS_ROOT /my/path/to/osiris/drp/
-
-$ source scripts/osirisSetup.csh
-Using OSIRIS_ROOT=/my/path/to/osiris/drp/
-Successfully setup OSIRIS DRP environment.
-The DRP is in /my/path/to/osiris/drp/
-You might want to add /my/path/to/osiris/drp/scripts to your PATH.
+export PATH=$PATH:/Applications/exelis/idl/bin
 ```
 
-You can add these lines to your ``.cshrc`` file or other startup profile if you want to set up the osiris environment variables for each of your shell sessions. Add lines like this to your profile:
+#### Environment Setup in CSH
+
+You can add these lines to your ``.cshrc`` file or other startup profile if you want to set up the osiris environment variables for each of your shell sessions. Add these lines to your profile:
 
 ```
 set OSIRIS_VERBOSE=0
@@ -163,6 +130,12 @@ source ${OSIRIS_ROOT}/scripts/osirisSetup.csh
 setenv PATH ${PATH}:${OSIRIS_ROOT}/scripts
 ```
 
+Remember if your IDL binary is not in your path, you should also add it to your ``.cshrc`` file, for example:
+
+```
+setenv PATH ${PATH}:/Applications/exelis/idl/bin
+```
+
 ### Running the Pipeline
 
 To run the pipeline, use ``run_odrp``. If you don't want the pipeline
@@ -187,4 +160,44 @@ The first time you run the tests, data will be downloaded so it will take longer
 
 ### Troubleshooting
 
-If you run into problems, please re-read this README.md, then read INSTALLPROBLEMS.md for some common installation problems.
+If you run into problems, please re-read this [README.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP), then read [INSTALLPROBLEMS.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP/blob/master/INSTALLPROBLEMS.md) for some common installation problems.
+
+OSIRIS DRP Project Contributors
+============================
+
+Project Coordinators
+--------------------
+* Jim Lyke (@jlyke-keck)
+* Tuan Do (@followthesheep)
+
+Alphabetical list of contributors
+---------------------------------
+* Anna Boehle (@aboehle)
+* Randy Campbell
+* Sam Chappell
+* Devin Chu
+* Mike Fitzgerald (@astrofitz)
+* Tom Gasawy
+* Christof Iserlohe
+* Alfred Krabbe
+* James Larkin
+* Jim Lyke (@jlyke-keck)
+* Kelly Lockhart
+* Jessica Lu
+* Etsuko Mieda
+* Mike McElwain
+* Marshall Perrin
+* Alex Rudy (@alexrudy)
+* Breann Sitarski
+* Andrey Vayner
+* Greg Walth
+* Jason Weiss
+* Tommer Wizanski
+* Shelley Wright
+
+(If you have contributed to the OSIRIS pipeline and your name is missing,
+please send an email to the coordinators, or
+If you have contributed to the OSIRIS pipeline and your name is missing,
+please send an email to the coordinators, or
+open a pull request for this [page](https://github.com/Keck-DataReductionPipelines/OsirisDRP/edit/master/AUTHORS.rst>)
+in the [OsirisDRP repository](https://github.com/Keck-DataReductionPipelines/OsirisDRP>)

backbone/SupportFiles/RPBconfig.xml

@@ -3,13 +3,9 @@
     adjchan_COMMON___Debug="1"
     ALL_COMMON___Colors="1"
 
-    assembcube_COMMON___CoeffFile         ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs.fits'
-    assembcube_COMMON___05_06CoeffFile    ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs_050222-060222.fits'
-    assembcube_COMMON___06_09CoeffFile    ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs_060223-091004.fits'
-    assembcube_COMMON___09_12CoeffFile    ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs_091005-120103.fits'
-    assembcube_COMMON___12_12CoeffFile    ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs_120104-121109.fits'
-    assembcube_COMMON___13_15CoeffFile    ='$OSIRIS_DRP_DATA_PATH/osiris_wave_coeffs_121110-151231.fits'
-    assembcube_COMMON___Filterfile        ='$OSIRIS_DRP_DATA_PATH/osiris_filter_info.list'
+    ALL_COMMON___CalibrationFile          ='${OSIRIS_DRP_DATA_PATH}/calibrations.xml'
+
+    assembcube_COMMON___Filterfile        ='${OSIRIS_DRP_DATA_PATH}/osiris_filter_info.list'
 
     caelum_mosaic_COMMON___Debug          = '0'
 
@@ -21,7 +17,7 @@
     calibrwave_COMMON___UpperLimitBad     = '5.'
     calibrwave_COMMON___NoiseMultiplier   = '10.'
     calibrwave_COMMON___InterPolType      = 'LINEAR'
-    calibrwave_COMMON___Filterfile        ='$OSIRIS_DRP_DATA_PATH/osiris_filter_info.list'
+    calibrwave_COMMON___Filterfile        ='${OSIRIS_DRP_DATA_PATH}/osiris_filter_info.list'
 
     combframes_COMMON___Combine_Method		 = 'AVERAGE'
 
@@ -32,7 +28,7 @@
     corrtilt_COMMON___Pressure   ="640."
     corrtilt_COMMON___Temperature="273."
 
-    detspecres_COMMON___Linefile         = '$OSIRIS_DRP_DATA_PATH/osiris_cal_line.list'
+    detspecres_COMMON___Linefile         = '${OSIRIS_DRP_DATA_PATH}/osiris_cal_line.list'
     detspecres_COMMON___FitHalfWindow    ='12'
     detspecres_COMMON___FitFunction      ='GAUSSIAN'
     detspecres_COMMON___FitTerms         ='4'
@@ -90,15 +86,15 @@
     mkwavcalfr_COMMON___MaxLag_px            = '20'
     mkwavcalfr_COMMON___DispFitSigma_fact    = '2.'
     mkwavcalfr_COMMON___DispFitIter          = '2'
-    mkwavcalfr_COMMON___Linefile             = '$OSIRIS_DRP_DATA_PATH/osiris_cal_line.list'
-    mkwavcalfr_COMMON___Filterfile           = '$OSIRIS_DRP_DATA_PATH/osiris_filter_info.list'
+    mkwavcalfr_COMMON___Linefile             = '${OSIRIS_DRP_DATA_PATH}/osiris_cal_line.list'
+    mkwavcalfr_COMMON___Filterfile           = '${OSIRIS_DRP_DATA_PATH}/osiris_filter_info.list'
     mkwavcalfr_COMMON___MedianLag            = '1'
 
     mkwavsmo_COMMON___Debug                  = '1'
     mkwavsmo_COMMON___Order                  = '2'
     mkwavsmo_COMMON___MedianWindow           = '5'
     mkwavsmo_COMMON___LimitPixel             = '2'
-    mkwavsmo_COMMON___FilterFile             = '$OSIRIS_DRP_DATA_PATH/osiris_filter_info.list'
+    mkwavsmo_COMMON___FilterFile             = '${OSIRIS_DRP_DATA_PATH}/osiris_filter_info.list'
 
     mosaicdpos_COMMON___Debug             = '1'
     mosaicdpos_COMMON___Magnification     = '1.'
@@ -122,10 +118,11 @@
     scaledskysub_COMMON___Scale_K_Continuum = "Yes"
     scaledskysub_COMMON___line_halfwidth = "4"
     scaledskysub_COMMON___show_plots = "Yes"
+    scaledskysub_COMMON___scale_fitted_lines_only = "No"
 
     spatrectif_COMMON___numiter="4"
     spatrectif_ARP_SPEC_numiter="40"
-    spatrectif_ORP_SPEC_numiter="25"
+    spatrectif_ORP_SPEC_numiter="4"
     spatrectif_COMMON___relaxation="1.0"
 
     srtrectdat_COMMON___Debug="1"
@@ -217,6 +214,7 @@
       <Argument Name="Line_Halfwidth" Type="float" Range="(1,20)" Default="4" />
       <Argument Name="Show_Plots" Type="enum" Range="YES|NO" Default="YES" />
       <Argument Name="Scale_K_Continuum" Type="enum" Range="YES|NO" Default="YES" />
+      <Argument Name="scale_fitted_lines_only" Type="enum" Range="YES|NO" Default="NO" />
   </Module> 
     <Module Name="Mosaic Frames" IDLFunc="mosaic_000" Comment="Mosaic Frames combines multiple OSIRIS cubes with the same PA using header keywords from TEL (default), NGS, and LGS. Combining frames may be performed with either AVERAGE (default), MEDIAN, or MEANCLIP.">
       <Argument Name="Combine_Method" Type="enum" Range="AVERAGE|MEDIAN|MEANCLIP|MEANCLIP2" Default="AVERAGE" />

backbone/drpRun.pro

@@ -18,7 +18,7 @@ PRO drpRun, QUEUE_DIR=queue_dir
         print, "*                                                   *"
         print, "*###################################################*"
         print, "*                                                   *"
-        print, "*                   VERSION 4.0.0                   *"
+        print, "*                   VERSION 4.1.0beta               *"
         print, "*                                                   *"
         print, "*           James Larkin, Shelley Wright,           *"
         print, "*            Jason Weiss, Mike McElwain,            *"