-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #629 from Keck-DataReductionPipelines/develop
Version 2.4.0
- Loading branch information
Showing
124 changed files
with
98,063 additions
and
5,796 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| Copyright (c) 2023, California Institute of Technology | ||
| All rights reserved. | ||
|
|
||
| Redistribution and use in source and binary forms for academic and other | ||
| non-commercial purposes with or without modification, are permitted provided | ||
| that the following conditions are met: | ||
|
|
||
| * Redistributions of source code must retain the above copyright notice, this | ||
| list of conditions and the following disclaimer. | ||
|
|
||
| * Redistributions in binary form must reproduce the above copyright notice, | ||
| this list of conditions and the following disclaimer in the documentation | ||
| and/or other materials provided with the distribution. | ||
|
|
||
| * Neither the name of the California Institute of Technology (Caltech) nor | ||
| the names of its contributors may be used to endorse or promote products | ||
| derived from this software without specific prior written permission. | ||
|
|
||
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" | ||
| AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE | ||
| IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE | ||
| ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE | ||
| LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | ||
| CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | ||
| SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | ||
| INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | ||
| CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | ||
| ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | ||
| POSSIBILITY OF SUCH DAMAGE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,21 @@ | ||
| # Data Reduction Pipeline for the Keck Planet Finder | ||
| # Data Reduction Pipeline (DRP) for the Keck Planet Finder (KPF) | ||
|
|
||
| [](http://shrek.caltech.edu:4444/job/KPF%20CI/) | ||
| [](https://kpf-pipeline.readthedocs.io/en/latest/) | ||
| [](https://coveralls.io/github/Keck-DataReductionPipelines/KPF-Pipeline?branch=master) | ||
|
|
||
| Facility data reduction pipline for the [Keck Planet Finder](https://exoplanets.caltech.edu/kpf/) high-resolution optical spectrograph. | ||
| The KPF DRP is the facility data reduction pipeline for the [Keck Planet Finder](https://exoplanets.caltech.edu/kpf/) high-resolution optical spectrograph at the W. M. Keck Observatory. It is under active development by the [NASA Exoplanet Science Institute](https://nexsci.caltech.edu), the [Caltech Institute of Technology](https://www.caltech.edu), and the [W. M. Keck Observatory](https://www.keckobservatory.org/). | ||
|
|
||
| Install instructions and documentation is available at [ReadTheDocs](https://kpf-pipeline.readthedocs.io/en/latest/) | ||
| ## Documention | ||
|
|
||
| Under active development by the [NASA Exoplanet Science Institute](https://nexsci.caltech.edu) and [Caltech Institute of Technology](https://www.caltech.edu). | ||
| Install instructions and documentation are available on [ReadTheDocs](https://kpf-pipeline.readthedocs.io/en/latest/) | ||
|
|
||
| ## Citation | ||
|
|
||
| If you use the KPF Data Reduction Pipeline in your research, please cite the following publication: | ||
|
|
||
| * Gibson et al. (2020, SPIE): [ADS entry](https://ui.adsabs.harvard.edu/abs/2020SPIE11447E..42G/abstract); kpf:gibson2020 in [BibTex file](kpf_bibliography.bib) | ||
|
|
||
| If there is no place to include the relevant citations in the text of the publication, please include the following acknowledgment (in LaTeX using the [provided BibTeX entry](kpf_bibliography.bib)): | ||
|
|
||
| "This research made use of the KPF Data Reduction Pipeline \citep{kpf:gibson2020}." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| ## Pipeline logger configurations | ||
| [LOGGER] | ||
| start_log = True | ||
| log_path = pipeline.log | ||
| log_level = info | ||
| log_verbose = True | ||
| log_directory = /data/logs/ | ||
|
|
||
| [ARGUMENT] | ||
| data_type = KPF | ||
| overwrite = True | ||
|
|
||
| # path to input and output data | ||
| output_dir = /data/ | ||
| output_dir_flat = /data/ | ||
| input_dir_root = /data/2D/ | ||
| masters_dir = /data/masters/ | ||
|
|
||
| # the subdirectory containing order trace result, L1 data, L2 data, L2 reweighted data, qlp, bary data | ||
| # output_trace = order_trace/ | ||
| output_trace = masters/ | ||
| output_extraction = L1/ | ||
| output_rv = L2/ | ||
| output_rv_reweighting = '' | ||
| output_qlp = QLP/ | ||
| output_barycorr = bary/ | ||
|
|
||
| # output_clip containing files with polygon clipping information when rectification_method is 'normal' or vertical' | ||
| # (not 'norect') | ||
| output_clip = clip_np/ | ||
|
|
||
| # suffix to 2D (L0 format), L1 and L2 data files | ||
| # for lev0 set with '_2D' suffix | ||
| output_lev0_suffix = _2D | ||
| output_lev1_suffix = _L1 | ||
| output_lev2_suffix = _L2 | ||
|
|
||
| # ccd list for KPF | ||
| ccd_list = ['GREEN_CCD', 'RED_CCD'] | ||
|
|
||
| # for order trace: | ||
| # flat_file: flat file for order trace process. updated upon new flat file. | ||
| # ccd_idx: index in ccd_list for the ccd to be processed in DRP recipe | ||
| #flat_file = KP.20221107.04689.77 | ||
| flat_file = /data/masters/20230527/kpf_20230527_master_flat.fits | ||
| order_trace_files = ['/code/KPF-Pipeline/static/kpf_20230418_order_trace_GREEN_CCD.csv', '/code/KPF-Pipeline/static/kpf_20230418_order_trace_RED_CCD.csv'] | ||
| fitting_poly_degree = 3 | ||
| ccd_idx = [0, 1] | ||
|
|
||
| # for spectral extraction: | ||
| # - update orders_per_ccd, start_order, orderlet_names per new order trace result | ||
| # orders_per_ccd: [<total order for each ccd> ..] | ||
| # orderlet_names: [[<list of extensions of 1D flux for each ccd>], ...], each extension is related to a ccd fiber. | ||
| # start_order: [<index of first orderlet of first order in each ccd>, ...], assuming there are 35 * 5 (or 32 * 5) traces for green (or red) in KPF. | ||
| # if all traces for green (or red) are identified by order trace process, start_order[0] ( or start_order[1]) is 0. | ||
| # if the first n traces for green (or red) are not in the order trace result, then start_order[0] (or start_order[1]) is -n. | ||
| # ex: start_order = [-1, -1] means the traces of sky fiber of first order for both green and red are missing in the order trace result. | ||
| # orderlet_widths_ccds: trace widths per orderlet per ccd to replace the trace widths from the order trace process | ||
| # - note: order_per_ccd, start_order, orderlet_names should (the outer list), orderlet_widths_ccds (the outer list) have size as that of ccd_list. | ||
| # the inner lists of orderlet_names and orderlet_widths_ccds are with the same size. | ||
| orders_per_ccd=[35,32] | ||
| start_order = [-1, 0] | ||
| orderlet_names = [['GREEN_SKY_FLUX', 'GREEN_SCI_FLUX1', 'GREEN_SCI_FLUX2', 'GREEN_SCI_FLUX3', 'GREEN_CAL_FLUX'], ['RED_SKY_FLUX', 'RED_SCI_FLUX1', 'RED_SCI_FLUX2', 'RED_SCI_FLUX3', 'RED_CAL_FLUX']] | ||
| #orderlet_widths_ccds = [[],[]] | ||
| orderlet_widths_ccds = [[-1, -1, -1, 1, -1],[-1, -1, -1, -1, -1]] | ||
|
|
||
| # rectification_method: norect|vertical|normal, method to rectify the trace. | ||
| # extraction_method: summ|optimal, method to do spectral extraction. | ||
| rectification_method = norect | ||
| extraction_method = optimal | ||
|
|
||
| # - fits with wavelength calibration data | ||
| # wls_fits: [ <wavelength solution file for each ccd>]. | ||
| # wave_to_ext: [ <extensions containing wavelength solution data for each ccd>] | ||
| wls_fits = ['masters/20230527/kpf_20230527_master_WLS_cal-LFC-eve_L1.fits', 'masters/20230527/kpf_20230527_master_WLS_cal-LFC-eve_L1.fits'] | ||
| wave_to_ext = [['GREEN_SCI_WAVE1', 'GREEN_SCI_WAVE2', 'GREEN_SCI_WAVE3', 'GREEN_SKY_WAVE', 'GREEN_CAL_WAVE'], ['RED_SCI_WAVE1', 'RED_SCI_WAVE2', 'RED_SCI_WAVE3', 'RED_SKY_WAVE', 'RED_CAL_WAVE']] | ||
| do_db_query_for_one_nearest_wls_master_file = False | ||
| cal_file_level = 1 | ||
| # contentbitmask = 3 means require at least GREEN and RED CCDs in the WLS master file database-queried nearest in time. | ||
| contentbitmask = 3 | ||
| cal_type_pairs = [['WLS','cal-lfc-morn'], ['WLS', 'cal-lfc-eve'], ['WLS', 'autocal-thar-all']] | ||
| # Maximum start-date age of WLS file relative to context.date_dir at 00:00:00 UT, otherwise fall back on wls_fits. | ||
| max_cal_file_age = '2 days' | ||
|
|
||
| # for rv: | ||
| # o/ou/outtrderlet_names_rv: [<extensions of L1 for radial velocity process, a subset of orderlet_names for each ccd>] | ||
| # reweighting_method: ccf_max|ccf_mean, getting the template file for reweighting by checking the maximum or mean ccf among the orders. | ||
| # ccf_ext: [<extension to contain ccf data for each ccd>, ...] | ||
| # rv_ext: extension containing rv result table | ||
| orderlet_names_rv = [['GREEN_SCI_FLUX1', 'GREEN_SCI_FLUX2', 'GREEN_SCI_FLUX3', 'GREEN_CAL_FLUX', 'GREEN_SKY_FLUX'],['RED_SCI_FLUX1', 'RED_SCI_FLUX2', 'RED_SCI_FLUX3', 'RED_CAL_FLUX', 'RED_SKY_FLUX']] | ||
| rv_correct_by_cal = False | ||
| reweighting_method = ccf_static | ||
| ccf_ext = ['GREEN_CCF', 'RED_CCF'] | ||
| rv_ext = RV | ||
| reweighting_enable_masks = [['espresso'], ['espresso']] | ||
| static_ccf_ratio = ['/code/KPF-Pipeline/static/static_green_ccf_ratio_2.csv', '/code/KPF-Pipeline/static/static_red_ccf_ratio_2.csv'] | ||
|
|
||
| # for ca_hk: | ||
| # hk_fiber_list: [<CA-HK spectrometer fibers>] | ||
| # hk_extract_exts: [<extension contaimng 1D extracted spectrum for each fiber in hk_fiber_list>, ...] | ||
| # hk_wave_exts: [<extension containing wavelength solution for each fiber in hk_fiber_list>, ...] | ||
| # hk_bias_fits: bias fits file for bias subtraction in CA-HK extraction | ||
| # hk_dark_fits: dark fits file for dark subtraction in CA-HK extraction | ||
| # hk_trace_path: CA-HK trace file for spectrum extraction | ||
| # hk_wavelength_path: [<wavelength solution data for each fiber>, ...] | ||
| hk_fiber_list = ['sci', 'sky'] | ||
| hk_extract_exts = ['CA_HK_SCI', 'CA_HK_SKY'] | ||
| hk_wave_exts = ['CA_HK_SCI_WAVE', 'CA_HK_SKY_WAVE'] | ||
| hk_dark_fits = None | ||
| hk_bias_fits = None | ||
| hk_trace_path = /code/KPF-Pipeline/kpfMaster_HKOrderBounds20220909.csv | ||
| hk_wavelength_path = ["masters/kpfMaster_HKwave20220909_sci.csv", "masters/kpfMaster_HKwave20220909_sky.csv"] | ||
|
|
||
| # for module process: | ||
| do_l0_to_2d = True | ||
| do_order_trace = True | ||
| do_spectral_extraction = True | ||
| do_rv = True | ||
| do_rv_reweighting = True | ||
| do_hk = True | ||
| do_wavecopy_in_sp = True | ||
| do_qlp = False | ||
| do_bk_subtraction = True | ||
| do_bc = True | ||
|
|
||
| # for L0->2D process | ||
| [WATCHFOR_L0] | ||
| # Define numbers of columns and rows in data section of channel images for each detector. | ||
| channel_datasec_ncols_green = 2040 | ||
| channel_datasec_nrows_green = 2040 | ||
| channel_datasec_ncols_red = 2040 | ||
| channel_datasec_nrows_red = 4080 | ||
| channel_orientation_ref_path_red = /code/KPF-Pipeline/static/kpfsim_ccd_orient_red_2amp.txt | ||
| channel_orientation_ref_path_green = /code/KPF-Pipeline/static/kpfsim_ccd_orient_green.txt | ||
| do_db_query_for_master_files = False | ||
| masterbias_path = /data/masters/20230527/kpf_20230527_master_bias.fits | ||
| masterdark_path = /data/masters/20230527/kpf_20230527_master_dark.fits | ||
| masterflat_path = /data/masters/20230527/kpf_20230527_master_flat.fits | ||
| prescan_reg = [0,4] | ||
| # overscan_method = clippedmean is n_sigma clipping and average over entire post-overscan strip. | ||
| # overscan_method = median is computed row by row. | ||
| overscan_method = clippedmean | ||
| n_sigma = 2.1 | ||
| # overscan_clip is number of columns at start and end of each row in post-overscan strip to ignore. | ||
| overscan_clip = 5 | ||
| overscan_order = 1 | ||
| lev0_ffi_ext_red = ['RED_CCD'] | ||
| lev0_ffi_ext_green = ['GREEN_CCD'] | ||
| quicklook = False | ||
| gain_keyword = 'CCDGAIN' | ||
| input_dir = /data/L0/ | ||
| output_dir = /data/2D/ | ||
|
|
||
| # config file associated with the modules | ||
| [MODULE_CONFIGS] | ||
| order_trace = modules/order_trace/configs/default_recipe_kpf_20220505.cfg | ||
| spectral_extraction = modules/spectral_extraction/configs/default_recipe_kpf_20220505.cfg | ||
| radial_velocity = modules/radial_velocity/configs/default_recipe_kpf_targ.cfg | ||
| hk_spectral_extraction = modules/ca_hk/configs/default_hk.cfg | ||
| quicklook = modules/quicklook/configs/default.cfg | ||
| bias_subtraction = modules/bias_subtraction/configs/default.cfg | ||
| flat_fielding = modules/flat_fielding/configs/default.cfg | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.