Skip to content

Commit

Permalink
Changed CRDS queries
Browse files Browse the repository at this point in the history
  • Loading branch information
@tddesjardins
tddesjardins committed May 31, 2018
1 parent 756bff4 commit 07f7bb8
Showing 1 changed file with 34 additions and 101 deletions.
135 changes: 34 additions & 101 deletions notebooks/acs_subarrays.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"source": [
"### Subarray Definitions\n",
"\n",
"A comprehensive summary of the subarray readout configuations can be found in [ACS ISR 2017-03](http://www.stsci.edu/hst/acs/documents/isrs/isr1703.pdf). For subarray data taken prior to Serviving Mission 4 (SM4), the 2-D bias and dark images are cut out of the full-frame reference files during calibration using `calacs`. However, after SM4 the clock speed of the subarray readout was different than the full-frame, resulting in the need to acquire bias images in each subarray mode to match observations. The subarray definitions in the flight software were updated at the start of Cycle 24 to match the clock speed of the readout, which solves the need for separate bias imaging. Most noticeably, the new subarray definitions allow users to reduce the number of rows in the image, but the number of columns will always be 2072 (2048 + 24 bias prescan columns). This is markedly different from the pre-Cycle 24 format in which both the number of columns and rows were reduced. For example, the old 512 subarray, which measured 512 rows x 534 columns in the raw image, is now 512 x 2072. In addition, the post-Cycle 24 2K subarray also includes 20 rows of virtual overscan to match the full-frame readout."
"A comprehensive summary of the subarray readout configuations can be found in [ACS ISR 2017-03](http://www.stsci.edu/hst/acs/documents/isrs/isr1703.pdf). For subarray data taken prior to Servicing Mission 4 (SM4), the 2-D bias and dark images are cut out of the full-frame reference files during calibration using `calacs`. However, after SM4 the clock speed of the subarray readout was different than the full-frame, resulting in the need to acquire bias images in each subarray mode to match observations. The subarray definitions in the flight software were updated at the start of Cycle 24 to match the clock speed of the readout, which solves the need for separate bias imaging. Most noticeably, the new subarray definitions allow users to reduce the number of rows in the image, but the number of columns will always be 2072 (2048 + 24 bias prescan columns). This is markedly different from the pre-Cycle 24 format in which both the number of columns and rows were reduced. For example, the old 512 subarray, which measured 512 rows x 534 columns in the raw image, is now 512 x 2072. In addition, the post-Cycle 24 2K subarray also includes 20 rows of virtual overscan to match the full-frame readout."
]
},
{
Expand All @@ -39,13 +39,13 @@
"\n",
"<div class=\"alert alert-info\">In an effort to provide the best-calibrated data possible in the archive, all custom subarray data have been calibrated by the ACS Team and future automated reprocessing has been disabled. Please see the [Status of ACS Data in MAST]() page for information regarding impacted data.</div>\n",
"\n",
"User-defined (or \"custom\") subarray data were an available-but-unsupported mode prior to SM4 (and then a restricted mode until Cycle 24 when their use was discontinued), however the calibration of these data is complicated within `CALACS`. The `CALACS` data pipeline utilizes the `OSCNTAB` reference file to define the physcial bias prescan and virtual bias overscan regions. During the `BLEVCORR` step in `ACSCCD`, the physical prescan regions are used to subtract the bias level from each row, and then the prescan and overscan regions are excised from the image. Because custom subarray data can include any or none of the prescan columns, they are not defined in `OSCNTAB`. We show how to update the table with a new definition for a custom subarray observation in the example below.\n",
"User-defined (or \"custom\") subarray data were an available-but-unsupported mode prior to SM4 (and then a restricted mode until Cycle 24 when their use was discontinued), however the calibration of these data is complicated within `CALACS`. The `CALACS` data pipeline utilizes the `OSCNTAB` reference file to define the physcial bias prescan and virtual bias overscan regions. During the `BLEVCORR` step in `ACSCCD`, the physical prescan regions are used to subtract the bias level from each row, and then the prescan and overscan regions are trimmed from the image. Because custom subarray data can include any or none of the prescan columns, they are not defined in `OSCNTAB`. We show how to update the table with a new definition for a custom subarray observation in the example below.\n",
"\n",
"The subtraction of the 2-D bias (\"superbias\") image from custom subarray data is also complicated by two factors:\n",
"\n",
"1. Pre-SM4: If the data do not contain any bias prescan columns, the `BLEVCORR` step will be skipped and the bias level will be entirely removed using the superbias image, which must be made from bias frames taken as part of the observing program to include the full bias level. This is in contrast to subarray data with bias prescan columns that use a cutout of the superbias image from the full-frame reference file.\n",
"\n",
"2. Post-SM4: As discussed above, differences in the clocking rate between subarray and full-frame observations taken after SM4 require subarray data to have corresponding bias frames taken with the same subarray readout defitions to create the superbias image.\n",
"2. Post-SM4: As discussed above, differences in the clocking rate between subarray and full-frame observations taken after SM4 require subarray data to have corresponding bias frames taken with the same subarray readout definitions to create the superbias image.\n",
"\n",
"The [CRDS](https://hst-crds.stsci.edu) is unable to match appropriately sized superbias reference files to custom subarray images. In cases where such a reference file is required, users will need to either create it themselves or contact the ACS Team via the [help desk](http://hsthelp.stsci.edu) for assistance."
]
Expand Down Expand Up @@ -96,11 +96,7 @@
"source": [
"#### Step 2: Update Header Keywords\n",
"\n",
"Next we will need to turn on the CTE correction switch in the primary header and add the names of the `PCTETAB` and `DRKCFILE` reference files to the header. These are the CTE correction parameters table and the CTE-corrected dark current image, respectively. This information can be retrieved using the Python [CRDS](https://hst-crds.stsci.edu) interface. Note that we have prefaced the names of the reference files with '`jref$`', which is the environment variable pointing to the directory containing ACS calibrartion data. You must set the JREF environment variable prior to proceding. Using a bash shell, this is accomplished by typing:\n",
"```\n",
"export JREF=\"/path/to/my/jref/\"\n",
"```\n",
"This example assumes that you have placed the appropriate calibration reference files in your JREF directory. Visit the [CRDS](https://hst-crds.stsci.edu) website to download calibration data."
"Next we will need to turn on the CTE correction switch in the primary header and add the names of the `PCTETAB` and `DRKCFILE` reference files to the header. These are the CTE correction parameters table and the CTE-corrected dark current image, respectively. This information can be retrieved using the [CRDS](https://hst-crds.stsci.edu) `bestrefs` tool."
]
},
{
Expand All @@ -110,30 +106,16 @@
"outputs": [],
"source": [
"from astropy.io import fits\n",
"from crds import getrecommendations\n",
"from crds import client\n",
"\n",
"client.set_crds_server('https://hst-crds.stsci.edu')\n",
"\n",
"# We will need several keyword values from the primary header\n",
"# to match the best reference files to this observation.\n",
"with fits.open('jd5702jmq_raw.fits') as hdu:\n",
" date = hdu[0].header['DATE-OBS']\n",
" time = hdu[0].header['TIME-OBS']\n",
" amps = hdu[0].header['CCDAMP']\n",
" gain = hdu[0].header['CCDGAIN']\n",
"\n",
"params = {'INSTRUME': 'ACS', 'DETECTOR': 'WFC', 'DATE-OBS': date, 'TIME-OBS': time, \n",
" 'CCDAMP': amps, 'CCDGAIN': gain}\n",
"\n",
"reffiles = getrecommendations(parameters=params, reftypes=['PCTETAB', 'DRKCFILE'], \n",
" observatory='hst')\n",
"import os\n",
"\n",
"# Now we update the primary header with the best reference files.\n",
"with fits.open('jd5702jmq_raw.fits', mode='update') as hdu:\n",
" hdu[0].header['PCTECORR'] = 'PERFORM'\n",
" hdu[0].header['PCTETAB'] = 'jref$' + reffiles['pctetab']\n",
" hdu[0].header['DRKCFILE'] = 'jref$' + reffiles['drkcfile']"
"\n",
"os.environ['CRDS_SERVER'] = 'https://hst-crds.stsci.edu'\n",
"os.environ['CRDS_PATH'] = './crds_cache'\n",
"os.environ['JREF'] = './crds_cache'\n",
"\n",
"!crds bestrefs --files 'jd5702jmq_raw.fits' --sync-references=1 --update-bestrefs"
]
},
{
Expand Down Expand Up @@ -173,7 +155,7 @@
"source": [
"from stwcs import updatewcs\n",
"\n",
"updatewcs.updatewcs('jd5702jmq_flc.fits')"
"updatewcs.updatewcs('jd5702jmq_flc.fits', use_db=False)"
]
},
{
Expand All @@ -182,7 +164,7 @@
"source": [
"### Example 2: Bias Prescan Subtraction of Custom Subarray Data\n",
"\n",
"Our example data come from a ramp filter observation of the galaxy PLS1345+12 (*HST* proposal 10206). The association name is J92SA0010, and we will only one image in the association: J92SA0W6Q. The raw image has 1180 columns and 1200 rows, which does not correspond to any entry in the `OSCNTAB` file, but a visual examination of the image shows that it contains bias prescan columns.\n",
"Our example data come from a ramp filter observation of the galaxy PLS1345+12 (*HST* proposal 10206). The association name is J92SA0010, and we will only use one image in the association: J92SA0W6Q. The raw image has 1180 columns and 1200 rows, which does not correspond to any entry in the `OSCNTAB` file, but a visual examination of the image shows that it contains bias prescan columns.\n",
"\n",
"The [acstools](http://acstools.readthedocs.io) package has been updated to include a tool for checking the `OSCNTAB` file if any entry matches the combination of readout amplifier, image size, and number of bias prescan columns for a given subarray observation. We will need to use several header keyword values to test if a subarray is in `OSCNTAB` and, if not, we will update the reference file appropriately.\n",
"\n",
Expand All @@ -204,11 +186,11 @@
"Observations.download_products(obs_table['obsid'], mrp_only=False,\n",
" productSubGroupDescription=['RAW', 'SPT'])\n",
"\n",
"files = glob.glob('mastDownload/HST/*W6Q/*.fits')\n",
"files = glob.glob('mastDownload/HST/*w6q/*.fits')\n",
"for f in files:\n",
" shutil.move(f, '.')\n",
" \n",
"shutil.rmtree('mastDownload')"
"#shutil.rmtree('mastDownload')"
]
},
{
Expand All @@ -229,9 +211,21 @@
"\n",
"# The name in the header is of the format jref$oscn_name.fits,\n",
"# therefore we need to split the string on the '$' character.\n",
"with fits.open('j92sa0w6q_raw.fits') as hdu:\n",
"with fits.open('j92sa0w6q_raw.fits', mode='update') as hdu:\n",
" prihdr = hdu[0].header\n",
" scihdr = hdu[1].header\n",
" \n",
" prihdr['DARKCORR'] = 'OMIT'\n",
" prihdr['BIASCORR'] = 'OMIT'\n",
" prihdr['SINKCORR'] = 'OMIT'\n",
" prihdr['FLATCORR'] = 'OMIT'\n",
" prihdr['PHOTCORR'] = 'OMIT'\n",
" \n",
"os.environ['CRDS_SERVER'] = 'https://hst-crds.stsci.edu'\n",
"os.environ['CRDS_PATH'] = './crds_cache'\n",
"os.environ['JREF'] = './crds_cache'\n",
"\n",
"!crds bestrefs --files 'j92sa0w6q_raw.fits' --sync-references=1 --update-bestrefs\n",
"\n",
"oscn_name = prihdr['OSCNTAB'].split('$')[-1]\n",
"\n",
Expand All @@ -250,24 +244,9 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Subarray Definition\n",
"-------------------\n",
"Readout Amplifier: A\n",
"X-size: 1180 pixels\n",
"Y-size: 1200 pixels\n",
"Leading prescan columns: 16\n",
"Trailing prescan columns: 0\n",
"Defined in OSCNTAB? True\n"
]
}
],
"outputs": [],
"source": [
"from acstools import utils_calib\n",
"\n",
Expand Down Expand Up @@ -311,32 +290,9 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=1</i>\n",
"<table id=\"table120704485080\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>CCDAMP</th><th>CCDCHIP</th><th>BINX</th><th>BINY</th><th>NX</th><th>NY</th><th>TRIMX1</th><th>TRIMX2</th><th>TRIMY1</th><th>TRIMY2</th><th>BIASSECTA1</th><th>BIASSECTA2</th><th>BIASSECTB1</th><th>BIASSECTB2</th><th>DESCRIPTION</th><th>VX1</th><th>VX2</th><th>VY1</th><th>VY2</th></tr></thead>\n",
"<thead><tr><th>str4</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>str67</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th></tr></thead>\n",
"<tr><td>A</td><td>1</td><td>1</td><td>1</td><td>1180</td><td>1200</td><td>16</td><td>0</td><td>0</td><td>0</td><td>11</td><td>16</td><td>0</td><td>0</td><td>Custom OSCN</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
"</table>"
],
"text/plain": [
"<Table length=1>\n",
"CCDAMP CCDCHIP BINX BINY NX NY ... DESCRIPTION VX1 VX2 VY1 VY2 \n",
" str4 int16 int16 int16 int16 int16 ... str67 int16 int16 int16 int16\n",
"------ ------- ----- ----- ----- ----- ... ----------- ----- ----- ----- -----\n",
" A 1 1 1 1180 1200 ... Custom OSCN 0 0 0 0"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"from astropy.table import Table\n",
"\n",
Expand Down Expand Up @@ -418,32 +374,9 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<i>Table length=1</i>\n",
"<table id=\"table120704874536\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>CCDAMP</th><th>CCDCHIP</th><th>BINX</th><th>BINY</th><th>NX</th><th>NY</th><th>TRIMX1</th><th>TRIMX2</th><th>TRIMY1</th><th>TRIMY2</th><th>BIASSECTA1</th><th>BIASSECTA2</th><th>BIASSECTB1</th><th>BIASSECTB2</th><th>DESCRIPTION</th><th>VX1</th><th>VX2</th><th>VY1</th><th>VY2</th></tr></thead>\n",
"<thead><tr><th>str4</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th><th>str67</th><th>int16</th><th>int16</th><th>int16</th><th>int16</th></tr></thead>\n",
"<tr><td>A</td><td>1</td><td>1</td><td>1</td><td>1180</td><td>1200</td><td>16</td><td>0</td><td>0</td><td>0</td><td>11</td><td>16</td><td>0</td><td>0</td><td>Custom OSCN</td><td>0</td><td>0</td><td>0</td><td>0</td></tr>\n",
"</table>"
],
"text/plain": [
"<Table length=1>\n",
"CCDAMP CCDCHIP BINX BINY NX NY ... DESCRIPTION VX1 VX2 VY1 VY2 \n",
" str4 int16 int16 int16 int16 int16 ... str67 int16 int16 int16 int16\n",
"------ ------- ----- ----- ----- ----- ... ----------- ----- ----- ----- -----\n",
" A 1 1 1 1180 1200 ... Custom OSCN 0 0 0 0"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"Table(new_table)"
]
Expand Down

0 comments on commit 07f7bb8

Please sign in to comment.