NIRISS SOSS SUBSTRIP96 flux drop

[stscijgbot-jp]

Issue JP-3745 was created on JIRA by Aarynn Carter:

BACKGROUND: For NIRISS SOSS data, the extract1d step of the pipeline uses (by default) the ATOCA algorithm, which is an optimal spectral extraction routine that disentangles the signals from the overlapping order 1 and order 2 traces. For the SUBSTRIP256 both orders are captured, but for the SUBSTRIP96 order 2 is only partially captured. 

PROBLEM: When using the SUBSTRIP96 subarray, we have identified a characteristic "flux drop" at ~1.5 micron in extracted order 2 spectra that is not observed in comparable SUBSTRIP256 datasets, and is not consistent with the expected astrophysical spectra. This issue was initially coupled with an erroneous wavelength calibration that has been resolved by JP-3588, but it is now clear it has an alternative origin. The current thinking of the SOSS team is that when using SUBSTRIP96 the default ATOCA extraction algorithm is biased by the partial capturing of the order 2 spectrum, or an interaction with the subarray padding, and produces the flux drop. 

An example of this can be seen for program GO-3596. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Just to get some context: JP-3588 is implementing the PASTASOSS approach that does a better job of matching real trace locations using adjustments based on PCWPOS, after which the ATOCA algorithm is applied as usual but at the updated locations.  This ticket is reporting a different issue with uncertain origin that persists even after the PASTASOSS change, potentially with the ATOCA algorithm itself or some other effect.  Is that about accurate?

[stscijgbot-jp]

Comment by Aarynn Carter on JIRA:

Hi David Law, that's right. From our testing this week the new JP-3588 implementation is looking good, but didn't resolve this issue, which was observed prior to the JP-3588 changes. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Assigning to NIRISS team while we sort out what the issue is, at which point we can reassign for the actual fix.  Rachel Plesha let me know what assistance NIRISS needs figuring out what's going on.

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

I think I located the issue. The order 2 trace is weighted in the soss_boxextract.get_box_weights function before doing the spectral extraction. For SUBSTRIP256, this works as intended. But for SUBSTRIP96 (which goes off the detector around column 1300), the order 2 trace is weighted incorrectly. 

This could be fixed by modifying the get_box_weights function for the subarray but I also found that passing an array of ones to the box_weights variable if the subarray is SUBSTRIP96 fixes the drop in flux.

The attached image shows the order 1 extracted spectrum with the flux drop if order 2 is (incorrectly) weighted (red line) as the pipeline currently does. The green line shows the extracted spectrum with the fix implemented.

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

In troubleshooting this, it looks like the weights are almost all 0 or 1 with only a 10 pixel transition of fractional values so I don't it's the taper that is playing a big role since the flux drop region is ~64 pixels wide.

However, if you look at the attached plots, the spec_profile trace (top right) and the order weights (bottom left and right) are very poorly aligned. The spec_profile is also poorly aligned with the actual data. In general, the order weights and the data overlap fine but only because the pixel weights are very generously padded (see order1_zoom.png). 

I suspect it's the intersection of the spec_profile and the weighted image that determines what gets extracted. So the reason setting an array of all ones as the order 2 weights works is that it expands the weighted trace to actually include all the spec_profile pixels at the blue end of order 2 (see order2_zoom.png).

So I think this solution works and can be implemented for this build but only as a placeholder until we redeliver a spec_profile reference file or Tyler Baines' work replaces it. That is, I think the extract1d results are trustworthy with these changes insofar as we trust the spec_profile reffile (which we don't entirely but it's something we're working on).

These changes have no impact on the SUBSTRIP256 extraction.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

The partial fix to set order 2 weights to 1.0 was included in #8983 and will be in build 11.2.

Setting this ticket to on hold for further study.