JP-3745: Fixed SUBSTRIP96 flux drop

[hover2pi]

Resolves JP-3745

Resolves #8780 .

When using the SUBSTRIP96 subarray, we have identified a characteristic "flux drop" at ~1.5 micron in extracted order 2 spectra that is not observed in comparable SUBSTRIP256 datasets, and is not consistent with the expected astrophysical spectra.

The order 2 trace is weighted in the soss_boxextract.get_box_weights function before doing the spectral extraction. For SUBSTRIP256, this works as intended. But for SUBSTRIP96 (which goes off the detector around column 1300), the order 2 trace is weighted incorrectly.

This could be fixed by modifying the get_box_weights function for the subarray but I also found that passing an array of ones to the box_weights variable for order 2 if the subarray is SUBSTRIP96 works. This is the only change I have made to the code and it resolves the issue.

The attached image shows the order 1 extracted spectrum with the flux drop if order 2 is (incorrectly) weighted (red line) as the pipeline currently does. The green line shows the extracted spectrum with the fix implemented.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[hover2pi]

In troubleshooting this, it looks like the weights are almost all 0 or 1 with only a 10 pixel transition of fractional values so I don't it's the taper that is playing a big role since the flux drop region is ~64 pixels wide.

However, if you look at the attached plots, the spec_profile trace (top right) and the order weights (bottom left and right) are very poorly aligned. The spec_profile is also poorly aligned with the actual data. In general, the order weights and the data overlap fine but only because the pixel weights are very generously padded (see order1_zoom.png). 

I suspect it's the intersection of the spec_profile and the weighted image that determines what gets extracted. So the reason setting an array of all ones as the order 2 weights works is that it expands the weighted trace to actually include all the spec_profile pixels at the blue end of order 2 (see order2_zoom.png).

So I think this solution works and can be implemented for this build but only as a placeholder until we redeliver a spec_profile reference file or future changes deprecate the reffile all together. That is, I think the extract1d results are trustworthy with these changes insofar as we trust the spec_profile reffile (which we don't entirely but it's something we're working on).

These changes have no impact on the SUBSTRIP256 extraction.

[tapastro]

I'm fine with the change - it appears as though the weights are not providing much usefulness, but from a results-based approach it looks good. Clearly, we need to take a closer look at why some of these functions are necessary, if at all.

Regression tests started here: https://github.com/spacetelescope/RegressionTests/actions/runs/12201245106

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.82%. Comparing base (1cc5a6c) to head (f422e50).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/extract_1d/soss_extract/soss_extract.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #8983       +/-   ##
===========================================
+ Coverage   66.64%   76.82%   +10.18%     
===========================================
  Files         375      495      +120     
  Lines       37946    45605     +7659     
===========================================
+ Hits        25289    35037     +9748     
+ Misses      12657    10568     -2089     

Flag	Coverage Δ		*Carryforward flag
nightly	77.42% <0.00%> (?)		Carriedforward from 89bd3e0

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[emolter]

Thanks @hover2pi, this looks ok to me as a temporary fix that we can get in by build 11.2, so LGTM pending a regression test run and a changelog entry.

However, it's not very satisfying that we haven't identified what is actually wrong with the definition of the order 2 trace when passed into get_box_weights. It's also probably not technically quite correct because we would no longer downweight pixels that are partially in, partially out of the box we are extracting. For now, I'd suggest adding a comment with something to the effect of, "this is a temporary fix, there are issues with the trace here". I might also suggest setting the status of the ticket "on hold" instead of "ready for testing" once this is merged so that we remember to revisit this in the future, but that would be @tapastro 's decision.

Would you please point me to the test data you're using for this?

I've been meaning to ping you, by the way, because I'm working on making a major refactor of the soss extraction code, including adding full unit test coverage. You can see my progress here. This is being targeted for build 11.3, so there is plenty of time, but it would probably be good for us to meet at some point to discuss the proposed changes. We can hopefully include a better fix for this problem in that refactor, as well as a regression test that ensures this flux drop does not recur

[melanieclarke]

@hover2pi @tapastro - did we want to get this into 11.2?

[tapastro]

Agree that this is a reasonable band-aid for the solution to get into 11.2, but we'll need to investigate the root cause and address it in the next build. Running regression tests here: https://github.com/spacetelescope/RegressionTests/actions/runs/12304740127

We'll make sure to either leave the existing ticket open or generate a new one to track work for 11.3.

[hover2pi]

@emolter Sorry, I just reread your comment. The data I was using to test this was for the target V-DR-Tau, specifically jw03596-o001_t001_niriss_clear-gr700xd-substrip96.fits.

Chatting about a SOSS extraction refactor would be great!

[melanieclarke]

Regression tests show no failures, probably because we are not testing any SUBSTRIP96 data. @emolter - can we add a regtest for this, for JP-3755?

[emolter]

Regression tests show no failures, probably because we are not testing any SUBSTRIP96 data. @emolter - can we add a regtest for this, for JP-3755?

definitely, planning on it

[melanieclarke]

I'd like to get this change in today, so I will go ahead and add a change fragment myself in a moment.