JP-3736: Add hybrid full-frame boolean to core exposure schema

[emolter]

Resolves JP-3736

Closes spacetelescope/jwst#8766

This PR adds a header keyword HYBRIDFF to indicate if the new hybrid readout mode was used for a NIRSpec MSA target acquisition.

Tasks

• update or add relevant tests
• update relevant docstrings and / or docs/ page
• Does this PR change any API used downstream? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • run jwst regression tests with this branch installed ("git+https://github.com/<fork>/stdatamodels@<branch>")

news fragment change types...

• changes/<PR#>.feature.rst: new feature
• changes/<PR#>.bugfix.rst: fixes an issue
• changes/<PR#>.doc.rst: documentation change
• changes/<PR#>.removal.rst: deprecation or removal of public API
• changes/<PR#>.misc.rst: infrastructure or miscellaneous change

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.52%. Comparing base (e458f1c) to head (1e16207).
Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #362      +/-   ##
==========================================
+ Coverage   67.51%   67.52%   +0.01%     
==========================================
  Files         114      114              
  Lines        5913     5916       +3     
==========================================
+ Hits         3992     3995       +3     
  Misses       1921     1921              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[emolter]

still waiting on confirmation of what the new header keyword should be

[braingram]

Does this correspond to the keyword dictionary ticket https://jira.stsci.edu/browse/JWSTKD-571?
If so I believe that ticket lists the keyword as HYBRIDFF.

[emolter]

sorry, yes, that is the one

[emolter]

starting regression tests here

[braingram]

Looks good to me. Where does this keyword get assigned?

[emolter]

Looks good to me. Where does this keyword get assigned?

It will be assigned to uncal files based on the keyword dictionary, is that what you're asking?

[braingram]

It will be assigned to uncal files based on the keyword dictionary, is that what you're asking?

Thanks! Yes. I didn't want to assume (based on the title) that it was defined in the uncal file (and not something derived and defined by the pipeline).

Perhaps @tapastro can give the approval as I'm very unfamiliar with the "uncal" formatting and would have more questions (like if this keyword name is already decided or pending discussion on the keyword dictionary ticket, etc).

[emolter]

There are a ton of regtest failures but they all appear to be due to the addition of this keyword to the uncal files (although it's pretty tedious to scroll through all the markdown, see my comment on this jira ticket)

Only level 3 products are affected, so this is likely because of the model blender. I'm not entirely sure why the blender picks this up when it's not populated in the level 2 files. @braingram I know you refactored ModelBlender recently, does it check the stdatamodels schemas explicitly or only the header keywords of the inputs?

[braingram]

Only level 3 products are affected, so this is likely because of the model blender. I'm not entirely sure why the blender picks this up when it's not populated in the level 2 files. @braingram I know you refactored ModelBlender recently, does it check the stdatamodels schemas explicitly or only the header keywords of the inputs?

ModelBlender does parse the schemas to figure out which attributes to blend and fills missing values with nans.

[emolter]

ok cool that's what I figured. So @tapastro do you think it's fine to leave as-is?

[tapastro]

I don't fully understand why this behavior exists for the new keyword, but for example cal_step entries don't appear in level 3 products - e.g. when I reprocess a level 3 IFU observation, I don't see an entry for either meta.cal_step.clean_flicker_noise or a FITS entry for 'S_CLNFNS'.

[braingram]

I see a column for S_CLNFNS in the regtest failures linked in this PR.

Linking to the column differrence is difficult (because of the length of the log) but opening the "raw logs" and Ctrl+F for S_CLNFNS I find:

2024-11-13T15:36:11.2610267Z �[1m�[31mE              Keyword TTYPE316 has different values:�[0m
2024-11-13T15:36:11.2610905Z �[1m�[31mE                 a> S_CHGMIG�[0m
2024-11-13T15:36:11.2611431Z �[1m�[31mE                 b> S_CLNFNS�[0m
2024-11-13T15:36:11.2612060Z �[1m�[31mE              Keyword TTYPE317 has different values:�[0m
2024-11-13T15:36:11.2612685Z �[1m�[31mE                 a> S_CLNFNS�[0m
2024-11-13T15:36:11.2613202Z �[1m�[31mE                 b> S_COMB1D�[0m

from a test_nirspec_mos_spec3[b000000030-s2d] failure.

Would you share the files you're testing (and the output)?

[braingram]

You may have to look at the "raw" fits file (or in extra_fits) since the model blender writes to the HDRTAB extension (which I don't see any schema definition for).

[tapastro]

I used that keyword as an example because I was testing on old cal files that don't have the step listed as skipped. Perhaps a better example would be S_FRSTFR - do you also see that listed?

I can test locally with a more uniform file.

[braingram]

Bingo!

2024-11-13T15:36:11.2631050Z �[1m�[31mE              Keyword TTYPE326 has different values:�[0m
2024-11-13T15:36:11.2631932Z �[1m�[31mE                 a> S_EXTR2D�[0m
2024-11-13T15:36:11.2632435Z �[1m�[31mE                 b> S_FRSTFR�[0m
2024-11-13T15:36:11.2633074Z �[1m�[31mE              Keyword TTYPE327 has different values:�[0m
2024-11-13T15:36:11.2633759Z �[1m�[31mE                 a> S_FRSTFR�[0m
2024-11-13T15:36:11.2634266Z �[1m�[31mE                 b> S_FLAT�[0m

[tapastro]

Ah, I've caught up and now realized my oversight - the test failures are in comparison against the HDRTAB extensions, which in my opinion are of little value - these keywords are not listed in the primary header when they do not have a filled value, so the perceived impact to users (cluttering up the primary header with nonsense NaN values) does not actually exist. I'm fine to merge.

[tapastro]

E.g.:

>>> b[5].data['S_RAMP']
chararray(['COMPLETE', 'COMPLETE', 'COMPLETE', 'COMPLETE'], dtype='<U8')
>>> b[0].header['S_RAMP']
'COMPLETE'
>>> b[5].data['S_CLNFNS']
array([nan, nan, nan, nan], dtype='>f8')
>>> b[0].header['S_CLNFNS']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tpauly/miniconda3/envs/jwstdev/lib/python3.12/site-packages/astropy/io/fits/header.py", line 170, in __getitem__
    card = self._cards[self._cardindex(key)]
                       ^^^^^^^^^^^^^^^^^^^^
  File "/Users/tpauly/miniconda3/envs/jwstdev/lib/python3.12/site-packages/astropy/io/fits/header.py", line 1725, in _cardindex
    raise KeyError(f"Keyword {keyword!r} not found.")
KeyError: "Keyword 'S_CLNFNS' not found."

[braingram]

Is that not expected since all of the combined models have that metadata undefined?

The model blender docs don't quite cover that exact case: https://jwst-pipeline.readthedocs.io/en/latest/jwst/model_blender/main.html#combined-model-metadata
But the "rule" is the combined metadata is the first model metadata + any blending (in this case nothing in model 1 and no valid values so no metadata assigned to the combined model).

For the table the docs mention that nan will be filled for missing values: https://jwst-pipeline.readthedocs.io/en/latest/jwst/model_blender/main.html#input-model-metadata-table

I agree that a column of nan isn't very useful. We could change the model blender output.

[tapastro]

Sorry, the quote is demonstrating what I believe to be intended behavior, and it reinforces my earlier statement that I am fine with the existing PR and its associated test failures. My misunderstanding was that the keyword was being "blended" into the main header despite not existing in the input headers, leading to spurious nan entries in the primary header. This is not the case, so I don't have any issues.

[tapastro]

If I had any change to suggest, it would be that we could improve usability of the HDRTAB extension by removing any column that is only nan-filled, but that is low priority and perhaps tricky to implement if we ever expect actual nan values. I don't believe we do, but there could be an exception lurking.