Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Roman jump detection to ramp fitting #215

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
73a0b94
Begin refactor of Cython code by moving it into its own module
WilliamJamieson Aug 14, 2023
d3d8c31
Clean up the fit_one_ramp function
WilliamJamieson Aug 14, 2023
afe4450
Split cython code into two modules
WilliamJamieson Aug 15, 2023
3a5961e
Add statistic computer
WilliamJamieson Aug 15, 2023
0ae6afd
Convert to nicer data types
WilliamJamieson Aug 15, 2023
62fe82b
Remove output struct
WilliamJamieson Aug 15, 2023
d21eaf1
Fold fit into ramp class object
WilliamJamieson Aug 15, 2023
1dc649c
Refactor variable names for fit_ramps
WilliamJamieson Aug 15, 2023
99fabcc
Add vector support
WilliamJamieson Aug 16, 2023
fa5a33c
More vectors
WilliamJamieson Aug 16, 2023
abe53f9
Move entirely to the fit being in cython, and dynamically vectorized
WilliamJamieson Aug 23, 2023
5d139f2
Rename main fit file
WilliamJamieson Aug 23, 2023
aabbe4e
Convert jump into an object
WilliamJamieson Aug 23, 2023
76eef89
Initial full jump algorithm
WilliamJamieson Aug 23, 2023
52cc6ab
Create a fixed parameter object
WilliamJamieson Aug 23, 2023
62a3b44
Begin process of moving computations to outer most loops
WilliamJamieson Aug 24, 2023
2a39a83
Combine jump and ramp
WilliamJamieson Aug 24, 2023
cb3b14a
Swap to passing structs
WilliamJamieson Aug 24, 2023
12aab35
Move fixed into its own file
WilliamJamieson Aug 24, 2023
5df44fa
Move ramp into its own file
WilliamJamieson Aug 24, 2023
97cd237
Remove jump file
WilliamJamieson Aug 24, 2023
c0fe85f
Add core documentation
WilliamJamieson Aug 24, 2023
43f08e3
Document the fixed data structure
WilliamJamieson Aug 24, 2023
97adae8
Document the ramp object
WilliamJamieson Aug 24, 2023
8be95bb
Fold theshold into an object so it doesn't have to be passed around
WilliamJamieson Aug 24, 2023
56d4f45
Drop public attributes as the things in question don't have to be acc…
WilliamJamieson Aug 24, 2023
7adc4b5
Fix cython lint issues
WilliamJamieson Aug 24, 2023
89d8332
Add initial ramps creator
WilliamJamieson Aug 25, 2023
b5bede3
Add MA table reader
WilliamJamieson Aug 25, 2023
ade66d6
Swap over to using "jump" for all the ramps
WilliamJamieson Aug 25, 2023
9d9fa15
Move from Ramp to Pixel
WilliamJamieson Aug 25, 2023
0e064c8
"fit" to "ramp_fit"
WilliamJamieson Aug 25, 2023
03f9139
Use C++ list templates instead of vectors
WilliamJamieson Aug 25, 2023
5721c9d
Use deque
WilliamJamieson Aug 25, 2023
20b2d9b
Collect ramp resultant indices
WilliamJamieson Aug 25, 2023
9d5be88
Clean up pre-computation code
WilliamJamieson Aug 25, 2023
d7877bd
Add some testing
WilliamJamieson Aug 30, 2023
e0981a4
Some partial fixes for the correction factor
WilliamJamieson Aug 31, 2023
98033dc
Better handle vector vs memory view for fixed data
WilliamJamieson Aug 31, 2023
a5d13ab
Finish testing fixed object
WilliamJamieson Aug 31, 2023
4aa6224
Add tests for building pixels
WilliamJamieson Aug 31, 2023
fac7c5b
Add more testing for ramp fitting
WilliamJamieson Sep 5, 2023
bdf1351
Fixed offset issue
WilliamJamieson Sep 6, 2023
6eca68a
Bugfix endpoint
WilliamJamieson Sep 6, 2023
8fffb21
Clean up final ramp fitter
WilliamJamieson Sep 6, 2023
b5de3e6
Change Thresh from class to struct
WilliamJamieson Sep 6, 2023
308c2ee
Add tests for fitting ramps with dq flags
WilliamJamieson Sep 20, 2023
1fa8078
Add testing of jump detection (with no jumps added) to existing tests
WilliamJamieson Sep 20, 2023
ea95dcd
add test with jumps in data
WilliamJamieson Sep 20, 2023
8cbc2cd
Clean up some indexing and variables
WilliamJamieson Sep 26, 2023
059c41e
Fix the jump detection indexing issues.
WilliamJamieson Sep 26, 2023
8c5ef0d
Use NaN for signalling degenerancy instead of 0
WilliamJamieson Sep 27, 2023
7ecd0d5
Update doc strings
WilliamJamieson Sep 27, 2023
a795cbd
Use enum to enumerate single vs double diff dimensions
WilliamJamieson Sep 27, 2023
68f9940
Clean up documentation further
WilliamJamieson Sep 27, 2023
ceef0e5
Record jumps
WilliamJamieson Sep 27, 2023
c35c06e
Use "jump" instead of "split"
WilliamJamieson Sep 27, 2023
c6fed09
Start to adapt code to use the new ramp_fit
WilliamJamieson Sep 27, 2023
dc071e4
Bugfix for off by one indexing for jump and add proper jump test
WilliamJamieson Oct 3, 2023
22c5fbf
Fix integration with rest of stcal
WilliamJamieson Oct 4, 2023
3e5c588
Move array building to cython
WilliamJamieson Oct 4, 2023
92e9508
Minor cleanups with enum values
WilliamJamieson Oct 4, 2023
d6f0227
Fix all negative indexing.
WilliamJamieson Oct 4, 2023
641c33b
Clean up style issues
WilliamJamieson Oct 4, 2023
d0891b0
Update changes
WilliamJamieson Oct 4, 2023
7399700
Remove unused variable
WilliamJamieson Oct 4, 2023
e7b6f4a
Change name of `DerivedData`
WilliamJamieson Oct 5, 2023
e86c939
Change name of Fixed to FixedValues
WilliamJamieson Oct 5, 2023
639573b
Change name of some FixedValues parameters
WilliamJamieson Oct 5, 2023
0778833
Clean up Pixel variable names
WilliamJamieson Oct 5, 2023
6398d0a
Clean up doc strings
WilliamJamieson Oct 5, 2023
26ab61f
Add t_bar_diff_sqrs
WilliamJamieson Oct 5, 2023
e7e5c87
Use new correction factor formulation
WilliamJamieson Oct 5, 2023
7111282
Remove ols_cas22_util module
WilliamJamieson Oct 5, 2023
a1c554a
Clean up test constants
WilliamJamieson Oct 5, 2023
167a0bf
Update the random ramp generator
WilliamJamieson Oct 5, 2023
ba82b14
Add full jump detection test with ramp fitting
WilliamJamieson Oct 5, 2023
39eabb5
Remove pure python implementation of cas22 ramp fitting
WilliamJamieson Oct 5, 2023
8e1b70c
Add ability to override jump detection parameters from defaults
WilliamJamieson Oct 6, 2023
208fef7
Add ability to get output dq array marking jump resultants
WilliamJamieson Oct 6, 2023
f7d25f5
Clean up essentially duplicated tests
WilliamJamieson Oct 6, 2023
f9ca47f
Clean up ramp_fit outputs so they are named tuples
WilliamJamieson Oct 6, 2023
d63b805
Update docstrings a little
WilliamJamieson Oct 6, 2023
d364ac8
Fix dq test to actually produce dq flags
WilliamJamieson Oct 6, 2023
4541e22
Add further documentation
WilliamJamieson Oct 6, 2023
3b9b68d
Remove a copy
WilliamJamieson Oct 6, 2023
46bb028
Clarify variable name
WilliamJamieson Oct 11, 2023
987790a
Remove all core function wrappers
WilliamJamieson Oct 11, 2023
366d2b0
Remove all remaining wrappers.
WilliamJamieson Oct 11, 2023
40303d2
Fix reshaping of data in/out of fitting routine
WilliamJamieson Oct 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,9 @@ dmypy.json

# Cython debug symbols
cython_debug/
src/stcal/ramp_fitting/ols_cas22/*.c
src/stcal/ramp_fitting/ols_cas22/*.cpp
src/stcal/ramp_fitting/ols_cas22/*.html

# setuptools-scm generated module
src/stcal/_version.py
Expand Down
6 changes: 6 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@

- Added ``alignment`` sub-package. [#179]

ramp_fitting
------------

- Refactor Casertano, et.al, 2022 uneven ramp fitting and incorporate the matching
jump detection algorithm into it. [#215]

Changes to API
--------------

Expand Down
30 changes: 26 additions & 4 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,31 @@
Options.docstrings = True
Options.annotate = False

extensions = [Extension('stcal.ramp_fitting.ols_cas22',
['src/stcal/ramp_fitting/ols_cas22.pyx'],
include_dirs=[np.get_include()],
extra_compile_args=['-std=c99'])]
extensions = [

Check warning on line 9 in setup.py

View check run for this annotation

Codecov / codecov/patch

setup.py#L9

Added line #L9 was not covered by tests
Extension(
'stcal.ramp_fitting.ols_cas22._core',
['src/stcal/ramp_fitting/ols_cas22/_core.pyx'],
include_dirs=[np.get_include()],
language='c++'
),
Extension(
'stcal.ramp_fitting.ols_cas22._fixed',
['src/stcal/ramp_fitting/ols_cas22/_fixed.pyx'],
include_dirs=[np.get_include()],
language='c++'
),
Extension(
'stcal.ramp_fitting.ols_cas22._pixel',
['src/stcal/ramp_fitting/ols_cas22/_pixel.pyx'],
include_dirs=[np.get_include()],
language='c++'
),
Extension(
'stcal.ramp_fitting.ols_cas22._fit_ramps',
['src/stcal/ramp_fitting/ols_cas22/_fit_ramps.pyx'],
include_dirs=[np.get_include()],
language='c++'
),
]

setup(ext_modules=cythonize(extensions))
220 changes: 0 additions & 220 deletions src/stcal/ramp_fitting/ols_cas22.pyx

This file was deleted.

4 changes: 4 additions & 0 deletions src/stcal/ramp_fitting/ols_cas22/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from ._fit_ramps import fit_ramps, RampFitOutputs
from ._core import Parameter, Variance, Diff, RampJumpDQ

__all__ = ['fit_ramps', 'RampFitOutputs', 'Parameter', 'Variance', 'Diff', 'RampJumpDQ']
58 changes: 58 additions & 0 deletions src/stcal/ramp_fitting/ols_cas22/_core.pxd
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
from libcpp.vector cimport vector
from libcpp.stack cimport stack
from libcpp.deque cimport deque


cdef struct RampIndex:
int start
int end


cdef struct RampFit:
float slope
float read_var
float poisson_var


cdef struct RampFits:
vector[RampFit] fits
vector[RampIndex] index
vector[int] jumps
RampFit average


cdef struct ReadPatternMetadata:
vector[float] t_bar
vector[float] tau
vector[int] n_reads


cdef struct Thresh:
float intercept
float constant


cpdef enum Diff:
single = 0
double = 1


cpdef enum Parameter:
intercept = 0
slope = 1


cpdef enum Variance:
read_var = 0
poisson_var = 1
total_var = 2


cpdef enum RampJumpDQ:
JUMP_DET = 4


cpdef float threshold(Thresh thresh, float slope)
cdef float get_power(float s)
cdef deque[stack[RampIndex]] init_ramps(int[:, :] dq)
cpdef ReadPatternMetadata metadata_from_read_pattern(list[list[int]] read_pattern, float read_time)
Loading
Loading