Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure memory efficiency when working with peak images #25

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

alex-l-kong
Copy link
Contributor

What is the purpose of this PR?

With 1000+ peak images, keeping them all stored in memory is infeasible. They should be saved and loaded in on an individual basis.

How did you implement your changes

This will change the peak image workflow. Currently, they're all housed in a single in-memory xarray. Instead:

  1. In extraction.coordinate_integration, save the peak images directly instead of storing them in the xarray. This will replace the functionality in plotting.save_peak_images
  2. Add a function plotting.plot_peak_hist to replace the notebook cell image_data.sel(peak=[desired_peak_hist], method="nearest").plot.hist(bins=bin_count)
  3. Update extraction.library_matching to use io_utils.list_files to retrieve the peak values instead of using the xarray dimension
  4. Change plotting.save_matched_peak_images to load the matched peak image from extraction_dir instead of indexing into the xarray directly

Remaining issues

Ongoing testing, full testing suite still needs to be implemented.

@alex-l-kong alex-l-kong self-assigned this Aug 29, 2024
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@coveralls
Copy link

coveralls commented Nov 20, 2024

Pull Request Test Coverage Report for Build 12037261796

Details

  • 38 of 38 (100.0%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.3%) to 97.705%

Totals Coverage Status
Change from base Build 11787590598: 0.3%
Covered Lines: 239
Relevant Lines: 239

💛 - Coveralls

@alex-l-kong alex-l-kong requested a review from kxleow November 21, 2024 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants