Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid dataframe copies to reduce memory footprint of analysis functions #86

Open
d33bs opened this issue Sep 5, 2024 · 0 comments · May be fixed by #140
Open

Avoid dataframe copies to reduce memory footprint of analysis functions #86

d33bs opened this issue Sep 5, 2024 · 0 comments · May be fixed by #140
Labels
enhancement New feature or request performance Related to performance (processing speed, resources, etc)

Comments

@d33bs
Copy link
Member

d33bs commented Sep 5, 2024 •
edited
Loading

When working with large Pandas dataframes a large amount of memory is required to process the cosmicqc analysis functions, which utilize df.copy(). We could avoid this memory consumption by focusing on only the dataframe columns which are required, appending to existing dataframes when necessary (at a lower resource expense).

Relates to #23
@d33bs d33bs added enhancement New feature or request performance Related to performance (processing speed, resources, etc) labels Sep 5, 2024
@d33bs d33bs mentioned this issue Nov 13, 2024
Open
@d33bs d33bs moved this to Todo in No image left behind Nov 18, 2024
@d33bs d33bs linked a pull request Nov 20, 2024 that will close this issue
Open
11 tasks
@d33bs d33bs moved this from Todo to Needs review in No image left behind Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request performance Related to performance (processing speed, resources, etc)
Projects
No image left behind
Status: Needs review
Milestone
No milestone
1 participant
@d33bs