Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed improvements with memory mapping #27

Open
TomMaullin opened this issue Apr 18, 2020 · 0 comments
Open

Speed improvements with memory mapping #27

TomMaullin opened this issue Apr 18, 2020 · 0 comments
Labels
Low priority This item is an extremely low priority. Not possible currently This item cannot be worked on at present but may one day be possible.

Comments

@TomMaullin
Copy link
Owner

Currently, when working with large arrays, the BLMM code utilizes the numpy memory map. This is extremely fast at first but as more memory maps are read in and out, the code appears to slow down, despite said memory maps being removed from memory and flushed. This appears to be a common problem on stack overflow and, as a result, perhaps alternative packages should be used.

I have tried h5py but found it's performance was worse than the numpy memory map. As all of these objects are built on the python mmap object, it may be best to wait until better support exists for this before trying out any other packages.

This issue is however a low priority, as the "slowing down" mentioned above is only observed for extremely large designs (likely much larger than the average user would ever want).

@TomMaullin TomMaullin added Low priority This item is an extremely low priority. Not possible currently This item cannot be worked on at present but may one day be possible. labels Apr 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Low priority This item is an extremely low priority. Not possible currently This item cannot be worked on at present but may one day be possible.
Projects
None yet
Development

No branches or pull requests

1 participant