Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Overload with racplusplus Package and 300K 512-Dimensional Data Points on 377G Memory #6

Open
lizcBold opened this issue Dec 4, 2023 · 1 comment

Comments

@lizcBold
Copy link

lizcBold commented Dec 4, 2023

Problem Description:
I am encountering a memory overload issue when using the "racplusplus" package to handle a dataset containing 300,000 data points, each consisting of a 512-dimensional vector. Despite having 377GB of available memory on my system, the package exhausts the memory during processing.

Could you please provide more details regarding the memory usage in Chapter 6 of your article - experimental evaluation? How much memory was required to work with the SIFT1M dataset?
Additionally, you mentioned selecting the batch size for calculating the distance matrix. I would like to know the values you used for this section - a number large enough for speedy results yet small enough to avoid memory overload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants
@lizcBold and others