New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider lowering memory footprint of MasterRecord #245

Open

purplefox opened this issue Sep 13, 2024 · 0 comments

Milestone

Contributor

purplefox commented Sep 13, 2024 •

edited

Loading

For a 1TB database shard, master-record is bout 10MB
For a 10TB database shard approx 100MB.

It's important master record is not to large as we store it in the object store on each metadata update.

This is not bad, but we can consider making it smaller still with some strategies:

using varint for integer fields
for rangestart and range end, we can store the range relative to the entry before it (for levels > 0) as we know entries are ordered by range and there are no overlapping ranges.

This could potentially allow us to halve the size.

purplefox added this to the 1.0 milestone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment