Optimise read planning #487
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kjnilsson
force-pushed
the
opt-read-planning
branch
2 times, most recently
from
9ed2089 to
e5445cf
Compare
kjnilsson
force-pushed
the
opt-read-planning
branch
2 times, most recently
from
0dcab3e to
79a1d39
Compare
|
minor: there were two typos I edited in the PR description that I think exist in the commit message as well |
Previously we had to do do a linear search through the list of
segment reference to find which file to read a given entry from.
This commit introduced a new module ra_lol which is a
"list of list" type of ordered data structure that replaces
the use of a plain list.
ra_lol keeps data in a list of 64 entry sized "rows" which
subtantially reduces the amount of work needed to find a
given segment ref as we first scan each row to find the
row with the segref in then we can find the fixed size row.
Even a server with 4096 segment only need to scan at most
128 to find any given entry.
This commit also fixes a few issues in segref compaction
and changes the type of the segment ref to a {ra_range:range(), filename()}
to work better with the ra_range module.
kjnilsson
force-pushed
the
opt-read-planning
branch
from
79a1d39 to
9eaf822
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously we had to do do a linear search through the list of
segment reference to find which file to read a given entry from.
This commit introduced a new module
ra_lolwhich is a"list of list" type of ordered data structure that replaces
the use of a plain list.
ra_lol keeps data in a list of 64 entry sized "rows" which
subtantially reduces the amount of work needed to find a
given segment ref as we first scan each row to find the
row with the segref in then we can find the fixed size row.
Even a server with 4096 segment only need to scan at most
128 to find any given entry.
This commit also fixes a few issues in segref compaction
and changes the type of the segment ref to a {ra_range:range(), filename()}
to work better with the ra_range module.
Fixes #486