-
Notifications
You must be signed in to change notification settings - Fork 663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add zfree_with_size to optimize sdsfree since we can get zmalloc_size from the header #453
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## unstable #453 +/- ##
============================================
- Coverage 70.07% 70.05% -0.03%
============================================
Files 110 110
Lines 59989 60085 +96
============================================
+ Hits 42037 42092 +55
- Misses 17952 17993 +41
|
I think the zmalloc_size is really expensive to update the |
@lipzhu the update of used_memory has 2 cost aspects, the zmalloc_size and also the fact it's an atomic operation.
|
@lipzhu good point about |
…to align the logic with sdsnewlen function. (#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the #453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]>
…to align the logic with sdsnewlen function. (valkey-io#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the valkey-io#453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]>
…to align the logic with sdsnewlen function. (valkey-io#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the valkey-io#453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]> Signed-off-by: Samuel Adetunji <[email protected]>
zfree updates memory statistics. It gets the size of the buffer from jemalloc by calling zmalloc_size. This operation is costly. We can avoid it if we know the buffer size. For example, we can calculate size of sds from the data we have in its header. This commit introduces zfree_with_size function that accepts both pointer to a buffer, and its size. zfree is refactored to call zfree_with_size. sdsfree uses the new interface for all but SDS_TYPE_5. Signed-off-by: Vadym Khoptynets <[email protected]>
sdsalloc returns sds's length for SDS_TYPE_5. It's not correct for SDS_TYPE_5. This patch makes sdsAllocSize call zmalloc_size for SDS_TYPE_5. sdsalloc is a lower level function that continues to return length for SDS_TYPE_5. Signed-off-by: Vadym Khoptynets <[email protected]>
Instead of zmalloc_size use s_malloc_size. Signed-off-by: Vadym Khoptynets <[email protected]>
Change the doc string for `zfree_with_size` Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Ok, I wasn't able to get as consistent of performance numbers. Diving into a perf analysis shows about a 20% reduction in CPU in sdsfree though. So it seems like there might be other bottlenecks, this is definitely a good optimization. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall it looks good to me, I have a question about an alternative, but it's not a blocker from me. If you still want to go with this approach, I think it's OK to merge.
Yes, let's go with this approach. |
Signed-off-by: Vadym Khoptynets <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, it is a good optimization.
Signed-off-by: Madelyn Olson <[email protected]>
Issue Introduced by #453. When we check the SDS _TYPE_5 allocation size we mistakenly used zmalloc_size which DOES take the PREFIX size into account when no malloc_size support. Later when we free we add the PREFIX_SIZE again which leads to negative memory accounting on some tests. Example test failure: https://github.com/valkey-io/valkey/actions/runs/9654170962/job/26627901497 Signed-off-by: ranshid <[email protected]>
Description
zfree updates memory statistics. It gets the size of the buffer from jemalloc by calling zmalloc_size. This operation is costly. We can avoid it if we know the buffer size. For example, we can calculate size of sds from the data we have in its header.
This commit introduces zfree_with_size function that accepts both pointer to a buffer, and its size. zfree is refactored to call zfree_with_size.
sdsfree uses the new interface for all but SDS_TYPE_5.
Benchmark
Dataset is 3 million strings. Each benchmark run uses its own value size (8192, 512, and 120). The benchmark is 100% write load for 5 minutes.