Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

histogram.go: use of memory can be made more efficient by using a circular buffer #70

Open
keep94 opened this issue Oct 6, 2021 · 0 comments

Comments

@keep94
Copy link
Contributor

keep94 commented Oct 6, 2021

This line here

h.priorTimedBinsList = h.priorTimedBinsList[1:]
makes inefficient use of memory. The bin that used to be at h.priorTimesBinList[0] doesn't go away, but it stays in the memory chunk allocated for the h.priorTimesBiList slice. If h.maxBins = 100, the capacity of the memory chunk holding the slice will eventually be 200. As bins are added and removed from the slice, the usable portion of the bins slice of 100 will walk up the memory chunk. When the bins slice reaches the end of the memory chunk of 200, go's append will allocate a new memory chunk of 200 and copy the bin slice of 100 to the start of that new memory chunk, and the bins slice will begin walking up that new memory chunk. What this means is that go programs using this histogram library will be doing more memory allocations and holding onto more memory than necessary. Also when the bins slice is near the end of its memory chunk, all of those TDigest objects in bins that fell off the beginning of the slice may not get GCed until the memory chunk gets GCed. A circular buffer would be good to use here. The circular buffer would only need to be allocated one time. Moreover, when the circular buffer starts overwriting old bins, it may be possible to reuse those old TDigest objects rather than allocate new ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants