Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamMetadata should have a mechanism to allow sequential access to the blocks #38

Open
skmonga opened this issue May 16, 2019 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@skmonga
Copy link
Contributor

skmonga commented May 16, 2019

The current implementation contains the bug in that the metadata has a last blockId maintained along with only a mapping of blockId and the MD5 checksum of it. Thus there is no way that blocks can be consumed in the order in which they were added to the stream thus not providing a sequential access pattern for the blocks. For the correction, a list of blocks can be used to fix it. Two cases need to be carefully looked into

  1. PutBlock with locking
  2. PutBlock without locking

For case 1, only one writer can append blocks at any time thus there is no race condition. However in case 2, multiple writers may be writing to the stream at the same time and there can be interleaving between the writers. It is not necessary that the last blockId should strictly increase and this interleaving is perfectly fine. So proper handling of race condition should be done i.e. atomic updates for the last blockId, list of blocks and mapping of blockId to the checksum as a whole should be done.

@skmonga skmonga added the bug Something isn't working label May 16, 2019
@skmonga skmonga self-assigned this May 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant