Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DynamoDB] Avoid OOMing from snapshotting #455

Merged
merged 2 commits into from
Aug 1, 2024
Merged

[DynamoDB] Avoid OOMing from snapshotting #455

merged 2 commits into from
Aug 1, 2024

Conversation

Tang8330
Copy link
Collaborator

@Tang8330 Tang8330 commented Aug 1, 2024

Right now, we're reading the entire json.gz file and then writing it.

DynamoDB chunks the files for 1.5M rows per file by default. However, if the row is really large - we're susceptible to OOM errors

@Tang8330 Tang8330 marked this pull request as ready for review August 1, 2024 18:14
@Tang8330
Copy link
Collaborator Author

Tang8330 commented Aug 1, 2024

Hmm, closing this. I think we need to throttle StreamJsonGzipFile, and this doesn't do that exactly.

@Tang8330 Tang8330 closed this Aug 1, 2024
@Tang8330 Tang8330 reopened this Aug 1, 2024
@Tang8330
Copy link
Collaborator Author

Tang8330 commented Aug 1, 2024

Actually, this is fine. We are using an unbuffered channel.

Unbuffered channels provide synchronous communication, meaning the sender will block until a receiver is ready to receive the value, and vice versa.

@Tang8330 Tang8330 merged commit 0142796 into master Aug 1, 2024
6 checks passed
@Tang8330 Tang8330 deleted the avoid-ddb-oom branch August 1, 2024 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants