Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reduce memory pressure while uploading files #13718

Merged
merged 5 commits into from
Apr 16, 2024
Merged

fix: reduce memory pressure while uploading files #13718

merged 5 commits into from
Apr 16, 2024

Conversation

sobolk
Copy link
Member

@sobolk sobolk commented Apr 16, 2024

Description of changes

This PR is a partial revert of https://github.com/aws-amplify/amplify-cli/pull/13493/files which caused #13681 .

I.e. using s3.upload API for all uploads puts a memory pressure due to aws/aws-sdk-js#2552 .

This PR changes file uploading logic to buffer all files below 5mb in memory. This partially restoring previous behavior but avoiding full revert to s3.putObject API call which doesn't have retries.

Issue #, if available

#13681

Description of how you validated changes

PR checks and E2E tests.

Manual verification on t2.nano with sample test project where ENOMEM has been reproduced.

Checklist

  • PR description included
  • yarn test passes
  • Tests are changed or added
  • Relevant documentation is changed or added (and PR referenced)
  • New AWS SDK calls or CloudFormation actions have been added to relevant test and service IAM policies
  • Pull request labels are added

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sobolk sobolk changed the title Fix upload leak fix: reduce memory pressure while uploading files Apr 16, 2024
@sobolk sobolk marked this pull request as ready for review April 16, 2024 22:10
@sobolk sobolk requested a review from a team as a code owner April 16, 2024 22:10
Copy link
Contributor

@edwardfoyle edwardfoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM. Did you also verify that uploading on a slow network still works as expected? See verification description in the previous PR: #13493 (comment)

@sobolk
Copy link
Member Author

sobolk commented Apr 16, 2024

Change LGTM. Did you also verify that uploading on a slow network still works as expected? See verification description in the previous PR: #13493 (comment)

I played a bit with 3g profile, seems to be working.

@sobolk sobolk merged commit 2691a0a into dev Apr 16, 2024
5 checks passed
@sobolk sobolk deleted the fix-upload-leak branch April 16, 2024 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants