fix(paramserver): file is too large error #98
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amitsingh21
approved these changes
Nov 8, 2024
pallabpain
force-pushed
the
fix/file-too-large-error
branch
2 times, most recently
from
e229fc4 to
9a2621f
Compare
ankitrgadiya
approved these changes
Nov 12, 2024
pallabpain
force-pushed
the
fix/file-too-large-error
branch
2 times, most recently
from
970382f to
99e6826
Compare
This commit fixes a specific case when a json/yaml file is uploaded to the paramserver and is stored by the backend in etcd. The issue existed till date because the SDK performs a stat to check the size of a file and determine whether to upload a file as blob or not. However, in cases where the file size is close to the upper limit i.e. 128 KiB, the API payload may inflate due to additional metadata and string conversion. In such cases, while the file size may be under 128KiB, the final data to be stored in etcd is greater than 128KiB and it throws a `file is too large` error. Fixes AB#33921
pallabpain
force-pushed
the
fix/file-too-large-error
branch
from
99e6826 to
8055507
Compare
ankitrgadiya
approved these changes
Nov 12, 2024
rr-github-ci-user
pushed a commit
that referenced
this pull request
Nov 12, 2024
## [2.1.1](v2.1.0...v2.1.1) (2024-11-12) ### Bug Fixes * **paramserver:** file is too large error ([#98](#98)) ([3f379ee](3f379ee)), closes [AB#33921](https://github.com/AB/issues/33921)
|
🎉 This PR is included in version 2.1.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When uploading configurations, if the size of a file is greater than 128 KiB, it is uploaded to a blob store. Otherwise, it is stored in etcd. The two operations have separate APIs that the SDK invokes based on the file size.
The max key value size for etcd is set to 128 KiB, which means that any value greater than that cannot be stored.
When an API call is made to save a file < 128 KiB, the backend stored the entire API payload instead of just the file content. The payload is of the form
{ "type": "FileNode", "content-type": "application/json", "data": "STRINGIFIED_DATA_READ_FROM_THE_FILE" }Further, the backend also injects a
metadatafield with timestamp and userid. So, let's say a file that is around115 KiBon disk becomes > 128 KiB by the time it is saved to etcd. Which leads to thefile too largeerrors.This PR handles the case by calculating the effective size and adds the metadata size in the calculation before invoking the right API.
Testing
Screen.sharing.-.2024-11-07.7_16_17.PM.mp4
Links