-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upload Refactor #606
Merged
Merged
Upload Refactor #606
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was seeing how to implement etags and got side-tracked on a mini-refactor. It's a bit hard to see in the diff but it does deduplicate a bunch of code we had. I went back and forth on it a bit so definitely don't mind to discuss it further.
I extended the worker with two methods:
func (w *worker) upload
func (w *worker) uploadMultiPart
that each wrap an upload and then handle the object and (potential) partial slab data with the bus. I moved this in and out of the upload manager but eventually figured that it should not be handled by the upload manager, at the cost of duplicating persisting the partial slabs and setting it on the object.
Most importantly we now spin a goroutine to persist packed slabs, so the request that pushes it over the limit isn't blocked by the upload. I think this is better and it gets rid of duplicate logic we had in regular upload vs multi-part uploads. (This is the thing you did initially and then ran into an error, so I hope the error you ran into is in a test we still have so it flags if I made the same mistake).
It also extends the
uploadConfig
with everything we can gather from the upload parameters and greatly simplifies the interface of the upload manager. Now we return the etag and a potential error but eventually I'll return theobject.Object
that has the etag.