Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload Refactor #606

Merged
merged 13 commits into from
Sep 21, 2023
Merged

Upload Refactor #606

merged 13 commits into from
Sep 21, 2023

Conversation

peterjan
Copy link
Member

@peterjan peterjan commented Sep 19, 2023

I was seeing how to implement etags and got side-tracked on a mini-refactor. It's a bit hard to see in the diff but it does deduplicate a bunch of code we had. I went back and forth on it a bit so definitely don't mind to discuss it further.

I extended the worker with two methods:
func (w *worker) upload
func (w *worker) uploadMultiPart

that each wrap an upload and then handle the object and (potential) partial slab data with the bus. I moved this in and out of the upload manager but eventually figured that it should not be handled by the upload manager, at the cost of duplicating persisting the partial slabs and setting it on the object.

Most importantly we now spin a goroutine to persist packed slabs, so the request that pushes it over the limit isn't blocked by the upload. I think this is better and it gets rid of duplicate logic we had in regular upload vs multi-part uploads. (This is the thing you did initially and then ran into an error, so I hope the error you ran into is in a test we still have so it flags if I made the same mistake).

It also extends the uploadConfig with everything we can gather from the upload parameters and greatly simplifies the interface of the upload manager. Now we return the etag and a potential error but eventually I'll return the object.Object that has the etag.

@peterjan peterjan changed the title Refactor Upload Upload Refactor Sep 19, 2023
@peterjan peterjan marked this pull request as ready for review September 20, 2023 08:08
worker/upload.go Outdated Show resolved Hide resolved
worker/upload.go Outdated Show resolved Hide resolved
worker/upload.go Outdated Show resolved Hide resolved
internal/testing/cluster_test.go Show resolved Hide resolved
worker/upload.go Outdated Show resolved Hide resolved
worker/upload.go Outdated Show resolved Hide resolved
worker/upload.go Outdated Show resolved Hide resolved
@peterjan peterjan self-assigned this Sep 21, 2023
@ChrisSchinnerl ChrisSchinnerl merged commit 823086c into master Sep 21, 2023
@ChrisSchinnerl ChrisSchinnerl deleted the pj/packed-slabs branch September 21, 2023 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants