Wait for ongoing uploads and downloads when shutting down the worker

[peterjan]

This PR makes the worker shut down gracefully and allows ongoing uploads and downloads to finish.

changed PR

[peterjan]

I moved it to DRAFT since it's more of a proposal now.

I propose we register ongoing uploads and downloads and wait for them to finish when shutting down the worker, this means we'll only use the shutdown context to prevent starting new uploads/downloads when we were blocking on memory. I added a middleware to check if the shutdown context was cancelled, and if that is the case we return service unavailable... this way we make sure we only handle ongoing requests and not take on any new ones.

I also want to change the way we handle the shutdown timeout, we now divide it by the number of services we have but that seems a little arbitrary. I see how it allows every service to shut down gracefully but at the same time it might needlessly prevent a service from shutting down gracefully simply because we restrict it to shutdownTimeout/10.

Maybe we can allocate (an arbitrary) 5 seconds to every service and reserve that. So we subtract that time from the shutdown timeout and then we just allow all services to use the remaining time, if we do timeout we still have that reserve to ensure we can gracefully shut down the rest. I feel this would give time to those services that need it, while ensuring we're not giving the opportunity to following services to complete their shutdown.

[ChrisSchinnerl]

Have you manually tested this a few times to see if it throws any unexpected errors?

[peterjan]

Have you manually tested this a few times to see if it throws any unexpected errors?

I manually tested it a bunch of times. Downloads and uploads are handled gracefully, and we do reserve a portion of the shutdown ctx to try and shut down the remaining services in case we time out. If the worker shut down times out though, we might not record some interactions and then we get:

2024-02-14T10:20:50+01:00       ERROR   worker.worker   worker/interactions.go:91       failed to record 1 price table updates on worker shutdown

[peterjan]

Converting it to DRAFT to prevent merge, ran into a 2024-02-14T10:40:07+01:00 ERROR uploadmanager worker/contract_lock.go:109 failed to send keepalive testing a shorter shutdown timeout.

[peterjan]

This was never scheduled and is not currently a priority.