Use Retry-After HTTP header for scheduling in progress

[Flix6x]

The FlexMeasures client uses polling to fetch the results of long-running jobs, such as creating a new schedule (see FlexMeasures/flexmeasures-client#2). The client would benefit from receiving an estimate on when to try fetching again. One way to determine this estimate is to base it on job.ended_at - job.enqueued_at of n relevant jobs:

• The last ones to have finished.
• Jobs in the same queue.
• Jobs for the same sensor.

We might consider storing this estimate of waiting time on the job description, which may be updated without influencing the job's hash.

Related to this, we should change the 400 status code to a 503 status code for the following cases:

• job.is_started
• job.is_queued
• job.is_deferred

Out of scope: storing the current expected waiting time is a property of the queue and would be an interesting property to keep track of, for monitoring. We can first monitor queue length and arrival rate, and use Little's law to compute the average waiting time.

[victorgarcia98]

Cool feature!

One possibility is to have a background job running periodically (say 5min) that recomputes the expected waiting times using the data from FinishedJobRegistry.

In order to avoid very fast polling or to have clients waiting too much for a job to finish, I would limit the waiting time estimates under a reasonable interval (e.g [10s, TTL]). Also the rate limiting feature should help us here.

Moreover, I would warn somewhere that the accuracy of the estimates will depend on the actual sample size.