Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Don't block when resolving chart versions #327

Open
parhamdoustdar opened this issue Jun 25, 2020 · 0 comments
Open

Don't block when resolving chart versions #327

parhamdoustdar opened this issue Jun 25, 2020 · 0 comments

Comments

@parhamdoustdar
Copy link
Contributor

parhamdoustdar commented Jun 25, 2020

Right now, when repo.FetchChartVersions is called, it waits for 2 seconds as a timeout.

This caused an incident when we tried to take down one of the chart repositories that Shipper depends on. As a large number of old releases were still around and were referring to this chart repository, the queues got clogged and we had to bring the chart repository back online.

Instead of having a delay, we should:

  • Change the repo to wait for a longer time period (e.g. 10 seconds) for the first time that a new repo has been encountered
  • Don't wait for resolution if it hasn't happened already while processing a job, and update the condition that we're waiting for the chart repository index to be fetched. That way we report to the user, and also move on to process releases that might be completely unaffected by this chart repository going down. In our incident, that would mean that new releases that are not using the older chart repository would still be handled just fine.

This would pretty much mean to remove the select statement from repo.FetchChartVersions, and move it to repo.Start, and add an argument like timeout to it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant