Refactor worker selection to facilitate adding multiple worker runtimes

[aoldershaw]

What does this PR accomplish?

Bug Fix | Feature | Documentation | Refactor

Make it easier to support multiple runtimes in the future by removing the responsibility of selecing a worker from the worker.Client. This also makes it possible to schedule multiple containers on the same worker, which would be useful for Services.

Changes proposed by this PR:

As was shown in #5223, the most natural way to support new runtimes is by constructing a new implementation of worker.Client (defining how to RunTaskStep, RunGetStep, etc.), rather than trying to find a more fine-grained common interface between worker runtimes up-front.

It's worth noting that there exists a worker.Worker interface already which seems to address some concerns of multiple runtimes - however, to me, worker.Worker doesn't really have a clear identity as an interface, as many of the interface's methods are just delegation to other things (DB, VolumeClient), and aren't runtime specific (though VolumeClient possibly will be). There are a few methods (FindContainerByHandle, FindOrCreateContainer, Fetch?) that are runtime specific and should probably be moved to worker.Client eventually, but that's out of scope for this PR.

The prior responsibilities of worker.Client included selecting a worker.Worker from the worker.Pool, and then running some (Garden specific) logic, including calling some methods on the worker.Worker to create containers. e.g. the flow for a get step is:

exec.GetStep.Run(...)
    -> worker.Client.RunGetStep()
        -> worker.Pool.FindOrChooseWorkerForContainer()
            <- worker.Worker
        -> worker.Worker.Fetch(...)
            -> Garden specific stuff

Now, worker.Client just represents a single worker, and worker.Pool gives worker.Clients. Each step first selects a worker.Client via the pool, and then runs the appropriate step. The new flow is:

exec.GetStep.Run(...)
    -> worker.Pool.SelectWorker()
        <- worker.Client
    -> worker.Client.RunGetStep()
        -> worker.Worker.Fetch(...)
            -> Garden specific stuff

This can also enable running services for task steps, e.g.

exec.TaskStep.Run(...)
    -> worker.Pool.SelectWorker()
        <- worker.Client
    -> worker.Client.RunService()
    -> worker.Client.RunTaskStep() // same worker

---

Additionally, this adds an ArtifactSourcer interface for converting runtime.Artifacts to worker.ArtifactSources. It doesn't have a super strong identity IMO, but it means we don't need to hold duplicated info on the worker.ContainerSpec and hackily modify the spec when running steps.

Notes to reviewer:

Contributor Checklist

• Followed Code of conduct, Contributing Guide & avoided Anti-patterns
• Signed all commits
• Added tests (Unit and/or Integration)
• Updated Documentation
• Added release note (Optional)

Reviewer Checklist

• Code reviewed
• Tests reviewed
• Documentation reviewed
• Release notes reviewed
• PR acceptance performed
• New config flags added? Ensure that they are added to the
  BOSH and
  Helm packaging; otherwise, ignored for
  the integration
  tests
  (for example, if they are Garden configs that are not displayed in the
  --help text).

[taylorsilva]

I'm so excited about this! Thanks for all the extra work you did :)
There's a lot to go through so I hope I didn't miss anything. The Artifact Streamer and Sourcer seem like good extractions for the volume fetching issue. I did notice they didn't have any tests though. Think you could add some for them?

[aoldershaw]

@taylorsilva sorry, took me a while to find the time to address your comments! I've added some tests for ArtifactStreamer and ArtifactSourcer. You may notice they look quite a bit different from the existing tests in atc/worker. tbh, I really don't like how atc/worker (and its existing tests) are currently structured - there's the appearance of loose coupling because everything is behind an interface, but in reality, everything is tightly coupled to the current set of abstractions, so making changes becomes a pain - especially because the tests are so mock-heavy.

I kinda want to replace the test suites with slightly higher level tests and remove the convoluted web of dependencies within the package (since many of these interfaces aren't really interfaces - we just use interfaces so we can make testing slightly easier). Really, the garden worker implementation (i.e. most of atc/worker) has only 3 dependencies - Garden, Baggageclaim, and the DB - if we can use in-memory fakes of these (not just dumb counterfeiter stubs), I think the code would be a lot simpler and the tests would be more telling of whether the code actually works.

....but anyway, that's for another PR, this one's already big enough!

[taylorsilva]

one nit about the one test's structure, but I'm gonna approve this anyways. Up to you if you wanna change it or not. Thanks for adding the test coverage :)