Form of branch-planner deployment

[squaremo]

As it's developed, the branch-planning capability #527 is simply an executable. It can be baked into a container images and will run fine, perhaps with minor adjustments, in a pod in a Kubernetes cluster. But there may be more convenient ways to deploy it.

The processes involved are:

• polling, which (at present) loads the config, fetches the PRs for each Terraform object mentioned in the config, and creates a "branch" Terraform object and source per PR.
• informer, which watches Terraform objects and will write plans back to PRs.

Here are some alternatives for deployment:

• status quo: run the executable in a pod (via a Deployment);
• run poller and informer as Runnable components with tf-controller
• adapt poller and informer to be controller-runtime controller(s), and run in its own Deployment;
• adapt poller and informer to be controller-runtime controller(s), and add to the tf-controller manager (behind a feature flag)

Status quo

This'll work.

Run as a process with tf-controller

The controller-runtime manager should be able to run anything with a Start(context.Context) method, so the branch-planning processes as written now could be added to the tf-controller manager.

The main benefit is that it is not another thing to run. The main downside is subjective: it seems like a bolt-on to tf-controller.
It may mean the branch-planning processes can share a client cache with tf-controller, which is a minor win for efficiency.

Run it as its own controller

Why run as a controller? One reason is that a reconciler being fed from a workqueue is pretty useful for doing stuff in Kubernetes. You can use the workqueue for retries as well as polling (ctrl.Result{RequeueAfter: pollingInterval}). Retries especially can be a pain to do yourself.

The poller could work well as a controller, because then it can react to the "original" Terraform objects changing, and use the queue to reschedule polls of the PR list, retry when it can't fetch the same. A complication is that the Terraform objects to consider come from a ConfigMap, which needs to be consulted regularly (I think this part of the design is provisional and may not last long, though).

The informer can work well as a controller, because it naturally watches Terraform objects and may need to retry.

Run as a controller in tf-controller

This has the properties of running as a controller, and those of running with tf-controller.

[yitsushi]

I don't think we wanted to keep the ConfigMap logic with a simple executable as it is now, and I don't think it's a long term solution, at least it should be, as it's not scalable at all and making is scalable we would have to implement a queue logic that's totally a waste of time and resources. We ended up with this because of simplicity, but I urge to move forward as soon as possible.

I would consider splitting the poller and informer parts into separate entities when we are talking about how to make it better.

Poller as a controller

The poller could be a simple controller with it's own CRD. With this approch we would have a lot more space to fine-tune when we really want per repo/tf object.

Examples:

---
apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: TerraformBranchBasedPlanner
metadata:
  name: all-infra
  namespace: infra
spec:
  interval: 10m
---
apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: TerraformBranchBasedPlanner
metadata:
  name: specific-resource
  namespace: team-a
spec:
  interval: 10m
  # "parent" terraform object. Intentionally left out the namespace, but we can use
  # a full namespacedObjectReference here.
  targetName: my-tf-stuff
  # name of the secret that contains the api token for the poller.
  secretName: token-secret
  branch:
    # It set, create resources only if head branch name matches this pattern.
    pattern: "^plan-"
    # Create resources only if the base branch is `main`. If not set, base can be anything.
    base: main
  # Create resources only (or create, but run plan only) if the Pull Request has at least
  # one approval
  #
  # other values:
  #  auto -> if it's open and matches all other criteria
  #  manual -> create one requested by a comment (`!create planner` for example)
  strategy: approved

It would still run the polling per TerraformBranchBasedPlanner which is not ideal if it's a wildcard resource (first example), but still much better than doing everything in one process.

Informer

The informer can be simply part of the main tf controller. If the reconciled Terraform object is a branch planner one (it has the labels), just post the plan when it's ready. The informer has a lot of extra stuff to check (is this a branch planner resource, where is the output, do we have output, i don't know what else), but most of them are already available at the time the main tf controller reconciles the object.

Integration

If branch planner is enabled we deploy a separate controller with the poller and set a "branch planner enabled" flag on the tf-conrtoller.

branchPlanner:
  enabled: true
  replicas: 3
  # How many tf objects should run at once if it reconciles a wildcard branch planner object.
  concurrency: 5
  image:
    repository: ghcr.io/weaveworks/branch-planner
    pullPolicy: IfNotPresent

Why separate controller? So it can be scaled independently from the main tf-controller. As we create a lot of extra Terraform objects, a user may want to keep the branch planner with replicas: 3 as it does not have much work to do, while the tf-conrtoller replicas: 5 as it has much more work to do. As an addition, the branch planner needs different cluster roles, for example creating TerraformObjects, but it does not need permissions like creating runner.

As a side-effect of the informer's job integrated into the main controller, a user can create Terraform objects for a specific pull request (without the branch planner watching the repo) by simply creating a new object with extra labels. I don't know if it's a good thing or not, but I like it :)