A Prefect Kubernetes operator

[chrisguidry]

I've been prototyping an operator for Kubernetes that can provision Prefect servers and workers with custom resources in the cluster. I'm looking for feedback from the community, especially from folks on platform teams who are provisioning Prefect for their organization.

The goal here is to make provisioning Prefect servers in a cluster as comfortable as any other Kubernetes resource, while covering many of the small details, like health probes and when database migrations are applied.

We're still quite early in this journey and would appreciate any feedback and feature requests you'd have for this operator. It's so early we haven't event published an image yet!

This idea started because I run Prefect on Kubernetes in a small environment where there is one central namespace running a Prefect server, then work pools with workers running in isolated namespaces with access to databases and other API secrets. I feel this is a common pattern, so I am trying to support both the use case of a single Prefect server per namespace and one where there is a Prefect server in the cluster and many worker using it.

In the initial design, we have two namespaced CRDs, PrefectServer and PrefectWorkPool:

apiVersion: prefect.io/v1
kind: PrefectServer
metadata:
  namespace: my-team
  name: my-prefect-server
spec:
  version: 3.0.0rc18
  sqlite:
    storageClassName: standard
    size: 1Gi
  settings:
  - name: PREFECT_ENABLE_COOL_FEATURE
    value: "True"

---

apiVersion: prefect.io/v1
kind: PrefectWorkPool
metadata:
  namespace: my-team
  name: my-k8s-pool
spec:
  version: 3.0.0rc17
  type: kubernetes
  server:
    namespace: my-team
    name: my-prefect-server
  workers: 2

The PrefectServer will own a Deployment and Service, as well as a PersistentVolumeClaim if necessary for SQLite-backed servers. The PrefectWorkPool will own a Deployment with the specified number of replicas.

The prototype is implemented in Go using the Operator SDK, so it can take advantage of the amazing and mature tooling around Kubernetes. I tried the Python kopf framework, which I found quite pleasant, but ultimately decided against due to the large warning at the top of their documentation.

[cicdw]

@chrisguidry could a Cloud user leverage the PrefectWorkPool CRD in your conception? Also - and this could be an ignorant question - is there any world in which the PrefectWorkPool CRD could own creating the jobs (and so avoiding the need for a worker deployment altogether), or is that getting out of scope?

[chrisguidry]

Yes! That was one something I had in mind, where in the server: section, it could look like this:

apiVersion: prefect.io/v1
kind: PrefectWorkPool
metadata:
  namespace: my-team
  name: my-k8s-pool
spec:
  version: 3.0.0rc17
  type: kubernetes
  server:
    apiUrl: https://api.prefect.cloud/accounts/.....
    apiKey:
      valueFrom:
        secretKeyRef:
          name: a-secret
          key: my-prefect-api-key
  workers: 2

This is not implemented just yet, but a sketch of how it could look.

[chrisguidry]

PrefectHQ/prefect-operator#42 to track this, I think this should definitely be in the plan.

[chrisguidry]

This was included in version 0.2.0 💪

[chrisguidry]

A status update: development on the operator has been coming along well, and we're getting close to the first release! Big thanks to @mitchnielsen and @parkedwards for all their contributions here. 🙇

[chrisguidry]

We've got our first (and second!) releases of prefect-operator out. Give them a whirl and let us know what you think: https://github.com/PrefectHQ/prefect-operator/releases/tag/0.2.0

We're sure there are still many details to iron out and we'd love feedback from anyone operating Prefect in Kubernetes. I'd say this version would be a great place to start for development and test workloads where ephemeral storage makes sense, for spinning up workers for a production PrefectWorkPool against an existing Prefect server or Prefect Cloud, and for running a PrefectServer against a PostgreSQL database you're managing.

We'll be doing a lot more testing internally over the coming weeks to make sure the PrefectServer handling of persistent storage is solid.