Tutorial 5 (controller-manager)

kube-controller-manager is responsible for synchronizing state between api server and the real world. The API server is responsible for recording the desired state of the cluster, then the controller manager is responsible for realizing it.

As the name suggests, the controller manager is not just one application. Rather, it's a single binary with a number of different components built into it.

The kube-controller-manager documentation lists the following controller managers:

attachdetach
bootstrapsigner
clusterrole-aggregation
cronjob
csrapproving
csrcleaner
csrsigning
daemonset
deployment
disruption
endpoint
garbagecollector
horizontalpodautoscaling
job
namespace
nodeipam
nodelifecycle
persistentvolume-binder
persistentvolume-expander
podgc
pv-protection
pvc-protection
replicaset
replicationcontroller
resourcequota
root-ca-cert-publisher
route
service
serviceaccount
serviceaccount-token
statefulset
tokencleaner
ttl
ttl-after-finished

We can't go through all of these, so we'll just concentrate on a few.

Credentials

kube-controller-manager accesses the API server. Like all other API clients, we'll generate a kubeconfig so the controller-manager can access the API server.

$ sudo kubeadm init phase kubeconfig controller-manager
[kubeconfig] Writing "controller-manager.conf" kubeconfig file

This will be written to /etc/kubernetes/controller-manager.conf

The `kube-controller-manager` pod

Just like before, we'll start out with a pod skeleton. The kubeadm command to generate a fully-generated manifest is sudo kubeadm init phase control-plane controller-manager, but we'll be creating this file from scratch.

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - image: k8s.gcr.io/kube-controller-manager:v1.13.4
    name: kube-controller-manager

Networking

In this case, hostNetwork is less about serving and more about access. If we didn't give the controller-manager hostNetwork, it would be segregated away from our api server. Normally Kubernetes would use a Software-defined networking to allow pods to access each other. But in this case, because we are using static pods that are required for Kubernetes to function, we rely on host networking.

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - image: k8s.gcr.io/kube-controller-manager:v1.13.4
    name: kube-controller-manager
  hostNetwork: true

Volumes

Certificate directory

We'll need the kubernetes pki directory. For one, we'll need to specify the CA to use to validate our connection to the kubernetes API. But we also need access to the Certificate Authority's private key, because some of the controllers are responsible for issuing certificates requesting through the API.

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - image: k8s.gcr.io/kube-controller-manager:v1.13.4
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs

Controller credentials

The pod will need access to the credentials we issued for it. Rather than mounting the entire /etc/kubernetes directory, which contains a number of sensitive files, we'll use the FileOrCreate type to just mount the supplied file.

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - image: k8s.gcr.io/kube-controller-manager:v1.13.4
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig

controller-manager arguments

It wouldn't be a Kubernetes component without a long list of command line arguments. There's an ongoing effort to move this type of configuration into configuration files, like we used to configure the kubelet. But for now, we'll make another argument list.

    command:
    - kube-controller-manager

Controller Credentials

We have both authentication and authorization arguments. Both of these are optional, with functionality degrading with each not supplied.

Since our kubeconfig is administrator-level, we can use it for both, along with the vanilla --kubeconfig:

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf

CA files

Since we may need to validate client certificates that have been passed through to us, we need to know which CA to use for that:

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt

Issuing Cluster-wide certificates

The csrsigning controller needs credentials to sign with. We'll use the same credentials that we use to validate the API server with:

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key

Service Accounts

The controller manager is in charge of issuing service accounts to all services that request them. Because these are signed by the service account secret key, we'll need to provide that:

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key

By default, the controller manager will use the credentials we've issued to it. However, these credentials are intentionally very limited, because it is better to have controller manager use its own service accounts. This makes it obvious which individual components are carrying out each action, rather than just one credential for the entire controller manager.

This needs to be explicitly enabled:

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true

Controllers

The --controllers flag specifies which of the many controllers to actually enable. Kubeadm enables all of them by default, and that's what you'd likely do in production. Instead, we will enable the bare minimum needed to get the cluster running.

bootstrapsigner is a controller that signs a ConfigMap with a set of tokens. This is part of the the process of adding new nodes.
csrapproving approves certificates requested via the Kubernetes api.
csrsigning actually signs certificates approved by csrapproving.
daemonset is responsible for synchronizing DaemonSet objects stored in the system with actual running pods.
deployment is is responsible for synchronizing Deployment objects stored in the system with actual running replica sets and pods.
disruption controls how much disruption can be applied to a pod.
endpoint joins services to pods.
job handles scheduling and cleaning up jobs.
namespace performs actions during namespace phases.
nodeipam handles IP Address Management for Kubernetes.
nodelifecycle handles node lifecycle events such as applying taints
podgc cleans up unneeded pods.
replicaset is responsible for synchronizing ReplicaSet objects stored in the system with actual running pods.
tokencleaner deletes expired tokens.
serviceaccount ensures a default ServiceAccount exists for every namespace
serviceaccount-token issues secrets for ServiceAccount objects

    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --controllers=bootstrapsigner,csrapproving,csrsigning,daemonset,deployment,disruption,endpoint,job,namespace,nodeipam,nodelifecycle,podgc,replicaset,tokencleaner,serviceaccount,serviceaccount-token

Installing the Pod

Let's assemble our finished yaml:

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - image: k8s.gcr.io/kube-controller-manager:v1.13.4
    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --controllers=bootstrapsigner,csrapproving,csrsigning,daemonset,deployment,disruption,endpoint,job,namespace,nodeipam,nodelifecycle,podgc,replicaset,tokencleaner,serviceaccount,serviceaccount-token
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig

Write this file out to /etc/kubernetes/manifests/kube-controller-manager.yaml

Check status

$ kubectl get pods --namespace kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
etcd-k8s-tutorial                      1/1     Running   0          4h49m
kube-apiserver-k8s-tutorial            1/1     Running   0          4h49m
kube-controller-manager-k8s-tutorial   1/1     Running   0          2m33s

If your kube-controller-manager container isn't dying immediately, congratulations, you've configured it correctly! Next, we'll get the scheduler working, at which point we'll have a mostly-functional Kubernetes cluster.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tutorial5.md

tutorial5.md

Tutorial 5 (controller-manager)

Credentials

The `kube-controller-manager` pod

Networking

Volumes

Certificate directory

Controller credentials

controller-manager arguments

Controller Credentials

CA files

Issuing Cluster-wide certificates

Service Accounts

Controllers

Installing the Pod

Check status

Files

tutorial5.md

Latest commit

History

tutorial5.md

File metadata and controls

Tutorial 5 (controller-manager)

Credentials

The kube-controller-manager pod

Networking

Volumes

Certificate directory

Controller credentials

controller-manager arguments

Controller Credentials

CA files

Issuing Cluster-wide certificates

Service Accounts

Controllers

Installing the Pod

Check status

The `kube-controller-manager` pod