kubeadm: support cert rotation or configurable cert expiration

[chendave]

What would you like to be added?

1. Add the feature of certs rotation which should be similar with kubelet client cert rotation, the certs will be automatically rotated when the expiration is approaching.
2. The valid period of certs signed by local CA is set to 1 year, we'd better to make it configurable instead of hard-coded as 1 year even that kubeadm is not used for production.
   https://github.com/kubernetes/kubernetes/blob/2047936f3f8ddd96498dcc812631b5368d14f3be/cmd/kubeadm/app/util/pkiutil/pki_helpers.go#L654-L657

Why is this needed?

kubeadm will refresh certs when the control plane are updated, and provide the command to renew certs. But haven't provided a feature to rotate the cert when the cert expiration is approaching. And actually, this feature is already implemented in kubelet.

[k8s-ci-robot]

@chendave: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chendave]

Technically, there should be no obstacle to implement this, but something worth to do it or not.

@neolit123 I am trying to find something I could contribute to kubeadm in the past few weeks, and found this is something might be considered as a feature. I'd like to hear from you, and I'd happy to draft a KEP if you are positive on this, thanks!

/sig cluster-lifecycle

[neolit123]

/transfer kubeadm

[neolit123]

hello, i moved this ticket to k/kubeadm where we are supposed to track kubeadm tickets..

Add the feature of certs rotation which should be similar with kubelet client cert rotation, the certs will be automatically rotated when the expiration is approaching.

this is not possible because kubeadm does not run as a service, kubeadm operates as a one-shot CLI execution tool.
there was an idea to create a kubeadm operator, but this proposal is stale and nobody is working on it:
kubernetes/enhancements#2505

note that the scope of the operator is more complex than just cert rotation.

do you want to work on the operator and revisit the old KEP or write a new one?
there was experimental operator code in this repository but it got removed at some point.

kubeadm will refresh certs when the control plane are updated, and provide the command to renew certs. But haven't provided a feature to rotate the cert when the cert expiration is approaching. And actually, this feature is already implemented in kubelet.

as mentioned above, kubelet is a service that runs constantly on the nodes, unlike kubeadm.

The valid period of certs signed by local CA is set to 1 year, we'd better to make it configurable instead of hard-coded as 1 year even that kubeadm is not used for production.
https://github.com/kubernetes/kubernetes/blob/2047936f3f8ddd96498dcc812631b5368d14f3be/cmd/kubeadm/app/util/pkiutil/pki_helpers.go#L654-L657

this has been discussed multiple times in the past, and was mostly rejected.
the problem with making it configurable for more that 1 year is that users will start signing certs for long periods of time and that makes the kubeadm cluster insecure. key sizes matter and 1 year is the industry recommended value.

for production users have multiple options:

• create a small "tool" that goes on kubeadm machines and rotates the certs, people are already doing that.
  it can be a controller or a bash script with SSH.
• instead of having kubeadm sign certificates for you use the kubeadm certs generate-csr option to generate CSRs and sign certificates with a long expiration time. then when you run kubeadm it will see that you have pre-populated certs.

[neolit123]

@chendave

also, forgot to mention, we are making a very good point here that we already support automatic cert renewal as part of upgrade:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal

kubeadm upgrade rotates certs and:

Note: It is a best practice to upgrade your cluster frequently in order to stay secure.

>1 year is not a secure practice and k8s releases 3 times a year. this means if you are using kubeadm upgrade it will rotate the certs for you 3 times!

[chendave]

Thanks @neolit123 for your feedback!

this is not possible because kubeadm does not run as a service

yes, I forgot that kubelet need to interlock with apiserver to achieve that and kubeadm is not a service.

do you want to work on the operator and revisit the old KEP or write a new one?

sure, I will take a look and get back to you later.

[neolit123]

ok, operator work is tracked here:
#2317

[chendave]

there was experimental operator code in this repository but it got removed at some point.

@neolit123 I did some research in the past few days, do you have any ideas why it's removed? it would be helpful if we have the experimental code, we can start the work on top of that.

[neolit123]

@chendave IIRC, it was experimental and we did not have the time to focus more work on it at the time.
it was removed because we did not want users to start using it, as it was.

if you'd like you can start working on a POC in a repository of yours.
and maybe share a design doc as well.