Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

[slieer]

Describe the bug

Rocky Linux release 8.10 (Green Obsidian)
minikube version: v1.34.0
kubectl version: Client Version: v1.31.2 Kustomize Version: v5.4.2 Server Version: v1.31.0
chectl version:
chectl/7.94.0 linux-x64 node-v18.18.0 or chectl/7.93.0 linux-x64 node-v18.18.0

chectl server:deploy --platform minikube
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...[1.31]
✔ Minikube preflight checklist
✔ Verify if kubectl is installed...[OK]
✔ Verify if minikube is installed...[OK]
✔ Verify if minikube is running...[OK]
✔ Enable minikube ingress addon...[Enabled]
✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io]
✔ Checking minikube version...[1.34.0]
✔ Create Namespace eclipse-che...[Exists]
✔ Install Cert Manager v1.8.2
✔ Apply resources...[Exists]
✔ Wait for Cert Manager pods ready...[OK]
✔ Install Dex
✔ Create Namespace dex...[Exists]
✔ Create Certificates...[Exists: /tmp/dex-ca.crt]
✔ Create ConfigMap dex-ca...[Updated]
✔ Create ServiceAccount dex...[Exists]
✔ Create ClusterRole dex...[Exists]
✔ Create ClusterRoleBinding dex...[Exists]
✔ Create Service dex...[Exists]
✔ Create Ingress dex...[Exists]
✔ Generate Dex username and password...[Exists]
✔ Create ConfigMap dex...[Exists]
✔ Create Deployment dex...[Exists]
✔ Configure API server
✔ Create /etc/ca-certificates directory...[Created]
✔ Copy Dex certificate into Minikube...[OK]
✔ Configure Minikube API server...[OK]
✔ Wait for Minikube API server...[OK]
✔ Start following Eclipse Che installation logs...[OK]
❯ Deploy Eclipse Che operator
❯ Install Dev Workspace operator
✔ Create Namespace devworkspace-controller...[Exists]
✖ Create Dev Workspace operator resources
→ issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged
Wait for Dev Workspace operator ready
Create ServiceAccount che-operator
Create RBAC
Wait for Cert Manager pods ready
Create Certificate che-operator-serving-cert
Create Issuer che-operator-selfsigned-issuer
Create Service che-operator-service
Create CRD checlusters.org.eclipse.che
Waiting
Create Deployment che-operator
Eclipse Che Operator pod bootstrap
Create ValidatingWebhookConfiguration org.eclipse.che
Create MutatingWebhookConfiguration org.eclipse.che
Create CheCluster Custom Resource
Error: Command server:deploy failed with the error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspaces.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspacetemplates.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured
customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured
serviceaccount/devworkspace-controller-serviceaccount unchanged
role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged
rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged
service/devworkspace-controller-manager-service unchanged
service/devworkspace-controller-metrics unchanged
deployment.apps/devworkspace-controller-manager configured
certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged
issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged See details: /home/skyworth/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1730961011828.
at newError (/usr/local/lib/chectl/lib/utils/utls.js:39:19)
at wrapCommandError (/usr/local/lib/chectl/lib/utils/command-utils.js:54:32)
at Deploy. (/usr/local/lib/chectl/lib/commands/server/deploy.js:82:65)
at Generator.throw ()
at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:167:69)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Cause: Error: Command failed with exit code 1: kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspaces.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspacetemplates.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured
customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured
serviceaccount/devworkspace-controller-serviceaccount unchanged
role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged
rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged
service/devworkspace-controller-manager-service unchanged
service/devworkspace-controller-metrics unchanged
deployment.apps/devworkspace-controller-manager configured
certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged
issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged
at makeError (/usr/local/lib/chectl/node_modules/execa/lib/error.js:60:11)
at handlePromise (/usr/local/lib/chectl/node_modules/execa/index.js:118:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

minikube kubectl -- get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-54f9d599b-mn52s 1/1 Running 4 (4m28s ago) 44h
cert-manager cert-manager-cainjector-648f59958c-ws8nk 1/1 Running 6 (4m29s ago) 44h
cert-manager cert-manager-webhook-7b845b56cb-k9gdj 1/1 Running 5 (4m29s ago) 44h
devworkspace-controller devworkspace-controller-manager-f54dbb6f6-vs55l 2/2 Running 0 3m12s
devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-2t9nt 2/2 Running 0 118s
devworkspace-controller devworkspace-webhook-server-7c4b65bdb9-kt9lf 2/2 Running 0 2m18s
dex dex-7687bb6d68-k68gc 1/1 Running 7 (3m57s ago) 44h
ingress-nginx ingress-nginx-admission-create-btgxv 0/1 Completed 0 45h
ingress-nginx ingress-nginx-admission-patch-46wbc 0/1 Completed 0 45h
ingress-nginx ingress-nginx-controller-857f8876df-dn89f 1/1 Running 4 (4m19s ago) 45h
kube-system coredns-d4ddbc888-72f9c 1/1 Running 5 (4m24s ago) 46h
kube-system coredns-d4ddbc888-xn45q 1/1 Running 4 (4m24s ago) 46h
kube-system etcd-minikube 1/1 Running 6 (4m28s ago) 46h
kube-system kube-apiserver-minikube 1/1 Running 3 (4m18s ago) 44h
kube-system kube-controller-manager-minikube 1/1 Running 5 (4m29s ago) 46h
kube-system kube-proxy-xqvwd 1/1 Running 5 (4m29s ago) 46h
kube-system kube-scheduler-minikube 1/1 Running 5 (4m28s ago) 46h
kube-system metrics-server-686dff4775-j2dhq 1/1 Running 8 (3m57s ago) 45h
kube-system storage-provisioner 1/1 Running 6 (4m29s ago) 46h
kubernetes-dashboard dashboard-metrics-scraper-c5db448b4-jdmwx 1/1 Running 4 (4m29s ago) 45h
kubernetes-dashboard kubernetes-dashboard-695b96c756-qfrdq 1/1 Running 5 (4m28s ago) 45h

Che version

7.93/ 7.94

Steps to reproduce

chectl server:deploy --platform minikube

Expected behavior

che install success.

Runtime

minikube

Screenshots

No response

Installation method

chectl/latest

Environment

Rocky Linux release 8.10 (Green Obsidian)

Eclipse Che Logs

No response

Additional context

No response

[tolusha]

It seems it is impossible to deploy DWO/Che operator on the latest Kubernetes version

[tolusha]

@slieer
Could reinstall minikube and deploy che one more time?
Currently I can't reproduce the issue.
Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

[slieer]

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

Thank you for your response. I'll try again.

[slieer]

kubectl apply -f /usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml

customresourcedefinition.apiextensions.k8s.io/devworkspaceoperatorconfigs.controller.devfile.io configured
customresourcedefinition.apiextensions.k8s.io/devworkspaceroutings.controller.devfile.io configured
serviceaccount/devworkspace-controller-serviceaccount unchanged
role.rbac.authorization.k8s.io/devworkspace-controller-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-edit-workspaces unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-proxy-role unchanged
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-role configured
clusterrole.rbac.authorization.k8s.io/devworkspace-controller-view-workspaces unchanged
rolebinding.rbac.authorization.k8s.io/devworkspace-controller-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-proxy-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/devworkspace-controller-rolebinding unchanged
service/devworkspace-controller-manager-service unchanged
service/devworkspace-controller-metrics unchanged
deployment.apps/devworkspace-controller-manager configured
certificate.cert-manager.io/devworkspace-controller-serving-cert unchanged
issuer.cert-manager.io/devworkspace-controller-selfsigned-issuer unchanged
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspaces.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspaces.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block
Error from server (Invalid): error when applying patch:
{"spec":{"conversion":{"webhook":{"clientConfig":{"caBundle":"Cg=="}}}},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "devworkspacetemplates.workspace.devfile.io", Namespace: ""
for: "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": error when patching "/usr/local/lib/chectl/templates/devworkspace-operator/kubernetes/combined.yaml": CustomResourceDefinition.apiextensions.k8s.io "devworkspacetemplates.workspace.devfile.io" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block

It's still like this. It looks like it's a cert-manager related issue. The current version of cert-manager in CHE is too low. Are there any plans to upgrade to the latest version from cert-manager.io?

[tolusha]

@slieer
There is no problem to udpate certmanager to a newer version.
Could you try the following on the clean minikube:

oc apply -f  https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/tags/v0.31.2/deploy/deployment/kubernetes/combined.yaml

[slieer]

@slieer Could reinstall minikube and deploy che one more time? Currently I can't reproduce the issue. Sometimes I have storage is (re)initializing problem. Maybe the latest minikube is not stable.

Thank you for your response. I'll try again.

OK， Thanks. Very thank you for your attention and response.

To address this issue, I think the first step should be to handle the self-signed certificate. However, the optional steps described in the documentation may not be accurate. I will try this method.

https://eclipse.dev/che/docs/stable/administration-guide/configuring-che-with-self-signed-certificate/

[tolusha]

@slieer
Please let me know if you need any help