[kube-prometheus-stack] Conflicting secret name

[applejag]

Describe the bug a clear and concise description of what the bug is.

Tried to do an update via helm upgrade, but got the error:

rendered manifests contain a resource that already exists. Unable to continue with update: Secret "prometheus-prometheus" in namespace "prometheus" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ri-prometheus-operator"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "prometheus"

It seems to be caused by this file: https://github.com/prometheus-community/helm-charts/blob/kube-prometheus-stack-54.1.0/charts/kube-prometheus-stack/templates/prometheus/secret.yaml

Having conflicts with a file that the Prometheus Operator generates here: https://github.com/prometheus-operator/prometheus-operator/blob/v0.69.1/pkg/prometheus/statefulset.go#L122-L148

What's your helm version?

version.BuildInfo{Version:"v3.13.2", GitCommit:"v3.13.2", GitTreeState:"", GoVersion:"go1.21.3"}

What's your kubectl version?

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.8

Which chart?

kube-prometheus-stack

What's the chart version?

54.1.0

What happened?

The secret from this chart is for Thanos stuff, and I don't even use Thanos. The secret that's blocking my upgrades is an empty secret.

What you expected to happen?

The secret deployed by this chart does not have a conflicting name.

How to reproduce it?

No response

Enter the changed values of values.yaml?

fullnameOverride: prometheus
cleanPrometheusOperatorObjectNames: true

Enter the command that you execute and failing/misfunctioning.

helm upgrade ri-prometheus-operator ./ -f path-to-my-values/values.yaml

Anything else we need to know?

No response

[aldemira]

I've the exact same problem. Not a permanent solution but I edited the secret manually with:
kubectl edit secret prometheus-prometheus -n prometheus
And added following lines (for some reason this object didn't have any annotations, which is odd.)

annotations:
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: prometheus

I've also changed:

labels:
managed-by: prometheus-operator

to:

labels:
app.kubernetes.io/managed-by: "Helm"

And upgrade worked without any issues after that.

[timebertt]

Seems to be caused by #3918 (version 52).
cc @thameezb

The secret is always created

helm-charts/charts/kube-prometheus-stack/templates/prometheus/secret.yaml

Line 1 in f61d899

{{- if .Values.prometheus.enabled }}

even if .Values.thanosRuler.thanosRulerSpec.alertmanagersConfig.secret is not set, which is the condition in the ThanosRuler object

helm-charts/charts/kube-prometheus-stack/templates/thanos-ruler/ruler.yaml

Lines 78 to 82 in f61d899

	{{- else if .Values.thanosRuler.thanosRulerSpec.alertmanagersConfig.secret }}
	alertmanagersConfig:
	key: alertmanager-configs.yaml
	name: {{ template "kube-prometheus-stack.thanosRuler.name" . }}
	{{- end }}

Note, that an empty secret will be created if .Values.thanosRuler.thanosRulerSpec.alertmanagersConfig.secret is not set:

helm-charts/charts/kube-prometheus-stack/templates/prometheus/secret.yaml

Lines 12 to 16 in f61d899

	{{- with .Values.prometheus.prometheusSpec.thanos.objectStorageConfig }}
	{{- if and .secret (not .existingSecret) }}
	object-storage-configs.yaml: {{ toYaml .secret | b64enc | quote }}
	{{- end }}
	{{- end }}

[mohamedezz96]

i am facing the same problem is there a solution instead editing the existing secret manual ?

[applejag]

We solved it by not setting cleanPrometheusOperatorObjectNames. The pod names are a bit ugly, but it works.