[kube-prometheus-stack] - Missing cluster label in metrics from kubelet, kube-state-metrics...

[bmgante]

Hi everyone, using kube-prometheus-stack and noticed that default dashboards use a label cluster for the metrics generated by kube-state-metrics, kubelet, etc. However, the metrics generated do not have this label.

Example of one metric used in one rule which is then used in several dashboard pannels:

container_cpu_usage_seconds_total{container="alertmanager", cpu="total", endpoint="https-metrics", id="/kubepods.slice/kubepods-pod87e2ed11_9a16_445c_9d57_f79f27f0d284.slice/cri-containerd-b3444f5ab945fd924295a552249424ec551f7f3a23020de8bbf3596e2495c3c0.scope", image="quay.io/prometheus/alertmanager:v0.25.0", instance="x.x.x.x:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="b3444f5ab945fd924295a552249424ec551f7f3a23020de8bbf3596e2495c3c0", namespace="monitoring", node="ip-10-30-131-229.ec2.internal", pod="alertmanager-kps-alertmanager-0", prometheus="monitoring/kps-prometheus", service="kps-kubelet"}

The rule which makes use of cluster label:

record:node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
expr:sum by (cluster, namespace, pod, container) (irate(container_cpu_usage_seconds_total{image!="",job="kubelet",metrics_path="/metrics/cadvisor"}[5m])) * on (cluster, namespace, pod) group_left (node) topk by (cluster, namespace, pod) (1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""}))

Any idea on how to fix this? Should this cluster label be added manually through the following values.yaml parameter?

## Additional labels for PrometheusRule alerts
  additionalRuleLabels: {}

Thanks

[Fr0s1]

I have the same problem, most of the dashboards show "No Data" when installing the latest Helm version 52.1.0 or downgrade to 51.9.4.

[knta7]

I have the same problem, most of the dashboards show "No Data" when installing the latest Helm version 52.1.0 or downgrade to 51.9.4.

if you are using k8s v1.24 or after and docker, docker plugin was removed from cAdvisor. See rancher/rancher#38934 (comment)

[deefdragon]

There is a PR open (#3863) But the issue appears to be upstream, and may need to be resolved elsewhere.