[kube-prometheus-stack] Configure additional aggregation labels + k8s.rules split

[jkroepke]

What this PR does / why we need it

I have the requirement to propagate labels from metrics to alerts.

For example, there is a label named "service_class" which should be present in alerts.

If there is an alternate solution, please let me know.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[jkroepke]

has someone a opinion on this or is it out of scope?

[QuentinBisson]

@jkroepke this part is not my area of expertise. Maybe @monotek or @GMartinez-Sisti can check this

[kgoralski]

If there is an alternate solution, please let me know.

I think an alternative solution is to make joins to kube_*_labels?

example:

kube_pod_labels * on (pod, namespace) group_right(label_team) (
  ...insert expression here
)

(sum by (namespace, pod) (
          max by(namespace, pod) (
            kube_pod_status_phase{job="kube-state-metrics", namespace=~".*", phase=~"Pending|Unknown"}
          ) * on(namespace, pod)   group_left(owner_kind) topk by(namespace, pod) (
            1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})
          )
        ) > 0) * on(namespace, pod) group_left(label_teamname) kube_pod_labels

Here is some additional context from kube_state_metrics maintainers why you may prefer joins

• Add an option to allow adding Kubernetes Labels as Prometheus Metric Labels per metric-series kubernetes/kube-state-metrics#1980 (comment)

[jkroepke]

@kgoralski I'm bit aware of this style of join, we already using it for define multiple thresholds (https://www.robustperception.io/using-time-series-as-alert-thresholds/)

However, in this case it does not feel an alternative for me.

Fine that the kube-state-metrics does not want to implement this, however thats something which is solvable with the metrics_relabel_config on a scrape configuration.

In my case, the metric kube_pod_status_phase already has the label "teamname" in my case and a join would not be helpful here.

[jkroepke]

@monotek or @GMartinez-Sisti can you take a look here?

[jkroepke]

The last run of the python script includes the split of k8s.rules

Ref: kubernetes-monitoring/kubernetes-mixin#882

@QuentinBisson @monotek or @GMartinez-Sisti Do you think, we need a new major bump?

[GMartinez-Sisti]

The last run of the python script includes the split of k8s.rules

Ref: kubernetes-monitoring/kubernetes-mixin#882

@QuentinBisson @monotek or @GMartinez-Sisti Do you think, we need a new major bump?

This is a big change 😅 , however it looks like the functionality is the same, just with some smaller groups. Can you confirm this is the situation?

[jkroepke]

@GMartinez-Sisti i compare the results with the upstream PR that I linked here. The changes are big, but the same split was done at the upstream repo, too.

[jkroepke]

I have to fix the names of the new rules

[jkroepke]

@GMartinez-Sisti I tested the changes on my local system with

helm upgrade -i kube-prometheus-stack . --set "defaultRules.additionalAggregationLabels={cluster,tenant}"

And I did not have any rule errors in prometheus.

[GMartinez-Sisti]

Thanks @jkroepke. Merging this as we can't run away from the rules split change for long.