[base-cluster] Grafana secret checking

[tmechsner]

I'm trying to add SMTP configuration to Grafana. It should work by adding SMTP details to the corresponding section in the grafana.ini which in the base-chart is filled from monitoring.grafana.config. However, I cannot add the password in plain text in the config, as this results in the following error:

Sensitive key 'smtp.password' should not be defi │
│ ned explicitly in values. Use variable expansion instead. You can disable this client-side validation by changing the value of assertNoLeakedSecrets.

At the same time I cannot add environment variables to Grafana to load the password from there, or disable the check by setting assertNoLeakedSecrets to false, as suggested here:
Related Grafana Issue 1
Related Grafana Issue 2

Am I missing something? Noone else using SMTP for Alerting? 😅

[cwrau]

Noone else using SMTP for Alerting? 😅

Actually yes, no one™️ (involved with this) is even using Grafana for alerting, let alone SMTP for sending alerts 😅

Best-practice is to write the alerts with PrometheusRules and configure the alertmanager to send the alerts somewhere (preferably an incident management tool like PagerDuty (which we support))

Currently only PagerDuty is supported (and free for small setups), if you have a different incident management tool you want to use feel free to open a feature request which we could then implement 😉

If you don't want to use an incident management tool, but instead fire one-shot alerts like with SMTP / Telegram / ..., we would have to implement this.

And last, if you want to set secret values for Grafana (for other reasons or because you want to stick with this alerting approach ((not recommended))) we would have to allow for that possibility as well.

[tmechsner]

Seems to be an ongoing discussion with people on both sides, actually. 😁 We are currently evaluating Better Stack as monitoring tool. It has a Prometheus Alertmanager Integration as well. The config would look like this:

route:
  receiver: betteruptime

receivers:
  - name: betteruptime
    webhook_configs:
      - url: <webhook-url>

Should be easy for you to add to the chart, I guess. Could you do this for us? :)