Emails not getting triggered for Custom Alert rules for Nginx

[sivaprakash123]

I have created Alert Rules for Nginx for a threshold limit on Nginx errors (4xx, 5xx). Below mentioned the alert rules for nginx. The alerts are not triggered though the expr is passed. Do i need to add/update labels or configuration in the prometheus-operator configuration for the nginx ?

Please let me know if any other details required

groups:
- name: nginx-alert.rules
  rules:
  - alert: nginx_failure_alert_warning
    annotations:
      message: Nginx error (4xx / 5xx) error count is more than 500 for the last 10
        minutes
      summary: dev Nginx Error rate is above 500
    expr: sum(count_over_time(nginx_http_requests_total{status!~"2..|3.."}[10m]))
      by (status, count) > 500
    labels:
      severity: warning

[gandham-santhosh]

@sivaprakash123 are you able to see these alerts being fired from the Prometheus dashboard? If it is showing in the fired alerts list then something to do with the notification configurations

[sivaprakash123]

@gandham-santhosh
Yes the alerts are in Firing state. Any specific configuration required for notification ? I have added the operator configuration below

Labels State Active Since Value
alertname="nginx_failure_alert_critical" severity="critical" status="404" FIRING 2024-05-06 05:55:16.008878185 +0000 UTC 1380
alertname="nginx_failure_alert_critical" severity="critical" status="419" FIRING 2024-05-06 07:19:46.008878185 +0000 UTC 1020

  - match_re:
      service: ".*nginx-public-ingress"
    receiver: "http_status_team"
    # Comment to ensure proper indentation while templating
    routes:
    # Comment to ensure proper indentation while templating
    - match:
        severity: "warning"
      receiver: "http_status_team_warning"
    # Comment to ensure proper indentation while templating
    - match:
        severity: "critical"
      receiver: "http_status_team_critical"
    # Comment to ensure proper indentation while templating
    - match:
        severity: "fatal"
      receiver: "http_status_team_fatal"

[sivaprakash123]

@gandham-santhosh I could all the email alerts are getting delivered to the default mailing list instead of different groups based on severity labels. How to consolidate this based on severity ?

[sivaprakash123]

@gandham-santhosh Please share your inputs