Skip to content

Commit

Permalink
Merge pull request #337 from logzio/poppulo
Browse files Browse the repository at this point in the history
Prometheus Alerts Migrator
  • Loading branch information
nico-shishkin authored Dec 20, 2023
2 parents 84d5046 + a34f5cf commit ad18c27
Show file tree
Hide file tree
Showing 2 changed files with 181 additions and 1 deletion.
180 changes: 180 additions & 0 deletions docs/shipping/Other/prometheus-alerts-migrator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
---
id: prometheus-alerts-migrator
title: Prometheus Alerts Migrator
overview: This Helm chart deploys the Prometheus Alerts Migrator as a Kubernetes controller, which automates the migration of Prometheus alert rules to Logz.io's alert format, facilitating monitoring and alert management in a Logz.io integrated environment.
product: ['metrics']
os: ['windows', 'linux']
filters: ['Other']
logo: https://dytvr9ot2sszz.cloudfront.net/logz-docs/shipper-logos/prometheusio-icon.svg
logs_dashboards: []
logs_alerts: []
logs2metrics: []
metrics_dashboards: []
metrics_alerts: []
drop_filter: []
---

This Helm chart deploys the Prometheus Alerts Migrator as a Kubernetes controller, which automates the migration of Prometheus alert rules to Logz.io's alert format, facilitating monitoring and alert management in a Logz.io integrated environment.

## Prerequisites

- Helm 3+
- Kubernetes 1.19+
- Logz.io account with API access

## Installing the Chart

To install the chart with the release name `logzio-prometheus-alerts-migrator`:

```shell
helm install \
--set config.rulesConfigMapAnnotation="prometheus.io/kube-rules" \
--set config.alerManagerConfigMapAnnotation="prometheus.io/kube-alertmanager" \
--set config.logzioAPIToken="your-logzio-api-token" \
--set config.logzioAPIURL="https://api.logz.io/" \
--set config.rulesDS="<<logzio_metrics_data_source_name>>" \
--set config.env_id="<<env_id>>" \
--set config.workerCount=2 \
logzio-prometheus-alerts-migrator logzio-helm/prometheus-alerts-migrator
```

## Configuration

The following table lists the configurable parameters of the Prometheus Alerts Migrator chart and their default values.

| Parameter | Description | Default |
| ----------------------------------------- | -------------------------------------------------- | -------------------------------- |
| `replicaCount` | Number of controller replicas | 1 |
| `image.repository` | Container image repository | logzio/prometheus-alerts-migrator|
| `image.pullPolicy` | Container image pull policy | IfNotPresent |
| `image.tag` | Container image tag | v1.0.0 |
| `serviceAccount.create` | Specifies whether a service account should be created | true |
| `serviceAccount.name` | The name of the service account to use | "" |
| `config.rulesConfigMapAnnotation` | ConfigMap annotation for rules | prometheus.io/kube-rules |
| `config.alertManagerConfigMapAnnotation` | ConfigMap annotation for alert manager configuration | prometheus.io/kube-alertmanager |
| `config.logzioAPIToken` | Logz.io API token | "" |
| `config.logzioAPIURL` | Logz.io API URL | https://api.logz.io/ |
| `config.rulesDS` | Data source for rules | IntegrationsTeamTesting_metrics |
| `config.env_id` | Environment ID | my-env-yotam |
| `config.workerCount` | Number of workers | 2 |
| `config.ignoreSlackText` | Ignore slack contact points `title` field | false |
| `config.ignoreSlackTitle` | Ignore slack contact points `text` field | false |
| `rbac.rules` | Custom rules for the Kubernetes cluster role | [{apiGroups: [""], resources: ["configmaps"], verbs: ["get", "list", "watch"]}] |

## Secret Management

The chart can optionally create a Kubernetes Secret to store sensitive information like the Logz.io API token. You can control the creation and naming of this Secret through the following configurations in the `values.yaml` file.

| Parameter | Description | Default |
| --------------- | ------------------------------------------------------------------ | ------------------- |
| `secret.create` | Determines whether a Secret should be created by the Helm chart. | `true` |
| `secret.name` | Specifies the name of the Secret to be used. | `logzio-api-token` |

### Using an Existing Secret

By default, the chart will create a Secret named `logzio-api-token`. You can change the name by setting `secret.name` to your preferred name. If you enable Secret creation, make sure to provide the actual token value in the `values.yaml` or via the `--set` flag.

```shell
helm install \
--set logzioAPIToken=your-logzio-api-token \
logzio-prometheus-alerts-migrator logzio-helm/prometheus-alerts-migrator
```

If you prefer to manage the Secret outside of the Helm chart (e.g., for security reasons or to use an existing Secret), set `secret.enabled` to `false` and provide the name of your existing Secret in `secret.name`.

Example of disabling Secret creation and using an existing Secret:

```shell
helm install \
--set secret.enabled=false \
--set secret.name=my-existing-secret \
logzio-prometheus-alerts-migrator logzio-helm/prometheus-alerts-migrator
```

In this case, ensure that your existing Secret `my-existing-secret` contains the necessary key (`token` in this context) with the appropriate value (Logz.io API token).

### Rules configMap Format

The controller is designed to process ConfigMaps containing Prometheus alert rules and Prometheus alert manager configuration. These ConfigMaps must be annotated with a specific key that matches the value of the `RULES_CONFIGMAP_ANNOTATION` or `ALERTMANAGER_CONFIGMAP_ANNOTATION` environment variables for the controller to process them.

### Example rules ConfigMap

Below is an example of how a rules configMap should be structured:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: logzio-rules
namespace: monitoring
annotations:
prometheus.io/kube-rules: "true"
data:
all_instances_down_otel_collector: |
alert: Opentelemetry_Collector_Down
expr: sum(up{app="opentelemetry-collector", job="kubernetes-pods"}) == 0
for: 5m
labels:
team: sre
severity: major
annotations:
description: "The OpenTelemetry collector has been down for more than 5 minutes."
summary: "Instance down"
```
* Replace `prometheus.io/kube-rules` with the actual annotation you use to identify relevant ConfigMaps. The data section should contain your Prometheus alert rules in YAML format.
* Deploy the ConfigMap to your cluster using `kubectl apply -f <configmap-file>.yml`.

Below is an example of how an alert manager ConfigMap should be structured:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: logzio-alertmanager
namespace: monitoring
annotations:
prometheus.io/kube-alertmanager: "true"
data:
alertmanager.yml: |
global:
# Global configurations, adjust these to your SMTP server details
smtp_smarthost: 'smtp.example.com:587'
smtp_from: '[email protected]'
smtp_auth_username: 'alertmanager'
smtp_auth_password: 'password'
# The root route on which each incoming alert enters.
route:
receiver: 'default-receiver'
group_by: ['alertname', 'env']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
# Child routes
routes:
- match:
env: production
receiver: 'slack-production'
continue: true
- match:
env: staging
receiver: 'slack-staging'
continue: true
# Receivers define ways to send notifications about alerts.
receivers:
- name: 'default-receiver'
email_configs:
- to: '[email protected]'
- name: 'slack-production'
slack_configs:
- api_url: 'https://hooks.slack.com/services/T00000000/B00000000/'
channel: '#prod-alerts'
- name: 'slack-staging'
slack_configs:
- api_url: 'https://hooks.slack.com/services/T00000000/B11111111/'
channel: '#staging-alerts'
```

* Replace p`rometheus.io/kube-alertmanager` with the actual annotation you use to identify relevant ConfigMaps. The data section should contain your Prometheus alert manager configuration in YAML format.

* Deploy the ConfigMap to your cluster using `kubectl apply -f <configmap-file>.yml`.
2 changes: 1 addition & 1 deletion static/manifest.json

Large diffs are not rendered by default.

0 comments on commit ad18c27

Please sign in to comment.