Using Prometheus

Using Prometheus with tng-bench to collect time series monitoring data

This page assumes that you have already installed tng-sdk-benchmark and an execution machine running vim-emu as it is described on this wikipage.

Setup

We now extend the setup and use a Prometheus DB (running on Machine 1) instance to automatically collect monitoring data from the execution machine (Machine 2). On Machine 2 we run a cAdvisor as well as Prometheus' NodeExporter to collect system statistics, like CPU usage etc.

All additional components will be deployed as Docker containers.

+-------------------------+      +----------------------------+
| +--------------------+  |      | +------------------------+ |
| |                    |  |      | |                        | |
| |     tng-bench      |  |      | |    tng-bench-emusrv    | |
| |(experiment control)|--+------+-> (vim-emu w. ctrl. API) | |
| |                    |  |      | |                        | |
| +--------------------+  |      | +------------------------+ |
|                         |      | +---------------+          |
|    +-----------------+  |  +---+-+   cAdvisor    |          |
|    |   Proemtheus    <--+--+   | +---------------+          |
|    |(time series DB) <--+--+   | +---------------+          |
|    +-----------------+  |  +---+-+ NodeExporter  |          |
|                         |      | +---------------+          |
|                         |      |                            |
|                         |      |     Machine 2: Target      |
|  Machine 1 (tng-bench)  |      |(vim-emu execution platform)|
+-------------------------+      +----------------------------+

Installation

Machine 2 (Executor)

Open additional ports: sudo ufw allow 9100:9120/tcp

Run Node exporter (as Docker container):

sudo docker run -d \
  --restart always \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  --publish=9100:9100 \
  --name=nodeexporter \
  quay.io/prometheus/node-exporter \
  --path.rootfs /host

Run cAdvisor (as Docker container):

sudo docker run \
  --restart always \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=9102:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

Machine 1 (tng-bench)

Go to ~/tng-sdk-benchmark/prometheus/:

Update the prometheus.yml to point to your executor machine.

Use docker-compose to bring up the Prometheus instance:

docker-compose up

If you encounter a permission error, try:

sudo addgroup nobody
sudo usermod -aG nobody nobody
sudo chown nobody:nobody prometheus-data

Usage

Prometheus will now continuously collect statistics from the executor machine and write them to disk. You can access the Prometheus GUI using at http://<ip-of-machine-1>:9090 and see the collected metrics.

Examples for metric queries/visualisation:

rate(node_cpu_seconds_total[1m])

rate(container_cpu_usage_seconds_total{name="mn.vnf0.vdu01.0"}[1m])

rate(suricata_stats_capture_kernel_packets[1m])