This repository contains the configuration scripts needed to deploy the metric monitoring system for INDIGO-DataCloud project.
The metric collector uses MetricsGrimoire, generating dashboards, using VizGrimoire, for each repository under indigo-dc GitHub organization.
It is Docker-based (both DB and consumer). The metric collector container uses a custom Docker image with all the tools needed for the metric gathering and displaying.
The MetricsDB container uses the official MariaDB image and it is recommended to use a Docker volume for storing the DB data. The current (production) deployment at CERN OpenStack infra uses a (cinder) volume that is mapped to the Docker volume being used by the Metrics DB container.
The initial DB configuration is customized via SQL scripts that the Docker image triggers at boot time.
Just run the run_container.sh script to deploy both the MetricsDB and the
collector containers.
This script expects a file in /root/.sqa/sqarc with all the following environment variables
defined:
- REPO_PATH
- MYSQL_ROOT_PASSWORD
- MYSQL_USER
- MYSQL_PASSWORD
- MYSQL_HOST
- GITHUB_TOKEN
Tackled by deploy_dashboards.py script, it needs credentials (passed through
positional arguments) in order to get access to the MetricsDB and to push dashboard files
to GitHub. Usually it will be ok with:
deploy_dashboards.py <repo> $MYSQL_USER $MYSQL_PASSWORD $MYSQL_HOST $GITHUB_TOKEN
Once deploy_dashboards.py has been triggered in the collector, the
dashboards will be accesible via:
https://indigo-dc.github.io/<repository-name>
which uses GitHub Pages for creating the websites.
In the current implementation, the metrics manager container exposes a service (port 5000, see
below) that will receive POSTs from GitHub webhooks to create/update the dashboard for the given
repository. For that purpose it uses Python GitHub Webhooks
tool, which will run the hook
for triggering the deploy_dashboard.py execution.