Prometheus Metrics

[seaneagan]

The helm operator supports release_condition_info and release_action_duration_seconds prometheus metrics. Similar metrics would be useful for the toolkit components. At least the code for this could be made generic and put in a common package for implementation in each controller:

helm_release_condition_info{namespace=gitops-system, name=podinfo, condition=Ready} -1
helm_release_action_duration_seconds{namespace=gitops-system, name=podinfo, action="install"} 34.05
git_repository_condition_info{namespace=gitops-system, name=config, condition=Ready} 1
git_repository_action_duration_seconds{namespace=gitops-system, name=config, action="download"} 12.20
# etc.

Or could create a single metric and use labels to distinguish group/value/kind (or maybe just kind is sufficient):

resource_condition_info{group="fluxcd", version=v2alpha1, kind=HelmRelease, namespace=gitops-system, name=podinfo, condition=Ready} -1
resource_action_duration_seconds{group="fluxcd", version=v2alpha1, kind=HelmRelease, namespace=gitops-system, name=podinfo, action=Ready} 34.05
resource_condition_info{group="fluxcd", version=v1alpha1, kind=GitRepository, namespace=gitops-system, name=config, condition=Ready} 1
resource_action_duration_seconds{group="fluxcd", version=v1alpha1, kind=GitRepository, namespace=gitops-system, name=config, action=download} 12.20

[stefanprodan]

All controllers expose metrics as provided by the controller-runtime (implemented in #94), for example:

# HELP controller_runtime_reconcile_errors_total Total number of reconciliation errors per controller
# TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="gitrepository"} 28

# HELP controller_runtime_reconcile_total Total number of reconciliations per controller
# TYPE controller_runtime_reconcile_total counter
controller_runtime_reconcile_total{controller="gitrepository",result="error"} 28
controller_runtime_reconcile_total{controller="gitrepository",result="requeue_after"} 2

# HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_count{name="gitrepository"} 13

We use the kind as the controller value across all toolkit components, so one can detect using promQL if a specific kind reconciler runs into errors or high latency. Adding the name and namespace as labels would result in high cardinality metrics and I'm reluctant to do so. Each controller issues events scoped to a specific kind/name/namespace. For troubleshooting, one would observe an error rate spike in the controller-runtime metrics and to find out which entity causes this, they will be inspecting the events or logs.

[stefanprodan]

The custom metrics implementation is tracked here #329