bugfix: Remove Prometheus Exemplars augmentation from counter metrics without suffix _total

[EvgeniaMartynova-thebeat]

Which problem is this PR solving?

In this PR we introduced support for Prometheus Exemplars for all counter and histogram metrics in clients and components. However according to OpenMetrics specification counter metric names should have suffix _total and histogram metrics names should have suffix _bucket in order to have the metrics augmented with exemplars. We found out that not following the requirement will bring a Service Monitor down.
In the logs of Service Monitor the following error would appear:

metric name X does not support exemplars

This error is coming from this func
Where there is a comment:

// Validate the name of the metric. It must have _total or _bucket as
// suffix for exemplars to be supported.

and here is the definition of function

Short description of the changes

Enabling exemplars only for counter metrics which have a name ending with a suffix _total.
_bucket suffix for Histograms is added by Prometheus itself, so we don’t need to worry about this. Please refer to the source code.

[mantzas]

@EvgeniaMartynova-thebeat Hi, I have a quick question: In order to support exemplars the name of the metric has to match a suffix.

1. Will this be fixed by prometheus?
2. If not would it make sense for us to provide our Counter, Gauge and Histogram constructors so that we support exemplars without the end users needing to provide the names? Even our own abstractions to support exemplars in the first place so that the type assertions are hidden?
   I am not saying that we should do this in this PR obviously but create a new issue for that.

[EvgeniaMartynova-thebeat]

@EvgeniaMartynova-thebeat Hi, I have a quick question: In order to support exemplars the name of the metric has to match a suffix.

1. Will this be fixed by prometheus?
2. If not would it make sense for us to provide our Counter, Gauge and Histogram constructors so that we support exemplars without the end users needing to provide the names? Even our own abstractions to support exemplars in the first place so that the type assertions are hidden?
   I am not saying that we should do this in this PR obviously but create a new issue for that.

Hi @mantzas,

1. I will create an issue for Prometheus.
2. This is a very nice idea I didn't think about this. We need to do it only for counters then because Prometheus is adding _bucket suffix for Histogram metric names under the hood and there is no exemplar support for Gauge metrics (please refer to this issue: Support for exemplar of other types of metrics, not just those containing the "_total" or "_bucket" suffixes prometheus/prometheus#9738). I can create a new ticket for patron and work on it.

[mantzas]

Hi @EvgeniaMartynova-thebeat ,
thanks let's add this also to histogram for consistency.
We still need to create a patron histogram to expose methods for exemplar and non-exemplar usage.
This abstraction should be a thin as possible e.g. native embedding/composition strategy.