Random "There is already an existing meter named" exception after adding "meterRegistry.config().commonTags"

[fedotxxl]

Hello. I use following modules:

 micronautVersion = "2.0.0"

    implementation "io.micronaut:micronaut-management"
    implementation "io.micronaut.micrometer:micronaut-micrometer-registry-prometheus:2.0.1"

According to this documentation I created MeterRegistryConfigurer to add common tag:

        @Singleton
        public static class CustomMicrometerMeterRegistryConfigurer implements MeterRegistryConfigurer {

            @Override
            public void configure(@NotNull MeterRegistry meterRegistry) {
                meterRegistry.config().commonTags("application", "my-application");
            }

            @Override
            public boolean supports(MeterRegistry meterRegistry) {
                return true;
            }

        }

After that metrics behavior becomes pretty unpredictable. After application restart, the following may occur:

1. Metrics returned by PrometheusMeterRegistry may contain application tag
2. Metrics returned by PrometheusMeterRegistry may not contain application tag
3. "Prometheus requires that all meters with the same name have the same set of tag keys" exception:

Caused by: java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'jvm_threads_states_threads' containing tag keys [state]. The meter you are attempting to register has keys [application, state].
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$16(PrometheusMeterRegistry.java:419)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932)
	at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:403)
	at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:205)
	at io.micrometer.core.instrument.MeterRegistry.lambda$gauge$1(MeterRegistry.java:295)
	at io.micrometer.core.instrument.MeterRegistry.lambda$registerMeterIfNecessary$5(MeterRegistry.java:559)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:612)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:566)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:559)
	at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:295)
	at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:190)
	at io.micrometer.core.instrument.composite.CompositeGauge.registerNewMeter(CompositeGauge.java:58)
	at io.micrometer.core.instrument.composite.CompositeGauge.registerNewMeter(CompositeGauge.java:27)
	at io.micrometer.core.instrument.composite.AbstractCompositeMeter.add(AbstractCompositeMeter.java:66)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at java.base/java.util.Collections$SetFromMap.forEach(Collections.java:5581)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.updateDescendants(CompositeMeterRegistry.java:216)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$add$5(CompositeMeterRegistry.java:141)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lock(CompositeMeterRegistry.java:184)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.add(CompositeMeterRegistry.java:133)
	at io.micronaut.configuration.metrics.micrometer.MeterRegistryFactory.compositeMeterRegistry(MeterRegistryFactory.java:69)
	at io.micronaut.configuration.metrics.micrometer.$MeterRegistryFactory$CompositeMeterRegistry0Definition.build(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1888)

Application is run by Intellij IDEA (not grade)

Why application behavior becomes random? Shouldn't it be predictable?

[fedotxxl]

I was able to add common tag by MeterFilter:

        @Bean
        @Singleton
        MeterFilter addCommonTagFilter() {
            return MeterFilter.commonTags(to.list(Tag.of("application", "my-application")));
        }

But still:

1. Your documentation says that it can be added by MeterRegistryConfigurer
2. It leads to unpredictable behaviour

[alvarosanchez]

I have been able to reproduce that not all metrics may have the common tag. However, I've never got an exception. Would you mind sharing a sample app?

[alvarosanchez]

This seems to work:

@Singleton
public class CustomMicrometerMeterRegistryConfigurer implements MeterRegistryConfigurer {

    @Override
    public void configure(MeterRegistry meterRegistry) {
        meterRegistry.config().meterFilter(MeterFilter.commonTags(singletonList(Tag.of("scope", "demo"))));
    }

    @Override
    public boolean supports(MeterRegistry meterRegistry) {
        return true;
    }

}

[jostick]

I was also facing the issue of tags randomly missing and reappearing between container restarts. The reason of this bug is described in #417. They are duplicates.

This also explains why registering a MeterFilter works reliably while using a MeterRegistryConfigurer as described in the Micronaut documentation does not.