QFE joining ring post upgrading to 2.14.1 causing gossip alerts to misfire

[rishabhkumar92]

Describe the bug

QFE joining ring post upgrading to 2.14.1 which is causing gossip alerts (ref) to misfire

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir 2.14.1
2. Noticed alert firing and confirmed that QFE has joined the ring

Expected behavior

QFE shouldn't join the ring and this behavior was working as expected till 2.13

Environment

• Infrastructure: Kubernetes

Additional Context

NA

[rishabhkumar92]

[rishabhkumar92]

Any update on this one? cc: @armandgrillet

[AelwynBo]

Hello,
I am facing the same issue in a setup with virtual machines since 2.14.0 upgrade.

[fgouteroux]

any news on this guys ?

[fgouteroux]

Before the version 2.14, when we make a request in /memberlist we got the message This instance doesn't use memberlist..

Something has change as now the queryfrontend is now a member in Memberlist Cluster. So the alert should be adapted like purposed in PR #10289

[pracucci]

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

[fgouteroux]

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

Does this behavior will change in next release ? For me it's ok if the query-frontend join memberlist, we know are many nodes we have in the cluster.

[pracucci]

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

Does this behavior will change in next release ? For me it's ok if the query-frontend join memberlist, we know are many nodes we have in the cluster.

I don't expect it to change.

[fgouteroux]

@pracucci thx for the explanation