Clarify internal usage of empty label values by federation

[alexgreenbank]

Ref: prometheus/prometheus#11024

Attempt to clarify the existing usage of empty label values by the federation system.

More details of that use case here: prometheus/prometheus#8726 (comment)

The goal of adding this line to the documentation is to point out the internal exception to the "MUST NOT contain any empty label names or values." rule and reinforce that it should not be relied upon anywhere else.

[beorn7]

See my comment. I think to really clarify #11024, we should also add a paragraph to the documentation of the external_labels config setting, explaining the (limited) use cases where you want an empty label there.

[beorn7]

🤔 I'm wondering if the federation use case would ever lead to the necessity to send an empty label value via remote write.

In different news, I think the most common case of utilizing an empty instance label is actually the Pushgateway and not federation. (The Pushgateway even adds an empty instance label implicitly, see last paragraph of this doc section.)

But again, this is only relevant for exposition and subsequent scrape, not for remote write. The samples ingested via federation or from a Pushgateway won't have any empty labels anymore when sent via remote write.

So maybe let's take it from a different perspective:

Suggested change

Whilst [Federation](https://prometheus.io/docs/prometheus/latest/federation/) relies upon an `instance` label with an empty value this is strictly an internal use case. Everything else MUST adhere to the `MUST NOT contain any empty label names or values` requirement above.

The existing use cases for empty label names (mostly when scraping a [federation endpoint](https://prometheus.io/docs/prometheus/latest/federation/) or the [Pushgateway](https://github.com/prometheus/pushgateway?tab=readme-ov-file#about-the-job-and-instance-labels) with the `honor_labels: true` setting) are irrelevant for remote write and thus not in conflict with the “MUST NOT contain any empty label names or values” requirement above.

[beorn7]

Just checked: So federation also adds an empty instance label implicitly (if there is no instance label already, i.e. pretty much what the PGW is doing). Not relevant here if you change the paragraph to what I have suggested, but it tells us that the federation documentation doesn't really explain that fact. So we should also update the federation documentation (over in prometheus/prometheus). It also makes the necessity to use an empty label in external_labels even rarer.