Specify target labels for relabeling via /clients

[elliot-resdiary]

The /clients endpoint's labels field would be the ideal way to provide static target-based labels for relabeling metrics. Our use-case requires each scrape target to have an environment, region and component tags which are crucial when we're using these metrics.

If we could add a --labels argument to pushprox-client that would be perfect for us. What do others think? I haven't used Prometheus for long so maybe there's a better way to do this, though I'd be glad to raise a PR for this if others also would like it.

[roidelapluie]

Static labels can be added by prometheus itself at scrape time.

[elliot-resdiary]

Static labels can be added by prometheus itself at scrape time.

Do you mean in a static config? I was hoping to use the /clients endpoint to discover targets along with the labels required to make their metrics useful.

[SuperQ]

It's an interesting idea about how to arrange labels in a reasonably consistent way to the /clients discovery interface. At the same time, avoiding arbitrary cardinality issues.

I haven't used Prometheus for long so maybe there's a better way to do this.

There are few different options here. Without a detailed conversation about your overall architecture, and design use case for PushProx, it's hard to say. But that's not really a conversation to be had over an issue.

[elliot-resdiary]

They would fulfill the same purpose as static __meta_* tags which are available with the other service discovery mechanisms. Originally we were planning on using __meta_azure_machine_tag_<tagname> before realizing we needed PushProx to traverse NAT.

A few other examples of tag meta labels I found are

__meta_consul_tags
__meta_ec2_tag_<tagkey>
__meta_openstack_tag_<tagkey>
__meta_gce_tags

[SuperQ]

If this is an Azure deployment, you should be running Prometheus inside of your VPCs to avoid NAT issues. Prometheus is meant to be distributed and deployed inside networks. PushProx is meant for more IoT use cases.

[cycwll]

@elliot-resdiary I have the same requires as your

[Eagleseb]

I agree this would be really nice to have, I could maybe have a look at the code base to implement that if someone could give me pointers