[outputs.prometheus_client] generates an error in 1.29.1 version

[wally007]

Relevant telegraf.conf

[agent]
  interval = "1m"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "1s"
  flush_interval = "10s"
  flush_jitter = "3s"
  precision = ""
  hostname = ""
  omit_hostname = false

[[outputs.prometheus_client]]
   listen = "localhost:9012"
   metric_version = 2
   export_timestamp = true

Logs from Telegraf

Dec 17 12:43:57 refinst-capture1 telegraf[2428]: 2023-12-17T12:43:57Z E! [telegraf] Error running agent: connecting output outputs.prometheus_client: error connecting to output "outputs.prometheus_client": Unknown scheme

System info

RHEL9.3, telegraf 1.29.1

Docker

No response

Steps to reproduce

1. Update telegraf from 1.28.5 to 1.29.1 with prometheus output plugin
2. Start Telegraf service
3. Watch the logs
   ...

Expected behavior

Service should start up and telegraf output plugin should work in 1.29 version same as 1.28

Actual behavior

Telegraf does not start and generates an error

Dec 17 12:43:57 refinst-capture1 telegraf[2428]: 2023-12-17T12:43:57Z E! [telegraf] Error running agent: connecting output outputs.prometheus_client: error connecting to output "outputs.prometheus_client": Unknown scheme

Additional info

Configuration that works in 1.28.5 version when upgraded to 1.29.1 fails to start Telegraf with an error

Dec 17 12:43:57 refinst-capture1 telegraf[2428]: 2023-12-17T12:43:57Z E! [telegraf] Error running agent: connecting output outputs.prometheus_client: error connecting to output "outputs.prometheus_client": Unknown scheme

When I remove the 1.29.1 version with yum remove telegraf and install 1.28.5 explicitly with yum install telegraf-1.28.5-1 - the same configuration as aboev works fine.

[powersj]

Hi @wally007,

Looks like a PR to add vsock support did not catch the case were no scheme was provided.

I have put up #14471 which in 20-30mins will have artifacts you can download and try out. Could you please give one of these artifacts a try and verify this resolves the issue?

Thanks

[dchiaramello]

Hello.
I faced the same problem.
I tested the PR artifact (Linux version, I only replaced the /usr/bin/telegraf file, however), but unfortunately the problem is still there:
[agent] Failed to connect to [outputs.prometheus_client], retrying in 15s, error was "Unknown scheme"
Regards

[dchiaramello]

Note that I "workarounded" the problem by adding the "http://" prefix in the listen attribute, in outputs.prometheus_client section (it used to be only "mgt:9273"):

[[outputs.prometheus_client]]
  ## Address to listen on.
  listen = "http://mgt:9273"

[powersj]

@dchiaramello can you please show the logs from the artifact and your config that still failed please?

@wally007 - can you try the artifacts please?

[dchiaramello]

@powersj

My telegraf.conf file is the following one:

[global_tags]
[agent]
  interval = "30s"
  round_interval = true
  metric_batch_size = 10000
  metric_buffer_limit = 100000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  omit_hostname = false

[[outputs.prometheus_client]]
  ## Address to listen on.
  listen = "mgt:9273"
  path = "/metrics"
  metric_version = 2
  tagexclude =  [ "url" ]

After having installed the artifact, systemctl status telegraf (after a restart) shows the following:

● telegraf.service - Telegraf
   Loaded: loaded (/usr/lib/systemd/system/telegraf.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2024-01-02 15:38:45 UTC; 7s ago
     Docs: https://github.com/influxdata/telegraf
 Main PID: 58388 (telegraf)
    Tasks: 7 (limit: 22694)
   Memory: 38.6M
   CGroup: /system.slice/telegraf.service
           └─58388 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Loaded inputs: chrony cpu disk diskio kernel kernel_vmstat mem net netstat nstat processes procstat (4x) statsd swap system x509_cert
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Loaded aggregators:
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Loaded processors:
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Loaded secretstores:
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Loaded outputs: prometheus_client
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! Tags enabled: host=AUSLAB005-CTL-01
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"AUSLAB005-CTL-01", Flush Interval:10s
Jan 02 15:38:45 AUSLAB005-CTL-01 telegraf[58388]: 2024-01-02T15:38:45Z E! [agent] Failed to connect to [outputs.prometheus_client], retrying in 15s, error was "Unknown scheme"
Jan 02 15:38:45 AUSLAB005-CTL-01 systemd[1]: Started Telegraf.

I don't know if these logs are OK for you, or if there's another way to retrieve them....

Regards,
Daniel

[dchiaramello]

Note that the mgt FQDN is resolved in the /etc/hosts, but it doesn't matter, I guess...

[dchiaramello]

And by using "http://mgt:9273" as listen value, I have the following log, in case it helps you somehow (but again, I don't think it will be the case):
Jan 02 15:44:54 AUSLAB005-CTL-01 telegraf[58605]: 2024-01-02T15:44:54Z I! [outputs.prometheus_client] Listening on http://192.105.5.2:9273/metrics

[powersj]

I don't know if these logs are OK for you, or if there's another way to retrieve them....

Can you get the full logs, I want to verify the version:

journalctl --no-pager --unit telegraf 

[dchiaramello]

Last lines after the "Stopped Telegraf." one are the following ones:

Jan 02 15:50:17 AUSLAB005-CTL-01 systemd[1]: Starting Telegraf...
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/chrony.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/controller.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/filebeat.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/sshd.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/system.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loading config: /etc/telegraf/telegraf.d/vnfcontroller.conf
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Starting Telegraf 1.30.0-56c500cc brought to you by InfluxData the makers of InfluxDB
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loaded inputs: chrony cpu disk diskio kernel kernel_vmstat mem net netstat nstat processes procstat (4x) statsd swap system x509_cert
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loaded aggregators:
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loaded processors:
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loaded secretstores:
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Loaded outputs: prometheus_client
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! Tags enabled: host=AUSLAB005-CTL-01
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"AUSLAB005-CTL-01", Flush Interval:10s
Jan 02 15:50:17 AUSLAB005-CTL-01 telegraf[58696]: 2024-01-02T15:50:17Z E! [agent] Failed to connect to [outputs.prometheus_client], retrying in 15s, error was "Unknown scheme"
Jan 02 15:50:17 AUSLAB005-CTL-01 systemd[1]: Started Telegraf.

[powersj]

@dchiaramello,

Really appeciate the output! thank you very much

listen = "mgt:9273"

It seems like the parse URI function thinks that "mgt" is the scheme. I am going to change this function up to default to listening on TCP and remove the unknown scheme error to keep backwards compatibility. I've updated the PR and new artifacts should be available in 20-30mins.

[dchiaramello]

OK - I'm about to leave now (being in Europe), I'll test it tomorrow

[dchiaramello]

Hello.

I confirm that, using latest PR (Telegraf 1.30.0-7cb0156d), Telegraf is successfully starting with listen = "mgt:9273" !!!

Thanks

[powersj]

Thank you!