Telegraf v1.33 doesn't write to InfluxDB3 using influxdb_v2 output plugin, whereas Telegraf v1.32 does

[barbaranelson]

Relevant telegraf.conf

[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
  debug = true


[[outputs.influxdb_v2]]
        urls = ["http://localhost:8181"]
        bucket = "telegraf_metrics_2"
        organization = "theorg"
        namedrop = ["internal*"]
        http_headers = {"Authorization" = "Bearer anythingwilldo"}

[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states
  report_active = true

[[inputs.disk]]
  ## By default stats will be gathered for all mount points.
  ## Set mount_points will restrict the stats to only the specified mount points.
  # mount_points = ["/"]
  ## Ignore mount points by filesystem type.
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.diskio]]

[[inputs.mem]]

[[inputs.system]]

[[inputs.swap]]

[[inputs.net]]
  ## By default, telegraf gathers stats from any up interface (excluding loopback)
  ## Setting interfaces will tell it to gather these explicit interfaces,
  ## regardless of status.
  # interfaces = ["eth0"]

Logs from Telegraf

Context

I am running Telegraf v1.33 on my mac, collecting local system metrics, and writing them to a local instance of InfluxDB3 (also on my mac). With Telegraf v1.32, the writes are successful and the data is written to the InfluxDB3 database. With Telegraf v1.33, the writes appear to be successful (from the Telegraf side) but are not written to the InfluxDB3 database.

Telegraf v1.32 output (good case)

2025-01-07T22:35:40Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2025-01-07T22:35:40Z D! [agent] Successfully connected to outputs.influxdb_v2
2025-01-07T22:35:40Z D! [agent] Starting service inputs
2025-01-07T22:35:50Z E! [inputs.cpu] Error in plugin: error getting CPU info: not implemented yet
2025-01-07T22:35:50Z E! [inputs.diskio] Error in plugin: error getting disk io info: not implemented yet
2025-01-07T22:35:51Z D! [outputs.influxdb_v2] Wrote batch of 38 metrics in 751.351125ms
2025-01-07T22:35:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2025-01-07T22:36:00Z E! [inputs.cpu] Error in plugin: error getting CPU info: not implemented yet
2025-01-07T22:36:00Z E! [inputs.diskio] Error in plugin: error getting disk io info: not implemented yet
2025-01-07T22:36:01Z D! [outputs.influxdb_v2] Wrote batch of 38 metrics in 742.0325ms
2025-01-07T22:36:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

InfluxDB3 output for Telegraf v1.32

2025-01-07T22:36:10.266085Z DEBUG hyper::proto::h1::io: parsed 7 headers
2025-01-07T22:36:10.266136Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2025-01-07T22:36:10.266173Z DEBUG influxdb3_server::http: Processing request request=Request { method: POST, uri: /api/v2/write?bucket=telegraf_metrics_2&org=theorg, version: HTTP/1.1, headers: {"host": "localhost:8181", "user-agent": "Telegraf/1.32.0 Go/1.23.0", "transfer-encoding": "chunked", "accept-encoding": "gzip", "content-encoding": "gzip", "content-type": "text/plain; charset=utf-8"}, body: Body(Streaming) }
2025-01-07T22:36:10.266217Z  INFO influxdb3_server::http: write_lp to telegraf_metrics_2
2025-01-07T22:36:10.266498Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xA (10 bytes)
2025-01-07T22:36:10.268136Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xF0 (240 bytes)
2025-01-07T22:36:10.268200Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xF0 (240 bytes)
2025-01-07T22:36:10.268217Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xF0 (240 bytes)
2025-01-07T22:36:10.268293Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xF0 (240 bytes)
2025-01-07T22:36:10.268386Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xF0 (240 bytes)
2025-01-07T22:36:10.268410Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xB5 (181 bytes)
2025-01-07T22:36:10.268499Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x4 (4 bytes)
2025-01-07T22:36:10.268521Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x8 (8 bytes)
2025-01-07T22:36:10.268536Z DEBUG hyper::proto::h1::conn: incoming body completed
2025-01-07T22:36:10.268698Z DEBUG influxdb3_write::write_buffer: write_lp to telegraf_metrics_2 in writebuffer
2025-01-07T22:36:11.002534Z  INFO influxdb3_wal::object_store: flushing WAL buffer to object store n_ops=1 min_timestamp_ns=1736289370000000000 max_timestamp_ns=1736289370000000000 wal_file_number=803 snapshot_details=None

Telegraf v1.33 output (bad case)

2025-01-07T22:17:42Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2025-01-07T22:17:42Z D! [agent] Successfully connected to outputs.influxdb_v2
2025-01-07T22:17:42Z D! [agent] Starting service inputs
2025-01-07T22:17:52Z D! [outputs.influxdb_v2] Wrote batch of 39 metrics in 4.270667ms
2025-01-07T22:17:52Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2025-01-07T22:18:02Z D! [outputs.influxdb_v2] Wrote batch of 48 metrics in 1.899333ms
2025-01-07T22:18:02Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

InfluxDB3 output for Telegraf v1.33

2025-01-07T22:17:52.202661Z DEBUG hyper::proto::h1::io: parsed 7 headers
2025-01-07T22:17:52.202688Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2025-01-07T22:17:52.202706Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x643 (1603 bytes)
2025-01-07T22:17:52.202780Z DEBUG hyper::proto::h1::io: flushed 208 bytes
2025-01-07T22:17:52.202798Z DEBUG hyper::proto::h1::conn: incoming body completed
2025-01-07T22:18:02.201346Z DEBUG hyper::proto::h1::io: parsed 7 headers
2025-01-07T22:18:02.201388Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2025-01-07T22:18:02.201418Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x8BC (2236 bytes)
2025-01-07T22:18:02.201496Z DEBUG hyper::proto::h1::io: flushed 208 bytes
2025-01-07T22:18:02.201525Z DEBUG hyper::proto::h1::conn: incoming body completed
2025-01-07T22:18:10.110542Z DEBUG hyper::proto::h1::io: parsed 7 headers
2025-01-07T22:18:10.110575Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2025-01-07T22:18:10.110596Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x888 (2184 bytes)
2025-01-07T22:18:10.110647Z DEBUG hyper::proto::h1::io: flushed 208 bytes
2025-01-07T22:18:10.110670Z DEBUG hyper::proto::h1::conn: incoming body completed
2025-01-07T22:18:10.111074Z DEBUG hyper::proto::h1::conn: read eof

System info

Telegraf 1.33 on mac

Docker

No response

Steps to reproduce

1. Start Influxdb3
2. Start Telegraf
3. Telegraf output will appear to be successfully writing
4. Influxdb3 will not be saving the data
   ...

Expected behavior

This is a regression from Telegraf 1.32, that with the same config file, successfully writes to InfluxDB3.

Actual behavior

Writes appear to succeed, from Telegraf output, but do not get written by InfluxDB3.

Additional info

No response

[btasker]

I reproed with 1.33 and then retested using the artefacts in #16383 (comment)

[[inputs.cpu]]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb3ent.svc.lan"]
  http_headers = {"Authorization" = "Bearer anythingwilldo"}
  token = ""
  organization = ""
  bucket = "telegraf"
  insecure_skip_verify = true
  content_encoding = "identity"

With 1.33 a pcap shows Influx returning a 400 with

{"error":"Authorization header was malformed and should be in the form 'Authorization: Bearer <token>'"}

With the build artefact, the write now succeeds with a 204 and the data can be queried back out

influxdb3 query -l influxql -d telegraf "select * from cpu where host='ratchett' order by time desc limit 5"
+------------------+---------------------+-----------+----+----------+---------------+-------------+------------------+-------------------+---------------------+-----------+------------+---------------+-------------+--------------------+--------------------+
| iox::measurement | time                | cpu       | f1 | host     | measured_from | usage_guest | usage_guest_nice | usage_idle        | usage_iowait        | usage_irq | usage_nice | usage_softirq | usage_steal | usage_system       | usage_user         |
+------------------+---------------------+-----------+----+----------+---------------+-------------+------------------+-------------------+---------------------+-----------+------------+---------------+-------------+--------------------+--------------------+
| cpu              | 2025-01-09T09:14:25 | cpu-total |    | ratchett |               | 0.0         | 0.0              | 90.56954538103223 | 0.18590501941859366 | 0.0       | 0.0        | 0.0           | 0.0         | 2.51816799005041   | 6.726381612855768  |