Plugin Vault: status 307 Temporary Redirect

[stackshadow]

Relevant telegraf.conf

[[inputs.vault]]
      url = "http://vaulty-0.dev:8200"
      token = "XXX"
      interval = "30s"

      [inputs.vault.tags]
        namespace = "dev"

Logs from Telegraf

2023-10-18T09:23:30Z E! [inputs.vault] Error in plugin: http://vaulty-0.dev:8200/v1/sys/metrics returned HTTP status 307 Temporary Redirect

System info

Telegraf 1.28.2 on Kubernetes v1.26.3

Docker

No response

Steps to reproduce

1. Use Vault in a cluster with 3 nodes
2. Add config to scrape all 3 nodes
3. Scrape of "standby"-node emit the "redirect-error" in the logs

Expected behavior

Follow the redirect

Actual behavior

Only the master-node will be scraped and for standby nodes we got these redirect-error

Additional info

No response

[powersj]

Hi,

Thanks for the report. Is this a new issue that was previously working? Or has this always occurred?

Follow the redirect

Do you happen to know what the redirected URLs are? If we do follow the redirect, does the new page actually produce real data?

Thanks

[stackshadow]

Well i think thats not an regression. So its not working before :)

Vault itself is in cluster mode, so one of the 3 nodes is the "leader" and all other vault-nodes are in standby. When an API-Call is made to an standby node, an 307 ( redirect ) will be done to the "leader" node. An curl will follow the http-redirect, but somehow this plugin does it not :)

For example:

We have three nodes:

• http://vaulty-0.dev:8200 <- standby
• http://vaulty-1.dev:8200 <- leader
• http://vaulty-2.dev:8200 <- standby

an curl will return this when do on a standby node:

curl -v http://vaulty-0.dev:8200/v1/sys/metrics
returns:

> GET /v1/sys/metrics HTTP/1.1
> Host: vaulty-0.dev:8200
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 307 Temporary Redirect
< Cache-Control: no-store
< Location: http://vaulty-1.dev:8200/v1/sys/metrics
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Date: Thu, 19 Oct 2023 06:43:48 GMT
< Content-Length: 0
< 
* Connection #0 to host vaulty-0.dev left intact

so http://vaulty-0.dev:8200 will be redirected to http://vaulty-1.dev:8200

and when the -L is used in curl, it will follow the redirect-url

https://everything.curl.dev/http/redirects#tell-curl-to-follow-redirects

I've build some internal telegraf plugins myself, so maybe i have on the weekend spare time to debugging this 😄

[srebhan]

@stackshadow can you please test the binary in #14153 available once CI finished the tests successfully. Let me know if this fixes the issue!

[stackshadow]

I had some wild week, thanks for your support, i will check if this works 👍