Cannot specify ?stale with ?consistent, conflicting semantics

[pvyaka01]

What did you do?
Ran consul_exporter with the following:
./consul_exporter --web.listen-address=":9107" --consul.health-summary --consul.server="http://localhost:8500" --consul.require_consistent

What did you expect to see?
Reads to be consistent

What did you see instead? Under which circumstances?
level=error ts=2020-10-20T14:36:00.501Z caller=consul_exporter.go:268 msg="Failed to query catalog for nodes" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"

Environment
Consul running on localhost (version 1.8.3)
consul_exporter-0.7.1.linux-amd64

• System information:

  insert output of uname -srm here

• consul_exporter version:

  Linux 3.10.0-1127.13.1.el7.x86_64 x86_64

• Consul version:

Consul v1.8.3
Revision a9322b9c7

• Prometheus version:

prometheus, version 2.18.2 (branch: HEAD, revision: a6600f564e3c483cc820bae6c7a551db701a22b3)
build user: root@130a411dd4ff
build date: 20200609-09:05:58
go version: go1.14.4

• Logs:
  level=info ts=2020-10-20T14:41:25.357Z caller=consul_exporter.go:486 msg="Starting consul_exporter" version="(version=0.7.1, branch=HEAD, revision=ae46e2ea980ecf1c961bf332b71160be707b5e98)"
  level=info ts=2020-10-20T14:41:25.357Z caller=consul_exporter.go:487 build_context="(go=go1.14.6, user=root@fdd571c00696, date=20200721-15:34:41)"
  level=info ts=2020-10-20T14:41:25.357Z caller=consul_exporter.go:538 msg="Listening on address" address=:9107
  level=error ts=2020-10-20T14:41:26.505Z caller=consul_exporter.go:268 msg="Failed to query catalog for nodes" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"
  level=error ts=2020-10-20T14:41:26.524Z caller=consul_exporter.go:294 msg="Failed to query for services" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"
  level=error ts=2020-10-20T14:41:26.525Z caller=consul_exporter.go:311 msg="Failed to query service health" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"
  level=error ts=2020-10-20T14:41:28.501Z caller=consul_exporter.go:268 msg="Failed to query catalog for nodes" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"
  level=error ts=2020-10-20T14:41:28.521Z caller=consul_exporter.go:294 msg="Failed to query for services" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"
  level=error ts=2020-10-20T14:41:28.522Z caller=consul_exporter.go:311 msg="Failed to query service health" err="Unexpected response code: 400 (Cannot specify ?stale with ?consistent, conflicting semantics.)"

[pvyaka01]

Also, i took out --consul.require_consistent flag and this is what i get:
level=error ts=2020-10-20T14:43:00.637Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49238->127.0.0.1:8500: read: connection reset by peer"
level=error ts=2020-10-20T14:43:00.637Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49236->127.0.0.1:8500: read: connection reset by peer"
level=error ts=2020-10-20T14:43:00.637Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49234->127.0.0.1:8500: read: connection reset by peer"
level=error ts=2020-10-20T14:43:00.637Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49232->127.0.0.1:8500: read: connection reset by peer"
level=error ts=2020-10-20T14:43:00.656Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": EOF"
level=error ts=2020-10-20T14:43:00.634Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49124->127.0.0.1:8500: read: connection reset by peer"
level=error ts=2020-10-20T14:43:00.661Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": EOF"

Replaced service name with "xxxxxxx"

I can make curl calls to that endpoint and that works fine and returns data:
curl "http://localhost:8500/v1/health/service/xxxxxxx?stale="

Am i missing anything?

[simonpasquier]

You can't require consistent reads and reading from non-leader servers at the same time (which makes total sense). Thus if you set --consul.require_consistent, you need also --no-consul.allow_stale. Maybe the exporter should forcibly disable stale reads when consistent reads are enabled.

https://github.com/hashicorp/consul/blob/24bc8451d54b9602256175f090dc79c0a63c3eb1/agent/http.go#L873-L877

[pvyaka01]

Yes, does make sense... did not know there's a flag "-no-consul.allow_stale", help doesn't show that flag. Will try.

However, without require_consistent, i see a constant barrage of these messages:
level=error ts=2020-10-20T14:43:00.634Z caller=consul_exporter.go:398 msg="Failed to query service health" err="Get "http://localhost:8500/v1/health/service/xxxxxxx?stale=\": read tcp 127.0.0.1:49124->127.0.0.1:8500: read: connection reset by peer"
I can do a curl and get a response in under a second but consul_exporter is constantly logging that error. Any ideas?

Thanks for the help!

[simonpasquier]

... -> 127.0.0.1:8500: read: connection reset by peer indicates that the server resets the connection so I would check the consul server's logs.