Sidecar: no supported bucket was configured, uploads will be disabled

[MaxDiOrio]

Using latest releases of Prometheus-Operator and Thanos Sidecar (v0.4.0

What happened - Launched Prom Op with a Thanos Sidecar and receive the following error in the sidecar: caller=sidecar.go:123 msg="no supported bucket was configured, uploads will be disabled"

What you expected to happen. The sidecar would connect to the S3 compatible Object Storage

How to reproduce it (as minimally and precisely as possible):

Have the following in the Prometheus Operator values yaml file:

    thanos:
      baseImage: improbable/thanos
      version: v0.4.0
      peers: thanos-peers.prod.cluster:10900
      objectStorageConfig:
        key: bucket.yaml
        name: thanos-objstore-config

I have a secret name thanos-objstore-config in the same namespace with a key bucket.yaml and value:

type: S3
config:
  endpoint: "minioingress.rook-minio.10.85.166.33.xip.io"
  bucket: "thanos"
  access_key: "MINIO_Access_Key"
  secret_key: "MINIO_Secret_Key"
  insecure: true
  signature_version2: false

The actual Thanos Store container has no problems with the bucket config used to access this same bucket.

[bwplotka]

I am pretty sure this is something with how operator configures Thanos. Can you double check what the actual Sidecar and Prometheus YAML looks like when it is generated and running on Kubernetes?

[MaxDiOrio]

That sounds right - the yaml doesn't have any reference to the argument.

args:
        - sidecar
        - --prometheus.url=http://127.0.0.1:9090/
        - --tsdb.path=/prometheus
        - --cluster.address=[$(POD_IP)]:10900
        - --grpc-address=[$(POD_IP)]:10901
        - --cluster.peers=thanos-peers.prod.cluster:10900

[MaxDiOrio]

It's strange - doing a dry-run in debug mode in helm shows that it is using the config:

thanos:
      baseImage: improbable/thanos
      objectStorageConfig:
        key: bucket.yaml
        name: thanos-objstore-config
      peers: thanos-peers.prod.cluster:10900
      version: v0.4.0

[bwplotka]

Hm, maybe then just not deployed yet?

[MaxDiOrio]

It's all definitely been deployed - the whole operator itself is working except for the bucket config. It's really strange.

[FUSAKLA]

Hi, what is the status of this? Did you managet to get the sidecar uploading the blocks?
From the discussion it sounds more related to the prometheus operator than actual thanos issue.

[MaxDiOrio]

I gave up on Thanos. Far too many moving pieces for the size of our environment. Especially when a core piece doesn’t work. I switched to remote writing our Prometheus instances to a VictoriaMetrics server and couldn’t be happier.

[FUSAKLA]

Ok, sorry to hear that but whatever works for you is fine.
So I'm closing this fo now.

[Radoslaw-Kutylowski]

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config:

I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

[pjuri]

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

This helped, thanks!

[Elter71]

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

Works thanks! <3

[pduchnovsky]

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

If anybody is looking when this happened:
prometheus-community/helm-charts#3918