K6_OTEL_EXPORTER_TYPE=http overrides K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090

[agilob]

Brief summary

Prometheus in version 2.54 has OTEL on http not GRPC, so to use it I'm using the following script command:

K6_OTEL_EXPORTER_TYPE=http K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090 K6_OTEL_HTTP_EXPORTER_URL_PATH=/api/v1/otlp/v1/metrics K6_OTEL_METRIC_PREFIX=agilob  ./k6 run --out experimental-opentelemetry main.js

k6 version

v0.55.0

OS

linux

Docker version and image (if applicable)

No response

Steps to reproduce the problem

but the script fails because defining K6_OTEL_EXPORTER_TYPE overrides my K6_OTEL_GRPC_EXPORTER_ENDPOINT:

❯ K6_OTEL_EXPORTER_TYPE=http K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090 K6_OTEL_HTTP_EXPORTER_URL_PATH=/api/v1/otlp/v1/metrics K6_OTEL_METRIC_PREFIX=agilob  ./k6 run --out experimental-opentelemetry main.js

         /\      Grafana   /‾‾/
    /\  /  \     |\  __   /  /
   /  \/    \    | |/ /  /   ‾‾\
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/

     execution: local
        script: main.js
        output: opentelemetry (http, https://localhost:4318/api/v1/otlp/v1/metrics)

ERRO[0000] can't shutdown OpenTelemetry metric provider  error="failed to upload metrics: Post \"https://localhost:4318/api/v1/otlp/v1/metrics\": dial tcp [::1]:4318: connect: connection refused"
     █ setup

Expected behaviour

.

Actual behaviour

But when I remove K6_OTEL_EXPORTER_TYPE, my exporter URL is correct, but obviously won't work:

❯ K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090 K6_OTEL_HTTP_EXPORTER_URL_PATH=/api/v1/otlp/v1/metrics K6_OTEL_METRIC_PREFIX=agilob ./k6 run --out experimental-opentelemetry main.js

         /\      Grafana   /‾‾/
    /\  /  \     |\  __   /  /
   /  \/    \    | |/ /  /   ‾‾\
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/

     execution: local
        script: main.js
        output: opentelemetry (grpc, prometheus:9090)

ERRO[0010] can't shutdown OpenTelemetry metric provider  error="failed to upload metrics: context deadline exceeded: rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded"

Now, the url is correct, but protocol isn't.

On top of it, If I add https:// to the exporter URL, it starts producing #4060 :

         /\      Grafana   /‾‾/
    /\  /  \     |\  __   /  /
   /  \/    \    | |/ /  /   ‾‾\
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/

     execution: local
        script: main.js
        output: opentelemetry (grpc, https://prometheus:9090)

ERRO[0000] can't shutdown OpenTelemetry metric provider  error="failed to upload metrics: failed to exit idle mode: invalid target address https://prometheus:9090, error info: address https://prometheus:9090:443: too many colons in address"

[agilob]

Played with it a bit more and have to say this gives funny outcome:

K6_OTEL_EXPORTER_TYPE=http \
K6_OTEL_GRPC_EXPORTER_INSECURE=false \
K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090 \
K6_OTEL_HTTP_EXPORTER_URL_PATH=/api/v1/otlp/v1/metrics \
K6_OTEL_METRIC_PREFIX=agilob

output: opentelemetry (http, https://localhost:4318/api/v1/otlp/v1/metrics)

Setting insecure exporter overrides exporter URL and protocol?

Yup, works as documented below: changes my protocol and exporter to the default :D

K6_OTEL_GRPC_EXPORTER_ENDPOINT	Configures the gRPC exporter endpoint. Default is localhost:4317.

[olegbespalov]

Hi @agilob

I'm not quite sure if I'm getting, what do you try to achieve by mixing HTTP (K6_OTEL_HTTP_*) and gRPC (K6_OTEL_GRPC_) configuration options?

Do you expect to work both exporters at the same time? 🤔

[agilob]

Please ignore the comment it's just as an addition to show more bugs in the configuration of otel exporters. After reporting the bug I played more with other flags to find more inconsistencies/bugs. The comment only shows that despite using http collector, k6 will configure url and secure port of grpc collector.

The first comment/post is the meaningful one.

I think this whole piece of code that configures otel collector type, path, port, secure port should be rewritten as currently it is mostly incorrect (secure grpc url and port is selected despite explicitly configuring http exporter). In this case exporter type must take precedence over port, path and security of exporter.

[TimotejKovacka]

I think this is tied to grafana/xk6-output-opentelemetry#25 as I have pointed out in the issue with some of the other OTEL configurations. The current approach to the overall OTEL configuration is unpredictable and in most cases clashes with the intended functionality leading to discrepancies. You should report your findings for the issue so when it is picked up all the relevant problems could be addressed.

[olegbespalov]

I'm still having a hard time to understand why where the bug is 🤔

When we provide the configuration like in example (mixing gRPC and HTTP), we still need to decide what is meaningful. And currently, it's K6_OTEL_EXPORTER_TYPE which defines that we'd like to export using HTTP. The rest of the options will either be default (like in the example HTTP's endpoint localhost:4318) or will be overwritten if provided.

❯ K6_OTEL_EXPORTER_TYPE=http K6_OTEL_GRPC_EXPORTER_ENDPOINT=prometheus:9090 K6_OTEL_HTTP_EXPORTER_URL_PATH=/api/v1/otlp/v1/metrics K6_OTEL_METRIC_PREFIX=agilob  ./k6 run --out experimental-opentelemetry main.js

So in theory, nicer UX will probably tell user that, K6_OTEL_GRPC_EXPORTER_ENDPOINT cannot be used because we've chosen the K6_OTEL_EXPORTER_TYPE=http, or what is expected behavior?