Protocol error when using gRPC

[yangyulely]

What happened?

Steps to Reproduce

1. We are using node auto-instrumentations to collect telemetry data and send to opentelemetry collector.
2. Our collector was deployed on AWS EKS, and create a nginx ingress to expose collecting API.
3. We try to connect to opentelemetry collector with gRPC protocol.

Expected Result

Collect telemetry data and send to opentelemetry collector.

Actual Result

Protocol error when connecting to collector endpoint.

Additional Details

Java auto instrumentations works fine with gRPC, but nodejs not.
Opentelemetry colector ingress yaml as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: otel-ingress
  namespace: otel
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  tls:
    - hosts:
        - otel.example.com
      secretName: otel-tls-secret
  ingressClassName: nginx
  rules:
    - host: otel.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: otel-collector-opentelemetry-collector
                port:
                  number: 4317

OpenTelemetry Setup Code

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel.example.com"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_INSECURE="true"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
export OTEL_SERVICE_NAME="grpc_nodejs_demo"
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
cd $HOME/grpc_nodejs_demo
node $HOME/grpc_nodejs_demo/index.js

package.json

{
  "name": "nodejs_demo",
  "version": "1.0.0",
  "description": "nodejs demo for otel collector",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "license": "ISC",
  "dependencies": {
    "@grpc/grpc-js": "^1.12.5",
    "@opentelemetry/api": "^1.7.0",
    "@opentelemetry/auto-instrumentations-node": "^0.40.3",
    "@opentelemetry/exporter-logs-otlp-proto": "^0.49.1",
    "@opentelemetry/sdk-logs": "^0.49.1",
    "@opentelemetry/sdk-node": "^0.49.1",
    "body-parser": "^1.20.2",
    "cors": "^2.8.5",
    "express": "^4.18.2",
    "node-fetch": "2.6.2"
  }
}

Relevant log output

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error
    at callErrorFromStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
    at ServiceClientImpl.export (/home/otel/grpc_nodejs_demo/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at OTLPTraceExporter.send [as _send] (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/util.js:76:39)
    at /home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:44:18
    at new Promise (<anonymous>)
    at OTLPTraceExporter._sendPromise (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:43:25)
    at Immediate.<anonymous> (/home/otel/grpc_nodejs_demo/node_modules/@opentelemetry/auto-instrumentations-node/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js:74:22)
    at process.processImmediate (node:internal/timers:478:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 13,
  details: 'Received RST_STREAM with code 2 triggered by internal client error: Protocol error',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Operating System and Version

Ubuntu 22.04

Runtime and Version

Node.js v20.11.0