NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors when using AWS X-Ray Exporter

[swapnil-dargude-rp]

Describe the bug
While using AWS Process credential provider, I encountered with an issue with the AWS OpenTelemetry Collector when trying to export metric and trace to AWS X-Ray . The collector fails with a error "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors". Below are the details of my setup and the error logs.

OpenTelemetry Collector Configuration (otel-collector-config.yaml)

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:55681
 
processors:
  batch:
 
exporters:
  awsxray:
    region: 'us-east-1'
  logging:
    verbosity: detailed
  awsemf:
    log_group_name: "log_group_name"
    log_stream_name: "log_stream_name"
    region: "us-east-1"
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]

docker-compose.yaml

  sr-otel-collector:
    container_name: otel-collector
    image: public.ecr.aws/aws-observability/aws-otel-collector:latest
    command: ["--config", "/etc/otel-collector-config.yaml"]
    networks:
      - proxy
    volumes:
      - ./OtelCollector/otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./config/.aws:/home/aoc/.aws
      - ./config/.aws/config:/home/aoc/.aws/config:ro
      - ./config/.aws/credentials:/home/aoc/.aws/credentials:ro
    environment:
      - AWS_PROFILE=rolesanywhere
      - AWS_REGION=us-east-1
      - HOME=/home/aoc
      - AWS_SDK_LOAD_CONFIG=1
      - AWS_CONFIG_FILE=/home/aoc/.aws/config
      - AWS_SHARED_CREDENTIALS_FILE=/home/aoc/.aws/credentials
    ports:
      - "55681:55681"
      - "4317:4317"
      - "4318:4318"

AWS ~/.aws/config:

[default]
region = us-east-1
output = json
[profile rolesanywhere]
credential_process = /home/aoc/.aws/aws_signing_helper credential-process --certificate /home/aoc/.aws/certificate.pem --private-key /home/aoc/.aws/decrypted_key.pem --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:XXXXXXXX:trust-anchor/xxxx-xxxx-xxxx-xxxx --profile-arn arn:aws:rolesanywhere:us-east-1:XXXXXXXXX:profile/xxxx-xxxx-xxxx-xxxx --role-arn arn:aws:iam::XXXXXXXXXX:role/rolesanywhere-role
region = us-east-1
output = json

AWS ~/.aws/credentials:

[default]
aws_access_key_id=XXXXXXXXXXXXXXXX
aws_secret_access_key=xxxxxx
region= us-east-1

What did you expect to see?
I expected the AWS OpenTelemetry Collector to successfully authenticate using AWS Process credential provider profile rolesanywhere and send trace data to AWS X-Ray.

What did you see instead?

Instead of the expected behavior, I encountered the following error in the logs:

{"kind": "exporter", "data_type": "traces", "name": "awsxray"}
2024-06-24T16:41:06.143Z	debug	[email protected]/awsxray.go:72	response error	{"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"}
2024-06-24T16:41:06.143Z	debug	[email protected]/awsxray.go:79	response: {
}	{"kind": "exporter", "data_type": "traces", "name": "awsxray"}
2024-06-24T16:41:06.143Z	error	exporterhelper/common.go:292	Exporting failed. Rejecting data.	{"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors", "rejected_items": 3}
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).send
	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:292
go.opentelemetry.io/collector/exporter/exporterhelper.NewTracesRequestExporter.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:134
go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces
	go.opentelemetry.io/collector/[email protected]/traces.go:25
go.opentelemetry.io/collector/processor/batchprocessor.(*batchTraces).export
	go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:414
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
	go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:261

Environment
OS: Red Hat Enterprise Linux (Build 4.18.0-305.88.1.el8_4.x86_64) (64-bit)
using docker container

Additional context
I have setup IAM roles anywhere for process credential provider

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stale for 30 days with no activity.

[forexpiphunter]

This issue needs to be opened again. I just deployed the latest version and getting the same error message.

{"L":"ERROR","T":"2024-11-26T18:50:08.218Z","M":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","repo":""}
2024/11/26 18:50:08 NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors