I can't connect x-ray console via x-ray daemon by docker

[lethienhoang]

Dockerfile
FROM amazonlinux RUN yum install -y unzip RUN curl -o daemon.zip https://s3.dualstack.ap-southeast-2.amazonaws.com/aws-xray-assets.ap-southeast-2/xray-daemon/aws-xray-daemon-linux-3.x.zip RUN unzip daemon.zip && cp xray /usr/bin/xray ENTRYPOINT ["/usr/bin/xray", "-t", "0.0.0.0:2000", "-b", "0.0.0.0:2000"] EXPOSE 2000/udp EXPOSE 2000/tcp
...
docker run \ --attach STDOUT \ -v ~/.aws/:/root/.aws/:ro \ --net=host \ -e AWS_REGION=ap-southeast-2 \ --name xray-daemon \ -p 2000:2000/udp \ xray-daemon -o

I have been set up my aws access key and aws secret access key in credential of .aws folder

The result of docker running is

• HTTP proxy server using x-ray endpoint: HTTPS://xray.ap-southeast-2.amazonaws.com
• Starting proxy http server on 127.0.0.1:2000

Then go to https://xray.ap-southeast-2.amazonaws.com/ on browser and got this error
<MissingAuthenticationTokenException> <Message>Missing Authentication Token</Message> </MissingAuthenticationTokenException>

My app is running normally . I still see X-Amzn-Trace-Id: Root=1-5dd25d16-b57eba6c376e7395d1645b16; in header of response but don't push them to x-ray console.

How to fix them. What's wrong in setting up x-ray daemon,.... ? Please help me.

[bhautikpip]

Hi @lethienhoang,

Looks like you have configured aws-xray-daemon correctly. I think aws x-ray endpoint will not be accessible to you because daemon is using that endpoint to send data to x-ray service. I would check back configuration with aws-console again since you are not able to see traces on the console.

Follow up questions from my side

1. You have mentioned that you are able to see traces but not on the x-ray console. Do you see messages like "successfully sent batch of 1 segments" in daemon logs ?

2. Can you provide more details about how you have configured your application with x-ray SDK ?

Hope this helps!

[lethienhoang]

Thanks I found solutions but I will ask another quétions

[Gayatri31]

@lethienhoang
I have facing same issue, can you please add your solution here.

[awssandra]

@Gayatri31 Can you post your X-Ray Daemon logs? Thanks!

[Gayatri31]

Hi @awssandra
Dockerfile:

FROM amazonlinux
ENV AWS_XRAY_DEBUG_MODE=true
RUN yum install -y unzip
RUN curl -o daemon.zip https://s3.us-east-1.amazonaws.com/aws-xray-assets.us-east-1/xray-daemon/aws-xray-daemon-linux-3.x.zip
RUN unzip daemon.zip && cp xray /usr/bin/xray
ENTRYPOINT ["/usr/bin/xray", "-t", "127.0.0.1:2000", "-b", "127.0.0.1:2000"]
EXPOSE 2000/udp
EXPOSE 2000/tcp

command: docker run --attach STDOUT -v ~/.aws/:/root/.aws/:ro --net=host -e AWS_REGION=us-east-1 --expose 2000 -p 2000:2000/tcp -p 2000:2000/udp xray-daemon -o -l debug

Logs:

Added the keys to .aws/credentials file, created an IAM user with permissions: AWSXrayFullAccess, AWSXRayDaemonWriteAccess, AmazonS3FullAccess

I'm running the cat segment.txt > /dev/udp/127.0.0.1/2000 command and expecting that it should connect the daemon and show the successfully segment sent message but nothing is happening neither ant error nor any successor.
Please suggest me the ways or where I went wrong in setting this up. so that the data should push to Xray.

segment.txt
{"format": "json", "version": 1}
{"trace_id": "1-594aed87-ad72e26896b3f9d3a27054bb", "id": "6226467e3f845502", "start_time": 1498082657.37518, "end_time": 1498082695.4042, "name": "test.elasticbeanstalk.com"}

[bhautikpip]

Hi @Gayatri31 ,

Installation looks fine by looking at the daemon logs. When I ran(using this cat segment.txt > /dev/udp/127.0.0.1/2000) segment.txt provided by you was receiving this error in daemon logs.

2020-05-08T11:50:30-07:00 [Error] Unprocessed segment: {
  ErrorCode: "InvalidTraceId",
  Id: "6226467e3f845502",
  Message: "Invalid segment. ErrorCode: InvalidTraceId"
}
2020-05-08T11:50:30-07:00 [Warn] Batch that contains unprocessed segments

Would you mind trying out below segment.txt file and see if you see any logs:

{"format": "json", "version": 1}
{"id": "6226467e3f845502", "name": "test.elasticbeanstalk.com", "start_time": 1498082657.37518, "trace_id": "1-5eb58dce-ccc2dd20fab9a82e48e1b354", "end_time": 1498082695.4042}

I would also recommend you to try out by sending actual segments and subsegments if your goal is to verify daemon installation is correct. Also, are you running daemon locally ?

[Gayatri31]

Hi @bhautikpip
Thanks for responding.

yes, I'm running daemon locally. My goal is to push data to Xray using cmd AWSXRay.setDaemonAddress('127.0.0.1:2000').
I tried with segment.txt provided by you and got a success message when I ran cat segment.txt > /dev/udp/127.0.0.1/2000 inside the container but with an error too.

2020-05-10T11:53:00Z [Info] Successfully sent batch of 1 segments (2.943 seconds)
2020-05-10T11:53:47Z [Debug] Send 1 telemetry record(s)
2020-05-10T11:54:49Z [Debug] Send 1 telemetry record(s)
2020-05-10T11:55:51Z [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed to unmarshal response error
	status code: 400, request id: 
caused by: UnmarshalError: failed decoding error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 34 30 30  20 42 61 64 20 52 65 71  |itle>400 Bad Req|
00000020  75 65 73 74 3c 2f 74 69  74 6c 65 3e 3c 2f 68 65  |uest</title></he|
00000030  61 64 3e 0d 0a 3c 62 6f  64 79 20 62 67 63 6f 6c  |ad>..<body bgcol|
00000040  6f 72 3d 22 77 68 69 74  65 22 3e 0d 0a 3c 63 65  |or="white">..<ce|
00000050  6e 74 65 72 3e 3c 68 31  3e 34 30 30 20 42 61 64  |nter><h1>400 Bad|
00000060  20 52 65 71 75 65 73 74  3c 2f 68 31 3e 3c 2f 63  | Request</h1></c|
00000070  65 6e 74 65 72 3e 0d 0a  3c 2f 62 6f 64 79 3e 0d  |enter>..</body>.|
00000080  0a 3c 2f 68 74 6d 6c 3e  0d 0a                    |.</html>..|

caused by: invalid character '<' looking for beginning of value

Is there any way to access daemon outside the container?

[srprash]

Hi @Gayatri31
Sorry we missed on replying to your question.
Since your dockerfile has the daemon listening on 127.0.0.1:2000 address, you are able to send trace data to it from within the container. To communicate with the daemon from outside the container, you will need to bind the daemon to 0.0.0.0:2000 (example dockerfile). Then you will need to set the address of your container running the daemon as value to the AWS_XRAY_DAEMON_ADDRESS environment variable on the host/container running your application instrumented with X-Ray SDK.

As for the UnmarshalError that you see in the daemon logs, it's for the failure to send the periodic telemetry data to X-Ray service. This failure doesn't impact sending of your trace data in any way. We are aware of this bug and will try to fix it ASAP.

[zaf187]

@srprash is there an issue number for the bug? i came across it this morning and couldn't figure out if it was something in my application or something in the sdk. thanks.

[awssandra]

Hi zaf187,

This is the issue tracking: #22

[srprash]

Hi all,
I'm closing this issue since the original concerned has been answered. As for the UnmarshalError: failed decoding error message issue, I have opened a new one to track it separately.

Thanks