SerializationError: failed decoding REST JSON error response

[tmuldoon]

I'm running AWS X-Ray daemon version 3.0.0 on my Windows 7 machine with the following options...

./xray_windows.exe -o -n us-east-1 -l debug

... and see SerializationErrors such as the following during idle periods.

2019-02-22T13:38:03-05:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value
2019-02-22T13:39:04-05:00 [Debug] Send 2 telemetry record(s)
2019-02-22T13:40:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:41:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:42:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:43:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:44:04-05:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value
2019-02-22T13:45:04-05:00 [Debug] Send 2 telemetry record(s)
2019-02-22T13:46:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:47:04-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:48:05-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:49:05-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:50:05-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:51:05-05:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value
2019-02-22T13:52:05-05:00 [Debug] Send 2 telemetry record(s)
2019-02-22T13:53:05-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:54:05-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:55:06-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:56:06-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:57:06-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:58:06-05:00 [Debug] Send 1 telemetry record(s)
2019-02-22T13:59:06-05:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value
2019-02-22T14:00:06-05:00 [Debug] Failed to send telemetry 2 record(s). Re-queue records. SerializationError: failed decoding REST JSON error response
caused by: invalid character '<' looking for beginning of value
2019-02-22T14:01:06-05:00 [Debug] Send 3 telemetry record(s)

Oddly enough, this doesn't seem to interfere with tracing. Is there some way to get more information?! Is this something that I should not worry about (since it's a debug message)?

[yogiraj07]

Hi @tmuldoon ,
Thank for posting the issue. The above debug message wont affect the daemon's ability to send X-Ray traces. The daemon send's telemetry records to X-Ray service to notify daemon health. From the above logs, the telemetry record receiving error are sent in the following interval. Example Send 3 telemetry record(s).

X-Ray daemon uses aws-go-sdk version 1.14.1.

I checked the git repo for aws-go-sdk and found that similar issue is present in the latest version of the aws-go-sdk : issue.

I will work with AWS GO SDK team to understand the scenario. Please stay tuned.

Thanks,
Yogi

[dagb]

I am experiencing the same error in the x-ray daemon log. Coincidentally no traces in X-Ray console, so seems like the problem affects x-ray monitoring.

[yogiraj07]

Hi @dagb ,
Can you help me understand the application environment, X-Ray SDK you are using and confirm the traces generated by the app are sent to X-Ray daemon address. By default the X-Ray SDK sends traces at UDP localhost:2000 and the daemon also listens on the same address. The above issue happens only when the daemon is ideal (not receiving any traces for longer time) and ideally shouldn't affect tracing abilities.

Thanks,
Yogi

[dagb]

Both the emitting app and X-Ray daemon runs in ECS. I have built the X-Ray daemon docker image with https://s3.dualstack.eu-west-1.amazonaws.com/aws-xray-assets.eu-west-1/xray-daemon/aws-xray-daemon-linux-3.x.zip

Port config of the X-Ray daemon container:
ENTRYPOINT ["/usr/bin/xray", "-b", "0.0.0.0:2000", "-t", "0.0.0.0:2000", "-n", "eu-west-1", "-o", "-l", "debug"]
EXPOSE 2000/udp
EXPOSE 2000/tcp

(In the log it states which X-Ray endpoint that is used:
Using Endpoint: https://xray.eu-west-1.amazonaws.com
)

The emitting app has environment variable
AWS_XRAY_DAEMON_ADDRESS=x-ray.local:2000

The app uses this X-Ray SDK (Maven dependency)

com.amazonaws
aws-xray-recorder-sdk-bom
2.2.1
pom
import

This was working for almost 4 days, the suddenly started to log errors (debug level), and no traces in the X-Ray console.

Will it help to restart the X-Ray container as a short term mitigation?

[dagb]

In addition I want to add that there is a test environment (different AWS account) using the same X-Ray daemon docker image. No errors and traces appears in the console.

[yogiraj07]

Hi @dagb ,
Thank you for letting me know the setup. There hasn't been any daemon update recently. From the description, the traces were sent almost for 4 days and suddenly no traces appeared in the X-Ray console.
One possibility is the DNS address of the X-Ray daemon "x-ray.local:2000" might have changed but this is not reflected on the SDK UDP emitter. Can you check the X-Ray SDK logs to check you see any errors? We have a feature request in the backlog for the JAVA SDK to periodically resolve the hostname. This is more of a SDK issue and not related to this daemon issue.

One solution I think is to restart X-Ray container so the X-Ray SDK resolves the DNS for the daemon address and the segments are received by the X-Ray Daemon.

Please feel free to open an issue if you still face the problem on JAVA SDK so we can do fast followup.

Thanks,
Yogi

[dagb]

Hi @yogiraj07 ,
My bad: this was caused by subnet (mis)configuration in ECS. I'm still a bit puzzled why it worked and stopped working, but at least I got to clean up some mess.

Thank for your feedback and directions.

[srprash]

The original concern in this issue has been resolved with the AWS SDK for Go v1.19.34
Resolving this issue to clear up some clutter. I have opened a new issue for tracking the ocassional UnmarshalError when calling PutTelemetryRecord.