Elastic APM Agent Fails to Send Data: 'Read timed out, response code is -1

[rbkuscu]

When the Elastic APM agent tries to send data to the APM server, it intermittently fails with the error "Read timed out, response code is -1." The agent logs also show warnings about the response body being null.

To Reproduce
Steps to reproduce the behavior:

1. Use this config:
   /bin/sh -c "java -Xms2G -Xmx4G -javaagent:/opt/deployment/elastic-apm-agent-1.48.1.jar -Delastic.apm.service_name=auth-service
   -Delastic.apm.server_url=
   https://elasticfleet.xxx.com/
   -Delastic.apm.hostname=10.90.xxx.xx -Delastic.apm.environment=production -Delastic.apm.application_packages=com.xxx.authentication.service -Dspring.profiles.active=production -jar /opt/deployment/auth-service/auth-service.jar --server.port=8081 >> /opt/deployment/auth-service/logs/auth-service.log"

2. Deploy the application with the above APM agent configuration.

3. Trigger transactions that should be reported to the APM server.

4. Observe the following error in the logs:
   2024-07-18 12:51:26,873 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null
   2024-07-18 12:51:26,873 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
   2024-07-18 12:51:26,873 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1
   2024-07-18 12:53:12,713 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null
   2024-07-18 12:53:12,713 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
   2024-07-18 12:53:12,713 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1

Expected behavior
I expected the Elastic APM agent to successfully send data to the APM server without encountering timeouts or null response body warnings.

Debug logs
2024-07-18 12:51:26,873 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null
2024-07-18 12:51:26,873 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
2024-07-18 12:51:26,873 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1
2024-07-18 12:53:12,713 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: null
2024-07-18 12:53:12,713 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
2024-07-18 12:53:12,713 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Read timed out, response code is -1

[simitt]

Hi @rbkuscu, could you please create a post in the discuss forum for this and categorize it with apm and server. Ensure to also look at the apm-server logs and include them if possible (sanitizing any potentially sensitive information first). It will allow other users to follow along with problems and discussions in one forum.

We aim for keeping the github repos for confirmed bugs and feature requests.