High latency when executing SNS publish operation

[m-combinator]

Hello,

I'm seeing some occasional latency spikes in production (up to 7 seconds) when publishing to SNS from EC2. According to datadog APM, the bottleneck is the call to SocketInputStream.read(byte[], int, int, int) which happens somewhere deep in the publish call. When looking at the sdk metrics in cloudwatch, the APICallDuration and ServiceCallDuration correlate with the numbers in datadog (> 6 seconds).

I'm using a r6a.2xlarge EC2 instance which runs in the same region as SNS (us-east-1)

I'm also using the SNS Java sync client and I'm creating it as follows

  private val metricsPublisher = CloudWatchMetricPublisher
    .builder()
    .cloudWatchClient(
      CloudWatchAsyncClient
        .builder()
        .httpClientBuilder(
          NettyNioAsyncHttpClient
            .builder()
        )
        .build()
    )
    .build()

  protected val client: SnsClient = SnsClient.builder
    .endpointOverride(URI.create(endpoint))
    .region(region)
    .overrideConfiguration(
      ClientOverrideConfiguration
        .builder()
        .addMetricPublisher(metricsPublisher)
        .build()
    )
    .build()

Any suggestions on possible fixes?

[debora-ito]

ServiceCallDuration is basically the time between request marshalling and response unmarshalling. If this metric is high it's probably due to network latency or service-side latency, meaning it's outside of the SDK control.

However, one metric that is included in ServiceCallDuration and is under SDK control is the time to acquire the connection from the pool (ConcurrencyAcquireDuration metric). Does ConcurrencyAcquireDuration correlates with the spikes?

[m-combinator]

@debora-ito Thanks for responding. ConcurrencyAcquireDuration is actually quite low (max 18 milliseconds in 5-minute period), so it does seems it's due to network or service-side latency. Any suggestions on possible fixes?

[debora-ito]

You can tune the client with customized timeouts so the request can fail faster, and set retries (if the default retries do not cover those errors) to make the application more resilient.

This blog post shows how to interpret the metrics and use them to tune the clients - https://aws.amazon.com/blogs/developer/tuning-the-aws-sdk-for-java-to-improve-resiliency/
Note that it uses Java SDK 1.x in the code samples to show how to set the configurations, but the same concept can be applied to Java SDK 2.x.