SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH #6708

loganmzz · 2024-12-02T18:36:31Z

Checkboxes for prior research

I've gone through Developer Guide and API reference
I've checked AWS Forums and StackOverflow.
I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have written a tool trying to resolve existing Terraform resource.

Everything is working fine with many services (Secret Manager, Security Group, EventBridge) and two regions (eu-central-1 and ap-northeast-1) but listing secrets on ap-northeast-1.

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

It results in:

AggregateError
        at internalConnectMultiple (node:net:1114:18)
        at internalConnectMultiple (node:net:1177:5)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      code: 'ETIMEDOUT',
      name: 'TimeoutError',
      '$metadata': { attempts: 3, totalRetryDelay: 209 },
      [errors]: [
        Error: connect ETIMEDOUT 18.179.209.155:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.209.155',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
          port: 443
        },
        Error: connect ETIMEDOUT 18.179.211.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.211.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
          port: 443
        },
        Error: connect ETIMEDOUT 52.198.52.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '52.198.52.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
          port: 443
        }
      ]
    }

Tried with AWS CLI I have no issue:

AWS_REGION=ap-northeast-1 aws secretsmanager list-secrets --filters 'Key=name,Values=...'

Regression Issue

Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

Observed Behavior

Error:

AggregateError
    at internalConnectMultiple (node:net:1114:18)
    at internalConnectMultiple (node:net:1177:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ETIMEDOUT',
  name: 'TimeoutError',
  '$metadata': { attempts: 3, totalRetryDelay: 157 },
  [errors]: [
    Error: connect ETIMEDOUT 18.179.209.155:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.209.155',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
      port: 443
    },
    Error: connect ETIMEDOUT 18.179.211.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.211.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
      port: 443
    },
    Error: connect ETIMEDOUT 52.198.52.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '52.198.52.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
      port: 443
    }
  ]
}

Expected Behavior

Valid HTTP response

Possible Solution

N/A

Additional Information/Context

No response

The text was updated successfully, but these errors were encountered:

zshzbh · 2024-12-04T23:17:12Z

Hey @loganmzz ,

Thanks for the feedback! However I can't reproduce this issue.

This is the code I have -

import { SecretsManager } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManager({region: 'ap-northeast-1'});

const response = await client.listSecrets();
console.log(response)

And the result is 200.

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'f40c7c36-63d2-459b-a6e3-XXX',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  SecretList: [
    {
      ARN: 'arn:aws:secretsmanager:ap-northeast-1:XXX:secret:prod/test/secret-XXX',
      CreatedDate: 2024-12-04T23:07:09.893Z,
      LastChangedDate: 2024-12-04T23:07:10.145Z,
      Name: 'prod/test/secret',
      SecretVersionsToStages: [Object],
      Tags: []
    }
  ]
}

This error indicates a network timeout when trying to connect to AWS Secrets Manager. You can manually setup timeout and retry attempts -

const client = new SecretsManager({
    region: 'ap-northeast-1',
    logger: console,
    maxAttempts: 3,
    retryMode: 'standard',
    requestTimeout: 3000,
    connectTimeout: 3000
});

Or if you think I miss anything, please add additional info.

Thanks!

loganmzz · 2024-12-05T13:30:48Z

Looks like timeouts are ignored. I increased them to 50000 and I still facing same issue in only 3s.

I run it several times (~ 10), and it passed once... May be just need to "really" increase timeout :( Any other way to increase them?

loganmzz · 2024-12-05T13:35:51Z

Just to be sure the information is shared, I have no issue with CLI (Python?)

loganmzz · 2024-12-18T08:28:29Z

@zshzbh At stated previously your setup doesn't seem to affect timeout. Any other suggestion on how to setup custom timeouts?

zshzbh · 2024-12-18T23:10:42Z

There's some steps I recommend -

Checking secrets in ap-northeast-1:
I don't actually have access to check AWS secrets or configurations. You'll need to verify this in your own AWS account using the AWS CLI or console.
Terraform and network connectivity:
You're right that Terraform is a third-party tool. When using it with AWS, it's a good practice to check network connectivity. You can do this by:
- Ensuring your local machine can reach AWS endpoints
- Verifying your AWS credentials are correctly configured
- Running a simple AWS CLI command to test connectivity
Adding middleware to the code:
The middleware you provided looks good. It will log the context, input, and output of AWS SDK calls. Here's how you can add it to your AWS SDK client:

import { S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-1" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

Setting up higher max attempts:
You can increase the maximum number of retry attempts by configuring the retry strategy. Here's an example:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { RetryStrategy } from "@aws-sdk/util-retry";

const client = new S3Client({
  region: "ap-northeast-1",
  maxAttempts: 5,  // Increase this number as needed
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 5000,  // 5 seconds
    socketTimeout: 5000,  // 5 seconds
  }),
  retryStrategy: new RetryStrategy(() => Promise.resolve(3000)),  // 3 seconds delay between retries
});

Sharing reproducible code:
If the above steps don't resolve the issue, please provide:
- The complete Terraform configuration file
- Any relevant AWS SDK code you're using
- Full error messages and stack traces
- The exact CLI commands you're running
- Any environment variables or AWS credentials setup

loganmzz · 2024-12-23T09:45:57Z

There's some steps I recommend -

Checking secrets in ap-northeast-1:
I don't actually have access to check AWS secrets or configurations. You'll need to verify this in your own AWS account using the AWS CLI or console.

No issue here. Everything is working fine with other clients (CLI, Terraform) and other services.

Terraform and network connectivity:
You're right that Terraform is a third-party tool. When using it with AWS, it's a good practice to check network connectivity. You can do this by:

Ensuring your local machine can reach AWS endpoints

Verifying your AWS credentials are correctly configured

Running a simple AWS CLI command to test connectivity

As stated it was already validated.

Adding middleware to the code:
The middleware you provided looks good. It will log the context, input, and output of AWS SDK calls. Here's how you can add it to your AWS SDK client:

import { S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-1" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

Strange... As long as I run tests without middleware I had the issue. After adding it, I had no more timeout issue... I removed middleware it still works, resetting networking connection it still works. I will try after rebooting later on.

Setting up higher max attempts:
You can increase the maximum number of retry attempts by configuring the retry strategy. Here's an example:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { RetryStrategy } from "@aws-sdk/util-retry";

const client = new S3Client({
  region: "ap-northeast-1",
  maxAttempts: 5,  // Increase this number as needed
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 5000,  // 5 seconds
    socketTimeout: 5000,  // 5 seconds
  }),
  retryStrategy: new RetryStrategy(() => Promise.resolve(3000)),  // 3 seconds delay between retries
});

I will give it a try if I can reproduce error.

Sharing reproducible code:
If the above steps don't resolve the issue, please provide:

The complete Terraform configuration file

Any relevant AWS SDK code you're using

Full error messages and stack traces

The exact CLI commands you're running

Any environment variables or AWS credentials setup

Here my test script:

const client = require('@aws-sdk/client-secrets-manager');

async function main() {
  const sm = new client.SecretsManager({
    region: 'ap-northeast-1',
    /*
    logger: console,
    maxAttempts: 3,
    retryMode: 'standard',
    requestTimeout: 100000,
    connectTimeout: 100000,
    */
  });

  // sm.middlewareStack.add(
  //   (next, context) => async (args) => {
  //     console.log("AWS SDK context", context.clientName, context.commandName);
  //     console.log("AWS SDK request input", args.input);
  //     const result = await next(args);
  //     console.log("AWS SDK request output:", result.output);
  //     return result;
  //   },
  //   {
  //     name: "MyMiddleware",
  //     step: "build",
  //     override: true,
  //   }
  // );

  const listSecrets = await sm.listSecrets();
  console.log(JSON.stringify(listSecrets.SecretList, undefined, 2));
}

main();

loganmzz added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 2, 2024

zshzbh self-assigned this Dec 4, 2024

zshzbh added p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Dec 4, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Dec 6, 2024

zshzbh added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Dec 18, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Dec 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH #6708

SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH #6708

loganmzz commented Dec 2, 2024 •

edited

Loading

zshzbh commented Dec 4, 2024 •

edited

Loading

loganmzz commented Dec 5, 2024 •

edited

Loading

loganmzz commented Dec 5, 2024

loganmzz commented Dec 18, 2024

zshzbh commented Dec 18, 2024 •

edited

Loading

loganmzz commented Dec 23, 2024

SecretManager | List secrets on ap-northeast-1: connect ETIMEDOUT/ENETUNREACH #6708

SecretManager | List secrets on ap-northeast-1: connect ETIMEDOUT/ENETUNREACH #6708

Comments

loganmzz commented Dec 2, 2024 • edited Loading

Checkboxes for prior research

Describe the bug

Regression Issue

SDK version number

Which JavaScript Runtime is this issue in?

Details of the browser/Node.js/ReactNative version

Reproduction Steps

Observed Behavior

Expected Behavior

Possible Solution

Additional Information/Context

zshzbh commented Dec 4, 2024 • edited Loading

loganmzz commented Dec 5, 2024 • edited Loading

loganmzz commented Dec 5, 2024

loganmzz commented Dec 18, 2024

zshzbh commented Dec 18, 2024 • edited Loading

loganmzz commented Dec 23, 2024

SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH #6708

SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH #6708

loganmzz commented Dec 2, 2024 •

edited

Loading

zshzbh commented Dec 4, 2024 •

edited

Loading

loganmzz commented Dec 5, 2024 •

edited

Loading

zshzbh commented Dec 18, 2024 •

edited

Loading