Latency issues w/ STS credentials

[yulius-take2]

Describe the bug

EKS uses the token based credentials to exchange the token w/ the key/secret pair. We were seeing random high latency of up to 5 seconds when calling dyanmodb.

We found the solution here.
https://www.rokt.com/engineering-blog/improving-app-latency-eks/

Summary of issues.

1. The sdk uses sts.amazonaws.com instead of the region specific url (sts.us-east-1.amazonaws.com). The DNS sometimes returns ips that are far away for some reason and the queries can take a couple of seconds.
2. When a token expires and a refresh happens, it is done in the current request's goroutine. The fix in the blog creates another go routine to periodically refresh the key/secret from the token to prevent slow queries.

Expected Behavior

STS should use a goroutine to refresh credentials in the background so queries don't pay for the latency caused by a refresh.
STS should use the region in the hostname. e.g sts.us-east-1.amazonaws.com instead of sts.amazonaws.com.

Current Behavior

STS token refresh happens inline w/ the request
STS uses sts.amazonaws.com to refresh the token without regard to the region.

Reproduction Steps

Use STS enable EKS. Do a poll request to dynamo or some other service. During refreshes, there will be latency issues.

Possible Solution

https://www.rokt.com/engineering-blog/improving-app-latency-eks/

Additional Information/Context

No response

SDK version used

v1.40.54

Environment details (Version of Go (go version)? OS name and version, etc.)

go version go1.17.9 linux/amd64/Linux social-service-api-develop-7f4f4bd4cb-lhjfg 5.4.181-99.354.amzn2.x86_64 #1 SMP Wed Mar 2 18:50:46 UTC 2022 x86_64 Linux

[rittneje]

The sdk uses sts.amazonaws.com instead of the region specific url (sts.us-east-1.amazonaws.com). The DNS sometimes returns ips that are far away for some reason and the queries can take a couple of seconds.

This can be addressed in one of three ways:

1. Set the eks.amazonaws.com/sts-regional-endpoints annotation on your ServiceAccount. https://github.com/aws/amazon-eks-pod-identity-webhook#eks-walkthrough
2. Set the AWS_STS_REGIONAL_ENDPOINTS environment variable in your pod spec.
3. Set the STSRegionalEndpoint field in your aws.Config.

[marselester]

I've put together some code based on Rokt's post https://github.com/marselester/awscreds. Maybe it will be useful to someone as a starting point.

[RanVaknin]

Hi all,

This is a pretty old issue ,and I see that there are number of suggestions / workaround on the thread.
If this still persists can you please give v2 a try and if it persists please open a separate issue on the v2 repo?

Thanks,
Ran~

[marselester]

Hi Ran, unfortunately I no longer can check if the issue persists.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.