Support credential profiles #74

mousedownmike · 2020-10-12T17:14:37Z

I just had some difficulty getting the Daemon running locally due to the way I have my credentials configured. To get things to work, I ultimately had to change my [default] credentials. This request is to add support for credential profiles. It's possible this can work another way, I just couldn't find it documented.

For reference, here is a sample credentials file.

[mike]
aws_access_key_id = AKIAZZZZZZZZZZZZ1234
aws_secret_access_key = abcdefghiljekls+JB2AupNJKv9
[corp]
aws_access_key_id = AKIAXXXXXXXXXXX5678
aws_secret_access_key = zyxwvutsrqponm+1CPiUFxYw
[default]
aws_access_key_id = AKIAZZZZZZZZZZZZ1234
aws_secret_access_key = abcdefghiljekls+JB2AupNJKv9

And here is the sample config file to go with that:

[profile mike]
region = us-east-1
[profile corp]
output = json
region = us-east-1
[profile corpdev]
output = json
region = us-east-1
role_arn = arn:aws:iam::123412341234:role/corp-dev-account-role
source_profile = corp
[default]
output = json
region = us-east-1

I would like to set an environment variable of AWS_PROFILE=corpdev or pass a command-line argument and have the Daemon use the specified profile's configuration/credentials.

I was able to confirm that the corpdev profile permissions were sufficient by running the following CLI:

aws xray put-trace-segments --trace-segment-documents "{\"trace_id\": \"1-5f84854a-b250fd33b4dd6208733a59df\", \"id\": \"ebd6c854cd28bca2\", \"start_time\": 1602520394, \"in_progress\": true, \"name\": \"Scorekeep-build\"}" --profile corpdev

I tried passing the role directly like -r "arn:aws:iam::123412341234:role/corp-dev-account-role" but that resulted in an AccessDenied error indicating the user from my [mike] (same as [default]) profile did not have permsission to assume that role (which is accurate).

In order to move on, I simply changed by [default] profile to be the [corp] profile and I pass the corp-dev-account-role ARN in as a -r command line option and it works.

This was pretty difficult to diagnose due to the default logging information available:

Without changing the log level, all of the credential errors are absorbed and the messages seem to indicate everything was sent fine but nothing appears in the console.
With logging set to dev, there are a number of JSON parsing errors that seem to indicate an error but they're apparently expected (SerializationError: failed decoding REST JSON error response #22).

The text was updated successfully, but these errors were encountered:

lupengamzn · 2020-10-19T20:19:55Z

Hi @mousedownmike ,

Thanks for the feedback

X-Ray Daemon is using AWS SDK Go to send requests to AWS service endpoint and based on AWS Go's documentation, you can specify which profile to use by setting AWS_PROFILE as an environment variable. That way AWS_PROFILE will be applied by AWS Go (and Daemon that is running locally).

We're planning to support passing a command-line argument when running Daemon locally and we'll put this in our backlog. Also you're always welcome to submit a PR to contribute.

Thanks

mousedownmike · 2020-10-19T20:58:35Z

@lupengamzn, I did try setting the AWS_PROFILE environment variable and it had no effect on running the daemon with the profile as I described it above (i.e. a profile that uses a source_profile).

mousedownmike · 2020-10-20T15:02:48Z

The curious thing is that I develop in Go and use the aws-sdk-go for my projects. In those projects, the AWS_PROFILE environment variable is respected and works for the profile configuration I defined above. Assuming the daemon is using a recent version of the SDK, this seems like a strange issue.

I will try to find some time to dig a little deeper into this and report back.

lupengamzn · 2020-10-26T23:19:41Z

Hey @mousedownmike ,

I just tried this on my end with X-Ray Daemon v3.2.0 that is using AWS SDK Go v1.25.42 and I can send traces to X-Ray through AWS Cli with the existence of source_file similar to your config file, but getting errors when running the Daemon. Can you provide some error logs here? I assume this may be due to the AWS SDK Go version, let me try this with an updated AWS Go version and get back to you soon.

Thanks

vissree · 2020-10-28T11:29:23Z

Hi @mousedownmike ,

Thanks for the feedback

X-Ray Daemon is using AWS SDK Go to send requests to AWS service endpoint and based on AWS Go's documentation, you can specify which profile to use by setting AWS_PROFILE as an environment variable. That way AWS_PROFILE will be applied by AWS Go (and Daemon that is running locally).

We're planning to support passing a command-line argument when running Daemon locally and we'll put this in our backlog. Also you're always welcome to submit a PR to contribute.

Thanks

Created a pull request for this. Could you please review?

mousedownmike · 2020-10-29T22:10:13Z

@lupengamzn I'm using the Docker image with the following Dockerfile.

FROM amazonlinux
RUN yum install -y unzip
RUN curl -o daemon.zip https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-3.x.zip
RUN unzip daemon.zip && cp xray /usr/bin/xray
ENTRYPOINT ["/usr/bin/xray", "-t", "0.0.0.0:2000", "-b", "0.0.0.0:2000"]
EXPOSE 2000/udp
EXPOSE 2000/tcp

This is the run configuration:

docker build -t xray-daemon .
&& docker run
-p 2000:2000/udp
--env AWS_REGION=us-east-1
--env AWS_PROFILE=myprofile
-v /home/mike/.aws:/root/.aws:ro
--net host
xray-daemon -l dev -o

Here is the output I'm getting:

2020-10-29T22:02:21Z [Info] Initializing AWS X-Ray daemon 3.2.0
2020-10-29T22:02:21Z [Debug] Listening on UDP 0.0.0.0:2000
2020-10-29T22:02:21Z [Info] Using buffer memory limit of 156 MB
2020-10-29T22:02:21Z [Info] 2496 segment buffers allocated
2020-10-29T22:02:21Z [Debug] Using proxy address: 
2020-10-29T22:02:21Z [Debug] Fetch region us-east-1 from environment variables
2020-10-29T22:02:21Z [Info] Using region: us-east-1
2020-10-29T22:02:21Z [Debug] ARN of the AWS resource running the daemon: 
2020-10-29T22:02:21Z [Debug] No Metadata set for telemetry records
2020-10-29T22:02:21Z [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
2020-10-29T22:02:21Z [Debug] Telemetry initiated
2020-10-29T22:02:21Z [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-east-1.amazonaws.com
2020-10-29T22:02:21Z [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
2020-10-29T22:02:21Z [Debug] Batch size: 50
2020-10-29T22:02:21Z [Info] Starting proxy http server on 0.0.0.0:2000
2020-10-29T22:02:38Z [Debug] Received request on HTTP Proxy server : /GetSamplingRules
2020-10-29T22:02:38Z [Debug] processor: sending partial batch
2020-10-29T22:02:38Z [Debug] processor: segment batch size: 1. capacity: 50
2020-10-29T22:03:02Z [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2020/10/29 22:03:02 http: proxy error: context canceled
2020-10-29T22:03:15Z [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2020-10-29T22:03:15Z [Warn] Delaying sending of additional batches by 0 seconds

For good measure, I just ran the daemon directly from the command line with the same setup and received the same errors.

$ export AWS_PROFILE=myprofile                                                                 
$ export AWS_REGION=us-east-1   
$ ./xray -l dev -o
2020-10-29T15:16:06-07:00 [Info] Initializing AWS X-Ray daemon 3.2.0
2020-10-29T15:16:06-07:00 [Debug] Listening on UDP 127.0.0.1:2000
2020-10-29T15:16:06-07:00 [Info] Using buffer memory limit of 156 MB
2020-10-29T15:16:06-07:00 [Info] 2496 segment buffers allocated
2020-10-29T15:16:06-07:00 [Debug] Using proxy address: 
2020-10-29T15:16:06-07:00 [Debug] Fetch region us-east-1 from environment variables
2020-10-29T15:16:06-07:00 [Info] Using region: us-east-1
2020-10-29T15:16:06-07:00 [Debug] ARN of the AWS resource running the daemon: 
2020-10-29T15:16:06-07:00 [Debug] No Metadata set for telemetry records
2020-10-29T15:16:06-07:00 [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
2020-10-29T15:16:06-07:00 [Debug] Telemetry initiated
2020-10-29T15:16:06-07:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-east-1.amazonaws.com
2020-10-29T15:16:06-07:00 [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com
2020-10-29T15:16:06-07:00 [Debug] Batch size: 50
2020-10-29T15:16:06-07:00 [Info] Starting proxy http server on 127.0.0.1:2000
2020-10-29T15:16:40-07:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules
2020-10-29T15:16:40-07:00 [Debug] processor: sending partial batch
2020-10-29T15:16:40-07:00 [Debug] processor: segment batch size: 1. capacity: 50
2020-10-29T15:17:04-07:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2020/10/29 15:17:04 http: proxy error: context canceled
2020-10-29T15:17:17-07:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2020-10-29T15:17:17-07:00 [Warn] Delaying sending of additional batches by 0 seconds

I'm running a pretty generic Ubuntu 20.04.1 LTS.

wangzlei · 2020-11-02T07:23:46Z

@mousedownmike NoCredentialProviders means AWS SDK cannot find the profile defined in environment variable AWS_PROFILE.
Please check your credential, does it contain [corpdev] ？

For reference, here is a sample credentials file.

[mike]
aws_access_key_id = AKIAZZZZZZZZZZZZ1234
aws_secret_access_key = abcdefghiljekls+JB2AupNJKv9
[corp]
aws_access_key_id = AKIAXXXXXXXXXXX5678
aws_secret_access_key = zyxwvutsrqponm+1CPiUFxYw
[default]
aws_access_key_id = AKIAZZZZZZZZZZZZ1234
aws_secret_access_key = abcdefghiljekls+JB2AupNJKv9

And here is the sample config file to go with that:

[profile mike]
region = us-east-1
[profile corp]
output = json
region = us-east-1
[profile corpdev]
output = json
region = us-east-1
role_arn = arn:aws:iam::123412341234:role/corp-dev-account-role
source_profile = corp
[default]
output = json
region = us-east-1

mousedownmike · 2020-11-02T17:25:09Z

@wangzlei yes, my credentials are set up like that. I was able to verify that the profiles work using the AWS CLI and I use them in my own Go code. I'm going to try building and running the daemon from the source when I have a spare moment. Would you recommend using the 3.2.0 tag to emulate the version that's being distributed?

lupengamzn · 2020-11-02T21:46:29Z

Hey @mousedownmike ,

~~Seems like it's an issue with AWS SDK Go, and we're contacting them for more information.~~

May I know why you're using source_profile and role_arn in config file? Are you trying to grant specific permissions to role_arn using credentials from source_profile? If so, we do provide configuration option for you to do that and you may try adding -r --role-arn when you configure Daemon, see link

mousedownmike · 2020-11-02T22:00:51Z

Lanks @lupengamzn , I have tried the --role-arn option but that still requires that my profile have use of that role. My scenario is brought up by my use of an Organization with multiple accounts. I have been able to remove discreet users from the sub accounts and instead use an assumed role from the "parent" account. It took a while to figure out the details but I have found that I can have a profile that uses my parent account credentials from a (non-default) named key and then assume the role for the sub-account. This way I can choose to use either my parent account profile to access resources in the parent account or I can use the sub-account role via the parent account credentials.

It works great with the cli and with my local Go programs that use the AWS SDK.. The daemon is the only tool where I've had issues which still makes me suspect I'm doing something wrong. When I have a moment I'll try running it from source to see if I can help discover the issue.

wangzlei · 2020-11-02T22:57:21Z

@mousedownmike Have you ever tried AWS_SDK_LOAD_CONFIG=1 before launching xray daemon?

mousedownmike · 2020-11-05T15:22:20Z

When I set AWS_SDK_LOAD_CONFIG=1 the daemon doesn't start and I get the following output:

$ export AWS_SDK_LOAD_CONFIG=1
$ export AWS_PROFILE=corpdev
$ ./xray -l dev -o
2020-11-05T07:18:27-08:00 [Info] Initializing AWS X-Ray daemon 3.2.0
2020-11-05T07:18:27-08:00 [Debug] Listening on UDP 127.0.0.1:2000
2020-11-05T07:18:27-08:00 [Info] Using buffer memory limit of 156 MB
2020-11-05T07:18:27-08:00 [Info] 2496 segment buffers allocated
2020-11-05T07:18:27-08:00 [Debug] Using proxy address: 
2020-11-05T07:18:27-08:00 [Error] Cannot fetch region variable from config file, environment variables and ec2 metadata.

awssandra · 2020-11-05T17:23:31Z

@mousedownmike

We'll take a closer look into this. Look like we're missing debugging logs around the credential resolution.

Thanks!

wangzlei · 2020-11-10T00:26:56Z

@mousedownmike yeah, that means your profile takes effect, but might forget to set export AWS_REGION=us-east-1

mousedownmike · 2020-11-10T00:36:06Z

@wangzlei you are correct, I did not have AWS_REGION set. Here's what I get with it set:

$ export AWS_PROFILE=corpdev  
$ export AWS_REGION=us-east-1                                                                         
$ export AWS_SDK_LOAD_CONFIG=1                                                                        
$ ./xray -l dev -o                                                                          
2020-11-09T16:30:05-08:00 [Info] Initializing AWS X-Ray daemon 3.2.0                                                    
2020-11-09T16:30:05-08:00 [Debug] Listening on UDP 127.0.0.1:2000                                                       
2020-11-09T16:30:05-08:00 [Info] Using buffer memory limit of 156 MB                                                    
2020-11-09T16:30:05-08:00 [Info] 2496 segment buffers allocated                                                         
2020-11-09T16:30:05-08:00 [Debug] Using proxy address:                                                                  
2020-11-09T16:30:05-08:00 [Debug] Fetch region us-east-1 from environment variables                                     
2020-11-09T16:30:05-08:00 [Info] Using region: us-east-1                                                                
2020-11-09T16:30:05-08:00 [Debug] ARN of the AWS resource running the daemon:                                           
2020-11-09T16:30:05-08:00 [Debug] No Metadata set for telemetry records                                                 
2020-11-09T16:30:05-08:00 [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com                                  
2020-11-09T16:30:05-08:00 [Debug] Telemetry initiated                                                                   
2020-11-09T16:30:05-08:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-east-1.amazonaws.com          
2020-11-09T16:30:05-08:00 [Debug] Using Endpoint: https://xray.us-east-1.amazonaws.com                                  
2020-11-09T16:30:05-08:00 [Debug] Batch size: 50                                                                        
2020-11-09T16:30:05-08:00 [Info] Starting proxy http server on 127.0.0.1:2000                                           
2020-11-09T16:31:05-08:00 [Debug] Skipped telemetry data as no segments found                                           
2020-11-09T16:31:09-08:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules                             
2020/11/09 16:31:10 http: proxy error: context canceled                                                                 
2020-11-09T16:31:42-08:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules
2020/11/09 16:31:42 http: proxy error: context canceled

mousedownmike · 2020-12-12T16:30:23Z

I finally found some time to dig into the code here and discovered that getDefaultSession() wasn't enabling the SharedConfigState option. When I change it to:

result, serr := session.NewSessionWithOptions(session.Options{SharedConfigState: session.SharedConfigEnable})

My profile is loaded (assuming I have AWS_PROFILE set to what I want). Additionally, I don't have to set the AWS_REGION environment variable because that is available in the session using:

es := getDefaultSession()
awsRegion = *es.Config.Region
if awsRegion == "" {
	_ = log.Error("Unable to fetch region from session config")
} else {
	log.Debugf("Fetched region %v from session config", awsRegion)
}

These changes are isolated to conn.go so I'll open a PR.

* #74 using SharedConfigState to enable credential profiles and regions set in session config * Using session region if noMetadata is set to true * Reverting import formatting and !noMetadata log.Errorf Co-authored-by: Lei Wang <[email protected]>

joebowbeer · 2021-09-27T16:32:50Z

This works now? Close this issue?

bhautikpip assigned lupengamzn Oct 12, 2020

lupengamzn added the Feature label Oct 19, 2020

wangzlei mentioned this issue Nov 10, 2020

add html anchor tag for go sdk credentials configuration #82

Merged

mousedownmike mentioned this issue Dec 12, 2020

SharedConfigState (work in progress) #86

Merged

srprash assigned wangzlei Dec 15, 2020

mousedownmike closed this as completed Sep 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support credential profiles #74

Support credential profiles #74

mousedownmike commented Oct 12, 2020

lupengamzn commented Oct 19, 2020

mousedownmike commented Oct 19, 2020

mousedownmike commented Oct 20, 2020

lupengamzn commented Oct 26, 2020

vissree commented Oct 28, 2020

mousedownmike commented Oct 29, 2020 •

edited

Loading

wangzlei commented Nov 2, 2020

mousedownmike commented Nov 2, 2020

lupengamzn commented Nov 2, 2020 •

edited

Loading

mousedownmike commented Nov 2, 2020

wangzlei commented Nov 2, 2020

mousedownmike commented Nov 5, 2020

awssandra commented Nov 5, 2020 •

edited

Loading

wangzlei commented Nov 10, 2020

mousedownmike commented Nov 10, 2020

mousedownmike commented Dec 12, 2020 •

edited

Loading

joebowbeer commented Sep 27, 2021

Support credential profiles #74

Support credential profiles #74

Comments

mousedownmike commented Oct 12, 2020

lupengamzn commented Oct 19, 2020

mousedownmike commented Oct 19, 2020

mousedownmike commented Oct 20, 2020

lupengamzn commented Oct 26, 2020

vissree commented Oct 28, 2020

mousedownmike commented Oct 29, 2020 • edited Loading

wangzlei commented Nov 2, 2020

mousedownmike commented Nov 2, 2020

lupengamzn commented Nov 2, 2020 • edited Loading

mousedownmike commented Nov 2, 2020

wangzlei commented Nov 2, 2020

mousedownmike commented Nov 5, 2020

awssandra commented Nov 5, 2020 • edited Loading

wangzlei commented Nov 10, 2020

mousedownmike commented Nov 10, 2020

mousedownmike commented Dec 12, 2020 • edited Loading

joebowbeer commented Sep 27, 2021

mousedownmike commented Oct 29, 2020 •

edited

Loading

lupengamzn commented Nov 2, 2020 •

edited

Loading

awssandra commented Nov 5, 2020 •

edited

Loading

mousedownmike commented Dec 12, 2020 •

edited

Loading