Application in pod not picking up role from service account

[anshumanr]

Describe the bug

My application (a k8s controller in Go) is running in a pod in AWS EKS. Some information about cluster & AWS Go SDK.

EKS version: 1.21
eksctl version: 0.77.0
AWS Go SDK verion: v1.44.28
Deploying using kubectl

I created a service account using eksctl as show below.

eksctl create iamserviceaccount --cluster ${EKS_CLUSTER_NAME} \
	--namespace tel \
	--name tel-controller-serviceaccount \
	--attach-policy-arn arn:aws:iam::xxxxxxxxx:policy/telcontrollerRoute53Policy \
	--override-existing-serviceaccounts --approve

The above role has a policy enabling to execute the route53:ChangeResourceRecordSets action.
In addition to the above, the above service account also has some K8s role & rolebinding attached to it during deployment.

The created service account

kubectl get sa/tel-controller-serviceaccount -n tel -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/eksctl-eks-tel-addon-iamserviceaccount-tel-t-Role1-1XSKAF2YBFFUU
  creationTimestamp: "2022-06-10T17:56:32Z"
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: tel-controller-serviceaccount
  namespace: tel
  resourceVersion: "2103"
  uid: ae6d4266-bf8f-4f78-901c-d7363cf406d1
secrets:
- name: tel-controller-serviceaccount-token-dc8gr

Describing the pod, shows me the necessary env variables & mounts.

Environment:
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::xxxx:role/eksctl-eks-tel-addon-iamserviceaccount-tel-t-Role1-1XSKAF2YBFFUU
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mwpzr (ro)

In addition to the above, for debugging, I added code to read the content of the above token file (harded above file name in code) & also get the caller identity (GetCallerIdentity()). I was able the read the token file from above location, so it is not a access problem. Decoded JWT token shows this

{
"aud": [
"sts.amazonaws.com"
],
"exp": 1654970669,
"iat": 1654884269,
"iss": "https://oidc.eks.us-east-1.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxx",
"kubernetes.io": {
"namespace": "tel",
"pod": {
"name": "tel-controller-64747b9b67-8gr88",
"uid": "39b0d09b-e187-4ae0-8ff0-8b66b48e7027"
},
"serviceaccount": {
"name": "tel-controller-serviceaccount",
"uid": "ae6d4266-bf8f-4f78-901c-d7363cf406d1"
}
},
"nbf": 1654884269,
"sub": "system:serviceaccount:tel:tel-controller-serviceaccount"
}

It looks correct to me.

Expected Behavior

Calling GetCallerIdentity() should have returned role set to the one set in AWS_ROLE_ARN - the IAM role attached to service account.

Current Behavior

Calling GetCallerIdentity() returns -

{
Account: "xxxxxxxxx",
Arn: "arn:aws:sts::xxxxxxxxx:assumed-role/eksctl-eks-tel-nodegroup-voice-NodeInstanceRole-1UTZ668NI10S8/i-0cdb286a5811595a7",
UserId: "AROAZUYK7F2GVF6F64MH7:i-0cdb286a5811595a7"
}

which is the nodegroup role of the EKS cluster.

Reproduction Steps

• create an IAM policy with some permissions. For my application it is named telcontrollerRoute53Policy and contains below permissions

{  
    "Version": "2012-10-17",  
    "Statement": [  
        {  
            "Sid": "VisualEditor0",  
            "Effect": "Allow",  
            "Action": "route53:ChangeResourceRecordSets",  
            "Resource": "*"  
        }  
    ]  
}  

• create service account. I used eksctl to do so

eksctl create iamserviceaccount --cluster ${EKS_CLUSTER_NAME}
--namespace tel
--name tel-controller-serviceaccount
--attach-policy-arn arn:aws:iam::xxxxxxxxx:policy/telcontrollerRoute53Policy
--override-existing-serviceaccounts --approve

• my cluster role & cluster role binding for attached to the above service account.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tel-controller-role
  labels:
    app: tel-controller
rules:
- apiGroups: [""]
  resources: [events]
  verbs: [create, delete, get, list, update, watch]
- apiGroups: ["networking.k8s.io"]
  resources: [ingressclasses]
  verbs: [get, list]
- apiGroups: ["", "networking.k8s.io"]
  resources: [services, ingresses]
  verbs: [create, get, list, patch, update, delete, watch]
- apiGroups: [""]
  resources: [configmaps]
  verbs: [create, delete, get, update]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: [get, create, update]
- apiGroups: [""]
  resources: [pods]
  verbs: [get, list, watch, update]
- apiGroups: ["", "networking.k8s.io"]
  resources: [services/status, ingresses/status]
  verbs: [update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tel-controller-rolebinding
  labels:
    app: tel-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tel-controller-role
subjects:
- kind: ServiceAccount
  name: tel-controller-serviceaccount
  namespace: tel

SSCCE code including rbac.yaml & deployment.yaml at https://github.com/anshumanr/sa

Possible Solution

No response

Additional Information/Context

No response

SDK version used

v1.44.28

Environment details (Version of Go (go version)? OS name and version, etc.)

go version go1.17.8 linux/amd64

[anshumanr]

I am not exactly sure why but using session.NewSession instead of session.New (which is deprecated anyway) seems to have solved this problem.

re-opened for comments, if any

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[anshumanr]

Confusion stemmed from the fact that the examples in this Route 53 doc use the old deprecated method (session.New). Maybe somebody at AWS could fix it.

[RanVaknin]

Hi there,

Thanks for the feedback. We are no longer updating docs for GO SDK v1, since this seems to have been resolved with the use of .newSession(), I feel like this is safe to close.

Thanks again for your feedback.
Ran~

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.