MSK: Pod Identity support

[rajarshp]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running main-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Hi Team,

Earlier I reported this issue - #287.
Though we didnt get a chance t use MSK after that, but now I have deployed akfka ui in AWS Rosa but it seems the pod identity issue is not resolved.

I can still see it is giving the same error

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) : [AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]): Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]) : [EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 81e6f31f-2ca4-XXXXXXXacca), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.]] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130) at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:175) at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.resolveCredentials(MSKCredentialProvider.java:162) at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:99) at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77) at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139) at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)

Expected behavior

It should able to connect MSK using Multi vpc cross account IAM role

Your installation details

Pull latest image
created a helm chat
deployed it in out Rosa env

Steps to reproduce

Create MSK
Enable Multi vpc for IAM
update cluster policy in MSK and Client end (Rosa)
Pull latest image
created a helm chat and provided MSK details
deployed it in AWS Rosa env

Screenshots

NA

Logs

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) : [AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]): Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]) : [EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 81e6f31f-2ca4-40f6-XXXXXXX69f9acca), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.]] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130) at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:175) at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.resolveCredentials(MSKCredentialProvider.java:162) at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:99) at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77) at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139) at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)

Additional context

NA

[Haarolean]

I don't really grasp what has to be done on our side here

[rajarshp]

latest aws-iam-auth jar has to be consumed and a new build need to be released. If this is not done yet.

[rajarshp]

Hi @Haarolean
Can you please release new version after updating the aws-msk-iam-auth jar. I fixed this error but now it seems affected by - aws/aws-msk-iam-auth#174

[Haarolean]

@rajarshp please verify, the build/image will be available shortly

[rajarshp]

@Haarolean let me know once the new image is pushed - I did pull the latest from ghcr.io/kafbat/kafka-ui:latest but issue is still there

[Haarolean]

latest is for the latest release.

https://github.com/kafbat/kafka-ui/pkgs/container/kafka-ui/291088073?tag=7be3325170d373ea882129a079c966951a2bb603

[rajarshp]

tried using this image, but it didn't solve the issue. It is still throwing the same error.

[rajarshp]

this issue is resolved

[taer]

Perfect timing. I just needed this. :)