Amazon MSK (provisioned) cluster is Offline.

[githubeto]

Describe

We are building MSK (Managed Streaming for Apache Kafka) and trying to connect to MSK from EKS (Elastic Kubernetes Service) pods using IRSA (IAM Roles for Service Accounts). No authentication method is configured for MSK.
However, the cluster appears offline, and brokers and other elements are not displayed on the screen. Additionally, error logs are being output.

---

Installation details

App Version = v1.0.0
Helm chart version = kafka-ui-1.4.9
Your application config : none.
Any IAAC configs : none.

Steps to reproduce

helm upgrade --install kafbat-ui -f kafka-ui-values.yaml kafbat-ui/kafka-ui

2. port-forward or create virtualservice(istio)

kafka-ui-values.yaml

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
    KAFKA_CLUSTERS_0_NAME: spdkube-stg-cdc-msk
    KAFKA_CLUSTERS_0_READONLY: "true"
    KAFKA_CLUSTERS_0_ZOOKEEPER: z-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'
    LOGGING_LEVEL_ROOT: info
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi

IAM role (arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui) policy.

{
    "Statement": [
        {
            "Action": [
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:Connect"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>",
            "Sid": "VisualEditor0"
        },
        {
            "Action": [
                "kafka-cluster:DeleteGroup",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopicDynamicConfiguration",
                "kafka-cluster:AlterTopicDynamicConfiguration",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:AlterClusterDynamicConfiguration",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:DescribeClusterDynamicConfiguration",
                "kafka-cluster:Connect",
                "kafka-cluster:DeleteTopic",
                "kafka-cluster:WriteData"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor1"
        },
        {
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor2"
        }
    ],
    "Version": "2012-10-17"
}

Logs

2024-11-22 11:24:48,175 ERROR [parallel-2] c.p.k.u.s.StatisticsService: Failed to collect cluster spdkube-stg-cdc-msk info
java.lang.IllegalStateException: Error while creating AdminClient for Cluster spdkube-stg-cdc-msk
	at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)
	at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:265)
	at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1886)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:340)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onError(MonoCacheTime.java:363)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
	at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

[githubeto]

I also tried this pattern.

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::735099427143:role/odc-stg-kafka-ui
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
    KAFKA_CLUSTERS_0_NAME: spdkube-stg-cdc-msk
#    KAFKA_CLUSTERS_0_READONLY: "true"
#    KAFKA_CLUSTERS_0_ZOOKEEPER: z-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181
#    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'
    LOGGING_LEVEL_ROOT: info
yamlApplicationConfig:
  kafka:
    clusters:
      - name: yaml
        bootstrapServers: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
        properties:
          security.protocol: SASL_SSL
          sasl.mechanism: AWS_MSK_IAM
          sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
          sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi

[Haarolean]

Timed out waiting for a node assignment

this is 99% IAM/firewall/access issue.
Please see this, might help.