Notification service Kafka error #1030

gohilamariappan · 2024-06-18T04:57:45Z

gohilamariappan
Jun 18, 2024

Hi Team!

We are trying to configure rc setup using docker compose and kubernetes. In both cases we are getting the kafka connection error in the notification service as below...
Can you please suggest.

##########
time="2024-06-05T06:11:51Z" level=error msg="Error in committing message localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)"
%3|1717567941.481|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
time="2024-06-05T06:12:21Z" level=error msg="Error in committing message localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)"
Consumer error: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
%3|1717567971.484|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
time="2024-06-05T06:12:51Z" level=error msg="Error in committing message localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)"
Consumer error: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 30 identical error(s) suppressed)
##########

Regards,
Gohila

thillaibirla · 2024-06-19T03:52:47Z

thillaibirla
Jun 19, 2024

@surendrasinghs - can a devops or a engineering team member help us with this issue?

0 replies

holashchand · 2024-06-19T09:43:39Z

holashchand
Jun 19, 2024
Collaborator

Hi @gohilamariappan, Seems like kafka bootstrap server is not configured. Please follow the below instructions to set it -

Configure KAFKA_BOOTSTRAP_SERVERS env for notification-service in configmap like link
Set the kafka url at values

And try again. Let me know (tag me) if it still not resolved, thanks

0 replies

gohilamariappan · 2024-06-20T05:40:15Z

gohilamariappan
Jun 20, 2024
Author

Hi @holashchand !

It seems like already we have already configured env variable KAFKA_BOOTSTRAP_SERVERS.
Now i tried the configuration as below but still connection issue.
Configuration:

containers:

name: notificationms-app
image: ghcr.io/sunbird-rc/sunbird-rc-notification-service
imagePullPolicy: IfNotPresent
memoryRequest: "250Mi"
cpuRequest: "100m"
memoryLimit: "500Mi"
cpuLimit: "200m"
containerPorts: [8765]
envVars:
- name: TRACK_NOTIFICATIONS
  value: false
- name: KAFKA_BOOTSTRAP_SERVERS
  value: kafkaapp-svc:9092
- name: kafka_bootstrap_address
  value: kafkaapp-svc:9092

ERROR:

time="2024-06-20T05:34:56Z" level=info msg="Starting certificate processor"
%3|1718861696.597|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
Consumer error: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
time="2024-06-20T05:34:56Z" level=error msg="Error in committing message GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)"
2024/06/20 05:34:56 Serving notification service at http://[::]:8765
%3|1718861696.730|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
Consumer error: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
time="2024-06-20T05:34:56Z" level=error msg="Error in committing message GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)"

###############

Please suggest.

Regards,
Gohila

1 reply

holashchand Jun 20, 2024
Collaborator

Hi @gohilamariappan , it seems it is still not able to read the env as it is showing localhost:9092 in the error
please change the keyword envVars to env and try again

gohilamariappan · 2024-06-20T08:19:36Z

gohilamariappan
Jun 20, 2024
Author

Hi @holashchand !

It seems env variables are coming in the deployment, becoz am able to see this variables inside a container.

/app/notification-service # env | grep kafka
kafka_bootstrap_address=kafkaapp-svc:9092
KAFKA_BOOTSTRAP_SERVERS=kafkaapp-svc:9092
###########
Now I have updated kafka service KAFKA_ADVERTISED_LISTENERS env variable with service name. Getting the below error.
########
time="2024-06-20T08:06:50Z" level=info msg="Starting certificate processor"
Consumer error: Subscribed topic not available: notify: Broker: Unknown topic or partition
time="2024-06-20T08:06:50Z" level=error msg="Error in committing message Subscribed topic not available: notify: Broker: Unknown topic or partition"
2024/06/20 08:06:50 Serving notification service at http://[::]:8765
##############

Please suggest.

Regards,
Gohila

1 reply

holashchand Jun 20, 2024
Collaborator

Hi @gohilamariappan, The topic is not existing, so it can be created with default name notify (configurable) manually or restart registry with notification_enabled and notification_async_enabled as true to automatically create the topic.

gohilamariappan · 2024-06-20T10:02:47Z

gohilamariappan
Jun 20, 2024
Author

Thank you! @holashchand

Tried registry notification conf changes, but that didn't worked(topics are not created automatically). Now i have created topic manually inside a container, it seems running.
Can you please share the list of topics which needs to be created for sunbird-rc setup.

One more suggestion needed in the below item.

Registry service is giving the below error. couldn't find which property its looking for

###########
09:55:11.450 [main] INFO d.s.registry.util.EntityParenter:242 - Adding index to shard: shard1 for definition: RevokedCredential
09:55:11.451 [main] INFO d.s.registry.sink.SqlgProvider:95 - Could not create unique index for empty properties
09:55:11.533 [main] ERROR d.s.registry.util.EntityParenter:282 - Failed Transaction creating index RevokedCredential: java.lang.RuntimeException: Property not found
at dev.sunbirdrc.registry.sink.SqlgProvider.lambda$createIndexOnVertex$0(SqlgProvider.java:127)
at java.util.Optional.orElseThrow(Optional.java:290)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndexOnVertex(SqlgProvider.java:127)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndexByIndexType(SqlgProvider.java:113)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndex(SqlgProvider.java:65)
at dev.sunbirdrc.registry.util.Indexer.createIndex(Indexer.java:78)
at dev.sunbirdrc.registry.util.EntityParenter.asyncAddIndex(EntityParenter.java:275)
at dev.sunbirdrc.registry.util.EntityParenter.ensureIndexExists(EntityParenter.java:243)
at dev.sunbirdrc.registry.util.EntityParenter.lambda$null$3(EntityParenter.java:222)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at dev.sunbirdrc.registry.util.EntityParenter.lambda$ensureIndexExists$4(EntityParenter.java:219)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at dev.sunbirdrc.registry.util.EntityParenter.ensureIndexExists(EntityParenter.java:214)
at dev.sunbirdrc.registry.util.EntityParenter$$FastClassBySpringCGLIB$$8750b526.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at dev.sunbirdrc.registry.util.EntityParenter$$EnhancerBySpringCGLIB$$90c59eb9.ensureIndexExists()
at dev.sunbirdrc.registry.app.AppStartupRunner.run(AppStartupRunner.java:37)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:783)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:773)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319)
at dev.sunbirdrc.registry.app.SunbirdRCApplication.main(SunbirdRCApplication.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

########
Please suggest.

Regards,
Gohila

1 reply

holashchand Jun 20, 2024
Collaborator

Hi @gohilamariappan's , Great to know that it's deployed successfully now! All the topics depends of the feature we use.

When notifications enabled, environment variable name for topic and default topic name are below
notification_topic:notify

When Async enabled to create entities asynchronously using async_enabled
kafka_create_entity_topic: create_entity
kafka_post_create_entity_topic:post_create_entity

When events enabled - can be used for custom services or sunbird metrics service
event_topic:events

gohilamariappan · 2024-06-24T05:42:41Z

gohilamariappan
Jun 24, 2024
Author

@holashchand Thank you so much!
Created events topic. Now Metrics service also running fine.
can u please update on the registry service error.
###########
09:55:11.450 [main] INFO d.s.registry.util.EntityParenter:242 - Adding index to shard: shard1 for definition: RevokedCredential
09:55:11.451 [main] INFO d.s.registry.sink.SqlgProvider:95 - Could not create unique index for empty properties
09:55:11.533 [main] ERROR d.s.registry.util.EntityParenter:282 - Failed Transaction creating index RevokedCredential: java.lang.RuntimeException: Property not found
at dev.sunbirdrc.registry.sink.SqlgProvider.lambda$createIndexOnVertex$0(SqlgProvider.java:127)
at java.util.Optional.orElseThrow(Optional.java:290)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndexOnVertex(SqlgProvider.java:127)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndexByIndexType(SqlgProvider.java:113)
at dev.sunbirdrc.registry.sink.SqlgProvider.createIndex(SqlgProvider.java:65)
at dev.sunbirdrc.registry.util.Indexer.createIndex(Indexer.java:78)
at dev.sunbirdrc.registry.util.EntityParenter.asyncAddIndex(EntityParenter.java:275)
at dev.sunbirdrc.registry.util.EntityParenter.ensureIndexExists(EntityParenter.java:243)
at dev.sunbirdrc.registry.util.EntityParenter.lambda$null$3(EntityParenter.java:222)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at dev.sunbirdrc.registry.util.EntityParenter.lambda$ensureIndexExists$4(EntityParenter.java:219)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at dev.sunbirdrc.registry.util.EntityParenter.ensureIndexExists(EntityParenter.java:214)
at dev.sunbirdrc.registry.util.EntityParenter$$FastClassBySpringCGLIB$$8750b526.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at dev.sunbirdrc.registry.util.EntityParenter$$EnhancerBySpringCGLIB$$90c59eb9.ensureIndexExists()
at dev.sunbirdrc.registry.app.AppStartupRunner.run(AppStartupRunner.java:37)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:783)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:773)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319)
Regards,
Gohila

1 reply

holashchand Jun 24, 2024
Collaborator

Hi @gohilamariappan, This is a registry startup error, it is resolved in the latest release, but needs to be updated the same in helms.
Latest release tag : v2.0.1
Although the above error should not affect much but if you are willing to update to the latest version, please follow the below steps -

Update the image tag registry service helm to v2.0.1.
Confirm if you have v2.0.1 for the desired service or not here https://github.com/orgs/Sunbird-RC/packages
Only these services are part of v2.0.1 release, registry claim-ms credential-schema credentials-service encryption-service identity-service id-gen-service metrics notification-service
And update similarly for respective services

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sunbird Registries & Credentials

Notification service Kafka error #1030

{{title}}

Replies: 6 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Sunbird Registries & Credentials

Notification service Kafka error #1030

gohilamariappan Jun 18, 2024

Replies: 6 comments · 4 replies

thillaibirla Jun 19, 2024

holashchand Jun 19, 2024 Collaborator

gohilamariappan Jun 20, 2024 Author

holashchand Jun 20, 2024 Collaborator

gohilamariappan Jun 20, 2024 Author

holashchand Jun 20, 2024 Collaborator

gohilamariappan Jun 20, 2024 Author

holashchand Jun 20, 2024 Collaborator

gohilamariappan Jun 24, 2024 Author

holashchand Jun 24, 2024 Collaborator

gohilamariappan
Jun 18, 2024

Replies: 6 comments 4 replies

thillaibirla
Jun 19, 2024

holashchand
Jun 19, 2024
Collaborator

gohilamariappan
Jun 20, 2024
Author

holashchand Jun 20, 2024
Collaborator

gohilamariappan
Jun 20, 2024
Author

holashchand Jun 20, 2024
Collaborator

gohilamariappan
Jun 20, 2024
Author

holashchand Jun 20, 2024
Collaborator

gohilamariappan
Jun 24, 2024
Author

holashchand Jun 24, 2024
Collaborator