[#222] Support Kafka based messaging in Hono. #237

b-abel · 2021-05-18T14:17:23Z

This fixes #222. It adds the support for Kafka-based messaging to the Hono chart. The property messagingNetworkType is used to select which type of configuration is added to the adapters and services (either amqp or kafka). The property kafkaMessagingClusterExample.enabled indicates whether the example Kafka cluster should be deployed and used.

b-abel · 2021-05-18T14:20:13Z

To be added:

TLS configuration
authentication against Kafka
a bit more documentation

b-abel · 2021-05-18T14:28:49Z

Replaced by updated version below.

b-abel · 2021-05-19T09:53:00Z

The chart used for the deployment of Kafka requires Helm version 3.1.0. With a lower version, the deployment might not work when the property kafkaMessagingClusterExample.enabled is set to true.

b-abel · 2021-05-19T09:55:10Z

I assume the currently failing lint check is caused by #239. This should be resolved with #240.

calohmn · 2021-05-19T12:16:11Z

@b-abel Can you rebase and update the chart version?

b-abel · 2021-05-19T12:21:13Z

Rebased.

b-abel · 2021-05-25T15:05:02Z

This includes now authentication and transport encryption.

b-abel · 2021-05-25T15:08:12Z

How to use Kafka-based Messaging with Authentication and TLS

Deploy to minikube

Deploy from the current branch:

# in directory packages/charts/hono
helm install --dependency-update -n hono --values ./profileKafkaMessaging.yaml eclipse-hono .

Send events

Download Hono's Command Line Client.
Change the path to the packages project in the following script block and run it to start a receiver:

# in directory where the hono-cli-*-exec.jar file has been downloaded to
export PATH_TO_PACKAGES=<path/to/the/packages/project>
export MY_TENANT=DEFAULT_TENANT
export KAFKA_IP=$(kubectl get service eclipse-hono-kafka-0-external --output="jsonpath={.status.loadBalancer.ingress[0]['hostname','ip']}" -n hono)
java -jar hono-cli-*-exec.jar --tenant.id=$MY_TENANT --spring.profiles.active=receiver,kafka --hono.kafka.commonClientConfig.bootstrap.servers=$KAFKA_IP:9094 --hono.kafka.commonClientConfig.security.protocol=SASL_SSL --hono.kafka.commonClientConfig.sasl.mechanism=SCRAM-SHA-512 --hono.kafka.commonClientConfig.sasl.jaas.config="org.apache.kafka.common.security.scram.ScramLoginModule required username=\"hono\" password=\"hono-secret\";" --hono.kafka.commonClientConfig.ssl.endpoint.identification.algorithm="" --hono.kafka.commonClientConfig.ssl.truststore.location=$PATH_TO_PACKAGES/charts/hono/example/certs/trustStore.jks --hono.kafka.commonClientConfig.ssl.truststore.password=honotrust

Send the event:

export MY_TENANT=DEFAULT_TENANT
export MY_DEVICE=sensor1
export MY_PWD=hono-secret
export HTTP_ADAPTER_IP=$(kubectl get service eclipse-hono-adapter-http-vertx --output="jsonpath={.status.loadBalancer.ingress[0]['hostname','ip']}" -n hono)
curl -i -u $MY_DEVICE@$MY_TENANT:$MY_PWD -H 'Content-Type: application/json' --data-binary '{"alarm": "fire"}' http://$HTTP_ADAPTER_IP:8080/event

Send and receive Commands

Start a simulated devices:

export MY_TENANT=DEFAULT_TENANT
export MY_DEVICE=sensor1
export MY_PWD=hono-secret
export MQTT_ADAPTER_IP=$(kubectl get service eclipse-hono-adapter-mqtt-vertx --output="jsonpath={.status.loadBalancer.ingress[0]['hostname','ip']}" -n hono)
mosquitto_sub -v -h $MQTT_ADAPTER_IP -u $MY_DEVICE@$MY_TENANT -P $MY_PWD -t command/+/+/req/#

Change the path to the packages project in the following script block and run it to send a command with Hono's CLI:

# in directory where the hono-cli-*-exec.jar file has been downloaded to
export PATH_TO_PACKAGES=<path/to/the/packages/project>
export MY_TENANT=DEFAULT_TENANT
export MY_DEVICE=4711
export KAFKA_IP=$(kubectl get service eclipse-hono-kafka-0-external --output="jsonpath={.status.loadBalancer.ingress[0]['hostname','ip']}" -n hono)
java -jar hono-cli-*-exec.jar --tenant.id=$MY_TENANT --device.id=$MY_DEVICE --spring.profiles.active=command,kafka --hono.kafka.commonClientConfig.bootstrap.servers=$KAFKA_IP:9094 --hono.kafka.commonClientConfig.security.protocol=SASL_SSL --hono.kafka.commonClientConfig.sasl.mechanism=SCRAM-SHA-512 --hono.kafka.commonClientConfig.sasl.jaas.config="org.apache.kafka.common.security.scram.ScramLoginModule required username=\"hono\" password=\"hono-secret\";" --hono.kafka.commonClientConfig.ssl.endpoint.identification.algorithm="" --hono.kafka.commonClientConfig.ssl.truststore.location=$PATH_TO_PACKAGES/charts/hono/example/certs/trustStore.jks --hono.kafka.commonClientConfig.ssl.truststore.password=honotrust

Connect external tools to the Kafka cluster

The following example shows how the Kafka CLI can be used as a northbound receiver.

First, download Kafka and extract it like this:

tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0

Change the path to the packages project in the following script block and run it to create a client config file:

export PATH_TO_PACKAGES=<path/to/the/packages/project>
cat <<EOS >> hono-client-sasl-ssl.config
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="hono" \
  password="hono-secret";
# disable hostname verification because the certificate does not contain load balancer IPs
ssl.endpoint.identification.algorithm=
ssl.truststore.location=$PATH_TO_PACKAGES/charts/hono/example/certs/trustStore.jks
ssl.truststore.password=honotrust
EOS

Now, start the console consumer on the events topic of the tenant:

export MY_TENANT=DEFAULT_TENANT
export MY_DEVICE=4711
export TOPIC=hono.event.$MY_TENANT
export KAFKA_IP=$(kubectl get service eclipse-hono-kafka-0-external --output="jsonpath={.status.loadBalancer.ingress[0]['hostname','ip']}" -n hono)
bin/kafka-console-consumer.sh --topic $TOPIC --bootstrap-server $KAFKA_IP:9094 --consumer.config ./hono-client-sasl-ssl.config

To list all topics in Kafka, run:

bin/kafka-topics.sh --bootstrap-server $KAFKA_IP:9094 --list --command-config ./hono-client-sasl-ssl.config

b-abel · 2021-05-26T11:35:48Z

Regarding the documentation I think it would be best to have the "How to..." part from above integrated somehow into Hono's Getting Started instead of adding more documentation here.

b-abel · 2021-05-27T11:37:44Z

When deploying the dispatch router, there is one port for TLS and one for plaintext connections. The Kafka cluster is now configured so that clients can only connect with TLS. At least with Bitnami's Helm chart, I haven't seen an easy way to provide both options at the same time. Instead of adding that manually, I tend to leave the configuration for TLS and rather document how to set up connections from the client with the truststore (I would add a new Getting Started guide for Kafka-based messaging to Hono's documentation).
A third option would be to create two different deployment profiles, each deploying different configurations of Kafka: one with TLS, the other without. WDYT?

b-abel · 2021-05-31T16:07:27Z

I tend to leave the configuration for TLS and rather document how to set up connections from the client with the truststore (I would add a new Getting Started guide for Kafka-based messaging to Hono's documentation).

The PR eclipse-hono/hono#2682 adds a Getting Started guide for Kafka-based messaging to Hono. It shows the required configuration of the Hono CLI to connect to the Kafka broker if it is configured for TLS only.

calohmn · 2021-06-01T11:01:35Z

When using minikube, minikube tunnel has to be running when installing the chart.

sophokles73 · 2021-06-01T11:42:32Z

When using minikube, minikube tunnel has to be running when installing the chart.

The chart itself can also use Node Ports instead of using LoadBalancer. We explicitly added the useLoadBalancer property for that purpose. I therefore would prefer, if we could also use kafka with Node Ports only (based on the value of the property), if possible.

b-abel · 2021-06-01T14:58:39Z

We explicitly added the useLoadBalancer property for that purpose. I therefore would prefer, if we could also use kafka with Node Ports only (based on the value of the property), if possible.

It is quite easy to configure the Bitnami Kafka chart to use Node Ports instead of LoadBalancer. What seems to be problematic is changing this depending on the useLoadBalancer property. The reason is that the Bitnami chart is configured in the file hono/values.yaml. Helm does not support templating in value.yaml files (for details see for example helm/helm#2492 and helm/helm#2133).
Unless someone comes up with a better workaround, the only way I currently see is to set the property kafka.externalAccess.service.type together with useLoadBalancer.

charts/hono/example/create_certs.sh

charts/hono/templates/_helpers.tpl

charts/hono/README.md

charts/hono/values.yaml

calohmn · 2021-06-01T17:31:41Z

charts/hono/example/ca_opts

+subjectKeyIdentifier = hash
+keyUsage             = keyAgreement,keyEncipherment,digitalSignature
+extendedKeyUsage     = serverAuth, clientAuth
+subjectAltName       = DNS.1:*.eclipse-hono-kafka-headless,DNS.2:*.eclipse-hono-kafka-headless.hono,DNS.3:*.eclipse-hono-kafka-headless.hono.svc.cluster.local,DNS.4:localhost


Is the *. prefix needed here?

I think so. The idea here is that clients directly address a pod using the headless service. The address is then <release-name>-0.<release-name>-headless.<namespace>.svc.cluster.local. The wildcard matches the pod name (would even work for multiple brokers).
But looking at this, I found another problem: the release names are part of the service names, and they are determined only during deployment. I created an issue for this: #252.

b-abel · 2021-06-03T08:09:40Z

Unless someone comes up with a better workaround, the only way I currently see is to set the property kafka.externalAccess.service.type together with useLoadBalancer.

@sophokles73 I added a check that lets the deployment fail if those two properties are not consistent with each other.

calohmn · 2021-06-03T13:09:47Z

charts/hono/README.md

+The following command provides a quickstart for Kafka based messaging:
+
+```bash
+helm install --dependency-update -n hono --set messagingNetworkType=kafka --set kafkaMessagingClusterExample.enabled=true --set amqpMessagingNetworkExample.enabled=false  eclipse-hono eclipse-iot/hono


How about adding a note here, that minikube tunnel needs to be running if using minikube.
I can imagine this being a common mistake when using minikube, with the install command running into a timeout in that case.

Done (and rebased).

This adds the support for the Kafka based messaging to the Hono chart. The property `messagingNetworkType` is used to select which type of configuration is added to the adapters and services (either `amqp` or `kafka`). The property `kafkaMessagingClusterExample.enabled` indicates whether the example Kafka cluster should be deployed and used. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

New certificates are created for the example Kafka broker. The broker is configured to expect TLS encrypted connections from clients. The truststore is added to all services and adapters that need to connect to Kafka. This is currently a "jks" file because in Kafka the support for the "pem" format was added in version 2.7 and Hono currently uses the Kafka clients in version 2.6. Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

calohmn

LGTM

sophokles73 · 2021-06-07T07:25:38Z

charts/hono/templates/hono-adapter-amqp/hono-adapter-amqp-vertx-secret.yaml

what do we need this for? We already configure the trusted-certs.pem which contains all the trusted certificates ....

Kafka does not support pem files before version 2.7. Hono uses the Kafka clients in version 2.6.

But Hono also supports JKS files so FMPOV we should either use JKS or PEM files, but not both ...

I can provide a PR that replaces the trusted-certs.pem with a JKS file if you like.

with a JKS file? My understanding is that we should be able to use only the truststore.jks file or does that contain other certs than trusted-certs.pem?

Right, trustStore.jks is the JKS file to be used everywhere instead of trusted-certs.pem.

I think you should indeed change it that way.

@sophokles73 Please have a look at #257.

Kafka does not support pem files before version 2.7. Hono uses the Kafka clients in version 2.6.

Is there a particular reason why we do not simply use the 2.7.x client and continue to use our PEM files?

Kafka does not support pem files before version 2.7. Hono uses the Kafka clients in version 2.6.

Is there a particular reason why we do not simply use the 2.7.x client and continue to use our PEM files?

Hono does not directly use the Kafka clients, but they are a transitive dependency of the Vert.x Kafka client. Currently, it is version 3.9.7 of vertx-kafka-client which depends on version 2.6.0 of the Kafka clients.

Another minor point might be client/broker compatibility. Kafka version 2.7 is pretty recent. While I find it hard to find general reliable information on that topic, it seems advisable to use brokers with an equal or higher version than the clients. So, this would potentially increase the minimal Kafka version required by Hono.

b-abel force-pushed the PR/#222_support_kafka_messaging branch from 1e16c9a to 5919c80 Compare May 19, 2021 06:47

b-abel force-pushed the PR/#222_support_kafka_messaging branch from 5919c80 to b62f797 Compare May 19, 2021 12:20

b-abel force-pushed the PR/#222_support_kafka_messaging branch from b62f797 to ac4fa72 Compare May 25, 2021 15:04

b-abel force-pushed the PR/#222_support_kafka_messaging branch 2 times, most recently from eb1fa73 to 4cd1115 Compare May 26, 2021 09:03

b-abel marked this pull request as ready for review May 26, 2021 11:35

b-abel requested review from calohmn and ctron as code owners May 26, 2021 11:35

b-abel mentioned this pull request May 28, 2021

[#2601] Add Getting Started guide with Kafka based messaging. eclipse-hono/hono#2682

Merged

calohmn added the Hono label Jun 1, 2021

b-abel force-pushed the PR/#222_support_kafka_messaging branch from 4cd1115 to 63d0169 Compare June 1, 2021 10:05

calohmn reviewed Jun 1, 2021

View reviewed changes

b-abel force-pushed the PR/#222_support_kafka_messaging branch from c62cd90 to 6289a8c Compare June 3, 2021 06:20

This was referenced Jun 3, 2021

[Hono] Add link to "Getting Started with Kafka" to NOTES.txt #253

Closed

[#253] Update NOTES.txt with information related to Kafka. #254

Merged

calohmn reviewed Jun 3, 2021

View reviewed changes

b-abel force-pushed the PR/#222_support_kafka_messaging branch from 07735a7 to d08ddaa Compare June 4, 2021 08:01

b-abel added 2 commits June 4, 2021 10:03

[eclipse#222] Add authentication for Kafka clients.

6b29136

Signed-off-by: Abel Buechner-Mihaljevic <[email protected]>

b-abel force-pushed the PR/#222_support_kafka_messaging branch from d08ddaa to acb227f Compare June 4, 2021 08:04

calohmn approved these changes Jun 4, 2021

View reviewed changes

calohmn merged commit 8513546 into eclipse:master Jun 4, 2021

sophokles73 reviewed Jun 7, 2021

View reviewed changes

This was referenced Jun 7, 2021

[Hono] Replace usage of pem truststore by JKS #257

Closed

how to assign parition id in dispatch router eclipse-hono/hono#2707

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#222] Support Kafka based messaging in Hono. #237

[#222] Support Kafka based messaging in Hono. #237

b-abel commented May 18, 2021

b-abel commented May 18, 2021 •

edited

Loading

b-abel commented May 18, 2021 •

edited

Loading

b-abel commented May 19, 2021

b-abel commented May 19, 2021

calohmn commented May 19, 2021

b-abel commented May 19, 2021

b-abel commented May 25, 2021

b-abel commented May 25, 2021 •

edited

Loading

b-abel commented May 26, 2021

b-abel commented May 27, 2021 •

edited

Loading

b-abel commented May 31, 2021

calohmn commented Jun 1, 2021

sophokles73 commented Jun 1, 2021 •

edited

Loading

b-abel commented Jun 1, 2021

calohmn Jun 1, 2021

b-abel Jun 2, 2021

b-abel commented Jun 3, 2021

calohmn Jun 3, 2021

b-abel Jun 4, 2021

calohmn left a comment

sophokles73 Jun 7, 2021

b-abel Jun 7, 2021

sophokles73 Jun 7, 2021

b-abel Jun 7, 2021

sophokles73 Jun 7, 2021

b-abel Jun 7, 2021

sophokles73 Jun 7, 2021

b-abel Jun 7, 2021

sophokles73 Jun 7, 2021

b-abel Jun 7, 2021

[#222] Support Kafka based messaging in Hono. #237

[#222] Support Kafka based messaging in Hono. #237

Conversation

b-abel commented May 18, 2021

b-abel commented May 18, 2021 • edited Loading

b-abel commented May 18, 2021 • edited Loading

b-abel commented May 19, 2021

b-abel commented May 19, 2021

calohmn commented May 19, 2021

b-abel commented May 19, 2021

b-abel commented May 25, 2021

b-abel commented May 25, 2021 • edited Loading

How to use Kafka-based Messaging with Authentication and TLS

Deploy to minikube

Send events

Send and receive Commands

Connect external tools to the Kafka cluster

b-abel commented May 26, 2021

b-abel commented May 27, 2021 • edited Loading

b-abel commented May 31, 2021

calohmn commented Jun 1, 2021

sophokles73 commented Jun 1, 2021 • edited Loading

b-abel commented Jun 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-abel commented Jun 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

calohmn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b-abel commented May 18, 2021 •

edited

Loading

b-abel commented May 18, 2021 •

edited

Loading

b-abel commented May 25, 2021 •

edited

Loading

b-abel commented May 27, 2021 •

edited

Loading

sophokles73 commented Jun 1, 2021 •

edited

Loading