quickstart-kafka-connect

Quick Start for Kafka Connect

This folder contains the resources required to build a version of the Quick Start app based on the Kafka Connect Lightstreamer Sink Connector plugin.

The diagram above illustrates how, in this setup, a stream of simulated market events is channeled from Kafka to the web client via the connector running on Kafka Connect and Lightstreamer Server.

As with the original Quick Start, this app is structured as a Docker Compose stack. The Docker Compose file comprises the following services:

1. broker: a Kafka broker, based on the Docker Image for Apache Kafka.

2. kafka-connect-lightstreamer-sink: Kafka Connect with the Lightstreamer Sink connector plugin, based on the Kafka Connect Lightstreamer Sink Connector Docker image example.

   The service starts a Kafka Connect worker in standalone mode. Kafka Connect's worker configuration and connector configuration are mounted on /config.

3. schema-registry: Schema Registry, based on the Confluent Docker Image for Schema Registry.

4. lightstreamer: Lightstreamer Server, which also includes the following:

   • A web client mounted on /lightstreamer/pages/QuickStart.
   • A Proxy adapter configuration mounted on /lightstreamer/adapters/kafka-connect-proxy.

5. producer: a native Kafka Producer, based on the provided Dockerfile file from the quickstart-producer producer sample client.

Run

1. Make sure you have Docker, Docker Compose, and Java 17 (or later) installed on your local machine.

2. From this directory, run the following:

   $ ./start-connect.sh
   ...
   ✔ Network quickstart-kafka-connect_default  Created                                               
   ✔ Container broker                          Started                                     
   ✔ Container schema-registry                 Started                                
   ⠋ Container lightstreamer                   Starting                                                                 
   ⠹ Container producer                        Starting                            
   ✔ Container kafka-connect                   Created  
   ...
   Services started. Now you can point your browser to http://localhost:8080/QuickStart to see real-time data.
   ...

3. Once all containers are ready, point your browser to http://localhost:8080/QuickStart.

4. After a few moments, the user interface starts displaying the real-time stock data.

   

5. To shutdown Docker Compose and clean up all temporary resources:

   $ ./stop-connect.sh