Skip to content

eskhisov/EventHubsMirrorMaker

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 

Repository files navigation

EventHubsMirrorMaker

This sample uses Azure Event Hubs Kafka MirrorMaker running on an Azure Container Instance and then it creates a second container as a load generator. Use the Cloud Shell in the Azure portal to give this a try. This will also work from bash if you have the Azure CLI installed.

Setup a few variables to make your life infinitely easier

rgname=mmrg$RANDOM
az group create --name $rgname --location westus
SOURCE=myehkafkasource$RANDOM
DEST=myehkafkadest$RANDOM

Create two Azure Event Hubs and get the connection strings

az eventhubs namespace create --name $SOURCE --resource-group $rgname -l westus --enable-kafka true
az eventhubs namespace create --name $DEST --resource-group $rgname -l westus --enable-kafka true
SOURCE_CON_STR=$(az eventhubs namespace authorization-rule keys list --resource-group $rgname --namespace-name $SOURCE --name RootManageSharedAccessKey --query "primaryConnectionString" --output tsv)
DEST_CON_STR=$(az eventhubs namespace authorization-rule keys list --resource-group $rgname --namespace-name $DEST --name RootManageSharedAccessKey --query "primaryConnectionString" --output tsv)

Create a container instance to run the built in Kafka perf test sender. The credentials are injected into the container as environment variables. Check out /ehmirror/perfstarter.sh for more details. The same pattern is used for the MirrorMaker container.

az container create --resource-group $rgname --name loadgenerator --image confluentinc/cp-kafka --gitrepo-url https://github.com/djrosanova/EventHubsMirrorMaker --gitrepo-mount-path /mnt/EventHubsMirrorMaker --command-line "/bin/bash ./mnt/EventHubsMirrorMaker/ehmirror/perfstarter.sh -d $SOURCE_CON_STR" --restart-policy OnFailure --environment-variables SOURCE_CON_STR="$SOURCE_CON_STR" 

Create a container instance to host Mirror Maker. It will see the topic that the above perf container is sending data to and copy it to the second Event Hub / topic.

az container create --resource-group $rgname --name mirrormaker --image confluentinc/cp-kafka --gitrepo-url https://github.com/djrosanova/EventHubsMirrorMaker --gitrepo-mount-path /mnt/EventHubsMirrorMaker --command-line "/bin/bash ./mnt/EventHubsMirrorMaker/ehmirror/mirrorstart.sh " --environment-variables SOURCE_CON_STR="$SOURCE_CON_STR" DEST_CON_STR="$DEST_CON_STR"

You can see the details from the mirror maker container with this command.

az container show --resource-group $rgname --name mirrormaker

Take a look at the logs from the container.

az container logs --resource-group $rgname --name mirrormaker
az container logs --resource-group $rgname --name loadgenerator 

Clean up all your stuff.

az container delete --resource-group $rgname --name mirrormaker
az container delete --resource-group $rgname --name loadgenerator 
az group delete --name $rgname

About

Azure Event Hubs Kafka MirrorMaker ACI

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 91.9%
  • Dockerfile 8.1%