From 886c607141610d1837227b2e13978132b4578e65 Mon Sep 17 00:00:00 2001 From: Binayak Pokharel Date: Thu, 14 Nov 2024 13:32:09 +1100 Subject: [PATCH] minor changes --- kafka/README.md | 87 +++++++++++++++++++++-------------------- kafka/node_collector.sh | 1 + 2 files changed, 46 insertions(+), 42 deletions(-) diff --git a/kafka/README.md b/kafka/README.md index 187f089..f21ce6f 100644 --- a/kafka/README.md +++ b/kafka/README.md @@ -1,54 +1,57 @@ This tool is used to collect information from a Kafka cluster to add in problem diagnosis or review. -# Design info: -There are two scripts used in instacollector tool. The `node_collector.sh` is supposed to be executed on each Kafka node. -The `cluster_collector.sh` can be executed on a machine connected to Kafka cluster e.g. user laptop or Jumpbox having connectivity -with Cassandra cluster. - -The node_collector.sh executes Linux and Kafka commands and copies configuration and log files required for cluster health check. -The cluster_collector.sh executes node_collector.sh on each Kafka node using ssh. The cluster_collector.sh accepts 4 user inputs - +Note: +* This won't work on versions before kafka 2 +* User is requested to change the path values in `node_collector.sh` before running any of the scripts, as by default the script uses `Kafka configuration file locations`, `data directory location`, and other setting locations as per **Apache Kafka** default setup. +* The term "VM" in environment of script `cluster_collector.sh` means if running in kernel. -Enter your kafka environment (vm/docker) : -Enter username for login on Kafka cluster nodes (Press Enter for default admin) : -Enter Identity file path: -Enter path of the command config file: -Enter file containing ip addresses/host/container names of Kafka cluster nodes: +# Design info: +There are two scripts used in instacollector tool for kafka. + +1. `node_collector.sh`: supposed to be executed on each Kafka node. It executes Linux and Kafka commands and copies configuration and log files required for cluster health checks. The user needs to modify the `KAFKA_HOME` path inside the script as per their configurations, the default value used is: +``` +KAFKA_HOME=/opt/kafka +``` +2. `cluster_collector.sh`: to be executed on a machine connected to Kafka cluster e.g. user laptop with a running docker or a running VM. It executes node_collector.sh on each Kafka node using ssh. The cluster_collector.sh requires 4 user inputs : +``` + * Enter your kafka environment (vm/docker) : + * [If VM] + * Enter username for login on Kafka cluster nodes (Press Enter for default admin) : + * Enter Identity file path: (the ssh key file in your local machine which is used to connect to the VMs) + * [If docker] + * Enter docker home directory: + * Enter path of the command config file: (kafka command-config file location on the kafka brokers) + * Enter file containing ip addresses/host/container names of Kafka cluster nodes: (the hosts file in your local machine) +``` +******************* +# Execution settings: +The `cluster_collector.sh` has setting of connecting to cluster nodes using the provided ssh key file or an id file. +If the ssh key has passphrase enabled then please use `ssh-agent` & `ssh-add` commands to add the passphrase before running `cluster_collector.sh` script. -# Execution settings: -The cluster_collector.sh has setting of connecting to cluster nodes using key file or id file. -If the ssh key has passphrase enabled then please use ssh-agent and ssh-add commands to add the passphrase before running cluster_collector.sh. If there is another method required for `ssh`, user is requested to change the script as applicable. -Alternatively, the node_collector.sh can also be executed on individual nodes if cluster_collector.sh is not useful in any case. - -The Kafka configuration file locations, data directory location and other settings are used as per Apache Kafka default setup. -User is requested to change those in node_collector.sh if other values are required. Below are the Kafka & Zookeeper related files -which are copied from different nodes. -Kafka Broker Files -******************* +Alternatively, the `node_collector.sh` can also be executed on individual nodes if `cluster_collector.sh` is not useful in any case. + + +Below are the Kafka & Zookeeper related files which will be copied from different nodes: +``` +Kafka Broker Files | Zookeeper Files +**********************************|*********************** +server.properties | zookeeper.properties +server.log | zoo.cfg +kafkaServer.out | log4j.properties +kafka-authorizer.log | zoo.log +controller.log | zookeeper_jaas.conf +state-change.log | zookeeper.out +kafka_server_jaas.conf | +kafka-topics/.sh | +kafka-topics/.sh | +kafka-broker-api-versions/.sh | +kafka-consumer-groups/.sh | server.properties -server.log -kafkaServer.out -kafka-authorizer.log -controller.log -state-change.log -kafka_server_jaas.conf -kafka-topics/.sh -kafka-topics/.sh -kafka-broker-api-versions/.sh -kafka-consumer-groups/.sh - -Zookeeper Files -**************** -zookeeper.properties -zoo.cfg -log4j.properties -zoo.log -zookeeper_jaas.conf -zookeeper.out - +``` **Note:** The scripts should be executed on bash shell. diff --git a/kafka/node_collector.sh b/kafka/node_collector.sh index 9e851d8..b5e1285 100755 --- a/kafka/node_collector.sh +++ b/kafka/node_collector.sh @@ -109,6 +109,7 @@ copy_config_files() { echo "$ip : ${FUNCNAME[0]} : Copying files" local config_files=( "$BROKER_CONFIG_PATH/server.properties" + "$BROKER_CONFIG_PATH/log4j.properties" "$BROKER_LOG_PATH/server.log" "$BROKER_LOG_PATH/kafkaServer.out" "$BROKER_LOG_PATH/kafka-authorizer.log"