From d71d384a6bd90142ca4dde95e7d6f000c0dfc292 Mon Sep 17 00:00:00 2001 From: Iulian Pascalau Date: Mon, 20 May 2024 14:21:36 +0300 Subject: [PATCH 1/3] - added documentation - added support scripts --- README.md | 252 +++++++++++++++++++++++++++++- scripts/config/functions.cfg | 252 ++++++++++++++++++++++++++++++ scripts/config/menu_functions.cfg | 112 +++++++++++++ scripts/config/variables.cfg | 56 +++++++ scripts/script.sh | 126 +++++++++++++++ 5 files changed, 796 insertions(+), 2 deletions(-) create mode 100644 scripts/config/functions.cfg create mode 100644 scripts/config/menu_functions.cfg create mode 100644 scripts/config/variables.cfg create mode 100755 scripts/script.sh diff --git a/README.md b/README.md index f2b6520..2358d7e 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,250 @@ -# mx-chain-keys-monitor-go -MultiversX keys monitor tool written in Go +[![Go Report Card](https://goreportcard.com/badge/github.com/multiversx/mx-chain-keys-monitor-go)](https://goreportcard.com/report/github.com/multiversx/mx-chain-keys-monitor-go) +[![Codecov](https://codecov.io/gh/multiversx/mx-chain-keys-monitor-go/branch/main/graph/badge.svg)](https://codecov.io/gh/multiversx/mx-chain-keys-monitor-go) +[![GitHub release (latest by date)](https://img.shields.io/github/v/release/multiversx/mx-chain-keys-monitor-go)](https://github.com/multiversx/mx-chain-keys-monitor-go/releases) +[![GitHub](https://img.shields.io/github/license/multiversx/mx-chain-keys-monitor-go)](LICENSE) + +# MultiversX keys monitor + +This tool allows the monitoring of BLS keys that can participate into the consensus, regardless the shard they are currently operating. +This is done by continuously polling the `/validator/statistics` API endpoint route. +It can monitor one or more networks so one instance of this tool is enough for all keys used on mainnet/testnet or devnet networks. +The monitored BLS keys will be defined as lists. Also, there is the possibility to define just an owner address that staked the BLS keys +and the application will automatically fetch the registered BLS keys on that identity. + +## Feature list +- [x] Keys monitor + - [x] Monitor any number of networks (mainnet/testnet/devnet) with one instance + - [x] Monitor any number of BLS keys defined in separate files + - [x] Automatically fetch the BLS keys staked by an address + - [x] Threshold definition on each set for the allowed rating drop + - [x] Configurable polling time for each definition set +- [x] Notification system + - [x] Integrated the [Pushover](https://pushover.net/) service that allows easy mobile device notifications + - [x] Integrated the SMTP email service to notify thorough emails the events encountered + - [X] Integrated the [Telegram bot](https://core.telegram.org/bots) notification service +- [x] System self-check messages + - [x] Integrated a self-check system that can send messages periodical messages about the status of the app +- [x] Scripts & installation support + - [x] Added scripts for easy setup & upgrade + +## Installation + +### Initial setup + +Although is possible, it is not recommended to run the application as `root`. For that reason, a new user is required to be created. +For example, the following script creates a new user called `ubuntu`. This script can be run as `root`. + +```bash +# host update/upgrade +apt-get update +apt-get upgrade +apt autoremove + +adduser ubuntu +# set a long password +usermod -aG sudo ubuntu +echo 'StrictHostKeyChecking=no' >> /etc/ssh/ssh_config + +visudo +# add this line: +ubuntu ALL=(ALL) NOPASSWD:ALL +# save & exit + +sudo su ubuntu +sudo visudo -f /etc/sudoers.d/myOverrides +# add this line: +ubuntu ALL=(ALL) NOPASSWD:ALL +# save & exit +``` + +### Repo clone & scripts init + +```bash +cd +git clone https://github.com/multiversx/mx-chain-keys-monitor-go +cd ~/mx-chain-keys-monitor-go/scripts +# the following init call will create ~/mx-chain-keys-monitor-go/scripts/config/local.cfg file +# and will copy the configs from ~/mx-chain-keys-monitor-go/cmd/monitor/config/example to ~/mx-chain-keys-monitor-go/cmd/monitor/config +# to avoid github pull problems +scripts.sh init +cd config +# edit the local.cfg file for the scripts setup +nano local.cfg +``` + +### local.cfg configuration + +The generated local.cfg file contains the following lines: + +```bash +#!/bin/bash +set -e + +CUSTOM_HOME=/home/ubuntu +CUSTOM_USER=ubuntu +GITHUBTOKEN="" +MONITOR_EXTRA_FLAGS="" + +#Allow user to override the current version of the monitor +OVERRIDE_VER="" +``` + +The `CUSTOM_HOME` and `CUSTOM_USER` will need to be changed if the current user is not `ubuntu`. +To easily figure out the current user, the bash command `whoami` can be used. + +It is strongly recommended to use a GitHub access token because the scripts consumes the GitHub APIs and +throttling might occur without the access token. + +The `MONITOR_EXTRA_FLAGS` can contain extra flags to be called whenever the application is started. +The complete list of the cli command can be found [here](./cmd/monitor/CLI.md) + +The `OVERRIDE_VER` can be used during testing to manually specify an override tag/branch that will be used when building +the application. If left empty, the upgrade process will automatically fetch and use the latest release. + +### Install + +After the `local.cfg` configuration step, the scripts can now install the application. +```bash +cd ~/mx-chain-keys-monitor-go/scripts +./script.sh install +``` + +### Application config + +After the application has been installed, it is now time to configure the application. +For this, you should edit the `config.toml` and `credentials.toml` files and add files containing lists of BLS keys or addresses in the +`~/mx-chain-keys-monitor-go/cmd/monitor/config` directory. + +The scripts init step already created some initial .toml and .list files to be ready to be used directly. + +Configuring the **config.toml** file: + +This file contains the general application configuration file. +```toml +[General] + ApplicationName = "Keys monitoring app" + # the application can send messages about the internal status at regular intervals + [General.SystemSelfCheck] + Enabled = true + DayOfWeek = "every day" # can also be "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday" and "Sunday" + Hour = 12 # valid interval 0-23 + Minute = 00 # valid interval 0-59 + PollingIntervalInSec = 30 + [General.Logs] + LogFileLifeSpanInMB = 1024 # 1GB + LogFileLifeSpanInSec = 86400 # 1 day + +[OutputNotifiers] + NumRetries = 3 + SecondsBetweenRetries = 10 + + # Uses Pushover service that can notify Desktop, Android or iOS devices. Requires a valid subscription. + # If you enable this notifier, remember to specify the credentials in credentials.toml file + [OutputNotifiers.Pushover] + Enabled = false + URL = "https://api.pushover.net/1/messages.json" + + # SMTP (email) based notification + # If you enable this notifier, remember to specify the credentials in credentials.toml file + # If you are using gmail server, please make sure you activate the IMAP server and use App passwords instead of the account's password + [OutputNotifiers.Smtp] + Enabled = false + To = "to@email.com" + SmtpPort = 587 + SmtpHost = "smtp.gmail.com" + + # Uses Telegram service that can notify Desktop, Android or iOS devices. Requires a running bot and the chat ID for + # the user that will be notified. + # If you enable this notifier, remember to specify the credentials in credentials.toml file + [OutputNotifiers.Telegram] + Enabled = false + URL = "https://api.telegram.org" + +[[BLSKeysMonitoring]] + AlarmDeltaRatingDrop = 1.0 # maximum Rating-TempRating value that will trigger an alarm, for the public testnet might use a higher value (2 or 3) + Name = "network 1" + ApiURL = "API URL 1" + ExplorerURL = "" + PollingIntervalInSeconds = 300 # 5 minutes + ListFile = "./config/network1.list" +``` + +* The `General` section + + - The `ApplicationName` can be any string, it will be used in the notification messages. + + - The `General.SystemSelfCheck` section can enable & configure the self check monitoring system +that will output a notification message once a day or week, depending on the configuration parameters. + + - The `General.Logs` will configure the internal logging system (for debugging purposes). + + +* The `OutputNotifiers` is the section containing the implemented notifiers. +There are 3 types of notifiers implemented: `Pushover`, `Smtp` and `Telegram`, each with its configuration sections. +The credentials for the notifiers are defined separately in the `credentials.toml` so that is the place where +passwords or access tokens will be specified. + + +* The `BLSKeysMonitoring` defines the section used on one network. + + The `config.toml` file accepts any number of this type of section. + + - The `AlarmDeltaRatingDrop` option will specify how large the difference between the "Epoch start rating" - "Current rating" is allowed before +emitting an alert. Usually, a value of `1` will suffice in most cases. On the public testnet, this value might be increased to `2` or even `3` to +reduce the number of false alarms due to the nature of the other nodes operating in the network. + + - The `Name` defines a string for the monitored network. Can be something like `Mainnet`, `Testnet`, `Devnet` or any kind of identification string. + + - The `ApiURL` defines the API url for that network. Examples here include `https://api.multiversx.com` for the mainnet, +`https://testnet-api.multiversx.com` for the testnet, and `https://devnet-api.multiversx.com` for the devnet. + + - The `ExplorerURL` is used whenever a BLS key alert message is emitted, to automatically include the link to that BLS key page. +If left empty, the message will still be emitted, but it will not contain any link. +Examples here include `https://explorer.multiversx.com` for the mainnet, +`https://testnet-explorer.multiversx.com` for the testnet, and `https://devnet-explorer.multiversx.com` for the devnet. + + - The `PollingIntervalInSeconds` represents the time in seconds between the calls on the API URL. + + - The `ListFile` will contain the name of the file containing BLS or identity keys. Refer to the example file called +`network1.list` to check how keys/identities can be defined. + +### Application start + +After editing the required config files, the application can be started. +```bash +cd ~/mx-chain-keys-monitor-go/scripts +./script.sh start +``` + +When the application starts, it automatically emits a notification message. This is valid also for the `stop` operation +issued by the `./script.sh stop` command. + +### Backup and upgrade + +It is a good practice to save the .toml, .list and the local.cfg files somewhere else just in case the application is cleaned up accidentally. +The upgrade call for the monitor app is done through this command: +```bash +cd ~/mx-chain-keys-monitor-go/scripts +./script.sh upgrade +``` + +### Uninstalling + +The application can be removed by executing the following script: +```bash +cd ~/mx-chain-keys-monitor-go/scripts +./script.sh cleanup +``` + +### Troubleshooting + +If the application fails to start (maybe there is a bad config in the .toml files), the following command can be issued: +```bash +sudo journalctl -f -u mx-chain-keys-monitor.service +``` + +Also, if the application misbehaves, the logs can be retrieved by using this command: +```bash +cd ~/mx-chain-keys-monitor-go/scripts +./script.sh get_logs +``` diff --git a/scripts/config/functions.cfg b/scripts/config/functions.cfg new file mode 100644 index 0000000..4b26a3a --- /dev/null +++ b/scripts/config/functions.cfg @@ -0,0 +1,252 @@ +#!/bin/bash +set -e + +function init { + if ! [ -f "$LOCAL_OVERRIDES" ]; then + echo "#!/bin/bash +set -e + +CUSTOM_HOME=\"/home/ubuntu\" +CUSTOM_USER=\"ubuntu\" +GITHUBTOKEN=\"\" +MONITOR_EXTRA_FLAGS=\"\" + +#Allow user to override the current version of the monitor +OVERRIDE_VER=\"\" +" > $LOCAL_OVERRIDES + + echo -e + echo -e "${GREEN}Created ./config/local.cfg - now will be a good time to add your scripts variables there...${NC}" + echo -e + else + echo -e + echo -e "${CYAN}Will not override the existing file ./config/local.cfg${NC}" + echo -e + fi + + copy_config_file "$WORKDIR/cmd/monitor/config/example/config.toml" "$WORKDIR/cmd/monitor/config/config.toml" + copy_config_file "$WORKDIR/cmd/monitor/config/example/credentials.toml" "$WORKDIR/cmd/monitor/config/credentials.toml" + copy_config_file "$WORKDIR/cmd/monitor/config/example/network1.list" "$WORKDIR/cmd/monitor/config/network1.list" + echo -e +} + +function copy_config_file { + if ! [ -f "$2" ]; then + cp $1 $2 + + echo -e "${GREEN}Copied $1 to $2...${NC}" + else + echo -e "${CYAN}Will not override the existing file $2${NC}" + fi + echo -e +} + +function check_variables { + #Check if CUSTOM_HOME exists + if ! [ -d "$CUSTOM_HOME" ]; then echo -e "${RED}Please configure your variables first ! (config/local.cfg (based on variables.cfg) --> CUSTOM_HOME & CUSTOM_USER)${NC}"; exit; fi +} + +function prerequisites { + #Prerequisites function + echo -e + echo -e "${GREEN}Updating system & installing some dependencies...${NC}" + echo -e + DISTRO=$(cat /etc/*-release | grep -w "ID") + + if [[ $DISTRO == *"linuxmint"* ]] || [[ $DISTRO == *"ubuntu"* ]] || [[ $DISTRO == *"debian"* ]] || [[ $DISTRO == *"elementary"* ]]; then + sudo DEBIAN_FRONTEND=noninteractive apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get -qy -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade > /dev/null + sudo DEBIAN_FRONTEND=noninteractive apt-get -qy -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install build-essential > /dev/null && sudo DEBIAN_FRONTEND=noninteractive apt-get -qy -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install git rsync curl zip unzip jq gcc wget > /dev/null + echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin" >> ~/.profile + echo "export GOPATH=$HOME/go" >> ~/.profile + else + echo "Unsupported DISTRO..." + exit + fi + + # Check if $CUSTOM_HOME exists and if not create it + if ! [ -d "$CUSTOM_HOME" ]; then mkdir -p $CUSTOM_HOME; fi + + #Limit journalctl size + echo -e + echo -e "${GREEN}Limiting journalctl node units logs size...${NC}" + echo -e + echo 'SystemMaxUse=4000M' | sudo tee -a /etc/systemd/journald.conf && echo 'SystemMaxFileSize=800M' | sudo tee -a /etc/systemd/journald.conf + sudo systemctl restart systemd-journald + + echo -e "${GREEN}Task completed.${NC}" + echo -e +} + +function paths { + # Export environment variables + export GOPATH=$HOME/go + export PATH=$GOPATH/bin:$PATH + export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin +} + +function go_lang { + #Identify machine architecture + ARCH=$(dpkg --print-architecture) + + #Check if golang is installed on system + if ! [ -x "$(command -v go)" ]; then + #Get the latest version of GO for amd64 & installing it + echo -e + echo -e "${RED}GO is not installed on your system${NC}" + echo -e + echo -e "${GREEN}The best working version of Go is:${CYAN}${GO_LATEST_TESTED}${NC}" + echo -e "${GREEN}Installing it now...${NC}" + echo -e + wget https://dl.google.com/go/${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + sudo tar -C /usr/local -xzf ${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + rm ${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + + else + #Check the installed version number + GOVERSION=$(go version | awk '{print $3}') + if [[ "$GOVERSION" < "${GO_LATEST_TESTED}" ]]; then + echo -e + echo -e "${RED}GO version is ${CYAN}$GOVERSION${RED} and the best working version is ${CYAN}${GO_LATEST_TESTED}${RED}... ${NC}" + #Detect go install method + GO_INSTALL_METHOD=$(which go) + + if [[ "$GO_INSTALL_METHOD" == "/usr/local/go/bin/go" ]]; then + #Installed by scripts. Go ahead and upgrade. + echo -e + echo -e "${GREEN}Your GO binary will pe upgraded to the minimum required version...${NC}" + sudo rm -rf /usr/local/go + wget -4 https://dl.google.com/go/${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + sudo tar -C /usr/local -xzf ${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + rm ${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz + + else + echo -e + echo -e "${RED}GO was not installed using the elrond scripts. Operation cannot continue...${NC}" + exit + fi + + else + echo -e + echo -e "${GREEN}GO is already installed: ${CYAN}$GOVERSION${NC}${GREEN}...skipping install${NC}" + echo -e + fi + fi +} + +function check_api_limit { + #Check API rate-limit + if [ ${GITHUB_API_LIMIT} -eq ${GITHUB_API_USED} ] && [[ -z "$GITHUBTOKEN" ]]; then + echo -e + echo -e "${RED}GitHub API Requests Limit for this IP has been reached !${NC}" + echo -e "${RED}The counter will reset at ${CYAN}${HUMAN_TIME}${RED}.${NC}" + echo -e + echo -e "${RED}We suggest using a GitHub Token to avoid this issue in the future.${NC}" + exit + else + echo -e + echo -e "${GREEN}Your GitHub API Requests is at ${CYAN}$GITHUB_API_USED${GREEN} out of a total of ${CYAN}$GITHUB_API_LIMIT${GREEN} (applied per hour)${NC}" + echo -e + fi +} + +function build_monitor { + #Build the monitor from latest tag + echo -e + echo -e "${GREEN}Building your monitor binary...${NC}" + echo -e + + #If OVERRIDE_VER is set SHOWVER should take it into account + [[ ! -z "$OVERRIDE_VER" ]] && SHOWVER=$(echo ${OVERRIDE_VER#${TMPVAR}}) + [[ ! -z "$OVERRIDE_VER" ]] && VERSION=$(echo $OVERRIDE_VER) + + #Building the node from the local directory + cd $WORKDIR + echo -e "${GREEN}Compiling binary with version ${CYAN}$VERSION${GREEN}...${NC}" + echo -e + + cd cmd/monitor + # alter/create the local.go file that is .gitignored + appVersion=$SHOWVER-0-$(git describe --tags --long | tail -c 11) + echo "package main + + func init() { + appVersion = \"$appVersion\" + }" > local.go + + git fetch + git checkout $VERSION + if [ "$VERSION" == "$SHOWVER" ]; then + # we are on a branch, we can do a git pull + git pull + fi + + go build + echo -e "${GREEN}Done !${NC}" +} + +function show_menu { +echo -e +echo -e +echo -e "${CYAN}MultiversX ${RED}$ATTENTION${CYAN} scripts options:${NC}" +echo -e +echo -e "${GREEN} 1) ${CYAN}init{GREEN} - init the scripts by creating the config/local.cfg file${NC}" +echo -e "${GREEN} 2) ${CYAN}install${GREEN} - Regular install process for the monitor${NC}" +echo -e "${GREEN} 3) ${CYAN}upgrade${GREEN} - Run the upgrade process for the monitor${NC}" +echo -e "${GREEN} 4) ${CYAN}start${GREEN} - Start the monitor${NC}" +echo -e "${GREEN} 5) ${CYAN}stop${GREEN} - Stop the monitor${NC}" +echo -e "${GREEN} 6) ${CYAN}cleanup${GREEN} - Remove everything from the host${NC}" +echo -e "${GREEN} 7) ${CYAN}get_logs${GREEN} - Get the logs from the monitor${NC}" +echo -e "${GREEN} 8) ${CYAN}quit${GREEN} - Exit this menu${NC}" +echo -e +echo -e +} + +function systemd { + #Add systemd monitor service & syslogd logging + echo -e + echo -e "${GREEN}Installing systemd unit for the monitor...${NC}" + echo -e + + #Create the service file for our node + echo "[Unit] + Description=MultiversX keys monitor + After=network-online.target + + [Service] + User=$CUSTOM_USER + WorkingDirectory=$WORKDIR/cmd/monitor + ExecStart=$WORKDIR/cmd/monitor/monitor -log-level *:DEBUG $MONITOR_EXTRA_FLAGS + StandardOutput=journal + StandardError=journal + Restart=always + RestartSec=3 + LimitNOFILE=4096 + + [Install] + WantedBy=multi-user.target" > mx-chain-keys-monitor.service + + #Move files to appropriate locations + if [ -f "/etc/systemd/system/mx-chain-keys-monitor.service" ]; then + echo -e + echo -e "${GREEN}Refreshing mx-chain-keys-monitor service (stop-disable-remove)...${NC}" + echo -e + sudo systemctl stop mx-chain-keys-monitor + sleep 1 + sudo systemctl disable mx-chain-keys-monitor.service + sleep 1 + sudo rm -f /etc/systemd/system/mx-chain-keys-monitor.service + sleep 1 + fi + sudo mv mx-chain-keys-monitor.service /etc/systemd/system/ + sudo systemctl enable mx-chain-keys-monitor.service +} + +function cleanup_files { + + #Cleanup Logs + echo -e + echo -e "${RED}Erasing previous monitor data (logs)...${NC}" + sudo rm -rf $WORKDIR/logs + #recreate erased folders + mkdir -p $WORKDIR/{logs} +} diff --git a/scripts/config/menu_functions.cfg b/scripts/config/menu_functions.cfg new file mode 100644 index 0000000..0027b21 --- /dev/null +++ b/scripts/config/menu_functions.cfg @@ -0,0 +1,112 @@ +#!/bin/bash +set -e + +source $SCRIPTPATH/config/variables.cfg +source $SCRIPTPATH/config/functions.cfg +# Load local overrides, .gitignored +LOCAL_OVERRIDES="$SCRIPTPATH/config/local.cfg" +if [ -f "$LOCAL_OVERRIDES" ]; then + source "$SCRIPTPATH/config/local.cfg" +fi + +function install { + check_variables + prerequisites + paths + go_lang + check_api_limit + build_monitor + systemd +} + +function upgrade { + echo -e + read -p "Do you want to go on with the upgrade (Default No) ? (Yy/Nn)" yn + echo -e + + case $yn in + [Yy]* ) + sudo systemctl stop mx-chain-keys-monitor + check_variables + check_api_limit + build_monitor + systemd + echo -e + echo -e "${CYAN}--> mx-chain-keys-monitor upgraded. Don't forget to start it...${NC}" + echo -e + ;; + + [Nn]* ) + echo -e "${GREEN}Fine ! Skipping upgrade on this machine...${NC}" + ;; + + * ) + echo -e "${GREEN}I'll take that as a no then... moving on...${NC}" + ;; + esac +} + +function start { + sudo systemctl daemon-reload + sudo systemctl start mx-chain-keys-monitor +} + +function stop { + sudo systemctl stop mx-chain-keys-monitor +} + +function cleanup { + paths + echo -e + read -p "Do you want to delete installed nodes (Default No) ? (Yy/Nn)" yn + echo -e + case $yn in + [Yy]* ) + echo -e "${RED}OK ! Cleaning everything !${NC}" + + echo -e + echo -e "${GREEN}Stopping MultiversX keys monitor binary on host ${CYAN}$HOST${GREEN}...${NC}" + echo -e + if [ -e /etc/systemd/system/mx-chain-keys-monitor.service ]; then sudo systemctl stop mx-chain-keys-monitor; fi + echo -e "${GREEN}Erasing unit file and monitor folder for MultiversX keys monitor...${NC}" + echo -e + if [ -e /etc/systemd/system/mx-chain-keys-monitor.service ]; then sudo rm /etc/systemd/system/mx-chain-keys-monitor.service; fi + if [ -d $CUSTOM_HOME/mx-chain-keys-monitor ]; then sudo rm -rf $CUSTOM_HOME/mx-chain-keys-monitor; fi + + #Reload systemd after deleting node units + sudo systemctl daemon-reload + + echo -e "${GREEN}Removing paths from .profile on host ${CYAN}$HOST${GREEN}...${NC}" + echo -e + sed -i 'N;$!P;$!D;$d' ~/.profile + + #----------------------------------------------------- + if [ -d "$GOPATH/pkg" ]; then sudo rm -rf $GOPATH/pkg; fi + ;; + + [Nn]* ) + echo -e "${GREEN}Fine ! Skipping cleanup on this machine...${NC}" + ;; + + * ) + echo -e "${GREEN}I'll take that as a no then... moving on...${NC}" + ;; + esac +} + +function get_logs { + #Get journalctl logs from the monitor + LOGSTIME=$(date "+%Y%m%d-%H%M") + #Make sure the log path exists + mkdir -p $CUSTOM_HOME/mx-chain-keys-monitor-logs + + echo -e + echo -e "${GREEN}Getting logs for MultiversX keys monitor...${NC}" + echo -e + sudo journalctl --unit mx-chain-keys-monitor >> $CUSTOM_HOME/mx-chain-keys-monitor-logs/mx-chain-keys-monitor-logs.log + + #Compress the logs and erase files + cd $CUSTOM_HOME/mx-chain-keys-monitor-logs/ && tar -zcvf mx-chain-keys-monitor-logs-$LOGSTIME.tar.gz *.log && rm *.log + echo -e + echo -e "${GREEN}---> Logs have been stored in the ${CYAN}~/mx-chain-keys-monitor-logs${GREEN} directory...${NC}" +} diff --git a/scripts/config/variables.cfg b/scripts/config/variables.cfg new file mode 100644 index 0000000..84c30d3 --- /dev/null +++ b/scripts/config/variables.cfg @@ -0,0 +1,56 @@ +#!/bin/bash +set -e + +#-------------------------------#### DON'T CHANGE THESE, use local.cfg instead ####-------------------------------# + +#Color definitions +RED='\x1B[0;31m' +CYAN='\x1B[0;36m' +GREEN='\x1B[0;32m' +NC='\x1B[0m' + +#Make sure curl and jq commands exist & if not install them +if ! [ -x "$(command -v curl)" ]; then sudo apt update; sudo apt install curl -qq -y; fi +if ! [ -x "$(command -v jq)" ]; then sudo apt update; sudo apt install jq -qq -y; fi + +CUSTOM_HOME="/home/ubuntu" +CUSTOM_USER="ubuntu" +GITHUBTOKEN="" +MONITOR_EXTRA_FLAGS="" + +#Make script aware of its location +SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )" +WORKDIR="$( cd "$(dirname "$0")" & cd .. ; pwd -P )" + +#Use the latest MultiversX tested GO Lang version +GO_LATEST_TESTED="go1.20.7" + +#Obtain the tag for the latest version node & configs +TMPVAR="tags/" + +if [ -z "$GITHUBTOKEN" ]; then + VERSION="tags/$(curl --silent "https://api.github.com/repos/multiversx/mx-chain-keys-monitor-go/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')" + RELEASENOTES=$(curl --silent "https://api.github.com/repos/multiversx/mx-chain-keys-monitor-go/releases/latest" | grep '"body": '| cut -f2- -d:) + GITHUB_RATE_LIMIT_RESPONSE=$(curl -s https://api.github.com/rate_limit) + + else + VERSION="tags/$(curl --silent -H "Authorization: token $GITHUBTOKEN" "https://api.github.com/repos/multiversx/mx-chain-keys-monitor-go/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')" + RELEASENOTES=$(curl --silent -H "Authorization: token $GITHUBTOKEN" "https://api.github.com/repos/multiversx/mx-chain-keys-monitor-go/releases/latest" | grep '"body": '| cut -f2- -d:) + GITHUB_RATE_LIMIT_RESPONSE=$(curl --silent -H "Authorization: token $GITHUBTOKEN" "https://api.github.com/rate_limit") + + fi + +SHOWVER=$(echo ${VERSION#${TMPVAR}}) + +SCRIPTS_LOCATION=$(pwd) +DIRECTORY_NAME=${PWD##*/} +ATTENTION=$(echo $ENVIRONMENT | tr [a-z] [A-Z]) + +#GitHub API related variables +GITHUB_API_LIMIT=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.limit) +GITHUB_API_USED=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.used) +GITHUB_API_RESET=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.reset) +HUMAN_TIME=$(date -d @"$GITHUB_API_RESET") + +#Allow user to override the current version of the monitor +OVERRIDE_VER="" \ No newline at end of file diff --git a/scripts/script.sh b/scripts/script.sh new file mode 100755 index 0000000..d1554c8 --- /dev/null +++ b/scripts/script.sh @@ -0,0 +1,126 @@ +#!/bin/bash +set -e + +#Make script aware of its location +SCRIPTPATH="$( cd -- "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 ; pwd -P )" + +source $SCRIPTPATH/config/variables.cfg +source $SCRIPTPATH/config/functions.cfg +source $SCRIPTPATH/config/menu_functions.cfg + +#See if the user has passed any command line arguments and if not show the menu +if [ $# -eq 0 ] + then + + show_menu #Show all the menu options + + COLUMNS=8 + PS3="Please select an action:" + options=("init" "install" "upgrade" "start" "stop" "cleanup" "get_logs" "quit") + + select opt in "${options[@]}" + do + + case $opt in + + `init`) + init + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'install') + install + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'upgrade') + upgrade + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'start') + start + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'stop') + stop + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'cleanup') + cleanup + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'get_logs') + get_logs + echo -e + read -n 1 -s -r -p " Process finished. Press any key to continue..." + clear + show_menu + ;; + + 'quit') + echo -e + echo -e "${GREEN}---> Exiting scripts menu...${NC}" + echo -e + break + ;; + + esac + + done + +else + +case "$1" in +'init') + init + ;; + +'install') + install + ;; + +'upgrade') + upgrade + ;; + +'start') + start + ;; + +'stop') + stop + ;; + +'cleanup') + cleanup + ;; + +'get_logs') + get_logs + ;; + +esac + +fi From f8fdf7ad8524019c65dbd4584bb6859521aeea08 Mon Sep 17 00:00:00 2001 From: Iulian Pascalau Date: Tue, 21 May 2024 10:21:09 +0300 Subject: [PATCH 2/3] - fixes after review --- README.md | 16 ++++++------- scripts/config/functions.cfg | 39 ++++++++++++++++--------------- scripts/config/menu_functions.cfg | 9 ++++--- scripts/config/variables.cfg | 14 +++++------ scripts/script.sh | 6 ++--- 5 files changed, 42 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 2358d7e..df48828 100644 --- a/README.md +++ b/README.md @@ -5,11 +5,11 @@ # MultiversX keys monitor -This tool allows the monitoring of BLS keys that can participate into the consensus, regardless the shard they are currently operating. +This tool allows the monitoring of BLS keys that can participate into the consensus, regardless of the shard they are currently operating in. This is done by continuously polling the `/validator/statistics` API endpoint route. It can monitor one or more networks so one instance of this tool is enough for all keys used on mainnet/testnet or devnet networks. The monitored BLS keys will be defined as lists. Also, there is the possibility to define just an owner address that staked the BLS keys -and the application will automatically fetch the registered BLS keys on that identity. +and the application will automatically fetch the registered BLS keys for that identity. ## Feature list - [x] Keys monitor @@ -19,11 +19,11 @@ and the application will automatically fetch the registered BLS keys on that ide - [x] Threshold definition on each set for the allowed rating drop - [x] Configurable polling time for each definition set - [x] Notification system - - [x] Integrated the [Pushover](https://pushover.net/) service that allows easy mobile device notifications + - [x] Integrated the [Pushover](https://pushover.net/) service to allow easy access to push-notifications on mobile devices - [x] Integrated the SMTP email service to notify thorough emails the events encountered - [X] Integrated the [Telegram bot](https://core.telegram.org/bots) notification service - [x] System self-check messages - - [x] Integrated a self-check system that can send messages periodical messages about the status of the app + - [x] Integrated a self-check system that can periodically send messages on the status of the app - [x] Scripts & installation support - [x] Added scripts for easy setup & upgrade @@ -31,7 +31,7 @@ and the application will automatically fetch the registered BLS keys on that ide ### Initial setup -Although is possible, it is not recommended to run the application as `root`. For that reason, a new user is required to be created. +Although it's possible, it is not recommended to run the application as `root`. For that reason, a new user is required to be created. For example, the following script creates a new user called `ubuntu`. This script can be run as `root`. ```bash @@ -92,7 +92,7 @@ OVERRIDE_VER="" The `CUSTOM_HOME` and `CUSTOM_USER` will need to be changed if the current user is not `ubuntu`. To easily figure out the current user, the bash command `whoami` can be used. -It is strongly recommended to use a GitHub access token because the scripts consumes the GitHub APIs and +It is strongly recommended to use a GitHub access token because the scripts consume the GitHub APIs and throttling might occur without the access token. The `MONITOR_EXTRA_FLAGS` can contain extra flags to be called whenever the application is started. @@ -111,7 +111,7 @@ cd ~/mx-chain-keys-monitor-go/scripts ### Application config -After the application has been installed, it is now time to configure the application. +After the application has been installed, it is now time to configure it. For this, you should edit the `config.toml` and `credentials.toml` files and add files containing lists of BLS keys or addresses in the `~/mx-chain-keys-monitor-go/cmd/monitor/config` directory. @@ -191,7 +191,7 @@ passwords or access tokens will be specified. - The `AlarmDeltaRatingDrop` option will specify how large the difference between the "Epoch start rating" - "Current rating" is allowed before emitting an alert. Usually, a value of `1` will suffice in most cases. On the public testnet, this value might be increased to `2` or even `3` to -reduce the number of false alarms due to the nature of the other nodes operating in the network. +reduce the number of false positive alarms due to the nature of the other nodes operating on the network. - The `Name` defines a string for the monitored network. Can be something like `Mainnet`, `Testnet`, `Devnet` or any kind of identification string. diff --git a/scripts/config/functions.cfg b/scripts/config/functions.cfg index 4b26a3a..aff870c 100644 --- a/scripts/config/functions.cfg +++ b/scripts/config/functions.cfg @@ -11,7 +11,7 @@ CUSTOM_USER=\"ubuntu\" GITHUBTOKEN=\"\" MONITOR_EXTRA_FLAGS=\"\" -#Allow user to override the current version of the monitor +# Allow user to override the current version of the monitor OVERRIDE_VER=\"\" " > $LOCAL_OVERRIDES @@ -42,12 +42,12 @@ function copy_config_file { } function check_variables { - #Check if CUSTOM_HOME exists + # Check if CUSTOM_HOME exists if ! [ -d "$CUSTOM_HOME" ]; then echo -e "${RED}Please configure your variables first ! (config/local.cfg (based on variables.cfg) --> CUSTOM_HOME & CUSTOM_USER)${NC}"; exit; fi } function prerequisites { - #Prerequisites function + # Prerequisites function echo -e echo -e "${GREEN}Updating system & installing some dependencies...${NC}" echo -e @@ -66,7 +66,7 @@ function prerequisites { # Check if $CUSTOM_HOME exists and if not create it if ! [ -d "$CUSTOM_HOME" ]; then mkdir -p $CUSTOM_HOME; fi - #Limit journalctl size + # Limit journalctl size echo -e echo -e "${GREEN}Limiting journalctl node units logs size...${NC}" echo -e @@ -85,12 +85,12 @@ function paths { } function go_lang { - #Identify machine architecture + # Identify machine architecture ARCH=$(dpkg --print-architecture) - #Check if golang is installed on system + # Check if golang is installed on system if ! [ -x "$(command -v go)" ]; then - #Get the latest version of GO for amd64 & installing it + # Get the latest version of GO for amd64 & installing it echo -e echo -e "${RED}GO is not installed on your system${NC}" echo -e @@ -102,16 +102,16 @@ function go_lang { rm ${GO_LATEST_TESTED}.linux-${ARCH}.tar.gz else - #Check the installed version number + # Check the installed version number GOVERSION=$(go version | awk '{print $3}') if [[ "$GOVERSION" < "${GO_LATEST_TESTED}" ]]; then echo -e echo -e "${RED}GO version is ${CYAN}$GOVERSION${RED} and the best working version is ${CYAN}${GO_LATEST_TESTED}${RED}... ${NC}" - #Detect go install method + # Detect go install method GO_INSTALL_METHOD=$(which go) if [[ "$GO_INSTALL_METHOD" == "/usr/local/go/bin/go" ]]; then - #Installed by scripts. Go ahead and upgrade. + # Installed by scripts. Go ahead and upgrade. echo -e echo -e "${GREEN}Your GO binary will pe upgraded to the minimum required version...${NC}" sudo rm -rf /usr/local/go @@ -134,7 +134,7 @@ function go_lang { } function check_api_limit { - #Check API rate-limit + # Check API rate-limit if [ ${GITHUB_API_LIMIT} -eq ${GITHUB_API_USED} ] && [[ -z "$GITHUBTOKEN" ]]; then echo -e echo -e "${RED}GitHub API Requests Limit for this IP has been reached !${NC}" @@ -150,16 +150,16 @@ function check_api_limit { } function build_monitor { - #Build the monitor from latest tag + # Build the monitor from latest tag echo -e echo -e "${GREEN}Building your monitor binary...${NC}" echo -e - #If OVERRIDE_VER is set SHOWVER should take it into account + # If OVERRIDE_VER is set SHOWVER should take it into account [[ ! -z "$OVERRIDE_VER" ]] && SHOWVER=$(echo ${OVERRIDE_VER#${TMPVAR}}) [[ ! -z "$OVERRIDE_VER" ]] && VERSION=$(echo $OVERRIDE_VER) - #Building the node from the local directory + # Building the node from the local directory cd $WORKDIR echo -e "${GREEN}Compiling binary with version ${CYAN}$VERSION${GREEN}...${NC}" echo -e @@ -202,12 +202,12 @@ echo -e } function systemd { - #Add systemd monitor service & syslogd logging + # Add systemd monitor service & syslogd logging echo -e echo -e "${GREEN}Installing systemd unit for the monitor...${NC}" echo -e - #Create the service file for our node + # Create the service file for our node echo "[Unit] Description=MultiversX keys monitor After=network-online.target @@ -225,7 +225,7 @@ function systemd { [Install] WantedBy=multi-user.target" > mx-chain-keys-monitor.service - #Move files to appropriate locations + # Move files to appropriate locations if [ -f "/etc/systemd/system/mx-chain-keys-monitor.service" ]; then echo -e echo -e "${GREEN}Refreshing mx-chain-keys-monitor service (stop-disable-remove)...${NC}" @@ -238,15 +238,16 @@ function systemd { sleep 1 fi sudo mv mx-chain-keys-monitor.service /etc/systemd/system/ + sudo systemctl daemon-reload sudo systemctl enable mx-chain-keys-monitor.service } function cleanup_files { - #Cleanup Logs + # Cleanup Logs echo -e echo -e "${RED}Erasing previous monitor data (logs)...${NC}" sudo rm -rf $WORKDIR/logs - #recreate erased folders + # recreate erased folders mkdir -p $WORKDIR/{logs} } diff --git a/scripts/config/menu_functions.cfg b/scripts/config/menu_functions.cfg index 0027b21..322450d 100644 --- a/scripts/config/menu_functions.cfg +++ b/scripts/config/menu_functions.cfg @@ -73,14 +73,13 @@ function cleanup { if [ -e /etc/systemd/system/mx-chain-keys-monitor.service ]; then sudo rm /etc/systemd/system/mx-chain-keys-monitor.service; fi if [ -d $CUSTOM_HOME/mx-chain-keys-monitor ]; then sudo rm -rf $CUSTOM_HOME/mx-chain-keys-monitor; fi - #Reload systemd after deleting node units + # Reload systemd after deleting node units sudo systemctl daemon-reload echo -e "${GREEN}Removing paths from .profile on host ${CYAN}$HOST${GREEN}...${NC}" echo -e sed -i 'N;$!P;$!D;$d' ~/.profile - #----------------------------------------------------- if [ -d "$GOPATH/pkg" ]; then sudo rm -rf $GOPATH/pkg; fi ;; @@ -95,9 +94,9 @@ function cleanup { } function get_logs { - #Get journalctl logs from the monitor + # Get journalctl logs from the monitor LOGSTIME=$(date "+%Y%m%d-%H%M") - #Make sure the log path exists + # Make sure the log path exists mkdir -p $CUSTOM_HOME/mx-chain-keys-monitor-logs echo -e @@ -105,7 +104,7 @@ function get_logs { echo -e sudo journalctl --unit mx-chain-keys-monitor >> $CUSTOM_HOME/mx-chain-keys-monitor-logs/mx-chain-keys-monitor-logs.log - #Compress the logs and erase files + # Compress the logs and erase files cd $CUSTOM_HOME/mx-chain-keys-monitor-logs/ && tar -zcvf mx-chain-keys-monitor-logs-$LOGSTIME.tar.gz *.log && rm *.log echo -e echo -e "${GREEN}---> Logs have been stored in the ${CYAN}~/mx-chain-keys-monitor-logs${GREEN} directory...${NC}" diff --git a/scripts/config/variables.cfg b/scripts/config/variables.cfg index 84c30d3..e959096 100644 --- a/scripts/config/variables.cfg +++ b/scripts/config/variables.cfg @@ -3,13 +3,13 @@ set -e #-------------------------------#### DON'T CHANGE THESE, use local.cfg instead ####-------------------------------# -#Color definitions +# Color definitions RED='\x1B[0;31m' CYAN='\x1B[0;36m' GREEN='\x1B[0;32m' NC='\x1B[0m' -#Make sure curl and jq commands exist & if not install them +# Make sure curl and jq commands exist & if not install them if ! [ -x "$(command -v curl)" ]; then sudo apt update; sudo apt install curl -qq -y; fi if ! [ -x "$(command -v jq)" ]; then sudo apt update; sudo apt install jq -qq -y; fi @@ -18,14 +18,14 @@ CUSTOM_USER="ubuntu" GITHUBTOKEN="" MONITOR_EXTRA_FLAGS="" -#Make script aware of its location +# Make script aware of its location SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )" WORKDIR="$( cd "$(dirname "$0")" & cd .. ; pwd -P )" -#Use the latest MultiversX tested GO Lang version +# Use the latest MultiversX tested GO Lang version GO_LATEST_TESTED="go1.20.7" -#Obtain the tag for the latest version node & configs +# Obtain the tag for the latest version node & configs TMPVAR="tags/" if [ -z "$GITHUBTOKEN" ]; then @@ -46,11 +46,11 @@ SCRIPTS_LOCATION=$(pwd) DIRECTORY_NAME=${PWD##*/} ATTENTION=$(echo $ENVIRONMENT | tr [a-z] [A-Z]) -#GitHub API related variables +# GitHub API related variables GITHUB_API_LIMIT=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.limit) GITHUB_API_USED=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.used) GITHUB_API_RESET=$(echo $GITHUB_RATE_LIMIT_RESPONSE | jq -r .resources.core.reset) HUMAN_TIME=$(date -d @"$GITHUB_API_RESET") -#Allow user to override the current version of the monitor +# Allow user to override the current version of the monitor OVERRIDE_VER="" \ No newline at end of file diff --git a/scripts/script.sh b/scripts/script.sh index d1554c8..1509009 100755 --- a/scripts/script.sh +++ b/scripts/script.sh @@ -1,18 +1,18 @@ #!/bin/bash set -e -#Make script aware of its location +# Make script aware of its location SCRIPTPATH="$( cd -- "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 ; pwd -P )" source $SCRIPTPATH/config/variables.cfg source $SCRIPTPATH/config/functions.cfg source $SCRIPTPATH/config/menu_functions.cfg -#See if the user has passed any command line arguments and if not show the menu +# See if the user has passed any command line arguments and if not show the menu if [ $# -eq 0 ] then - show_menu #Show all the menu options + show_menu # Show all the menu options COLUMNS=8 PS3="Please select an action:" From 42d3f71e7d5bf57f809eb9fcc190ac00082be989 Mon Sep 17 00:00:00 2001 From: Iulian Pascalau Date: Tue, 21 May 2024 10:42:49 +0300 Subject: [PATCH 3/3] - fix in readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index df48828..4a94f7d 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ cd ~/mx-chain-keys-monitor-go/scripts # the following init call will create ~/mx-chain-keys-monitor-go/scripts/config/local.cfg file # and will copy the configs from ~/mx-chain-keys-monitor-go/cmd/monitor/config/example to ~/mx-chain-keys-monitor-go/cmd/monitor/config # to avoid github pull problems -scripts.sh init +script.sh init cd config # edit the local.cfg file for the scripts setup nano local.cfg