A script to gather valuable data for providing to support engineer
This script is mostly intended as tool for support engineers. If a user reports a problem, the output of the scripts should cover the most common questions a support engineer will have.
The intention is not to collect as much data as possible but collect what is helpful in most cases and providing an overview whithout drowning support in too much data.
Please note: Even though this script is meant to just collect really useful data, it does still collect a lot. Browse through any output before you send it so that you don't send any confidential data without noticing.
Please note as well: The main intention of the script is to provide data when asked for it. While it might be helpful when working with commercial support, sending unwanted big blobs of data into community channels might lower your chances of getting help. Always try the troubleshooting section of the Icinga 2 documentation before requesting any community help. The upcoming "suggestion mode" of this script might help with troubleshooting as well.
Later versions might include options to collect this "usually helpful" set of data or lots of data for deep diving (e.g. whole logfiles, etc.)
This script has several different modes:
- Default mode: Give a quick overview over your setup for professional or community support
- Tarball mode: Create a tarball of the configuration of your setup for thorough investigation by professional support
- Suggestion mode: Search things to improve and possible problems and give advice (still to be implemented)
If your setup consists of multiple hosts you can run on it on every node to get a more thorough overview. If you have e.g. dedicated nodes for graphing the script will work and only analyse the tools you installed on the node. (Although details about Graphers are still missing)
Just use the script without any parameter.
# ./icinga-diagnostics.sh
Just run the script with the -t parameter.
# ./icinga-diagnostics.sh -t
Be aware that this collects your whole configuration including passwords, etc. Think before you send this to someone!
The script will thell you where you can find your tarball when it's finished. You can still remove data from the tarball before sending it to someone.
- -z : Calculate zones with their endpoints (very time consuming on large setups)
- -g : add
gdboutput
gdb Output is only useful for debugging problems in the Icinga 2 binary, not for configuration problems. For gdb to work you have to install gdb and the "degbugging symbols" for all the programs you want to have their information added. e.g. for Icinga 2 the corresponding package is icinga2-debuginfo on RedHat/CentOS or icinga2-dbg on Debian or Ubuntu. In addition you will need the icinga-gdb file containing the gdb commands in the directory where icinga2-diagnostics resides.
In the folder debug-configuration you find a sample configuration file you can place in a zone where you have problems with executing checks with Icinga. This will create some services that check for the presence of some script interpreters including version and include paths on the host. You will get extra services showing you the user and other data about this specific host.
This is especially helpful if you have problems with finding the reason why a specific check is not able to be executed. A common situation is that you can run a check as root or icinga user via shell but the Icinga daemon can't.
This configuration is not intended to be running all the time. Just copy it into to zone you want to debug and when you're finished, remove it from Icinga.
The script currently works on the following operating systems:
- RHEL/CentOS/Oracle Linux/Scientific Linux
- SLES (thanks, wnieder)
- FreeBSD (thanks, @larsengels)
- Debian (still some functionality is missing)
The script currently works with the following software components:
- Icinga 2
- Icinga Web 2
The script should work on all Operating systems supported by Icinga 2
The script should gather informations about the most common addons to Icinga 2 and collect data relevant to monitoring setups.
e.g.
- MariaDB/MySQL
- PostgresSQL
- Graphite
- PNP4Nagios
- InfluxDB
- Elasticsearch
- Logstash
This is an abrreviated output from a run on a Icinga 2 Vagrant box
# ./icinga-diagnostics.sh
### Icinga 2 Diagnostics ###
# Version: 0.0
# Run on icinga2 at Thu Jan 11 18:02:14 UTC 2018
Running as root
## OS ##
OS Version: CentOS Linux release 7.4.1708 (Core)
Hypervisor: Running virtually on a VirtualBox hypervisor
CPU cores: 2
RAM: 1.8G
SELinux: Permissive
Firewall: inactive
# Icinga 2 #
## Packages: ##
Icinga 2 Version : 2.8.0.187.g025abc3
Done checking packages. See Anomaly section if something odd was found.
Features:
Disabled features: command compatlog elasticsearch gelf influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker debuglog graphite ido-mysql mainlog notification
Check intervals:
88 * check_interval = 300, Host
1 * check_interval = 60, Host
113 * check_interval = 60, Service
13 * check_interval = 30, Service
5 * check_interval = 5, Service
3 * check_interval = 300, Service
Used commands (numbers are relative to each other, not showing configured objects):
28093 /etc/icinga2/scripts/mail-service-notification.sh
14120 /usr/lib64/nagios/plugins/check_mysql_health
10146 /usr/lib64/nagios/plugins/check_http
6428 /usr/lib64/nagios/plugins/check_ping
4396 /usr/lib64/nagios/plugins/check_load
3415 /usr/lib64/nagios/plugins/check_ssh
1758 /usr/lib64/nagios/plugins/check_dns
1758 /usr/lib64/nagios/plugins/check_disk
880 /usr/lib64/nagios/plugins/check_procs
879 /usr/lib64/nagios/plugins/check_users
879 /usr/lib64/nagios/plugins/check_swap
528 /usr/bin/icingacli
information/cli: Icinga application loader (version: v2.8.0-187-g025abc3)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: icinga2
warning/globals.getHostGeoLocation: Cannot find 'be' in GeoLocationShort
warning/ApplyRule: Apply rule 'many-dummy' (in /etc/icinga2/demo/many.conf: 36:1-36:39) for type 'Notification' does not match anywhere!
warning/ApplyRule: Apply rule 'many-dummy' (in /etc/icinga2/demo/many.conf: 41:1-41:42) for type 'Notification' does not match anywhere!
warning/ApplyRule: Apply rule 'many-test-0' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'many-test-1' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere!
warning/ApplyRule: Apply rule 'many-test-2' (in /etc/icinga2/demo/many.conf: 16:3-16:32) for type 'Service' does not match anywhere!
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 4 ApiUsers.
information/ConfigItem: Instantiated 2 FileLoggers.
information/ConfigItem: Instantiated 28 Notifications.
information/ConfigItem: Instantiated 1 UserGroup.
information/ConfigItem: Instantiated 2 Users.
information/ConfigItem: Instantiated 203 CheckCommands.
information/ConfigItem: Instantiated 3 NotificationCommands.
information/ConfigItem: Instantiated 4 HostGroups.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 89 Hosts.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 134 Services.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).
# Icinga Web 2 #
Packages:
icingaweb2-2.5.0.2.g890013c-0.2017.11.28+2.el7.icinga.noarch
package php is not installed
httpd-2.4.6-67.el7.centos.6.x86_64
Icinga Web 2 Modules:
MODULE VERSION STATE DESCRIPTION
businessprocess 2.1.0 enabled A Business Process viewer and modeler
cube 1.0.1 enabled Cube for Icinga Web 2
director 1.3.2 enabled Director - Config tool for Icinga 2
doc 2.5.0 enabled Documentation module
grafana 1.1.10 enabled Grafana - A perfdata visualisation module
map 1.0.4 enabled Map - Visualize your hosts and service status
monitoring 2.5.0 enabled Icinga monitoring module
businessprocess via git - "00e2f1886a9b07244e8dad237776b629fad59c0a"
cube via git - "7ba3feb71601fd2433e0b787ea87dddf53878e49"
director via git - "c4a97692df23e428a2eb6f1be41a7c25ea7b19a4"
doc via release archive/package
grafana via git - "5ba8995a15fb4b1e72232e3dffd4b45c6bb89ab8"
map via git - "ec7fed3e4085f98f6f30f3557d85d5ed498e2a7d"
monitoring via release archive/package
Icinga Web 2 commandtransport configuration:
[icinga 2 api localhost]
transport = "api"
host = "localhost"
port = "5665"
username = "icingaweb2"
password = MASKED
Director is release 1.3.2
Director was installed as a git clone
# Anomalies found #
* Director is installed but no release archive was used for installation. (Please note that it still could the code of a release)