From 88525ebf5e43cc86aea66dc75ec83db58233883b Mon Sep 17 00:00:00 2001 From: Tao Liu Date: Thu, 5 Sep 2024 15:49:07 +1200 Subject: [PATCH] Introduce vmcore creation notification to kdump Motivation ========== People may forget to recheck to ensure kdump works, which as a result, a possibility of no vmcores generated after a real system crash. It is unexpected for kdump. It is highly recommended people to recheck kdump after any system modification, such as: a. after kernel patching or whole yum update, as it might break something on which kdump is dependent, maybe due to introduction of any new bug etc. b. after any change at hardware level, maybe storage, networking, firmware upgrading etc. c. after implementing any new application, like which involves 3rd party modules etc. Though these exceed the range of kdump, however a simple vmcore creation status notification is good to have for now. Design ====== Kdump currently will check any relating files/fs/drivers modified before determine if initrd should rebuild when (re)start. A rebuild is an indicator of such modification, and kdump need to be rechecked. This will clear the vmcore creation status specified in $VMCORE_CREATION_STATUS. Vmcore creation check will happen at "kdumpctl (re)start/status", and will report the creation success/fail status to users. A "success" status indicates previously there has been a vmcore successfully generated based on the current env, so it is more likely a vmcore will be generated later when real crash happens; A "fail" status indicates previously there was no vmcore generated, or has been a vmcore creation failed based on current env. User should check the 2nd kernel log or the kexec-dmesg.log for the failing reason. $VMCORE_CREATION_STATUS is used for recording the vmcore creation status of the current env. The format will be like: success 1718682002 Which means, there has been a vmcore generated successfully at this timestamp for the current env. Usage ===== [root@localhost ~]# kdumpctl restart kdump: kexec: unloaded kdump kernel kdump: Stopping kdump: [OK] kdump: kexec: loaded kdump kernel kdump: Starting kdump: [OK] kdump: Notice: No vmcore creation test performed! [root@localhost ~]# kdumpctl test [root@localhost ~]# kdumpctl status kdump: Kdump is operational kdump: Notice: Last successful vmcore creation on Tue Jun 18 16:39:10 CST 2024 [root@localhost ~]# kdumpctl restart kdump: kexec: unloaded kdump kernel kdump: Stopping kdump: [OK] kdump: kexec: loaded kdump kernel kdump: Starting kdump: [OK] kdump: Notice: Last successful vmcore creation on Tue Jun 18 16:39:10 CST 2024 The notification for kdumpctl (re)start/status can be disabled by setting VMCORE_CREATION_NOTIFICATION in /etc/sysconfig/kdump === v3 -> v2: Always mount $VMCORE_CREATION_STATUS(/var/crash/vmcore-creation.status)'s device for 2nd kernel, in case /var is a seperate device than rootfs's device. v4 -> v3: Add "kdumpctl test" as the entrance for performing the kdump test. v5 -> v4: Fix the mounting failure issue in fadump. v6 -> v5: Add new argument as customized mount point for add_mount/to_mount. v7 -> v6: a. Code refactoring based on Philipp's suggestion. b. Only mount $VMCORE_CREATION_STATUS(/var/crash/vmcore-creation.status)'s device when needed. c. Add "--force" option for "kdumpctl test", to support the automation test script QE may perform. d. Add check in "kdumpctl test" that $VMCORE_CREATION_STATUS can only be on local drive. v8 -> v7: a. Rebased the patch on top of upstream commit e2b8463b5. b. Code refactoring based on Philipp's suggestion. c. Updated the "test" entry of kdumpctl.8. Signed-off-by: Tao Liu --- dracut/99kdumpbase/kdump.sh | 11 +++++ dracut/99kdumpbase/module-setup.sh | 1 + gen-kdump-sysconfig.sh | 4 ++ kdump-lib-initramfs.sh | 33 +++++++++++++++ kdumpctl | 67 +++++++++++++++++++++++++++++- kdumpctl.8 | 10 +++++ mkdumprd | 16 ++++++- 7 files changed, 139 insertions(+), 3 deletions(-) diff --git a/dracut/99kdumpbase/kdump.sh b/dracut/99kdumpbase/kdump.sh index dea83ee4..e35a76d4 100755 --- a/dracut/99kdumpbase/kdump.sh +++ b/dracut/99kdumpbase/kdump.sh @@ -308,11 +308,22 @@ do_final_action() { } do_dump() { + if [ -d /vmcorestatus ]; then + _vmcore_creation_status="/vmcorestatus/$VMCORE_CREATION_STATUS" + else + _vmcore_creation_status="/sysroot/$VMCORE_CREATION_STATUS" + fi + + set_vmcore_creation_status 'clear' "$_vmcore_creation_status" + eval "$DUMP_INSTRUCTION" _ret=$? if [ $_ret -ne 0 ]; then + set_vmcore_creation_status 'fail' "$_vmcore_creation_status" derror "saving vmcore failed" + else + set_vmcore_creation_status 'success' "$_vmcore_creation_status" fi return $_ret diff --git a/dracut/99kdumpbase/module-setup.sh b/dracut/99kdumpbase/module-setup.sh index 50dd67cc..05019453 100755 --- a/dracut/99kdumpbase/module-setup.sh +++ b/dracut/99kdumpbase/module-setup.sh @@ -1095,6 +1095,7 @@ install() { inst "/usr/bin/printf" "/sbin/printf" inst "/usr/bin/logger" "/sbin/logger" inst "/usr/bin/chmod" "/sbin/chmod" + inst "/usr/bin/dirname" "/sbin/dirname" inst "/lib/kdump/kdump-lib-initramfs.sh" "/lib/kdump-lib-initramfs.sh" inst "/lib/kdump/kdump-logger.sh" "/lib/kdump-logger.sh" inst "$moddir/kdump.sh" "/usr/bin/kdump.sh" diff --git a/gen-kdump-sysconfig.sh b/gen-kdump-sysconfig.sh index e436d4b0..983cf001 100755 --- a/gen-kdump-sysconfig.sh +++ b/gen-kdump-sysconfig.sh @@ -47,6 +47,10 @@ KDUMP_IMG="vmlinuz" #What is the images extension. Relocatable kernels don't have one KDUMP_IMG_EXT="" +# Enable vmcore creation notification by default, disable by setting +# VMCORE_CREATION_NOTIFICATION="" +VMCORE_CREATION_NOTIFICATION="yes" + # Logging is controlled by following variables in the first kernel: # - @var KDUMP_STDLOGLVL - logging level to standard error (console output) # - @var KDUMP_SYSLOGLVL - logging level to syslog (by logger command) diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index f7b58a2c..67c87ba2 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -9,6 +9,7 @@ KDUMP_CONFIG_FILE="/etc/kdump.conf" FENCE_KDUMP_CONFIG_FILE="/etc/sysconfig/fence_kdump" FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send" LVM_CONF="/etc/lvm/lvm.conf" +VMCORE_CREATION_STATUS="/var/crash/vmcore-creation.status" # Read kdump config in well formated style kdump_read_conf() @@ -171,3 +172,35 @@ kdump_get_ip_route_field() { echo "$1" | sed -n -e "s/^.*\<$2\>\s\+\(\S\+\).*$/\1/p" } + +# $1: success/fail/clear +# $2: status_file +set_vmcore_creation_status() +{ + _status=$1 + _status_file=$2 + _dir=$(dirname "$_status_file") + + [[ -d "$_dir" ]] || mkdir -p "$_dir" + + _mnt_op=$(get_mount_info OPTIONS target "$_dir" -f) + case $_mnt_op in + ro*) + dinfo "remounting the vmcore status target in rw mode." + mount -o remount,rw "$(findmnt -n -o TARGET --target $_dir)" + ;; + esac + + case "$_status" in + success | fail) + dinfo "saving vmcore status file to $_status_file" + echo "$_status $(date +%s)" > "$_status_file" + ;; + clear) + rm -f "$_status_file" + ;; + *) + return + esac + sync -f "$_dir" +} diff --git a/kdumpctl b/kdumpctl index 612a159a..82b7fb15 100755 --- a/kdumpctl +++ b/kdumpctl @@ -185,6 +185,8 @@ rebuild_initrd() else rebuild_kdump_initrd fi + + set_vmcore_creation_status 'clear' "$VMCORE_CREATION_STATUS" } #$1: the files to be checked with IFS=' ' @@ -1043,6 +1045,7 @@ start() start_dump || return dinfo "Starting kdump: [OK]" + check_vmcore_creation_status return 0 } @@ -1653,6 +1656,63 @@ _should_reset_crashkernel() { [[ $(kdump_get_conf_val auto_reset_crashkernel) != no ]] && systemctl is-enabled kdump &> /dev/null } +check_vmcore_creation_status() +{ + local _status _timestamp _status_date + + [[ ${VMCORE_CREATION_NOTIFICATION,,} == "yes" ]] || return + + if [[ ! -s $VMCORE_CREATION_STATUS ]]; then + dwarn "Notice: No vmcore creation test performed!" + return + fi + + read -r _status _timestamp < "$VMCORE_CREATION_STATUS" + _status_date="$(date -d "@$_timestamp")" + + if [[ "$_status" == "success" ]]; then + dinfo "Notice: Last successful vmcore creation on $_status_date" + else + dwarn "Notice: Last NOT successful vmcore creation on $_status_date" + fi +} + +kdump_test() +{ + local _dir + + if ! is_kernel_loaded "$DEFAULT_DUMP_MODE"; then + derror "Kdump needs be operational before test." + exit 1 + fi + + _dir=$(dirname "$VMCORE_CREATION_STATUS") + if ! [[ -d "$_dir" ]]; then + derror "Vmcore status dir $_dir not exist." + exit 1 + fi + + if ! lsblk $(get_mount_info SOURCE target "$_dir") > /dev/null; then + derror "$VMCORE_CREATION_STATUS must on local drive" + exit 1 + fi + + if [[ ! "$1" == "--force" ]]; then + read -p "DANGER!!! Will perform a kdump test by crashing the system, proceed? (y/N): " input + case $input in + [Yy] ) + dinfo "Start kdump test..." + ;; + * ) + dinfo "Operation cancelled." + exit 0 + ;; + esac + fi + set_vmcore_creation_status 'clear' "$VMCORE_CREATION_STATUS" + echo c > /proc/sysrq-trigger +} + main() { # Determine if the dump mode is kdump or fadump @@ -1684,6 +1744,7 @@ main() EXIT_CODE=3 ;; esac + check_vmcore_creation_status exit $EXIT_CODE ;; reload) @@ -1728,8 +1789,12 @@ main() reset_crashkernel_for_installed_kernel "$2" fi ;; + test) + shift + kdump_test "$@" + ;; *) - dinfo $"Usage: $0 {estimate|start|stop|status|restart|reload|rebuild|reset-crashkernel|propagate|showmem}" + dinfo $"Usage: $0 {estimate|start|stop|status|restart|reload|rebuild|reset-crashkernel|propagate|showmem|test}" exit 1 ;; esac diff --git a/kdumpctl.8 b/kdumpctl.8 index 29a6119f..f6a7070c 100644 --- a/kdumpctl.8 +++ b/kdumpctl.8 @@ -70,6 +70,16 @@ Note: The memory requirements for kdump varies heavily depending on the used hardware and system configuration. Thus the recommended crashkernel might not work for your specific setup. Please test if kdump works after resetting the crashkernel value. +.TP +.I test [--force] +Test the kdump by actually trigger the system crash & dump, and check if a +vmcore can really be generated successfully based on current config and +environment. After system reboot back to normal, check the test result +by "kdumpctl status". + +If the optional parameter [--force] is provided, there will be no interact +before triggering the system crash. Dangerous though, this option is meant +for automation testing. .SH "SEE ALSO" .BR kdump.conf (5), diff --git a/mkdumprd b/mkdumprd index 24688ecd..19333a0c 100644 --- a/mkdumprd +++ b/mkdumprd @@ -73,9 +73,10 @@ has_dracut_module() # caller should ensure $1 is valid and mounted in 1st kernel to_mount() { - local _target=$1 _fstype=$2 _options=$3 _sed_cmd _new_mntpoint _pdev + local _target=$1 _fstype=$2 _options=$3 _new_mntpoint=$4 + local _sed_cmd _pdev - _new_mntpoint=$(get_kdump_mntpoint_from_target "$_target") + _new_mntpoint="${_new_mntpoint:-$(get_kdump_mntpoint_from_target "$_target")}" _fstype="${_fstype:-$(get_fs_type_from_target "$_target")}" _options="${_options:-$(get_mntopt_from_target "$_target")}" _options="${_options:-defaults}" @@ -429,4 +430,15 @@ if ! is_fadump_capable; then fi fi +status_target=$(get_target_from_path $(dirname "$VMCORE_CREATION_STATUS")) + +if [[ $(get_root_fs_device) != "$status_target" ]]; then + new_mntpoint=$(echo /vmcorestatus/$(get_mntpoint_from_target "$status_target") \ + | tr -s "/") + add_mount "$status_target" "" "" "$new_mntpoint" +elif ! is_fadump_capable && \ + ! [[ ${dracut_args[@]} == *"$(kdump_get_persistent_dev $status_target)"* ]]; then + add_mount "$status_target" +fi + dracut "${dracut_args[@]}" "$@"