GRPC crashes if we are doing accounts flush and disconnect the consumer

[gritsly]

We have started the grpc plugin with snapshot_plugin_channel_capacity: 200000000, immediately it was whirligig that connected first to the plugin. So i stopped the whirligig service and it triggered a plugin crash, logs below:

Aug 22 11:11:30 pit17 solana-rpc.sh[52797]: [2024-08-22T11:11:30.156441224Z INFO  solana_runtime::snapshot_bank_utils] Rebuilding bank from full snapshot /solana/snapshots/snapshot/tmp-snapshot-archive-ZvOhIJ/snapshots/285109815/285109815 and incremental snapshot Some("/solana/snapshots/snapshot/tmp-snapshot-archive-h7sCho/snapshots/285132851/285132851")
Aug 22 11:11:58 pit17 solana-rpc.sh[52797]: [2024-08-22T11:11:58.122700777Z INFO  solana_accounts_db::accounts_db] Background account hasher has started
Aug 22 11:12:00 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:00.426197302Z INFO  solana_accounts_db::accounts_db] generating index: 1134/443264 slots... (567/s)
Aug 22 11:12:02 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:02.431009149Z INFO  solana_accounts_db::accounts_db] generating index: 71062/443264 slots... (17765/s)
Aug 22 11:12:04 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:04.432368574Z INFO  solana_accounts_db::accounts_db] generating index: 123658/443264 slots... (20609/s)
Aug 22 11:12:05 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:05.762660084Z INFO  yellowstone_grpc_geyser::grpc] client #0: removed
Aug 22 11:12:05 pit17 solana-rpc.sh[52797]: thread '<unnamed>' panicked at yellowstone-grpc-geyser/src/plugin.rs:135:35:
Aug 22 11:12:05 pit17 solana-rpc.sh[52797]: failed to send message to startup queue: channel closed
Aug 22 11:12:05 pit17 solana-rpc.sh[52797]: stack backtrace:
Aug 22 11:12:06 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:06.436343408Z INFO  solana_accounts_db::accounts_db] generating index: 171518/443264 slots... (21439/s)
Aug 22 11:12:08 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:08.243706329Z INFO  solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=140112862i bg_waiting_percent=2.9386966228485107 bg_throttling_wait_percent=0 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=256i max_in_bin_disk=218584i count_from_bins_disk=140184921i median_from_bins_disk=1035i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=140112862i deletes=0i active_threads=28i items=0i keys=0i ms_per_age=5040i buckets_scanned=8794i flush_scan_us=96i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=8800i disk_index_failed_resizes=112i disk_index_max_size=315818i disk_index_new_file_us=2625844i disk_index_resize_us=43188759i disk_index_flush_file_us=154i disk_index_index_file_size=10721592240i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=31824657i disk_index_flush_mmap_us=7692530i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
Aug 22 11:12:08 pit17 solana-rpc.sh[52797]: [2024-08-22T11:12:08.439119930Z INFO  solana_accounts_db::accounts_db] generating index: 222899/443264 slots... (22289/s)
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    0:     0x7fc77716346c - std::backtrace_rs::backtrace::libunwind::trace::ha637c64ce894333a
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    1:     0x7fc77716346c - std::backtrace_rs::backtrace::trace_unsynchronized::h47f62dea28e0c88d
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    2:     0x7fc77716346c - std::sys_common::backtrace::_print_fmt::h9eef0abe20ede486
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:67:5
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    3:     0x7fc77716346c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hed7f999df88cc644
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:44:22
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    4:     0x7fc77718ae20 - core::fmt::rt::Argument::fmt::h1539a9308b8d058d
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/rt.rs:142:9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    5:     0x7fc77718ae20 - core::fmt::write::h3a39390d8560d9c9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/mod.rs:1120:17
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    6:     0x7fc7771610af - std::io::Write::write_fmt::h5fc9997dfe05f882
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/io/mod.rs:1762:15
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    7:     0x7fc777163254 - std::sys_common::backtrace::_print::h894006fb5c6f3d45
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:47:5
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    8:     0x7fc777163254 - std::sys_common::backtrace::print::h23a2d212c6fff936
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:34:9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:    9:     0x7fc777164857 - std::panicking::default_hook::{{closure}}::h8a1d2ee00185001a
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   10:     0x7fc7771645bf - std::panicking::default_hook::h6038f2eba384e475
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:292:9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   11:     0x7fc777164cd8 - std::panicking::rust_panic_with_hook::h2b5517d590cab22e
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:779:13
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   12:     0x7fc777164b89 - std::panicking::begin_panic_handler::{{closure}}::h233112c06e0ef43e
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:649:13
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   13:     0x7fc777163936 - std::sys_common::backtrace::__rust_end_short_backtrace::h6e893f24d7ebbff8
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   14:     0x7fc777164922 - rust_begin_unwind
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   15:     0x7fc776d0f8f5 - core::panicking::panic_fmt::hbf0e066aabfa482c
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   16:     0x7fc776e4f4b4 - <yellowstone_grpc_geyser::plugin::Plugin as solana_geyser_plugin_interface::geyser_plugin_interface::GeyserPlugin>::update_account::h4edd987b1bdcfb64
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   17:     0x559eb3c7930c - solana_geyser_plugin_manager::accounts_update_notifier::AccountsUpdateNotifierImpl::notify_plugins_of_account_update::hd339c04d54a4ba56
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   18:     0x559eb3c78ccb - <solana_geyser_plugin_manager::accounts_update_notifier::AccountsUpdateNotifierImpl as solana_accounts_db::accounts_update_notifier_interface::AccountsUpdateNotifierInterface>::notify_account_restore_from_snapshot::hb300007ebfde8474
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   19:     0x559eb4f8561a - solana_accounts_db::accounts_db::geyser_plugin_utils::<impl solana_accounts_db::accounts_db::AccountsDb>::notify_account_restore_from_snapshot::hed8ca769fcc0ea4b
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   20:     0x559eb49a0825 - std::sys_common::backtrace::__rust_begin_short_backtrace::he5bfa6bc11f6716c
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   21:     0x559eb49a2c1a - core::ops::function::FnOnce::call_once{{vtable.shim}}::h0389b849a0e92f29
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   22:     0x559eb5a59d65 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc7eafaff61e32df9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   23:     0x559eb5a59d65 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h6ba4a5de48dd2304
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:   24:     0x559eb5a59d65 - std::sys::unix::thread::Thread::new::thread_start::he469335aef763e45
Aug 22 11:12:09 pit17 solana-rpc.sh[52797]:                                at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]:   25:     0x7fc77a636609 - start_thread
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]:                                at /build/glibc-LcI20x/glibc-2.31/nptl/pthread_create.c:477:8
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]:   26:     0x7fc77a3e5353 - clone
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]:                                at /build/glibc-LcI20x/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]:   27:                0x0 - <unknown>
Aug 22 11:12:10 pit17 solana-rpc.sh[52797]: fatal runtime error: Rust cannot catch foreign exceptions
Aug 22 11:12:40 pit17 solana-rpc.sh[52792]: /home/solana/bin/solana-rpc.sh: line 190: 52797 Aborted                 (core dumped) start_solana
Aug 22 11:12:40 pit17 systemd[1975]: solana-rpc.service: Main process exited, code=exited, status=134/n/a
Aug 22 11:12:40 pit17 systemd[1975]: solana-rpc.service: Failed with result 'exit-code'.
Aug 22 11:12:42 pit17 systemd[1975]: solana-rpc.service: Scheduled restart job, restart counter is at 1.
Aug 22 11:12:42 pit17 systemd[1975]: Stopped Solana RPC Node.