"Deadline Exceed" Error reported during normal I/O (no flip, no restart)

[yamingk]

Even run CI build, we can see this error, like this test case (there is no flip, no restarts, just pure I/O): https://github.com/eBay/HomeStore/blob/master/src/tests/test_raft_repl_dev.cpp#L618

this is seen both file and real drives

root@test-vol-test-kernel-hdd-001-aj2no:/output# grep Deadline log
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 1 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 2 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 4 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 8 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 16 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 32 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 64 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 128 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 256 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:50-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 512 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded
[05/16/24 16:06:51-07:00] [E] [test_raft_repl_dev_replica_0] [2448] [proto_mesg_factory.cpp:81:log_every_nth] Failed 1024 time(s) in the last 60 seconds to send unidirectional data_service_request to 127.0.0.1:4002, error: Deadline Exceeded