Issue 452: Replication latency metrics and monitoring

[yamingk]

This change is needed when debugging replication long run issues, when grpc complaining "Deadline Exceeded" issues and cause a serious of other issues.

Changes:

1. add different phase of latency to narrow down each phase of latency during a replication request lifecycle.
2. removed the assert to now allowing HS to boot with one HDD.

Testing:

0. Used below command line to verify this change:

Command line: 
1. Locally with file: // <<<< each replica starts with one local file, the --dev_size_mb parameter requires you have enough free space on your local drive. It allows test to run longer without going out of space.
./bin/test_raft_repl_dev --gtest_filter=*Snapshot_and_Compact* --num_io=999999 --override_config nuraft_mesg_config.mesg_factory_config.data_request_deadline_secs:120 --config_path ./config --dev_size_mb=2048600 --snapshot_distance=0  

2. With real drives on real env: // <<<< each replica starts with only one HDD drive, replica-0: /dev/data1, replica-1: /dev/data2, replica-2: /dev/data3
./test_raft_repl_dev --gtest_filter=*Snapshot_and_Compact* --replica_dev_list /dev/data1,/dev/data2,/dev/data3 --num_io=999999 --override_config nuraft_mesg_config.mesg_factory_config.data_request_deadline_secs:120 --config_path ./config/ --snapshot_distance=0

2. All existing test_raft_repl_dev passed

[06/20/24 16:29:37-07:00] [I] [test_raft_repl_dev_replica_2] [229807] [iomgr_timer.cpp:126:operator()] Removing recurring per-thread timer fd 54 device
[       OK ] RaftReplDevTest.GCReplReqs[06/20/24 16:29:37-07:00] [I] [test_raft_repl_dev_replica_2] [229521] [test_raft_repl_dev.cpp:263:TearDown] Waiting for repl dev to get destroyed
 (23537 ms)
[----------] 9 tests from RaftReplDevTest (198435 ms total)

[----------] Global test environment tear-down
[==========] 9 tests from 1 test suite ran. (198435 ms total)
[  PASSED  ] 9 tests.

[xiaoxichen]

lgtm