Dead cluster member

[sp1ff]

Hi, not sure if I'm in the right place.
I'm experimenting with openraft & running a little test cluster. I stood up three nodes & formed the cluster (much along the lines of test-cluster.sh in the sample app). Leader was elected, I could write data, all was well.
Then I killed the leader to see what would happen (i.e. kill -9). The other two elected a leader and the cluster continues to take writes-- cool.
My question is, how do I re-add the first node, once I start it again? Pulling metrics from the remaining nodes shows that they still think the dead node is part of the cluster, which confuses me. I mean, they knew that node died and responded by electing a new leader... why didn't they eject it from the cluster?
Re-starting the dead node starts it in learner mode-- I get that. But adding it as a learner to the leader doesn't seem to work?

[drmingdrmer]

Hi, not sure if I'm in the right place.

Yes, it's the right place:)

My question is, how do I re-add the first node, once I start it again?

Just restart the node with the same store.
Raft requires its store to be persistent. Although in this example for simplicity, it just uses an in-memory store for testing.
There is a sample that restarts node: https://github.com/datafuselabs/openraft/blob/762c5a32746895463e68529adf8830ecbe058e92/openraft/tests/append_entries/t30_append_inconsistent_log.rs#L90-L94

Pulling metrics from the remaining nodes shows that they still think the dead node is part of the cluster, which confuses me.
I mean, they knew that node died and responded by electing a new leader... why didn't they eject it from the cluster?

This is a regular assumption about for a distributed consensus protocol:
The cluster always tries its best to form a majority with all live nodes. Nodes go down but are still expected to go back online sometime in the future.
A node will not be used only when the administrator informs the cluster to remove it, by sending a change_membership request to the leader.

Re-starting the dead node starts it in learner mode-- I get that. But adding it as a learner to the leader doesn't seem to work?

The dead node is still a member of the cluster.
The cluster will reject re-adding it as learner.