need solutions for leader replacement / failover

[jchappelow]

The network will be quite fragile if a leader needs to be replaced i.e. it cannot be brought back with the same private key. We need at least a simple solution for final release IMO.

Deterministically picking a replacement leader from the validator set should be relatively straightforward. For instance, it can be based on last block's app_hash. Deciding when to proceed with replacement is harder.

Alternatively, the new leader can be picked an specified with admin commands in an election process e.g. admin command: validators promote [desired new leader id].

Other challenges?

[jchappelow]

Say leader vaporizes, and there can be no blocks to make an on-chain election, I can see an admin command to force a leader change. For a validator that runs the command, they would accept block proposals signed by a new leader. When enough validators do the same, the chain moves again. I think this would necessitate a special case where the node itself becomes the leader, which would require a little work to have CE change roles on the fly.

To support sync from genesis on such a chain, there would need to be some other magic however. I'm thinking some p2p comms to propagate aggregate validators' approvals, which would then have to be recorded in the first block from the new leader. It would necessitate a special field in the block header. This could be subject to a supermajority as it amounts to a coup.