Significant fixes to suspected failure modes for builder

[nyospe]

Closes #173

This PR:

Prevents scenarios where the highest view builder is removed, and guarantees that the highest view builder task will always handle incoming proposals if there is not a valid parent builder task.

Key places to review:

Everything. But especially the changes to remove_handles in service.rs

Testing:

This needs to be incorporated into the builder, and a deployment using this should no longer fall over almost immediately.

[jbearer]

This generally makes sense to me, but I'd like @QuentinI or someone more familiar with the builder than me to approve.

I do have one question: why, when we don't have a matching builder state for a proposal, do we default to the highest builder state? It seems like it should be the highest builder state less than the proposal. Is there a code path where we end up processing a proposal with a builder state that is newer than it, and what are the implications of this?

[nyospe]

This generally makes sense to me, but I'd like @QuentinI or someone more familiar with the builder than me to approve.

Added @QuentinI, but I'm afraid that most of this was entirely written by Himanshu, so I'm not sure anyone has deep familiarity, aside from me.

I do have one question: why, when we don't have a matching builder state for a proposal, do we default to the highest builder state? It seems like it should be the highest builder state less than the proposal. Is there a code path where we end up processing a proposal with a builder state that is newer than it, and what are the implications of this?

It does make a certain sense for the highest builder state with a parent less than the proposal's parent view (we can call this highest builder state less than the proposal for short) to handle it, but a) we don't have a guarantee that such a state exists, absent the proposal's parent, and b) the logic for each builder state's task to determine whether it would be the correct task to handle this proposal, without ever either having all tasks ignore the proposal, or having multiple tasks branch from the same proposal, would be rather cumbersome, and potentially fraught with race conditions. And both the highest builder state overall and the highest builder state with a parent less than the proposal's parent view have drawbacks. The highest overall may result in some transactions that weren't included due to reorg being permanently dropped. The highest builder state less than the proposer will result in duplicate transactions.

[jbearer]

Ok. Given no one really has expertise with this code, I will approve based on my best-effort understanding

[QuentinI]

As Nathan mentioned, I'm not familiar with this code either, but to my understanding this looks good FWIW