Dispatcher error handler and retrier features

[kamikazechaser]

All the below depend on the correct RPC response. An error handler should be written to correctly propagate the error type from the RPC node. This is a bit difficult for sidechains e.g. Celo which implement custom response error messages.

• RPC error handler with core error return types

Gas bump

Resign a transaction and bump the gas price by 15%. On the otx table, the old transaction should be invalidated and a new one created in a single tx. Gas limit issues can also be included here but ideally, we set a generous gas limit that on the gas oracle itself.

• Invalidated flag on otx table
• Resigner

Low nonce issues

Ideally this kind of error should be near-impossible to get if the signer implementation is correct as we maintain the nonce sequence offline. However, in the rare event that it does occur, a special noop tx which attempt a random low gas sign should be attempted to close the gap.

• Special noop tx tyoe signer
• Periodic health check to chech if a tx is mined/reverted. If not, we may have any of the issues described here.

Out of balance

Ideally, we never face this issue. A lock should be applied on the account to prevent further tx sign requests. There should always be a pre-check on the dispatcher before it is attempted.

• Gas lock implementation
• Balance check middleware.

[kamikazechaser]

A retrier logger and health check has been added to confirm if the dispatch error handler works in the first place before we can proceed to write complex retrying logic.

[kamikazechaser]

When performing an invalidation:

• Set true on replaced
• Use same nonce

Nonce gaps should be semi-automatic. Meaning that a manual range should be provided by the admin to close any gaps. This prevents accidental overwriting of signed txs. Draw inspiration from Louis's old unlocker tool.

[kamikazechaser]

Gas bump issue is blocked because the cel2 node does not reject gas prices lower than celo floor. We can enforce a gasFloor ourselves in the gas oracle as a workaround for now. But this means that until the Celo team investigates and fixes the issue, we cannot test dispatch issues around low gas prices.