bug: Operation status labels are reset across relayer restarts

[daniel-savu]

Problem

Is your feature request related to a problem? Please describe.

The relayer has an enum of possible statuses a pending operation (i.e. pending message) can have here. New statuses are assigned based on where a message is in its submission lifecycle. For instance, all messages are created with a FirstPrepareAttempt status, then if metadata is fetched successfully (e.g. validator signatures for the multisig ISM) the message's status is updated to ReadyToSubmit here, then if the transaction on the destination chain is successfully dispatched the status is updated to Confirm(SubmittedBySelf) here, and after the message is finalized on the destination it is removed from in-memory queues.

This status is also persisted to the local db when it is updated so upon restarting all old messages should be loaded into the prep queue with their old statuses (such as Retry(GasPaymentNotFound)).

The problem is that relayer restarts causes all old messages to be loaded into the prep queue with a FirstPrepareAttempt status, as can be observed below for celo. The red colour chunk is always reset upon restart (you may even need to zoom in to see it), although all of those messages showing up as green have been attempted before.

https://abacusworks.grafana.net/goto/8cJ2XfIHg?orgId=1

Solution

Statuses should stay consistent across restarts by storing and retrieving them from the local db. Understand and fix why this is not the case