Track replication lag in otel metrics and log warning when it gets too high

[KyleAMathews]

We really need visibility into this.

[alco]

@KyleAMathews

@kevin-dp and I have been discussing the best way to measure the "replication lag". There are multiple options:

• if we're interested in the amount of WAL that's kept around between the replication slot's starting LSN and the latest LSN in Postgres, we can track that by periodically querying Postgres for its current LSN and comparing that to the slot's LSN
• alternatively, we could measure the difference between the commit time of the transaction that's being processed and the time it gets written to the shape log or served to a client. But here we'd have to also take into account the clock difference between Postgres and the Electric instance

Regarding the warning, can that be setup in Honeycomb? Or do we want a configuration option for Electric to specify when it needs to log. I imagine the threshold will be different for different clients based on their database write rate and other parameters.

[balegas]

Why not do both? I think it is useful to know the amount of bytes that are pending in the WAL and what is the latency for electric to write a transaction into the log

[KyleAMathews]

Yeah both sound great. I didn't know we could get the actual time diff so didn't mention it but that'd be great to have as well.

Yeah warnings from your otel collector is a lot more flexible — people can already set warnings from postgres data directly. We can potentially add warnings directly in the Electric logs as well but a good sequencing of work is to first gather the data and then later decide exactly how to communicate it.