Expand metrics on pg_stat_replication to include lag expressed as time.

[ahjmorton]

Expands the metrics exposed from pg_stat_replication to include lag as reported from the wal sender perspective. Also adds a collector for pg_stat_wal_receiver for monitoring from the standby side.

The following examples are from a locally running streaming replica setup and calling postgres_exporter

Primary:

# HELP postgres_stat_replication_flush_lag_seconds flush_lag as reported by the pg_stat_replication view converted to seconds
# TYPE postgres_stat_replication_flush_lag_seconds gauge
postgres_stat_replication_flush_lag_seconds{application_name="walreceiver",client_addr="172.28.0.3",state="streaming",sync_state="sync"} 0.002844
# HELP postgres_stat_replication_lag_bytes delay in bytes pg_wal_lsn_diff(pg_current_wal_lsn(), replay_location)
# TYPE postgres_stat_replication_lag_bytes gauge
postgres_stat_replication_lag_bytes{application_name="walreceiver",client_addr="172.28.0.3",state="streaming",sync_state="sync"} 0
# HELP postgres_stat_replication_replay_lag_seconds replay_lag as reported by the pg_stat_replication view converted to seconds
# TYPE postgres_stat_replication_replay_lag_seconds gauge
postgres_stat_replication_replay_lag_seconds{application_name="walreceiver",client_addr="172.28.0.3",state="streaming",sync_state="sync"} 0.00317
8
# HELP postgres_stat_replication_write_lag_seconds write_lag as reported by the pg_stat_replication view converted to seconds
# TYPE postgres_stat_replication_write_lag_seconds gauge

Standby:

# HELP postgres_wal_receiver_replay_lag_bytes delay in standby wal replay bytes pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn())::float
# TYPE postgres_wal_receiver_replay_lag_bytes gauge
postgres_wal_receiver_replay_lag_bytes{status="streaming"} 0
# HELP postgres_wal_receiver_replay_lag_seconds delay in standby wal replay seconds EXTRACT (EPOCH FROM now() - pg_last_xact_replay_timestamp()
# TYPE postgres_wal_receiver_replay_lag_seconds gauge
postgres_wal_receiver_replay_lag_seconds{status="streaming"} 0

References

• Offical documentation for version 10 of PG on pg_stat_replication

[rnaveiras]

hey @ahjmorton

Firstly, thank you so much for this. I will review the PR in detail in the next few days.

I think you can drop support for Postgres older than 10, as 9.6 reached the end of life. See https://www.postgresql.org/support/versioning/

That will make the changes a bit easier, I think

[rnaveiras]

@ahjmorton I didn't have time to go into details here, but could you share more details about these metrics disappearing?

Later you make a conditional have a value to report these metrics. We should avoid that, as metrics that disappear from Prometheus are complex to deal with

https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

[ahjmorton]

@ahjmorton I didn't have time to go into details here, but could you share more details about these metrics disappearing?

If the WAL sender process didn't have an activity for some period of time (not sure how long, might need to look at source) then Postgres would null out those "*_lag" fields.

Later you make a conditional have a value to report these metrics. We should avoid that, as metrics that disappear from Prometheus are complex to deal with

I was trying to avoid reporting metrics for nodes that didn't have a WAL sender process however those would be eliminated by the query as they wouldn't have a row in pg_stat_replication.
Going to change this to default to zero in case of null.

[ttamimi]

I tested this branch on my local machine with a streaming replication setup, and I loaded it with tens of thousands of records of test data. I can confirm that the five new metrics introduced in this PR all appeared for me with non-zero values.

Metrics from the primary PostgreSQL instance

postgres_stat_replication_flush_lag_seconds{
  application_name="walreceiver",
  client_addr="<nil>",
  instance="localhost:9187",
  job="postgresql",
  state="streaming",
  sync_state="async"
}
postgres_stat_replication_replay_lag_seconds{
  application_name="walreceiver",
  client_addr="<nil>",
  instance="localhost:9187",
  job="postgresql",
  state="streaming",
  sync_state="async"
}
postgres_stat_replication_write_lag_seconds{
  application_name="walreceiver",
  client_addr="<nil>",
  instance="localhost:9187",
  job="postgresql",
  state="streaming",
  sync_state="async"
}

Metrics from the replica

postgres_wal_receiver_replay_lag_bytes{
  instance="localhost:9188",
  job="postgresql",
  status="streaming"
}
postgres_wal_receiver_replay_lag_seconds{
  instance="localhost:9188",
  job="postgresql",
  status="streaming"
}

[ttamimi]

Where is pg_log_location_diff referenced? We aren't using it anywhere in this code, are we? Maybe a third party library? It is definitely getting queried though. I tested the exporter with a streaming replication setup on my machine and I saw the following errors in the logs of the primary PostgreSQL instance:

2022-06-14 10:32:45.934 BST [94070] STATEMENT:  select * from pg_log_location_diff;
2022-06-14 10:32:47.403 BST [94070] ERROR:  relation "pg_log_location_diff" does not exist at character 15

Even Google has no clue what pg_log_location_diff is!!

[ahjmorton]

Hey @rnaveiras and @ttamimi . I'm happy to leave this PR open and work on getting the metrics in there. Reckon it's worth me taking a look? Realise it's been a long time