Investigate slow full sync

[corverroos]

Problem to Solve

Reports from operators indicate that syncing omega can take more than 24 hours. Our join-omega-nightly also timed out after 6 hours.

We need to figure out why syncing is slow, or more precisely, what can one expect in terms of blocks/s when syncing? We also need to figure out of this is impacted by peers? Are some peers slower than others?

Proposed Solution

Pushing metrics to grafana is a good idea, but metrics unfortunately don’t show which peers we are syncing from.

I think that would be the next thing to do. During join-omega-nightly somehow figure out what peers we are syncing from and the network bandwidth being used from that peer (network bandwidth per IP).