Make z2 work again - and incidentally, make networks more robust to stalls.

[rrw-zilliqa]

No description provided.

[JamesHinshelwood]

Sorry for taking ages to get to this :(

[JamesHinshelwood]

Kademlia already does this periodically by itself:
https://github.com/libp2p/rust-libp2p/blob/master/protocols/kad/src/behaviour.rs#L990-L996

It also has logic to do this more sanely by triggering a bootstrap when new peers are added and avoid making more than one bootstrap at once.

Does z2 work without this change? Can you see if the automatic bootstrap is working? We can also adjust the automatic_bootstrap_throttle configuration.

[rrw-zilliqa]

z2 doesn't work without this change, sadly...

The logic here is that, at least at startup, a node starts before the bootstrap; it fails to add the peer (because it's not up yet), and drops the bootstrap. Now it won't add the peer again, even if it comes up, and z2 then deadlocks essentially forever.

I'll see if automatic_bootstrap_throttle will work, but if it's currently set to less than a few hours, I suspect it is already triggering. We could perhaps omit the .kademlia_bootstrap(), but the .add_address() definitely isn't optional.

This also relates to one of the crashing bugs which I have utterly failed (sorry!) to tag as such and now can't find, where all peers get dropped and the network then dissociates.

[rrw-zilliqa]

@JamesHinshelwood - OK. I've checked. You can't get away without re-adding the address, but (as expected) kademlia will eventually bootstrap itself automatically.

[github-actions]

Bencher Report

Branch	users/richard/make-z2-work-again
Testbed	self-hosted

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
process-empty/process-empty	📈 view plot 🚷 view threshold	8.46 (-7.82%)	10.40 (81.34%)
produce-full/produce-full	📈 view plot 🚷 view threshold	2,251.10 (+10.46%)	2,632.43 (85.51%)

🐰 View full continuous benchmarking report in Bencher

[JamesHinshelwood]

@shawn-zil Can you check if your change (#2030) also fixes z2 and thus supersedes this PR? Thanks

[shawn-zil]

@shawn-zil Can you check if your change (#2030) also fixes z2 and thus supersedes this PR? Thanks

I can confirm that I am able to deploy the latest main (ae64d06) to perftest using the latest z2 command. So, I will close this PR for now. We can reopen if it necessary.

[rrw-zilliqa]

@shawn-zil - that's not what this PR attempts to fix. It fixes the behaviour when you try to run Zilliqa 2 with the z2 command locally. I'll see if current main works ..

[rrw-zilliqa]

I think that if you run:

rm -rf /tmp/d
./scripts/z2 run /tmp/d 0-4

On current main a8bed8b6 , you get a network which forms but cannot move past the genesis block because none of the peers can see any other peer. If you run the same command with this PR, the network forms and starts producing blocks (judged by grepping the logs with RUST_LOG=debug for the `### proposing' log line).

@shawn-zil - are you seeing something different?

[shawn-zil]

I'm having trouble getting z2 run to work on my setup.
But if the issue is due to libp2p's external address, I wonder if this would help?

zq2/zilliqa/src/p2p_node.rs

Line 136 in be194da

.with_hide_listen_addrs(false)