Relayer can still crash upon startup due to bad RPCs

[tkporter]

Problem

Despite most relayer crashes being fixed (the ones to do with contract sync building here #4811), it's still possible to fail e.g. here

hyperlane-monorepo/rust/main/agents/relayer/src/relayer.rs

Lines 138 to 143 in 58425a2

	let mailboxes = settings
	.build_mailboxes(settings.destination_chains.iter(), &core_metrics)
	.await?;
	let validator_announces = settings
	.build_validator_announces(settings.origin_chains.iter(), &core_metrics)
	.await?;

Some additional context:

• build_mailboxes and build_validator_announces (but possibly other calls from the from_settings function as well) iterate the list of configured chains and if any error occurs (such as because of a flaky RPC) it gets propagated, crashing the relayer.
• as more chains are relayed between, the chance of one of the RPCs being flaky increases
• instead of crashing, the relayer should at least log an error
• the relayer should also record a critical error (example here) for that chain if an error occurs, so we alert on it

Solution

We probably don't want to take down everything if one chain is having issues

[kamiyaa]

Talked with @daniel-savu about this

Takeaways:

1. Ensure no unwrap()s in tasks the relayer spawns
2. replace try_join_all with join_all perhaps. Such that 1 failed future does not affect all futures

• potentially have some sort of iterator pattern that yields Ok/Err for these futures
• maybe https://docs.rs/futures/latest/futures/stream/struct.FuturesUnordered.html

3. Unittesting panic!() s in all tasks spawned by relayer to ensure liveliness is not lost when a single task dies