Flaky tests in hercules

[koxu1996]

It seems that tests are failing in Hercules randomly.

[marijanp]

Could you provide a list which ones are flaky?
Also note that our CI machine is not very powerful, pushing to and updating multiple PR's will cause the system load to go above 300% which could cause some runtime delays with tests that spawn a casper network and await only a limited time for results until they fail/

[koxu1996]

E2E tests are sometimes failing with Success key not found in JSON.

For example job #2179 (commit 2eda67a) failed first, then succeded when I trigerred "re-run effects and failures":

[marijanp]

Yeah, if you scroll up in the logs you will see that the cctl service didn't start properly. cctl is a oneshot and this

kairos/kairos-test-utils/src/cctl.rs

Line 153 in 9519070

let timed_out = start_time.elapsed().as_secs() > 90;

check will make it fail to start. This is due to the high demand of ressources especially if there are many builds running at the moment (We have a notification stream on zulip which reports the current system load of the CI machine).

The solution is to increase that timeout that was linkem, or reduce the amount of concurrent builds.