Debug PR #2028

[DonggeLiu]

Debug two new errors observed in #2028 with the logs from #2038.

Error 1: cloud build timeout

Error 2: Fuzz target binary not found.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-libafl --fuzzers libafl

[DonggeLiu]

Experiment 2024-08-25-2039-libafl data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-libafl-1 --fuzzers libafl

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-25-2039-bases-1 --fuzzers afl aflplusplus libfuzzer

[DonggeLiu]

Experiment 2024-08-25-2039-libafl-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

Experiment 2024-08-25-2039-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[addisoncrump]

This is starting to look like a Heisenbug...

[DonggeLiu]

This is starting to look like a Heisenbug...

LOL, let's merge your PR if these two work fine.
We can always revert if the bug reappears.

I will merge my PR to add more logs so that we can better observe the bug once it reoccurs.

[tokatoka]

For the 1st one some of them stopped at 15 minutes
For the 2nd one some of them stopped at 6h so it was not really working

[tokatoka]

Now it is showing the final report.
But for the 2nd one in the final report,
None of {afl aflplusplus libfuzzer} is listed in the report, despite they are the fuzzers being tested

[tokatoka]

and as i said,
for the 2nd report, https://storage.googleapis.com/www.fuzzbench.com/reports/2024-08-25-2039-bases-1/data.csv.gz
the data.csv.gz does contain the data for afl, aflplusplus, and libfuzzer. but all their experiment are terminated after 6 hours (= 21600 seconds)

for the 1st one, too.
at least for this one, we have complete result for most targets on libafl except for the 5 missing NaN.

[tokatoka]

although I said "terminated after 6 hours".
but this is not really correct
because in the experiment-folder
https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-08-25-2039-bases-1/experiment-folders/curl_curl_fuzzer_http-afl/trial-3106096/corpus/

here it does have corpus for the last 23 hours. So in the end the fuzzer did run successfully!. but the problem is just about the measurement not working

[DonggeLiu]

Thanks for tracking the experiment progress, @tokatoka
It's perfectly normal for the experiment report to pause for a few hours:

1. The experiment trial VM might be preempted. Hence, that trial has to be re-run.
2. The measurer might take a while to produce results.

As a result, we can expect a 1-day experiment report to take up to 3 days : )