Archive coverage data alongside corpus archives (from AFL++ fork) #2028

addisoncrump · 2024-08-14T15:22:27Z

Supercedes #2020. Moving so we (AFL++ people) can collaborate on this PR.

From the original:

Currently, only corpora are saved in the archive and the summaries of coverage are provided at the end of the experiment. This change simply incorporates the saving of the coverage data snapshots next to the trial corpus snapshots.

addisoncrump · 2024-08-16T11:58:00Z

@DonggeLiu Can we try to do a baseline experiment with this PR again? 🙂 It is fully rebased to the latest changes.

I will integrate the analysis changes once there is a public baseline to point the analysis example at.

DonggeLiu · 2024-08-17T01:26:07Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-2028-bases-1 --fuzzers afl aflplusplus libafl libfuzzer

DonggeLiu · 2024-08-17T01:29:52Z

Hi @addisoncrump,I started a test exp above.
Experiment 2024-08-17-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

If it works well and you'd like to run a full exp (23 hours), could you please rebase to adopt this change?
I forgot to revert temp changes in a previous PR.

Thanks!

addisoncrump · 2024-08-19T01:11:03Z

Rebased. The experiment looks good, all the coverage samples were archived.

DonggeLiu · 2024-08-19T01:54:53Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-19-2028-bases-1 --fuzzers afl aflplusplus libafl libfuzzer

DonggeLiu · 2024-08-19T02:03:21Z

Experiment 2024-08-19-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

addisoncrump · 2024-08-19T13:35:36Z

It seems to still not be hitting the measurer...

DonggeLiu · 2024-08-21T00:16:08Z

This is a really strange, because 2024-08-19-2028-bases-1 has a list of errors about merging coverage summary:

But 2024-08-17-2028-bases-1 did not have any:

DonggeLiu · 2024-08-21T00:20:16Z

QQ: Is the only thing change between those 2 experiments?

BTW, I noticed this runtime crash in libafl. I don't think it could cause the failure, but it might be interesting to you:
https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-08-19-2028-bases-1/experiment-folders/libxml2_xml-libafl/trial-3070882/results/

It did not happen in 2024-08-17-2028-bases-1, maybe because that experiment was very short?

A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?

addisoncrump · 2024-08-21T06:45:31Z

@tokatoka random libafl crash 🙃

addisoncrump · 2024-08-21T06:59:48Z

I confirmed that the only difference is that commit, yes.

Let's add some more debugging and run a very short run with all the benchmarks, I guess?

tokatoka · 2024-08-21T07:01:30Z

ohh i see. so this is why my experiment didn't complete either

tokatoka · 2024-08-21T07:03:09Z

A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?

but it should not affect other fuzzers such as aflplusplus runs right?

tokatoka · 2024-08-21T11:23:43Z

@tokatoka random libafl crash 🙃

can you reproduce?
i used the same setup on fuzzbench but cannot reproduce

tokatoka · 2024-08-21T14:55:42Z

I updated. @DonggeLiu
Could you run the same command again to see if it fixes the problem or not?

DonggeLiu · 2024-08-22T00:41:11Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-22-2028-bases-1 --fuzzers libafl

DonggeLiu · 2024-08-22T00:45:34Z

A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?

Also running an experiment without libafl to help verify this theory.

DonggeLiu · 2024-08-22T00:45:48Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-22-2028-bases-2 --fuzzers afl aflplusplus libfuzzer

DonggeLiu · 2024-08-22T01:09:50Z

Experiment 2024-08-22-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

Experiment 2024-08-22-2028-bases-2 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

addisoncrump · 2024-08-22T11:01:35Z

bases-1 seems to be working fine, but bases-2 is not hitting the measurer still.

tokatoka · 2024-08-22T11:31:31Z

so it looks like the libafl crash is not the cause of this

tokatoka · 2024-08-22T16:24:24Z

btw for base-1 it seems all the fuzzers are stuck after 10:45m so it was not a successful run either...

DonggeLiu · 2024-08-23T00:18:25Z

Ops there were a DB issue yesterday which affected both experiments.
Let me re-run them

DonggeLiu · 2024-08-23T00:42:32Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-23-2028-libafl --fuzzers libafl

DonggeLiu · 2024-08-23T00:45:40Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-23-2028-bases --fuzzers afl aflplusplus libfuzzer

tokatoka · 2024-08-23T10:21:36Z

we have experiment-folder but not report. so the measurement is still broken

tokatoka · 2024-08-23T12:22:46Z

For the report:
2024-08-23-2028-libafl and 2024-08-23-2028-bases is missing
but 2024-08-23-2036-bases-1 (from the other PR) is there

For the experiment-data: nothing is missing.

tokatoka · 2024-08-23T23:45:39Z

Btw if the experiment on my branch and https://www.fuzzbench.com/reports/experimental/2024-08-23-dgfuzz/index.html ← this experiment are working. would it be possible that the changes in this PR caused the measurement failure??

DonggeLiu · 2024-08-24T09:44:26Z

Experiment 2024-08-23-2028-libafl data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

Experiment 2024-08-23-2028-bases data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

DonggeLiu · 2024-08-24T11:53:06Z

@addisoncrump would this happen to be related to this PR?

It could be due to this error:

I think the gsutil rm error is at least benign, because @tokatoka shows 2024-08-23-2036-bases-1 can generate a report:

Btw if the experiment on my branch and https://www.fuzzbench.com/reports/experimental/2024-08-23-dgfuzz/index.html ← this experiment are working. would it be possible that the changes in this PR caused the measurement failure??

and it also has gsutil rm error, but not llvm-profdata error:

There was another build error (discussed in #2038, as shown above), but I am sure that one is benign and unrelated to the missing report in the experiment.

addisoncrump · 2024-08-24T20:27:52Z

I'm really not sure how it could be. I think it will require manual inspection to understand the root cause here. I don't really understand why this would work locally but not in the cloud environment for these reasons, since we should expect the same errors.

Looking at the changeset: I don't see why anything I did would have affected this, especially since we see inconsistent generation of reports. The only thing I can think of that might cause this would be rate limiting with the bucket or similar.

DonggeLiu · 2024-08-25T03:19:12Z

Looking at the changeset: I don't see why anything I did would have affected this, especially since we see inconsistent generation of reports. The only thing I can think of that might cause this would be rate limiting with the bucket or similar.

Yep, I did not think of any reason from this PR either. Yet this seems to be the only place that we can reproduce the no report error: I was trying to reproduce the Fuzz target binary not found. error in this PR, but it did not work either.

Could you please cherry-pick commits from #2038, or rebase your PR on it?
Hopefully those commits can help us understand the cause.

Never mind, I created #2039 for this to keep your PR clean.

This is weird: With the same commits, that experiment works.

Let's wait a bit longer and if that experiments proves the error is flaky, we should be able to merge this.
Not sure we can can consistently reproduce it here though, maybe because we run two experiments together?

addisoncrump · 2024-10-13T15:59:18Z

Hey, what remains for this PR? We settled that the flakiness was not associated to this PR, unless I'm mistaken.

DonggeLiu · 2024-10-13T23:14:44Z

Hey, what remains for this PR? We settled that the flakiness was not associated to this PR, unless I'm mistaken.

You are right, I will merge this.

DonggeLiu · 2024-10-13T23:14:54Z

/gcbexp skip

addisoncrump · 2024-10-20T17:27:29Z

Any chance we could run another baselines experiment with this? Would be good to have this data in the pocket.

DonggeLiu · 2024-10-21T02:36:09Z

Any chance we could run another baselines experiment with this? Would be good to have this data in the pocket.

Yep sure!
Running at a new PR as this PR has been merged.

…oogle#2028) Supercedes google#2020. Moving so we (AFL++ people) can collaborate on this PR. From the original: > Currently, only corpora are saved in the archive and the summaries of coverage are provided at the end of the experiment. This change simply incorporates the saving of the coverage data snapshots next to the trial corpus snapshots. --------- Co-authored-by: Toka <[email protected]>

addisoncrump mentioned this pull request Aug 14, 2024

Archive coverage data alongside corpus archives #2020

Closed

addisoncrump force-pushed the archive-coverage branch 2 times, most recently from 637743c to c8c6ad3 Compare August 16, 2024 11:56

addisoncrump added 11 commits August 19, 2024 03:09

archive coverage data alongside corpus archives

003b4aa

format

c96f66c

upload to the right place

90a0a51

direction???

7821dea

format

1cbeb69

don't put it in corpus lol

8cdbdb1

llvm-cov no-warn

1b48924

weird presubmit bug fix

97adb0b

ok, no no-warn

6a072a7

dummy change for experiment runner

3af5fdf

revert changes in coverage_utils; remove warning lines in save

6723889

addisoncrump force-pushed the archive-coverage branch from c8c6ad3 to 6723889 Compare August 19, 2024 01:10

addisoncrump mentioned this pull request Aug 19, 2024

Fixup libafl_fuzzbench-based fuzzers #2032

Merged

DonggeLiu mentioned this pull request Aug 22, 2024

Seed experiment #2025

Open

DonggeLiu mentioned this pull request Aug 24, 2024

More informative logs to help debugging #2038

Draft

DonggeLiu mentioned this pull request Aug 25, 2024

Debug PR #2028 #2039

Closed

DonggeLiu approved these changes Oct 13, 2024

View reviewed changes

DonggeLiu merged commit 2920e74 into google:master Oct 13, 2024
58 of 62 checks passed

DonggeLiu mentioned this pull request Oct 21, 2024

[EXP ONLY] Re-run experiments for 2028 #2052

Open

Archive coverage data alongside corpus archives (from AFL++ fork) #2028

Archive coverage data alongside corpus archives (from AFL++ fork) #2028

Conversation

addisoncrump commented Aug 14, 2024

addisoncrump commented Aug 16, 2024 • edited Loading

DonggeLiu commented Aug 17, 2024

DonggeLiu commented Aug 17, 2024

addisoncrump commented Aug 19, 2024

DonggeLiu commented Aug 19, 2024

DonggeLiu commented Aug 19, 2024

addisoncrump commented Aug 19, 2024

DonggeLiu commented Aug 21, 2024

DonggeLiu commented Aug 21, 2024 • edited Loading

addisoncrump commented Aug 21, 2024

addisoncrump commented Aug 21, 2024

tokatoka commented Aug 21, 2024

tokatoka commented Aug 21, 2024 • edited Loading

tokatoka commented Aug 21, 2024 • edited Loading

tokatoka commented Aug 21, 2024

DonggeLiu commented Aug 22, 2024

DonggeLiu commented Aug 22, 2024

DonggeLiu commented Aug 22, 2024

DonggeLiu commented Aug 22, 2024

addisoncrump commented Aug 22, 2024

tokatoka commented Aug 22, 2024

tokatoka commented Aug 22, 2024

DonggeLiu commented Aug 23, 2024

DonggeLiu commented Aug 23, 2024

DonggeLiu commented Aug 23, 2024

tokatoka commented Aug 23, 2024

tokatoka commented Aug 23, 2024 • edited Loading

tokatoka commented Aug 23, 2024

DonggeLiu commented Aug 24, 2024

DonggeLiu commented Aug 24, 2024 • edited Loading

addisoncrump commented Aug 24, 2024

DonggeLiu commented Aug 25, 2024 • edited Loading

addisoncrump commented Oct 13, 2024

DonggeLiu commented Oct 13, 2024

DonggeLiu commented Oct 13, 2024

addisoncrump commented Oct 20, 2024

DonggeLiu commented Oct 21, 2024 • edited Loading

addisoncrump commented Aug 16, 2024 •

edited

Loading

DonggeLiu commented Aug 21, 2024 •

edited

Loading

tokatoka commented Aug 21, 2024 •

edited

Loading

tokatoka commented Aug 21, 2024 •

edited

Loading

tokatoka commented Aug 23, 2024 •

edited

Loading

DonggeLiu commented Aug 24, 2024 •

edited

Loading

DonggeLiu commented Aug 25, 2024 •

edited

Loading

DonggeLiu commented Oct 21, 2024 •

edited

Loading