Fix & update benchmarks and fuzzers

[DonggeLiu]

1. Fix and update (to the latest tag/release) the (nontrivial) benchmarks and fuzzers to adapt to Ubuntu:20.04 and Python3.10.8.
2. Delete the trivial/buggy benchmarks and fuzzers.
   Base: Upgrade base images, benchmark images, and Python. #1526

Benchmarks

'[x]' means it can build and run under the new environment.

Support

• All benchmarks in upgrade images #1441.

  • systemd_fuzz-link-parser.
  • systemd_fuzz-varlink.
  • njs_njs_process_script_fuzzer.
  • libxslt_xpath.
  • openh264_decoder_fuzzer.
  • libarchive_libarchive_fuzzer.
  • ffmpeg_ffmpeg_demuxer_fuzzer.
    • I apt install some packages that were previously built manually, to make its build.sh consistent with OSS-Fuzz, hope that would be alright.
  • mruby-2018-05-23.
  • openssl_x509.
  • php_php-fuzz-parser-2020-07-25.
  • php_php-fuzz-execute.
  • zstd_stream_decompress.
  • poppler_pdf_fuzzer.
  • grok_grk_decompress_fuzzer.

• Bug Benchmarks (temporarily converted to coverage benchmarks):

  • arrow_parquet-arrow-fuzz
  • aspell_aspell_fuzzer
  • ffmpeg_ffmpeg_demuxer_fuzzer
  • file_magic_fuzzer
  • grok_grk_decompress_fuzzer
  • libarchive_libarchive_fuzzer
  • libgit2_objects_fuzzer
  • libhevc_hevc_dec_fuzzer
  • libhtp_fuzz_htp
  • libxml2_libxml2_xml_reader_for_file_fuzzer
  • matio_matio_fuzzer
  • mruby-2018-05-23
  • muparser_set_eval_fuzzer
  • njs_njs_process_script_fuzzer
  • openh264_decoder_fuzzer
  • php_php-fuzz-execute
  • php_php-fuzz-parser-2020-07-25
  • poppler_pdf_fuzzer
  • proj4_standard_fuzzer
  • quickjs_eval-2020-01-05
  • stb_stbi_read_fuzzer
  • systemd_fuzz-varlink
  • usrsctp_fuzzer_connect
  • wireshark_fuzzshark_ip

• Other benchmarks?

  • libpng.
  • bloaty.

• Change benchmark dir name to reflect their current version.

Deprecate

Nothing so far, may add later.

Undetermined

1. poppler
   • Failed to build in FuzzBench and OSS-Fuzz (CI cancelled).
   • Can add it back when it can build.
   • Can deprecate if it continues fail to build.

Fuzzers

Fix and update (to the latest tag/release) the (nontrivial) fuzzers to adapt to Ubuntu:20.04 and Python3.10.8.
Maybe also delete the outdated/trivial/buggy ones?

We want to support
([x] means it is up-to-date and passed the standard and oss-fuzz categories of CI tests and did not break in the bug category before the 5-hour timeout):

• afl
• aflfast
• afl++
• aflsmart
• centipede (failing due to the weak reference issue, which should be acceptable?)
• eclipser
• fairfuzz
• honggfuzz
• libafl
• libFuzzer
• mopt
• klee (Not sure if we want to remove it. It is actively maintained, but it seems we do not use it in our reports for some reason).
• symcc_aflplusplus (I want to add SymCC-related fuzzers, as SymCC supports concolic execution and seems impactful. But I failed to fix its errors after trying for two days. SymCC still relies on Clang-10 and Python2, yet it also tries to support LLVM-15 in very recent PRs. Maybe let's wait for a while until it can get rid of the ancient dependencies and becomes stable on LLVM-15?)

Not used as default fuzzers in general comparison evaluations

• centipede_function_filter (Do we use it for daily fuzzing evaluations?)
• introspector_driven_focus (Do we use it for daily fuzzing evaluations?)

Deprecate

([x] means it is up-to-date and passed the standard and oss-fuzz categories of CI tests and did not break in the bug category before the 5-hour timeout):

• All variations
• entropic
• neuzz
• pythia
• fafuzz
• tortoisefuzz
• wingfuzz
• weizz
• fuzzolic_aflplusplus_z3
• nautilus
• gramatron
• token_level
• afl_2_52_b
• libfuzzer_dataflow
• lafintel (Do we want to keep this? It has not been updated in the past 6 years.)

Undecided

Nothing so far.

[DonggeLiu]

I noticed that we are using this repo for SymCC instead of the official repo.
Currently, it fails to build and I am planning to fix it.
Shall I continue using that repo or switch to the official one? Thanks! @jonathanmetzman

[DonggeLiu]

Which AFL shall we keep?
I noticed that we have AFL (which is archived) and AFL-2.52b (which has not been updated in 2 years).
Shall we keep both of them or stop supporting one of them?

[DonggeLiu]

I reckon the bug category is still needed by the competition, maybe we exclude them from daily experiment and CI tests, and only keep them for the competition?

I've tried to make bug benchmarks compatible with libfuzzer in Ubuntu:20.04 and Python-3.10.8. CI tests were not able to test them all due to the 5-hour time limit.

[DonggeLiu]

Interesting. GitLab GNOME packages (e.g. libxml2, libxslt) are not available.
Our benchmarks are broken because of it.

[DonggeLiu]

The general rule I followed is to consider libFuzzer as the default fuzzer and libpng-1.6.38 as the default benchmark.
If a benchmark B can build with libFuzzer then we can keep B, and if a fuzzer A failed to build with B, then we add A to the list of unsupported_fuzzers in B's benchmark.yaml.
Similarly, if a fuzzer A can build with libpng-1.6.38, then we keep A, and if A cannot build with another benchmark B, then we add A to the list of unsupported_fuzzers in B's benchmark.yaml.

In addition, we deprecate a fuzzer A is A is no longer maintained or relies on very old dependencies.

[vanhauser-thc]

Which AFL shall we keep? I noticed that we have AFL (which is archived) and AFL-2.52b (which has not been updated in 2 years). Shall we keep both of them or stop supporting one of them?

IMHO the archive one should be kept because still people like to reference against it. 2.52b should be dropped though.

[DonggeLiu]

Which AFL shall we keep? I noticed that we have AFL (which is archived) and AFL-2.52b (which has not been updated in 2 years). Shall we keep both of them or stop supporting one of them?

IMHO the archive one should be kept because still people like to reference against it. 2.52b should be dropped though.

Yep, thanks!
We think the same: This PR keeps AFL and drops AFL-2.52b : )

Are there any other fuzzers that you would suggest keeping/dropping?

[vanhauser-thc]

I think lafintel, mopt, aflfast and fairfuzz can be dropped. aflsmart is also pretty outdated, but its one of the few structured fuzzing implementations so that is why I would keep it.

[jonathanmetzman]

Interesting. GitLab GNOME packages (e.g. libxml2, libxslt) are not available. Our benchmarks are broken because of it.

Let me close and reopen and see if that fixes things

[DonggeLiu]

Interesting. GitLab GNOME packages (e.g. libxml2, libxslt) are not available. Our benchmarks are broken because of it.

Let me close and reopen and see if that fixes things

Ah sorry, that was fixed automatically on the following day.
They were only down for one night.

[DonggeLiu]

I think lafintel, mopt, aflfast and fairfuzz can be dropped. aflsmart is also pretty outdated, but its one of the few structured fuzzing implementations so that is why I would keep it.

Thanks!
Yeah, lafintel has not been maintained for years, I excluded it in this PR.
I will have another PR dedicated to excluding more of those fuzzers (and benchmarks) later.

[DonggeLiu]

OK, I will merge this back to its base branch as discussed.
Will have another PR to remove redundant fuzzers/benchmarks.