Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade base images, benchmark images, and Python. #1526

Merged
merged 54 commits into from
Dec 5, 2022
Merged

Upgrade base images, benchmark images, and Python. #1526

merged 54 commits into from
Dec 5, 2022

Conversation

DonggeLiu
Copy link
Contributor

I guess this branch has what we need? Not sure why it does not has a PR though.
I make it up-to-date with the master branch so that we can discuss here.

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Oct 14, 2022 •
edited
Loading

A quick update:
I am working on merging modifications from other PR to here.
Currently, I have a docker/base-image/Dockerfile that is based on focal and has python3.10.8 (the latest version).
I also updated the pip packages in requirements.txt to the latest version and managed to install them successfully in docker/base-image/Dockerfile.
I will push later.

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Oct 14, 2022 •
edited
Loading

TODOs in this PR:

  1. Check if there is any conflict with the new version of Ubuntu, python, and pip packages (like Update ubuntu to jammy and python to 3.10 #1521).
  2. Check if this PR missed anything in Upgrade to python3.10 #1493.
  3. Use a newer OSS-Fuzz docker image (like upgrade images #1441).
  4. Final check for conflicts.

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Oct 14, 2022 •
edited
Loading

TODOs in future PRs:

  1. Fix new lint/formatter issues, as we are using a newer version of pylint now. WIP at Fix lint #1529.
  2. Update documentation, done in Update documentation #1533.

@jonathanmetzman
Copy link
Contributor

btw you can just checkout https://github.com/google/fuzzbench/pull/1441/files locally and start pushing to that PR. I don't think we need another one.

jonathanmetzman
jonathanmetzman reviewed Oct 14, 2022
View reviewed changes
docker/image_types.yaml Outdated
@@ -49,7 +49,6 @@
- 'parent_image=gcr.io/fuzzbench/builders/coverage/{benchmark}-intermediate'
depends_on:
- 'coverage-{benchmark}-builder-intermediate'
- 'base-image'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this removal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this removal?

I do not know, it comes with this branch of yours : )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you prefer me to revert this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah lets rever this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is causing the CI failure as we discussed

@DonggeLiu
Copy link
Contributor Author

CI failed because we've updated pylint.

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Oct 18, 2022 •
edited
Loading

I've fixed all meaningful lint errors in #1529, which involves more than 60+ files.
There are many lint errors I consider not worth fixing and plan to let pylint ignore them, could you please take a look over there and let me know if you'd like me to fix them?

Once that PR is ready, I will merge its branch here.

@DonggeLiu DonggeLiu mentioned this pull request Oct 19, 2022
Merged
@DonggeLiu DonggeLiu mentioned this pull request Oct 20, 2022
Closed
@DonggeLiu DonggeLiu changed the title Upgrade base image to 20.04 Upgrade base images, benchmark images, and Python. Oct 21, 2022
@DonggeLiu
Fix & update benchmarks and fuzzers (#1541)
94cc1d3 
1. Fix and update (to the latest tag/release) the (nontrivial)
benchmarks and fuzzers to adapt to `Ubuntu:20.04` and `Python3.10.8`.
2. Fix and update (to the latest tag/release) the (nontrivial) fuzzers to
adapt to `Ubuntu:20.04` and `Python3.10.8`.
3. Delete the trivial/buggy benchmarks and fuzzers.
@DonggeLiu
Merge branch 'master' into upgrade-base-image
a1307ea
@DonggeLiu
Copy link
Contributor Author

It is OK to let presubmit fail for now: I will bring in the PR that fixes it at the end.

@DonggeLiu
Benchmark popper does not support wingfuzz
e87097d
@DonggeLiu
Do not report unused args as errors (#1559)
3878b68 
Some projects use `-Werror` to turn all warnings into errors.
This affects `Centipede` as we do not separate build and linking flags
as it expects, which leads to `unused-command-line-argument` warnings.
This PR disables turning that specific warning into errors and keeps the
rest the same.

See [the same PR from
OSS-Fuzz](google/oss-fuzz#9030) for more info
and [the error in this PR](google/oss-fuzz#8990)
for its use case.
@DonggeLiu
Remove wingfuzz from systemd
9e67151
@DonggeLiu
Fix lint (#1529)
2ba4957 
This PR fixes new lint errors from the latest `pylint`, which is
required as #1526 upgrades `Python` to `3.10`.
I reckon I have fixed all meaningful errors in the *`FuzzBench`
framework*, not sure if the rest is worth fixing or could be ignored.
@DonggeLiu
Merge branch 'master' into upgrade-base-image
aee441d
@DonggeLiu
Fix test_build_all_measurers
63f0eaa
@DonggeLiu
Fix typing hint
a011d7a
@DonggeLiu
Temporarily convert bug/oss-fuzz benchmarks to code
ce36cc9
@DonggeLiu
Update libpng's name in tests
3d9c7f1
@DonggeLiu
Copy link
Contributor Author

@jonathanmetzman : Please let me know if Fix test_build_all_measurers makes sense.
I was unsure if I completely understood the previous code (which failed the unittest), so I was trying to fix it based on its function description.

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Nov 25, 2022 •
edited
Loading

Not sure what is the best way to fix validate_fuzzers_and_benchmarks in presubmit:
I guess we don't want to delete the temporary fuzzers and variations now, but also do not want to always check them in CIs.

That is also the only thing left to pass all CI tests.

@DonggeLiu
Copy link
Contributor Author

Not sure what is the best way to fix validate_fuzzers_and_benchmarks in presubmit: I guess we don't want to delete the temporary fuzzers and variations now, but also do not want to always check them in CIs.

That is also the only thing left to pass all CI tests.

@jonathanmetzman : A gentle ping on this : )
Would now be a good time to delete the temporary fuzzers or remove them from CIs, so that we can merge this PR to master?
Also, I noticed that there are experiments running for weeks (r-2022-11-11-um-prioritize-2209607) or months (r-2022-10-13-um-final-3-2201231), is that intended?
Thanks!

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Dec 2, 2022 •
edited by jonathanmetzman
Loading

@jonathanmetzman:
Here is the full list of current fuzzers, could you please also tick the ones that you'd like to keep? Thanks!
I will delete the rest.

  • afl
  • aflfast
  • aflplusplus
  • aflsmart
  • centipede
  • eclipser
  • fairfuzz
  • honggfuzz
  • libafl
  • libfuzzer
  • mopt
  • klee # To Be Fixed.
  • symcc_aflplusplus # To Be Fixed.
  • afl_2_52_b
  • aflcc
  • afl_collision_free
  • afl_double_timeout
  • afl_maxmap
  • afl_no_favfactor
  • afl_no_favored
  • afl_no_trim
  • aflplusplus_classic
  • aflplusplus_cmplog
  • aflplusplus_cmplog_1585
  • aflplusplus_cmplog_1587
  • aflplusplus_cmplog_r
  • aflplusplus_dict2file
  • aflplusplus_frida
  • aflplusplus_muttfuzz
  • aflplusplus_optimal
  • aflplusplus_pcguard
  • aflplusplus_qemu
  • aflplusplus_qemu_tracepc
  • aflplusplus_tracepc
  • aflplusplus_um_parallel
  • aflplusplus_um_prioritize
  • aflplusplus_um_prioritize_75
  • aflplusplus_um_random
  • aflplusplus_um_random_3
  • aflplusplus_um_random_6
  • aflplusplus_um_random_75
  • aflplusplus_zafl
  • aflpp_random_default
  • aflpp_random_no_favs
  • aflpp_random_wrs
  • aflpp_random_wrs_rf
  • aflpp_random_wrs_rf_rp
  • aflpp_random_wrs_rp
  • afl_qemu
  • afl_random_favored
  • afl_scheduling_lifo
  • afl_scheduling_random
  • afl_score_max
  • afl_score_min
  • afl_score_no_novel_prioritization
  • afl_score_random
  • afl_splicing_mutation
  • afl_two_instances
  • afl_um_parallel
  • afl_um_prioritize
  • afl_um_random
  • afl_virginmap
  • centipede_function_filter
  • coverage
  • eclipser_aflplusplus
  • eclipser_um_parallel
  • eclipser_um_prioritize
  • eclipser_um_prioritize_75
  • eclipser_um_random
  • eclipser_um_random_75
  • entropic
  • entropic_execute_final
  • fafuzz
  • fuzzolic_aflplusplus_fuzzy
  • fuzzolic_aflplusplus_z3
  • glibfuzzer
  • gramatron
  • grimoire
  • honggfuzz_qemu
  • honggfuzz_um_parallel
  • honggfuzz_um_prioritize
  • honggfuzz_um_prioritize_75
  • honggfuzz_um_random
  • honggfuzz_um_random_75
  • introspector_driven_focus
  • lafintel
  • libafl_text
  • libfuzzer_dataflow
  • libfuzzer_dataflow_load
  • libfuzzer_dataflow_pre
  • libfuzzer_dataflow_store
  • libfuzzer_exeute_final
  • libfuzzer_focus_idx0
  • libfuzzer_focus_idx1
  • libfuzzer_focus_idx2
  • libfuzzer_focus_idx3
  • libfuzzer_focus_idx4
  • libfuzzer_focus_idx5
  • libfuzzer_focus_idx6
  • libfuzzer_focus_idx7
  • libfuzzer_focus_idx8
  • libfuzzer_fork_parallel
  • libfuzzer_um_parallel
  • libfuzzer_um_prioritize
  • libfuzzer_um_prioritize_75
  • libfuzzer_um_random
  • libfuzzer_um_random_75
  • manul
  • nautilus
  • neuzz
  • pythia_bb
  • pythia_effect_bb
  • symcc_afl
  • symcc_aflplusplus_single
  • symcc_afl_single
  • symqemu_aflplusplus
  • token_level
  • tortoisefuzz
  • weizz_qemu
  • wingfuzz

@jonathanmetzman
Copy link
Contributor

jonathanmetzman commented Dec 2, 2022 •
edited
Loading

@jonathanmetzman: Here is the full list of current fuzzers, could you please also tick the ones that you'd like to keep? Thanks! I will delete the rest.

We need to keep coverage for sure.
Let's do the upgrade and then delete in a subsequent PR (OK if CI fails)
Explain why we need to delete some of these please. Is it because they need to be fixed to pass linting?
Anyway, I've marked some I think we should keep

@DonggeLiu
Copy link
Contributor Author

DonggeLiu commented Dec 2, 2022 •
edited
Loading

We need to keep coverage for sure. Let's do the upgrade and then delete in a subsequent PR (OK if CI fails)

Sure! I will merge this on Monday, then.

Explain why we need to delete some of these please. Is it because they need to be fixed to pass linting?

Yep, it is mainly to avoid the validate_fuzzers_and_benchmarks error from presubmit.

@DonggeLiu DonggeLiu merged commit a6a5a9a into master Dec 5, 2022
@DonggeLiu DonggeLiu deleted the upgrade-base-image branch December 5, 2022 00:27
This was referenced Dec 5, 2022
Merged
Closed
Closed
Closed
Open
@DonggeLiu DonggeLiu mentioned this pull request Aug 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonathanmetzman jonathanmetzman Awaiting requested review from jonathanmetzman

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DonggeLiu @jonathanmetzman