[WIP] Random weight and favored #6

jiradeto · 2021-07-19T01:41:30Z

This PR is basically a combination of random favored idea #4 and weight random selection #5.

Squashed commit of the following: commit e5c1aae Author: Jiradet Ounjai <[email protected]> Date: Tue Jun 29 18:24:04 2021 +0700 fixed incorrect exec_per_sec formula commit 6c87212 Author: Jiradet Ounjai <[email protected]> Date: Tue Jun 29 00:46:24 2021 +0700 implemented boost_fast_seqs commit 2bdcd30 Author: Jiradet Ounjai <[email protected]> Date: Tue Jun 22 17:40:13 2021 +0700 address PR feedback commit f043b57 Author: Jiradet Ounjai <[email protected]> Date: Tue Jun 22 13:56:29 2021 +0700 record the actual number of fuzz commit 50a865d Author: Jiradet Ounjai <[email protected]> Date: Tue Jun 22 00:05:30 2021 +0700 enable boost and throuttle inputs commit 2ecdb45 Author: Jiradet Ounjai <[email protected]> Date: Thu Jun 17 18:06:22 2021 +0700 assign weighed random to each seed commit c757066 Author: Jiradet Ounjai <[email protected]> Date: Thu Jun 17 16:45:37 2021 +0700 random double in interval 0.0-1.0 commit 0069e15 Author: Jiradet Ounjai <[email protected]> Date: Thu Jun 17 14:16:30 2021 +0700 new way to initialize minimum rand commit 454f30d Author: Jiradet Ounjai <[email protected]> Date: Thu Jun 10 15:51:26 2021 +0700 add missing header commit 1ec660d Author: Jiradet Ounjai <[email protected]> Date: Wed Jun 9 22:55:57 2021 +0700 random uniformly the favored seeds

jiradeto · 2021-07-19T02:05:48Z

I am setting up the fuzzbench experiment with the following fuzzers.

vanilla AFL
the latest setting of the random favored PR (increase_boost_inputs && base_weight_fac == 4.0 && max_weight_fac_decr == 3.75)
combination of random favored and weight random selection (this PR with the customzed values same as fuzzer no.2 — increase_boost_inputs && base_weight_fac == 4.0 && max_weight_fac_decr == 3.75)
combination of random favored and weight random selection (this PR with default parameters)

@wuestholz, could you please help to confirm that I have correct fuzzers for the evaluation?

wuestholz · 2021-07-19T07:51:21Z

@jiradeto Thanks! This will be a local experiment, right?

Did you already run an experiment to compare increase_boost_inputs with new_increase_boost_fast_seqs (from #4 (comment))? If not we should also include just the increase_boost_inputs configuration.

jiradeto · 2021-07-19T12:16:03Z

@jiradeto Thanks! This will be a local experiment, right?

@wuestholz, yes I mean the local experiment.

So, based on your your comment in another PR (#4 (comment)), I assume the current setting means the default value of increase_boost_inputs which is:

default && max_weight_fac_incr == 15.0

If this is the case, I will run local fuzzbench with the following fuzzers:

vanilla AFL
the increase_boost_inputs setting of the random favored PR (max_weight_fac_incr == 15.0)
combination of random favored and weight random selection (this PR with max_weight_fac_incr == 15.0)
combination of random favored and weight random selection (this PR with default parameters)

Could you please tell me if I am missing something?

wuestholz · 2021-07-19T12:58:14Z

@jiradeto Great! Thanks!

I think we don't need the fourth configuration.

Could you please also update this PR to make 15.0 the default value?

wuestholz · 2021-07-21T13:53:51Z

Let's add the following flags:

DISABLE_WRS: disables weighted random selection
DISABLE_RF: disables random favorites
ENABLE_UF: enables uniformly random favorites (i.e., weight always 1)
DISABLE_FAVS: disables favorites by considering every seed to be a favorite

- add flag to toggle weighted rando selection

jiradeto · 2021-07-22T11:57:15Z

DISABLE_RF: disables random favorites

@wuestholz, does this mean that we disable both random weighted selection (PR #5) and uniformly random favorites (PR #4)?

wuestholz · 2021-07-22T13:14:51Z

DISABLE_RF: disables random favorites

@wuestholz, does this mean that we disable both random weighted selection (PR #5) and uniformly random favorites (PR #4)?

@jiradeto No, just uniformly random favorites (PR #4).

jiradeto · 2021-07-22T15:39:33Z

DISABLE_RF: disables random favorites

@wuestholz, does this mean that we disable both random weighted selection (PR #5) and uniformly random favorites (PR #4)?

@jiradeto No, just uniformly random favorites (PR #4).

@wuestholz OK, that makes sense.

But I am unsure when to use ENABLE_UF since the default weight is already set to 1.0 and untouched if additional parameters enable_throttle_inputs, enable_boost_fast_seqs, enable_boost_inputs are not enabled.

wuestholz · 2021-07-22T20:58:42Z

DISABLE_RF: disables random favorites

@wuestholz, does this mean that we disable both random weighted selection (PR #5) and uniformly random favorites (PR #4)?

@jiradeto No, just uniformly random favorites (PR #4).

@wuestholz OK, that makes sense.

But I am unsure when to use ENABLE_UF since the default weight is already set to 1.0 and untouched if additional parameters enable_throttle_inputs, enable_boost_fast_seqs, enable_boost_inputs are not enabled.

@jiradeto I see. I suggest we replace enable_throttle_inputs, enable_boost_fast_seqs, and enable_boost_inputs with a single flag ENABLE_UF. When it is not set we will use enable_boost_fast_seqs and enable_boost_inputs.

We can probably remove the enable_throttle_inputs flag and the corresponding code.

afl-fuzz.c

wuestholz · 2021-07-23T08:47:53Z

@jiradeto Great! Thanks for the latest changes. There's only a small typo I noticed. I think we could try to set up a fuzzbench experiment with the following fuzzers:

afl (vanilla AFL)
afl_no_favs: DISABLE_WRS && DISABLE_RF && !ENABLE_UF && DISABLE_FAVS
afl_wrs_rf: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS
afl_rf: DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS
afl_rf_u: DISABLE_WRS && !DISABLE_RF && ENABLE_UF && !DISABLE_FAVS
afl_wrs: !DISABLE_WRS && DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS

Can you think of another setup that would be interesting?

jiradeto · 2021-07-23T15:03:50Z

@wuestholz. Thanks for your recommendation! May be we could also try with the following setting:

afl_rf_u_wrs: !DISABLE_WRS && !DISABLE_RF && ENABLE_UF && !DISABLE_FAVS

BTW, you mean the local fuzzbench experiment or the public one?

wuestholz · 2021-07-23T15:24:31Z

@jiradeto Sure, why not. :) Thanks!

I was thinking a public one.

jiradeto · 2021-07-23T16:55:50Z

@wuestholz, I feel that I'd rather at least have a local experiment result before I can go with the public one. Maybe we should start with the local experiment to see whether the above configurations behave properly?

wuestholz · 2021-07-23T20:35:22Z

@jiradeto Sure, that's also fine with me. 👍

wuestholz · 2021-08-12T09:02:08Z

@jiradeto As discussed, I would first do the following:

Afterwards, it would be great if you could run a local experiment that compares the following:

afl (vanilla AFL)
afl_wrs_rf: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS && DISABLE_RP
afl_wrs_rf_u: !DISABLE_WRS && !DISABLE_RF && ENABLE_UF && !DISABLE_FAVS && DISABLE_RP
afl_wrs_rf_t: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS && DISABLE_RP && changes below

The changes for the last configuration are:

      if (!enable_uniformly_random_favorites) {
        // enable_boost_inputs
        double base_weight_fac_boost_inputs = 1.0;
        double max_weight_fac_incr = 7.0;
        double scale_fac_boost_inputs = 0.001;
        double num_selections = (double)q->num_fuzzed;
        weight *= base_weight_fac_boost_inputs + max_weight_fac_incr / (scale_fac_boost_inputs * num_selections + 1.0);

        // enable_boost_fast_seqs
        double base_weight_fac_boost_fast = 8.0;
        double max_weight_fac_decr = 7.0;
        double scale_fac_boost_fast = 0.005;
        double execs_per_sec = 1000000.0 / (double) q->exec_us;
        weight *= base_weight_fac_boost_fast - max_weight_fac_decr / (scale_fac_boost_fast*execs_per_sec + 1.0);
      }

…_seed [WIP] Assign probability for random params per seed

[WIP] Randomize fuzzing params with prob

jiradeto · 2021-08-15T13:25:29Z

@wuestholz thank you for your detailed guidance. The fuzzbench evaluation for the changes you requested is available and can access using the following links:

wuestholz · 2021-08-15T20:07:14Z

@jiradeto Thanks a lot! Looks like it's slightly better. If you agree I suggest we update the code.

afl-fuzz.c

jiradeto added 3 commits July 19, 2021 08:15

remove duplicate rand_double

3d7fc09

change default value of max_weight_fac_incr to 15.0

31d9667

wuestholz mentioned this pull request Jul 22, 2021

[WIP] Weighted random seed selection #5

Closed

- use 1.0 probability fo fuzz new inputs

c10878b

- add flag to toggle weighted rando selection

revert AFL's original mechanism to select favorites

db648a1

jiradeto added 2 commits July 22, 2021 23:38

revert update_bitmap_score function

45244dd

reformat code

9c31732

jiradeto added 2 commits July 23, 2021 12:47

remove enable_throttle_inputs

d52c800

add enable_uniformly_random_favories flag

8027a2b

wuestholz reviewed Jul 23, 2021

View reviewed changes

afl-fuzz.c Outdated Show resolved Hide resolved

fix typo

9400fab

wuestholz added the ready for public benchmarking label Jul 23, 2021

fix redefinition

63e1623

randomize fuzzing params with prob

71fe3cf

jiradeto added 2 commits July 28, 2021 16:37

simplify random function

a216c7b

randomize more parameters

7e9c248

jiradeto mentioned this pull request Jul 28, 2021

[WIP] Randomize fuzzing params with prob #7

Merged

jiradeto added 4 commits July 29, 2021 20:44

remove logging

0906916

assign random params prob per seed

43c7877

include math.h

cfdd58c

avoid using math

881d74b

jiradeto added 3 commits August 12, 2021 18:07

Merge pull request #9 from Practical-Formal-Methods/random_params_per…

265b49d

…_seed [WIP] Assign probability for random params per seed

remove blank line

d22d152

Merge pull request #7 from Practical-Formal-Methods/random_params

f31d164

[WIP] Randomize fuzzing params with prob

This was referenced Aug 12, 2021

[WIP] No favorites #3

Closed

[WIP] Random favorites #4

Closed

jiradeto added 2 commits August 16, 2021 13:40

adjust variables default value

8abcf93

remove unused env variable

01e143d

jiradeto mentioned this pull request Sep 2, 2021

Port random fuzzing parameters to AFLplusplus Practical-Formal-Methods/AFLplusplus#1

Open

wuestholz reviewed Sep 3, 2021

View reviewed changes

afl-fuzz.c Outdated Show resolved Hide resolved

adjust default parameter

0e10d78

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Random weight and favored #6

[WIP] Random weight and favored #6

jiradeto commented Jul 19, 2021

jiradeto commented Jul 19, 2021 •

edited

Loading

wuestholz commented Jul 19, 2021

jiradeto commented Jul 19, 2021 •

edited

Loading

wuestholz commented Jul 19, 2021

wuestholz commented Jul 21, 2021 •

edited

Loading

jiradeto commented Jul 22, 2021 •

edited

Loading

wuestholz commented Jul 22, 2021

jiradeto commented Jul 22, 2021 •

edited

Loading

wuestholz commented Jul 22, 2021

wuestholz commented Jul 23, 2021 •

edited

Loading

jiradeto commented Jul 23, 2021 •

edited

Loading

wuestholz commented Jul 23, 2021 •

edited

Loading

jiradeto commented Jul 23, 2021

wuestholz commented Jul 23, 2021

wuestholz commented Aug 12, 2021

jiradeto commented Aug 15, 2021

wuestholz commented Aug 15, 2021

[WIP] Random weight and favored #6

Are you sure you want to change the base?

[WIP] Random weight and favored #6

Conversation

jiradeto commented Jul 19, 2021

jiradeto commented Jul 19, 2021 • edited Loading

wuestholz commented Jul 19, 2021

jiradeto commented Jul 19, 2021 • edited Loading

wuestholz commented Jul 19, 2021

wuestholz commented Jul 21, 2021 • edited Loading

jiradeto commented Jul 22, 2021 • edited Loading

wuestholz commented Jul 22, 2021

jiradeto commented Jul 22, 2021 • edited Loading

wuestholz commented Jul 22, 2021

wuestholz commented Jul 23, 2021 • edited Loading

jiradeto commented Jul 23, 2021 • edited Loading

wuestholz commented Jul 23, 2021 • edited Loading

jiradeto commented Jul 23, 2021

wuestholz commented Jul 23, 2021

wuestholz commented Aug 12, 2021

jiradeto commented Aug 15, 2021

wuestholz commented Aug 15, 2021

jiradeto commented Jul 19, 2021 •

edited

Loading

jiradeto commented Jul 19, 2021 •

edited

Loading

wuestholz commented Jul 21, 2021 •

edited

Loading

jiradeto commented Jul 22, 2021 •

edited

Loading

jiradeto commented Jul 22, 2021 •

edited

Loading

wuestholz commented Jul 23, 2021 •

edited

Loading

jiradeto commented Jul 23, 2021 •

edited

Loading

wuestholz commented Jul 23, 2021 •

edited

Loading