run benchmark script

[Justinyu1618]

New Benchmark Config and Benchmark script

Run Groups

For each model, you can specify any number of 'run groups' with a unique group id. Each run group allows you to specific a set of hyperparams to run the model with.
You can also include/exclude attacks, customize if they will be shown on the final benchmark graph etc.

Hyperparameters

Each hparams config is a dictionary where keys are json access paths in the base config of the model, and the value is an array of potential hparam values.
The cartesian product of hparam values is calculated and a model will be run on each. For K hparams, we will run N_0 * ... * N_K models where N_i is the length
of the values array for hparam i.

Attacks:

Which attacks to run on which defense can be configured using attack settings. These consist of include, exclude and a special @all token.
Attacks can be included/excluded globally, per model, and per run group

Benchmark Config Fields:

{
config_path: path to base folder for config files
defense_models/attack_models: configs for each defense/attack model
    [model name]
        attacks: (optional, configure attacks to be run on this model)
        run_groups: (see Run Groups above)
            [group id]
                hparams: (see Hyperparameters above)
                attacks: (optional, configue attacks to be run on this group)
default_attacks: settings for which attacks will be default be run on all defense models
}

TODOS:

• will need to change experiment naming system
• what to do with attack hp? should run all attacks variations on every defense?
• option to only run attack on best performing defense