Experiments with gradient-based hyperparameter optimization

For my master's project.

Abstract

Gradient-based hyperparameter optimization algorithms have the potential to scale to numbers of individual hyperparameters proportional to the number of elementary parameters, unlike other current approaches. Some candidate completions of DrMAD, one such algorithm that updates the hyperparameters after fully training the parameters of the model, are explored, with experiments tuning per-parameter L2 regularization coefficients on CIFAR10 with the DenseNet architecture. Experiments with DenseNets on CIFAR10 are also conducted with an adaptive method, which updates the hyperparameters during the training of elementary parameters, tuning per-parameter learning rates and L2 regularization. The experiments do not establish the utility of either method, but the adaptive method shows some promise, with further experiments required.

Primary references:

DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks (code)

Gradient-based Hyperparameter Optimization through Reversible Learning (code)

Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (code)

Online Learning Rate Adaptation with Hypergradient Descent

On-Line Step Size Adaptation

Parameter Adaptation in Stochastic Optimization

Densely Connected Convolutional Networks

The code here is edited from https://github.com/bigaidream-projects/drmad (and https://github.com/HIPS/hypergrad), and https://github.com/Lasagne/Recipes/tree/master/papers/densenet.

The CPU version (cpu-ver) is written in Python and uses https://github.com/HIPS/autograd, while the GPU version (gpu-ver) is written in Theano.