Enabling Binary Neural Network Training on the Edge

In this repository we introduce a low-cost binary neural network training strategy exhibiting sizable memory footprint and energy reductions while inducing little to no accuracy loss vs Courbariaux & Bengio's standard binary neural network (BNN) training approach, thereby increasing the viability of deep neural network training on the edge.

These resource decreases are primarily enabled by keeping activations exclusively in binary format throughout both forward and backward propagation, while the standard BNN training method stores activations only in forward propagation, not backward propagation. Please checkout our paper for more details.

Repo organization

We first emulated our BNN traning method using TensorFlow on GPUs, then implemented it in C++ targetting Raspberry Pi 3B+'s ARM CPU. We also include our implementations of the standard BNN training method on Raspberry Pi written in both C++ and Python with TensorFlow, which we used as baselines for performance comparison. Memory profilers were also included with our Raspberry Pi prototypes to show our method's memory savings.

• rpi_prototype/cpp_bnn: C++ implementations of both standard and our BNN training methods targetting Raspberry Pi (with valgrind-based memory profiler).
• rpi_prototype/python_bnn: Python (TF) implementation of the standard method targetting Raspberry Pi (with PyPI memory profiler).
• training_emulation_on_gpu: Emulation of our method using TF on GPUs.

Results

Results Obtained from GPU Emulation

Dataset	Model	Batch size	Top-1 accuracy (%)	Modelled memory footprint savings (x↓)
MNIST	Multilayer Perceptron	100	96.90	2.78
CIFAR-10	CNV	100	83.08	4.17
SVHN	CNV	100	94.28	4.17
CIFAR-10	BinaryNet	100	89.90	3.71
SVHN	BinaryNet	100	95.93	3.71
ImageNet	ResNetE-18	4096	57.04	3.78
ImageNet	Bi-Real-18	4096	54.45	3.78

Results Measured from Training BNNs on Raspberry Pi 3B+

Dataset	Model	Batch size	Training time per batch (s)	Memory footprint (MiB)	Energy per batch (J)
MNIST	Multilayer Perceptron	200	0.98	5.00 (52x↓ vs. Keras)	1.42
CIFAR-10	BinaryNet	40	121.20	154.37 (Keras doesn't fit)	136.53

Citation

If you make use of this code, please acknowledge us by citing our paper:

@inproceedings{BNN_ON_EDGE,
    author = {Wang, Erwei and Davis, James J. and Moro, Daniele and Zielinski, Piotr and Lim, Jia Jie and Coelho, Claudionor and Chatterjee, Satrajit and Cheung, Peter Y. K. and Constantinides, George A.},
    title = {Enabling Binary Neural Network Training on the Edge},
    year = {2021},
    doi = {10.1145/3469116.3470015},
    booktitle = {Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning}
    }