Accelerating Training of Sparse DNNs

It is a truth universally acknowledged that deep neural networks are heavily overparametrized, and pruning techniques can reduce weight counts by an order of magnitude. Specialized accelerator architectures have been proposed to convert this sparsity to energy and latency improvements at inference time by not fetching the pruned weights and not carrying out the corresponding multiplications.

Comparatively little attention, however, has been paid to the problem of efficiently training sparse networks. Typically, a network is first trained without pruning, then gradually pruned and retrained to recover accuracy — a process which requires even more time and energy than training an unpruned network.

In this talk, we will present a training algorithm that obtains a pruned DNN directly by dynamically following the most productive gradient during optimization; this results in state-of-the-art pruning ratios without compromising the accuracy of the trained classifier. We will also discuss architectural challenges to accelerating such sparse training algorithms, and outline an architecture that can train sparse networks in much less time and energy than an equivalent unpruned DNN.