Dirichlet Pruning for Neural Network Compression

Dirichlet Pruning for Neural Network Compression

We introduce Dirichlet pruning, a novel post-processing technique to transform a large neural network model into a compressed one. Dirichlet pruning is a form of structured pruning which assigns the Dirichlet distribution over each layer's channels in convolutional layers (or neurons in fully-connected layers), and estimates the parameters of the distribution over these units using variational inference.The learned distribution allows us to remove unimportant units, resulting in a compact architecture containing only crucial features for a task at hand. Our method is extremely fast to train. The number of newly introduced Dirichlet parameters is only linear in the number of channels, which allows for rapid training, requiring as little as one epoch to converge. We perform extensive experiments, in particular on larger architectures such as VGG and WideResNet (45% and 52% compression rate, respectively) where our method achieves the state-of-the-art compression performance and provides interpretable features as a by-product.

用于神经网络压缩的Dirichlet修剪

我们介绍Dirichlet修剪，这是一种将大型神经网络模型转换为压缩模型的新颖后处理技术。Dirichlet修剪是结构修剪的一种形式，它在卷积层（或完全连接的层中的神经元）的每一层通道上分配Dirichlet分布，并使用变分推断来估计这些单元上的分布参数。.. 学习到的分发使我们可以删除不重要的单元，从而形成一个紧凑的体系结构，其中仅包含当前任务的关键功能。我们的方法训练起来非常快。新引入的Dirichlet参数的数量仅在通道数量上是线性的，这允许快速训练，只需要一个历元即可收敛。我们进行了广泛的实验，尤其是在较大的体系结构上，例如VGG和WideResNet（分别为45％和52％的压缩率），其中我们的方法实现了最先进的压缩性能并作为副产品提供了可解释的功能。 （阅读更多）