FILTER PRE-PRUNING FOR IMPROVED FINE-TUNING OF QUANTIZED DEEP NEURAL NETWORKS

Abstract

Deep Neural Networks(DNNs) have many parameters and activation data, and these both are expensive to implement. One method to reduce the size of the DNN is to quantize the pre-trained model by using a low-bit expression for weights and activations, using fine-tuning to recover the drop in accuracy. However, it is generally difficult to train neural networks which use low-bit expressions. One reason is that the weights in the middle layer of the DNN have a wide dynamic range and so when quantizing the wide dynamic range into a few bits, the step size becomes large, which leads to a large quantization error and finally a large degradation in accuracy. To solve this problem, this paper makes the following three contributions without using any additional learning parameters and hyperparameters. First, we analyze how batch normalization, which causes the aforementioned problem, disturbs the fine-tuning of the quantized DNN. Second, based on these results, we propose a new pruning method called Pruning for Quantization (PfQ) which removes the filters that disturb the fine-tuning of the DNN while not affecting the inferred result as far as possible. Third, we propose a workflow of fine-tuning for quantized DNNs using the proposed pruning method(PfQ). Experiments using well-known models and datasets confirmed that the proposed method achieves higher performance with a similar model size than conventional quantization methods including fine-tuning.

1. INTRODUCTION

DNNs (Deep Neural Networks) greatly contribute to performance improvement in various tasks and their implementation in edge devices is required. On the other hand, a typical DNN (He et al., 2015; Simonyan & Zisserman, 2015) has the problem that the implementation cost is very large, and it is difficult to operate it on an edge device with limited resources. One approach to this problem is to reduce the implementation cost by quantizing the activations and weights in the DNN. Quantizing a DNN using extremely low bits, such as 1 or 2 bits has been studied by Courbariaux et al. (2015) and Gu et al. (2019) . However, it is known that while such a bit reduction has been performed for a large model such as ResNet (He et al., 2015) , it has not yet been performed for a small model such as MobileNet (Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019) , and is very difficult to apply to this case. For models that are difficult to quantize, special processing for the DNN is required before quantization. On the other hand, although fine-tuning is essential for quantization with extremely low bit representation, few studies have been conducted on pre-processing for easy fine-tuning of quantized DNNs. In particular, it has been experimentally shown from previous works (Lan et al., 2019; Frankle & Carbin, 2018) that, regardless of quantization, some weights are unnecessary for learning or in fact disturb the learning process. Therefore, we focused on the possibility of the existence of weights that specially disturb the finetuning quantized DNN and to improve the performance of the quantized DNN after fine-tuning by removing those weights.

