Browse > Article
http://dx.doi.org/10.22156/CS4SMB.2020.10.08.023

Application and Performance Analysis of Double Pruning Method for Deep Neural Networks  

Lee, Seon-Woo (Electric Computer Engineering, Inha University)
Yang, Ho-Jun (Computer Engineering, Inha University)
Oh, Seung-Yeon (Computer Engineering, Inha University)
Lee, Mun-Hyung (Computer Engineering, Inha University)
Kwon, Jang-Woo (Computer Engineering, Inha University)
Publication Information
Journal of Convergence for Information Technology / v.10, no.8, 2020 , pp. 23-34 More about this Journal
Abstract
Recently, the artificial intelligence deep learning field has been hard to commercialize due to the high computing power and the price problem of computing resources. In this paper, we apply a double pruning techniques to evaluate the performance of the in-depth neural network and various datasets. Double pruning combines basic Network-slimming and Parameter-prunning. Our proposed technique has the advantage of reducing the parameters that are not important to the existing learning and improving the speed without compromising the learning accuracy. After training various datasets, the pruning ratio was increased to reduce the size of the model.We confirmed that MobileNet-V3 showed the highest performance as a result of NetScore performance analysis. We confirmed that the performance after pruning was the highest in MobileNet-V3 consisting of depthwise seperable convolution neural networks in the Cifar 10 dataset, and VGGNet and ResNet in traditional convolutional neural networks also increased significantly.
Keywords
Model Compression; Model Light Weight; Deep Learning; Pruning; Network-Slimming;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Z. Liu, M. Sun, T. Zhou, G. Huang, T. Darrell. (2019). Rethinking the Value of Network Pruning, International Conference on Learning Representations (ICLR) Seq
2 A. Morcos, H. Yu, M. Paganini & Y. Tian. (2019). One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems (pp. 4932-4942).
3 A. Krizhevsky & Hinton, G. (2009). Learning multiple layers of features from tiny images.
4 I. J. Goodfellow et al. (2013, Nov). Challenges in Representation Learning: A report on three machine learning contests. In International Conference on Neural Information Processing (pp. 117-124). Springer, Berlin, Heidelberg.
5 E. Barsoum, C. Zhang, C. C. Ferrer & Z. Zhang. (2016, October). Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (pp. 279-283).
6 A. Wong. (2019, August). NetScore: Towards universal metrics for large-scale performance analysis of deep neural networks for practical on-device edge usage. In International Conference on Image Analysis and Recognition (pp. 15-26). Springer, Cham.
7 M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard & Q. V. Le. (2019). Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2820-2828).
8 P. Ramachandran, B. Zoph & Q. V. Le. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941.
9 I. Loshchilov & F. Hutter. (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983.
10 A. Aimar et al. (2018). Nullhop: A flexible convolutional neural network accelerator based on sparse representations of feature maps. IEEE transactions on neural networks and learning systems, 30(3), 644-656. DOI : 10.1109/TNNLS.2018.2852335   DOI
11 M. Schmidt, G. Fung & R. Rosales. (2007, September). Fast optimization methods for l1 regularization: A comparative study and two new approaches. In European Conference on Machine Learning (pp. 286-297). Springer, Berlin, Heidelberg. DOI : 10.1007/978-3-540-74958-5_28
12 M. Sandler, A. Howard, M. Zhu, A. Zhmoginov & L. C. Chen. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510-4520).
13 E. Real, A. Aggarwal, Y. Huang & Q. V. Le. (2019, July). Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, 33, 4780-4789. DOI : 10.1609/aaai.v33i01.33014780
14 S. Karen & Z. Andrew. (2014), Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556.
15 K. He, X. Zhang, S. Ren & J. Sun. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
16 C. Szegedy et al. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
17 A. Howard. et al. (2019). Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1314-1324).
18 G. H. Andrew, Z. Menglong, C. Bo, K. Dmitry, W. Weijun, W. Tobias, A. Marco & A. Hartwig. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint arXiv:1704.04861.
19 N. I, Forrest. H. Song, W. M. Matthew, A. Khalid, J. D. William & K. Kurt. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv preprint arXiv:1602.07360.
20 X. Zhang, X. Zhou, M. Lin & J. Sun. (2018). Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6848-6856).
21 F. Chollet. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1251-1258).
22 H. Song, M. Huizi & J. D. William (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
23 Gholami, A., Kwon K., Wu B., Tai Z., Yue X., Jin P., Zhao S., Keutzer K., (2018. June). Squeezenext: Hardware-aware neural network design. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 1638-1647).
24 L. Yann, S. D. John & A. S. Sara. (1990). Optimal brain damage. In Advances in neural information processing systems (pp. 598-605).
25 S. Han, J. Pool, J. Tran & W. Dally. (2015). Learning both weights and connections for efficient neural network. In Advances in neural information processing systems (pp. 1135-1143).
26 R. Reed, (1993). Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5), 740-747. DOI : 10.1109/72.248452   DOI
27 N. Lee, T. Ajanthan & P. H. Torr. (2018). Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340.
28 Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan & C. Zhang, (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2736-2744).
29 Y. He, X. Zhang & J. Sun, (2017). Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1389-1397).
30 M. Tan & Q. V. Le. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946.
31 M. Tan & Q. V. Le. (2019). Mixconv: Mixed depthwise convolutional kernels. CoRR, abs/1907.09595
32 J. H. Luo, J. Wu & W. Lin. (2017). Thinet: A filter level pruning method for deep neural network compression. The IEEE International Conference on Computer Vision (ICCV) (pp. 5058-5066)