Browse > Article
http://dx.doi.org/10.6109/jkiice.2018.22.9.1165

Apply Locally Weight Parameter Elimination for CNN Model Compression  

Lim, Su-chang (Department of Computer Engineering, Sunchon National University)
Kim, Do-yeon (Department of Computer Engineering, Sunchon National University)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.22, no.9, 2018 , pp. 1165-1171 More about this Journal
Abstract
CNN requires a large amount of computation and memory in the process of extracting the feature of the object. Also, It is trained from the network that the user has configured, and because the structure of the network is fixed, it can not be modified during training and it is also difficult to use it in a mobile device with low computing power. To solve these problems, we apply a pruning method to the pre-trained weight file to reduce computation and memory requirements. This method consists of three steps. First, all the weights of the pre-trained network file are retrieved for each layer. Second, take an absolute value for the weight of each layer and obtain the average. After setting the average to a threshold, remove the weight below the threshold. Finally, the network file applied the pruning method is re-trained. We experimented with LeNet-5 and AlexNet, achieved 31x on LeNet-5 and 12x on AlexNet.
Keywords
CNN; Pruning; Parameter Compression; Model Compression;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Y. J. Kim and E.G. Kim, "Image based Fire Detection using Convolutional Neural Network," Journal of the Korea Institute of Information and Communication Engineering, vol. 20, no. 9, pp. 1649-1656, Sep. 2016.   DOI
2 S. C. Lim, S.H. Kim, Y.H. Kim, and D.Y. Kim, "Training Network Design Based on Convolution Neural Network for Object Classification in few class problem," Journal of the Korea Institute of Information and Communication Engineering, vol. 21, no. 1, pp. 144-150, Jan. 2017.   DOI
3 Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackal, "Backpropagation applied to handwritten zip code recognition," International Journal of Neural Computation, vol. 1, no. 4, pp. 541-551, Dec. 1989.   DOI
4 O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet large scale visual recognition challenge," International Journal of Computer Vision, vol. 115, no. 3, pp. 211-252, Dec. 2015.   DOI
5 A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proceedings of the 25th Annual International Conference on Advances in Neural Information Processing Systems, California:CA, pp. 1106-1114, 2012.
6 Y. Gong, L. Liu, M.Yang, and L. Bourdev. (2014, December). "Compressing deep convolutional networks using vector quantization," arXiv preprint [Online. pp.1-10. Available: https://arXiv:1412.6115.
7 V. Vanhoucke, A. Senior, and M. Z Mao, "Improving the speed of neural networks on cpus," in Proceedings of Deep Learning and Unsupervised Feature Learning NIPS Workshop, Vol. 1. pp.1-8, Dec. 2011.
8 V. Chandrasekhar, J. Lin, Q. Liao, O. Morère, A. Veillard, L. Duan, and T. Poggio, "Compression of Deep Neural Networks for Image Instance Retrieval," in Proceedings of Data Compression Conference, Utah:UT, pp. 300-309, 2017.
9 W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, "Compressing neural networks with the hashing trick," in Proceedings of International Conference on Machine Learning, Lille:FR, pp. 2285-2294, 2015.
10 M. Lin, Q. Chen, and S. Yan. (2014, March). "Network in network," arXiv preprint [Online]. pp.1-10. Available: https://arXiv:1312.4400.
11 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," in Proceedings of Computer Vision and Pattern Recognition, Ohio:OH, pp. 1-9, 2014.
12 Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.   DOI
13 S. Srinivas, and R. V. Babu. (2016, August). "Learning Neural Network Architectures using Backpropagation," arXiv preprint [Online]. pp.1-14. Available: https://arXiv:1511.0549.
14 X. Liu, and T. Yatish. (2016, March). "Pruning of Winograd and FFT Based Convolution Algorithm," CS231n: Convolutional Neural Networks for Visual Recognition [Online]. pp.1-7. Available: http://cs231n.stanford.edu/reports/2016/pdfs/117 _Report.pdf
15 Y. Jia, E. Shelhamer, J. Deonahue, S. Karayev, J. Long, R. Girshick, and T. Darrell, "Caffe: Convolutional architecture for fast feature embedding." in Proceedings of the 22nd ACM international conference on Multimedia, New York: NY, pp. 675-678, 2014.
16 E. L Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, "Exploiting linear structure within convolutional networks for efficient evaluation," in Proceedings of the 27th Annual International Conference on Advances in Neural Information Processing Systems, Montreal:CA, pp. 1269-1277, 2014.
17 Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang, "Deep fried convnets," in Proceedings of the IEEE International Conference on Computer Vision, Massachusetts:MA, pp. 1476-1483, 2015.
18 S. Han, J. Pool, J. Tran, and W. J. Dally, "Learning both weights and connections for efficient neural networks," in Proceedings of the 28th Annual International Conference on Advances in Neural Information Processing Systems, Montreal:CA, pp. 1135-1143, 2015.
19 M. D Collins, and P. Kohli. (2014. December). "Memory bounded deep convolutional networks," arXiv preprint [Online]. pp.1-10. Available: https://arXiv:1412.1442.
20 S. Srinivas, and R V. Babu. (2015, July). "Data-free parameter pruning for deep neural networks," arXiv preprint [Online]. pp.1-12. Available: https://arXiv:1507.06149.