Browse > Article
http://dx.doi.org/10.5909/JBE.2020.26.7.862

Pruning for Robustness by Suppressing High Magnitude and Increasing Sparsity of Weights  

Cho, Incheon (Department of Computer Science and Engineering, Kyung Hee Univ.)
Ali, Muhammad Salman (Department of Computer Science and Engineering, Kyung Hee Univ.)
Bae, Sung-Ho (Department of Computer Science and Engineering, Kyung Hee Univ.)
Publication Information
Journal of Broadcast Engineering / v.26, no.7, 2021 , pp. 862-867 More about this Journal
Abstract
Although Deep Neural Networks (DNNs) have shown remarkable performance in various artificial intelligence fields, it is well known that DNNs are vulnerable to adversarial attacks. Since adversarial attacks are implemented by adding perturbations onto benign examples, increasing the sparsity of DNNs minimizes the propagation of errors to high-level layers. In this paper, unlike the traditional pruning scheme removing low magnitude weights, we eliminate high magnitude weights that are usually considered high absolute values, named 'reverse pruning' to ensure robustness. By conducting both theoretical and experimental analyses, we observe that reverse pruning ensures the robustness of DNNs. Experimental results show that our reverse pruning outperforms previous work with 29.01% in Top-1 accuracy on perturbed CIFAR-10. However, reverse pruning does not guarantee benign samples. To relax this problem, we further conducted experiments by adding a regularization term for the high magnitude weights. With adding the regularization term, we also applied conventional pruning to ensure the robustness of DNNs.
Keywords
convolutional neural networks; image classification; adversarial attack; pruning;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Carlini, Nicholas, and David Wagner. "Towards evaluating the robustness of neural networks," In 2017 ieee symposium on security and privacy (sp), pp. 39-57, May 2017.
2 K. He, X. Zhang, S. Ren, and J. Sun, ''Deep residual learning for image recognition,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770-778.
3 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, ''Going deeper with convolutions,''in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1-9.
4 A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, ''MobileNets: Efficient convolutional neural networks for mobile vision applications,'' 2017, arXiv:1704.04861.
5 S. Ren, K. He, R. B. Girshick, and J. Sun, ''Faster R-CNN: Towards re- al time object detection with region proposal networks,'' in Proc. NIPS, 2015, pp. 91-99.
6 H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, ''Pruning filters for efficient ConvNets,'' 2016, arXiv:1608.08710.
7 Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
8 R. Girshick, J. Donahue, T. Darrell, and J. Malik, ''Rich feature hierarchies for accurate object detection and semantic segmentation,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580-587.
9 J. Long, E. Shelhamer, and T. Darrell, ''Fully convolutional networks f or semantic segmentation,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3431-3440.
10 Y. He, P. Liu, Wang Z. Hu, and Y. Yang, "Filter pruning via geometric median for deep convolutional neural networks acceleration," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 4340-4349.
11 S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, ''Learned step size quantization,'' 2019, arXiv:1902.08153.
12 Z. Wang, X. Cheng, G. Sapiro, and Q. Qiu, ''ACDC: Weight sharing in atom-coefficient decomposed convolution,'' 2020, arXiv:2009.02386.
13 V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, ''Speeding-up convolutional neural networks using fine-tuned CP- decomposition,'' in Proc. 3rd Int. Conf. Learn. Represent., (ICLR) Conf. Track, 2015, pp. 2-12.
14 Y. Guo, A. Yao, and Y. Chen, ''Dynamic network surgery for efficient DNNs,'' in Proc. NIPS, 2016, pp. 1387-1395.
15 Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).
16 Weng, Tsui-Wei, et al. "Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach," In International Conference on Learning Representations, 2018.
17 Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. "Towards deep learning models resistant to adversarial attacks," In International Conference on Learning Representations, 2018.
18 Han. S., Pool. J., Tran. J., and Dally. W. J. "Learning both weights and connections for efficient neural network.", in Proc. Advances in neural information processing systems, Jun. 2015, pp.1135 - 1143.
19 Molchanov, D., Ashukha, A., and Vetrov, D. "Variational dropout sparsifies deep neural networks," In International Conference on Machine Learning ,pp. 2498-2507, July 2017.
20 Lu, J., Issaranon, T., and Forsyth, D. Safetynet: Detecting and rejecting adversarial examples robustly. In Proceedings of the IEEE International Conference on Computer Vision, pp. 446-454, 2017.
21 L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834-848, Apr. 2017.   DOI
22 Guo, Y., Zhang, C., Zhang, C., and Chen, Y. "Sparse dnns with improved adversarial robustness," Advances in Neural Information Processing Systems, 31:242-251, 2018
23 Ye, S., Xu, K., Liu, S., Cheng, H., Lambrechts, J.-H., Zhang, H., Zhou, A., Ma, K., Wang, Y., and Lin, X. "Adversarial robustness vs. model compression, or both?," In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 111-120, 2019.