Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Pruning for Robustness by Suppressing High Magnitude and Increasing Sparsity of Weights

  • Cho, Incheon (Department of Computer Science and Engineering, Kyung Hee Univ.) ;
  • Ali, Muhammad Salman (Department of Computer Science and Engineering, Kyung Hee Univ.) ;
  • Bae, Sung-Ho (Department of Computer Science and Engineering, Kyung Hee Univ.)
  • Received : 2021.10.25
  • Accepted : 2021.11.30
  • Published : 2021.12.20
Download PDF
⟨ Previous Next ⟩

Abstract

Although Deep Neural Networks (DNNs) have shown remarkable performance in various artificial intelligence fields, it is well known that DNNs are vulnerable to adversarial attacks. Since adversarial attacks are implemented by adding perturbations onto benign examples, increasing the sparsity of DNNs minimizes the propagation of errors to high-level layers. In this paper, unlike the traditional pruning scheme removing low magnitude weights, we eliminate high magnitude weights that are usually considered high absolute values, named 'reverse pruning' to ensure robustness. By conducting both theoretical and experimental analyses, we observe that reverse pruning ensures the robustness of DNNs. Experimental results show that our reverse pruning outperforms previous work with 29.01% in Top-1 accuracy on perturbed CIFAR-10. However, reverse pruning does not guarantee benign samples. To relax this problem, we further conducted experiments by adding a regularization term for the high magnitude weights. With adding the regularization term, we also applied conventional pruning to ensure the robustness of DNNs.

Keywords

Acknowledgement

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (2018R1C1B3008159). Also, this research was a result of a study on the "HPC Support" Project, supported by the 'Ministry of Science and ICT' and NIPA.

References

  1. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, ''Going deeper with convolutions,''in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1-9.
  2. K. He, X. Zhang, S. Ren, and J. Sun, ''Deep residual learning for image recognition,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770-778.
  3. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, ''MobileNets: Efficient convolutional neural networks for mobile vision applications,'' 2017, arXiv:1704.04861.
  4. R. Girshick, J. Donahue, T. Darrell, and J. Malik, ''Rich feature hierarchies for accurate object detection and semantic segmentation,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580-587.
  5. S. Ren, K. He, R. B. Girshick, and J. Sun, ''Faster R-CNN: Towards re- al time object detection with region proposal networks,'' in Proc. NIPS, 2015, pp. 91-99.
  6. J. Long, E. Shelhamer, and T. Darrell, ''Fully convolutional networks f or semantic segmentation,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3431-3440.
  7. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834-848, Apr. 2017. https://doi.org/10.1109/TPAMI.2017.2699184
  8. H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, ''Pruning filters for efficient ConvNets,'' 2016, arXiv:1608.08710.
  9. Y. He, P. Liu, Wang Z. Hu, and Y. Yang, "Filter pruning via geometric median for deep convolutional neural networks acceleration," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 4340-4349.
  10. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, ''Learned step size quantization,'' 2019, arXiv:1902.08153.
  11. Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
  12. Z. Wang, X. Cheng, G. Sapiro, and Q. Qiu, ''ACDC: Weight sharing in atom-coefficient decomposed convolution,'' 2020, arXiv:2009.02386.
  13. V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky, ''Speeding-up convolutional neural networks using fine-tuned CP- decomposition,'' in Proc. 3rd Int. Conf. Learn. Represent., (ICLR) Conf. Track, 2015, pp. 2-12.
  14. Han. S., Pool. J., Tran. J., and Dally. W. J. "Learning both weights and connections for efficient neural network.", in Proc. Advances in neural information processing systems, Jun. 2015, pp.1135 - 1143.
  15. Y. Guo, A. Yao, and Y. Chen, ''Dynamic network surgery for efficient DNNs,'' in Proc. NIPS, 2016, pp. 1387-1395.
  16. Molchanov, D., Ashukha, A., and Vetrov, D. "Variational dropout sparsifies deep neural networks," In International Conference on Machine Learning ,pp. 2498-2507, July 2017.
  17. Carlini, Nicholas, and David Wagner. "Towards evaluating the robustness of neural networks," In 2017 ieee symposium on security and privacy (sp), pp. 39-57, May 2017.
  18. Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).
  19. Lu, J., Issaranon, T., and Forsyth, D. Safetynet: Detecting and rejecting adversarial examples robustly. In Proceedings of the IEEE International Conference on Computer Vision, pp. 446-454, 2017.
  20. Weng, Tsui-Wei, et al. "Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach," In International Conference on Learning Representations, 2018.
  21. Guo, Y., Zhang, C., Zhang, C., and Chen, Y. "Sparse dnns with improved adversarial robustness," Advances in Neural Information Processing Systems, 31:242-251, 2018
  22. Ye, S., Xu, K., Liu, S., Cheng, H., Lambrechts, J.-H., Zhang, H., Zhou, A., Ma, K., Wang, Y., and Lin, X. "Adversarial robustness vs. model compression, or both?," In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 111-120, 2019.
  23. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. "Towards deep learning models resistant to adversarial attacks," In International Conference on Learning Representations, 2018.