대한임베디드공학회논문지 (IEMEK Journal of Embedded Systems and Applications)

대한임베디드공학회 (Institute of Embedded Engineering of Korea)
권호 표지
DOI QR코드

DOI QR Code

신경망 근사에 의한 다중 레이어의 클래스 활성화 맵을 이용한 블랙박스 모델의 시각적 설명 기법

Visual Explanation of Black-box Models Using Layer-wise Class Activation Maps from Approximating Neural Networks

  • 투고 : 2021.06.22
  • 심사 : 2021.07.22
  • 발행 : 2021.08.31
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

In this paper, we propose a novel visualization technique to explain the predictions of deep neural networks. We use knowledge distillation (KD) to identify the interior of a black-box model for which we know only inputs and outputs. The information of the black box model will be transferred to a white box model that we aim to create through the KD. The white box model will learn the representation of the black-box model. Second, the white-box model generates attention maps for each of its layers using Grad-CAM. Then we combine the attention maps of different layers using the pixel-wise summation to generate a final saliency map that contains information from all layers of the model. The experiments show that the proposed technique found important layers and explained which part of the input is important. Saliency maps generated by the proposed technique performed better than those of Grad-CAM in deletion game.

키워드

과제정보

이 논문은 정부 (과학기술통신부)의 재원으로 한국연구재단 (No. 2019R1F1A1061941) 및 전북대학교 인공지능응용기술연구센터의 지원을 받아 수행된 연구임.

참고문헌

  1. Sample, Ian, "Computer says no: Why Mmaking AIs fair, Accountable and Transparent is Crucial". the Guardian. last modified, Vol. 5, No. 1, pp. 1-15, 2017.
  2. A. Holzinger, M. Plass, K. Holzinger, G.C. Crisan, C.M. Pintea, V Palade, "A Glass-box Interactive Machine Learning Approach for Solving NP-hard Problems with the Human-in-the-loop.", arXiv preprint arXiv:1708.01104, 2017
  3. L. Edwards, M. Veale. "Slave to the Algorithm: Why a Right to an Explanation is Probably not the Remedy you are Looking for." Duke L. & Tech, Vol. 16, No. 18, 2017.
  4. B. Zhou, A. Khosla, A. Lapedriza, "Learning Deep Features for Discriminative Localization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., pp. 2921-2929, 2016.
  5. R.R. Selvaraju, M. Cogswell, A. Das, "Grad-cam: Visual Explanations from Deep Networks via Gradient-based Localization." Proceedings of the IEEE International Conference on Computer Vision., pp. 618-626, 2017.
  6. M.T. Ribeiro, S. Singh, C. Guestrin, "Why Should I trust you?: Explaining the Predictions of any Classifier." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 1135-1144, 2016.
  7. V. Petsiuk, A. Das, K. Saenko, "Rise: Randomized Input Sampling for Explanation of black-box models.", arXiv preprint arXiv:1806.07421, 2018
  8. C. Fong, A. Vedaldi, "Interpretable Explanations of Black Boxes by Meaningful Perturbation." Proceedings of the IEEE International Conference on Computer Vision., pp. 3429-3437, 2017
  9. G. Hinton, O. Vinyals, J. Dean, "Distilling the Knowledge in a Neural Network.", arXiv preprint arXiv:1503.02531, 2015.
  10. S.H. Woo, J.C. Park, J.Y. Lee, I.S. Kweon "Cbam: Convolutional Block Attention Module." Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
  11. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin, "Attention is all you Need." Advances in Neural Information Processing Systems., pp. 5998-6008, 2017.
  12. F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, X. Tang, "Residual Attention Network for Image Classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., pp. 3156-3164, 2017.
  13. J. Hu, L. Shen, G. Sun, "Squeeze-and-excitation Networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., pp. 7132-7141, 2018.
  14. X. Wang, R. Girshick, A. Gupta, K. He, "Non-local Neural Networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., pp. 7794-7803, 2018.
  15. J. Deng, W. Dong, R. Socher, .LJ. Li, K Li, L. Fei-Fei "Imagenet: A Large-scale Hierarchical Image Database." 2009 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, pp. 248-255, 2009.
  16. C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie, "The Caltech-ucsd Birds-200-2011 dataset.", 2011.
  17. J. Krause, M. Stark, J. Deng, L. Fei-Fei, "3d Object Representations for Fine-grained Categorization." Proceedings of the IEEE International Conference on Computer Vision Workshops., pp. 554-561, 2013.
  18. S. Maji, E. Rahtu, J. Kannala, M. Blaschko, "Fine-grained Visual Classification of Aircraft.", arXiv preprint arXiv:1306.5151, 2013.