Journal of Software Engineering Society (소프트웨어공학소사이어티 논문지)

Software Engineering Society (한국정보과학회 소프트웨어공학 소사이어티)
권호 표지

Development of an Image Data Augmentation Apparatus to Evaluate CNN Model

CNN 모델 평가를 위한 이미지 데이터 증강 도구 개발

  • 최영원 (부산대학교 전기전자컴퓨터공학과) ;
  • 이영우 (부산대학교 전기전자컴퓨터공학과) ;
  • 채흥석 (부산대학교 전기컴퓨터공학부)
  • Received : 2020.02.28
  • Accepted : 2020.03.01
  • Published : 2020.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

As CNN model is applied to various domains such as image classification and object detection, the performance of CNN model which is used to safety critical system like autonomous vehicles should be reliable. To evaluate that CNN model can sustain the performance in various environments, we developed an image data augmentation apparatus which generates images that is changed background. If an image which contains object is entered into the apparatus, it extracts an object image from the entered image and generate s composed images by synthesizing the object image with collected background images. A s a method to evaluate a CNN model, the apparatus generate s new test images from original test images, and we evaluate the CNN model by the new test image. As a case study, we generated new test images from Pascal VOC2007 and evaluated a YOLOv3 model with the new images. As a result, it was detected that mAP of new test images is almost 0.11 lower than mAP of the original test images.

CNN 모델이 이미지 분류와 객체 탐지 등 여러 분야에 활용됨에 따라, 자율주행자동차와 같이 안전필수시스템에 사용되는 CNN 모델의 성능은 신뢰할 수 있어야 한다. 이에 CNN 모델이 다양한 환경에서도 성능을 유지하는지 평가하기 위해 배경을 변경한 이미지를 생성하는 이미지 데이터 증강 도구를 개발한다. 이미지 데이터 증강 도구에 객체가 존재하는 이미지를 입력하면, 해당 이미지로부터 객체 이미지를 추출한 후 수집한 배경 이미지 내에 객체 이미지를 합성하여 새로운 이미지를 생성한다. CNN 모델 성능 평가 방법으로 개발한 도구를 사용하여 기존 테스트 이미지로부터 새로운 테스트 이미지를 생성하고, 생성한 새로운 테스트 이미지로 CNN 모델을 평가한다. 사례 연구로 Pascal VOC2007 테스트 데이터로부터 새로운 테스트 이미지를 생성하고, 새로운 테스트 이미지로 YOLOv3 모델을 평가하였다. 그 결과 기존 테스트 이미지의 mAP 보다 새로운 테스트 이미지의 mAP가 약 0.11 더 낮아지는 것을 확인하였다.

Keywords

References

  1. P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, "Importance Estimation for Neural Network Pruning," in CVPR. pp. 11264-10272, 2019.
  2. A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," in NIPS, pp. 1097-1105, 2012.
  3. C. Szegedy. W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going Deeper with Convolutions," in CVPR, pp. 1-9, 2015.
  4. K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in CVPR, pp. 770-778, 2016.
  5. C. Szegedy, A. Toshev, and D. Erhan, "Deep Neural Networks for Object Detection," in NIPS, pp. 2553-2561, 2013.
  6. C. Szegedy, A. Toshev, and D. Erhan, "Deep Neural Networks for Object Detection," in NIPS, pp. 2553-2561, 2013.
  7. A. Dosovitskiy, G. Ros, F. Codevilla , A. Lopez, and V. Koltun, "CARLA: An Open Urban Driving Simulator," in CoRL,, pp. 1-16, 2017.
  8. S. Segura, G. Fraser, A.B. Sanchez, and A. Ruiz-Corts, "A Survey on Metamorphic Testing," IEEE TSE, Vol. 42, No. 9, pp. 805-824, 2016.
  9. T. Y. Chen, S. C. Cheung, and S. M. Yiu, "Metamorphic Testing: A New Approach for Generating Next Test Cases," Computer Science, Hong Kong Univ. of Science and Technology, Tech. HKUST-CS98-01, 1998.
  10. J. Canny, "A Computational Approach to Edge Detection," IEEE TPAMI, Vol. 8, No. 6 , pp. 679-698, 1986. https://doi.org/10.1109/TPAMI.1986.4767851
  11. C . Rother, V. Kolmogorov, and A. Blake, "'Grabcut': Interactive Foreground Extraction Using Iterated Graph Cuts," in ACM TOG, Vol. 23 , No. 3, pp. 309-314, 2004.
  12. J. Redmon, and A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv preprint arXiv: 1804.02767, 2018.
  13. S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in NIPS, pp. 91-99, 2015.
  14. K . He, G. Gkioxari, P. Dollar, and R. Girshick, "Mask R-CNN," in ICCV, pp. 2961-2969, 2017.
  15. T .-Y. Lin, M.Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in ECCV, pp. 740-755, 2014.
  16. Y. Tian, K. Pei, S. Jana, and B. Ray, "DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars," in ICSE , pp. 303-314, 2018.
  17. M. Koziarski and B. Cyganek, "Image Recognition with Deep Neural Networks in Presence of Noi se-Dealing with and Taking Advantage of Distortions," ICAE, Vol. 24, No. 4, pp. 337-349, 2017.
  18. M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, "The Pascal Visual Object Classes (VOC) Challenge ," IJCV, Vol. 88, No. 2, pp. 303-338, 2010. https://doi.org/10.1007/s11263-009-0275-4
  19. J. Choi, D. Chun, H. Kim, and H.-j. Lee, "Gaussian YOLOv3 : An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving," in ICCV, pp. 502-511, 2019
  20. Q. Mao, H. Sun, Y. Liu, and R. Jia, "Mini-YOLOv3 : Real-Time Object Detector for Embedded Applications," IEEE Access, Vol. 7, No. 1, pp. 133529-133538, 2019. https://doi.org/10.1109/ACCESS.2019.2941547
  21. H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother, "Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes," IJCV, Vol. 126, No. 9, pp. 961-972, 2018 https://doi.org/10.1007/s11263-018-1070-x