Smart Media Journal (스마트미디어저널)

THE KOREAN INSTITUTE OF SMART MEDIA (한국스마트미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Road Image Recognition Technology based on Deep Learning Using TIDL NPU in SoC Enviroment

SoC 환경에서 TIDL NPU를 활용한 딥러닝 기반 도로 영상 인식 기술

  • 신윤선 (한동대학교 전산전자공학부) ;
  • 서주현 (한동대학교 전산전자공학부) ;
  • 이민영 (한동대학교 일반대학원 전산전자공학과) ;
  • 김인중 (한동대학교 전산전자공학부)
  • Received : 2022.10.05
  • Accepted : 2022.12.08
  • Published : 2022.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Deep learning-based image processing is essential for autonomous vehicles. To process road images in real-time in a System-on-Chip (SoC) environment, we need to execute deep learning models on a NPU (Neural Procesing Units) specialized for deep learning operations. In this study, we imported seven open-source image processing deep learning models, that were developed on GPU servers, to Texas Instrument Deep Learning (TIDL) NPU environment. We confirmed that the models imported in this study operate normally in the SoC virtual environment through performance evaluation and visualization. This paper introduces the problems that occurred during the migration process due to the limitations of NPU environment and how to solve them, and thereby, presents a reference case worth referring to for developers and researchers who want to port deep learning models to SoC environments.

자율주행 자동차에서 딥러닝 기반 영상처리는 매우 중요하다. 자동차를 비롯한 SoC(System on Chip) 환경에서 실시간으로 도로 영상을 처리하기 위해서는 영상처리 모델을 딥러닝 연산에 특화된 NPU(Neural Processing Unit) 상에서 실행해야 한다. 본 연구에서는 GPU 서버 환경에서 개발된 7종의 오픈소스 딥러닝 영상처리 모델들을 TIDL (Texas Instrument Deep Learning) NPU 환경에 이식하였다. 성능 평가와 시각화를 통해 본 연구에서 이식한 모델들이 SoC 가상환경에서 정상 작동함을 확인하였다. 본 논문은 NPU 환경의 제약으로 인해 이식 과정에 발생한 문제들과 그 해결 방법을 소개함으로써 딥러닝 모델을 SoC 환경에 이식하려는 개발자 및 연구자가 참고할 만한 사례를 제시한다.

Keywords

Acknowledgement

이 논문은 2020년도 정부(산업통상자원부)의 재원으로 한국산업기술평가관리원의 지원을 받아 수행된 연구임 (No. 20009775). 본 연구는 과학기술정보통신부의 소프트웨어중심대학 지원사업(2017-0-00130)의 지원을 받아 수행하였음.

References

  1. K. He, X. Zhang. S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Proc. of the IEEE conference on computer vision and pattern recognition CVPR, pp. 770-778, 2016. 
  2. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4510-4520, 2018. 
  3. P. Chao, C. Kao, Y. Ruan, C. Huang, and Y. Lin, "HarDNet: A Low Memory Traffic Network," Proc. of the IEEE/CVF international conference on computer vision (ICCV), pp. 3552-3561, 2019. 
  4. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg, "SSD: Single Shot MultiBox Detector," European Conference on Computer Vision (ECCV), pp. 21-37. Cham. 2016. 
  5. A. Bochkovskiy, C. Wang, and H. M. Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection," arXiv preprint, arXiv:2004.10934, Apr. 2020. 
  6. T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, "Focal Loss for Dense Object Detection," Proc. of the IEEE international conference on computer vision (ICCV), pp. 2980-2988, 2017. 
  7. N. Dvornik, K. Shmelkov, J. Mairal, and C. Schmid, "BlitzNet: A Real-Time Deep Network for Scene Understanding," Proc. of the IEEE international conference on computer vision (ICCV), pp. 4154-4162, 2017. 
  8. L. Tabelini, R. Berriel, T. M. Paixao, C. Badue, A. F. D. Souza, and T. Oliveira-Santos, "Keep your Eyes on the Lane: Real-time Attention-guided Lane Detection," Proc. of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 294-302, 2021. 
  9. T. Tan and G. Cao, "FastVA: Deep Learning Video Analytics Through Edge Processing and NPU in Mobile," IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 1947-1956, IEEE, 2020. 
  10. A. Ignatov, R. Timofte, A. Kulik, S. Yang, K. Wang, F. Baum, M. Wu, L. Xu, and L. V. Gool, "AI Benchmark: All About Deep Learning on Smartphones in 2019," 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3617-3635, IEEE, 2019. 
  11. M. Mathew, K. Desappan, P. K. Swami, S. Nagori, and B. M. Gopinath, "Embedded low-power deep learning with tidl," Texas Instrum., Dallas, TX, USA, Tech. Rep. SPRY314, 2018. 
  12. K. He, X. Zhang. S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," Proc. of the IEEE transactions on pattern analysis and machine intelligence (TPAMI), vol. 37, no. 9, pp. 1904-1916, 2015.  https://doi.org/10.1109/TPAMI.2015.2389824
  13. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollar, "Microsoft COCO: Common Objects in Context," European Conference on Computer Vision, (ECCV), pp. 740-755. 2014. 
  14. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The Cityscapes Dataset for Semantic Urban Scene Understanding," Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213-3223, 2016. 
  15. J. Borrego-Carazo, D. Castells-Rufas, E. Biempica, and J. Carrabina, "Resource-Constrained Machine Learning for ADAS: A Systematic Review," in IEEE Access, vol. 8, pp. 40573-40598, 2020. https://doi.org/10.1109/access.2020.2976513