Smart Media Journal (스마트미디어저널)

THE KOREAN INSTITUTE OF SMART MEDIA (한국스마트미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Apple detection dataset with visibility and deep learning detection using adaptive heatmap regression

가시성을 표시한 사과 검출 데이터셋과 적응형 히트맵 회귀를 이용한 딥러닝 검출

  • 유태웅 (전북대학교 컴퓨터인공지능학부/영상정보신기술연구센터) ;
  • 서다솜 (전북대학교 전자정보공학부(컴퓨터 공학） ) ;
  • 김민우 (전북대학교병원 의생명연구원) ;
  • 이슬기 (국립원예특작과학원 과수과) ;
  • 오일석 (전북대학교 컴퓨터인공지능학부/영상정보신기술연구센터)
  • Received : 2023.10.30
  • Accepted : 2023.11.17
  • Published : 2023.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

In the fruit harvesting field, interest in automatic robot harvesting is increasing due to various seasonality and rising harvesting costs. Accurate apple detection is a difficult problem in complex orchard environments with changes in light, vibrations caused by wind, and occlusion of leaves and branches. In this paper, we introduce a dataset and an adaptive heatmap regression model that are advantageous for robot automatic apple harvesting. The apple dataset was labeled with not only the apple location but also the visibility. We propose a method to detect the center point of an apple using an adaptive heatmap regression model that adjusts the Gaussian shape according to visibility. The experimental results showed that the performance of the proposed method was applicable to apple harvesting robots, with MAP@K of 0.9809 and 0.9801 when K=5 and K=10, respectively.

과실 수확 분야에서 다양한 계절성과 수확 비용 상승 등으로 자동 로봇 수확에 대한 관심이 증가하고 있다. 빛의 변화, 바람에 의한 진동, 나뭇잎 및 가지 겹침 등 복잡한 과수원 환경에서 정확한 사과 검출은 어려운 문제이다. 본 논문에서는 로봇 자동 사과 수확에 유리한 데이터셋과 적응형 히트맵 회귀 모델을 소개한다. 사과 데이터셋은 사과 위치뿐만 아니라 가시성을 같이 레이블링하였다. 가시성에 따라 가우시안 모양을 조절하는 적응형 히트맵 회귀 모델을 사용하여 사과 중심점을 검출하는 방법을 제안한다. 실험 결과 MAP@K가 K=5와 K=10일 때 0.9809, 0.9801로 사과 수확 로봇에 응용 가능한 성능을 나타내었다.

Keywords

Acknowledgement

본 논문은 농촌진흥청 연구사업(과제번호: PJ015618)의 지원에 의해 이루어진 것임.

References

  1. K. Zhang, K. Lammers, P. Chu, Z. Li, and R. Lu, "System design and control of an apple harvesting robot," Mechatronics, vol. 79, 102644, Nov. 2021. 
  2. H. J. Cho, "A Study on Environmental Factor Recommendation Technology based on Deep Learning for Digital Agriculture," Smart Media Journal, Vol. 12, No. 5, pp. 65-72, Jun. 2023.  https://doi.org/10.30693/SMJ.2023.12.5.65
  3. J. H. Bae, J. H. Lee, G H. Yu, G. J. Kwon, J. Y. Kim, "A Study about Learning Graph Representation on Farmhouse Apple Quality Images with Graph Transformer," Smart Media Journal, Vol. 12, No. 1, pp. 9-16, Feb. 2023.  https://doi.org/10.30693/SMJ.2023.12.1.9
  4. J. H. Bang, J. Park, S. W. Park, "A System for Determining the Growth Stage of Fuit Tree Using a Deep Learning-Based Object Detection Model," Smart Media Journal, Vol. 11, No. 4, pp. 9-18, May. 2022.  https://doi.org/10.30693/SMJ.2022.11.4.9
  5. Y. Tang, M. Chen, C. Wang, L. Luo, J. Li, G. Lian, and X. Zou, "Recognition and localization methods for vision-based fruit picking robots: A review" Frontiers in Plant Science, vol. 11, 2020. 
  6. J. Naranjo-Torres, M. Mora, R. Hernandez-Garcia, R.J. Barrientos, C. Fredes, and A. Valenzuela, "A review of convolutional neural network applied to fruit image processing," Applied Sciences, 10(10), 3443, 2020. 
  7. H. Kang, and C. Chen, "Fast implementation of real-time fruit detection in apple orchards using deep learning," Computers and Electronics in Agriculture, vol. 168, 105108, 2020. 
  8. N. Hani, P. Roy and V. Isler, "MinneApple: A benchmark dataset for apple detection and segmentation," in IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 852-858, April 2020, doi: 10.1109/LRA.2020.2965061. 
  9. S. Bargoti and J.P. Underwood, "Deep fruit detection in orchards," 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 2017, pp. 3626-3633, doi: 10.1109/ICRA.2017.7989417. 
  10. S. Bargoti, and J.P. Underwood, "Image segmentation for fruit detection and yield estimation in apple orchards," Journal of Field Robotics, vol. 34, issue 6, pp. 1039-1060, 2017.  https://doi.org/10.1002/rob.21699
  11. J. Gene-Mola, R. Sanz-Cortiella, J.R. Rosell-Polo, J.R. Morros, J. Ruiz-Hidalgo, V. Vilaplana, and E. Gregorio, "Fuji-SfM dataset: A collection of annotated images and point clouds for Fuji apple detection and location using structure-from-motion photogrammetr. Data in brief, vol. 30, 105591, Jun. 2020. 
  12. F. Xiao, H. Wang, Y. Xu, and R. Zhang, "Fruit detection and recognition based on deep learning for automatic harvesting: an overview and review," Agronomy, vol. 13, issue 6, 1625, 2023. 
  13. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once- Unified, real-time object detection," In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, May. 2016. 
  14. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.Y. Fu, and AC.. Berg, "Ssd: Single shot multibox detector," In Computer Vision - ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 -14, Proceedings, Part I 14, Springer International Publishing, pp. 21-37, 2016. 
  15. S. Wan, and S. Goudo, "Faster R-CNN for multi-class fruit detection using a robotic vision system," Computer Networks, vol. 168, 107036, Feb. 2020. 
  16. P. Chu, Z. Li, K. Lammers, R. Lu, and X. Liu," Deep learning-based apple detection using a suppression mask R-CNN," Pattern Recognition Letters, vol. 147, pp. 206-211, Jul. 2021.  https://doi.org/10.1016/j.patrec.2021.04.022
  17. G. Lin, Y. Tang, X. Zou, J. Xiong, and J. Li," Guava detection and pose estimation using a low-cost RGB-D sensor in the field," Sensors, vol. 19, issue 2, 428, 2019. 
  18. K. Bresilla, G.D. Perulli, A. Boini, B. Morandi, L.-C. Grappadelli, and L. Manfrini, "Single-shot convolution neural networks for real-time fruit detection within the tree," Frontiers in plant science, vol. 10, 611, 2019. 
  19. Y. Tian, G. Yang, Z. Wang, H. Wang, E. Li, and Z. Liang, "Apple detection during different growth stages in orchards using the improved YOLO-V3 mode," Computers and electronics in agriculture, vol. 157, pp. 417-426, 2019.  https://doi.org/10.1016/j.compag.2019.01.012
  20. L. -C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, "DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834-848, 1 April 2018, doi: 10.1109/TPAMI.2017.2699184. 
  21. B. Yan, P. Fan, X. Lei, Z. Liu, and F. Yang, "A real-time apple targets detection method for picking robot based on improved YOLOv5," Remote Sensing, vol. 13, issue 9, 1619, 2021. 
  22. J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," Proceedings of the IEEE coherence on computer vision and pattern recognition, USA, Jun. 2018. 
  23. W. Chen, J. Zhang, B. Guo, Q. Wei, and Z. Zhu, "An apple detection method based on des-YOLO v4 algorithm for harvesting robots in complex environment," Mathematical Problems in Engineering, vol. 2021, 1-12, 2021.  https://doi.org/10.1155/2021/7351470
  24. G. Yang, J. Wang, Z. Nie, H. Yang, and S. Yu, "A lightweight YOLOv8 tomato detection algorithm combining feature enhancement and attention.," Agronomy, vol. 13 issue 7, 1824, 2023. 
  25. G. Liu, Z. Hou, H. Liu, J. Liu, W. Zhao, and K. Li, "TomatoDet: Anchor-free detector for tomato detection," Frontiers in Plant Science, 13, 942875, Aug. 2022. 
  26. X. Zhou, D. Wang, and P. Krahenbuhl. "Objects as points," arXiv preprint arXiv:1904.07850, 2019.
  27. K. Zhao, and W.Q. Yan. "Fruit detection from digital images using CenterNet," Geometry and Vision: First International Symposium, ISGV 2021, pp. 313-326, Auckland, New Zealand, January 28-29, 2021. 
  28. O. Ronneberger, P. Fischer, T. Brox, "U-Net: convolutional networks for biomedical image segmentation," Medical Image Computing and Computer-Assisted Intervention : MICCAI 2015, Lecture Notes in Computer Science, vol. 9351, pp. 234-241, Nov. 2015. 
  29. P. Cremonesi, Y. Koren, and R. Turrin, "Performance of recommender algorithms on top-n recommendation tasks," In Proceedings of the fourth ACM conference on Recommender systems, pp. 39-46, Sep. 2010.