The Journal of Korea Robotics Society (로봇학회논문지)

Korea Robotics Society (한국로봇학회)
권호 표지
DOI QR코드

DOI QR Code

Design of Vehicle-mounted Loading and Unloading Equipment and Autonomous Control Method using Deep Learning Object Detection

차량 탑재형 상·하역 장비의 설계와 딥러닝 객체 인식을 이용한 자동제어 방법

  • Soon-Kyo Lee (Future Transport Logistics Research Center, Korea Railroad Research Institute) ;
  • Sunmok Kim (Dept. Electrical Engineering, Kwangwoon University) ;
  • Hyowon Woo (Dept. Electrical Engineering, Kwangwoon University) ;
  • Suk Lee (Future Transport Logistics Research Center, Korea Railroad Research Institute) ;
  • Ki-Baek Lee (Dept. Electrical Engineering, Kwangwoon University)
  • Received : 2023.11.06
  • Accepted : 2023.12.18
  • Published : 2024.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

Large warehouses are building automation systems to increase efficiency. However, small warehouses, military bases, and local stores are unable to introduce automated logistics systems due to lack of space and budget, and are handling tasks manually, failing to improve efficiency. To solve this problem, this study designed small loading and unloading equipment that can be mounted on transportation vehicles. The equipment can be controlled remotely and is automatically controlled from the point where pallets loaded with cargo are visible using real-time video from an attached camera. Cargo recognition and control command generation for automatic control are achieved through a newly designed deep learning model. This model is designed to be optimized for loading and unloading equipment and mission environments based on the YOLOv3 structure. The trained model recognized 10 types of palettes with different shapes and colors with an average accuracy of 100% and estimated the state with an accuracy of 99.47%. In addition, control commands were created to insert forks into pallets without failure in 14 scenarios assuming actual loading and unloading situations.

Keywords

Acknowledgement

This work is supported by the Korea Agency for Infrastructure Technology Advancement(KAIA) grant funded by the Ministry of Land, Infrastructure and Transport (Grant RS-2021-KA163201). Additionally, this paper was researched through the 2021 Kwangwoon University Outstanding Researcher Support Project

References

  1. G. Garibotto, S. Masciangelo, P. Bassino, and M. Ilic, "Computer vision control of an intelligent forklift truck," Conference on Intelligent Transportation Systems, Boston, MA, USA, pp. 589-594, 1997, DOI: 10.1109/ITSC.1997.660540. 
  2. M. Li, S. Gu, G. Chen, and Z. Zhu, "A RFID-based Intelligent Warehouse Management System Design and Implementation," 2011 IEEE 8th International Conference on e-Business Engineering, Beijing, China, pp. 178-184, 2011, DOI: 10.1109/ICEBE.2011.28. 
  3. M. Seelinger and J.-D. Yoder, "Automatic Pallet Engagment by a Vision Guided Forklift," 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, pp. 4068-4073, 2005, DOI: 10.1109/ROBOT.2005.1570744. 
  4. J. Pages, X. Armangue, J. Salvi, J. Freixenet, and J. Marti, "A Computer Vision System for Autonomous Forklift Vehicles in Industrial Environments," 9th Mediterranean Conference on Control and Automation MEDS, Dubrovnik, Croatia, pp. 1-6, 2001, [Online], http://eia.udg.es/~qsalvi/papers/2001-MEDS.pdf. 
  5. T. Li, B. Huang, C. Li, and M. Huang, "Application of convolution neural network object detection algorithm in logistics warehouse," The Journal of Engineering, vol. 2019, no. 23, pp. 9053-9058, Dec., 2019, DOI: 10.1049/joe.2018.9180. 
  6. Y.-Y. Li, X.-H. Chen, G.-Y. Ding, S. Wang, W.-C. Xu, B.-B. Sun, and Q. Song, "Pallet detection and localization with RGB image and depth data using deep learning techniques," 2021 6th International Conference on Automation, Control and Robotics Engineering (CACRE), Dalian, China, pp. 306-310, 2021, DOI: 10.1109/CACRE52464.2021.9501390. 
  7. J. Ren, Y. Pan, P. Yao, Y. Hu, W. Gao, and Z. Xue, "Deep Learning-Based Intelligent Forklift Cargo Accurate Transfer System," Sensors, vol. 22, no. 21, pp. 8437, Nov., 2022, DOI: 10.3390/s22218437. 
  8. S. V. Carata, M. Ghenescu, and R. Mihaescu, "Real-Time Detection of Unrecognized Objects in Logistics Warehouses Using Semantic Segmentation," Mathematics, vol. 11, no. 11, pp. 2445, May, 2023, DOI: 10.3390/math11112445. 
  9. H. Yin, C. Chen, C. Hao, and B. Huang, "A Vision-based inventory method for stacked goods in stereoscopic warehouse," Neural Computing and Applications, vol. 34, pp. 20773-20790, Jul., 2022, DOI: 10.1007/s00521-022-07551-4. 
  10. C. Mok, I. Baek, Y. S. Cho, Y. Kim, and S. B. Kim, "Pallet recognition with multi-task learning for automated guided vehicles," Applied Sciences, vol. 11, no. 24, Dec., 2021, DOI: 10.3390/app112411808. 
  11. Y. Shao, Z. Fan, B. Zhu, M. Zhou, Z. Chen, and J. Lu, "A Novel Pallet Detection Method for Automated Guided Vehicles Based on Point Cloud Data," Sensors, vol. 22, no. 20, Oct., 2022, DOI: 10.3390/s22208019. 
  12. M. Zaccaria, R. Monica, and J. Aleotti, "A Comparison of Deep Learning Models for Pallet Detection in Industrial Warehouses," 2020 IEEE 16th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, pp. 417-422, 2020, DOI: 10.1109/ICCP51029.2020.9266168. 
  13. Y. Mo, Z. Sun, and C. Yu, "EventTube: An Artificial Intelligent Edge Computing Based Event Aware System to Collaborate With Individual Devices in Logistics Systems," IEEE Transactions on Industrial Informatics, vol. 19, no. 2, pp. 1823-1832, Feb., 2023, DOI: 10.1109/TII.2022.3189177. 
  14. M. Liu, X. Xu, X. Wang, Q. Jiang, and C. Liu, "Intelligent monitoring method of tridimensional storage system based on deep learning," Environmental Science and Pollution Research, vol. 29, pp. 70464-70478, May, 2022, DOI: 10.1007/s11356-022-20658-4. 
  15. H. Xiong, J. Wu, Q. Liu, and Y. Cai, "Research on abnormal object detection in specific region based on Mask R-CNN," International Journal of Advanced Robotic Systems, vol. 17, no. 3, May, 2020, DOI: 10.1177/1729881420925287. 
  16. Z. Li, K. Lu, Y. Zhang, Z. Li, and J.-B. Liu, "Research on Energy Efficiency Management of Forklift Based on Improved YOLOv5 Algorithm," Journal of Mathematics, vol. 2021, Dec., 2021, DOI: 10.1155/2021/5808221. 
  17. J. Redmon, and A. Farhadi, "Yolov3: An incremental improvement," arXiv:1804.02767, Apr., 2018, DOI: 10.48550/arXiv.1804.02767. 
  18. T. Allison, "Freight-handling technologies and industrial building design: freighthouse and warehouse facilities of the Chicago junction railway, 1900-30," Industrial Archaeology Review, vol. 36, no. 2, pp. 109-127, Dec., 2014, DOI: 10.1179/0309072814Z.00000000034. 
  19. A. Andelkovic and M. Radosavljevic, "Improving order-picking process through implementation of warehouse management system," Strategic Management, vol. 23, no. 1, pp. 3-10, Jan., 2018, DOI: 10.5937/STRAMAN1801003A. 
  20. M. Brambilla, E. Ferrante, M. Birattari, and M. Dorigo, "Swarm robotics: a review from the swarm engineering perspective," Swarm Intelligence, vol. 7, no. 1, pp. 1-41, Jan., 2013, DOI: 10.1007/s11721-012-0075-2. 
  21. M. Al-Obaidy and R. Al-Azawi, "Cluster-based Algorithm for Energy Optimization of Swarmed Robots using Swarm Intelligence," 2019 Sixth HCT Information Technology Trends (ITT), Ras Al Khaimah, United Arab Emirates, pp. 202-207, 2019, DOI: 10.1109/ITT48889.2019.9075119. 
  22. J. A. Cano, F. Salazar, R. A. Gomez-Montoya, and P. Cortes, "Disruptive and conventional technologies for the support of logistics processes: a literature review," International Journal of Technology, vol. 12, no. 3, pp. 448-460, Jul., 2021, DOI: 10.14716/ijtech.v12i3.4280. 
  23. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, pp. 580-587, 2014, DOI: 10.1109/CVPR.2014.81. 
  24. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, "Ssd: Single shot multibox detector," Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, Netherlands, pp. 11-14, 2016, DOI: 10.1007/978-3-319-46448-0_2. 
  25. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 779-788, 2016, DOI: 10.1109/CVPR.2016.91. 
  26. J. Redmon and A. Farhadi, "YOLO9000: better, faster, stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 7263-7271, 2017, DOI: 10.1109/CVPR.2017.690. 
  27. G. Scheithauer and J. Terno, "A new heuristic for the pallet loading problem," Operations Research Proceeding 1995, Passau, Germany, pp. 84-89, 1996, DOI: 10.1007/978-3-642-80117-4_15. 
  28. E. G. Birgin, R. Morabito, and F. H. Nishihara, "A note on an L-approach for solving the manufacturer's pallet loading problem," Journal of Operations Research Society, vol. 56, no. 12, pp. 1448-1451, Mar, 2005, DOI: 10.1057/palgrave.jors.2601960. 
  29. L. Wang, S. Shang, and Z. Wu, "Research on Indoor 3D Positioning Algorithm Based on WiFi Fingerprint," Sensors, vol. 23, no. 1, pp. 153, Dec., 2022, DOI: 10.3390/s23010153. 
  30. K. C. Wu and C. J. Ting, "A two-phase algorithm for solving the manufacturer's pallet loading problem," 2007 IEEE International Conference on Industrial Engineering and Engineering Management, Singapore, pp.1574-1578, 2007, DOI: 10.1109/IEEM.2007.4419457. 
  31. S. Ahn, K. Yoon, and J. Park, "A best-first branch and bound algorithm for the pallet-loading problem," International Journal of Production Research, vol.53, no. 3, pp.835-849, Jul., 2014, DOI: 10.1080/00207543.2014.935824. 
  32. H. Zhang, Q. Wang, C. Yan, J. Xu, and B. Zhang, "Research on UWB Indoor Positioning Algorithm under the Influence of Human Occlusion and Spatial NLOS," Remote Sensing, vol. 14, no. 24, pp. 6338, Dec., 2022, DOI: 10.3390/rs14246338.