Browse > Article
http://dx.doi.org/10.3837/tiis.2021.02.003

High-Quality Coarse-to-Fine Fruit Detector for Harvesting Robot in Open Environment  

Zhang, Li (College of Information and Electrical Engineering, China Agricultural University)
Ren, YanZhao (College of Information and Electrical Engineering, China Agricultural University)
Tao, Sha (College of Information and Electrical Engineering, China Agricultural University)
Jia, Jingdun (College of Information and Electrical Engineering, China Agricultural University)
Gao, Wanlin (College of Information and Electrical Engineering, China Agricultural University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.2, 2021 , pp. 421-441 More about this Journal
Abstract
Fruit detection in orchards is one of the most crucial tasks for designing the visual system of an automated harvesting robot. It is the first and foremost tool employed for tasks such as sorting, grading, harvesting, disease control, and yield estimation, etc. Efficient visual systems are crucial for designing an automated robot. However, conventional fruit detection methods always a trade-off with accuracy, real-time response, and extensibility. Therefore, an improved method is proposed based on coarse-to-fine multitask cascaded convolutional networks (MTCNN) with three aspects to enable the practical application. First, the architecture of Fruit-MTCNN was improved to increase its power to discriminate between objects and their backgrounds. Then, with a few manual labels and operations, synthetic images and labels were generated to increase the diversity and the number of image samples. Further, through the online hard example mining (OHEM) strategy during training, the detector retrained hard examples. Finally, the improved detector was tested for its performance that proved superior in predicted accuracy and retaining good performances on portability with the low time cost. Based on performance, it was concluded that the detector could be applied practically in the actual orchard environment.
Keywords
Fruit Detection; Coarse-to-Fine; Synthetic Dataset; Harvesting Robot; Multitask Cascaded;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 D. Pise and G. D. Upadhye, "Grading of harvested mangoes quality and maturity based on machine learning techniques," in Proc. of 2018 International Conference on Smart City and Emerging Technology (ICSCET), 2018, pp. 1-6.
2 W. Xu, S. Lee, and E. Lee, "A Robust Method for Partially Occluded Face Recognition," KSII Transactions on Internet and Information Systems, vol. 9, no. 7, pp. 2667-2682, 2015.   DOI
3 J. Gene-Mola, V. Vilaplana, J. R. Rosell-Polo, J. R. Morros, J. Ruiz-Hidalgo, and E. Gregorio, "KFuji RGB-DS database: Fuji apple multi-modal images for fruit detection with color, depth and range-corrected IR data," Data in Brief, vol. 25, 2019.
4 L. Fu, Y. Feng, Y. Majeed, X. Zhang, J. Zhang, M. Karkee, and Q. Zhang, "Kiwifruit detection in field images using Faster R-CNN with ZFNet," IFAC-PapersOnLine, vol. 51, no. 17, pp. 45-50, 2018.   DOI
5 S. Tu, J. Pang, H. Liu, N. Zhuang, Y. Chen, C. Zheng, H. Wan, and Y. Xue, "Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images," Precision Agriculture, pp. 1-20, 2020.
6 W. Mao, B. Ji, J. Zhan, X. Zhang, and X. Hu, "Apple location method for the apple harvesting robot," in Proc. of the 2nd International Congress on Image and Signal Processing, pp. 1-5, 2009.
7 L. Zhang, G. Gui, A. M. Khattak, M. Wang, W. Gao, and J. Jia, "Multi-task cascaded convolutional networks based intelligent fruit detection for designing automated robot," IEEE Access, vol. 7, pp. 56028-56038, 2019.   DOI
8 X. D. Bai, Z. G. Cao, Y. Wang, Z. H. Yu, X. F. Zhang, and C. N. Li, "Crop segmentation from images by morphology modeling in the CIE L* a* b* color space," Computers and Electronics in Agriculture, vol. 99, pp. 21-34, 2013.   DOI
9 Z. Guo, Y. Shen, A. K. Bashir, M. Imran, N. Kumar, D. Zhang, and K. Yu, "Robust Spammer Detection Using Collaborative Neural Network in Internet of Thing Applications," IEEE Internet of Things Journal, early access.
10 Y. Wang, G. Gui, T. Ohtsuki, and F. Adachi, "Multi-task learning for generalized automatic modulation classification under non-Gaussian noise with varying SNR conditions," IEEE Transactionsa on Wireless Commuications, early access.
11 N. Hani, P. Roy, and V. Isler, "Minneapple: A benchmark dataset for apple detection and segmentation," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 852-858, 2020.   DOI
12 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779-788, 2016.
13 S. Ren. K. He, R. Girshick, and J. Sun, "Faster R-CNN Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions on Pattern Analysis and Machine Intellignece, vol. 39, no. 6, pp. 1137-1149, 2017.
14 L. Zhang, J. Jia, G. Gui, X. Hao, W. Gao, and M. Wang, "Deep learning based improved classification system for designing tomato harvesting robot," IEEE Access, vol. 6, pp. 67940-67950, 2018.   DOI
15 S. Bargoti and J. P. Underwood, "Image segmentation for fruit detection and yield estimation in apple orchards," Journal of Field Robotics, vol. 34, no. 6, pp. 1039-1060, 2017.   DOI
16 X. Liu, S. W. chen, S. Aditya, N. Sivakumar, S. Dcunha, C. Qu, C. J. Taylor, J. Das, and V. Kumar, "Robust fruit counting: Combining deep learning, tracking, and structure from motion," in Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1045-1052, 2018.
17 H. Gan, W. S. Lee, V. Alchanatis, R. Ehsani, and J. K. Schueller, "Immature green citrus fruit detection using color and thermal images," Computers and Electronics in Agriculture, vol. 152, pp. 117-125, 2018.   DOI
18 I. Sa, Z. Ge, F. Dayoub, B. Upcroft, T. Perez, and C. McCool, "Deepfruits: A fruit detection system using deep neural networks," Sensors, vol. 16, no. 8, pp. 1-23, 2016.
19 S. Bargoti and J. Underwood, "Deep fruit detection in orchards," in Proc. of 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3626-3633, 2017.
20 Z. Guo, K. Yu, Y. Li, G. Srivastava, and J. C. W. Lin, "Deep Learning-Embedded Social Internet of Things for Ambiguity-Aware Social Recommendations," IEEE Transactions on Network Science and Engineering, early access.
21 D. Dwibedi, I. Misra, and M. Hebert, "Cut, paste and learn: Surprisingly easy synthesis for instance detection," in Proc. of IEEE International Conference on Computer Vision, pp. 1310-1319, 2017.
22 Y. Lin, Y. Tu, Z. Dou, L. Chen, and S. Mao, "Contour stella image and deep learning for signal recognition in the physical layer," IEEE Transactions on Cognitive Communications and Networking, early access.
23 M. P. Arakeri and Lakshmana, "Computer vision based fruit grading system for quality evaluation of tomato in agriculture industry," Procedia Computer Science, vol. 79, pp. 426-433, 2016.   DOI
24 K. Yu, L. Lin, M. Alazab, L. Tan, and B. Gu, "Deep Learning-Based Traffic Safety Solution for a Mixture of Autonomous and Manual Vehicles in a 5G-Enabled Intelligent Transportation System," IEEE Transactions on Intelligent Transportation Systems, pp. 1-11, 2020.
25 Y. Lin, M. Wang, X. Zhou, G. Ding, and S. Mao, "Dynamic spectrum interaction of UAV flight formation communication with priority: A deep reinforcement learning approach," IEEE Transactions on Cognitive Communications and Networking, vol. 6, no. 3, pp. 892-903, 2020.   DOI
26 T. Nishi, S. Kurogi, and K. Matsuo, "Grading fruits and vegetables using RGB-D images and convolutional neural network," in Proc. of 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-6, 2017.
27 L. Hou, Q. Wu, Q. Sun, H. Yang, and P. Li, "Fruit recognition based on convolution neural network," in Proc. of the 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 18-22, 2016.
28 S. A. Akbar, S. Chattopadhyay, N. M. Elfiky, and A. Kak, "A novel benchmark RGBD dataset for dormant apple trees and its application to automatic pruning," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 347-354, 2016.
29 J. Liu, J. Tan, J. Qin, and X. Xiang, "Smoke Image Recognition Method Based on the optimization of SVM parameters with Improved Fruit Fly Algorithm," KSII Transactions on Internet and Information Systems, vol. 14, no. 8, pp. 3534-3549, 2020.   DOI
30 J. Zhang, L. He, M. Karkee, Q. Zhang, X. Zhang, and Z. Gao, "Branch detection for apple trees trained in fruiting wall architecture using depth features and Regions-Convolutional Neural Network (R-CNN)," Computers and Electronics in Agriculture, vol. 155, pp. 386-393, 2018.   DOI
31 L. Zhang, J. Jia, Y. Li, W. Gao, and M. Wang, "Deep Learning based Rapid Diagnosis System for Identifying Tomato Nutrition Disorders," KSII Transactions on Internet and Information Systems, vol. 13, no. 4, pp. 2012-2027, 2019.   DOI
32 G. Zeng, "Fruit and vegetables classification system using image saliency and convolutional neural network," in Proc. of the 3rd Information Technology and Mechatronics Engineering Conference (ITOEC), pp. 613-617, 2017.
33 G. Gui, M. Liu, F. Tang, N. Kato, and F. Adachi, "6G: Opening new horizons for integration of comfort, security and intelligence," IEEE Wireless Communications, vol. 27, no. 5, pp. 126-132, Oct. 2020.   DOI
34 Y. Wen, K. Zhang, Z. Li, and Y. Qiao, "A discriminative feature learning approach for deep face recognition," in Proc. of European Conference on Computer Vision, pp. 499-515, 2016.
35 J. R. A. Farhadi and J. Redmon, "YOLOv3: An incremental improvement," arXiv:1804.02767, 2018.
36 J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proc. of 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
37 C. Rother, V. Kolmogorov, and A. Blake, "'GrabCut' interactive foreground extraction using iterated graph cuts," ACM Transations Graphics, vol. 23, no. 3, pp. 309-314, 2004.
38 A. Shrivastava, A. Gupta, and R. Girshick, "Training region-based object detectors with online hard example mining," in Proc.of IEEE Conference on Computer Vision and Pattern Recognition, pp. 761-769, 2016.
39 K. Bresilla, G. D. Perulli, A. Boini, B. Morandi, L. C. Grappadelli, and L. Manfrini, "Single-shot convolution neural networks for real-time fruit detection within the tree," Frontier Plant Science, vol. 10, 2019.
40 S. Siddesha, S. K. Niranjan, and V. N. M. Aradhya, "Segmentation of coconut crop bunch from tree images," in Proc. of 2016 Second International Conference on Cognitive Computing and Information Processing (CCIP), pp. 1-6, 2016.
41 J. Lu, J. Hu, G. Zhao, F. Mei, and C. Zhang, "An in-field automatic wheat disease diagnosis system," Computers and Electronics in Agriculture, vol. 142, pp. 369-379, 2017.   DOI
42 Z. M. Khaing, Y. Naung, and P. H. Htut, "Development of control system for fruit classification based on convolutional neural network," in Proc. of IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pp. 1805-1807, 2018.
43 Y. Tian, G. Yang, Z. Wang, H. Wang, E. Li, and Z. Liang, "Apple detection during different growth stages in orchards using the improved YOLO-V3 model," Computers Electronics in Agriculture, vol. 157, pp. 417-426, 2019.   DOI
44 M. Nagle, K. Intani, G. Romano, B. Mahayothee, V. Sardsud, and J. Muller, "Determination of surface color of 'all yellow'mango cultivars using computer vision," International Journal of Agricultural and Biological Engineering, vol. 9, no. 1, pp. 42-50, 2016.
45 S. Tu, Y. Xue, C. Zheng, Y. Qi, H. Wan, and L. Mao, "Detection of passion fruits and maturity classification using Red-Green-Blue Depth images," Biosystems Engineering, vol. 175, pp. 156-167, 2018.   DOI
46 L. M. Azizah, S. F. Umayah, S. Riyadi, C. Damarjati, and N. A. Utama, "Deep learning implementation using convolutional neural network in mangosteen surface defect detection," in Proc. of the 7th IEEE International Conference on Control System, Computing and Engineering (ICCSCE), pp. 242-246, 2017.
47 A. Durand-Petiteville, S. Vougioukas, and D. C. Slaughter, "Real-time segmentation of strawberry flesh and calyx from images of singulated strawberries during postharvest processing," Computers and Electronics in Agriculturre, vol. 142, pp. 298-313, 2017.
48 J. Lu, W. S. Lee, H. Gan, and X. Hu, "Immature citrus fruit detection based on local binary pattern feature and hierarchical contour analysis," Biosystems Engineering, vol. 171, pp. 78-90, 2018.   DOI
49 W. Ahmad, S. M. A. Shah, and A. Irtaza, "Plants Disease Phenotyping using Quinary Patterns as Texture Descriptor," KSII Transactions on Internet and Information Systems, vol. 14, no. 8, pp. 3312-3327, 2020.   DOI
50 J. Ma, K. Du, L. Zhang, F. Zheng, J. Chu, and Z. Sun, "A segmentation method for greenhouse vegetable foliar disease spots images using color information and region growing," Computers and Electronics in Agriculture, vol. 142, pp. 110-117, 2017.   DOI
51 A. Aquino, B. Millan, M. P. Diago, and J. Tardaguila, "Automated early yield prediction in vineyards from on-the-go image acquisition," Computers and Electronics in Agriculture, vol. 144, pp. 26-36, 2018.   DOI
52 N. Behroozi-Khazaei and M. R. Maleki, "A robust algorithm based on color features for grape cluster segmentation," Computers and Electronics in Agriculturre, vol. 142, pp. 41-49, 2017.
53 A. Mohapatra, S. Shanmugasundaram, and R. Malmathanraj, "Grading of ripening stages of red banana using dielectric properties changes and image processing approach," Computers and Electronics in Agriculture, vol. 143, pp. 100-110, 2017.   DOI
54 A. Gongal, S. Amatya, M. Karkee, Q. Zhang, and K. Lewis, "Sensors and systems for fruit detection and localization: A review," Computers and Electronics in Agriculture, vol. 116, pp. 8-19, 2015.   DOI
55 Y. Shao, "Supervised global-locality preserving projection for plant leaf recognition," Computers and Electronics in Agriculture, vol. 158, pp. 102-108, 2019.   DOI
56 A. Yadav, N. Sengar, A. Issac, and M. K. Dutta, "Image processing based acrylamide detection from fried potato chip images using continuous wavelet transform," Computers and Electronics in Agriculture, vol. 145, pp. 349-362, 2018.   DOI
57 Y. Shi, W. Huang, J. Luo, L. Huang, and X. Zhou, "Detection and discrimination of pests and diseases in winter wheat based on spectral indices and kernel discriminant analysis," Computers and Electronics in Agriculture, vol. 141, pp. 171-180, 2017.   DOI
58 R. Hassankhani and H. Navid, "Potato sorting based on size and color in machine vision system," Journal of Agricultural Science, vol. 4, no. 5, pp. 235-244, 2012.