KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Efficient Recognition of Easily-confused Chinese Herbal Slices Images Using Enhanced ResNeSt

  • Qi Zhang (School of Medical Information and Engineering, Guangdong Pharmaceutical University) ;
  • Jinfeng Ou (School of Medical Information and Engineering, Guangdong Pharmaceutical University) ;
  • Huaying Zhou (School of Medical Information and Engineering, Guangdong Pharmaceutical University)
  • Received : 2024.02.14
  • Accepted : 2024.07.11
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Chinese herbal slices (CHS) automated recognition based on computer vision plays a critical role in the practical application of intelligent Chinese medicine. Due to the complexity and similarity of herbal images, identifying Chinese herbal slices is still a challenging task. Especially, easily-confused CHS have higher inter-class and intra-class complexity and similarity issues, the existing deep learning models are less adaptable to identify them efficiently. To comprehensively address these problems, a novel tiny easily-confused CHS dataset has been built firstly, which includes six pairs of twelve categories with about 2395 samples. Furthermore, we propose a ResNeSt-CHS model that combines multilevel perception fusion (MPF) and perceptive sparse fusion (PSF) blocks for efficiently recognizing easilyconfused CHS images. To verify the superiority of the ResNeSt-CHS and the effectiveness of our dataset, experiments have been employed, validating that the ResNeSt-CHS is optimal for easily-confused CHS recognition, with 2.1% improvement of the original ResNeSt model. Additionally, the results indicate that ResNeSt-CHS is applied on a relatively small-scale dataset yet high accuracy. This model has obtained state-of-the-art easily-confused CHS classification performance, with accuracy of 90.8%, far beyond other models (EfficientNet, Transformer, and ResNeSt, etc) in terms of evaluation criteria.

Keywords

Acknowledgement

This work is supported by Guangdong Administration of Traditional Chinese Medicine, China (No.20221221 and No.20231221); the College Student Innovation and Entrepreneurship Training Program of Guangdong Province (No. 202310573014) and Special Fund for Science and Technology Innovation Strategy of Guangdong Province ("Climbing Program")(No. pdjh2023b0273).

References

  1. Han, M., Zhang, J., Zeng, Y., Hao, F., & Ren, Y., "A Novel Method of Chinese Herbal Medicine Classification Based on Mutual Learning," Mathematics, vol.10, no.9, 2022.
  2. Hu, H., & Chung, C. C., "The innovation and modernisation of 'herbal pieces' in China: System evolution and policy transitions (1950s-2010s)," European Journal of Integrative Medicine, vol.7, no.6, pp.645-649, 2015.
  3. Xie, Z. W., ""Differentiation of symptoms and discussion of quality" in traditional experience identification of traditional Chinese medicine varieties," Lishizhen Medicine and Materia Medica Research, vol.5, no.3, pp.19-21, 1994.
  4. Zhang, Y., Wan, H., and Tu, S. Q., "Technical review and case study on classification of Chinese herbal slices based on computer vision," Journal of Computer Applications, vol.42, no.10, pp.3224-3234, 2022.
  5. Tang, Y., Wang, Y., Li, J., Zhang, W., Wang, L., Zhai, X., & Han, A., "Classification of Chinese Herbal Medicines by deep neural network based on orthogonal design," in Proc. of 2021 IEEE 4th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp.574-583, 2021.
  6. Cai, C., Liu, S., Wang, L., Yang, B., Zhi, M., Wang, R., & He, W., "Classification of Chinese Herbal Medicine Using Combination of Broad Learning System and Convolutional Neural Network," in Proc. of 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp.3907-3912, 2019.
  7. Zhu, X., Zhu, M., & Ren, H., "Method of plant leaf recognition based on improved deep convolutional neural network," Cognitive Systems Research, vol.52, pp.223-233, 2018.
  8. Kadir, A., Nugroho, L. E., Susanto, A., & Santosa, P. I., "Leaf Classification Using Shape, Color, and Texture Features," International Journal of Computer Trends and Technology (IJCTT), vol.1, no.3, pp.225-230, 2011.
  9. Dehan, L., Jia, W., Yimin, C., & Hamid, G., "Classification of Chinese Herbal medicines based on SVM," in Proc. of 2014 International Conference on Information Science, Electronics and Electrical Engineering, vol.1, pp.453-456, 2014.
  10. Mahajan, S., Raina, A., Gao, X. Z., & Kant Pandit, A., "Plant Recognition Using Morphological Feature Extraction and Transfer Learning over SVM and AdaBoost," Symmetry, vol.13, no.2, 2021.
  11. Xing, C., Huo, Y., Huang, X., Lu, C., Liang, Y., & Wang, A., "Research on Image Recognition Technology of Traditional Chinese Medicine Based on Deep Transfer Learning," in Proc. of 2020 International Conference on Artificial Intelligence and Electromechanical Automation (AIEA), pp.140-146, 2020.
  12. Simonyan, K., & Zisserman, A., "Very Deep Convolutional Networks for Large-Scale Image Recognition," in Proc. of International Conference on Learning Representations, 2015.
  13. Krizhevsky, A., Sutskever, I., & Hinton, G. E., "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol.60, no.6, pp.84-90, 2017.
  14. He, K., Zhang, X., Ren, S., & Sun, J., "Deep Residual Learning for Image Recognition," in Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770-778, 2016.
  15. Chollet, F., "Xception: Deep Learning with Depthwise Separable Convolutions," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1251-1258, 2017.
  16. Lee, S., Choi, G., Park, H.-C., Choi, C., "Automatic Classification Service System for Citrus Pest Recognition Based on Deep Learning," Sensors, vol.22, no.22, 2022.
  17. Dilshad, N., Khan, T., Song, J., "Efficient Deep Learning Framework for Fire Detection in Complex Surveillance Environment," Computer Systems Science and Engineering, vol.46, no.1, pp.749-764, 2023.
  18. Wang, W., Tian, W., Liao, W., Cai, B., & Li, B., "Identifying Chinese Herbal Medicine by Image with Three Deep CNNs," in Proc. of CCEAI '21: Proceedings of the 5th International Conference on Control Engineering and Artificial Intelligence, pp.1-8, 2021.
  19. Sun, X., & Qian, H., "Chinese Herbal Medicine Image Recognition and Retrieval by Convolutional Neural Network," PLoS ONE, vol.11, no.6, 2016.
  20. Huang, F., Yu, L., Shen, T., & Jin, L., "Chinese Herbal Medicine Leaves Classification Based on Improved AlexNet Convolutional Neural Network," in Proc. of 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), vol.1, pp.1006-1011, 2019.
  21. Zhao, P., "Explore the identification of Chinese herbal medicine based on the VGG-16 model," in Proc. of the 3rd International Conference on Signal Processing and Machine Learning. vol.4, pp. 645-650, 2023.
  22. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q., "Densely Connected Convolutional Networks," in Proc. of the IEEE conference on computer vision and pattern recognition, pp.2261-2269, 2017.
  23. Liu, S., Chen, W., & Dong, X., "Automatic Classification of Chinese Herbal Based on Deep Learning Method," in Proc. of 2018 14th International conference on natural computation, fuzzy systems and knowledge discovery (ICNC-FSKD), pp.235-238, 2018.
  24. Liu, S., Chen, W., Li, Z., & Dong, X., "Chinese Herbal Classification Based on Image Segmentation and Deep Learning Methods," in Proc. of Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery: Proceedings of the ICNC-FSKD 2021, pp.267-275, 2022.
  25. Hao, W., Han, M., Yang, H., Hao, F., & Li, F., "A novel Chinese herbal medicine classification approach based on EfficientNet," Systems Science & Control Engineering, vol.9, no.1, pp.304-313, 2021.
  26. Wu, C.,TAN, C. Q.,Huang, Y. L. Wu, C. J., Chen, H., "Intelligent Identification of Fritillariae Cirrhosae Bulbus,Crataegi Fructus and Pinelliae Rhizoma Based on Deep Learning Algorithms," Chinese Journal of Experimental Traditional Medical Formulae, vol.26, no.21, pp.195-201, 2020.
  27. Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C., "CMT: Convolutional Neural Networks Meet Vision Transformers," in Proc. of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.12165-12175, 2022.
  28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I., "Attention is all you need," in Proc. of NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp.6000-6010, 2017.
  29. Niu, Z., Zhong, G., & Yu, H., "A review on the attention mechanism of deep learning," Neurocomputing, vol.452, pp.48-62, 2021.
  30. Wang, Y., Feng, Y., Zhang, L., Zhou, J. T., Liu, Y., Goh, R. S. M., & Zhen, L., "Adversarial multimodal fusion with attention mechanism for skin lesion classification using clinical and dermoscopic images," Medical Image Analysis, vol.81, 2022.
  31. Shanshan, W., Tao, Z., Fei, L., ZhenPing, R., Zhen, Y., Shu, Z., & ZhiQiang, Z., "A Synergic Neural Network For Medical Image Classification Based On Attention Mechanism," in Proc. of 2022 Asia Conference on Algorithms, Computing and Machine Learning (CACML), pp.82-87, 2022.
  32. Zhu, H., Wang, J., Wang, S. H., Raman, R., Gorriz, J. M., & Zhang, Y. D., "An Evolutionary Attention-Based Network for Medical Image Classification," International Journal of Neural Systems, vol.33, no.3, 2023.
  33. Xu, Y., Wen, G., Hu, Y., Luo, M., Dai, D., Zhuang, Y., & Hall, W., "Multiple attentional pyramid networks for Chinese herbal recognition," Pattern Recognition, vol.110, 2021.
  34. Miao, J., Huang, Y., Wang, Z., Wu, Z., & Lv, J., "Image recognition of traditional Chinese medicine based on deep learning," Frontiers in Bioengineering and Biotechnology, vol.11, 2023.
  35. Liu, Z., Mao, H., Wu, C. Y., Feichtenhofer, C., Darrell, T., & Xie, S., "A ConvNet for the 2020s," in Proc. of the IEEE/CVF conference on computer vision and pattern recognition, pp.11966-11976, 2022.
  36. Pan, X., Ge, C., Lu, R., Song, S., Chen, G., Huang, Z., & Huang, G., "On the Integration of SelfAttention and Convolution," in Proc. of the 2022 IEEE/CVF conference on computer vision and pattern recognition, pp.805-815, 2022.
  37. Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., Li, M., Smola, A., "ResNeSt: Split-Attention Networks," in Proc. of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.2735-2745, 2022.
  38. Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K., "Aggregated Residual Transformations for Deep Neural Networks," in Proc. of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp.5987-5995, 2017.
  39. Nan, F., Zeng, Q., Xing, Y., & Qian, Y., "Single Image Super-Resolution Reconstruction based on the ResNeXt Network," Multimedia Tools and Applications, vol.79, pp.34459-34470, 2020.
  40. Go, J. H., Jan, T., Mohanty, M., Patel, O. P., Puthal, D., & Prasad, M., "Visualization Approach for Malware Classification with ResNeXt," in Proc. of 2020 IEEE Congress on Evolutionary Computation (CEC), pp.1-7, 2020.
  41. Wang, J., Mo, W., Wu, Y., Xu, X., Li, Y., Ye, J., & Lai, X., "Combined Channel Attention and Spatial Attention Module Network for Chinese Herbal Slices Automated Recognition," Frontiers in Neuroscience, vol.16, 2022.
  42. Tan, D. Q. et al., "Research on identification of confusing TCM decoction pieces by integrating of improved residual network," China Digital Medicine, vol.18, no.6, pp.42-50, 2023.
  43. Yang, Z., Wang, X., Hong, W., Zhang, S., Yang, Y., Xia, Y., & Yang, R., "The pharmacological mechanism of Chinese herbs effective in treating advanced ovarian cancer: Integrated metaanalysis and network pharmacology analysis," Frontiers in Pharmacology, vol.13, 2022.
  44. Sarwinda, D., Paradisa, R. H., Bustamam, A., & Anggia, P., "Deep Learning in Image Classification using Residual Network (ResNet) Variants for Detection of Colorectal Cancer," Procedia Computer Science, vol.179, pp.423-431, 2021.
  45. Showkat, S., & Qureshi, S., "Efficacy of Transfer Learning-based ResNet models in Chest X-ray image classification for detecting COVID-19 Pneumonia," Chemometrics and Intelligent Laboratory Systems, vol.224, 2022.
  46. Lou, M., Zhou, H. Y., Yang, S., Yu, Y., "TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition," arXiv:2310.19380, 2023.
  47. Sun, X., Ponce, J., Wang, Y. X., "Revisiting Deformable Convolution for Depth Completion," in Proc. of 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.1300-1306, 2023.
  48. Goyal, P., Dollar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour," arXiv:1706.02677, 2017.