Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Methodology for Classifying Hierarchical Data Using Autoencoder-based Deeply Supervised Network

오토인코더 기반 심층 지도 네트워크를 활용한 계층형 데이터 분류 방법론

  • Kim, Younha (Graduate School of Business IT, Kookmin University) ;
  • Kim, Namgyu (Graduate School of Business IT, Kookmin University)
  • 김윤하 (국민대학교 비즈니스IT전문대학원) ;
  • 김남규 (국민대학교 비즈니스IT전문대학원)
  • Received : 2022.09.08
  • Accepted : 2022.09.24
  • Published : 2022.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, with the development of deep learning technology, researches to apply a deep learning algorithm to analyze unstructured data such as text and images are being actively conducted. Text classification has been studied for a long time in academia and industry, and various attempts are being performed to utilize data characteristics to improve classification performance. In particular, a hierarchical relationship of labels has been utilized for hierarchical classification. However, the top-down approach mainly used for hierarchical classification has a limitation that misclassification at a higher level blocks the opportunity for correct classification at a lower level. Therefore, in this study, we propose a methodology for classifying hierarchical data using the autoencoder-based deeply supervised network that high-level classification does not block the low-level classification while considering the hierarchical relationship of labels. The proposed methodology adds a main classifier that predicts a low-level label to the autoencoder's latent variable and an auxiliary classifier that predicts a high-level label to the hidden layer of the autoencoder. As a result of experiments on 22,512 academic papers to evaluate the performance of the proposed methodology, it was confirmed that the proposed model showed superior classification accuracy and F1-score compared to the traditional supervised autoencoder and DNN model.

최근 딥 러닝 기술의 발전으로 인해, 텍스트, 이미지 등 비정형 데이터 분석에 딥 러닝 알고리즘을 적용하는 연구가 활발히 수행되고 있다. 그중 텍스트 분류는 학계 및 업계에서 오랜 기간 연구되어 온 분야로, 분류의 성능을 향상시키기 위해 계층형 레이블 등 데이터 자체의 특성을 활용하기 위한 다양한 시도가 이루어지고 있다. 하지만 계층적 분류를 위해 주로 사용되는 하향식 접근법은 상위 레벨의 오분류가 하위 레벨의 정분류 기회를 차단한다는 한계가 있다. 따라서, 본 연구에서는 레이블의 계층적인 관계를 고려하면서도 상위 레벨의 분류가 하위 레벨의 분류를 차단하지 않도록 하여 분류 성능을 향상시키기 위해, 오토인코더 기반 심층 지도 네트워크를 활용한 계층형 데이터 분류 방법론을 제안한다. 제안 방법론은 오토인코더의 잠재변수에 하위 레이블을 예측하는 주 분류기를 추가하고, 인코더의 은닉층에 상위 레벨의 레이블 예측하는 보조 분류기를 추가하여 End-to-End 학습을 진행한다. 제안 방법론의 성능을 평가하기 위하여 국내 논문 데이터 총 22,512건에 대한 실험을 수행한 결과, 제안 모델이 기존의 지도 오토인코더 및 DNN 모델에 비해 분류 정확도와 F1-Score에서 우수한 성능을 나타냄을 확인하였다.

Keywords

References

  1. 김동규, 이동욱, 박장원, 오성우, 권성준, 이인용, & 최동원. (2022). KB-BERT: 금융 특화 한국어 사전학습 언어모델과 그 응용. 지능정보연구, 28(2), 191-206 https://doi.org/10.13088/JIIS.2022.28.2.191
  2. 김무성, & 김남규. (2021). 다중 레이블 분류의 정확도 향상을 위한 스킵 연결 오토인코더 기반 레이블 임베딩 방법론. 지능정보연구, 27(3), 175-197. https://doi.org/10.13088/JIIS.2021.27.3.175
  3. 신병진, 이종훈, 한상진, & 박충식. (2021). ICT 인프라 이상탐지를 위한 조건부 멀티모달 오토인코더에 관한 연구. 지능정보연구, 27(3), 57-73. https://doi.org/10.13088/JIIS.2021.27.3.057
  4. 유태우, 김윤욱, 정하민, 유현수, & 안용학. (2018). 효율적인 사물 이미지 분류를 위한 계층적 이미지 분류 체계의 설계 및 구현. 융합보안논문지, 18(3), 53-59.
  5. 이상아, & 신효필. (2020). 감정 분석을 위한 BERT 사전학습모델과 추가 자질 모델의 결합. 한국정보과학회 학술발표논문집, 275-277.
  6. 임소라, & 권용진. (2017). 특허문서 필드의 기능 적 특성을 활용한 IPC 다중 레이블 분류. 인터넷정보학회논문지, 18(1), 77-88. https://doi.org/10.7472/JKSII.2017.18.1.77
  7. 한국과학기술정보연구원. (2021, 09.08). 국내 논문 전문 텍스트 데이터셋. 한국과학기술정보연구원. https://doi.org/10.23057/38.
  8. 한국과학기술정보연구원. (2022, 04.04). 논문 연구분야 분류 데이터. 한국과학기술정보연구원. https://doi.org/10.23057/50.
  9. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2006). Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19.
  10. Cai, L., & Hofmann, T. (2004). Hierarchical document categorization with support vector machines. In Proceedings of the thirteenth ACM international conference on Information and knowledge management, 78-87.
  11. Chen, Y., Wang, Y., Gu, Y., He, X., Ghamisi, P., & Jia, X. (2019). Deep learning ensemble for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(6), 1882-1897. https://doi.org/10.1109/JSTARS.2019.2915259
  12. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171-4186
  13. Dou, Q., Chen, H., Jin, Y., Yu, L., Qin, J., & Heng, P. A. (2016). 3D deeply supervised network for automatic liver segmentation from CT volumes. International conference on medical image computing and computerassisted intervention, 149-157.
  14. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 580-587.
  15. Kaikhah, K. (2004). Automatic text summarization with neural networks. 2004 2nd International IEEE Conference on'Intelligent Systems'. Proceedings (IEEE Cat. No. 04EX791), 1, 40-44.
  16. Le, L., Patterson, A., & White, M. (2018). Supervised autoencoders: improving generalization performance with unsupervised regularizers. Advances in neural information processing systems, 31.
  17. Lee, C. Y., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015). Deeply-supervised nets. Artificial intelligence and statistics, 562-570.
  18. Li, R., Wang, X., Huang, G., Yang, W., Zhang, K., Gu, X., Tran, S. N., Garg, S., Alty, J., & Bai, Q. (2022). A comprehensive review on deep supervision: theories and applications. arXiv preprint arXiv:2207.02376.
  19. Mishra, D., Chaudhury, S., Sarkar, M., & Soin, A. S. (2018). Ultrasound image segmentation: a deeply supervised network with attention to boundaries. IEEE Transactions on Biomedical Engineering, 66(6), 1637-1648.
  20. Parida, S., Villatoro-Tello, E., Kumar, S., Motlicek, P., & Zhan, Q. (2020). Idiap submission to swiss-german language detection shared task. SwissText/KONVENS.
  21. Pereira, G. T., Santos, B. Z., & Cerri, R. (2018). A genetic algorithm for transposable elements hierarchical classification rule induction. 2018 IEEE Congress on Evolutionary Computation (CEC), 1-8.
  22. Phyu, T. N. (2009). Survey of classification techniques in data mining. Proceedings of the international multiconference of engineers and computer scientists, 1(5).
  23. Romero, M., Finke, J., & Rocha, C. (2022). A top-down supervised learning approach to hierarchical multi-label classification in networks. Applied Network Science, 7(1), 1-17. https://doi.org/10.1007/s41109-021-00435-x
  24. Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986). Parallel distributed processing: explorations in the microstructure of cognition. Cambridge, MA, USA: MIT Press.
  25. Secker, A. D., Davies, M. N., Freitas, A. A., Timmis, J., Mendao, M., & Flower, D. R. (2007). An experimental comparison of classification algorithms for hierarchical prediction of protein function. Expert Update (Magazine of the British Computer Society's Specialist Group on AI), 9(3), 17-22.
  26. Shen, Z., Liu, Z., Li, J., Jiang, Y. G., Chen, Y., & Xue, X. (2017). Dsod: learning deeply supervised object detectors from scratch. Proceedings of the IEEE international conference on computer vision, 1919-1927.
  27. Silla, C. N., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1), 31-72. https://doi.org/10.1007/s10618-010-0175-9
  28. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
  29. Ullah, M. A., Marium, S. M., Begum, S. A., & Dipa, N. S. (2020). An algorithm and method for sentiment analysis using the text and emoticon. ICT Express, 6(4), 357-360. https://doi.org/10.1016/j.icte.2020.07.003
  30. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, 1096-1103.
  31. Vlasenko, B., Prasad, R., & Magimai.-Doss, M. (2021). Fusion of acoustic and linguistic information using supervised autoencoder for improved emotion recognition. Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge, 51-59.
  32. Wang, J., & Perez, L. (2017). The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit, 11, 1-8.
  33. Zhang, G. P. (2000). Neural networks for classification: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(4), 451-462. https://doi.org/10.1109/5326.897072
  34. Zhu, X., & Bain, M. (2017). B-CNN: branch convolutional neural network for hierarchical classification. arXiv preprint arXiv:1709.09890.