Browse > Article
http://dx.doi.org/10.7472/jksii.2022.23.5.145

Automatic Target Recognition Study using Knowledge Graph and Deep Learning Models for Text and Image data  

Kim, Jongmo (Dept. of Industrial Engineering, Sungkyunkwan University)
Lee, Jeongbin (Dept. of Industrial Engineering, Sungkyunkwan University)
Jeon, Hocheol (Agency for Defense Development)
Sohn, Mye (Dept. of Industrial Engineering, Sungkyunkwan University)
Publication Information
Journal of Internet Computing and Services / v.23, no.5, 2022 , pp. 145-154 More about this Journal
Abstract
Automatic Target Recognition (ATR) technology is emerging as a core technology of Future Combat Systems (FCS). Conventional ATR is performed based on IMINT (image information) collected from the SAR sensor, and various image-based deep learning models are used. However, with the development of IT and sensing technology, even though data/information related to ATR is expanding to HUMINT (human information) and SIGINT (signal information), ATR still contains image oriented IMINT data only is being used. In complex and diversified battlefield situations, it is difficult to guarantee high-level ATR accuracy and generalization performance with image data alone. Therefore, we propose a knowledge graph-based ATR method that can utilize image and text data simultaneously in this paper. The main idea of the knowledge graph and deep model-based ATR method is to convert the ATR image and text into graphs according to the characteristics of each data, align it to the knowledge graph, and connect the heterogeneous ATR data through the knowledge graph. In order to convert the ATR image into a graph, an object-tag graph consisting of object tags as nodes is generated from the image by using the pre-trained image object recognition model and the vocabulary of the knowledge graph. On the other hand, the ATR text uses the pre-trained language model, TF-IDF, co-occurrence word graph, and the vocabulary of knowledge graph to generate a word graph composed of nodes with key vocabulary for the ATR. The generated two types of graphs are connected to the knowledge graph using the entity alignment model for improvement of the ATR performance from images and texts. To prove the superiority of the proposed method, 227 documents from web documents and 61,714 RDF triples from dbpedia were collected, and comparison experiments were performed on precision, recall, and f1-score in a perspective of the entity alignment..
Keywords
Automatic Target Recognition; Text-image Graph Conversion; Graph Entity Alignment; Knowledge Graph-based Target Recognition;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Wang, H., et al., "Consensus-aware visual-semantic embedding for image-text matching," in European Conference on Computer Vision, Springer, 2020. https://doi.org/10.1007/978-3-030-58586-0_2   DOI
2 Matsumurr, J., et al., "Exploring advanced technologies for the future combat systems program," RAND ARROYO CENTER SANTA MONICA CA, 2002. https://doi.org/10.7249/mr1332   DOI
3 Huang, Z., Z. Pan, and B. Lei, "What, where, and how to transfer in SAR target recognition based on deep CNNs,"IEEE Transactions on Geoscience and Remote Sensing, 58(4), p. 2324-2336, 2019. https://doi.org/10.1109/tgrs.2019.2947634   DOI
4 Mithun, N.C., et al., "Webly supervised joint embedding for cross-modal image-text retrieval," in Proceedings of the 26th ACM international conference on Multimedia, 2018. https://doi.org/10.1145/3240508.3240712   DOI
5 Shi, B., et al., "Knowledge Aware Semantic Concept Expansion for Image-Text Matching," in IJCAI, 2019. https://doi.org/10.24963/ijcai.2019/720   DOI
6 Kim, S., W.-J. Song, and S.-H. Kim, "Double weight-based SAR and infrared sensor fusion for automatic ground target recognition with deep learning," Remote Sensing, 10(1), p. 72, 2018. https://doi.org/10.3390/rs10010072   DOI
7 Zheng, Z., et al., "Dual-path convolutional image-text embeddings with instance loss," ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(2), p. 1-23, 2020. https://doi.org/10.1145/3383184   DOI
8 Sakla, W., G. Konjevod, and T.N. Mundhenk, "Deep multi-modal vehicle detection in aerial ISR imagery," in 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2017. https://doi.org/10.1109/wacv.2017.107   DOI
9 Zhang, D., et al., "Multi-modal graph fusion for named entity recognition with targeted visual guidance," in Proceedings of the AAAI Conference on Artificial Intelligence, 2021. https://doi.org/10.1609/aaai.v35i16.17687   DOI
10 Lang, C., A. Braun, and A. Valada, "Contrastive object detection using knowledge graph embeddings," Computer Vision and Pattern Recognition, 2021. https://doi.org/10.48550/arXiv.2112.11366   DOI
11 Yan, H., et al., "TENER: adapting transformer encoder for named entity recognition," Computation and Language, 2019. https://doi.org/10.48550/arXiv.1911.04474   DOI
12 Xu, C., et al., "An Optimal Faster-RCNN Algorithm for Intelligent Battlefield Target Recognition," in 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), IEEE, 2020. https://doi.org/10.1109/icaica50127.2020.9181857   DOI
13 Jo, S.-H., et al., "A study on building knowledge base for intelligent battlefield awareness service," Journal of the Korea Society of Computer and Information, 25(4), p. 11-17, 2020. https://doi.org/10.9708/jksci.2020.25.04.011   DOI
14 Birant, D. and A. Kut, "ST-DBSCAN: An algorithm for clustering spatial-temporal data," Data & knowledge engineering, 60(1), p. 208-221, 2007. https://doi.org/10.1016/j.datak.2006.01.013   DOI