Journal of Korean Society of Industrial and Systems Engineering (산업경영시스템학회지)

Society of Korea Industrial and System Engineering (한국산업경영시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Test and Evaluation Procedures of Defense AI System linked to the ROK Defense Acquisition System

국방획득체계와 연계한 국방 인공지능(AI) 체계 시험평가 방안

  • Received : 2023.11.09
  • Accepted : 2023.11.27
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

In this research, a new Test and Evaluation (T&E) procedure for defense AI systems is proposed to fill the existing gap in established methodologies. This proposed concept incorporates a data-based performance evaluation, allowing for independent assessment of AI model efficacy. It then follows with an on-site T&E using the actual AI system. The performance evaluation approach adopts the project promotion framework from the defense acquisition system, outlining 10 steps for R&D projects and 9 steps for procurement projects. This procedure was crafted after examining AI system testing standards and guidelines from both domestic and international civilian sectors. The validity of each step in the procedure was confirmed using real-world data. This study's findings aim to offer insightful guidance in defense T&E, particularly in developing robust T&E procedures for defense AI systems.

Keywords

References

  1. Calabrese, J., Esponda, S., and Pesado, P.M., Framework for Data Quality Evaluation Based on ISO/IEC 25012 and ISO/IEC 25024, VIII Conference on Cloud Computing, Big Data & Emerging Topics (Modalidad virtual, 8 al 10 de septiembre de 2020), 2020.
  2. Chang, W., ISO/IEC JTC 1/SC 42 (AI)/WG 2 (data) data quality for analytics and machine learning (ML), 2022, Information Technology Laboratory.
  3. Cho, K.T., Lee, S.Y., Lee, H.M., Kim, S.H., and Jeong, H.M., Enhancing the Efficiency and Reliability for M&S based Test and Evaluation System Development, Journal of the Korea society for Simulation, 2012 21.1 pp. 89-96. https://doi.org/10.9709/JKSS.2012.21.1.089
  4. Cho, Q., Han, M., Ryu, H., Lee, J., and Kim, S., A Study on Development of Test and Evaluation Methods for AI-based Image Surveillance Systems in Defense, Journal of Applied Reliability, 2023 Vol. 23, No. 1, pp. 97-104. https://doi.org/10.33162/JAR.2023.3.23.1.097
  5. Choi, M., Choi, E., Song, Y., Kim, J., Park, S.T., Kwon, O., and Cho, N., Simulation Based Reinforcement Learning for the Intelligence Behavior of Autonomous Weapon System, Journal of the Korea Society For Simulation, 2023, Vol. 32, No. 2, pp. 91-111.
  6. Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., and Batra, D., Sim2real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?, IEEE Robotics and Automation Letters, 2020, Vol.5, No. 4. pp. 6670-6677. https://doi.org/10.1109/LRA.2020.3013848
  7. Geron, A., Hands-On Machine Learning with ScikitLearn, Keras, and TensorFlow, H. S. Park Ed., Republic of Korea: Hanbit Publishing Network, 2020, pp. 61.
  8. https://www.etnews.com/20231012000315.
  9. https://www.iso.org/standard/35749.html.
  10. ISO/IEC JTC 1/SC 42/WG2, <>, 2022.
  11. ISO/IEC JTC 1/SC 42/WG2, <>, 2022.
  12. ISO/IEC JTC 1/SC 42/WG2, <>, 2022.
  13. ISO/IEC JTC 1/SC 42/WG2, <>, 2022.
  14. ISO/IEC JTC 1/SC 42/WG2, <>, 2022.
  15. ISO/IEC JTC 1/SC 7, <>, 2015.
  16. Kim, B.-S. and Yun, K., A study on the test evaluation of automated-vehicles based on artificial intelligence, Spring Conf, of KSAE, 2019, pp. 722-722.
  17. Kim, B.S. and Yun, K., A Study on the Test Evaluation of Automated-Vehicles Based on Artificial Intelligence, The Korean Society of Automotive Engineers, Annual Spring Conference, 2019, pp. 722-722.
  18. Kim, M.Y. and Noh, S.C., Acquisition Process of Military Weapon System Using Artificial Intelligence, Proceedings of Symposium of the Korean Institute of Communications and Information Sciences, 2022.
  19. Kohavi, R., A study of cross-validation and bootstrap for accuracy estimation and model selection, in Proc. 14th Int. Joint Conf. on Artif. Intell., 1995, pp. 1137-1143.
  20. Kurrek, P., Jocas, M., Zoghlami, F., Stoelen, M., Salehi, V., Ai motion control-a generic approach to develop control policies for robotic manipulation tasks, Proceedings of the Design Society: International Conference on Engineering Design, 2019, Vol. 1, No. 1. Cambridge University Press.
  21. Lee, Y., Jeon, I., and Kim, S., A Study on the Development of Artificial Intelligence Weapon System Test and Evaluation Method: Focusing on the Performance Evaluation of the Classification Model, Reliability Applied Research, 2022, Vol. 22, No. 1, pp. 1-9. https://doi.org/10.33162/JAR.2022.3.22.1.001
  22. Ministry of Science and ICT, Korea Information and Communication Technology Association, Trustworthy Artificial Intelligence Development Guide (Plan), 2023.
  23. National Information Society Agency (Vol 3.0), Republic of Korea: Guide for Constructing Artificial Intelligence Training Data, 2023.
  24. Park, J.H., A Study on the V&V Process of M&S for the Test and Evaluation, Journal of the Korea Academia-Industrial Cooperation Society, 2019, Vol. 20, No. 9, pp. 397-404.
  25. Porter, D.J. and Dennis, J.W., Test & evaluation of ai-enabled and autonomous systems: A literature review, Institute for Defense Analyses, 2020.
  26. Priestley, M., O'Donnell, F., and Simperl, E., A survey of data quality requirements that matter in ML development pipelines, ACM Journal of Data and Information Quality, 2023, Vol. 15, No. 2, pp. 1-39. https://doi.org/10.1145/3592616
  27. Rangineni, S., An Analysis of Data Quality Requirements for Machine Learning Development Pipelines Frameworks, International Journal of Computer Trends and Technology, 2023, Vol. 71, No. 9, pp. 16-27. https://doi.org/10.14445/22312803/IJCTT-V71I8P103
  28. Schnelle, S. and Favaro, F. M., ADS Standardization Landscape: Making Sense of its Status and of the Associated Research Questions, arXiv preprint arXiv, 2023, 2306.17682.
  29. Shin, J.H., Data quality verification method for artificial intelligence learning, Journal of Electronic Engineering, 2021, Vol. 48, No. 7, pp. 28-34.
  30. Torchiano, M., Vetro, A., and Iuliano, F., Preserving the benefits of Open Government Data by measuring and improving their quality: An Empirical Study, IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), 2017, Vol. 1. IEEE.