Journal of Korean Society of Archives and Records Management (한국기록관리학회지)

Korean Society of Archives and Records Management (한국기록관리학회)
권호 표지
DOI QR코드

DOI QR Code

Test Set Construction for Quality Evaluation of NAK Portal's Search Service and the Status Analysis

국가기록포털 검색서비스 품질 점검을 위한 평가셋 구축 및 현황 분석

  • 나정호 (전북대학교 일반대학원 기록관리학과) ;
  • 소현기 (전북대학교 일반대학원 기록관리학과) ;
  • 염경록 ((주) 아이와즈) ;
  • 이정옥 (국가기록원 서비스정책과) ;
  • 오효정 (전북대학교 문헌정보학과, 문화융복합아카이빙연구소)
  • Received : 2022.10.17
  • Accepted : 2022.11.08
  • Published : 2022.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

The ultimate record management's purpose is preservation and utilization. However, the National Archives of Korea (NAK)s Portal has problems such as search system aging and search tools dualization. As a result, the users' search satisfaction is not satisfied, and the improvement demand increases. This study aimed to evaluate the NAK's search quality as a preliminary study for NAK search system advancement. To this end, we analyzed the current status of CAMS and NAK's Portal. Then, we established the test sets and evaluated the NAK's Portal quality from the user's point of view. Evaluation results were analyzed using Precision, Recall, F-score, and MRR. The analysis results showed that the overall search performance was low, particularly in the "advanced subject search," which showed low performance in Precision, Recall, and MRR. Thus, improvement is urgently needed. The test sets established for this study are expected to be used as a basis for objectively measuring the improvement of the search performance after the NAK search system advancement.

기록을 관리하는 궁극적인 목적은 안전한 보존과 효율적 활용이다. 그러나 공공기록물 관리의 수장인 국가기록원의 검색서비스인 국가기록포털은 검색 시스템이 도입된 지 10년이 넘어 노후화되었고, 검색도구도 이원화되어 있다. 또한, 국가기록물을 열람하고자 하는 이용자의 검색 만족도를 충족시키지 못해 개선이 필요하다. 본 연구는 국가기록물 검색 고도화를 위한 사전 연구로, 현재의 국가기록원 검색시스템의 검색 품질 현황을 파악하는 것을 목적으로 한다. 이를 위해 국가기록원에서 사용 중인 검색도구인 CAMS와 국가기록포털의 현황을 분석하고 검색서비스 품질을 점검하기 위한 평가셋을 구축, 이용자 관점에서 국가기록포털의 검색 품질 평가를 실시하였다. 국가기록포털의 성능평가를 위한 척도로는 정확률과 재현율, F-score, MRR 등을 활용, 다양한 각도에서 분석하였다. 검색 품질 평가결과 전반적으로 검색 성능이 낮은 것으로 나타났으며, 특히 '주제별상세검색'의 경우 정확률, 재현율, MRR이 모두 낮게 나와 개선이 시급할 것으로 보인다. 본 연구를 통해 구축된 평가셋이 향후 국가기록원 검색시스템의 고도화 이후 시스템의 성능이 얼마나 개선되었는지를 객관적으로 측정할 수 있는 기틀로 활용되기를 기대한다.

Keywords

Acknowledgement

본 논문은 2022년 국가기록관리 활용기술 연구개발(R&D) 사업'의 연구비를 지원받아 수행되었음.

References

  1. Chang, InHo (2011). Developing and evaluating an ontology-based legal retrieval system. Journal of the Korean Society for Library and Information Science, 45(2), 345-366. https://doi.org/10.4275/KSLIS.2011.45.2.345
  2. Jang, Hee-Jung (2012). A study on evaluation of national archives websites. Journal of Korean Society of Archives and Records Management, 12(2), 51-70. https://doi.org/10.14404/JKSARM.2012.12.2.051
  3. Kim, Jang-hwan & Lee, Eun Byol (2015). A Study on the Development of the National Assembly Archives and Records Integrated Management System. Journal of Korean Society of Archives and Records Management, 15(2), 103-136. https://doi.org/10.14404/JKSARM.2015.15.2.103
  4. Kim, Jihyun (2012). A study on users' perception of reference services in national archives of korea. Journal of Korean Society of Archives and Records Management, 12(1), 167-187. https://doi.org/10.14404/JKSARM.2012.12.1.167
  5. Kim, Hyun Hee (2007). An Experimental Study on the Retrieval Efficiency of the FRBR Based Bibliographic Retrieval System. Journal of Korean Library and Information Science Society, 38(3), 223-246. https://doi.org/10.16981/kliss.38.3.200709.223
  6. Lee, Byung Gil & Kim, Heesop (2013). Design and Evaluation of an Individual Instance-based Ontology Retrieval System for Archival Records of the "Saemaul Movement". Journal of Korean Society of Archives and Records Management, 13(3), 67-97. https://doi.org/10.14404/JKSARM.2013.13.3.067
  7. Lee, Hyojin & Kim, Jihyun (2021). A Study on Improvement Plans for National Archives of Korea Website's Search Service through Its Usability Evaluation. Journal of Korean Society of Archives and Records Management, 21(3), 187-215. https://doi.org/10.14404/JKSARM.2021.21.3.187
  8. Lee, Kyung Nam (2013). Evaluation of search functions of the standard records management systems. The Korean Journal of Archival Studies, 37, 273-305. https://doi.org/10.20923/kjas.2013.37.273
  9. Lee, Sang Gin & Chung, Jason (2011). An Empirical Study on the Marketing Performance of e-Tradeusing Search Engine Optimization. International Commerce and Information Review, 13(1), 3-28. https://doi.org/10.15798/kaici.13.1.201103.3
  10. Lee, Yoon-Ryeong & Rieh, Hae-Young (2014). A study on the improvement direction for online finding aids: based on the assessment of National Archives. Journal of Korean Society of Archives and Records Management, 14(1), 75-100. https://doi.org/10.14404/JKSARM.2014.14.1.075
  11. National Archives of Korea (2014). National Archives of Korea White Paper 2013. Daejeon: National Archives of Korea.
  12. National Archives of Korea (2015). National Archives of Korea White Paper 2014. Daejeon: National Archives of Korea.
  13. National Archives of Korea (2021). National Archives of Korea Statistical Yearbook 2021. Daejeon: National Archives of Korea.
  14. Noh, Meung-Hoan (2020). The Way for Data Archive and Records/Archive Management in the 4th Industrial Revolution Era : Critical Reviews and Mid- and Long-term Proposals for the National Archives' Administrative Information Dataset Records/Archive Management Implementation Plan. The Korean Journal of Archival, Information and Cultural Studies, 11, 7-44. https://doi.org/10.23035/kaics.2020.1.11.007
  15. Park, Mi-Sung (2009). Task Review of INEX Book Search Track. Journal of Korean Library and Information Science Society, 40(4), 199-225. https://doi.org/10.16981/kliss.40.4.200912.199
  16. Park, Minsoo & Hyun, Mi-Hwan (2011). Usability of the national science and technology information system. Journal of the Korean BIBLIA Society for library and Information Science, 22(4), 5-19. https://doi.org/10.14699/kbiblia.2011.22.4.005
  17. Park, Min Soo & Seo, Eun-Gyoung (2012). Analyzing User Satisfaction of the Standard Records Management System. Journal of Korean Society of Archives and Records Management, 12(1), 189-207. https://doi.org/10.14404/JKSARM.2012.12.1.189
  18. Park, Sang Hyun & Kim, Heesop (2014). Design and Implementation of an EAD-based Integrated Retrieval System for the Archives in Korea. Journal of Korean Society of Archives and Records Management, 14(1), 101-124. https://doi.org/10.14404/JKSARM.2014.14.1.101
  19. Park, Soyeon, Lee, Joon-Ho & Jeon, Jiwoon (2006). Evaluation of the documents from the web-based question and answer service. Journal of the Korean society for library and information science, 40(2), 299-314. https://doi.org/10.4275/KSLIS.2006.40.2.299
  20. Park, Soyeon & Lee, Joon-Ho (2008). Comparative evaluation of the unified search services provided by major Korean search portals. Journal of Korean library and information science society, 39(1), 265-278. https://doi.org/10.16981/kliss.39.1.200803.265
  21. Seol, Moon-Won (2022). An Exploratory Investigation of Archival Reference Services in the National Archives of Korea. Korean Society of Archives and Records Management, 22(3), 103-124. http://dx.doi.org/10.14404/JKSARM.2022.22.3.103
  22. Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1-47. https://doi.org/10.1145/505282.505283
  23. Voorhees, E. M. (1999). TREC-8 Question Answering Track Report. Proceedings of the 8th Text Retrieval Conference, 77-82.