Browse > Article
http://dx.doi.org/10.14404/JKSARM.2022.22.4.025

Test Set Construction for Quality Evaluation of NAK Portal's Search Service and the Status Analysis  

Jeong Ho, Na (전북대학교 일반대학원 기록관리학과)
Hyeon-Gi, So (전북대학교 일반대학원 기록관리학과)
Gyung Rok, Yeom ((주) 아이와즈)
Jung-Ok, Lee (국가기록원 서비스정책과)
Hyo-Jung, Oh (전북대학교 문헌정보학과, 문화융복합아카이빙연구소)
Publication Information
Journal of Korean Society of Archives and Records Management / v.22, no.4, 2022 , pp. 25-43 More about this Journal
Abstract
The ultimate record management's purpose is preservation and utilization. However, the National Archives of Korea (NAK)s Portal has problems such as search system aging and search tools dualization. As a result, the users' search satisfaction is not satisfied, and the improvement demand increases. This study aimed to evaluate the NAK's search quality as a preliminary study for NAK search system advancement. To this end, we analyzed the current status of CAMS and NAK's Portal. Then, we established the test sets and evaluated the NAK's Portal quality from the user's point of view. Evaluation results were analyzed using Precision, Recall, F-score, and MRR. The analysis results showed that the overall search performance was low, particularly in the "advanced subject search," which showed low performance in Precision, Recall, and MRR. Thus, improvement is urgently needed. The test sets established for this study are expected to be used as a basis for objectively measuring the improvement of the search performance after the NAK search system advancement.
Keywords
National Archives of Korea (NAK) Portal; Information Retrieval; Finding Aids; Retrieval Evaluation; Test Set;
Citations & Related Records
Times Cited By KSCI : 13  (Citation Analysis)
연도 인용수 순위
1 Chang, InHo (2011). Developing and evaluating an ontology-based legal retrieval system. Journal of the Korean Society for Library and Information Science, 45(2), 345-366. https://doi.org/10.4275/KSLIS.2011.45.2.345   DOI
2 Jang, Hee-Jung (2012). A study on evaluation of national archives websites. Journal of Korean Society of Archives and Records Management, 12(2), 51-70. https://doi.org/10.14404/JKSARM.2012.12.2.051   DOI
3 Kim, Jang-hwan & Lee, Eun Byol (2015). A Study on the Development of the National Assembly Archives and Records Integrated Management System. Journal of Korean Society of Archives and Records Management, 15(2), 103-136. https://doi.org/10.14404/JKSARM.2015.15.2.103   DOI
4 Kim, Jihyun (2012). A study on users' perception of reference services in national archives of korea. Journal of Korean Society of Archives and Records Management, 12(1), 167-187. https://doi.org/10.14404/JKSARM.2012.12.1.167   DOI
5 Kim, Hyun Hee (2007). An Experimental Study on the Retrieval Efficiency of the FRBR Based Bibliographic Retrieval System. Journal of Korean Library and Information Science Society, 38(3), 223-246. https://doi.org/10.16981/kliss.38.3.200709.223   DOI
6 Lee, Byung Gil & Kim, Heesop (2013). Design and Evaluation of an Individual Instance-based Ontology Retrieval System for Archival Records of the "Saemaul Movement". Journal of Korean Society of Archives and Records Management, 13(3), 67-97. https://doi.org/10.14404/JKSARM.2013.13.3.067   DOI
7 Lee, Hyojin & Kim, Jihyun (2021). A Study on Improvement Plans for National Archives of Korea Website's Search Service through Its Usability Evaluation. Journal of Korean Society of Archives and Records Management, 21(3), 187-215. https://doi.org/10.14404/JKSARM.2021.21.3.187   DOI
8 Lee, Kyung Nam (2013). Evaluation of search functions of the standard records management systems. The Korean Journal of Archival Studies, 37, 273-305. https://doi.org/10.20923/kjas.2013.37.273   DOI
9 Lee, Sang Gin & Chung, Jason (2011). An Empirical Study on the Marketing Performance of e-Tradeusing Search Engine Optimization. International Commerce and Information Review, 13(1), 3-28. https://doi.org/10.15798/kaici.13.1.201103.3   DOI
10 Lee, Yoon-Ryeong & Rieh, Hae-Young (2014). A study on the improvement direction for online finding aids: based on the assessment of National Archives. Journal of Korean Society of Archives and Records Management, 14(1), 75-100. https://doi.org/10.14404/JKSARM.2014.14.1.075   DOI
11 National Archives of Korea (2014). National Archives of Korea White Paper 2013. Daejeon: National Archives of Korea.
12 National Archives of Korea (2015). National Archives of Korea White Paper 2014. Daejeon: National Archives of Korea.
13 National Archives of Korea (2021). National Archives of Korea Statistical Yearbook 2021. Daejeon: National Archives of Korea.
14 Park, Min Soo & Seo, Eun-Gyoung (2012). Analyzing User Satisfaction of the Standard Records Management System. Journal of Korean Society of Archives and Records Management, 12(1), 189-207. https://doi.org/10.14404/JKSARM.2012.12.1.189   DOI
15 Noh, Meung-Hoan (2020). The Way for Data Archive and Records/Archive Management in the 4th Industrial Revolution Era : Critical Reviews and Mid- and Long-term Proposals for the National Archives' Administrative Information Dataset Records/Archive Management Implementation Plan. The Korean Journal of Archival, Information and Cultural Studies, 11, 7-44. https://doi.org/10.23035/kaics.2020.1.11.007   DOI
16 Park, Mi-Sung (2009). Task Review of INEX Book Search Track. Journal of Korean Library and Information Science Society, 40(4), 199-225. https://doi.org/10.16981/kliss.40.4.200912.199   DOI
17 Park, Minsoo & Hyun, Mi-Hwan (2011). Usability of the national science and technology information system. Journal of the Korean BIBLIA Society for library and Information Science, 22(4), 5-19. https://doi.org/10.14699/kbiblia.2011.22.4.005   DOI
18 Park, Sang Hyun & Kim, Heesop (2014). Design and Implementation of an EAD-based Integrated Retrieval System for the Archives in Korea. Journal of Korean Society of Archives and Records Management, 14(1), 101-124. https://doi.org/10.14404/JKSARM.2014.14.1.101   DOI
19 Park, Soyeon, Lee, Joon-Ho & Jeon, Jiwoon (2006). Evaluation of the documents from the web-based question and answer service. Journal of the Korean society for library and information science, 40(2), 299-314.   DOI
20 Park, Soyeon & Lee, Joon-Ho (2008). Comparative evaluation of the unified search services provided by major Korean search portals. Journal of Korean library and information science society, 39(1), 265-278. https://doi.org/10.16981/kliss.39.1.200803.265   DOI
21 Voorhees, E. M. (1999). TREC-8 Question Answering Track Report. Proceedings of the 8th Text Retrieval Conference, 77-82.
22 Seol, Moon-Won (2022). An Exploratory Investigation of Archival Reference Services in the National Archives of Korea. Korean Society of Archives and Records Management, 22(3), 103-124. http://dx.doi.org/10.14404/JKSARM.2022.22.3.103   DOI
23 Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1-47. https://doi.org/10.1145/505282.505283   DOI