A Case Study on Metadata Extractionfor Records Management Using ChatGPT (챗GPT를 활용한 기록관리 메타데이터 추출 사례연구)
-
- Journal of Korean Society of Archives and Records Management
- /
- v.24 no.2
- /
- pp.89-112
- /
- 2024
Metadata is a crucial component of record management, playing a vital role in properly managing and understanding the record. In cases where automatic metadata assignment is not feasible, manual input by records professionals becomes necessary. This study aims to alleviate the challenges associated with manual entry by proposing a method that harnesses ChatGPT technology for extracting records management metadata elements. To employ ChatGPT technology, a Python program utilizing the LangChain library was developed. This program was designed to analyze PDF documents and extract metadata from records through questions, both with a locally installed instance of ChatGPT and the ChatGPT online service. Multiple PDF documents were subjected to this process to test the effectiveness of metadata extraction. The results revealed that while using LangChain with ChatGPT-3.5 turbo provided a secure environment, it exhibited some limitations in accurately retrieving metadata elements. Conversely, the ChatGPT-4 online service yielded relatively accurate results despite being unable to handle sensitive documents for security reasons. This exploration underscores the potential of utilizing ChatGPT technology to extract metadata in records management. With advancements in ChatGPT-related technologies, safer and more accurate results are expected to be achieved. Leveraging these advantages can significantly enhance the efficiency and productivity of tasks associated with managing records and metadata in archives.
Predicting IT trends has been a long and important subject for information systems research. IT trend prediction makes it possible to acknowledge emerging eras of innovation and allocate budgets to prepare against rapidly changing technological trends. Towards the end of each year, various domestic and global organizations predict and announce IT trends for the following year. For example, Gartner Predicts 10 top IT trend during the next year, and these predictions affect IT and industry leaders and organization's basic assumptions about technology and the future of IT, but the accuracy of these reports are difficult to verify. Social media data can be useful tool to verify the accuracy. As social media services have gained in popularity, it is used in a variety of ways, from posting about personal daily life to keeping up to date with news and trends. In the recent years, rates of social media activity in Korea have reached unprecedented levels. Hundreds of millions of users now participate in online social networks and communicate with colleague and friends their opinions and thoughts. In particular, Twitter is currently the major micro blog service, it has an important function named 'tweets' which is to report their current thoughts and actions, comments on news and engage in discussions. For an analysis on IT trends, we chose Tweet data because not only it produces massive unstructured textual data in real time but also it serves as an influential channel for opinion leading on technology. Previous studies found that the tweet data provides useful information and detects the trend of society effectively, these studies also identifies that Twitter can track the issue faster than the other media, newspapers. Therefore, this study investigates how frequently the predicted IT trends for the following year announced by public organizations are mentioned on social network services like Twitter. IT trend predictions for 2013, announced near the end of 2012 from two domestic organizations, the National IT Industry Promotion Agency (NIPA) and the National Information Society Agency (NIA), were used as a basis for this research. The present study analyzes the Twitter data generated from Seoul (Korea) compared with the predictions of the two organizations to analyze the differences. Thus, Twitter data analysis requires various natural language processing techniques, including the removal of stop words, and noun extraction for processing various unrefined forms of unstructured data. To overcome these challenges, we used SAS IRS (Information Retrieval Studio) developed by SAS to capture the trend in real-time processing big stream datasets of Twitter. The system offers a framework for crawling, normalizing, analyzing, indexing and searching tweet data. As a result, we have crawled the entire Twitter sphere in Seoul area and obtained 21,589 tweets in 2013 to review how frequently the IT trend topics announced by the two organizations were mentioned by the people in Seoul. The results shows that most IT trend predicted by NIPA and NIA were all frequently mentioned in Twitter except some topics such as 'new types of security threat', 'green IT', 'next generation semiconductor' since these topics non generalized compound words so they can be mentioned in Twitter with other words. To answer whether the IT trend tweets from Korea is related to the following year's IT trends in real world, we compared Twitter's trending topics with those in Nara Market, Korea's online e-Procurement system which is a nationwide web-based procurement system, dealing with whole procurement process of all public organizations in Korea. The correlation analysis show that Tweet frequencies on IT trending topics predicted by NIPA and NIA are significantly correlated with frequencies on IT topics mentioned in project announcements by Nara market in 2012 and 2013. The main contribution of our research can be found in the following aspects: i) the IT topic predictions announced by NIPA and NIA can provide an effective guideline to IT professionals and researchers in Korea who are looking for verified IT topic trends in the following topic, ii) researchers can use Twitter to get some useful ideas to detect and predict dynamic trends of technological and social issues.
Recently, big data analysis has developed into a field of interest to individuals and non-experts as well as companies and professionals. Accordingly, it is utilized for marketing and social problem solving by analyzing the data currently opened or collected directly. In Korea, various companies and individuals are challenging big data analysis, but it is difficult from the initial stage of analysis due to limitation of big data disclosure and collection difficulties. Nowadays, the system improvement for big data activation and big data disclosure services are variously carried out in Korea and abroad, and services for opening public data such as domestic government 3.0 (data.go.kr) are mainly implemented. In addition to the efforts made by the government, services that share data held by corporations or individuals are running, but it is difficult to find useful data because of the lack of shared data. In addition, big data traffic problems can occur because it is necessary to download and examine the entire data in order to grasp the attributes and simple information about the shared data. Therefore, We need for a new system for big data processing and utilization. First, big data pre-analysis technology is needed as a way to solve big data sharing problem. Pre-analysis is a concept proposed in this paper in order to solve the problem of sharing big data, and it means to provide users with the results generated by pre-analyzing the data in advance. Through preliminary analysis, it is possible to improve the usability of big data by providing information that can grasp the properties and characteristics of big data when the data user searches for big data. In addition, by sharing the summary data or sample data generated through the pre-analysis, it is possible to solve the security problem that may occur when the original data is disclosed, thereby enabling the big data sharing between the data provider and the data user. Second, it is necessary to quickly generate appropriate preprocessing results according to the level of disclosure or network status of raw data and to provide the results to users through big data distribution processing using spark. Third, in order to solve the problem of big traffic, the system monitors the traffic of the network in real time. When preprocessing the data requested by the user, preprocessing to a size available in the current network and transmitting it to the user is required so that no big traffic occurs. In this paper, we present various data sizes according to the level of disclosure through pre - analysis. This method is expected to show a low traffic volume when compared with the conventional method of sharing only raw data in a large number of systems. In this paper, we describe how to solve problems that occur when big data is released and used, and to help facilitate sharing and analysis. The client-server model uses SPARK for fast analysis and processing of user requests. Server Agent and a Client Agent, each of which is deployed on the Server and Client side. The Server Agent is a necessary agent for the data provider and performs preliminary analysis of big data to generate Data Descriptor with information of Sample Data, Summary Data, and Raw Data. In addition, it performs fast and efficient big data preprocessing through big data distribution processing and continuously monitors network traffic. The Client Agent is an agent placed on the data user side. It can search the big data through the Data Descriptor which is the result of the pre-analysis and can quickly search the data. The desired data can be requested from the server to download the big data according to the level of disclosure. It separates the Server Agent and the client agent when the data provider publishes the data for data to be used by the user. In particular, we focus on the Big Data Sharing, Distributed Big Data Processing, Big Traffic problem, and construct the detailed module of the client - server model and present the design method of each module. The system designed on the basis of the proposed model, the user who acquires the data analyzes the data in the desired direction or preprocesses the new data. By analyzing the newly processed data through the server agent, the data user changes its role as the data provider. The data provider can also obtain useful statistical information from the Data Descriptor of the data it discloses and become a data user to perform new analysis using the sample data. In this way, raw data is processed and processed big data is utilized by the user, thereby forming a natural shared environment. The role of data provider and data user is not distinguished, and provides an ideal shared service that enables everyone to be a provider and a user. The client-server model solves the problem of sharing big data and provides a free sharing environment to securely big data disclosure and provides an ideal shared service to easily find big data.
Recently concerns have been raised due to the unbalanced supply of crops: the price of crops has been unstable and at one point the price went up so high that the word Agflation(agriculture+ inflation) was coined. Korea, in particular, is a small-sized country and needs to secure the stable supply of crops by investing in the produce importation at a national level. Investment in foreign produce importation is becoming more important as a measure for sufficient supply of crops, limited supply of domestic crops, weakened farming conditions worldwide, as well as recent changes in the use of crops due to the development of bio-fuels, influence of carbon emission on crops, the price increase in crops, and influx of foreign hot money. However, there are many problems with investing in foreign produce importation: lack of support from the government; lack of farming information and technology; difficulty in securing the capital; no immediate pay-off from the investment and insufficient management. Although foreign produce is originally more price-competitive than domestic produce, it loses its competiveness in the process of importation (due to high tariffs) and poor distribution system, which makes it difficult to sell in Korea. Therefore, investment in foreign produce importation is being questioned for feasibility; to make it possible, foreign produce must maintain the price-competitiveness. Especially, harvest of agricultural products depends on natural and geographical conditions of each country and those products have indigenous properties, so distribution system according to import and export of agricultural products should be treated more carefully than that of other industries. Distribution costs are differentiated into each item and include cost of sorting and wrapping, cost of wrapping materials, cost of domestic transport, cost of international transport and cost of clearing customs for import and export. So transporting and storing agricultural products generates considerable costs compared with other products. Also, due to upgrade of dietary life, needs for stability, taste and visible quality toward food including agricultural products are being raised and wrong way of storage causes decomposition of food and loss of freshness, making the storage more difficult than that in room temperature, so storage and transport in distribution of agricultural products needs specialty. In addition, because lack of specialty in distribution and circulation such as storage and wrapping does not solve limit factors in distance, the distribution and circulation has been limited to a form of import and export within short-distant region. Therefore, need for distribution out-sourcing which can satisfy specialty in managing distribution and circulation and it is needed to establish more effective distribution system. However, existing distribution system of agricultural products is exposed to various problems including problems in distribution channel, making distribution and strategy for distribution and those problems are as follows. First, in case of investment in overseas agricultural industry, stable supply of the products is difficult because areas of production are dispersed widely and influenced by outer factors due to including overseas distribution channels. Also, at the aspect of quality, standardization of products is difficult, distribution system is quite complicated and unreasonable due to long distribution channels according to international trade and financial and institutional support is not enough. Especially, there are quite a lot of ineffective factors including multi level distribution process, dramatic gap between production cost and customer's cost, lack of physical distribution facilities and difficulties in storage and transport due to lack of wrapping containers. Besides, because import and export of agricultural products has been manages under the company's own distribution according to transaction contract between manufacturers and exporting company, efficiency is low due to excessive investment in fixed costs and lack of specialty in dealing with agricultural products causes fall of value of products, showing the limit to lose price-competitiveness. Especially, because lack of specialty in distribution and circulation such as storage and wrapping does not solve limit factors in distance, the distribution and circulation has been limited to a form of import and export within short-distant region. Therefore, need for distribution out-sourcing which can satisfy specialty in managing distribution and circulation and it is needed to establish more effective distribution system. Second, among tangible and intangible services which promote the efficiency of the whole distribution, a function building distribution environment which includes distribution information, system for standard and inspection, distribution finance, system for diversification of risks, education and training, distribution administration and tax system is wanted. In general, such a function building distribution environment is difficult to be changed and supplement innovatively because its effect compared with investment does not appear immediately despite of its necessity. Especially, in case of distribution of agricultural products, as a function of collecting and distributing is performed individually through various channels, the importance of distribution information and standardization is getting more focus due to the problem of repetition of work and lack of specialty. Also, efficient management of distribution is quite difficult due to lack of professionals in distribution, so support to professional education is needed. Third, though effort to keep self-sufficiency ratio of staple food, rice is regarded as important at the government level, level of dependency on overseas of others crops is high. Therefore, plan for stable securing food resources aside from staple food is also necessary. Especially, governmental organizations of agricultural products distribution in Korea are production-centered and have unreasonable structure whose function at the aspect of distribution and consumption is quite insufficient. And development of new distribution channels which can deal with changes in distribution environment and they do not achieve actual results of strategy for distribution due to non-positive strategy for price distribution. That is, it implies the possibility that base for supply will become vulnerable because it does not mediate appropriate interests on total distribution channels such as manufacturers, wholesale dealers and vendors by emphasizing consumer protection excessively in the distribution of agricultural products. Therefore, this study examined fundamental concept and actual situation for our investment to overseas agriculture, drew necessities, considerations, problems, etc. of overseas agricultural investment and suggested improvements at the level of distribution for price competitiveness of agricultural products cultivated in overseas under five aspects; government's indirect support, distribution's modernization and distribution information function's strengthening, government's political support for distribution facility, transportation route, load and unloading works' improvement, price competitiveness' securing, professional manpower's cultivation by education and training, etc. Here are some suggestions for foreign produce importation. First, the government should conduct a survey on the current distribution channels and analyze the situation to establish a measure for long-term development plans. By providing each agricultural area with a guideline for planning appropriate production of crops, the government can help farmers be ready for importation, and prevent them from producing same crops all at the same time. Government can sign an MOU with the foreign government and promote the importation so that the development of agricultural resources can be stable and steady. Second, the government can establish a strategy for an effective distribution system by providing farmers and agriculture-related workers with the distribution information such as price, production, demand, market structure and location, feature of each crop, and etc. In order for such distribution system to become feasible, the government needs to reconstruct the current distribution system, designate a public organization for providing distribution information and set the criteria for level of produce quality, trade units, and package units. Third, the government should provide financial support and a policy to seek an efficient distribution channel for foreign produce to be delivered fresh: the government should expand distribution facilities (for selecting, packaging, storing, and processing) and transportation vehicles while modernizing old facilities. There should be another policy to improve the efficiency of unloading, and to lower the cost of distribution. Fourth, it is necessary to enact a new law covering exceptional cases for importing produce in order to maintain the price competitiveness; currently the high tariffs is keeping the imported produce from being distributed domestically. However, the new adjustment should be made carefully within the WTO regulations since it can create a problem from giving preferential tariffs. The government can also simplify the distribution channels in order to reduce the cost in the distribution process. Fifth, the government should educate distributors to raise the efficiency and to modernize the distribution system. It is necessary to develop human resources by educating people regarding the foreign agricultural environment, the produce quality, management skills, and by introducing some successful cases in advanced countries.