• Title/Summary/Keyword: Dirichlet problem

Search Result 175, Processing Time 0.02 seconds

A Study on Educational Data Mining for Public Data Portal through Topic Modeling Method with Latent Dirichlet Allocation (LDA기반 토픽모델링을 활용한 공공데이터 기반의 교육용 데이터마이닝 연구)

  • Seungki Shin
    • Journal of The Korean Association of Information Education
    • /
    • v.26 no.5
    • /
    • pp.439-448
    • /
    • 2022
  • This study aims to search for education-related datasets provided by public data portals and examine what data types are constructed through classification using topic modeling methods. Regarding the data of the public data portal, 3,072 cases of file data in the education field were collected based on the classification system. Text mining analysis was performed using the LDA-based topic modeling method with stopword processing and data pre-processing for each dataset. Program information and student-supporting notifications were usually provided in the pre-classified dataset for education from the data portal. On the other hand, the characteristics of educational programs and supporting information for the disabled, parents, the elderly, and children through the perspective of lifelong education were generally indicated in the dataset collected by searching for education. The results of data analysis through this study show that providing sufficient educational information through the public data portal would be better to help the students' data science-based decision-making and problem-solving skills.

A Proposal of a Keyword Extraction System for Detecting Social Issues (사회문제 해결형 기술수요 발굴을 위한 키워드 추출 시스템 제안)

  • Jeong, Dami;Kim, Jaeseok;Kim, Gi-Nam;Heo, Jong-Uk;On, Byung-Won;Kang, Mijung
    • Journal of Intelligence and Information Systems
    • /
    • v.19 no.3
    • /
    • pp.1-23
    • /
    • 2013
  • To discover significant social issues such as unemployment, economy crisis, social welfare etc. that are urgent issues to be solved in a modern society, in the existing approach, researchers usually collect opinions from professional experts and scholars through either online or offline surveys. However, such a method does not seem to be effective from time to time. As usual, due to the problem of expense, a large number of survey replies are seldom gathered. In some cases, it is also hard to find out professional persons dealing with specific social issues. Thus, the sample set is often small and may have some bias. Furthermore, regarding a social issue, several experts may make totally different conclusions because each expert has his subjective point of view and different background. In this case, it is considerably hard to figure out what current social issues are and which social issues are really important. To surmount the shortcomings of the current approach, in this paper, we develop a prototype system that semi-automatically detects social issue keywords representing social issues and problems from about 1.3 million news articles issued by about 10 major domestic presses in Korea from June 2009 until July 2012. Our proposed system consists of (1) collecting and extracting texts from the collected news articles, (2) identifying only news articles related to social issues, (3) analyzing the lexical items of Korean sentences, (4) finding a set of topics regarding social keywords over time based on probabilistic topic modeling, (5) matching relevant paragraphs to a given topic, and (6) visualizing social keywords for easy understanding. In particular, we propose a novel matching algorithm relying on generative models. The goal of our proposed matching algorithm is to best match paragraphs to each topic. Technically, using a topic model such as Latent Dirichlet Allocation (LDA), we can obtain a set of topics, each of which has relevant terms and their probability values. In our problem, given a set of text documents (e.g., news articles), LDA shows a set of topic clusters, and then each topic cluster is labeled by human annotators, where each topic label stands for a social keyword. For example, suppose there is a topic (e.g., Topic1 = {(unemployment, 0.4), (layoff, 0.3), (business, 0.3)}) and then a human annotator labels "Unemployment Problem" on Topic1. In this example, it is non-trivial to understand what happened to the unemployment problem in our society. In other words, taking a look at only social keywords, we have no idea of the detailed events occurring in our society. To tackle this matter, we develop the matching algorithm that computes the probability value of a paragraph given a topic, relying on (i) topic terms and (ii) their probability values. For instance, given a set of text documents, we segment each text document to paragraphs. In the meantime, using LDA, we can extract a set of topics from the text documents. Based on our matching process, each paragraph is assigned to a topic, indicating that the paragraph best matches the topic. Finally, each topic has several best matched paragraphs. Furthermore, assuming there are a topic (e.g., Unemployment Problem) and the best matched paragraph (e.g., Up to 300 workers lost their jobs in XXX company at Seoul). In this case, we can grasp the detailed information of the social keyword such as "300 workers", "unemployment", "XXX company", and "Seoul". In addition, our system visualizes social keywords over time. Therefore, through our matching process and keyword visualization, most researchers will be able to detect social issues easily and quickly. Through this prototype system, we have detected various social issues appearing in our society and also showed effectiveness of our proposed methods according to our experimental results. Note that you can also use our proof-of-concept system in http://dslab.snu.ac.kr/demo.html.

Case Study of Big Data-Based Agri-food Recommendation System According to Types of Customers (빅데이터 기반 소비자 유형별 농식품 추천시스템 구축 사례)

  • Moon, Junghoon;Jang, Ikhoon;Choe, Young Chan;Kim, Jin Gyo;Bock, Gene
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.40 no.5
    • /
    • pp.903-913
    • /
    • 2015
  • The Korea Agency of Education, Promotion and Information Service in Food, Agriculture, Forestry and Fisheries launched a public data portal service in January 2015. The service provides customized information for consumers through an agri-food recommendation system built-in portal service. The recommendation system has fallowing characteristics. First, the system can increase recommendation accuracy by using a wide variety of agri-food related data, including SNS opinion mining, consumer's purchase data, climate data, and wholesale price data. Second, the system uses segmentation method based on consumer's lifestyle and megatrends factors to overcome the cold start problem. Third, the system recommends agri-foods to users reflecting various preference contextual factors by using recommendation algorithm, dirichlet-multinomial distribution. In addition, the system provides diverse information related to recommended agri-foods to increase interest in agri-food of service users.

A Study on Analysis of national R&D research trends for Artificial Intelligence using LDA topic modeling (LDA 토픽모델링을 활용한 인공지능 관련 국가R&D 연구동향 분석)

  • Yang, MyungSeok;Lee, SungHee;Park, KeunHee;Choi, KwangNam;Kim, TaeHyun
    • Journal of Internet Computing and Services
    • /
    • v.22 no.5
    • /
    • pp.47-55
    • /
    • 2021
  • Analysis of research trends in specific subject areas is performed by examining related topics and subject changes by using topic modeling techniques through keyword extraction for most of the literature information (paper, patents, etc.). Unlike existing research methods, this paper extracts topics related to the research topic using the LDA topic modeling technique for the project information of national R&D projects provided by the National Science and Technology Knowledge Information Service (NTIS) in the field of artificial intelligence. By analyzing these topics, this study aims to analyze research topics and investment directions for national R&D projects. NTIS provides a vast amount of national R&D information, from information on tasks carried out through national R&D projects to research results (thesis, patents, etc.) generated through research. In this paper, the search results were confirmed by performing artificial intelligence keywords and related classification searches in NTIS integrated search, and basic data was constructed by downloading the latest three-year project information. Using the LDA topic modeling library provided by Python, related topics and keywords were extracted and analyzed for basic data (research goals, research content, expected effects, keywords, etc.) to derive insights on the direction of research investment.

Problem Identification and Improvement Measures through Government24 App User Review Analysis: Insights through Topic Model (정부24 앱 사용자 리뷰 분석을 통한 문제 파악 및 개선방안: 토픽 모델을 통한 통찰)

  • MuMoungCho Han;Mijin Noh;YangSok Kim
    • Smart Media Journal
    • /
    • v.12 no.11
    • /
    • pp.27-35
    • /
    • 2023
  • Fourth Industrial Revolution and COVID-19 pandemic have boosted the use of Government 24 app for public service complaints in the era of non-face-to-face interactions. there has been a growing influx of complaints and improvement demands from users of public apps. Furthermore, systematic management of public apps is deemed necessary. The aim of this study is to analyze the grievances of Government 24 app users, understand the current dissatisfaction among citizens, and propose potential improvements. Data were collected from the Google Play Store from May 2, 2013, to June 30, 2023, comprising a total of 6,344 records. Among these, 1,199 records with a rating of 1 and at least one 'thumbs-up' were used for topic modeling analysis. The analysis revealed seven topics: 'Issues with certificate issuance,' 'Website functionality and UI problems,' 'User ID-related issues,' 'Update problems,' 'Government employee app management issues,' 'Budget wastage concerns ((It's not worth even a single star) or (It's a waste of taxpayers' money)),' and 'Password-related problems.' Furthermore, the overall trend of these topics showed an increase until 2021, a slight decrease in 2022, but a resurgence in 2023, underscoring the urgency of updates and management. We hope that the results of this study will contribute to the development and management of public apps that satisfy citizens in the future.