• Title/Summary/Keyword: Datasets as Topic

검색결과 33건 처리시간 0.023초

Effects of Preprocessing on Text Classification in Balanced and Imbalanced Datasets

  • Mehmet F. Karaca
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제18권3호
    • /
    • pp.591-609
    • /
    • 2024
  • In this study, preprocessings with all combinations were examined in terms of the effects on decreasing word number, shortening the duration of the process and the classification success in balanced and imbalanced datasets which were unbalanced in different ratios. The decreases in the word number and the processing time provided by preprocessings were interrelated. It was seen that more successful classifications were made with Turkish datasets and English datasets were affected more from the situation of whether the dataset is balanced or not. It was found out that the incorrect classifications, which are in the classes having few documents in highly imbalanced datasets, were made by assigning to the class close to the related class in terms of topic in Turkish datasets and to the class which have many documents in English datasets. In terms of average scores, the highest classification was obtained in Turkish datasets as follows: with not applying lowercase, applying stemming and removing stop words, and in English datasets as follows: with applying lowercase and stemming, removing stop words. Applying stemming was the most important preprocessing method which increases the success in Turkish datasets, whereas removing stop words in English datasets. The maximum scores revealed that feature selection, feature size and classifier are more effective than preprocessing in classification success. It was concluded that preprocessing is necessary for text classification because it shortens the processing time and can achieve high classification success, a preprocessing method does not have the same effect in all languages, and different preprocessing methods are more successful for different languages.

Identifying Topic-Specific Experts on Microblog

  • Yu, Yan;Mo, Lingfei;Wang, Jian
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제10권6호
    • /
    • pp.2627-2647
    • /
    • 2016
  • With the rapid growth of microblog, expert identification on microblog has been playing a crucial role in many applications. While most previous expert identification studies only assess global authoritativeness of a user, there is no way to differentiate the authoritativeness in a particular aspect of topics. In this paper, we propose a novel model, which jointly models text and following relationship in the same generative process. Furthermore, we integrate a similarity-based weight scheme into the model to address the popular bias problem, and use followee topic distribution as prior information to make user's topic distribution more precisely. Our empirical study on two large real-world datasets shows that our proposed model produces significantly higher quality results than the prior arts.

Too Much Information - Trying to Help or Deceive? An Analysis of Yelp Reviews

  • Hyuk Shin;Hong Joo Lee;Ruth Angelie Cruz
    • Asia pacific journal of information systems
    • /
    • 제33권2호
    • /
    • pp.261-281
    • /
    • 2023
  • The proliferation of online customer reviews has completely changed how consumers purchase. Consumers now heavily depend on authentic experiences shared by previous customers. However, deceptive reviews that aim to manipulate customer decision-making to promote or defame a product or service pose a risk to businesses and buyers. The studies investigating consumer perception of deceptive reviews found that one of the important cues is based on review content. This study aims to investigate the impact of the information amount of review on the review truthfulness. This study adopted the Information Manipulation Theory (IMT) as an overarching theory, which asserts that the violations of one or more of the Gricean maxim are deceptive behaviors. It is regarded as a quantity violation if the required information amount is not delivered or more information is delivered; that is an attempt at deception. A topic modeling algorithm is implemented to reveal the distribution of each topic embedded in a text. This study measures information amount as topic diversity based on the results of topic modeling, and topic diversity shows how heterogeneous a text review is. Two datasets of restaurant reviews on Yelp.com, which have Filtered (deceptive) and Unfiltered (genuine) reviews, were used to test the hypotheses. Reviews that contain more diverse topics tend to be truthful. However, excessive topic diversity produces an inverted U-shaped relationship with truthfulness. Moreover, we find an interaction effect between topic diversity and reviews' ratings. This result suggests that the impact of topic diversity is strengthened when deceptive reviews have lower ratings. This study contributes to the existing literature on IMT by building the connection between topic diversity in a review and its truthfulness. In addition, the empirical results show that topic diversity is a reliable measure for gauging information amount of reviews.

Digital Epidemiology: Use of Digital Data Collected for Non-epidemiological Purposes in Epidemiological Studies

  • Park, Hyeoun-Ae;Jung, Hyesil;On, Jeongah;Park, Seul Ki;Kang, Hannah
    • Healthcare Informatics Research
    • /
    • 제24권4호
    • /
    • pp.253-262
    • /
    • 2018
  • Objectives: We reviewed digital epidemiological studies to characterize how researchers are using digital data by topic domain, study purpose, data source, and analytic method. Methods: We reviewed research articles published within the last decade that used digital data to answer epidemiological research questions. Data were abstracted from these articles using a data collection tool that we developed. Finally, we summarized the characteristics of the digital epidemiological studies. Results: We identified six main topic domains: infectious diseases (58.7%), non-communicable diseases (29.4%), mental health and substance use (8.3%), general population behavior (4.6%), environmental, dietary, and lifestyle (4.6%), and vital status (0.9%). We identified four categories for the study purpose: description (22.9%), exploration (34.9%), explanation (27.5%), and prediction and control (14.7%). We identified eight categories for the data sources: web search query (52.3%), social media posts (31.2%), web portal posts (11.9%), webpage access logs (7.3%), images (7.3%), mobile phone network data (1.8%), global positioning system data (1.8%), and others (2.8%). Of these, 50.5% used correlation analyses, 41.3% regression analyses, 25.6% machine learning, and 19.3% descriptive analyses. Conclusions: Digital data collected for non-epidemiological purposes are being used to study health phenomena in a variety of topic domains. Digital epidemiology requires access to large datasets and advanced analytics. Ensuring open access is clearly at odds with the desire to have as little personal data as possible in these large datasets to protect privacy. Establishment of data cooperatives with restricted access may be a solution to this dilemma.

자아 중심 네트워크 분석과 동적 인용 네트워크를 활용한 토픽모델링 기반 연구동향 분석에 관한 연구 (Combining Ego-centric Network Analysis and Dynamic Citation Network Analysis to Topic Modeling for Characterizing Research Trends)

  • 유소영
    • 정보관리학회지
    • /
    • 제32권1호
    • /
    • pp.153-169
    • /
    • 2015
  • 이 연구에서는 토픽 모델링 결과 해석의 용이성을 위하여, 동적 인용 네트워크를 활용하여 LDA 기반 토픽 모델링의 토픽 수를 설정하고 중복 배치된 주요 키워드를 자아 중심 네트워크 분석을 통해 재배치하여 제시하는 방법을 제안하였다. 'White LED' 두 분야의 논문 데이터를 이용하여 분석한 결과, 동적 인용 네트워크 분석을 통해 형성된 분석대상 문헌집단에 혼잡도에 따른 토픽수를 사용하고 중복 분류된 토픽 내 주요 키워드를 자아중심 네트워크 분석 기법을 적용하여 재배치한 결과가 토픽 간의 중복도가 가장 낮은 것으로 나타났다. 따라서 동적 인용 네트워크 및 자아 중심 네트워크 분석을 적용함으로써 토픽모델링에 의한 분석 결과를 보완하는 다면적인 연구 동향 분석이 가능할 것으로 보인다.

텍스트 마이닝 기법을 이용한 컴퓨터공학 및 정보학 분야 연구동향 조사: DBLP의 학술회의 데이터를 중심으로 (Investigation of Topic Trends in Computer and Information Science by Text Mining Techniques: From the Perspective of Conferences in DBLP)

  • 김수연;송성전;송민
    • 정보관리학회지
    • /
    • 제32권1호
    • /
    • pp.135-152
    • /
    • 2015
  • 이 논문의 연구목적은 컴퓨터공학 및 정보학 관련 연구동향을 분석하는 것이다. 이를 위해 텍스트마이닝 기법을 이용하여 DBLP(Digital Bibliography & Library Project)의 학술회의 데이터를 분석하였다. 대부분의 연구동향 분석 연구가 계량서지학적 연구방법을 사용한 것과 달리 이 논문에서는 LDA(Latent Dirichlet Allocation) 기반 다항분포 토픽모델링 기법을 이용하였다. 가능하면 컴퓨터공학 및 정보학과 관련된 광범위한 자료를 수집하기 위해서 DBLP에서 컴퓨터공학 및 정보학과 관련된 353개의 학술회의를 수집 대상으로 하였으며 2000년부터 2011년 기간 동안 출판된 236,170개의 문헌을 수집하였다. 토픽모델링 결과와 주제별 문헌 수, 주제별 학술회의 수를 조사하여 2000년부터 2011년 사이의 주제별 상위 저자와 주제별 상위 학술회의를 제시하였다. 주제동향 분석 결과 네트워크 관련 연구 주제 분야는 성장 패턴을 보였으며, 인공지능, 데이터마이닝 관련 연구 분야는 쇠퇴 패턴을 나타냈고, 지속 패턴을 보인 주제는 웹, 텍스트마이닝, 정보검색, 데이터베이스 관련 연구 주제이며, HCI, 정보시스템, 멀티미디어 시스템 관련 연구 주제 분야는 성장과 하락을 지속하는 변동 패턴을 나타냈다.

사용자 입력 문장에서 우울 관련 감정 탐지 (Detects depression-related emotions in user input sentences)

  • 오재동;오하영
    • 한국정보통신학회논문지
    • /
    • 제26권12호
    • /
    • pp.1759-1768
    • /
    • 2022
  • 본 논문은 AI Hub에서 제공하는 웰니스 대화 스크립트, 주제별 일상 대화 데이터세트와 Github에 공개된 챗봇 데이터세트를 활용하여 사용자의 발화에서 우울 관련 감정을 탐지하는 모델을 제안한다. 우울 관련 감정에는 우울감, 무기력을 비롯한 18가지 감정이 존재하며, 언어 모델에서 높은 성능을 보이는 KoBERT와 KoELECTRA 모델을 사용하여 감정 분류 작업을 수행한다. 모델별 성능 비교를 위해 우리는 데이터세트를 다양하게 구축하고, 좋은 성능을 보이는 모델에 대해 배치 크기와 학습률을 조정하면서 분류 결과를 비교한다. 더 나아가, 사람은 동시에 여러 감정을 느끼는 것을 반영하기 위해, 모델의 출력값이 특정 임계치보다 높은 레이블들을 모두 정답으로 선정함으로써, 다중 분류 작업을 수행한다. 이러한 과정을 통해 도출한 성능이 가장 좋은 모델을 Depression model이라 부르며, 이후 사용자 발화에 대해 우울 관련 감정을 분류할 때 해당 모델을 사용한다.

What Topics Have Been Studied in Korean Mathematics Education for 15 Years: Latent Topic Modeling Analysis

  • Hwang, Jihyun
    • 한국수학교육학회지시리즈D:수학교육연구
    • /
    • 제24권4호
    • /
    • pp.313-335
    • /
    • 2021
  • The purpose of this research is to identify topics discussed by Korean mathematics education studies and examine research trends for 15 years. I applied latent Dirichlet allocation (LDA) to the original text datasets including English abstracts of 3,157 articles published in eight journals indexed by the Korean Citation Index (KCI) from 1997 to 2019. I identified an LDA model with 60 topics, then research trends in 2,884 articles between 2002 and 2018 were as follows; mathematics educators have paid most attention to teacher education through 2010 to 2015 and curriculum analysis after 2016. The findings in this research can contribute to understand what have been discussed in Korean mathematics education society as well as what will and need to be emphasized more in the future compared to the global research trends. In addition, LDA has potentials to identify topics and keywords of manuscripts newly written and submitted to any journals in addition to information provided by authors.

Human Posture Recognition: Methodology and Implementation

  • Htike, Kyaw Kyaw;Khalifa, Othman O.
    • Journal of Electrical Engineering and Technology
    • /
    • 제10권4호
    • /
    • pp.1910-1914
    • /
    • 2015
  • Human posture recognition is an attractive and challenging topic in computer vision due to its promising applications in the areas of personal health care, environmental awareness, human-computer-interaction and surveillance systems. Human posture recognition in video sequences consists of two stages: the first stage is training and evaluation and the second is deployment. In the first stage, the system is trained and evaluated using datasets of human postures to ‘teach’ the system to classify human postures for any future inputs. When the training and evaluation process is deemed satisfactory as measured by recognition rates, the trained system is then deployed to recognize human postures in any input video sequence. Different classifiers were used in the training such as Multilayer Perceptron Feedforward Neural networks, Self-Organizing Maps, Fuzzy C Means and K Means. Results show that supervised learning classifiers tend to perform better than unsupervised classifiers for the case of human posture recognition.

Large Language Models: A Guide for Radiologists

  • Sunkyu Kim;Choong-kun Lee;Seung-seob Kim
    • Korean Journal of Radiology
    • /
    • 제25권2호
    • /
    • pp.126-133
    • /
    • 2024
  • Large language models (LLMs) have revolutionized the global landscape of technology beyond natural language processing. Owing to their extensive pre-training on vast datasets, contemporary LLMs can handle tasks ranging from general functionalities to domain-specific areas, such as radiology, without additional fine-tuning. General-purpose chatbots based on LLMs can optimize the efficiency of radiologists in terms of their professional work and research endeavors. Importantly, these LLMs are on a trajectory of rapid evolution, wherein challenges such as "hallucination," high training cost, and efficiency issues are addressed, along with the inclusion of multimodal inputs. In this review, we aim to offer conceptual knowledge and actionable guidance to radiologists interested in utilizing LLMs through a succinct overview of the topic and a summary of radiology-specific aspects, from the beginning to potential future directions.