• Title/Summary/Keyword: Generative Summarization

Search Result 9, Processing Time 0.024 seconds

Empirical Study for Automatic Evaluation of Abstractive Summarization by Error-Types (오류 유형에 따른 생성요약 모델의 본문-요약문 간 요약 성능평가 비교)

  • Seungsoo Lee;Sangwoo Kang
    • Korean Journal of Cognitive Science
    • /
    • v.34 no.3
    • /
    • pp.197-226
    • /
    • 2023
  • Generative Text Summarization is one of the Natural Language Processing tasks. It generates a short abbreviated summary while preserving the content of the long text. ROUGE is a widely used lexical-overlap based metric for text summarization models in generative summarization benchmarks. Although it shows very high performance, the studies report that 30% of the generated summary and the text are still inconsistent. This paper proposes a methodology for evaluating the performance of the summary model without using the correct summary. AggreFACT is a human-annotated dataset that classifies the types of errors in neural text summarization models. Among all the test candidates, the two cases, generation summary, and when errors occurred throughout the summary showed the highest correlation results. We observed that the proposed evaluation score showed a high correlation with models finetuned with BART and PEGASUS, which is pretrained with a large-scale Transformer structure.

Improving Abstractive Summarization by Training Masked Out-of-Vocabulary Words

  • Lee, Tae-Seok;Lee, Hyun-Young;Kang, Seung-Shik
    • Journal of Information Processing Systems
    • /
    • v.18 no.3
    • /
    • pp.344-358
    • /
    • 2022
  • Text summarization is the task of producing a shorter version of a long document while accurately preserving the main contents of the original text. Abstractive summarization generates novel words and phrases using a language generation method through text transformation and prior-embedded word information. However, newly coined words or out-of-vocabulary words decrease the performance of automatic summarization because they are not pre-trained in the machine learning process. In this study, we demonstrated an improvement in summarization quality through the contextualized embedding of BERT with out-of-vocabulary masking. In addition, explicitly providing precise pointing and an optional copy instruction along with BERT embedding, we achieved an increased accuracy than the baseline model. The recall-based word-generation metric ROUGE-1 score was 55.11 and the word-order-based ROUGE-L score was 39.65.

Information Video Summarization and Keyword-based Video Tracking System (정보성 동영상 요약 및 키워드 기반 영상검색 시스템)

  • Gihun Kim;Mikyeong Moon
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2023.07a
    • /
    • pp.701-702
    • /
    • 2023
  • 비대면 교육이 증가함에 따라 강의, 특강과 같은 정보성 동영상의 수가 급격히 많아지고 있다. 이러한 정보성 동영상을 보아야 하는 학습자들은 자원과 시간을 효율적으로 활용할 수 있는 동영상 이해 및 학습 시스템이 필요하다. 본 논문에서는 GPT-3 모델과 KoNLPy 사용하여 동영상 요약을 수행하고 키워드 기반 해당 영상 프레임으로 바로 갈 수 있는 시스템의 개발내용에 대해 기술한다. 이를 통해 동영상 콘텐츠를 효과적으로 활용하여 학습자들의 학습 효율성을 향상시킬 수 있을 것으로 기대한다.

  • PDF

Research on the Development Direction of Language Model-based Generative Artificial Intelligence through Patent Trend Analysis (특허 동향 분석을 통한 언어 모델 기반 생성형 인공지능 발전 방향 연구)

  • Daehee Kim;Jonghyun Lee;Beom-seok Kim;Jinhong Yang
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.16 no.5
    • /
    • pp.279-291
    • /
    • 2023
  • In recent years, language model-based generative AI technologies have made remarkable progress. In particular, it has attracted a lot of attention due to its increasing potential in various fields such as summarization and code writing. As a reflection of this interest, the number of patent applications related to generative AI has been increasing rapidly. In order to understand these trends and develop strategies accordingly, future forecasting is key. Predictions can be used to better understand the future trends in the field of technology and develop more effective strategies. In this paper, we analyzed patents filed to date to identify the direction of development of language model-based generative AI. In particular, we took an in-depth look at research and invention activities in each country, focusing on application trends by year and detailed technology. Through this analysis, we tried to understand the detailed technologies contained in the core patents and predict the future development trends of generative AI.

A Study on Korean Generative Question-Answering with Contextual Summarization (문맥 요약을 접목한 한국어 생성형 질의응답 모델 연구)

  • Jeongjae Nam;Wooyoung Kim;Sangduk Baek;Wonjun Lee;Taeyong Kim;Hyunsoo Yoon;Wooju Kim
    • Annual Conference on Human and Language Technology
    • /
    • 2023.10a
    • /
    • pp.581-585
    • /
    • 2023
  • Question Answering(QA)은 질문과 문맥에 대한 정보를 토대로 적절한 답변을 도출하는 작업이다. 이때 입력으로 주어지는 문맥 텍스트는 대부분 길기 때문에 QA 모델은 이 정보를 처리하기 위해 상당한 컴퓨팅 자원이 필요하다. 이 문제를 해결하기 위해 본 논문에서는 요약 모델을 활용한 요약 기반 QA 모델 프레임워크를 제안한다. 이를 통해 문맥 정보를 효과적으로 요약하면서도 QA 모델의 컴퓨팅 비용을 줄이고 성능을 유지하는 것을 목표로 한다.

  • PDF

Generative Evidence Inference Method using Document Summarization Dataset (문서 요약 데이터셋을 이용한 생성형 근거 추론 방법)

  • Yeajin Jang;Youngjin Jang;Harksoo Kim
    • Annual Conference on Human and Language Technology
    • /
    • 2023.10a
    • /
    • pp.137-140
    • /
    • 2023
  • 자연어처리는 인공지능 발전과 함께 주목받는 분야로 컴퓨터가 인간의 언어를 이해하게 하는 기술이다. 그러나 많은 인공지능 모델은 블랙박스처럼 동작하여 그 원리를 해석하거나 이해하기 힘들다는 문제점이 있다. 이 문제를 해결하기 위해 설명 가능한 인공지능의 중요성이 강조되고 있으며, 활발히 연구되고 있다. 연구 초기에는 모델의 예측에 큰 영향을 끼치는 단어나 절을 근거로 추출했지만 문제 해결을 위한 단서 수준에 그쳤으며, 이후 문장 단위의 근거로 확장된 연구가 수행되었다. 하지만 문서 내에 서로 떨어져 있는 근거 문장 사이에 누락된 문맥 정보로 인하여 이해에 어려움을 줄 수 있다. 따라서 본 논문에서는 사람에게 보다 이해하기 쉬운 근거를 제공하기 위한 생성형 기반의 근거 추론 연구를 수행하고자 한다. 높은 수준의 자연어 이해 능력이 필요한 문서 요약 데이터셋을 활용하여 근거를 생성하고자 하며, 실험을 통해 일부 기계독해 데이터 샘플에서 예측에 대한 적절한 근거를 제공하는 것을 확인했다.

  • PDF

A Lecture Summarization Application Using STT (Speech-To-Text) and ChatGPT (STT(Speech-To-Text)와 ChatGPT 를 활용한 강의 요약 애플리케이션)

  • Jin-Woong Kim;Bo-Sung Geum;Tae-Kook Kim
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2023.11a
    • /
    • pp.297-298
    • /
    • 2023
  • COVID-19 가 사실상 종식됨에 따라 대학 강의가 비대면 온라인 강의에서 대면 강의로 전환되었다. 온라인 강의에서는 다시 보기를 통한 복습이 가능했지만, 대면강의에서는 녹음을 통해서 이를 대체하고 있다. 하지만 다시 보기와 녹음본은 원하는 부분을 찾거나 내용을 요약하는데 있어서 시간이 오래 걸리고 불편하다. 본 논문에서는 강의 내용을 STT(Speech-to-Text) 기술을 활용하여 텍스트로 변환하고 ChatGPT(Chat-Generative Pre-trained Transformer)로 요약하는 애플리케이션을 제안한다.

A Study on Evaluating Summarization Performance using Generative Al Model (생성형 AI 모델을 활용한 요약 성능 평가 연구 )

  • Gyuri Choi;Seoyoon Park;Yejee Kang;Hansaem Kim
    • Annual Conference on Human and Language Technology
    • /
    • 2023.10a
    • /
    • pp.228-233
    • /
    • 2023
  • 인간의 수동 평가 시 시간과 비용의 소모, 주석자 간의 의견 불일치, 평가 결과의 품질 등 불가피한 한계가 발생한다. 본 논문에서는 맥락을 고려하고 긴 문장 입출력이 가능한 ChatGPT를 활용한 한국어 요약문 평가가 인간 평가를 대체하거나 보조하는 것이 가능한가에 대해 살펴보았다. 이를 위해 ChatGPT가 생성한 요약문에 정량적 평가와 정성적 평가를 진행하였으며 정량적 지표로 BERTScore, 정성적 지표로는 일관성, 관련성, 문법성, 유창성을 사용하였다. 평가 결과 ChatGPT4의 경우 인간 수동 평가를 보조할 수 있는 가능성이 있음을 확인하였다. ChatGPT가 영어 기반으로 학습된 모델임을 고려하여 오류 발견 성능을 검증하고자 한국어 오류 요약문으로 추가 평가를 진행하였다. 그 결과 ChatGPT3.5와 ChatGPT4의 오류 요약 평가 성능은 불안정하여 인간을 보조하기에는 아직 어려움이 있음을 확인하였다.

  • PDF

A Proposal of a Keyword Extraction System for Detecting Social Issues (사회문제 해결형 기술수요 발굴을 위한 키워드 추출 시스템 제안)

  • Jeong, Dami;Kim, Jaeseok;Kim, Gi-Nam;Heo, Jong-Uk;On, Byung-Won;Kang, Mijung
    • Journal of Intelligence and Information Systems
    • /
    • v.19 no.3
    • /
    • pp.1-23
    • /
    • 2013
  • To discover significant social issues such as unemployment, economy crisis, social welfare etc. that are urgent issues to be solved in a modern society, in the existing approach, researchers usually collect opinions from professional experts and scholars through either online or offline surveys. However, such a method does not seem to be effective from time to time. As usual, due to the problem of expense, a large number of survey replies are seldom gathered. In some cases, it is also hard to find out professional persons dealing with specific social issues. Thus, the sample set is often small and may have some bias. Furthermore, regarding a social issue, several experts may make totally different conclusions because each expert has his subjective point of view and different background. In this case, it is considerably hard to figure out what current social issues are and which social issues are really important. To surmount the shortcomings of the current approach, in this paper, we develop a prototype system that semi-automatically detects social issue keywords representing social issues and problems from about 1.3 million news articles issued by about 10 major domestic presses in Korea from June 2009 until July 2012. Our proposed system consists of (1) collecting and extracting texts from the collected news articles, (2) identifying only news articles related to social issues, (3) analyzing the lexical items of Korean sentences, (4) finding a set of topics regarding social keywords over time based on probabilistic topic modeling, (5) matching relevant paragraphs to a given topic, and (6) visualizing social keywords for easy understanding. In particular, we propose a novel matching algorithm relying on generative models. The goal of our proposed matching algorithm is to best match paragraphs to each topic. Technically, using a topic model such as Latent Dirichlet Allocation (LDA), we can obtain a set of topics, each of which has relevant terms and their probability values. In our problem, given a set of text documents (e.g., news articles), LDA shows a set of topic clusters, and then each topic cluster is labeled by human annotators, where each topic label stands for a social keyword. For example, suppose there is a topic (e.g., Topic1 = {(unemployment, 0.4), (layoff, 0.3), (business, 0.3)}) and then a human annotator labels "Unemployment Problem" on Topic1. In this example, it is non-trivial to understand what happened to the unemployment problem in our society. In other words, taking a look at only social keywords, we have no idea of the detailed events occurring in our society. To tackle this matter, we develop the matching algorithm that computes the probability value of a paragraph given a topic, relying on (i) topic terms and (ii) their probability values. For instance, given a set of text documents, we segment each text document to paragraphs. In the meantime, using LDA, we can extract a set of topics from the text documents. Based on our matching process, each paragraph is assigned to a topic, indicating that the paragraph best matches the topic. Finally, each topic has several best matched paragraphs. Furthermore, assuming there are a topic (e.g., Unemployment Problem) and the best matched paragraph (e.g., Up to 300 workers lost their jobs in XXX company at Seoul). In this case, we can grasp the detailed information of the social keyword such as "300 workers", "unemployment", "XXX company", and "Seoul". In addition, our system visualizes social keywords over time. Therefore, through our matching process and keyword visualization, most researchers will be able to detect social issues easily and quickly. Through this prototype system, we have detected various social issues appearing in our society and also showed effectiveness of our proposed methods according to our experimental results. Note that you can also use our proof-of-concept system in http://dslab.snu.ac.kr/demo.html.