• Title/Summary/Keyword: sentence summarization

Search Result 69, Processing Time 0.035 seconds

Document Summarization Using Mutual Recommendation with LSA and Sense Analysis (LSA를 이용한 문장 상호 추천과 문장 성향 분석을 통한 문서 요약)

  • Lee, Dong-Wook;Baek, Seo-Hyeon;Park, Min-Ji;Park, Jin-Hee;Jung, Hye-Wuk;Lee, Jee-Hyong
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.22 no.5
    • /
    • pp.656-662
    • /
    • 2012
  • In this paper, we describe a new summarizing method based on a graph-based and a sense-based analysis. In the graph-based analysis, we convert sentences in a document into word vectors and calculate the similarity between each sentence using LSA. We reflect this similarity of sentences and the rarity scores of words in sentences to define weights of edges in the graph. Meanwhile, in the sense-based analysis, in order to determine the sense of words, subjectivity or objectivity, we built a database which is extended from the golden standards using Wordnet. We calculate the subjectivity of sentences from the sense of words, and select more subjective sentences. Lastly, we combine the results of these two methods. We evaluate the performance of the proposed method using classification games, which are usually used to measure the performances of summarization methods. We compare our method with the MS-Word auto-summarization, and verify the effectiveness of ours.

Multi-Topic Meeting Summarization using Lexical Co-occurrence Frequency and Distribution (어휘의 동시 발생 빈도와 분포를 이용한 다중 주제 회의록 요약)

  • Lee, Byung-Soo;Lee, Jee-Hyong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2015.07a
    • /
    • pp.13-16
    • /
    • 2015
  • 본 논문에서는 어휘의 동시 발생 (co-occurrence) 빈도와 분포를 이용한 회의록 요약방법을 제안한다. 회의록은 일반 문서와 달리 문서에 여러 세부적인 주제들이 나타나며, 잘못된 형식의 문장, 불필요한 잡담들을 포함하고 있기 때문에 이러한 특징들이 문서요약 과정에서 고려되어야 한다. 기존의 일반적인 문서요약 방법은 하나의 주제를 기반으로 문서 전체에서 가장 중요한 문장으로 요약하기 때문에 다중 주제 회의록 요약에는 적합하지 않다. 제안한 방법은 먼저 어휘의 동시 발생 (co-occurrence) 빈도를 이용하여 회의록 분할 (segmentation) 과정을 수행한다. 다음으로 주제의 구분에 따라 분할된 각 영역 (block)의 중요 단어 집합 생성, 중요 문장 추출 과정을 통해 회의록의 중요 문장들을 선별한다. 마지막으로 추출된 중요 문장들의 위치, 종속 관계를 고려하여 최종적으로 회의록을 요약한다. AMI meeting corpus를 대상으로 실험한 결과, 제안한 방법이 baseline 요약 방법들보다 요약 비율에 따른 평가 및 요약문의 세부 주제별 평가에서 우수한 요약 성능을 보임을 확인하였다.

  • PDF

Automatic Quality Evaluation with Completeness and Succinctness for Text Summarization (완전성과 간결성을 고려한 텍스트 요약 품질의 자동 평가 기법)

  • Ko, Eunjung;Kim, Namgyu
    • Journal of Intelligence and Information Systems
    • /
    • v.24 no.2
    • /
    • pp.125-148
    • /
    • 2018
  • Recently, as the demand for big data analysis increases, cases of analyzing unstructured data and using the results are also increasing. Among the various types of unstructured data, text is used as a means of communicating information in almost all fields. In addition, many analysts are interested in the amount of data is very large and relatively easy to collect compared to other unstructured and structured data. Among the various text analysis applications, document classification which classifies documents into predetermined categories, topic modeling which extracts major topics from a large number of documents, sentimental analysis or opinion mining that identifies emotions or opinions contained in texts, and Text Summarization which summarize the main contents from one document or several documents have been actively studied. Especially, the text summarization technique is actively applied in the business through the news summary service, the privacy policy summary service, ect. In addition, much research has been done in academia in accordance with the extraction approach which provides the main elements of the document selectively and the abstraction approach which extracts the elements of the document and composes new sentences by combining them. However, the technique of evaluating the quality of automatically summarized documents has not made much progress compared to the technique of automatic text summarization. Most of existing studies dealing with the quality evaluation of summarization were carried out manual summarization of document, using them as reference documents, and measuring the similarity between the automatic summary and reference document. Specifically, automatic summarization is performed through various techniques from full text, and comparison with reference document, which is an ideal summary document, is performed for measuring the quality of automatic summarization. Reference documents are provided in two major ways, the most common way is manual summarization, in which a person creates an ideal summary by hand. Since this method requires human intervention in the process of preparing the summary, it takes a lot of time and cost to write the summary, and there is a limitation that the evaluation result may be different depending on the subject of the summarizer. Therefore, in order to overcome these limitations, attempts have been made to measure the quality of summary documents without human intervention. On the other hand, as a representative attempt to overcome these limitations, a method has been recently devised to reduce the size of the full text and to measure the similarity of the reduced full text and the automatic summary. In this method, the more frequent term in the full text appears in the summary, the better the quality of the summary. However, since summarization essentially means minimizing a lot of content while minimizing content omissions, it is unreasonable to say that a "good summary" based on only frequency always means a "good summary" in its essential meaning. In order to overcome the limitations of this previous study of summarization evaluation, this study proposes an automatic quality evaluation for text summarization method based on the essential meaning of summarization. Specifically, the concept of succinctness is defined as an element indicating how few duplicated contents among the sentences of the summary, and completeness is defined as an element that indicating how few of the contents are not included in the summary. In this paper, we propose a method for automatic quality evaluation of text summarization based on the concepts of succinctness and completeness. In order to evaluate the practical applicability of the proposed methodology, 29,671 sentences were extracted from TripAdvisor 's hotel reviews, summarized the reviews by each hotel and presented the results of the experiments conducted on evaluation of the quality of summaries in accordance to the proposed methodology. It also provides a way to integrate the completeness and succinctness in the trade-off relationship into the F-Score, and propose a method to perform the optimal summarization by changing the threshold of the sentence similarity.

A Korean Text Summarization System Using Aggregate Similarity (도합유사도를 이용한 한국어 문서요약 시스템)

  • 김재훈;김준홍
    • Korean Journal of Cognitive Science
    • /
    • v.12 no.1_2
    • /
    • pp.35-42
    • /
    • 2001
  • In this paper. a document is represented as a weighted graph called a text relationship map. In the graph. a node represents a vector of nouns in a sentence, an edge completely connects other nodes. and a weight on the edge is a value of the similarity between two nodes. The similarity is based on the word overlap between the corresponding nodes. The importance of a node. called an aggregate similarity in this paper. is defined as the sum of weights on the links connecting it to other nodes on the map. In this paper. we present a Korean text summarization system using the aggregate similarity. To evaluate our system, we used two test collection, one collection (PAPER-InCon) consists of 100 papers in the field of computer science: the other collection (NEWS) is composed of 105 articles in the newspapers and had built by KOROlC. Under the compression rate of 20%. we achieved the recall of 46.6% (PAPER-InCon) and 30.5% (NEWS) and the precision of 76.9% (PAPER-InCon) and 42.3% (NEWS).

  • PDF

Automatic Document Summary Technique Using Fuzzy Theory (퍼지이론을 이용한 자동문서 요약 기술)

  • Lee, Sanghoon;Moon, Seung-Jin
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.3 no.12
    • /
    • pp.531-536
    • /
    • 2014
  • With the very large quantity of information available on the Internet, techniques for dealing with the abundance of documents have become increasingly necessary but the problem of processing information in the documents is still technically challenging and remains under study. Automatic document summary techniques have been considered as one of critical solutions for processing documents to retain the important points and to remove duplicated contents of the original documents. In this paper, we propose a document summarization technique that uses a fuzzy theory. Proposed summary technique solves the ambiguous problem of various features determining the importance of the sentence and the experiment result shows that the technique generates better results than other previous techniques.

Definition Sentences Recognition Based on Definition Centroid

  • Kim, Kweon-Yang
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.17 no.6
    • /
    • pp.813-818
    • /
    • 2007
  • This paper is concerned with the problem of recognizing definition sentences. Given a definition question like "Who is the person X?", we are to retrieve the definition sentences which capture descriptive information correspond variously to a person's age, occupation, of some role a person played in an event from the collection of news articles. In order to retrieve as many relevant sentences for the definition question as possible, we adopt a centroid based statistical approach which has been applied in summarization of multiple documents. To improve the precision and recall performance, the weight measure of centroid words is supplemented by using external knowledge resource such as Wikipedia and redundant candidate sentences are removed from candidate definitions. We see some improvements obtained by our approach over the baseline for 20 IT persons who have high document frequency.

Issues and Empirical Results for Improving Text Classification

  • Ko, Young-Joong;Seo, Jung-Yun
    • Journal of Computing Science and Engineering
    • /
    • v.5 no.2
    • /
    • pp.150-160
    • /
    • 2011
  • Automatic text classification has a long history and many studies have been conducted in this field. In particular, many machine learning algorithms and information retrieval techniques have been applied to text classification tasks. Even though much technical progress has been made in text classification, there is still room for improvement in text classification. In this paper, we will discuss remaining issues in improving text classification. In this paper, three improvement issues are presented including automatic training data generation, noisy data treatment and term weighting and indexing, and four actual studies and their empirical results for those issues are introduced. First, the semi-supervised learning technique is applied to text classification to efficiently create training data. For effective noisy data treatment, a noisy data reduction method and a robust text classifier from noisy data are developed as a solution. Finally, the term weighting and indexing technique is revised by reflecting the importance of sentences into term weight calculation using summarization techniques.

Sentence Summarization of News Articles (뉴스 기사의 문장 요약)

  • Choi, DongHyun;Shin, Ji-Ae;Choi, Key-Sun
    • Annual Conference on Human and Language Technology
    • /
    • 2007.10a
    • /
    • pp.269-275
    • /
    • 2007
  • 텔레비전 뉴스에서 부제목을 만들거나, 문장을 PDA나 휴대폰과 같은 작은 화면에 출력하고 싶은 경우, 가능한 방법은 두 가지가 있다. 첫번째는 사람에 의해 직접 만드는 방식이다. 두번째는 자동화된 문장 요약 시스템을 사용하는 방법이다. 따라서 문장 요약 알고리즘은 그 중요성이 계속해서 커지고 있다. 본 논문에서는 구문 트리의 서브 트리가 변화할 수 있는 규칙을 제시하는 방법에 (1)공기 정보와 (2) 문법적으로 올바른 구조를 유지하기 위해 핵심적인 부분(주요 문법 구조) 및 같이 요약되어야 할 절을 표시하는 휴리스틱, (3)주어진 문장이 포함된 글의 제목 정보를 추가로 사용하여 문장 요약을 실행하였다. 본 시스템의 결과와 기존의 요약 방식을 비교하는 실험을 분야 전공자들에 의한 주관적 평가로 수행한 결과, 본 시스템의 알고리즘이 기존에 사용되던 구문서브트리 변환 방법보다 중요한 부분 및 문법적으로 올바른 부분을 많이 유지하는 요약임을 확인하였다.

  • PDF

Product Review Summarization through Review Sentence Analysis (상품평 분석을 통한 상품 평가 요약 시스템)

  • Kim, Je-Sang;Jung, Gun-Young;Gwan, In-Ho;Lee, Hyun-Ah
    • Annual Conference on Human and Language Technology
    • /
    • 2013.10a
    • /
    • pp.113-115
    • /
    • 2013
  • 다수의 상품평 요약은 인터넷 쇼핑몰 고객에게 편의를 제공할 수 있다. 본 논문에서는 상품평 요약 시스템의 성능 향상을 위한 방안을 제안한다. 시스템은 크게 상품평의 평가 항목 추출과 극성 사전 생성, 극성 판별 단계로 구성된다. 평가 항목 추출에서는 외부 연관도의 영향력을 줄이고, 극성 사전 생성에서는 단어 거리 평균을 적용한다. 제안한 방식을 사용하였을 때 평가 항목에 대한 문장의 극성 판별 시 90.8%의 정확율을 보였다.

  • PDF

Query-Based Document Summarization using Important Sentence Selection Heuristics and MMR. (중요 문장추출 휴리스틱과 MMR을 이용한 질의기반 문서요약.)

  • Kim, Dong-Hyun;Lee, Seung-Woo;Lee, Gary Geun-Bae
    • Annual Conference on Human and Language Technology
    • /
    • 2002.10e
    • /
    • pp.285-291
    • /
    • 2002
  • 본 논문은 자연어 검색엔진에서의 검색결과에 대한 HIT LIST[6]와 검색 문서의 요약을 위하여 질의 기반의 3단계 문서요약을 제안한다. 첫째단계로 IR에 주어지는 질의를 유의어 DB를 통해 질의확장을 거친다. 둘째로 질의와 검색문서상의 문장의 유사도 계산을 통해 문장의 중요도 점수를 구한다. 좀더 정확한 요약을 위해 4가지 방법론을 적용하여 각 문장의 중요도를 ranking한다. 셋째로 MMR (Maximal Marginal Relevance)방식을 적용하여 요약 시 중복이 되는 부분을 줄인다. 이때 요약 압축률을 임의로 조절할 수 있다. 실험은 KORDIC의 신문기사로 구성된 문서요약 테스트 집합을 사용하여 좋은 요약결과를 얻었다.

  • PDF