• The KIPS Transactions:PartB
• /
• v.13B no.2 s.105
• /
• pp.155-162
• /
• 2006

The medical thesaurus, MeSH (Medical Subject Heading), has been used as a controlled vocabulary thesaurus for English medical paper indexing for a long time. In this paper, we propose an automatic cross language keyword assignment method, which assigns English MeSH index terms to the abstract of a Korean medical paper. We compare the performance with the indexing performance of human indexers and the authors. The procedure of index term assignment is that first extracting Korean MeSH terms from text, changing these terms into the corresponding English MeSH terms, and calculating the importance of the terms to find the highest rank terms as the keywords. For the process, an effective method to solve spacing variants problem is proposed. Experiment showed that the method solved the spacing variant problem and reduced the thesaurus space by about 42%. And the experiment also showed that the performance of automatic keyword assignment is much less than that of human indexers but is as good as that of authors.

• The Journal of the Acoustical Society of Korea
• /
• v.21 no.3
• /
• pp.274-284
• /
• 2002

Word segmentation errors occurring in text preprocessing often insert incorrect words into recognition vocabulary and cause poor language models for Korean large vocabulary continuous speech recognition. We propose an automatic word segmentation algorithm using Markov chains and syllable-based n-gram language models in order to correct word segmentation error in teat corpora. We assume that a sentence is generated from a Markov chain. Spaces and non-space characters are generated on self-transitions and other transitions of the Markov chain, respectively Then word segmentation of the sentence is obtained by finding the maximum likelihood path using syllable n-gram scores. In experimental results, the algorithm showed 91.58% word accuracy and 96.69% syllable accuracy for word segmentation of 254 sentence newspaper columns without any spaces. The algorithm improved the word accuracy from 91.00% to 96.27% for word segmentation correction at line breaks and yielded the decomposition accuracy of 96.22% for compound-noun decomposition.

• Annual Conference on Human and Language Technology
• /
• 2003.10d
• /
• pp.47-53
• /
• 2003

본 논문은 정제된 대용량 말뭉치로부터 얻은 음절 n-gram과 어절 통계를 이용한 한국어 자동 띄어쓰기 시스템을 제안한다. 한 문장 내에서 최적의 띄어쓰기 위치는 Viterbi 알고리즘에 의해 결정된다. 통계 기반 연구에 고유한 문제인 데이터 부족 문제, 학습 말뭉치 의존 문제를 개선하기 위하여 말뭉치를 확장하고 실험을 통해 얻은 매개변수를 사용하고 최장 일치 Viable Prefix를 찾아 어절 목록에 추가한다. 본 연구에 사용된 학습 말뭉치는 33,641,511어절로 구성되어 있으며 구어와 문어를 두루 포함한다.

• KIPS Transactions on Software and Data Engineering
• /
• v.3 no.7
• /
• pp.271-276
• /
• 2014

Short message service(SMS) in a mobile communication environment is a very convenient method. However, it caused a serious side effect of generating spam messages for advertisement. Those who send spam messages distort or deform SMS sentences to avoid the messages being filtered by automatic filtering system. In order to increase the performance of spam filtering system, we need to recover the distorted sentences into normal sentences. This paper proposes a method of normalizing the various types of distorted sentence and extracting keywords through automatic word spacing and compound noun decomposition.

• Proceedings of the Korean Information Science Society Conference
• /
• 2007.06a
• /
• pp.59-60
• /
• 2007

휴대폰과 메신저 등 통신 환경에서 사용되는 표준어가 아닌 SMS의 변형된 어휘 및 띄어쓰기 오류를 자동으로 교정하여 형태소 분석 및 품사 태깅의 성능 저하 문제를 방지하는 문자열 오류의 교정 방법을 제안하였다. 통신 어휘들의 문자열 사전 구축 방법으로 통신어휘집을 기반으로 수동으로 구축하는 방법과 수작업으로 구축된 말뭉치로부터 자동으로 변형된 문자열을 추출하는 방법, 그리고 문맥을 고려하는 방법을 비교-분석하고 실험 및 성능 평가 결과를 제시하였다.

• Proceedings of the Korean Information Science Society Conference
• /
• 2003.04c
• /
• pp.528-530
• /
• 2003

우리말에서 띄어쓰기와 버금갈 정도로 어려운 것이 우리말의 발음이다. 이에 실생활에서 혼란스럽게 사용되는 발음법과 그로 인해 올바른 발음의 선택에 대한 어려움을 덜어낼 수 있도록 표준어 규정의 표준 발음법에 따른 한국어 음소열 자동 생성기를 구현하여 교육용으로 쓸 수 있도록 하는 것이 이 논문의 목적이다.

• Speech Sciences
• /
• v.8 no.2
• /
• pp.83-90
• /
• 2001

We proposed a probabilistic approach of using syllable bigrams to the word-spacing problem. Syllable bigrams are extracted and the frequencies are calculated for the large corpus of 12 million words. Based on the syllable bigrams, we performed three experiments: (1) automatic word-spacing, (2) detection and correction of word-spacing errors for spelling checker, and (3) automatic insertion of a space at the end of line in the character recognition system. Experimental results show that the accuracy ratios are 97.7 percent, 82.1 percent, and 90.5%, respectively.

• The Journal of Korean Association of Computer Education
• /
• v.12 no.2
• /
• pp.87-96
• /
• 2009

We propose a Korean spacing-error correction system that requires small memory usage although the proposed method is a mixture of rule-based and statistical methods. In addition, to train the proposed model to be robust in mobile colloquial sentences in which spelling errors and omissions of functional words are frequently occurred, we propose a method to automatically transform typical colloquial corpus to mobile colloquial corpus. The proposed system uses statistical information of syllable uni-grams in order to increase coverages on new syllable patterns. Then, the proposed system uses error correction rules of two or more grams of syllables in order to increase accuracies. In the experiments on fake mobile colloquial sentences, the proposed system showed relatively high accuracy of 92.10% (93.80% in typical colloquial corpus, 94.07% in typical balanced corpus) spite of small memory usage of about 1MB.

• Journal of Korean language education
• /
• v.23 no.1
• /
• pp.77-100
• /
• 2012

This study investigates issues in relation to text segmenting, in particular, line breaks in Korean language textbooks. Research on L1 and L2 reading has shown that readers process texts by chunking (grouping words into phrases or meaningful syntactic units) and, therefore, phrase-cued texts are helpful for readers whose syntactic knowledge has not yet been fully developed. In other words, it would be important for language textbooks to avoid awkward syntactic divisions at the end of a line, in particular, those textbooks for beginners and intermediate level learners. According to our analysis of a number of major Korean language textbooks for beginner-level learners, however, many textbooks were found to display line-breaks of awkward syntactic division. Moreover, some textbooks displayed frequent instances where a single word (or eojeol in the case of Korean) is split between different lines. This can hamper not only learners' learning of the rules of spaces between eojeols in Korean, but also learners' development in automatic word recognition, which is an essential part of reading processes. Based on the findings of our textbook analysis and of existing research on reading, this study suggests ways to overcome awkward line-breaks in Korean language textbooks.

• Proceedings of the Korean Society for Cognitive Science Conference
• /
• 2000.06a
• /
• pp.216-223
• /
• 2000

웹 문서는 일반 문서들과 달리 자유로운 형식으로 기술되어 있고, 원문에 태그나 코드 등 불필요한 내용들을 많이 포함하고 있어 언어 처리에 바로 사용하기에 적합하지 못하다. 본 논문은 인덱싱 대상 문서로 사용되는 웹 문서를 자동으로 수집하여, 문장 단위로 정렬된 문서로 제작, 관리하는 통합 전처리 시스템인 Web Tagger의 구조와 전처리 방법을 소개한다. Web Tagger는 문서 정제, 문장 분할, 띄어쓰기의 과정을 거쳐 웹 문서에서 표준화된 정보를 추출하고, 형태소 분석기를 포함한 응용 시스템의 목적에 맞게 XML형식의 원문 코퍼스를 자동으로 생성하고 관리한다. '정규문법(Regexp)', '휴리스틱', '품사 인덱스 참조', 'C4.5를 사용한 학습 규칙' 등의 다양한 전처리 기법은 형태소 분석 정확도 향상과 시스템 안정성 보장에 기여한다.