• Proceedings of the IEEK Conference
• /
• 2001.06d
• /
• pp.29-32
• /
• 2001

Visual speech information improves the performance of speech recognition, especially in noisy environment. We have tested the various spatial-temporal features for the Korean lipreading and evaluated the performance by using a hidden Markov model based classifier. The results have shown that the direction as well as the magnitude of the movement of the lip contour over time is useful features for the lipreading.

• The Journal of the Acoustical Society of Korea
• /
• v.21 no.1
• /
• pp.19-26
• /
• 2002

This paper defines the visual basic speech units, visemes and investigates various visual features of a lip for the effective Korean lipreading. First, we analyzed the visual characteristics of the Korean vowels from the database of the lip image sequences obtained from the multi-speakers, thereby giving a definition of seven Korean vowel visemes. Various spatio-temporal features of a lip are extracted from the feature points located on both inner and outer lip contours of image sequences and their classification performances are evaluated by using a hidden Markov model based classifier for effective lipreading. The experimental results for recognizing the Korean visemes have demonstrated that the feature victor containing the information of inner and outer lip contours can be effectively applied to lipreading and also the direction and magnitude of the movement of a lip feature point over time is quite useful for Korean lipreading.

• Journal of the Korea Society of Computer and Information
• /
• v.11 no.2 s.40
• /
• pp.159-167
• /
• 2006

Lipreading has been suggested as one of the methods to improve the performance of speech recognition in noisy environment. However, existing methods are developed and implemented only in software. This paper suggests a hardware design for real-time lipreading. For real-time processing and feasible implementation, we decompose the lipreading system into three parts; image acquisition module, feature vector extraction module, and recognition module. Image acquisition module capture input image by using CMOS image sensor. The feature vector extraction module extracts feature vector from the input image by using parallel block matching algorithm. The parallel block matching algorithm is coded and simulated for FPGA circuit. Recognition module uses HMM based recognition algorithm. The recognition algorithm is coded and simulated by using DSP chip. The simulation results show that a real-time lipreading system can be implemented in hardware.

• Proceedings of the Korean Institute of Intelligent Systems Conference
• /
• 1993.06a
• /
• pp.903-906
• /
• 1993

Lipreading through visual processing techniques help provide some useful systems for the hearing impaired to learn communication assistance. This paper proposes a method to understand spoken words by using visual images taken by a camera with a video-digitizer. The image is processed to obtain the contours of lip, which is approximated into a hexagon. The pattern lists, consisting of lengths and angles of hexagon, are compared and computed to get the fuzzy similarity between two lists. By similarity matching, the mouth shape is recognized as the one which has the pronounced voice. Some experiments, exemplified by recognition of the Japanese vowels, are given to show feasibilities of this method.

• Speech Sciences
• /
• v.8 no.3
• /
• pp.123-132
• /
• 2001

In recent years research on HCI technology has been very active and speech recognition is being used as its typical method. Its recognition, however, is deteriorated with the increase of surrounding noise. To solve this problem, studies concerning the multimodal HCI are being briskly made. This paper describes automated lipreading for bimodal speech recognition on the basis of image- and speech information. It employs audio-visual DB containing 1,074 words from 70 voice and tri-viseme as a recognition unit, and state tied HMM as a recognition model. Performance of automated recognition of 22 to 1,000 words are evaluated to achieve word recognition of 60.5% in terms of 22word recognizer.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 1998.11a
• /
• pp.181-185
• /
• 1998

Recently, research and developmental direction of communication system is concurrent adopting voice data and face image in speaking to provide more higher recognition rate then in the case of only voice data. Therefore, we present a method of lipreading in speech image sequence by using the 3-D facial shape model. The method use a feature information of the face image such as the opening-level of lip, the movement of jaw, and the projection height of lip. At first, we adjust the 3-D face model to speeching face image sequence. Then, to get a feature information we compute variance quantity from adjusted 3-D shape model of image sequence and use the variance quality of the adjusted 3-D model as recognition parameters. We use the intensity inclination values which obtaining from the variance in 3-D feature points as the separation of recognition units from the sequential image. After then, we use discrete HMM algorithm at recognition process, depending on multiple observation sequence which considers the variance of 3-D feature point fully. As a result of recognition experiment with the 8 Korean vowels and 2 Korean consonants, we have about 80% of recognition rate for the plosives and vowels.

• The Journal of Korean Institute of Communications and Information Sciences
• /
• v.35 no.7C
• /
• pp.598-603
• /
• 2010

In this paper, we work for Lipreading using visual information for ship environment. Lipreading is studied for using image information including lips of a speaker at the existing speech recognition system. This technique is a compensation method to increase recognition rate decreasing remarkably in noisy circumstances. Proposed way improved the rate of recognition improving methode of preprocessing using the Gabor Filter for Ship Environment. The experiment were carried out under changing of light with time in the ship environment with lip image. For Comparing with recognition, make a compare with between method of lip region of interest (ROI) before Gabor filtering and after Gabor filtering. In the case of using method of lip ROI before Gabor filtering, the result of the experiments applying to the proposed ways recognition resulting in 44% of recognition.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2009.01a
• /
• pp.135-140
• /
• 2009

본 논문은 한국어 8 단모음을 인식하는 자동 독화 신경망 시스템을 구축한 것이다. 얼굴의 특정들은 휘도와 채도 성분으로 인하여 다양한 색 공간에서 다양한 표현 값을 갖는다. 이를 이용하여 각 표현 값들을 증폭하거나 축소, 대비시킴으로서 얼굴 특정들을 추출되게 하였다. 눈과 코, 안쪽 입의 외곽선, 이의 외곽선을 찾았고, 그 후 한국어 8모음 발화시 구분되게 변화는 값들을 파라미터로 설정하였다. 한국어 8모음을 발화하는 2400개의 자료를 모아 분석하고 이 분석을 바탕으로 신경망 시스템을 구축하여 실험하였다. 이 실험에 정상인 5명이 동원되었고, 사람들 사이에 있는 관찰 오차를 정규화를 통하여 수정하였다. 5명으로 분석하였고, 5명으로 인식 실험하여 좋은 결과를 얻었다.

• Journal of the Korea Computer Industry Society
• /
• v.8 no.4
• /
• pp.281-288
• /
• 2007

In this paper, we set 6 interesting points around lips. Analyzed and characterized is the distance change of these 6 interesting points when people pronounces 5 vowels of Korean language. 450 data are gathered and analyzed. Based on this analysis, the system is constructed and the recognition experiments are performed. In this system, we used the camera connected to computer to measure the distance vector between 6 interesting points. In the experiment, 80 normal persons were sampled. The observational error between samples was corrected using normalization method. We analyzed with 30 persons and experimented with 50 persons. We constructed three recognition systems and of those the neural net system gave the best recognition result of 87.44 %.

• Journal of the Korea Society of Computer and Information
• /
• v.14 no.4
• /
• pp.39-46
• /
• 2009

Many lip region detection methods have been developed in PC environment. But the existing methods are difficult to run on real-time in resource limited mobile devices. To solve the problem, this paper proposes a real-time lip region detection method for lipreading in Mobile device. It detects face region by using adaptive face color information. After that, it detects lip region by using geometrical relation between eyes and lips. The proposed method is implemented in a smart phone with Intel PXA 270 embedded processor and 386MB memory. Experimental results show that the proposed method runs at the speed 9.5 frame/see and the correct detection rate was 98.8% for 574 images.