Medical Diagnosis Algorithm Based on Tongue Image on Mobile Device

Abstract

In traditional Chinese medical (TCM) science, tongue images can be observed for medical diagnosis; however, the tongue diagnosis of TCM is influenced by the subjective factors of doctors, and the diagnosis results vary from person to person. Quantitative TCM tongue diagnosis can improve the accuracy of diagnosis and increase the application value. In this paper, digital image processing and pattern recognition technologies are employed on mobile device to classify tongue images collected in different health states. First, through grayscale integral projection processing, the trough is found to localize the tongue body. Then the tongue body image is transferred from RGB color space to HSV color space, and the average H and S values are considered as the color features. Finally, the diagnosis results are obtained according to the relationship between the color characteristics and physical symptoms.

I. INTRODUCTION

Tongue diagnosis is one of the most widely used diagnostic methods in traditional Chinese medicine [1]. The benefit of a tongue diagnosis is that it is simple and straightforward: people who need a health check can quickly determine their pathology through a regular tongue diagnosis.

 

1.1 Tongue segmentation algorithm

At present, the category of the tongue segmentationalgorithm mainly include color model transformation, active contour model, watershed algorithm, neural network and so on.

It is the easiest way to think of the separation of the tongue and other parts by color [2]. [3] observed the H channel data distribution characteristics of the image, and used it as a key segmentation factor, combined with the Ichannel data to segment. The author judges the largest part of the Unicom area in the segmentation result as the tongue, and finally corrects the final segmentation result by the morphological algorithm. Under the technical conditions at that time, it was not possible to accurately distinguish the color of the tongue, so the color of the middle part of the mechanical tongue was taken as the color of the tongue coating, and the color of the tongue took the color of theedge part of the tongue. At the same time, due to the difference in hardware facilities and methods, the results of tongue color recognition are also different. Because the color model conversion method based on color difference is not ideal, researchers have proposed a new active contourmodel algorithm [4]. Ref. [5] converted the original image from the initial coordinate system to the polar coordinatesystem, performed boundary enhancement and boundary extraction, and binarizing the result to the initial boundary of the contour. In [6], authors used the double-snake energy function algorithm for tongue segmentation, which improved the accuracy by nearly 10% compared to the ordinary snake algorithm. The authors of [7] used the C2G2FSnake algorithm for tongue segmentation, whichincreases the curve rate and reduces the complexity of the algorithm. In [8], athors used the maximum inter-class variance method to binarize the original tongue image, and then use the mathematical morphology method to modify the boundary of the binary image. The algorithm combinesthe maximum inter-class variance and mathematical morphology. Segmentation accuracy. It can be seen that the active contour model [9] has attracted the attention of researchers because of the high accuracy of segmentation results and the smooth contour curve. The watershed algorithm usually brings the problem of excessivesegmentation. The researchers have proposed some improvement measures for this problem. Marker control is a commonly used improvement method.

The watershed algorithm usually brings the problem of excessive segmentation. The researchers have proposed some improvement measures for this problem. Markercontrol is a commonly used improvement method. [10] proposed an image segmentation algorithm based on fast two-step mark control. The tongue segmentation technique of the color model conversion method has graduallymatured. [11] proposed to first find an initial object region, by transforming and thresholding the morphological components of the image in the HSI color space and morphological operations, and then image clustering the RGB components of the initial object region to find the rootof the tongue. The gap region between the upper lip and the upper lip is finally removed by means of the gap region toremove the false tongue region such as the upper lip, and the tongue is extracted from the initial target region. The tongue segmentation algorithm is summarized as shown in Table 1.

 

Table 1. Tongue segmentation threshold table.

 

2. Tongue feature extraction algorithm

In recent years, there have been few studies on tongue image feature extraction algorithms, and their research focuses on how to comprehensively and effectively acquiredisease-related features.

[12] tried to use wavelet transform to extract the colorand texture features of different parts of the tongue image, and statistical analysis of these feature data, and finally toclassify the healthy tongue image and the diseased tongue image. The authors of [13] believe that most tongue featureclassification methods do not take into account regionalinformation. Therefore, the study uses a color-texturesegmentation algorithm to obtain a series of homogeneous regions, and classifies these regions according to the Earth Mover distance, and finally performs corresponding featureanalysis. [14] studied different methods to obtain color and texture feature information in the tongue image, and based on this information, the appendicitis disease diagnosis experiment, the experimental results have higher accuracy. [15] classified the shape of the tongue based on the geometric characteristics of the tongue. The study corrected the skew of the tongue with three geometric criteria and classified the shape of the tongue with seven geometric features. In order to obtain better classification results, theresearch used the analytic hierarchy process to increase the weight of relevant factors and used the fuzzy fusion framework to express the certainty and accuracy between these factors and the tongue category.

At the same time, in order to avoid the subjective and qualitative problems of traditional tongue diagnosis, several computer-aided diagnosis systems have been proposed [16]. For example, [17] proposed a Bayesian network-based system capable of identifying five different diseases withan accuracy of approximately 75%. [18] used the modelestablished by Bayesian decision to map the tongue imagefeatures of the sample with the existing classification results and identified the color of the tongue, which played a certain role in the clinical tongue diagnosis. [19] proposed a hybrid image segmentation algorithm combining region-based methods with boundary-based automaticclassification methods.

At present, mobile phone penetration rate in somecountries is almost 100%. Most people carry mobile phones with them. The built-in sensors of smart phones (such asmagnetometers, accelerometers, cameras) enable the development of new sensor systems to measure the status of telephone users and his surrounding environment, so the study of a person who monitors his health through a tonguediagnosis of a smartphone [20] is beginning to rise. The concept of computerized tongue diagnosis is not new, but the research on tongue diagnosis based on mobile phones is relatively rare. Much of the work in this area has been based on the assumption that the tongue image comes from a well-controlled environment and can only be used by Chinesemedicine practitioners. [21] developed an Android-based automatic tongue diagnosis application based on Canny algorithm. However, due to various lighting conditions, they do not discuss how to calibrate the image color.

 

II. TONGUE IMAGE DIAGNOSIS ALGORITHM

 

1. Tongue image segmentation

 

1.1. Image graying

In order to diagnose the color of the tongue image, it is very important to extract the image of the tongue position. Most of the tongue segmentation algorithm first converts the image into gray image. The gray-scale integral projection algorithm adopted in this paper also needs to bedivided first. The image is converted to a grayscale image. The use of the simple and color singular representation of the gray image can better segment the region. In this paper, the weighted average method is used to obtain the gray image, which is used to prepare the segmented tongue image. There are several common grayscale processing methods (R, G, and B represent the three primary colors, namely, red, green, and blue):

Maximum method:

GRAY = max(R,G,B)       (1)

Component method:

GRAY = R or G or B       (2)

Average method:

GRAY = R+G+B       (3)

Weighted average method

GRAY = 0.299×R + 0.578×G + 0.114×B       (4)

 

1.2. Grayscale integral projection

In a tongue image, at the junction of two objects, the gray value changes greatly, and the transition of the gray value is prone to occur. According to the distributioncharacteristics of the color of the tongue image, the tongue and the non-region have relatively obvious boundaries and the body area of tongue is a whole in a block shape. According to these characteristics of the image, we canchoose the gray integral projection algorithm to determinethe position of the tongue image. The gray integral projection divides the region by the gray integral value in acertain direction and selects the divided region according to the condition. In this paper, the gray level is integrated into the horizontal and vertical directions of the image todetermine the position of the tongue.

\(\mathrm{GS}(\mathrm{x})=\int_{\& \mathrm{y} 1}^{\& \mathrm{y} 2} \mathrm{GRAY}(\mathrm{x}, \mathrm{y}) \mathrm{dy}\)       (5)

\(\mathrm{GS}(\mathrm{y})=\int_{\mathcal{E} \times 1}^{\& \mathrm{X} 2} \mathrm{GRAY}(\mathrm{x}, \mathrm{y}) \mathrm{d} \mathrm{x}\)       (6)

GS(x) represents horizontal gray-scale integration, GS(y) represents vertical integration, and GRAY(x,y) represents gray (x,y) coordinates in [x1,x2],[y1,y2] images. Afterobtaining the gray level integral projection of the horizontal and vertical directions of the tongue image, respectively, the distribution curve of the gray level integral in the horizontal and vertical directions will be drawn, and the distribution will show many peaks and troughs, but we know that in the tongue In the image, the tongue and non-tongue regions have different brightness values. Due to the existence of the edge of the tongue, troughs will begenerated, and the position of the trough can be found todetermine the region of the tongue.

After obtaining the gray integral projection value, the projection function curve is fitted with smooth lines. As shown in Fig 1, the distribution and difference of the gray value of the image can be clearly seen, and then the appropriate interval is set to find the minimum value of the gray integral. (This article limits the distance between twoadjacent minimum points to be no less than 20 pixels).

 

Fig. 1. Grayscale integral projection result. (a) source image, (b) Vertical gray level integration, (c) Horizontal gray levelintegration.

\(|\mathrm{MIN}(\mathrm{i})-\mathrm{MIN}(\mathrm{i}+1)| \geq 20 \mathrm{px}\)       (7)

where MIN(i) represents the i minimum value, and pxrepresents that the pixel unit obtains the minimum valuepoint, and the two extreme points in the horizontal direction and the vertical direction are respectively selected as the boundary of the tongue to determine the position of the tongue. The red "*" point in the figure indicates: horizontalintegral projection or vertical integral projection trough position tongue coating color classification.

 

2.1 Tongue classification

 

2.1.1 RGB and HSV color space

Color information is an important component in image processing technology. In many image processing algorithms, such as image segmentation algorithm, edge detection algorithm, and algorithm implementation are based on accurate color information. Nowadays, there aremany color spaces, such as RGB space, HSI space, HSV space, etc., which can be used to accurately represent a color image. Different color spaces determine their different research roles. Facing an actual image processing problem, how to It is especially important to choose theright color space. This article will introduce the RGB colorspace and the HSV color space.

The RGB color space is based on the three primary colors of human vision - red (R), green (G), and blue (B). The color space is considered to be red (R), green (G), and blue (B). Proper mixing can cause perception of all colors in thespectrum. In the RGB color space, each image is composed of one pixel, and the color of each pixel is composed of three components of R, G, and B, which together constitute the color feature of the image. In the tongue image, the R, G, and B values of various tongue colors are not much different, and they do not show certain regularity. In order to better classify the tongue image according to the tongue color, it is necessary to select a more suitable color space, and RGB. The space is converted to other color spaces. This article uses the HSV color space.

The HSV color space is based on human visual perception. In the HSV color space, three components areno longer used to represent the image color, only the H component is used, and the S component represents the depth of the color, called saturation. The S component and the H component play an important role in the colorclassification of the tongue image. The luminance (V) component indicates the degree of lightness and darkness of the color, and the range is [0, 1]. It is these characteristics of the HSV color space that are more suitable for the classification of the color features of the tongue image thanthe RGB color space. This paper mainly uses the mean value of the H and S components of the HSV color space todetermine the color classification threshold.

 

2.1.2. RGB and HSV space conversion

After experimenting and analyzing the tongue image indifferent color spaces, it is found that the HSV color space is more suitable for the color classification of the tongue image. After segmenting the image of the tongue position, the space of the segmented tongue is converted and converted. The formula is as in formula (2.8). The range of R, G, and B is: [0-255], max represents the maximum of the three values of R, G, and B, and min represents theminimum of the three values of R, G, and B.

\(\left\{\begin{array}{ll} {60^{*}(\mathrm{G}-\mathrm{B}) /(\mathrm{SV})} & {\mathrm{S} \neq 0 \text { and } \max =\mathrm{R}} \\ {\mathrm{H}=\left\{60^{*}(2+(\mathrm{B}-\mathrm{R}) /(\mathrm{SV}))\right.} & {\mathrm{S} \neq 0 \text { and } \max =\mathrm{G}} \\ {60^{*}(4+(\mathrm{B}-\mathrm{R}) /(\mathrm{SV}))} & {\mathrm{S} \neq 0 \text { and } \max =\mathrm{B}} \\ {\mathrm{S}=} & {\max } \\ {\mathrm{V}=(\max -\min ) / \max } \end{array}\right.\)       (8)

 

2.1.3. Threshold classification

Common tongue colors are pale white, red, purple, black, etc. The common tongue coating color is divided intoyellow, white, gray and so on. For these color types, this paper divides the color detection of tongue image into fivecategories: light red tongue, white moss, yellow moss, red tongue and purple black tongue. After the experimentalanalysis of the collected tongue image, under the HSV colorspace, according to H The mean value of S determines the classification threshold of the five types of tongue images. After converting the RGB color space to the HSV space, separate the chromaticity (H), saturation (S), and brightness (V), and only use H to represent the color, and remove thered, green, and blue colors in the RGB color space. Correlation is a good guarantee for the classification of tongue image color. Under the HSV color space, the average values of H and S are obtained respectively, H̅and S̅. Under the HSV space, the H̅ values of the tongue images differ greatly and have a certain regularity, that is, from the red tongue. , yellow moss, purple black tongue, white moss and red tongue, H̅ value increased in turn, but the white moss and red tongue are basically the same. However, their H̅ values are larger than those of othercategories, and their S̅ value is observed. The S̅ value of the reddish tongue is greater than 0.20, while the S̅value of white moss is less than 0.20. For these two points, we can well classify the tongue image into the fivecategories mentioned above according to the two values of H̅ and S̅.

The tongue image segmentation is performed by taking the tongue image sample obtained on the Internet and the tongue image sample obtained by taking photos of itself, and then calculating the H̅ and S̅ values in the HSV color space to determine the color classification of the tongue image as shown in Table 2. The range of H̅ in Table 2 is [0, 255], the unit is degree, and the range of S̅is [0, 1].

 

Table 2. Tongue segmentation threshold table.

 

III. EXPERIMENTAL RESULTS AND ANALYSIS

By downloading the already-collected tongue image and the mobile device to start the tongue diagnosis system to obtain the tongue image for detection, the JPG format picture is used for testing, and the detection result is recorded. Good results have been achieved in both tonguesegmentation and tongue color diagnosis.

 

3.1. Grayscale image of tongue

Before performing the segmentation of the tongue, the tongue image should be processed to convert the image into a grayscale image. Fig 2 shows the grayscale result of the experimental tongue image.

 

Fig. 2. Tongue image grayscale result.

 

3.2. Tongue segmentation

The gray level integral projection method is used todetermine the positional condition of the tongue is slightly harsh, and the photographing light is required to be good, and the collected tongue image basically does not containother objects. Under such conditions, a better segmentationeffect can be obtained. As shown in Fig 3 below, the image inside the white rectangle is the position of the tongue determined by the gray integral projection algorithm, which can accurately locate the position of the tongue, but the lack of part of the non-tongue part in the white rectangle is To acertain extent, it affects the color detection behind.

 

Fig. 3. Tongue segmentation result image.

 

3.3. Tongue image color diagnosis

After the position of the tongue is determined, the average of R, G, and B is separately classified into the RGB space of the segmented tongue image to classify the colorof the tongue image, and the same type of tongue image is found, and the R, G, and B values are found. There is noconvergence to a certain interval, and there is no regularity, as shown in Table 3. In terms of R-means, only the red tongue value is more prominent than the value of any category, while in other categories, the mean values of Gand B do not show any regularity. In the RGB color space, it is basically impossible to find a suitable classificationstandard, which is seriously affected by the high correlation of its color components.

Then, the cut tongue image is converted into the HSV space, and the collected sample images are color-classified according to the H̅ and S̅ thresholds of the tongue-segment threshold table in Table 4, which can better detect the red tongue, yellow tongue, white tongue, purple blacktongue and red tongue five categories. 48 tongue images were tested, 41 of which were correctly diagnosed, and the diagnostic rate was over 85.4%.

 

Table 3. Different RGB image range distribution.

 

Table 4. HSV color space recognition.

 

Ⅳ. CONCLUSIONS

Through the experimental results, it is found that the accurate recognition of the color of the tongue image is based on the accurate determination of the position of the tongue. If the tongue image is not correctly segmented, the

H̅ and S̅  under the HSV color space are used to judgethe color. Significant. The gray-scale integral projectionalgorithm used in this paper does not accurately determinethe position of the tongue. The four corners of the segmented rectangular tongue image still contain non-tongue images. Although it is a small amount, it still affects the back. Color detection. In order to improve the accuracy of the segmentation of the tongue, it is necessary to achieveaccurate segmentation of the tongue after the approximatedetermination of the position of the tongue. At the sametime, in the system design, the image should be roughly judged before the diagnosis of the tongue image. The image of the tongue object is not detected.

The detection of H̅ and S̅ values in the HSV colorspace is very good in the five categories of red tongue, yellow moss, white moss, purple black tongue and red tongue. However, this paper only divides the color into fivecategories, and the later research It can be considered that the color category is added based on this classification, and the category is subdivided. Different types of diseases indifferent positions of the tongue are a very good research direction for image feature extraction of different parts of the tongue. When the classification threshold is determined, many sample images are downloaded from the website, and their collection environments are not the same, which has agreater impact on the classification threshold determination. At the same time, the sample base is too small when the threshold is determined. The sample collected in the sameenvironment should be used to correct the colorclassification threshold of the tongue image. At the sametime, the sample size required is as large as possible. Theaccurate classification threshold is the guarantee of thereliability of tongue diagnosis.