Automatic Classification Algorithm for Raw Materials using Mean Shift Clustering and Stepwise Region Merging in Color

컬러 영상에서 평균 이동 클러스터링과 단계별 영역 병합을 이용한 자동 원료 분류 알고리즘

Abstract

In this paper, we propose a classification model by analyzing raw material images recorded using a color CCD camera to automatically classify good and defective agricultural products such as rice, coffee, and green tea, and raw materials. The current classifying agricultural products mainly depends on visual selection by skilled laborers. However, classification ability may drop owing to repeated labor for a long period of time. To resolve the problems of existing human dependant commercial products, we propose a vision based automatic raw material classification combining mean shift clustering and stepwise region merging algorithm. In this paper, the image is divided into N cluster regions by applying the mean-shift clustering algorithm to the foreground map image. Second, the representative regions among the N cluster regions are selected and stepwise region-merging method is applied to integrate similar cluster regions by comparing both color and positional proximity to neighboring regions. The merged raw material objects thereby are expressed in a 2D color distribution of RG, GB, and BR. Third, a threshold is used to detect good and defective products based on color distribution ellipse for merged material objects. From the results of carrying out an experiment with diverse raw material images using the proposed method, less artificial manipulation by the user is required compared to existing clustering and commercial methods, and classification accuracy on raw materials is improved.

본 논문에서는 카메라로부터 입력된 영상으로부터 쌀, 커피, 녹차 등 다양한 원료를 양품과 불량품으로 자동 분류하기 위한 분류 모델을 제안한다. 현재 농산물 원료 분류를 위해서 주로 숙달된 노동력의 육안 선택에 의존하고 있지만 작업시간이 길어질수록 반복적인 작업에 의해 분류 능력이 현저히 떨어지는 문제점이 있다. 노동력에 부분적으로 의존하는 기존 제품의 문제점을 해결하기 위해, 본 논문에서는 평균-이동 클러스터링 알고리즘과 단계별 영역 병합 알고리즘을 결합하는 비전기반 자동 원료 분류 알고리즘을 제안한다. 우선 입력 원료 영상에서 평균-이동 클러스터링 알고리즘을 적용하여 영상을 N개의 클러스터 영역으로 분할한다. 다음단계에서 N개의 클러스터 영역 중에서 대표 영역을 선택하고 이웃 영역들의 영역의 색상과 위치 근접성을 기반으로 단계별 영역 병합 알고리즘을 적용하여 유사한 클러스터 영역을 병합한다. 병합된 원료 객체는 RG, GB, BR의 2D 색상 분표로 표현되고, 병합된 원료 객체에 대해 색상 분포 타원을 만든다. 이후 미리 실험적으로 설정된 임계값을 적용하여 원료를 양품과 불량품을 구분한다. 다양한 원료 영상에 대해 본 논문에서 제안하는 알고리즘을 적용한 결과 기존의 클러스터링 알고리즘이나 상업용 분류 방법에 비해 사용자의 인위적 조작이 덜 필요하고 분류성능이 우수한 결과를 나타냄을 알 수 있었다.

Ⅰ. Introduction

As production of agricultural products and primary raw materials increases rapidly and the transactions between nations increase on a large scale, the necessity for automatically classifying defective raw materials is increasing. Such automatic classification of raw materials can be applied not only to agricultural products but also to diverse materials such as ores and waste materials. Therefore it is an essential procedure for the processing of good secondary commodities.

The method of classifying agricultural products mainly used at present is visual selection by skilled laborers. However, such a method requires skilled laborers with several years of experience, and classification ability may drop owing to repeated labor for a long period of time. To resolve such problems, a method of classifying raw materials using the brightness information of raw materials in images received from a mono CCD camera has been developed. Although this method is effective for grain products of a single color such as rice, wheat, or barley, it has the drawback of poor selection performance for raw materials of diverse colors such as coffee, green tea, ore, and waste materials.

To resolve this problem, trichromatic CCD or color CCD cameras have been used recently; these can enhance classification performance even for raw materials of diverse colors. Raw material classifiers commercialized until now use the colors of raw materials [1], and pictures of the raw materials are taken by a camera after the raw material is submitted, as shown in Fig. 1(a). Later, the entire raw material color distribution is displayed on the screen and an ellipse which falls under the threshold value is drawn like Fig. 1 (b), after which good and defective products are classified using an air pump depending on whether the test raw material has the color included in the relevant ellipse. However, this method has a problem: if the studied raw material itself contains diverse colors or impurities, misclassification increases when the test raw material is submitted as the range of the threshold values grows, eventually leading to a situation where the user must adjust the threshold value again.

Fig. 1.Example of commercialized raw material classifier [1] (a) Principle of existing CCD-camera-based raw material classifiers . (b) Error of a threshold setting by ellipse of the existing raw material classifiers. According to the ellipse threshold, good and defective products are classified using an air pump depending on whether the test raw material has the color included in the relevant ellipse 그림 1. 상업용 원료 분류기의 예 [1] (a) CCD 카메라 기반 원료 분류기의 원리, (b) 기존 원료 분류기의 타원에 의한 임계값 설정의 에러. 타원 임계값에 따라 테스트 원료가 타원 안에 포함되는지 여부에 따라 에어 펌프에 의해 양품과 불량품으로 분류됨

In this paper, we propose an algorithm that resolves the problem with existing color CCD-camera-based raw material classifiers and improves classification performance, as shown in Fig. 2, by expanding the existing raw material classification product [1] and the basic research [2]. The main contributions and the major steps of the proposed algorithm are as follows:

Fig. 2.Diagram of the overall algorithm of the proposed automatic raw material classifier 그림 2. 제안하는 자동 원료 분류기의 전체적인 알고리즘 다이어그램

 

Ⅱ. Raw Material Region Segmentation using Mean-Shift Clustering and Stepwise Region Merging

1. Image Background Removal

As pictures of raw materials are acquired with a fixed background as shown in Fig. 3(a), in order to distinguish the background and foreground, the color value of the input image on the first row is assumed to be the background color, and a foreground map of only the raw material is left by removing the background from the image on the basis of the estimated color (Fig. 3(b)). As the background of the raw material boundary is not perfectly removed in this process, the background pixels remaining on the edges of the raw material are removed by carrying out morphological erosion [4] based on the raw material part of the foreground map (Fig. 3(c)). For morphological operation, we use 5x5 sized kernel of rectangle shape.

Fig. 3.The background removal process: (a) raw material input image, (b) the foreground after background removal, (c) the image to which morphological erosion is applied 그림 3. 배경제거 처리 과정: (a) 원료 입력 영상, (b) 배경제거 후의 전경, (c) 모폴로지 침식을 적용한 후의 영상

2. Mean-Shift Clustering

In the second step, a mean-shift clustering algorithm [3] is applied to the image to which morphological erosion was applied. Clustering means dividing data into a random number of groups based on similarity by analyzing the position of the data distribution and other characteristics. The representative algorithms include K-means clustering [5] and mean-shift clustering.

While the K-means clustering algorithm exhibits fast classification performance, it has the shortcomings of requiring the number of clusters to be designated randomly by the user and of classification performance varying significantly depending on the initial value of the clusters. In contrast to the K-means clustering algorithm, the number of clusters in the mean-shift clustering algorithm does not need to be designated and the initial value of clusters is not required to be considered. Therefore, in this study mean-shift clustering is applied only to the foreground clusters of the background map generated earlier. Mean-shift clustering is a non-parametric clustering technology in which clustering is possible without any preliminary knowledge about the number of clusters. Moreover, as it is not influenced by the shape of the clusters, it is suitable for the division of less predictable images, e.g., of leukocyte cells of irregular shapes. Mean-shift clustering for image segmentation is the method of finding the local maxima of the probability density function that indicates the color distribution of image pixels in color space. First, among the color values in the given spatial kernel, the mean position of pixels that have a color similar to that of the center color of the current kernel and the mean value in the color space are calculated using Equation (1). In Equation (1), the mean value mh represents the mean-shift vector.

Here, xt represents the center of the kernel, x is one point in the kernel, h is the size of the kernel window, and g represents the kernel function.

Next, the same work is performed again after moving the kernel to a new point xt+1 using Equation (2) and the same work as above is repeated until the point converges to the point where the maximum value in the window kernel exists.

When the color value of this maximum point is changed to the pixel value of the current spatial position, image segmentation creates an effect whereby the color values in the spatial region are uniform. In this paper, to carry out a mean-shift suitable for each region, the size of the kernel for image space was set to 20, the kernel size in color space was 15, the minimum region size per cluster was 25, and the mean-shift was carried out in the RG color space. After the mean-shift clustering, the window region is designated based on a random starting pixel in the foreground map. The color of the starting pixel in this window region is defined as the base color value, and similar data are found. The mean coordinates of the data found becomes the new centroid, and the mean of the color becomes the new base color value. The above work is repeated again, shifting the window region to the new centroid. This calculation work is repeated until the centroid converges, and the base color value obtained last becomes the base color value of the starting pixel. The above work is repeated for all other pixels to obtain the base color value for each. Later, clusters can be designated based on the similarity between base color values.

3. Stepwise Region Merging

If the raw material has multiple colors, some objects may be divided into several clusters (regions) through clustering work. These multiple clusters can be merged into one close to the actual object by improving the region approach of Ko et al. [6]. As shown in Fig. 4(b), the largest region of each raw material is selected as the seed region after applying mean-shift clustering. Next, positional proximity and color similarity between the seed cluster and the surrounding clusters are measured for merging of each region (Fig. 4(c)).

Fig. 4.Stepwise region-merging steps: (a) image input, (b) mean-shift clustering and seed region selection, (c) region merging using positional proximity and color similarity, (d) final region-merging result. If positional proximity and color similarity are below their threshold values, the two clusters are merged and the merged cluster becomes the new seed cluster. The detailed merging process is described in Algorithm 1 그림 4. 단계별 영역병합 과정: (a) 입력 영상, (b) 평균 이동 클러스터링과 초기 영역 선정, (c) 위치근접성과 색상유사성을 이용한 영역병합, (d) 최종 병합 결과. 위치근접성과 색상유사성이 임계값 이하이면 두 군집은 병합되고 병합된 군집은 새로운 초기 군집이 됨. 자세한 병합과정에 대한 설명은 알고리즘 1을 참조

4. Color Distribution Conversion of Raw Material Object and Ellipse Construction Threshold Value

The color distribution of the raw material object is expressed in the 2D color spaces of RG, GB, and BR based on object data after the raw material objects are merged. An ellipse is generated in such a way that only the main components of the raw material can be included without including noise, etc., in each color distribution diagram expressed as such. The ellipse is used as the threshold value of the raw material classifier. First, the main components and the gradient of the object are determined by calculating the linear regression function from the color distribution diagram created for each raw material color. Additionally, the major and minor axis values required for the ellipse are derived by obtaining the width and height of the object (O). First, the centroid of the object is determined as shown in Fig. 5(a), and the tilt angle (θ) is calculated using the least moment of inertia for each data set.

where Cx and Cy are the centroid of an object and a pixel within an object, respectively, and (p, q) is the order of central moment C. The tilt angle is calculated; then the minimum rectangle enclosing the object is obtained. The bounding rectangle is the smallest rectangle enclosing the object that is also aligned with its orientation θ. Here, we calculate the bounding rectangle from the object boundary points (x,y) and orientation (θ) using Equation (5) [7].

where a and b are the standard points to estimate major and minor axises.

Fig. 5.Calculation of the ellipse threshold value by estimating the color distribution’s central axis, (a) calculation of the center value of the color distribution, (b) calculation of the major and minor axes of the color distribution, and (c) generation of the final color ellipse threshold value 그림 5. 색상분포의 중심축 추정을 통한 타원 임계값 계산, (a) 색상분포의 중심값 계산, (b) 색상분포의 최대최소축 계산, (c) 최종 색상 타원 임계값 생성

From each boundary point, we can search for amax and bmax . Finally, we can estimate the boundary rectangle immediately with length l1 = amax - amin and l2 = bmax - bmin . Between the two lengths, the longer is defined as the major axis (Rmax) and the other is defined as the minor axis (Rmin) (Fig. 5 (b)).

The ellipse threshold value enclosing the relevant object can be generated by color distribution as shown in Fig. 5(c) using the gradient, major axis value, minor axis value, and data center value.

 

Ⅲ. Experiment and Performance Evaluation

In this paper, a classification performance experiment was executed on rice, coffee, and plastic raw materials, and a comparative experiment was carried out with the K-means clustering algorithm [4] and the commercial algorithm of Daewon GSI Co., Ltd. [1].

For evaluation, the ellipse region obtained by each method is compared with the region of interest determined by a human to evaluate the rate of error between the two regions. As there is no specific method for evaluating the performance of ellipse estimation, the evaluation method as shown in Equations (6) to (8) proposed by [8][9] was modified.

where SM and SR represent the cardinality of M and R, respectively, M represents the set of pixels in the manually estimated ellipse, R represents the set of pixels in the estimated ellipse by proposed algorithm, SU and SO represent the inaccuracy of under-extraction and over-extraction, respectively, and AVG represents the accuracy between the manually estimated ellipse and the systematically estimated ellipse. For the experiment, rice and coffee products used most widely for classification of agricultural products and plastic materials used most widely for classification of waste materials were classified. The experimental tests consisted of 14 color raw material images collected from the commercial product of [1], including 62 rices, 14 coffees, and 11 plastics.

Fig. 6.Evaluation of classification rate performance for three raw materials (plastic, coffee, and rice) 그림 6. 세 가지 원료 (플라스틱, 커피, 쌀)에 대한 분류율 성능 평가

As the performance rate on average of the proposed algorithm was about 84.9 % while that of K-means clustering [4] was 3.47 % and that of the commercial algorithm [1] was 41.4 %, it is clear that the performance of the proposed algorithm is superior to that of the other two algorithms. In particular, the classification rate of K-means clustering was the lowest, which is thought to be because the initial center point value of the K-means clustering algorithm is selected randomly and because the algorithm is sensitive to noise in the clustering process. It can be seen that, as the plastic used for the experiment included diverse colors, performance was lower than for other raw materials by 18.5 % on average. It can be seen that, as for overall performance, the classification rate of the proposed method was improved compared to that of the existing commercial products by about 40 % because it segments raw materials into objects and sets the threshold value on the basis of the color distribution model for the segmented objects.

Figures 7 and 8 show the screen for the result where coffee or rice, respectively, are segmented automatically and converted into color distributions objectified using the proposed method and where the ellipse threshold value is set at the same time.

Fig. 7.Automatic segmentation of coffee material using the proposed method and setup for setting ellipse threshold value by color 그림 7. 제안하는 방법과 색상에 의한 타원 임계값 설정을 이용한 커피 원료의 자동 분할

Fig. 8.Automatic segmentation of rice material using the proposed method and setup for setting the ellipse threshold value by color 그림 8. 제안하는 방법과 색상에 의한 타원 임계값 설정을 이용한 쌀 원료의 자동 분할

 

IV. Conclusion

In this paper, we proposed an algorithm that can effectively differentiate good products out of grain, ore, or other raw materials using RGB data from color images. The existing method sets the color threshold value using K-means clustering or other methods, but after mapping raw material to color spaces, such methods encounter the problem of requiring a user to correct the threshold region because, since existing methods are sensitive to noise, an accurate ellipse cannot be generated.

In this paper, to resolve this problem we proposed an algorithm whereby regions are generated in RGB space, applying the mean-shift clustering algorithm without mapping the raw material image directly to the color space, and regions are merged into significant objects using the stepwise merging algorithm. An algorithm that is robust in the presence of noise and requires no additional work of the user was developed by mapping the result of merged objects to each color space in order to estimate the threshold value ellipse.

In future studies, we intend to widen the scope of application to the classification of ores in addition to that of grains or waste materials, and to design a system with a fast classification rate using smaller memory by improving classification speed.