Chessboard and Pieces Detection for Janggi Chess Playing Robot

Abstract

Vision system is an indispensable part of constructing the chess-playing robot. Chessboard detection and pieces localization in the captured image of robot's camera are important steps for processes followed such as pieces recognition, move calculation, and robot controlling. We present a method for detecting the Janggi chessboard and pieces based on the edge and color feature. Hough transform combined with line extraction is used for segmenting the chessboard and warping it to form the rectangle shape in order to detect and interpolate the lines of chessboard. Then we detect the existence of pieces and their side by applying the saliency map and checking the color distribution at piece locations. While other methods either work only with the empty chessboard or do not care about the piece existence, our method could detect sufficiently side and position of pieces as well as lines of the chessboard even if the occlusion happens.

1. INTRODUCTION

As technology develops rapidly in present day, robot systems are gradually being applied into every aspect of normal life. Among those, robot applications for playing physical chess currently attract a large amount of research interests. The system of chess playing robots usually consists of four main modules: image processing module for detecting the board and localizing pieces, pattern recognition module for recognizing the piece type, artificial intelligence module for deciding the chess moves, and controlling module for handling movements of robot's arm.

Our work focuses on detecting the Janggi chessboard and pieces which supports for modules followed. Although there are many researches in board detection, they all either work with western chessboard or do not consider the pieces. With Janggi and Chinese chessboards, which are mainly drawn with lines, the situation is quite different. We could divide the methods of board detection into two categories: corner-based and line-based detection. The first class of researches [1]-[5] tries to find the corners of squares by using algorithms like Harris or SUSAN corner detection with post processing steps to remove fake corners. As mentioned by Tam et al. [6], the advantage of these approaches is that it has high tolerance against camera distortion. However, in case of Janggi chess game in which the corners are occluded by pieces, these methods are not appropriate. The second class of researches [6]-[8] focuses on detecting lines instead of corners. These approaches are inaccurate with image distortion. One solution for this problem is to apply a pre-processing step to undistort the image beforehand. Line-based methods could accept small occlusion. However, in case the line is fully filled with pieces, we could have missing lines. Tam et al. [6] proposed a method for calculating corner points of the western chessboard. From the intersection points of detected lines, they try to predict other corner points of the chessboard by using geometric calculations. The problem is that because we do not know exactly the area of the chessboard in the image, we could not verify whether predicted points are the right corners or not. With pieces detection, a simple approach is to apply image subtraction [1]. The empty chessboard image will be subtracted from the image having both pieces and board. However, this method requires the stability of camera and chessboard which is hard to apply because of the illumination changes, the movability of the robot, and chessboard displacements.

In this paper, we propose a method for detecting the Janggi chessboard and pieces by making use of the edge and color features. Lines of the chessboard are located in the image with line filtering and Hough transform. The chessboard is then segmented and warped to rectangular shape in which we could easily reconstruct the missing lines of chessboard. Supposing that the pieces always locate at intersection points, as soon as the grid is formed, we could apply the saliency map to find out the pieces locations. Finally, we analyse the color distribution at image areas of pieces and divide them into two groups. Fig. 1 illustrates the overview of our detection work.

Our contribution consists of four main parts: (1) a new framework for detecting the location and side of pieces, (2) a novel line filtering and detection method based on the structure of inner lines and the distribution of local maximum points in Hough space that could extract efficiently the four borders and inner lines of chessboard, (3) a line reconstruction strategy for occlusion cases and (4) a side and location detection method using the information of lines and pieces color distribution.

The paper is organized as follows. In section 2, we present methods used for lines detection and board segmentation. Section 3 describes our approach for lines reconstruction. Location and side of pieces are found out in Section 4. Section 5 shows our experiment results compared with other board detection methods. Section 6 draws out the conclusion and our future researches.

Fig. 1.The flow of chessboard and pieces detection system

 

2. CHESSBOARD SEGMENTATION

The target of this step is to specify the borders of chessboard as well as its lines. Segmenting the chessboard helps us narrowing the processed image area and supports the line reconstruction and pieces detection steps after. Our chessboard segmentation method consists of three main steps: line filtering, line detection, and board segmentation.

A. Line filtering

Line-based detection approaches usually require generating a binary image of the chessboard beforehand. For this purpose, Canny edge detectors could be used. As showed in Fig. 2, this method has a problem that it detects not only the board's lines but also the edges of the background.

Fig. 2.Canny edge detector for the Janggi chessboard; (a) original image; (b) Canny edge detector; (c)(d) lines of chessboard and their representation in scaled images of (a) and (b).

Our desire is to keep the line information of the chessboard while removing the others as much as possible. In Fig. 2 (c) and (d), we could see that two close parallel lines in the edge binary image represent each line on chessboard. Besides that, the color variance in two sides of those lines is not significant. By making use of these attribute, we could remove most of unwanted lines. For each pixel p in the binary image, we calculate the gradient direction gp. Then we follow the ray r = p + n*gp (n > 0). If another edge pixel q is found with where W is line width threshold and gp is nearly opposite with gq (gp ~ -gq), then we consider the color similarity in reverse directions at p and q as in Fig. 3(a). If all conditions are matched, we have a line candidate point cpq = (p + q)/2. Fig. 3 (b) and (c) show the result of line extraction after applying the method for all pixels. We could see that the method removes most of noise in the background and keeps candidate lines on the chessboard.

B. Line detection

Based on detected line pixels, Hough transform [10] is used:

with x, y are coordinate of line points and M, N are the size of image. The purpose of the Hough transform is to perform groupings of line point into line candidates by performing an explicit voting procedure over two parameters θ and ρ.

In Hough space, each point represents one line in image spaces. By choosing a number of local maximum points, we could detect strong lines in the image as showed in Fig.4 (a) and (b). In Fig. 4(c), horizontal and vertical lines of the chessboard form two line patterns. We could apply RANSAC line fitting [9] to remove isolated points which are diagonal or noised lines and classify the remaining ones into vertical and horizontal groups. In case the number of horizontal and vertical lines is larger than defined value, we have to choose lines based on known information: firstly the Janggi chessboard has nine vertical lines and ten horizontal lines; secondly the direction and the distance between adjacent lines in each group are not too different. After that, lines in each group are sorted by ρ to decide the order. Fig. 4(d) shows our result for empty chessboard detection.

Fig. 3.Proposed method for line extraction; (a) p is a pixel in the edge image. By searching in the direction of gradient at p to find q and considering the distance of p and q and color similarity of two sides, we decide the line candidate point cpq ; (b)(c) results of our method when apply for all pixels

Fig. 4.Line detection with Hough transform; (a)(b) Local maximum points in Hough space and correspond lines in image space; (c)(d) Final results of line detection by applying line fitting and refining steps)

If only the empty chessboard is considered, our work will be almost done in this step. The problem appears when we have line missing due to the existence of pieces on the board as showed in Fig. 5. In this case, we have to reconstruct the missing lines. In Fig. 4(d), we could see that the direction of lines and distance between them is variable. Therefore, it is hard to calculate correctly the missing position. The next section will present our method for segmenting the board which supports for reconstruction and detection steps following.

Fig. 5.Line missing because of occlusion.

C. Board segmentation

In this section, we try to segment the chessboard form the background by finding four borders of the board. Similar with line detection, we could use Hough transform to detect the edges. The difference is that the binary image will not include pixels of the inner lines. Using the line filtering methods, we remove the narrow parallel lines in the Canny image instead of extract them. The result of line removing as in Fig. 6(a) still contains pixels of pieces on the board. These pixels form connected components in the binary image. Let W and H are the width and height of components; we will eliminate ones that satisfy the condition: low_ratio < W/H < high_ratio and low_range < W, H < high_range. By setting the ratio and range threshold appropriately, we have binary image in Fig. 6(b) in which the connected components of pieces are almost taken out. The edges of chessboard are detected by selecting local maxima points in Hough space. For making sure that all four borders of the board are detected, we have to choose larger than four maxima points. Among those, we have to take four lines that are nearest to and have the same direction with lines of the chessboard which are detected in Section 2. The final results of chessboard segmentation are showed in Fig. 7.

Fig. 6.Pre-processing for detecting borders of the board; (a) Binary image of the chessboard with lines removed; (b) connected components corresponding to pieces are removed.

Fig. 7.Results of chessboard segmentation. Borders of the board are specified.

 

3. LINE RECONSTRUCTION

The aim of line reconstruction is to interpolate missing lines in detection step due to occlusion. In order to restore the lines more precisely, we change the board into the original rectangular shape. Lines are then recovered by using known information of the chessboard such as the number of lines, the constant distance and direction between lines etc.

A. Rectangular warping

As presented before, this step will rectify the chessboard into an easier form to find the missing lines. We want the rectified image keep the maximum information of the original one. At first, we have to know the size and the ratio of two dimensions of the chessboard. The ratio value could be obtained easily when we know the number of vertical and horizontal lines. We define the ratio r = 10/11 which is the ratio of the width and height of the board because the Janggi chessboard has ten horizontal unit distances and eleven vertical unit distances. The side lengths of the chessboard are denoted by W1 and W2 for the upper and lower borders and by H1 and H2 for the left and right borders. Let Wm = max(W1, W2) and Hm = max(H1, H2).

If Wm / Hm > r, we choose W = Wm and H = W / r; otherwise, we choose H = Hm and W = rH. When the size of target image is specified, we could easily compute the homography matrix by using four correspondent corner points. Fig. 8 demonstrates the warping process.

B. Line reconstruction

In rectangular form, lines of the chessboard have the horizontal and vertical direction. On the other hand, the distance between adjacent lines is nearly constant. Combining these attributes with known number of lines in the board, we could detect the missing location and interpolate new lines. Fig. 9 shows one case of line reconstruction. The dot lines are recovered ones. d1 is the distance between two neighbour parallel lines and d2 is the distance from one line to the chessboard border. Because these two distances are larger than W / 10 or H / 11 which are the constant distance between vertical and horizontal lines respectively, we could know that we lack some lines in those locations. Restored lines have to satisfy the direction and distance constrains. In case of Fig. 9, reconstructed lines will be interpolated at missing locations.

Fig. 8.Results of chessboard rectification. Segmented region in the input image is warped to rectangular form

Fig. 9.Line reconstruction; (a) recover the lines by measuring the distance from each line to the others or to the border; (b) the result in real image

 

4. PIECES DETECTION

In this section, we will propose our method for detecting the pieces location and its side on the board. This work consists of two main steps, localizing the pieces on the board and classifying them into two groups.

A. Pieces detection

We assume that pieces always locate at intersection points of detected lines. With this constrain, we reduce the search area to small specific image windows centred by intersection locations. Regions of pieces usually have high contrast comparing to the board color. Based on this characteristic, we apply the salient image detection method proposed by Achanta et al. [11]. The contrast value at each color pixel is evaluated as the distance between the average color vector of the pixels of an image sub region with the average color vector of the pixels of its neighbourhood according to the equation below.

The saliency value ci,j at pixel position (i, j) is the Euclidean distance between the average color vector vq of the N1 pixels of inner region R1 and vq of the N2 pixels of the outer region R2 (Fig. 10(a)). Fig. 10(b) shows the saliency map of a warped chessboard image.

Fig. 10.Saliency map; (a) The sub region R1 and its neighbourhood R2; (b) Saliency map of rectified chessboard image, high value of saliency at pieces locations. The size of R1 is 5 x 5 and of R2 is 125 x 125. The size of image is 486 x 441.

Let d is the size of a square in the chessboard. Considering the window size d x d centered at each intersection point, we calculate the sum of saliency value for each window. Ones that have the total saliency value larger than a specific threshold would denote the existence of pieces.

B. Side grouping

In the previous step, we detected the positions of pieces on the grid of chessboard. In this section, we apply a further process for grouping them into two sides based on their colors feature. The image windows at detected locations have three main color: the color of the board, the piece, and the character. Among those colors, the most difference is in the character. By analysing the color distribution of image windows, we could classify the pieces to their side. In order to separate the brightness and chroma, the YCbCr color space is used. We only consider the Cb and Cr components. If two image windows have different color distributions, the histograms of Cb or Cr channel will be also different. Fig. 11 shows three image windows and correspondent histograms in both Cb and Cr channel. With two pieces at the same side, the blue pieces, the shape of histogram in both Cb and Cr channel is also almost same. However, when looking at the red piece window image, we could see that the form of its histogram is different with the histogram of two blue pieces. For the purpose of checking the similarity of two histograms, we calculate the intersection value by the below equation.

In which n is the number of color value, hist1 and hist2 are two histogram functions. Two image windows will be consider in a same side if the intersection value of both Cb and Cr histogram is larger than a specified threshold. Fig. 12 shows the final result of detection which consists of lines, pieces, and side of pieces.

Fig. 11.Histogram of Cb and Cr channel for image window, (a) three image windows (b) histograms for Cb channel (c) histogram for Cr channel

Fig. 12.Final result of chessboard and pieces detection

 

5. EXPERIMENT RESULTS

This section presents the results of our method in detecting the chessboard and pieces of Janggi chess game. The chessboard is detected in different views and various kinds of occlusion. At first we warp the chessboard image to rectangular shape by using the methods described in section 3.1. When the form of chessboard is rectangle, lines and pieces are easily detected. After that, we perform the inverse warping to transform detected positions back to the original image. We also compare our method with two common chessboard detection approaches, Harris corner detection and Canny line detection combined with Hough transform. With Harris corner detection method, the threshold for deciding the corner points is selected to detect as much as intersection points on the board as possible. However, as mentioned before, the corner-based methods are not suitable for occlusion cases which will lead to incorrect detections. Besides that, this method also detects unwanted points outside the board. In case of Hough transform and Canny line detection approach, we apply Hough transform for Canny binary edge image and select the local maxima in Hough space. Due to lacking refining steps, this method detects lines that do not belong to the horizontal and vertical line sets. It also fails to detect lines hidden by pieces. Fig.13 shows the results of our experiments in which not only lines are detected sufficiently but also pieces information is found.

 

6. CONCLUSION

This paper has presented our proposed method for Janggi chessboard and pieces detection. Our method could detect lines of the chessboard even if there are occlusions. Besides that, it could also extract positions of pieces and classify pieces into two sides. Knowing the piece locations, we will continue with recognition problem in the future researches. Our final target is to create a vision system that helps the robot acquiring all important information of chessboard and pieces in the image.