Automatic Extraction of Route Information from Road Sign Imagery

Abstract

With the advances of the big-data process technology, acquiring the real-time information from the massive image data taken by a mobile device inside a vehicle will be possible in the near future. Among the information that can be found around the vehicle, the route information is needed for safe driving. In this study, the automatic extraction of route information from the road sign imagery was dealt with. The scope of the route information in this study included the route number, route type, and their relationship with the driving direction. For the recognition of the route number, the modified Tesseract OCR (Optical Character Recognition) engine was used after extracting the rectangular-road-sign area with the Freeman chain code tracing algorithm. The route types (expressway, highway, rural highway, and municipal road) are recognized using the proposed algorithms, which are acquired from colour space analysis. Those road signs provide information about the route number as well as the roads that may be encountered along the way. In this study, such information was called “OTW (on the way)” or “TTW (to the way)” which between the two should be indicated is determined using direction information. Finally, the route number is matched with the direction information. Experiments are carried out with the road sign imagery taken inside a car. As a result, route numbers, route number type, OTW or TTW are successfully recognized, however some errors occurred in the process of matching TTW number with the direction.

1. Introduction

Those devices to gather the massive imagery from the roads have emerged lately; such as cameras designed to be installed in cars and mobile devices designed for drivers. With the rapid advances in the big-data technology, the real-time analysis of such massive imagery will be possible in the near future. This will be enable drivers or autonomous vehicles to immediately obtain information that are important and meaningful for driving. For that reason, the automatic extraction of transportation information from the massive imagery obtained from the roads has become a significant issue of late.

The Most of previous researches on transportation information extraction from massive road imagery focused on traffic signs and used colour space analysis, histogram analysis, the support vector machine, and other computer vision techniques. Benallal and Meunier (2003) developed a computer vision system for embedding in a car to enable traffic signs to be identified and located. For their study, they adopted the colour segmentation strategy based on the behaviour of the RGB components of several traffic signs from sunrise to sunset. The extraction of information from traffic signs through pattern matching and edge detection is proposed by Broggi et al. (2007). A support vector machine approach for classifying the pixels of traffic signs is presented by Bascon et al. (2008). In the latter, normalized fast Fourier transform is used for determining the shape of the traffic sign. Lee and Yun (2013) developed a traffic sign recognition van in which a GPS, an IMU, a DMI, a camera, and a laser sensor are installed. Using sun van, they determined the positions of the traffic signs with the laser sensor and other positioning sensor, and recognized the attribute data of traffic signs with the support vector machine. A natural scene is difficult to analyse due to many unpredicted weather and illumination conditions. This principle is also applied to traffic signs. Zakir (2011) proposed a method of resolving such problems (e.g., colour illumination and occlusion, etc.). Lee et al. (2015) also tried to solve the illumination problem by proposing an illumination invariant traffic sign recognition method using the modified census transform and Fisher vector coding with the Gaussian mixture model. Chong (2014) proposed text area detection from road sign imagery, suggesting an IRBP (incremental right-to-left blob projection) method to overcome the limitations of template-matching-based text area detection.

The above approaches concentrate on the extraction of information from traffic sign imagery, and there have been relatively few researches on the automatic extraction of information from road sign imagery. The automatic extraction of road information from road sign imagery can be categorized into three types: text information extraction, direction information extraction, and route information extraction. For the researches related with text recognition in road signs, various computer vision techniques (e.g., image segmentation, edge detection, colour space analysis, pattern analysis) are sequentially applied (Kim et al., 2013; Gonzalez et al., 2012; Huang et al., 2012). With regard to direction information extraction, composing a skeleton model with the Hausdroff distance calculation approach (Sastre et al., 2005) and the arrow shape template matching approach (Vavilin and Jo, 2009; Kim et al., 2013) has been tried. Kim et al. (2013) extracted the arrow regions with their proposed line scan algorithm, and extracted the directions of the arrowheads through template image matching. Although the route information is one of the most important information around the vehicle for manual safety and autonomous driving, there have been a few studies related with the extraction of route number information. In manually driving, the driver will be embarrassed with wrong route number information, which could induce the accident. Also, the car checks the prescheduled routes with the automatic recognition of route information in autonomous driving. Soetedjo et al. (2010) developed an algorithm for extracting the symbols and text regions from road signs with a normalized RGB chromaticity diagram by detecting only the route number area, without recognizing the number itself. Vavilin and Jo (2009) extracted some route numbers, arrow regions, and texts with colour segmentation and geometry. Those route number pixels are extracted in this approach, but the pixel group could not be recognized as a number, and the approach can extract a route number that had been laid on the arrow region. From studying the previous researches, we could know that the researcher was able to develop a route number region extraction method to differentiate the route number from other previous developed techniques. There are challenges to be overcome; however, in recognizing the route number type and matching it with the direction information.

In the paper, an approach for extracting route number information from road sign imagery is introduced. This paper is organized as follows; the following section presents the proposed route number recognition algorithm. The closed-curve areas are extracted using the chain code tracing algorithm, and non-route number areas are excluded by using the regularized size of the route number in the road sign. The route type recognition algorithm is proposed after performing colour space analysis. The route number recognition with Tesseract is also introduced in this section. Secondly, an algorithm matching the extracted route number with the direction information will be discussed with defining terminologies of the “OTW (on the way)” and “TTW (to the way).” A recognition algorithm to determine, which of OTW and TTW, is proposed based on the arrowhead’s direction and position in the road sign. The matching of route numbers and directions will be dealt with in the section. Lastly, the experiment results, achievements, and limitations of the proposed algorithm will be discussed, followed by the study’s conclusions.

 

2. Recognition of Route Number

In South Korea, the size (height and width) and shape of the route number in the road signs should follow the specifications in Korea Road Act. In the act, there are four types of routes: expressway, highway, rural highway, and municipal road. The Fig. 1 shows examples of the routes described in Road Sign Regulation published by the Ministry of Land, Infrastructure, and Transport of the Republic of Korea. In the Fig. 1, “H” represents the height of the Korean word in the road sign. As can be seen in the Fig. 1, all the routes numbers are in a closed-curve area. Therefore, the closed-curve areas are first extracted for the recognition of the route number.

Fig. 1.(a) Expressway; (b) highway; (c) rural highway; and (d) municipal road in the Road Sign Regulation (source: MOLIT (2014))

All closed-curves are extracted using the Freeman chain code tracing algorithm. The Chain codes are used to present a boundary consisting of connected straight-line segments with a specified length and direction (Gonzalez and Wood, 1992). The original colour image is converted into a binary image (black and white). From searching the whole image, a point on the boundary is selected, and its coordinates are transmitted. The encoder then moves along the boundary of the region, so as transmits a symbol representing the direction of this movement at each step. If the encoder returns to the starting position, then the boundary is defined as the closed-curve. After extracting the closed-curve areas, the rectangular closed-curve areas (or the so-called “smallest enclosing boxes”) are determined. The Fig. 2 shows the procedures; firstly, the longest pair of points is determined for each point. Each pair composed of one line and the lines are projected into a line and a sample axis. The longest projected lines in the line and sample axes are the height and width of rectangular closed-curve area. The Fig. 3 shows the results of the rectangular closed-curve area extraction overlaid on the original image. The red rectangles in the Fig. 3 denote the extracted closed-curve areas. As seen in the Fig. 3, not only the route number areas can be extracted but also boundaries of the image and letters. The extracted letters are the Korean letter “ㅇ” and the English letters “o,” “g,” “a,” “e,” and “d.”

Fig. 2.Process of determining the rectangular closed-curve area

Fig. 3.Extracted all rectangular closed-curve area

Among the extracted closed-curve areas, the non-route number areas are excluded by using the size regulation defined in the “Road Sign Regulation.” In this regulation, the sizes of the road signs and the height of the Korean word in the road signs are regulated by road sign type (e.g., three-direction notice sign in a two-lane road, three-direction sign in a two-lane road, three-direction notice sign in a more-than-two-lane road, etc.) For an instance, the width and height of a three-direction sign in a two-lane road should be 445 and 220 cm. In the same two-lane road, the height of the Korean word should be 30 cm. Based on the ratio of the width and height of the road sign image, and the Korean word’s “height” (“H”), road signs can be classified. Then the Korean word’s height in the image coordinate (Himg) can be determined in proportion to the image size of road signs. As mentioned earlier, the size of the route number area is regulated with “H,” as shown in the Fig. 1. Therefore, among all the rectangular closed-curve areas (RAclose), the route number area RArout is calculated as follows:

where i = 1, 2, ⋯⋯N (N is the maximum number of extracted rectangular closed-curve areas) and, RAhi is the height of RAi, and RAwi is the width of RAi). The Fig. 4 presents the determined rectangular closed-curve areas for the route number. As can be seen in the Fig. 4, the route number areas containing 30, 32, and 616 were extracted.

Fig. 4.Extracted rectangular closed-curve areas for the route number

Based on the colour differences, the route numbers can be classified. As discussed earlier, there are four types of routes: expressway, highway, rural highway, and municipal road. The route number is enclosed in the symbol, which denotes the type of route. The salient difference between the different symbols is the local colour distribution. In the case of the expressway, the upper part of the symbol coloured the red. The most of highway and rural highway symbols are filled with the blue and orange colours, respectively. Also, the saturation value for municipal road symbols is higher than others because a big portion of the symbols is coloured white. With the previously proposed algorithm, the route number area for an expressway contains only a number, as can be seen in the Fig. 4. In case of the expressway’s red colour characteristic, if there is a red pixel at the upper part of rectangle, the rectangular closed-curve area should be expanded for it. To differentiate the red (expressway), blue (highway), and orange (municipal road) symbols, the hue image was analysed. To extract the white colour distribution ratio, the saturation image was analysed. With these heuristic analyses, the route-number-type classification process presented in the Fig. 5 is proposed. In the Fig 5, R, O, W, and B are defined as follows;

Fig. 5.Flowchart for the route number type classification method

in which NRhs is a cardinality of pixel set Rhs, which is

where j = 1, 2, ⋯⋯M (M is the total number of pixels in the rectangular route number area), and s(Rhsj) is the saturation value for the pixel, and H(Rhsj) is the hue value for the pixel, while Nsm20 is a cardinality of the pixel set whose saturation value is more than 20 in the rectangular route number area.

in which NOhs is a cardinality of pixel set Ohs, which is

in which NBhs is a cardinality of pixel set Bhs, which is

Applying equations (2)-(8) and the process presented in the Fig. 5, the rectangular route number areas containing the numbers 30, 32, and 616 are classified as an expressway, a highway, and a rural highway, respectively.

After extracting the rectangular route number areas, the numbers are recognized with Tesseract, which is an open-source OCR (optical character recognition) engine. A Tesseract is a free software program released under Apache License ver. 2.0, and its development has been sponsored by Google since 2006 (Wikipedia, 2015). The Tesseract engine was originally developed as a proprietary software at Hewlett Packard Labs, and was released as an open-source engine in 2005. Although Tesseract is considered one of the most accurate open-source OCR engines at present (Willis, 2006), many misrecognized numbers could be found when if its provided dictionary is directly applied. For instances, the number “1” could be recognized as the English small letter “l,” and the number “101” could be recognized as the number “1” or the Korean word “이.” To avoid such misrecognitions, the route number’s font in a road sign was constructed as a learning data, which Tesseract engine could modify signs based on. After the application of the modified Tesseract engine, the route numbers 30, 32, and 616 are already correctly recognized, as shown in the Fig. 4.

 

3. Matching Extracted Route Number with Direction Information

In Chapter 2, a method of recognizing route numbers and their type (i.e., expressway, highway, rural highway, or municipal road) was proposed. The present chapter deals with matching of the route number and type with direction information. Before the discussion of method, there are terminologies to be defined; OTW (On the Way) and TTW (To the way). The OTW means “the driver is on the way,” as the TTW means “drivers will encounter the way.” If Expressway 7 is located towards the east (as indicated by the east direction arrow), it means “If you go to the eastern direction, you will find yourself on Expressway 7”; with the terminologies, this line can be simply expressed as “North OTW Expressway 7.” If the Highway 4 is located near the northern direction arrowhead and is not located on the arrow itself, it means “If you go to the northern direction, you will encounter the Highway 4”; which could be simply expressed as “Straight TTW Highway 4.” In the Fig. 4, three kinds of information can be extracted: “Straight TTW Expressway 30,” “Straight OTW Highway 32,” and “Right TTW Rural Highway.” These terminologies will be used in this paper for matching the route number and the direction information.

The direction information is recognized with the algorithm proposed by Kim et al. (2013). The seed area for the arrow is firstly extracted using the line scan method in Kim et al. (2013), prior to extract the arrow area with the region growing method. After detecting the feature points using the good-features tracking algorithm, the template-matching algorithm is applied to extract the arrowhead. Around the feature points, five kinds of arrowhead template expressing the east, west, north, northeast, and northwest directions are matched in the order. Finally, the direction information can be extracted by selecting the template that produces a high correlation coefficient value.

OTW or TTW route numbers are differently located depending on the type of road sign. In MOLIT (2014), there are three types of road signs containing route numbers. The Fig. 6 presents these; the first one is DNS (direction notice sign) in the Fig. 6(a) that contains 2/3 DNSs and 2/3 direction in MOLIT (2014); the second one is DS (direction sign), which contains 2/3 DSs, as shown in Fig. 6(b), while the third one refers SDNS (simple direction notice sign), which contains 2/3 DNSs. These three road sign types can be differentiated from one another based on the ratio of the width and height of the road sign imagery following to the sizes of road signs in Road Sign Regulation. As the route numbers are differently located depending on the type of road sign, a different recognition method needs to be applied for each type.

Fig. 6.Three types of road signs containing the route number: (a) direction notice sign (DNS); (b) direction sign (DS); and (c) simple direction notice sign (SDNS)

OTW or TTW for route numbers are recognized after composing closed polygon for the case of DNS. The Fig. 7 shows the concept of composing closed polygon for DNS. The closed polygon is comprised of the end of extracted arrowheads and the base point, which is generated by perpendicular projecting the end of highest arrowhead onto lower end of the image. A, B, and C denotes the end of extracted arrowheads, and D is a base point in the Fig. 7. After composing the closed polygon, the position of extracted route numbers is examined. If the extracted route number is inside the closed polygon, then it is classified as OTW. The other route number, located outside of the closed polygon, is classified as TTW, while the direction of each route numbers is acquired from its nearest arrowhead. With these method, following information is acquired in the Fig. 7: “Left OTW highway 37 and 47”, “Right OTW highway 47”, “Straight OTW highway 37”, “Left TTW highway 25”, “Right TTW highway 27”, and “Straight TTW highway 26”.

Fig. 7.Concept of composing a close polygon in the case of DNS

The recognition of OTW or TTW for route numbers starts from the image segmentation in the case of DS and SDNS. As stated earlier, DS and SDNS have two or three direction signs. The image is equally divided perpendicularly into two parts (two-direction road sign) or three parts (three-direction road sign) in the case of DS. Likewise, the image is equally divided horizontally into two or three parts in the case of SDNS. Then, the existence of the route number from the end of arrowhead to the opposite direction of arrowhead is searched. If the route numbers are detected during searching, the route numbers are classified as OTW. Otherwise, the route numbers are classified as TTW. The searching is processed within each segmented image. The Fig. 8 presents the concept of image segmentation and determination in either OTW, or TTW. In the Fig. 8, the image is divided into three equal parts since it is a three-direction sign. In left part of Fig. 8, route number 513 and 21 are detected during the search. Therefore, the following information is acquired with the suggested method: “Left TTW Highway 21”, “Left OTW Rural Highway 513”, “Straight OTW Highway 21”, “Straight TTW Expressway 35”, and “Right TTW Highway 17” in the case of Fig. 8(a). “Straight OTW Highway 32” in the case of Fig. 8(b).

Fig. 8.Concept of image segmentation and OTW or TTW determination in the case of DS (a) and SDNS (b)

 

4. Experiment and Results

The proposed algorithm is applied to road sign imagery. The used imagery is acquired by MMS (Mobile Mapping System) equipped with Ladybug5 sensor, which is generated by PointGrey corporation, and laser sensor LMS 151, which is generated by SICK corporation. During the acquisition of image, the velocity of MMS is about 60 km/h. The imagery is orthorectified by using point cloud from laser sensor to manually crop the road sign section. The processing time for one image is less than one second in the Intel(R) Core™ i5-2400 CPU @3.10 GHz computer. Among the acquired imagery, the partially occluded imagery by branches or wire is excluded for the experiment.

The Fig. 9 presents imageries that were used in the experiment, and the results lists in the Table 1. In the Fig. 9, each image presents each case; (a)-(d) of DNS cases, (e)-(f) of DS cases, and (g)-(h) of SDNS cases. As seen in Table 1 and Fig. 9, all the route numbers and their types are correctly recognized. Also, The OTW or TTW recognitions results successfully. Some errors occurred, however, during the matching of the route number with the directions. By comparing with the extracted results (Table 1) and the experiment imageries (Fig. 9), those of TTW Highway 34, TTW Highway 1, and TTW Expressway 30 have supposed to be matched with the ‘Straight’, but they turned out to be the ‘Left’ in the extracted results. The reason for error occurrence is that the route number’s direction is determined based on the distance between numbers and arrowhead. For an instance, the distance between the route number 34 and the the same route number and the northern direction arrowhead. Compared with other cases, it can be concluded that such problems occur when the place’s name is relatively long. Therefore, if the route number is matched with the place’s name after matching the place’s name with the arrowhead, such errors will not occur.

Fig. 9.Imagery used for the experiment in this study: (a)-(d) are DNS cases; (e)-(f) were DS cases; and (g)-(h) are SDNS cases

Table 1.Experiment results for the items listed in Fig. 9

 

5. Conclusions

This study dealt with the extraction of route information (number and type, OTW or TTW, and direction) from road sign imageries. For the recognition of route number, the rectangular closed-curve areas for route numbers were extracted using the Freeman chain code tracing algorithm, while the size of the route number was regulated by Road Sign Regulation. The route number was extracted using the modified Tesseract software with a cropped route number area image. The route number was classified using the proposed process and equations. The equations were generated through colour space analysis with the hue and saturation value percentage for the route number area. The arrowheads were used for determining that the route number was either OTW (On the Way) or TTW (To the way). A different determination method was proposed for each route type. The extracted route numbers were matched with the direction information, based on the arrowhead direction and the distance between the route number and the arrowhead. With the experiment results, we could conclude in that the route number, route number type, OTW, or TTW were successfully recognized, but some errors occurred in matching the TTW route number with the direction information. The future research will be aimed at improving the algorithm for matching the route number with the direction information by using the sequential matching method, which matches place’s name and route number after matching the arrowhead and place’s name.