Lane Detection and Tracking Using Classification in Image Sequences

Abstract

We propose a novel lane detection method based on classification in image sequences. Both structural and statistical features of the extracted bright shape are applied to the neural network for finding correct lane marks. The features used in this paper are shown to have strong discriminating power to locate correct traffic lanes. The traffic lanes detected in the current frame is also used to estimate the traffic lane if the lane detection fails in the next frame. The proposed method is fast enough to apply for real-time systems; the average processing time is less than 2msec. Also the scheme of the local illumination compensation allows robust lane detection at nighttime. Therefore, this method can be widely used in intelligence transportation systems such as driver assistance, lane change assistance, lane departure warning and autonomous vehicles.

1. Introduction

In many advanced driver assistance systems (ADASs), land detection and tracking are practical utilized to detect dangerous situations and take an active part in the driving [1]. The main processes of these systems usually consist of road detection, lane detection and obstacle detection. A robust lane detection method enables the systems to easily distinguish obstacles and traffic structures. Therefore, the lane detection can be applied as the core process for ADASs.

For lane detection, various methods based on Hough transform (HT) are widely used [2-5]. HT-based methods have several bottlenecks in the development of commercializable systems as follows [2]: 1) Standard HT has complex computation for every mapped parametric point, 2) Standard and modified HT are often difficult to determine artifacts and lane boundaries, 3) They must be applied after preprocessing such as edge detection, and 4) They are sensitive to shadows, road marks and so on.

Methods using tangent vectors were shown to be quite robust on well-marked lanes, but they may fail to detect poor-marked lanes [6, 7]. And Lane-finding in ANother domAin (LANA) algorithm [8] using frequency domain features was proposed, but it also used another representation of edge.

Several methods using pixel clustering are proposed; however, they need excessive computation time. Mathematical model-based methods can be used to fit the lane boundaries; however, simple models cannot correctly estimate lane information and complicated ones are vulnerable to disturbance [9].

Classification methods based on training for lane or road detection were proposed; however, their performance may not be guaranteed when road and illumination conditions are not encountered in training [2, 10, 11]. The performances of some state-of-the-arts methods using multiple kernel sparse representation based on classifiers [12] or adaptive hypergraph learning [13] seem to have high performances; however, they need complex features and classifiers.

In this paper, we propose a robust lane detection scheme based on the use of both structural and statistical features of candidate lane marks. Lane marks are usually characterized by bright line features. To discriminate the lane marks from various types of road marks such as arrows, rectangles or diamond-shaped road marks, the horizontal projection function of the candidate shape and its derivation are fed to the neural network. The performance of the lane detection is shown to be robust to the variation of illumination conditions.

In addition, the internal angle of the vanishing point to track lanes in the current frame is used to locate the lane marks if track lanes are not found in the next frame. Most of all other methods are designed for daytime environments; however, the proposed method can be used at nighttime. The average processing time of this algorithm is less than 2msec so it is fast enough for real-time operation. It is overwhelmingly faster than other methods using HT and pixel clustering.

 

2. Classification-Based Lane Mark Detection

Because the terms related to roadways are confusedly described in many places, we first define some terms as shown in Fig. 1. A lane is a guidance region for use by a single line of vehicle, and lane marks are the solid or dash painted lines to divide lanes, traffic lines are both side lines to separate lanes, and a vanishing point is the cross point of both side traffic lines.

Fig. 1.Definition of terms related to roadways.

To detect and track traffic lanes, the overall scheme of the proposed method is shown in Fig. 2. Traffic marks belonging to lane marks are brighter than road color, so we detect bright evidences in traffic images. Local illumination conditions are, however, various because of camera angles and lighting conditions, so we first compensate the local illumination variables by a DC-notch filter [14]. The input luminance image f(x,y) is used for lane detection, so the filtered image g(x,y) is calculated by

where is the local average calculated by a N×N local convolution mask.

Fig. 2.Overall scheme of the proposed method.

Bright evidences are segmented by binarization using the threshold method. The threshold is simply calculated by the Otsu’s method. However, the traffic marks are not obviously observed in g(x,y) in nighttime images as shown in Fig. 3. So we perform the contrast stretching as follow:

where Cg and denote the contrast ranges of g(x,y) and , respectively, is the stretched image, and i5 is the intensity of the upper 5 percent for the histogram of g(x,y). It means is represented as g(x,y)Ⅹ255/i5 for g(x,y) below i5 and 255 for values upper i5 to linearly enhance the contrast of the most popular intensity level (lower 95 percent).

Fig. 3.f(x,y), g(x,y) and .

Followed by binarization, bright evidence regions are labeled into Bi (0

where w is the width of B ,and the differential of the projection function p'(y) defined by

to calculate features. By the above clues, we use standard deviations of p(y) and p'(y), σ(p(y)) and σ(p'(y)), for features of classification.

Fig. 4.p(y) and p'(y) of Bi belonging to lane marks.

Fig. 5.p(y) and p'(y) of Bi not belonging to lane marks.

In addition, the size, position and thickness are useful information for classifying lane marks, so we use the width and height of bounding boxes ( w and h ), the x-y coordinate of center of bounding boxes (cx and cy), and the average thickness, that is the mean of p(y), μ(p(y)), as features. Because these features ( w, h, cx, cy, μ(p(y)), σ(p(y)) and σ(p'(y)) are obvious for classifying traffic lines and others, simple or typical classification methods, e.g. heuristic rules, k-NN (nearest neighbor), neural networks and so on, can be applied. It is hard to find out the appropriate heuristic rules that can classify traffic lanes and other road marks accurately. Moreover, k-NN method needs a large number of training data set. Neural network method, however, can classify with a few training of simple features. In this paper, we used a very simple classification by heuristic rules and the neural network as shown in Fig. 6, where the variables described τ are the thresholds for classification. In the neural network, we adopted an MLP (multiple layer perceptron). The network consists of an input layer, an output layer and one hidden layer. As shown in Fig. 7, the number of nodes for input, hidden and hidden layers are 4, 8 and 2, respectively. The last input node is used for training the bias, and the number of hidden nodes is determined to be twice as large as the size of the input layer. For training we use the error back propagation algorithm.

Fig. 6.Classification rule for traffic line detection.

Fig. 7.The structure of the neural network.

After classification, we take candidate traffic lines Ci (0

Table 1.Parameters for traffic lines.

The direction of traffic lines (d) are classified into left, right and alternative ones by subregion intensity as shown in Fig. 8(a), and the average intensity of subregions (μ(Ri)) are used as follows:

where λ is the positive margin for alternative marks because the marks may be tilted in the opposite direction when the vehicle changes the lane or is driven in a curved lane. The other parameters are calculated using Bi by detecting two points (p1 and p2) on upper and lower positions as shown in Fig. 8(b). The slope of traffic lines (m) is the slope of p1 and p2. The intersection points between traffic line and top and bottom horizontal lines (pt and pb) can be calculated using the equation of straight line set up by p1 and p2.

Fig. 8.Subregion division (a) and calculation of line information (b).

 

3. Lane Detection and Tracking

Because a lane is marked with two traffic lines, we detect left and right traffic lines for lane detection. After detecting both side traffic lines, the left and right traffic lines of the tracked lane at the current frame are Lt and Rt, respectively.

If many left lane marks are detected, we reject traffic lines having low values of two similarities for lane width and slope from Ci such that d=l or d=a.

After this rejection, we track Ci having the minimum distance between and . We use the same algorithm to track right lane marks.

When only one side traffic line is tracked, we estimate the other side traffic line as shown in Fig. 9. In our estimation method, we assume two constraints, 1) the vertical position of the vanishing point of the tracked lane is not changed, and 2) the internal angle of the vanishing point is not changed. When the only left side lane mark is tracked, the rotated angle of the left traffic line is calculated by

So, the slope of the estimated right traffic line is

and the horizontal position of the vanishing point of the tracked lane is calculated by

The line information of and is can be updated by (10). The same method can be utilized to estimate untracked left traffic lines.

Fig. 9.Estimation of untracked traffic line.

When a vehicle is changing the lane with the left (or right) direction, a left (or right) traffic line is changed to a right (or left) line. At this time, the lane should not be tracked, so we determine the state of changing lane by two OR conditions as follows:

where c and s denote the center and side horizontal ranges, respectively. When the state of changing lane is detected, a traffic lane can be detected without tracking as the same detection with the initial state.

 

4. Experimental Results

The proposed method was implemented in Visual C++, and tested on a PC (Intel® Core™ i7-3770 CPU @ 3.40GHz 3.40GHz). The test image sequences were grabbed from one smart phone and four car black boxes with 320×240 resolution.

Detection results of lane marks are shown in Fig. 10, where ROI (region of interest) is manually set by the camera angle and it is represented in magenta. Detected left marks are represented in red and the right marks are represented in blue. As shown in Fig. 10, most of all lane marks are successfully detected, and other traffic marks are not detected.

Fig. 10.Experimental results of lane mark detection.

Lane detection results of some tested images are shown in Fig. 11, where the high performance of the proposed method is revealed in various illumination and road conditions. In addition, curved lanes are verisimilarly detected because the lanes can be approximated by line segments in ROI areas.

Fig. 11.Experimental results of lane detection and tracking.

When the driving car is changing the lane, the tracked lane is successfully changed as shown in Fig. 12. The following graph of Fig. 12 shows at consecutive frames, where the red lines indicate the state of changing lane.

Fig. 12.An example of changing lane.

The average processing time is 1.930 msec. It is fast enough for real-time operation and much faster than other methods. The processing time is inconceivable to HT-based methods. Quantitative comparison with other methods is not feasible since standard database images are not provided. The proposed method has, however, a remarkable simple computation in lane detection researches, and the method is capable of correctly detecting and tracking lanes in various operating conditions. In addition, HT-based methods may not detect short traffic lines as shown in Fig. 13, and may detect non traffic lines such as crosswalk, road crack and shadow as shown in Figs. 14 and 15. However, our method can classify crosswalk and do not detect road crack and shadow as traffic lines because we do not use edge information.

Fig. 13.A scene in which short traffic lines are observed.

Fig. 14.A scene in which many lines are observed.

Fig. 15.Scenes in which edges which are similar to traffic lines are observed.

 

5. Conclusion

In this paper, we proposed a novel lane detection method based on classification. Because the preprocessing and extracted features are very robust for classification, this method can be said to accurately detect traffic lanes. To estimate untracked traffic lines, the inner angle of two side traffic lines are used. In addition, the state of changing lane is determined in order to decide the driving lane. The processing is very simple and fast enough to be determined, the average time of processing is less than 2msec, so it is very suitable for real-time systems. Furthermore, most of currently used methods are concerned only daytime but our method can robustly detect traffic lanes even at nighttime by using appropriate preprocessing. Therefore, the proposed method can greatly contribute to intelligence transportation systems (ITSs) such as driver assistance, lane change assistance, lane departure warning and so forth. The proposed method can detect short dashed traffic lanes much accurately than HT-based methods but there are still some cases that cannot be detected short dashed traffic lines when the camera is viewing down excessively. For future works, we are going to develop an algorithm that can detect vehicles ahead in order to propose a smart car black-box system using the proposed methods.