An Approach for Security Problems in Visual Surveillance Systems by Combining Multiple Sensors and Obstacle Detection

Abstract

As visual surveillance systems become more and more common in human lives, approaches based on these systems to solve security problems in practice are boosted, especially in railway applications. In this paper, we first propose a robust snag detection algorithm and then present a railway security system by using a combination of multiple sensors and the vision based snag detection algorithm. The system aims safety at several repeatedly occurred situations including slope protection, inspection of the falling-object from bridges, and the detection of snags and foreign objects on the rail. Experiments demonstrate that the snag detection is relatively robust and the system could guarantee the security of the railway through these real-time protections and detections.

1. Introduction

With the development of railway techniques and the construction of infrastructures, the upcoming modernization of cities make the rails become one of the most commonly used means to transport cargos and passengers. Compared with other forms of transport such as highway and airplanes, railway transport is capable of high levels of passenger and cargo utilization, better customer comfort, high energy efficiency, limited use of space and less noise pollution [1], etc. Especially, when compared with the highway, the railway does not have the problem of traffic congestion. While as the railway transport is closely correlated to people’s lives and property, the safety of railways is the most significant issue that people care. Accidents in railway lines can be mainly categorized into collision with snags and derailment [2]. One reason that may cause derailment is the rail facility itself, which requires consistent and frequent inspections and maintenances. For example, rail defects lead to rail breaks and further to derailment, and they are generally inspected manually, but manual inspection is slow, laborious, and potentially hazardous [3], so recently they can be detected through vision based techniques [4]. Another cause that results in derailment is erroneous estimates or mistaken operations by engine drivers. It can only be avoided by the caution of engine drivers. Collision with snags, such as sudden appearances of objects, animals or human beings on the railways, obstacles on the railways, etc., is very dangerous to railway transport. Unfortunately, due to the fact that a number of rail lines constructed in mountainous or desert regions, these situations are fully possible to happen. Thus, it becomes very important to have a system that examines the safety of these lines.

With the development of electronics, sensors are widely employed in practical systems [5-7]. Many efforts have been done to establish a detection and protection system for rail lines and most of them were based on wireless sensor network [8-13]. In [8], Kalpana Sharma et al. established a wireless acoustic sensors to detect rail defects such as cracks and breakages in the railway tracks. Zeinab Sam Daliri et al. [9] employed electromagnetic and ultrasonic sensors to recognize flaws in the railway. S. Ramesh [10] presents a system for detecting the railway cracks based on Infrared Rays (IR) sensors. In [11], a wireless sensor network was constructed to monitor the rail lines. Emad Aboelela et al. [12] developed a wireless sensor network in order to reduce the occurrence rate of accidents and improving the efficiency of railroad maintenance activities. Kalaimathi et al. [13] presented a railway surveying system combined by a Global positioning System (GPS) module and Micro Electro Mechanical Switch (MEMS) based track detector (ultrasonic sensors utilized) to detect cracks in the railway tracks. For the recognition of snags and foreign objects on the rail, vigorous solutions were proposed. In general, they can be categorized into two classes, sensor based approaches [14-17] and vision based techniques [18-22]. Sensors that are widely employed in the snag detection on the rail include a barrier of emitters and receivers [14], combination of video cameras and radars (LIDAR) [16, 17], infrared systems [15], and so on. Vision based approaches were widely employed in many areas [23-28] and basic tasks in railways have been studied by many researches. For instance, F. Kaleli et al. [21] extracted rail track by using dynamic programming. In [29], small parallel segments were evaluated to extract rails, and Bogdan Tomoyuki Nassu and Masato Ukai [30] employed matching edge features to choose rail patterns. Qi Z et al. [20] used HOG feature and a region-growing algorithm to extract railways. Algorithms using the ground plane and height information were described in [19] to detect obstacles in railways. Pankaj Jain et al. [35] proposes a visual obstacle detection algorithm based on image segmentation. M. Ruder et al. developed a system capable of monitoring the track and detecting objects in front of a train [22]. To sum up, a variety of threatens can induce a collision of a train and various sensors and vision based algorithms are employed to protect the train from these potential threatens.

In this paper we focus on three repeatedly occurred situations including slope protection, inspection of the falling-object from bridges, and the detection of snags and foreign objects on the rail. We propose a hierarchical system of wireless sensor network to communicate between sensor nodes and the servers, and develop a vision based algorithm to check the safety of railways. In the proposed system, switching sensors and slope protection nets are employed to detect landslide, displacement sensors and fall prevention nets are utilized in the inspection of falling-object from bridges, and a camera is installed to detect snags on the rail. The contributions of this paper include: 1) proposal of a vision based algorithm to detect snags on the railway; 2) mixed use of the wireless sensor network composed by multiple sensors and the vision based algorithm to enhance the safety of rail lines.

The rest of the paper is arranged as follows. Section 2 gives a description of the overall multi-sensor system and we present the snag detection algorithm in Section 3. The setup of the system and experiments are delineated and executed in Section 4 and we come to conclusion in Section 5.

 

2. Multi-sensor System for Railway Security

In this section, we propose a multi-sensor system for railway security. Three situations are taken into consideration, including slope protection, inspection of the falling-object from bridges, and the detection of snags. The construction of the network is presented in Section 2.1 and the action of sensors is described in Section 2.2.

2.1 Wireless sensor network

A wireless sensor network is established to communicate between the sensor nodes and the data collecting terminal. A sensor node consists of a microcontroller, a RAM, a short-range radio transmitter, and a small power source. Sensor nodes are used to perceive the environment and collect data [31]. Fig. 1 shows a typical wireless sensor network of our system. Multiple sensor nodes are connected with a sink node through Zigbee network and the sink node is in charge of sending data to the data collecting terminal.

Fig. 1.A typical wireless sensor network

The proposed multi-sensor system is composed by a number of wireless sensor nodes that are deployed along the rail lines. Multi-layer routing is also required since the railway security should not only be examined on site, but also be checked in a remote control center which is continuously monitored by the authority and the authority can request for the location of an abnormity. In our system, sensors are placed at fixed locations such as bridges, tunnels and some special points, and cameras are installed at places where there is a possibility of landslide covering rails with stones. Fig. 2 presents the wireless sensor network for our railway security system. The sensors gather the information from the environment and a router (included in the sink node in Fig. 1) reorganizes the data collected by the sensors except cameras at the lowest layer (layer 0) and forwards these data packages to the next higher layer (layer 1). The video information captured by the cameras at layer 0 is transmitted via 2.4G wireless network. By using the wireless modules (Zigbee, 2.4G wireless network) the information is passed to the data collecting terminal. The data collecting terminal might need to be placed on site. However, the railway security should also be able to be examined at a control center where authority can get the information of an abnormity and command. This makes a web server necessary and is deployed at layer 2 in Fig. 2. The data collecting terminal transmits data packages to the web server through the Ethernet network.

Fig. 2.Multi-sensor system for railway security

2.2 Action of sensors

Three situations are considered in the proposed railway system. For slope protection, it deals with the situation when landslide occurs. The slope protection net first conducts to block soil and stones from tumbling down. Then the switching sensor functions and sends information up to the web server. Authority can see an online scene acquired by the installed cameras and alarm trains along this abnormal rail line. Slope protection is usually set in the mountainous sections of railways. The inspection of the falling-object from bridges is implemented by the displacement sensor. A protection net is equipped on the bridge when there are rail lines under the bridge. When object falling from the bridge, the displacement sensor set on the protection net will act and alert the system about the falling object. The last function of the railway security system is snag detection on the railway. The cameras mounted along the rail lines transfer real-time video information to the web server and a vision based snag detection algorithm is executed to decide whether there is a snag on the railway, and if there is an obstacle, it should be localized.

 

3. Snag Detection

The distinguishing of moving trains from moving snags is one of core issues to verify the performance of snag detection. The snag in this paper is defined as any objects that are hindering or might potentially hinder the running of trains. Therefore, trains should not be detected as snags and we employ a superpixel based detection algorithm to identify trains and exclude them from snag candidates. This is described in Section 3.1. As the snag detection algorithm is based on background subtraction algorithm, which requires a stable background model, and the background model is delineated in Section 3.2. We present the description of snag detection algorithm in Section 3.3.

3.1 Superpixel based train detector

Since the rails are not always horizontal lines or vertical lines, it is difficult to annotate a running train in an image by a bounding box (as shown in the left image of Fig. 3, there are numerous noises within the bounding box); otherwise we need to annotate the train manually pixel by pixel. In order to reduce the load of tedious hand-labelling and at the same time balance the precision of labelling, in this paper we employ superpixels as the basic element to process the training of trains. For a training image, superpixels are first extracted by SLIC algorithm [25] (as shown in the top right of Fig. 3). Then the image can be represented by a number of superpixels (about 300, a parameter that inputs to SLIC) and trains are a part of these superpixels (as shown in the bottom right of Fig. 3).

Fig. 3.Annotations of trains based on superpixels

In the learning process of the prior train detector, we employed Histograms of Oriented Gradients (HOG) [32] as the extracted feature and Support Vector Machine (SVM) [33] as the learning algorithm. Features extracted from superpixels that are annotated as trains are positive features and otherwise they are used as negative features.

In the detection process, a blob is considered as a superpixel [25]. For a current frame, several operations are performed (such as the calculation of difference frame, see Section 3.3 for more details) to obtain abnormal pixels, and these pixels agglomerate into blobs. The decision on one blob is based on the decisions of features involving in the blob. If the number of positive features beats the number of negative features, the blob is determined as a train blob; otherwise it is decided as a non-train blob.

3.2 Background model

As the background subtraction algorithm is a pre-processing step in our snag detection algorithm, a stable background model exerts a tremendous influence to the performance of snag detection. A robust background subtraction algorithm should be able to deal with illumination changes due to light or weather, reflections on screens (indoor), etc. The main task focuses on the update of the background model. We employed a running average with selectivity model to update the background model [32], which is described in Eq. (1) where ft-1 represents the previous frame.

We denote the current and previous background model by and , respectively. α suggests a learning rate and a typical value is 0.05. This equation connects the current background model with the classification (background or foreground) of the previous frame, which prevents the background model from being contaminated by non-background pixels.

3.3 Snag detection algorithm

Snag detection is the most important part of the proposed railway security system. In this work we employ a combination of image processing and machine learning techniques to achieve a high performance of rail snag detection.

Fig. 4 shows the flowchart of snag detection. The video sequence is first processed by difference analysis, which is done on two adjacent frames for each pixel. Define ft-1 and ft as the intensity values of two adjacent frames, and as cameras are fixed, the reference frame (represented by Br) can be obtained. The difference frame is defined as the average of two adjacent different frames compared with the reference frame as shown in Eq. (2). The update of the reference frame is described in Section 3.2 (Eq. (1)).

Fig. 4.Flowchart of snag detection

Blobs are formulated by identifying and analyzing strongly connected components for the difference frame calculated in the previous step. A binary image is first generated from the difference frame and the threshold is set to 15 in our experiments, and then 8-connected component is labeled to extract blobs. In order to adapt the proposed algorithm to the illumination change of frames, two blob conditions are regulated. One is that the total number of pixels in a blob should be larger than a threshold thre1 (set to 50 in our experiments) and the other condition is that the average difference intensity of a blob (normalized by 255) should be larger than a threshold thre2 (set as 0.1 in our experiments). The two conditions are represented by Eq. (3) where Mb is the binary mask of the bth blob (b=1,..., B, B is the total number of blobs) and N is the total number of pixels in the blob mask. Dt,i suggests the intensity value of the position corresponding to index i in the current difference frame Dt. This processing can filter out noises and only blobs that satisfy these two conditions are considered as foreign objects and passed to the next step.

The foreign objects detected by the blob algorithm could be a moving train along the rail line and this situation should be excluded from the abnormities. An object is believed to be a running train based on that the detection of the object is a train and the moving path is along the rail track. A prior train detector that is delineated in Section 3.1 is learnt to decide whether the object is a train. Since the camera is fixed, the rail track template can be obtained beforehand.

The moving path restriction on a blob is examined by the coverage pixels of the template (Ti in Eq. (4) is the template value for index i) and the blob mask (as shown in Eq. (4), thre3 is set to 50). Snags are identified as the detected blobs barring from the ones that are recognized as trains and satisfy the moving path restriction. We summarize the snag detection algorithm in Algorithm 1.

 

4. Experiments

In this section, we will evaluate the performance of the snag detection algorithm and the railway security system. Experiments related to the snag detection algorithm are shown in Section 4.1. The setup and actions of the railway security system are reported in Section 4.2.

4.1 Experimental results for snag detection

Three parameters (thre1, thre2, thre3) are involved in the snag detection algorithm, and are tuned on a validation dataset which includes more than 500 frames and is different from both the training dataset and the test dataset. The results are shown in Table 1. It is clear that the accuracy performance is similar for different values of parameters thre1 and thre3. The performance obtained under the value of 50 (both thre1 and thre3) achieves slightly better than the others and this value is used in the test experiments. For thre2, 0.1 performs the best, which is employed in the test experiments.

Table 1.Results for parameter tuning

We demonstrate the proposed snag detection algorithm in two aspects. First, the effectiveness of the constraints is investigated. The calculation of the difference frame is influenced by the illumination as we only do a subtraction operator between the online frames and the reference frame. The illumination impact can be filtered out by the two constrains we described in Section 3 (Eq. (3)). Fig. 5 shows experimental results of difference frames. In the left figure of Fig. 5, four blobs are constructed. However, in fact only one blob is a snag, and others are products of illumination changes in difference frames (original images can be found in the third row of Fig. 6) and are removed by the constrains. In the right figure of Fig. 5, only one blob is formed, but this blob is a running train and we should not recognize it as a snag. When the blob inputs to the prior train detector, the detector determines the blob as train. At the same time, the rail template is also matched. Therefore, this blob is confirmed as non-snag (see Fig. 7 for the detection results).

Fig. 5.Difference frames. Left: The 66th difference frame from Sequence 1 and four blobs are constructed. Right: The 56th difference frame from Sequence 2 and one blob is constructed.

Fig. 6.Snag detection in Sequence 1. The first row and third row: original frames (the snag is labelled by a red ellipse); the second and last row: detection results

Fig. 7.Snag detection results in Sequence 2

The overall snag detection is tested on several video sequences and parts of the results are shown in Fig. 6 and Fig. 7. In Sequence 1 there is a stone rolling down from the mountains (abnormity situation), while in Sequence 2 (normal situation) a train is moving on the rail line, which should not be detected as a snag. We report a series of original frames and corresponding detection results in Fig. 6 and Fig. 7. Table 2 reports the comparative experiments for snag detection on seven video sequences between the proposed snag detection and the obstacle detection based on image segmentation [35]. The accuracy is calculated by the percentage of the number of correctly detected frames (two situations: the frame is background and detected as a non-snag frame; the frame is a snag frame and detected as a snag frame) and the total number of frames. From these results, we can see that the proposed system is able to output both the appearance of the snag (annotated by a word “obstacle!” in Fig. 6) and the location of the snag (filled with blue on the original image). Compared with the segment method in [35], the proposed method achieves a much better and more robust performance, as our performance is slightly worse than that of [35] (video 5: ours: 95.1%; [35]: 95.9%) in only one sequence and is superior or identical to that of segment method [35] in other sequences.

Table 2.Results of snag detection on seven video sequences

4.2 Tests on the railway security system

We simulate the railway security system on a railway platform. The platform displays a complex rail line with cluttered backgrounds. There are mountains along the railways and also bridges over the rail lines. Fig. 8 reveals a top view of the platform and the deployment of sensors in the proposed wireless sensor network.

Fig. 8.Deployment of sensors for the railway security system (An abnormal state of the railway security system (a snag appears))

Three kinds of sensors including switching sensor, displacement sensor, and camera for snag detection are installed on the platform. Any of these sensors or the snag detection algorithm discovers an abnormity, it will send a signal to the system, and a signal light is lightened to indicate the state at the location of each sensor mounted. Fig. 8 also discloses a state of the system. The red light denotes for an abnormity and the green light suggests a normal state. Fig. 8 signifies that a snag is detected and other locations where sensors are mounted are in a normal state. The abnormity of the railway line can be reflected through the combination of the sensors and the visual algorithm based on the surveillance system.

 

5. Conclusions and Future Works

In this paper, we present a railway security system via a mixed use of wireless sensor network and vision based techniques. A vision based snag detection algorithm is proposed and we employ multiple sensors to implement slope protection and falling-object protection from bridges. These multiple sensors and cameras compose a wireless sensor network. If an abnormity occurs, the information can be transferred to a long-distance control center. The system guarantees the security of the railway through these real-time protections and detections. In the future, we will extend the system with more types of sensors to observe more potential dangers for trains and exploit more video based algorithms such as train tracking to apply in the system.