Object Tracking based on Relaxed Inverse Sparse Representation

Abstract

In this paper, we develop a novel object tracking method based on sparse representation. First, we propose a relaxed sparse representation model, based on which the tracking problem is casted as an inverse sparse representation process. In this process, the target template is able to be sparsely approximated by all candidate samples. Second, we present an objective function that combines the sparse representation process of different fragments, the relaxed representation scheme and a weight reference prior. Based on some propositions, the proposed objective function can be solved by using an iteration algorithm. In addition, we design a tracking framework based on the proposed representation model and a simple online update manner. Finally, numerous experiments are conducted on some challenging sequences to compare our tracking method with some state-of-the-art ones. Both qualitative and quantitative results demonstrate that the proposed tracking method performs better than other competing algorithms.

1. Introduction

As one of the important problems in computer vision and pattern recognition, object tracking plays a critical role in many research lines (e.g., motion analysis, video compression and activity recognition) and has many useful applications in realistic scene (e.g., traffic control, human computer interface and video surveillance) [1][2]. The traditional tracking methods often work well under some well-controlled conditions or track some specific objects [3][4] (such as human, car, face and so on); while online visual tracking aims to track any object in realistic conditions [5]. It is very difficult to develop an effective online tracking method for many challenging factors [6], which mainly include illumination variation, pose change, partial occlusion, scale change, background clutter and so on.

From the perspective of adopted theories and techniques, online visual tracking algorithms can be categorized into three classes: tracking methods based on state estimation, tracking methods based on online classifiers and tracking methods based on template matching. First, tracking methods based on state estimation consider the tracking problem as a state estimation problem and reclusively estimate the states of the tracked target, such as Kalman filter [7], particle filter [8][9] and so on. This type of tracking methods mainly focus on designing an effective motion model and lacks of the discussion of a robust appearance model, thus, leads to an unstable tracking performance. Second, tracking methods based on online classifiers (usually called discriminative trackers) treat the tracking problem as a local detection problem, which aims to distinguish the tracked object from its local surroundings and learn robust online classifiers to capture appearance changes of both object and background during the tracking process. Thus, many classical and state-of-the-art machine learning algorithms can be used to slove the tracking problem, including support vector tracking (SVT) [10], ensemble tracking (EST) [11], online boosting tracking (OBT) [12][13], semi-supervised boosting tracking (SemiBT) [14], multiple instance learning (MIL) [15], tracking-learning-detection (TLD) [16], to name a few. However, this kind of trackers usually achieves not good performance in terms of accuracy since the number of collected positive and negative samples is limited in the tracking process.

During the tracking process, tracking methods based on template matching search for a most likely image region being of the highest similarity or the smallest distance to the tracked object. In 1981, Lucas and Kanade [17] propose an iterative image registration method, which is the basis of the optical flow tracking algorithm. In 2003, Comaniciu et al. [7] present a kernel-based tracking framework, which exploits a spatial kernel function to measure the similarities between the tracked object and candidates and uses the Mean Shift method to achieve a fast matching. In 2008, Ross et al. [18] propose an incremental visual tracking (IVT) method based on online subspace learning, which learns an incremental principle component analysis (PCA) subspace in an online fashion. The IVT method is able to handle the illumination variation and pose change due to the PCA assumption, but is sensitive to outliers (such as partial occlusion and background clutter). Kwon and Lee [19] adopt a sparse PCA method to select multiple color and edge templates, which is robust to many challenging cases (such as illumination variation, scale change, pose change, non-rigid motion and so on). However, this method is too complex to be applied in real tracking problems.

Recently, the sparse representation theory has been widely used in the fields of image processing and computer vision [20]. Motivated by the success of sparse representation for face recognition [21], Mei et al. [22] introduce sparse representation into the tracking filed and propose a L1 tracker, which uses a series of object and trivial templates to sparsely represent the tracked object. After that, many researchers improve the original L1 tracker in terms of both speed and accuracy. Based on the original L1 tracker, Mei et al. [23] compute a minimum error bound of each candidate by using the L2-norm minimization problem and discard those candidates with large reconstruction errors, which effectively reduces the number of complicated L1-norm minimizations in each frame. Then, Bao et al. [24] introduce an accelerate proximal gradient (APG) method to speed up the solution process of the L1-norm minimization. Besides, a lot of researchers have also attempted to improve the L1 tracker in different aspects, such as considering both positive and negative templates [25], adopting different optimization techniques [26], modeling the relationships among different candidates [27], combining subspace and sparse representation models [28][29] and so on.

Inspired by the “template matching”-based trackers (especially the “sparse representation”-based ones), this paper presents a novel tracking method based on the proposed relaxed inverse sparse representation model. The contributions of this work are mainly four folds. First, we treat the tracking problem as an inverse sparse representation process and propose a novel objective function to depict this idea. The proposed objective function integrates the sparse representation, relaxed representation and weight prior in a unified framework. Second, we design an iteration algorithm to effectively solve our objective functions based on three propositions. In addition, the proposed representation model is embed into a Bayesian inference framework for designing a robust tracker, in which a simple template update method is introduced. Finally, many experiments are conducted on some challenging image sequences to compare the proposed tracking method with other state-of-the-art trackers. The experimental results demonstrate that our tracker achieves good performance than other tracking methods.

The rest of this paper is organized as follows. Section 2 introduces the proposed tracking framework with the inverse sparse representation method, including motivation, problem formulation, objective function and so on. In Section 4, some experiments are conducted to evaluate the proposed tracker and compare it with many state-of-the-art algorithms. Finally, Section 5 concludes this paper.

 

2. Object Tracking based on Relaxed Inverse Sparse Representation

2.1 Motivation

This paper is motivated by the recent success of sparse representation in visual tracking [19] and object recognition [27] [28]. The basic idea of the original L1 tracker [19] is that each candidate d can be sparsely represented by a set of object (T) and trivial templates (I), in which the coding coefficient vector can be solved by the following L1 minimization problem, i.e.,

in which is the coding coefficient vector and encourage a sparse solution. However, in the tracking problem, it requires to maintain many candidates to approximate the probability of the object’s state. Thus, it needs to calculate many L1 minimization problems, which will make the tracker very slow. In recent, some researchers have studied another line in sparse representation, sparsity induced similarity [27]. This research line treats the solution obtained by sparse representation as a similarity measurement. From this view, the tracking problem can be viewed as an inverse sparse representation process, i.e., representing the template by using a set of candidates rather than coding each candidate by using templates. The core idea of the inverse sparse representation process is illustrated in Fig. 1.

Fig. 1.The overall framework of the proposed tracking algorithm.

We note that the coefficient vetcor can be viewed as the similarity degrees between different candidates and the object template according to the sparsity induced similarity framework[27]. Compared with the original L1 tracker [19], this idea merely sloves one L1 minination problem (i.e., ) to determine the likelihood values of different candidates. In the tracking process, the target’s appearance may experience some unexpected noises or outliers, such as partial occlusion, local illumination variation and so on. However, the holistic representation in equation (2) cannot deal with this dilemma. Thus, to alleviate this problem, we divide the observation patch (for both object template y and candidates D into M fragments), and then the representation process in equation (2) can be converted into M sub-processes ym ≈ Dmxm. But in this fragment scheme, different fragments are treated to be of equal importance and therefore cannot make the tracker aviod the effect of outliers. To emphasize the differences of different fragments and avoids the effect of outliers (such as partial occlusion), the representation coefficient vectors should be similar but not same [28]. This idea can be described by using the term , which is a weighted average vector to make different coefficient vectors be similar and are weights of different fragments for depicting the differences among coefficient vectors. Based on the above-mentioned discussions, we present the objective function of the proposed representation model in the next subsection. Based on the proposed model, i.e., the relaxed inverse sparse representation, the overall framework of the tracking method is illustrated in Fig. 1.

2.2 Problem formulation

In this work, we cast the tracking problem as a relaxed inverse sparse representation problem, the core objective function of which can be defined as

where the first term denotes the reconstruction error based on coding coefficient vectors, the second term is the L1 regularization term that aims to encourage the coding coefficients being sparse, the third term is the relaxed term that measures the consistency (or dis-consistency) among different coding coefficient vectors, and the last one is a Kullback–Leibler divergence term that makes sure the weight vector w should be more similar a reference weight vector w' (that is, w' provides a prior on w), λ, μ, η are parameters to balance different terms.

We note that can be viewed as an overall sparisty induced similarity measure between the template y and the candidate dictionary D. The optimal can be obtained by optimizing the following optimization problem,

We note that the objective function (3) is a unified framework to combine several key components, such as the inverse sparse representation process, the fragment scheme and the weight adaptive scheme. The inverse sparse representation process makes the tracker use different candidates to sparsely represent the object template, in which the representation coefficients can be viewed as the observation likelihood values of those candidates. The fragment scheme makes the inverse sparse representation process be in the fragment level rather than holistic level, which effectively exploit the differences of different fragments in representation process. In addition, the weight adaptive scheme could determine the important degrees of different fragments in an online manner. Thus, the combination of these key components in a unified objective function is able to benefit obtaining accurate observation likelihood during the tracking process. The effectiveness of these components can be demonstrated in the experiment section.

2.3 Problem solution

To the best of our knowledge, there is no closed-from solution for the optimization problem (4). So, we propose an iteration algorithm to estimate , and w based on the following three propositions.

Proposition 1: Given the optimal solutions and w*, the optimal coefficient vectors can be obtained by solving individual sparse coding problems, which can be solved by the LASSO method.

If the optimal values and w* are given, the minimization of the problem (5) can be converted into the following optimization problem,

where the objective function is defined as

It is easy to see that the objective function (7) can be viewed as a sum of M individual functions, i.e., , where

where , I is an indentify matrix. From equation (7), we can see that the optimization problem (6) can be modified as M sub-optimization problems, i.e., , each of which is a standard LASSO problem that can be optimized by using the SPAMS (SPArse Modeling Software) package (http://spams-devel.gforge.inria.fr/).

Proposition 2: Given the optimal solutions and w*, the optimal average coefficient vector can be solved by a simple weighted average operator.

If the optimal values and w* are given, the optimal average coefficient vector can be obtained by solving the following problem,

where the objective function is defined as . This problem is a standard least squares problem and its closed-from solution can be obtained by setting the derivation to zero.

So, the optimal solution of the average coefficient vector can be obtained by . Due to the non-negativity of the coefficient vectors , the optimal average vector is also negative.

Proposition 3: Given the optimal solutions and , the optimal weight vector w* can be obtained M by product operators separately.

If the optimal values and are given, the optimization problem (4) can be converted into the minimization problem (11).

in which the objective function that is a sum of M sub-equations . Thus, the optimal weight vector w* can be obtained by solving M sub-problems . By setting the derivation ∂J3,m(wm)/∂wm to zero, the optimal value w*m can be obtained.

Thus, we can obtain , the physical meaning of which is very intuitive. The former component measures the inconsistency between the coding coefficient vector and the average coding vector ; and the later one makes the solution be similar with the reference weight vector.

By the above-mentioned three propositions, the optimization problem (4) can be solved iteratively. The iteration algorithm for solving the optimization problem (4) is presented in Table 1. The iteration operations will be terminated when a stopping criterion is met, e.g., the difference of the average coefficient vector () or a maximal number of iteration steps.

Table 1.The iteration algorithm for solving the optimization problem (4)

2.4 The tracking framework

Based on the proposed relaxed sparse representation model, we develop a tracking algorithm by using the Bayesian inference framework. In general, object tracking can be casted as a Bayesian inference problem in a hidden Markov model. Given continuous observation image patches Dt = {d1,d2,…,dt} up to the t-th frame, the aim is to infer the hidden state variable zt recursively, i.e.,

where p(ztlzt-1) stands for the motion model between two consecutive frames and p(dtlzt) is the observation model that estimates the likelihood function for each candidate. The overall framework of the tracking method has been illustrated in Fig. 1. Similar to [16], the affine transform with six parameters is adopted to depict the motion model p(ztlzt-1), in which xt={xt,yt,θt,st,αt,φt} denote the x, y translations, rotation angle, scale, aspect ratio, and skew in the t-th frame. Then the random walk process are adopted to describe the state transition, i.e., p(ztlzt-1) = N(zt;zt-1,ψ), where is a diagonal covariance matrix.

In the t-th frame, we solve the following optimization problem,

where denotes the template, stands for the m-th sub-dictionary related with N candidate states, is the weight vector of the (t-1)-th frame. After solving equation (13) by the iteration algorithm in Table 1, the likelihood of each candidate can be measured by , Then the optimal state inferred by the Bayesian framework. We also note that the reference weight vector is initialized as and then is updated frame by frame.

During the tracking process, it is necessary to update the template of the tracked object to capture the appearance changes. After obtaining the optimal state z*, we extract its corresponding image patch d* and update the target template in a fragment-based manner, i.e.,

in which η=0.95 is a update rate and ε=0.1 is a predefined threshold.

 

3. Experiments

In this paper, we implement our tracker in the MATLAB platform, which runs 18 frames per second on a PC machine with Intel i5-M560 CPU (2.67 GHz) with 2 GB memory. For each image sequence, the bounding box of the tracked object is manually labeled in the first frame for initializing our tracker. The affine parameters are set to be ψ=diag(4,4,le-2,5e-3,le-3,le-3) for sampling candidates in each frame. To model the appearance feature of each patch, we firstly resize each patch to 32×32 pixels and then divide it into 4×4( = 16) fragments. 600 particles are used to balance the effectiveness and efficiency. The parameter for sparse regularization λ is set as 0.1, and other parameters are set to be μ = 0.01 and η = 0.01.

We adopt many challenging video clips to evaluate the proposed tracker in comparison with ten recent trackers, including five sparisty-based tracking algorithms (accelerated proximal gradient L1 (L1) [21], multitask tracking (MTT) [24], local sparse appearance tracking (LSAT) [29], two view sparse represenation (TVSR) [35]) and other six methods (fragment-based tracking (Frag) [30], incremental visual tracking (IVT) [15], multiple instance learning (MIL) [12], visual tracking decomposition (VTD) [16], tracking-learning-detection (TLD) [13] and probability continuous outlier model (PCOM) [34]). The challenging factors of these video clips include partial occlusion, illumination variation, scale change, pose change, background clutter and so on. Both qualitative and quantitative results are presented as follows.

3.1 Qualitative Results

Fig. 2 and Fig. 3 provide some qualitative results to compare our tracker with other four tracking methods, including two baseline algorithms (IVT and MIL) and two sparse ones (L1 and MTT). In Fig. 2, our tracker is compared with other tracking methods when partial occlusions occur during the tracking process. It can be seen from this figure that our tracker performs better than other algorithms in handling these cases due to the fragment-based scheme and relaxed sparse representation. The IVT method is very sensitive to partial occlusions as its adopted PCA representation model cannot depict outliers (e.g., Occlusion2 and Caviar1). The MIL method uses the multiple instance learning technique to deal with the ambiguity of positive samples, however this algorithm cannot handle the case when the tracked object occluded by other objects with similar appearance (e.g., Caviar1 and Caviar2).

Fig. 2.An illustration of selected tracking results of different trackers in partial occlusions.

Fig. 3.An illustration of selected tracking results of different trackers in other conditions.

Although the FragT method also uses the fragment-based object representation to handle partial occlusions, it performs poorly in more complex conditions (e.g., Occlusion2 and Caviar2). Both the L1 and MTT trackers are motivated by sparse representation, which adopts a series of trivial templates to model outliers (i.e., partial occlusions) explicitly. But they also achieve not good performance as the traditional sparse representation model cannot effectively model the relationships among candidates (e.g., Caviar1 and Caviar2). Besides, Fig. 3 show representative results on other five challenging videos. It can be seen from this figure that the proposed tracking algorithm achieves good performance in dealing with pose variation (DavidIndoor), illumination change (DavidIndoor, Singer1, Car4) and cluttered background (Car11 and Deer). In addition, Fig. 4 demonstrates the tracking results of different algorithms on two challenging image sequences in the PETS dataset (http://www-prima.imag.fr/PETS04/index.html).

Fig. 4.An illustration of selected tracking results of different trackers in two challenging sequences from the PETS dataset.

3.2 Quantitative Results

To evaluate our tracker and other trackers quantitatively, we use two popular criteria, i.e., center location error (CLE) and F-meansure (F). The center location error is defined as the Euclidean distance between the center location of the ground truth and the center location obtained by a tracker. It is obvious that a good tracker intends to obtain small CE values in the test sequences. However, this rule does not consider scale and rotation changes. In addition, we also adopt the F-meansure (F) [5] to further evaluate different trackers (the F rule is defined in equation (15)).

in which RG and RT are ground truth and the tracked bounding boxes respectively, Pr(RT,RG) = area(RT∩RG)/area(RG), Re(RT,RG) = area(RT∩RG)/area(RT), and area(X) denotes the area of the region X. Table 2 and Table 3 report the average CLE and F values for different tracking algorithms on the test image sequences, from which we can see that the proposed RISR tracking method achieves better performance than other state-of-the-art trackers.

Table 2.Average center location errors (ACLE) of different trackers.

Table 3.Average overlap rates (AOR) of different trackers.

3.3 The effects of parameters and components

In this subsection, we investigate the effects of two critical parameters, the sparsity regularization parameter λ and the fragment number M. First, the choice of parameter λ is a critical parameter that controls the sparsity level. Fig. 5 (a) demonstrates the tracking performance (i.e., F-measure) with different λ values. If λ is too small, the solution will be too trivial and not sparse, which will introduce too much noise in inferring the similarities of different candidates. On the other hand, if λ is too large, the sparisty will be over-emphasized, which may lead to select a very small number of candidates. Thus, the tracking performance is also not good. Second, the number of fragments is also very important to our tracker (the performance of different fragments is shown in Fig. 5 (b)). If the fragment number is very small, the tracker cannot achieve good performance as it is not able to model outliers (such as partial occlusions or local illumination changes) effectively by using a very small number of fragments. For another, if the fragment number is too large, the size of each fragment will be too small to capture sufficient visual information, which will also leads the tracker’s performance is not satisfying. Based on the reported results in Fig. 5, we set λ = 0.1 and M = 16 as default values in this work.

Fig. 5.The effects of critical parameters.

In addition, Fig. 6 show the effects of different components, in which L1 denotes the original L1 tracker [21], ISR denotes the tracking method based on inverse sparse representation process without using the fragment scheme, FISR is the fragment-based ISR tracker, and RISR indicates the final tracking method that combines the inverse sparse representation, fragment scheme and adaptive weight scheme within a unified framework. From this figure, we can see that both fragment and adaptive weight schemes facilitate the improvements of tracking performance.

Fig. 6.The effects of different components.

 

4. Conclusion

In this work, we develop a novel online object tracking algorithm based on our relaxed inverse sparse representation algorithm. First, we treat the tracking problem as an inverse sparse representation process, in which a given template of the tracked object can be sparsely represented by the candidate samples in each frame. In addition, we introduce a relaxed constraint term to make the inverse sparse representation process be more flexible and a weight prior as a reference of the relaxed term. Then, a novel tracking method is designed based on the proposed representation model and a simple online update manner within the Bayesian framework. Last but not least, we conduct many experiments to compare our tracker with other recent trackers. The results show that our tracker performs better than other compared tracking algorithms.