Efficient Tracking as Linear Program on Weak Binary Classifiers ? Michael Grabner1,3 , Christopher Zach2 , and Horst Bischof3 1

3

Microsoft Photogrammetry Graz, Austria [email protected] 2 Department of Computer Science University of North Carolina, Chapel Hill [email protected] Institute for Computer Graphics and Vision Graz University of Technology [email protected]

Abstract. This paper demonstrates how a simple, yet effective, set of features enables to integrate ensemble classifiers in optical flow based tracking. In particular, gray value differences of pixel pairs are used for generating binary weak classifiers, forming the respective object representation. For the tracking step an affine motion model is proposed. By using hinge loss functions, the motion estimation problem can be formulated as a linear program. Experiments demonstrate robustness of the proposed approach and include comparisons to conventional tracking methods.

1

Introduction

Considering object tracking as a binary classification problem has been well studied in recent years [2, 7, 16, 1, 18]. The principle idea is to learn a discriminant function which distinguishes a particular object from non target regions. For learning such a function many techniques have been proposed in machine learning literature whereas in particular SVMs and ensemble methods have demonstrated success within object tracking. While the procedure of learning such classifiers has been well studied there is still the question how to integrate them in tracking applications for the purpose of parameter estimation of motion models with higher degrees of freedom. To estimate how an image region has moved from one frame to the next, the naive way is to apply the classifier in an exhaustive manner over a search region [7, 16, 2] to create a likelihood map where the peak is typically interpreted as the objects new location. However, for tracking typically we would like to handle several degrees of freedom of the target object, e.g. affine deformations for planar objects, where exhaustive search simply becomes to cumbersome. Williams ?

This work has been sponsored by the Austrian Joint Research Project Cognitive Vision under projects S9103-N04 and S9104-N04.

2

et al. [18] overcomes this limitations of exhaustive search by training so called displacement experts [8, 13] for multiple directions of the target region. In particular, shifted samples of the target object are collected for training multiple Relevance Vector Machines (RVMs) which are distinctive for particular directions and finally are used for making predictions even for more than two degrees of freedom. Probably the work most related to ours is Avidan’s Support Vector Tracking (SVT) [1]. SVT integrates SVM as a powerful detection method for the purpose of object tracking. In particular, a pre-trained SVM is used in combination with the optical flow constraint which maximizes, instead of minimizing the sum of squared pixel differences, the SVM score between two successive frames. Similar to the idea of Avidan, we propose a technique to combine powerful ensemble methods [4] with optical flow based tracking. Ensemble methods are characterized by linearly combining individual expert decisions. The goal is to select a set of weak classifiers with a certain amount of diversity in their decisions to increase overall performance. Different methods have been proposed for constructing such an ensemble where bagging [3] and boosting [5] probably belong to the most widely used ones. These methods have been successfully applied to object detection tasks [17] as well as recently for tracking tasks [7, 2]. However, they are typically limited to translational movement estimation. In this work we propose to create weak hypothesis from grayvalue responses at random pixel positions. Simple pixel comparisons have been widely used in many applications such as pattern matching or classification tasks. Pele [15] uses simple comparisons of pixel intensities for pattern matching and illustrates results which are competitive to well established local image descriptors. Furthermore, the Census transform [19] proposed for dense matching has a similar idea. Pixels are compared with a simple greater than comparison of a center pixel to its neighborhood to generate a bit ordering of the neighborhood and though achieving invariance to changes in gain or bias. Recently, in [10, 14] pixel comparisons have been applied for real-time keypoint recognition. Simple binary tests based on gray value comparisons are used for solving large multiclass problems (with classes greater than 100) and again - robustness and effectiveness of these methods are mainly due to the employed pixel-pair features. The basic advantage of building weak hypothesis from pixel-based features is that it enables to apply the idea of linearization from Lucas and Kanade [12] and thereby to perform tracking through optimizing the classifier output. In particular, we illustrate the concept for tracking affine motions which can be replaced by simpler or more complex warping functions. The tracking step itself is formulated as linear program wheres weak binary classifiers form the basis for the optimization step. The outline of the paper is as follows. In Section 2 we present how to fuse an ensemble classifier with optical flow and formulate the problem of displacement estimation within linear programming. Section 3 presents processed sequences and the benefit versus a conventional sum-of-squared differences tracking based approach.

3

2

Affine Tracking with Pixel-Pair Features

This section describes our approach for object tracking by combining learned weak classifiers with the optical flow constraint. If (u, v) denote the displacement vectors, then the brightness constancy assumption of corresponding pixel yields to the optical flow constraint, ∂I ∂I ∂I =u +v . ∂t ∂x ∂y

(1)

Avidan [1] integrated this constraint into the support vector framework to enable tracking of a learned object representation stored in the support vectors. Due to the specific choice of the kernel, support vector tracking can be performed by solving linear systems of equations. Note, that [1] only addresses the estimation of pure translations and does not incorporate a more complex affine motion model. In this work we propose the combination of weak and simple classifiers with the optical flow constraint, which naturally yields to a linear program to determine the update of the motion parameters (see Section 2.2). Our underlying objective function to be minimized is a sum of hinge loss functions, which is more robust to outliers than the often used sum of squared differences. In combination with the underlying pixel-pair features, which are invariant with respect to gain changes, we consider our tracking approach very robust to a large range of illumination changes and occlusions. 2.1

Object Representation

The learned object representation is an ensemble of weak classifiers, hi , each associated with a pixel pair, and the corresponding voting weights αi > 0. A weak classifier hi is represented by two pixel positions, pi1 and pi2 , and an intensity difference threshold bi . The response of the classifier hi for a given image I and motion parameters A is then
hi (I, A) = sgn I(A(pi1 )) − I(A(pi2 )) − bi ∈ {+1, −1}, (2) where A(p) denotes the original position p transformed by the motion hypothesis A. Thus, the weak classifier returns a positive response, if the intensity difference of two pixels is larger than a learned threshold. Combining the weak classifiers results in a strong classifier, X αi hi (I, A). (3) i

Generally, the location of the tracked object can be determined by maximizing the response of this strong classifier over A for the provided image I: X A? = max αi hi (I, A), (4) A

i

4

which is a highly non-convex optimization task and therefore hard to solve. Exhaustive sampling is only feasible for low-dimensional parameter spaces for A, e.g. when searching only for translational motion. In order to allow affine motion tracking at a suitable speed, we use an optical flow based search instead of an exhaustive one. 2.2

Displacement Estimation through Linear Programming

In this section we combine the weak classifiers derived from pixel pairs with the optical flow constraint. The obtained formulation allows the optimization for the full affine tracking parameters. The current tracking hypothesis comprising the position and orientation of the tracked object is represented by the following affine matrix A   a11 a12 a13 A= . (5) a21 a22 a23 Thus, a (homogeneous) pixel position p in the reference image has the corresponding position A(p) = A · p in the current image. Recall from the last P section, that directly searching for optimal motion parameters A? = maxA i αi hi (I, A) is generally not feasible for real-time or interactive performance due to its high degree of non-convexity. The non-convexity has two sources: first, the images seen as a 2D function are usually not convex. Second, the thresholding induced by the sign(·) function in the weak classifier definition introduces additional local minima. The first source of non-convexity is addressed by locally linearizing the image intensities, since tracking is typically applied on video sequences. Hence, we can assume small changes in A for consecutive frames, i.e. A(t) = A(t−1) + ∆A(t) with ∆A(t) small. This allows us to employ a first order Taylor approximation for the current image I (t) , I (t) (p(t) ) = I (t) (p(t−1) + ∆p(t) ) D E ≈ I (t) (p(t−1) ) + ∆A(t) , ∇A I (t) (p(t−1) ) ,

(6)

where we use the shorter notations p(t) = A(t) · p and ∆p(t) = ∆A(t) · p. Since the linearization of pixel intensities is only valid for a small neighborhood, the proposed tracking procedure is embedded in a coarse-to-fine approach. At this point, we incorporate the brightness constancy assumption typically utilized in optical flow determination. ∇A I is the derivative of the image intensities with respect to the affine parameters akl , e.g. ∂I ∂x ∂I ∂I = = x. ∂a11 ∂x ∂a11 ∂x

(7)

Obviously, the binary response functions hi are not convex and maximizing P α h is highly vulnerable to reach only local optima. The pixel intensity i i i ˜ i (I, A) := I(A · pi ) − I(A · pi ) − bi , are linear and therefore convex, differences, h 2 1 P ˜ but optimizing αi hi in conjunction with the Taylor approximation in Eq. 7

5

results in unbounded solutions (since the objective function is then linear with respect to the (unbounded) affine matrix parameters). Therefore, we employ a ˜ i , i.e. the new penalty different approach by applying a hinge loss function on h n o ˜ i (I, A) . Positive responses of a weak function is now fi (I, A) := max 0, −h classifier do not contribute to the objective function, whereas negative responses are basically weighted according to their confidence. Note that the hinge loss function is a convex function, but not differentiable everywhere. Nevertheless, as P it is derived in the following, the accumulated hinge loss function αi fi (A, I (t) ) can be optimized by linear programming. Explicitly, the objective function is now n o X X ˜ i (I (t) , A) . αi fi (A, I (t) ) = αi max 0, −h (8) (t) ˜ i (I, A) = I (t) (A · pi ) − I (t) (A · pi ) + bi . J (t) (A) is posWe introduce Ji (A) = −h 2 1 i itive if the tracked parameters violate the respective pixel feature. Inserting the ˜ i and incorporating the first order image intensity approximation definition of h (Eq. 7) yields n X X αi fi (A, I (t) ) = αi max 0, (9) i

D Eo (t) (t) Ji (A(t) ) + ∆A(t) , ∇A Ji (A(t−1) ) , with ∆A containing the unknowns. This minimization task can be solved using linear programming by introducing new unknowns Li (for the loss induced by the weak classifier i) with n D Eo (t) (t) Li = max 0, Ji (A(t−1) ) + ∆A, ∇A Ji (A(t−1) ) (10) for given ∆A. Minimizing Eq. 9 reduces now to the following equivalent linear program: X αi Li min ∆A,Li

i

Li ≥ 0 Li ≥

(11)

(t) Ji (A(t−1) )

D E (t) + ∆A, ∇A Ji (A(t−1) ) ,

which can be solved by standard linear programming methods. In order to obtain a meaningful solution even in the case that all Li are zero, we slightly extend the linear program above: X X min αi Li + ε Akl ∆A,Li ,Akl

i

k,l

Li ≥ 0 D E (t) (t) Li ≥ Ji (A(t−1) ) + ∆A, ∇A Ji (A(t−1) ) Akl ≥ ∆akl Akl ≥ −∆akl .

(12)

6

Akl represents |∆akl |, hence the objective function includes a L1 penalizing term on ∆A. ε is a small constant, which enforces zero updates if all weak classifiers are already satisfied by the preceding tracking parameters. If all hi (I (t) , A(t−1) ) return a positive response (i.e. Li = 0 for A(t−1) ), the obtained update ∆A is zero in this case. A feasible initial solution for ∆A, Akl and Li is given by ∆A ≡ 0, Akl ≡ 0, and Li is determined by Eq. 10, respectively. Further, all αi are positive and the Li are bounded from below, hence the optimal objective value is always bounded, too. Thus, a bounded optimal solution is always obtained. The update ∆A(t) for the affine parameters is the returned optimal solution for ∆A, and the new affine parameters are set to A(t) = A(t−1) + ∆A(t) . 2.3

Online Selection of Hypothesis

The tracking step has been formulated in a general way meaning that any ensemble learning algorithm (e.g. [9, 5, 6, 11]) can be applied for constructing the strong classifier and thereby bring a meaningful variety in the selected set of weak classifiers. Note that the learning algorithm itself is not the focus of this work but instead investigates the fusion of ensemble classifiers with optical flow based tracking. In our case we have decided for integration of an online boosting variant [6] which has recently also applied to tracking applications [7]. Due

Algorithm 1 Ensemble Classifier based Optical Flow Tracking Input: Frames I (1) , ..., I (n) and object region r(1) 1. Initialization: (a) Repeat for each pyramid level: P i. Initialize strong classifier H(x) = i αi hi (I, E) with αi = 1 2. For each new frame I (t) do: (a) Repeat for each pyramid level: i. Determine change in affine parameters ∆A:

X

min ∆A,Li ,Akl

αi Li + ε

i

X

Akl

k,l

ii. Update affine parameters: A ← A + ∆A (b) Repeat for each pyramid level: i. Generate random affine warps ii. Apply online boosting for updating strong classifier H(x)

to the usage of an online classifier we perform updates during tracking with positive and negative samples. For this purpose, we generate a set of random

7

affine transformations for making negative updates while the tracked region is assumed to be a positive sample. Movements larger than a few pixels result in large linearization errors and typically lead to tracking failures. For making the approach more robust against larger movements, we incorporate a pyramidal representation of classifiers which are then used on the different levels for displacement estimations respectively, see Algorithm 1. Therefore, on each image pyramid level a separate classifier is initialized for the target region. The movement is estimated at each pyramid level, starting from the lowest level which then provides the initial guess for the next level. Even if subsampling introduces errors, coarser levels of the pyramid provide a good estimate of the initial position for the next level and so helps to tackle the problem of larger movements. Further on, it increases robustness against image noise as well as reflectance or lighting changes. The upper bounds for the paramters of the motion model are identical for all levels.

3

Experiments

For tracking targets with the presented approach we manually selected the target object to be tracked. Afterwards pairs of pixels are randomly chosen over the target region. Different criteria for selecting pixels have been tested (i.e. points on edges or near edges), however it turned out, that random initialization is a good choice since it has the advantage not to limit the approach to certain object appearances. As mentioned before for each layer of the pyramid an individual classifier is trained to perform online feature selection. The strong classifier complexity has been set to 150 weak classifiers whereas the proposed feature pool provided 600 possible pixel comparisons. In addition the affine parameters between consecutive frames has been limited to a factor of 0.03 whereas the translational components do not exceed 3 pixels for a single layer (note that the settings of these parameters is not critical). The approach has been implemented in Matlab and runs with about 1 fps. Several sequences have been used for testing the approach. The first one presented within this section is shown in Figure 1. The object is accurately localized throughout the whole sequence. Figure 2 presents a sequence containing a higher amount of artifacts. Illumination changes as well as rather large inter frame motions are handled by the approach in a robust manner and the affine parameters are correctly estimated. The approach has also been tested on sequences which are typical for the purpose of autonomous driving, see Figure 3. In particular it shows a car entering a tunnel where shuttering of the camera complicates the tracking task. Due to the low resolution and the simple requirement of localization within image space, the affine warping function has been replaced by a translational model. As depicted in the lower row of the sequence, drastic appearance changes are handled. A comparison to a simple SSD tracker is presented in 4 whereas the SSD measure simply accounts for the same set of pixels which are considered by our approach. The explanation for making our

8

(a) Frame 8

(b) Frame 68

(c) Frame 100 (d) Frame 147 (e) Frame 200

Fig. 1. This sequence is available at http://esm.gforge.inria.fr/ESMdownloads.html and has been chosen in order to demonstrate the ability to track affine motions. The upper row presents the individual frames whereas the target itself is depicted in more detail below. The approach presents accurate localization of the target over the whole sequence.

(a) Frame 26

(b) Frame 37

(c) Frame 57

(d) Frame 91

(e) Frame 123

Fig. 2. Throughout the sequence appearance artifacts of the target object are present. Illumination changes as well as spots are handled by the approach and do not confuse the tracker when estimating the affine parameters.

approach more robust against the artifacts are simple the type of features in combination to the robust loss function used within the approach.

4

Conclusion

The paper has presented an approach which allows to apply ensemble classification techniques for the purpose of object tracking with motion models. Simple decisions resulting from intensity tests on pixel pairs are used as weak hypothesis which are finally combined to a strong classifier. The specific type of hypothesis

9

(a) Frame 26

(b) Frame 37

(c) Frame 57

(d) Frame 91

(e) Frame 123

Fig. 3. For this sequence a simple translational warping function has been used, since the target region is of very low resolution. Again strong changes in appearance are present and successfully handled by the approach.

(a) Frame 1

(b) Frame 57

(c) Frame 111

(d) Frame 200

Fig. 4. Comparison of simple SSD tracker (top) and our approach (bottom). Due to the specific loss function of our approach and the possibility to limit the amount of warping parameters the approach turns out to be more robust to standard methods.

allows us to fuse classical optical flow based tracking with an ensemble classifier and thereby makes it possible to use this classification techniques for displacement estimation of more complex motion models. The estimation of parameters of the warping function between successive frames is formulated as a linear program. The loss function used within the optimization step shows robustness which is demonstrated in several challenging sequences.

10

References 1. S. Avidan. Support vector tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(8):1064–1072, 2004. 2. S. Avidan. Ensemble tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(2):261–271, 2007. 3. L. Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996. 4. T. G. Dietterich. Ensemble methods in machine learning. In Proceedings International Workshop on Multiple Classifier Systems, pages 1–15, 2000. 5. Y. Freund and R.E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997. 6. H. Grabner and H. Bischof. On-line boosting and vision. In Proceedings Conference Computer Vision and Pattern Recognition, volume 1, pages 260–267, 2006. 7. H. Grabner, M. Grabner, and H. Bischof. Real-time tracking via on-line boosting. In Proceedings British Machine Vision Conference, volume 1, pages 47–56, 2006. 8. F. Jurie and M. Dhome. A simple and efficient template matching algorithm. In Proceedings International Conference on Computer Vision, volume 2, pages 544– 549, 2001. 9. J. Z. Kolter and M. A. Maloof. Dynamic weighted majority: A new ensemble method for tracking concept drift. In Proceedings International Conference on Data Mining, pages 123–130, 2003. 10. V. Lepetit. Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9):1465–1479, 2006. 11. N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and Computation, 108(2):212–261, 1994. 12. B. D. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings International Joint Conference on Artificial Intelligence, pages 674–679, 1981. 13. J. Matas and A. Hilton K. Zimmermann, T. Svoboda. Learning efficient linear predictors for motion estimation. In Indien Conference on Computer Vision, Graphics and Image Processing, pages 445–456, 2006. ¨ 14. Mustafa Ozuysal, Vincent Lepetit, Fran¸cois Fleuret, and Pascal Fua. Feature harvesting for tracking-by-detection. In Proceedings European Conference on Computer Vision, volume 3, pages 592–605, 2006. 15. O. Pele and M. Werman. Robust real time pattern matching using bayesian sequential hypothesis testing. To appear in IEEE Transaction Pattern Analysis and Machine Intelligence., 2008. 16. M. Tian, W. Zhang, and F. Liu. On-line ensemble svm for robust object tracking. In Proceedings Asian Conference on Computer Vision, pages 355–364, 2007. 17. P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proceedings Conference Computer Vision and Pattern Recognition, volume I, pages 511–518, 2001. 18. O. Williams, A. Blake, and R. Cipolla. Sparse bayesian learning for efficient visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27:1292–1304, 2005. 19. R. Zabih and J. Woodfill. Non-parametric local transforms for computing visual correspondence. In Proceedings European Conference on Computer Vision, volume 2, pages 151–158, 1994.