IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 5, MAY 2012

693

Improved Hand Tracking System Jing-Ming Guo, Senior Member, IEEE, Yun-Fu Liu, Student Member, IEEE, Che-Hao Chang, and Hoang-Son Nguyen

Abstract—This paper presents an improved hand tracking system using pixel-based hierarchical-feature AdaBoosting (PBHFA), skin color segmentation, and codebook (CB) background cancelation. The proposed PBH feature significantly reduces the training time by a factor of at least 1440 compared to the traditional Haar-like feature. Moreover, lower computation and high tracking accuracy are also provided simultaneously. Yet, one of the disadvantages of the PBHFA is the false positive which is the consequence of the appearance of complex background in positive samples. To effectively reduce the false positive rate, the skin color segmentation and the foreground detection by applying the CB model are catered for rejecting all of the candidates which are not hand targets. As documented in the experimental results, the proposed system can achieve promising results, and thus it can be considered as an effective candidate in handling practical applications which require hand postures. Index Terms—AdaBoost, hand detection, hand tracking, hierarchical feature, skin color segmentation.

I. Introduction

H

AND POSTURES are powerful means for communication among humans. They facilitate a recipient to understand what a speaker is communicating. In daily life, some common postures are also used, such as “Stop,” “OK,” “Yes,” and “No.” Many applications are designed by using the motion of hand, such as human–computer interfaces, robot control, and communication with the deaf. Yet, to use the motion of hand for supporting visual aspects of interaction, it is necessary to track the hand in real time. In general, the main challenge of building a real-time system is to maintain a good balance between the accuracy and the processing time. The objective of these applications requires fast detection to process information carried by the hand postures while maintaining a good detection rate. In particular, the hand tracking is rather difficult because most of the backgrounds change across frames, and the hand shape is a complex object to detect. Kolsch and Turk [1] proposed a new method to detect a hand using image cues obtained from the optical flow and a color probability distribution. The pyramid-based Kanade–Lucas–

Manuscript received January 23, 2011; revised May 24, 2011 and August 12, 2011; accepted September 22, 2011. Date of publication December 1, 2011; date of current version May 1, 2012. This work was supported by the National Science Council, Taiwan, under Contract NSC 100-2221-E-011-103MY3. This paper was recommended by Associate Editor J. Luo. The authors are with the Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106, Taiwan (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TCSVT.2011.2177192

Tomasi (KLT) feature tracking was employed as their first modality because it shows excellent performance on quickly moving rigid objects, and can be processed very efficiently [2], [3]. The method can deal with dynamic backgrounds and light condition changing. Moreover, it can follow a rapid hand movement despite arbitrary finger-configuration changes. However, the color of a hand must be learned with a normalized-RGB histogram, which is contrasted to the background color in the image around the hand. It means that there is no other part with skin color from the same person appearing in the reference background area. Thus, this approach cannot classify wooden objects that are not within the reference background area during learning. As a result, the wooden objects will be considered as foreground. Zhu et al. [4] introduced a novel statistical approach to hand segmentation based on Bayes decision theory. The Gaussian mixture model with the restricted expectation and maximization algorithm [5] were used to build the color models of a hand and background for a given image. These models can classify each pixel in an image as either a hand pixel or a background pixel. The advantage of this approach is able to segment hand region from various backgrounds and light conditions with unknown color of a hand. Unfortunately, the method proposed in [4] cannot distinguish multitargets in an image and the error rate of 11.5% is a bit too high. Moreover, it takes about 0.1 s for each image on a personal computer platform, which does not support real-time scenarios. Binh and Ejima [6] applied skin color tracking for face detection to extract the hand region by separating a hand from a face. An image is classified into regions according to the human skin color by thresholding with a proper threshold value. Subsequently, the face region is detected and removed to reserve the hand region. By doing this, the hand regions of each input image can be extracted in real time. Other skin models such as [7] can be embedded into the above systems to achieve good skin segmentation. Nevertheless, these approaches have the same problem as those methods using human skin detection; they cannot distinguish between skin regions or some objects with similar color or shape as the hand. Thus, this is not a reliable modality, which requires users to wear long-sleeve shirt to avoid the problem of misclassifying the arm region as a part of the hand. Other approaches were proposed to use a learning-based method such as AdaBoost, Gaussian mixture model (GMM), and SVM [8]. AdaBoost is a well-known algorithm to maintain a good classification capability. With this approach, choosing feature for classification is highly important. Local binary pattern (LBP) [9] is one of the powerful features with low

c 2011 IEEE 1051-8215/$26.00 

694

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 5, MAY 2012

computation. Its grayscale invariant characteristic has coped with many of practical computer vision problems. Another feature used in Chen et al.’s work [10], called Haar-like feature [11], was adopted for hand detection. They collected 1712 positive samples (including four different postures) with different scales, and 500 negative samples (only backgrounds) for experiment. Compared with other methods which operate on multiple scales, this approach has high accuracy, which reduces the processing time by using the integral image. However, as described in their paper [10], the positive samples used for training do not contain any complex background, and the results show that the detection is simply on uniform background. Thus, the background subtraction is required before detecting, and the Gaussian filter is employed to reduce noise. Thus, the overall performance of the system highly relies on the subtraction. Just et al. [12] also employed AdaBoost for both of the hand classification and recognition. An additional feature, called modified census transform (MCT), was used to transform the original image into another feature space using 3 × 3 kernels and then comparing to its neighborhood. Then, the two-class strong classifier of the AdaBoost is trained to distinguish the posture and nonposture. The training and testing sets include images of size 30 × 30 for ten postures. However, in the experiment, images are cropped and the posture is put perfectly centered in each image, and this limitation does not guarantee a good detection for different scales when scanning image in real time. Donoser and Bischof [13] applied appearance-based approach to combine a state-ofthe-art interest point tracker with efficiently calculated color likelihood maps. First, the color distribution of the hand is built by using the Gaussian model. Second, the color distribution of the hand is combined with maximally stable extremal region (MSER) detector. The MSER is one of the good interest region detectors in computer vision proposed by Mikolajczyk– Schmid [14]. The color likelihood map is used to compute ordered set for each pixel. Based on the pixel ordering, MSERs connect regions which can be detected in any image and the hand region can be segmented with high accuracy. After detection phase, the system tracks the hand based on the detected targets in frame. However, the system cannot deal with the nonhand target that is similar to the color model. In this paper, a hand tracking system is proposed to combine the novel pixel-based hierarchical-feature for AdaBoosting (PBHFA), skin color detection, and codebook (CB) foreground detection model to locate a hand in real time. Subsequently, the tracking system is employed to trace the hand as a moving object. As a result, the system can achieve a high detection rate and accurate tracking while maintaining a fast processing speed with the capability to deal with problems in fields of hand detection and object tracking. The remainder of this paper is organized as follows. The proposed system is introduced in Section II. Section III describes the tracking method. Experimental results are presented in Section IV. Finally, Section V draws the conclusions. II. Proposed Hand Detection System One of the differences between hand detection and face detection is that most of the positive samples officially provided

Fig. 1. Examples of the background influences. (a) Background’s variation at a specific position. (b) False detected results.

in hand detection [15] are with complex background behind the objects of interest. Yet, most of the official positive samples of face [16] contain only face area; as shown in Fig. 1(a), most of the positive samples for training contain various backgrounds as indicated with the green squares and these areas are considered as a part of the object. It causes the false positive as shown in Fig. 1(b), and thus reduces the tracking rate. Nonetheless, the appearance of background in the positive training samples can benefit the proposed method to gain the advantage of detecting hand against complex background. This is the main difference to other methods when most of the former approaches tried to separate the pure hand region from background, which made their systems not to solve the complexity background problem effectively. Thus, the proposed hand tracking system employs the combination of pixel-based hierarchical (PBH) feature, skin color segmentation, and CB background subtraction as organized in Fig. 2 to yield effective tracking performance. Fig. 3 shows the advantages of proposed system to reduce the false positive. A. Proposed PBH Features The Haar-like features are linear combinations of the simple rectangle features. However, in a subwindow of size M × N, there can be a large number of possible features (in total 816 246 when M × N = 36 × 36), which makes the training process time consuming. Although the Haar-like features have satisfactory detection results, the tedious training time can make specific environment feature adaptation infeasible. The core problem of the Haar-like features is that most of the

GUO et al.: IMPROVED HAND TRACKING SYSTEM

695

Since a hand always appears at the center of the used training samples, the probabilities of the hand presence have obvious tendency (0 or 1). Conversely, backgrounds are unpredictable, thus the probabilities of the background are close to 0.5. This probability characteristic is catered for the order table O(x, y) to obtain the useful features in this paper. Enter value (1 ∼ M × N) into O(x, y) with the probability value in position (x, y). This means, when the probability of position (x, y) is the maximum, set O(x, y) = 1, and O(x, y) = 2 when the probability in position (x, y) is the second highest, and so forth. Step 5) All possible features Fj in a M×N subwindow can be produced according to the order table O(x, y). First, the summation of the pixel values in the subwindow of a training image is calculated and denoted as sum. The feature values Fj is computed as Fig. 2. Structure of the proposed hand detection using PBH, skin color segmentation, and CB model.

Fj = sum −

 k=j 

 Xi (x, y), where O(x, y) = k

(1)

k=1

Fig. 3.

False positive reduction with the proposed system.

features are not good candidates for practical use, and thus it is highly desirable to produce an alternative set of candidate features for the corresponding structure. Many researchers have indicated that most features in a subwindow do not demonstrate good discrimination ability in classifying face and nonface patterns. Only few best features are sufficient in achieving good detection results, and the AdaBoost is employed to select those few best features from a huge number of features. Unfortunately, this procedure takes a long training time, and thus the PBH features are proposed to reduce the number of features. The PBH features can significantly reduce the training time for hand detection, while maintaining good detection results. The procedure of the PBH feature generation is described as follows. Step 1) Given a training positive and negative samples Xi , Xj of size M × N. Step 2) Process the samples with the histogram equalization to reduce the influence of lighting effect. Step 3) Calculate the average value Ti of each sample Xi and use Ti as a threshold for binarizing the corresponding Xi to yield a binary image Bi . Step 4) Given Bi , calculate the probability of black pixel occurrence at each position (x, y) and obtain P(x, y).

where Xi (x, y) denotes the pixel value of the training image Xi and Xj at position (x, y). Step 6) Finally, these features and the training image Xi and Xj are fed into the AdaBoost algorithm. Each PBH feature can be considered as a weak classifier. The number of all possible features in an image of size M × N should be smaller or equal to M × N, since some of the probabilities obtained in Step 3 may be identical. Nevertheless, these features are the best which provide excellent capability for classification. All of the weak classifiers use samples and AdaBoost for training phase. The input images cooperate with the order table from the training phase for generating the corresponding feature values. The strong classifier is built by the AdaBoost algorithm. Subsequently, to detect various sizes of hand with the subwindow of a fixed size of M × N in detection phase, the input frames will be resized by a scale factor, an example is shown in Fig. 4. By using the PBH features, when a subwindow is examined, some previously computed pixels in (j − 1)th weak classifier are not required to be recomputed in jth weak classifier. In other words, only new pixels in jth weak classifier are required to be considered. The new feature value can be obtained by removing the new points in current weak classifier as shown in Fig. 5, in which the black points represent new points in the current weak classifier and the gray ones are those previously computed. The main point is that the training time is thus extremely shorter than that of the Haar-like features, and is able to achieve good detection performance. B. AdaBoosting for Real-Time Hand Detection The aim of boosting is to improve the classification performance of any given learning algorithms. First, all input images are fed into the training. This phase includes many classifiers; each weak classifier ht is made from a feature ft and a threshold θt such that the minimal error εt is produced

696

Fig. 4.

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 5, MAY 2012

Example of detecting hands of various sizes.

Fig. 6. Skin color segmentation with complex background. (a) Original image. (b) Segmented result.

reduce the false positives. The range of the skin colors heavily rely on the lighting conditions, thus the HSV color model which is robust to lighting change is adopted for skin color localization. One of the most important advantages of this color model in skin color segmentation is that it allows users to intuitively specify the boundary of the skin color class in terms of the hue and saturation. In this paper, the ranges of the hue and saturation are set in between 0° and 5° and 0.23 to 0.68, respectively, as specified in [17]. Fig. 6 shows one of the segmentation results. D. Foreground Detection Fig. 5.

Strong classifier equivalent concept using PBH features.

when performing the classification as follows:  1, if pt ft (x) < pt θt ht (x) = 0, otherwise

(2)

where pt denotes the polarity used for indicating the direction of the inequality. In this case, the feature ft denotes the proposed PBH calculated by (1). For each feature ft , the training stage calculates all of the PBH features of the samples. The features are then sorted, and the best threshold value is used to yield the trained weak classifier. The best threshold can be obtained by dividing all the samples into hands or nonhand, and then look for an optimal threshold to induce the smallest classifier error. Second, the final hypothesis H(x)  is a combination of T weak classifiers H(x) = Tt=1 αt ht (x), where variable αt denotes the weight for each weak classifier (the details can be referred in [11]) found during boosting process. The strong classifier can be obtained by combining the selected weak classifiers from the AdaBoosting, and which is reorganized as follows: ⎧ T T  ⎨  1 : αt ht (x) ≥ 21 αt C(x) = (3) t=1 t=1 ⎩ 0 : otherwise. The strong classifier can make a binary decision to determine whether a subwindow is a hand or a nonhand. Notably, if the hand region is centered in the positive images of the training process, the detection results will be more accurate. C. HSV Color Space In the practical usage stage, a detected hand by the PBH– AdaBoost algorithm is further filtered by the skin color to

The first level of tracking is the detection of targets of interest. A successful approach toward this detection is the foreground–background segregation. Unfortunately, most of the background and foreground in the video sequence are nonstationary in practice, such as waving trees, rippling water, and light conditions. One of the popular methods in foreground detection is the mixture of Gaussian (MoG) [18], [19]. The main idea of the GMM is the construction of the background distributions. These distributions are used to classify the pixels to foreground and background. Yet, the MoG still has some disadvantages: the low learning rate makes it difficult to cope the sudden change problem. Conversely, slowly moving objects will be considered as background. Moreover, it normally causes high false positive rate. For these, Kim et al. [20] proposed the CB model for foreground detection. The advantages of the CB include fast processing speed, capability to handle scenes with dynamic backgrounds, and robust to handle illumination variations. Also, by using the CB, the background can be classified into many layers based on the applications. The concept of the CB is to train background pixel pixelwise over a period of time. Sample values at each pixel are clustered as a set of codewords. The combination of multiple codewords can model the mixed backgrounds. In this paper, the former technique CB is adopted for yielding better results of hand detection. Fig. 7 shows the foreground detection using the CB model. The white pixels indicate the foreground while the background is illustrated by black pixels. Yet, as it can be seen in Fig. 8, one disadvantage of the system is that the foreground will disappear when it stands still for a long time in front of the camera. To solve this problem, a “buffer” is employed to store the history of each tracking target. This buffer is updated frame by frame. If one target is detected and tracking, the value in buffer

GUO et al.: IMPROVED HAND TRACKING SYSTEM

697

Fig. 7. Example of foreground detection using the CB model. (a) Original image. (b) Detected result.

Fig. 8. Example of foreground classified as background when it stands still for a long time. (a) Original image. (b) Classified result.

associates to the frame that is set as 1, and it is set to 0 if there is no tracked target. When a hand stands still in front of the camera, at first, it is tracked and thus the corresponding value in the buffer is 1. Yet, the hand might be eliminated when it stands still after some frames. For this, the values in the buffer are summed up over frames, and the history of the target can trigger a decision made by the system to keep tracking or not. The length of the buffer depends on the number of frames for training the CB. In this paper, the CB module utilizes 200 frames for the training phase, and thus the buffer should be 200 frames. If the summation of the values in the buffer is greater than a threshold T , where 0 < T ≤ 200, it means that the target should be kept tracking even it is determined as background. Fig. 9 demonstrates a successful case by applying the strategy introduced above. In the first frame, the background is constructed to detect the foreground, and the hand target is located with the proposed system in the next frame. In the last frame, even the CB model considers the hand as background, the target is still detected.

III. Hand Tracking Methodology Hand tracking is an application in field of object tracking. In the proposed system, the hand tracking phase is the second step after hand detection. Many algorithms for objects tracking were presented in the literature. These methods can be divided into three categories [21]: point tracking (Kalman filter [22]), kernel tracking (mean shift [23] and KLT tracker [24]), and silhouette tracking (variation method [24] and condensation algorithm [25]). In this paper, the Euclidean distance is chosen to track the hand because of its simplicity and effective result in practice. The Euclidean distance between two points A(x1 , y1 ), B(x2 , y2 ) as shown in Fig. 10(a) is calculated as

Fig. 9. Tracking hand against (b) Foreground detection.

follows: Eucliddist(A, B) =

CB

model.

(a)

Processing

(x1 − x2 )2 + (y1 − y2 )2 .

image.

(4)

Assume that in current frame, two hands are detected. The center of each hand is denoted as Ci (xi , yi ), and the hand area is limited within a square with the parameter ri = Eucliddis(Ci , Boti ), where the variable Boti is denoted on the bottom-right corner of Fig. 10(b). The green line in Fig. 10(b) indicates the trace of the hand in previous frames. The tracker in frame t has two parameters: centert (x, y) and the distance dt . Now, the task is to determine center t (x, y) belonging to which hand. Because the detection was implemented in real time, the distance between the center of hand I in frame t and frame (t+1) is mostly less than ri . Thus, the distance disti (x, y) between Ci (xi , yi ) and center t−1 (x, y) is computed. If the minimum value of disti (x, y) is less than r1 , then Ci (xi , yi ) is the new point for tracking. According to Fig. 10(b), the satisfactory condition for tracking is dist1 (x, y) < r1 . Then, the centert (x, y) is assigned to C1 (x1 , y1 ). As a result, even a traced hand suddenly disappears; the system is still able to track the correct object after it appears again.

IV. Experimental Results In this paper, the public Sebastien Marcel’s hand posture database [15] is employed in our experiment, and some of the samples are shown in Fig. 11. In this database, six different postures are involved, including “A,” “B,” “C,” “Point,” “Five,” and “V,” and the numbers of each postures in the database are organized in Table I. However, the samples in both “Point”

698

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 5, MAY 2012

Fig. 11.

Six postures examples of Sebastien Marcel’s database [15].

Fig. 12.

Examples of (a) “Point” and (b) “Five” datasets.

Fig. 10. Hand tracking example. (a) Euclidean distance. (b) Tracking methodology. TABLE I Property of S´ebastien Marcel Database [15] Postures A B C Point Five V

Amount (No. of Samples) 1328 487 572 1395 654 435

and “Five” sets have a wide variety of shapes, and some of them are even in unreasonable shapes, as shown in Fig. 12. Thus, the two datasets are excluded in our experiments. In the reduced database, 80% of them are adopted for the training, and in total 200 PBH-based weak classifiers are employed for the following experiments. Moreover, the remained database (20%) is adopted for the following simulation. A. Hand Detection Before testing, all of samples are processed with the histogram equalization for resisting the lighting effect. For the testing samples, various sizes of samples are tested, such as 20 × 20, 24 × 24, and 36 × 36. The results showed that the training samples of size 36 × 36 can yield the best detection rate among the three different sizes, since a bigger sample can provide more features than that of a smaller one. Yet, bigger samples also require more training time for the classifiers, which reduces the contribution of this paper in yielding extremely low training time and fewer computations in detection. Based upon the above reasons, this size is set at 36×36 in our paper. In addition, the processing time also highly relies on the

TABLE II Training Time Comparison Among Three Different Features for Hand Detection (Suppose a Subwindow is of Size 36 × 36)

LBP feature [9] Haar-like feature [11] Proposed PBH feature

Possible Features for Training Training Time 1296 Around 45 min More than 100 000 More than 10 days Less than 1296 Around 10 min

complexity of the background. In our experiment, the detection time for uniform background is around 4 ms as opposed to the 5–20 ms for complex background. Fig. 13 shows some practical detection results. Notably, both of the uniform and complex backgrounds are involved in the experiments. Table II shows the training time comparisons among three different features, including the LBP feature [9], the Haar-like feature [11], and the proposed PBH feature. The combination of the LBP feature and the AdaBoost is similar to that of the Haar-like feature combining with the Adaboost. The LBP feature in fact is a decimal value obtained from the bit-stream formed by determining whether the value of the current position is greater or smaller than its eight neighbors. Likewise, the Haar-Like feature is a number yielded from the integral image calculation for the further AdaBoost training. Thus, the Adaboost can be easily applied for the LBP features. Since

GUO et al.: IMPROVED HAND TRACKING SYSTEM

699

TABLE III Computation Complexity Comparison in Testing Phase Among Three Different Features for Hand Detection (Suppose a Subwindow is of Size M × N ) Feature-Type Operation Number Suppose subwindow of size M × N (K denotes the number of weak classifiers)

LBP Feature [9]

Haar-Like Feature [11]

Proposed PBH Feature

15K (where 8K for multiplication and 7K for addition)

Integral image

Less than 2 × M × N (addition/ subtraction)

Suppose subwindow of size 36×36

3000 (where 1600 for multiplication and 1400 for addition; supposed 200 features are used)

(M × N − 1) + (M − 1) (N − 1) (addition/ subtraction) K weak classifier (suppose all are tworectangle feature ) 7K (addition/subtraction) Case 1: 3920 (suppose 200 features are tworectangle features) (addition/subtraction) Case 2: > 3920 (suppose three-rectangle or four-rectangle features are used) (addition/ subtraction)

Less than 2592 (addition/subtraction)

TABLE IV True-Positive Detection Rate Comparisons Among Three Different Features with Hand

“A” posture “B” posture “C” posture “V” posture

Fig. 13.

LBP Feature [9] Haar-Like Feature [10] Proposed PBH Feature 87.28% 87.52% 88.39% 90.08% 92.57% 92.25% 88.04% 90.54% 91.19% 86.77% 93.54% 92.77%

Detection results with the PBH features. Fig. 14.

the Haar-like feature creates a big feature pool for finding the weak classifier combination, much more training time than the proposed PBH by a factor of 1440 (10 days/10 min) is required. The number of the LBP features is directly affected by the subwindow size, yet due to the fact that the number of the PBH features is usually lower than the size of the subwindow (some of the probabilities of black points may be identical, as explained in the Section II-A), the training time is lower than that of the LBP feature, in which the feature number of the LBP is identical to the size of a subwindow. Table III documents the computational complexity comparisons “in testing phase” of the three different features when applied to practical detection. The upper part shows the complexity in terms of the subwindow size, M × N. The complexity of the Haar-like features is further divided

Hand tracking results.

into integral image evaluation and K weak classifiers value calculation. The lower part of this table shows the complexity when M×N = 36×36. The best scenario by using LBP feature and Haar-like feature are 3000 and 3920, respectively, when K (=200) weak classifiers are used. For the Haar-like feature, when three-rectangle or four-rectangle features are involved, the number of operations is higher than 3920. Conversely, the number of operations with PBH features is 2592, which is less than that of the LBP and Haar-like features. Table IV shows the correct detection rate comparisons among the LBP feature [9] with AdaBoost algorithm [11], Chen et al.’s method [10], which adopted Haar-like feature [11], and the proposed method. Although the recognition rates of these methods are close, the proposed method still provides the lowest training and detection time.

700

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 22, NO. 5, MAY 2012

appears again, the system recognizes it as the object of interest, and keeps tracking. Because the distance between the centers of a tracked object of consecutive frames is often very short, the system mostly can track the correct hand among many interfering targets. Table V summarizes the advantages and disadvantages among various methods in the field of hand tracking. Complex background is the key issue of the hand detection, and most of the former researchers tried to solve this problem. For this, the proposed system can be considered as a good candidate for providing both efficient processing time and detection accuracy. V. Conclusion

Fig. 15. Results of tracking methodology with multiobjects. (a) Frame 1514. (b) Frame 1546. (c) Frame 1584. (d) Frame 1619. TABLE V Capability of Dealing with Problems Among Various Methods

[1]

[13]

Proposed method

Advantages -Able to deal with complex background, light condition changes -Able to track in real time

-Able to segment hand region from complex backgrounds -Able to detect in real time -No skin color occlusion -Able to track in real time -No skin color occlusion -Able to detect with complex background -Short training time -High accuracy

Disadvantages -Background and color of the hand must be learned in training process -Cannot detect with background that is different from the reference backgrounds -Cannot deal with the nonhand target which is similar to the color model -False positive rate relies on the complexity of background

B. Hand Tracking The main objective of this paper is to track a hand. In this implementation, the Logitech webcam is used for testing in real time, and the captured images are of size 320 × 240 under 15 f/s with natural lighting conditions. The HP laptop with Core 2 Duo 2.4 GB and 4 GB RAM is employed as the platform. Fig. 14 shows the tracking results with indoor/outdoor environments. Also, the tracking methodology guarantee that only the hand of interest is tracked even there are many targets in the frame. Fig. 15 shows another scenario in which there are two targets in frame 1514, and the “small hand” is the object we are interested. In frame 1546, the “small hand” disappears from the scene, and it simply leaves the “big hand” as a target. Nonetheless, the system does not track with the “big hand” instead because it is not the tracked object in the previous frame. In frames 1584 and 1619, when the “small hand”

In this paper, a hand tracking system was proposed by using the PBHFA, skin color segmentation, and CB model. The goal of this paper was to use PBH feature to reduce the required training time and further reduce the required computation in tracking phase. According to the experimental results, the above tasks were achieved, meanwhile the tracking accuracy was still maintained in high level as that of the Haarlike feature. The superiorities of the PBH also induced some benefits for the applications. 1) Short training time: This feature makes the proposed features that can be applied for self-learning hand tracking system, since the proposed features only require little time for adapting with unusual positive hands. 2) Low computation complexity: It indicates that the proposed features algorithm can also be embedded on lower price systems. On the other hand, the combination of PBH, skin color segmentation, and CB model was employed to achieve some additional advantages. The task of the skin color segmentation was to separate hand region from background, which can remove most of the background. Yet, there were some backgrounds which had color in skin tone, and it was the main reason of applying the CB model. The advantage of the CB model was to be able to subtract moving foreground and background with high accuracy. The background was trained in every frame and therefore it detect the moving hand in front of a camera. As presented in experimental results, the system showed promising performance. References [1] M. Kolsch and M. Turk, “Fast 2D hand tracking with flocks of features and multi-cue integration,” in Proc. CVPRW, vol. 10. Jun. 2004, p. 158. [2] B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. Imag. Understand. Workshop, 1981, pp. 121–130. [3] J. Shi and C. Tomasi, “Good features to track,” in Proc. IEEE Conf. Comput. Vision Patt. Recog., Jun. 1994, pp. 593–600. [4] X. Zhu, J. Yang, and A. Waibel, “Segmenting hands of arbitrary color,” in Proc. IEEE Int. Conf. Automat. Face Gesture Recog., Mar. 2000, pp. 446–453. [5] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Royal Statist. Soc., vol. 39, no. 1, pp. l–38, 1977. [6] N. D. Binh and T. Ejima, “Hand gesture recognition using fuzzy neural network,” in Proc. Int. Conf. Graphics, Vision Image Process., Dec. 2005, pp. 362–368.

GUO et al.: IMPROVED HAND TRACKING SYSTEM

[7] A. Cheddad, J. Condell, K. Curran, and P. M. Kevitt, “A skin tone detection algorithm for an adaptive approach to steganography,” Signal Process. J., vol. 89, pp. 2465–2478, Dec. 2009. [8] C. Bergamini, L. S. Oliveira, A. L. Koerich, and R. Sabourin, “Combining different biometric traits with one-class classification,” Signal Process. J., vol. 89, pp. 2117–2127, Nov. 2009. [9] T. Ojala, M. Pietikaine, and D. Harwood, “A comparative study of texture measures with classification based on featured distributions,” Patt. Recog., vol. 29, no. 1, pp. 51–59, Jan. 1996. [10] Q. Chen, N. D. Georganas, and E. M. Petriu, “Hand gesture recognition using Haar-like features and a stochastic context-free grammar,” IEEE Trans. Instrument. Measurement, vol. 57, no. 8, pp. 1562–1571, Aug. 2008. [11] P. Viola and M. J. Jones, “Robust real-time face detection,” Int. J. Comput. Vision, vol. 57, no. 2, pp. 137–154, 2004. [12] A. Just, Y. Rodriguez, and S. Marcel, “Hand posture classification and recognition using the modified census transform,” in Proc. 7th Int. Conf. Automat. Face Gesture Recog., Apr. 2006, pp. 351–356. [13] M. Donoser and H. Bischof, “Real time appearance based hand tracking,” in Proc. Int. Conf. Patt. Recog., Dec. 2008, pp. 1–4. [14] K. Mikolajczyk and C. Schmid, “A performance evaluation of local descriptors,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 27, no. 10, pp. 1615–1630, Oct. 2005. [15] Hand Posture Database [Online]. Available: http://www.idiap.ch/ resources/gestures [16] Face Database [Online]. Available: http://www.face-rec.org/databases [17] S. L. Phung, A. Bouzerdoum, and D. Chai, “Skin segmentation using color pixel classification: Analysis and comparison,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 27, no. 1, pp. 148–154, Jan. 2005. [18] C. Stauffer and W. E. L. Grimson, “Adaptive background mixture models for real-time tracking,” in Proc. IEEE Comput. Soc. Conf. Comput. Vision Patt. Recog., vol. 2. Jun. 1999, pp. 246–252. [19] C. Stauffer and W. E. L. Grimson, “Learning patterns of activity using real-time tracking,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 22, no. 8, pp. 747–757, Aug. 2000. [20] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis, “Real-time foreground-background segmentation using codebook model,” RealTime Imag., vol. 11, no. 3, pp. 172–185, Jun. 2005. [21] A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Comput. Survey, vol. 38, no. 4, pp. 1–45, Dec. 2006. [22] D. Terzopoulos and R. Szeliski, Tracking with Kalman Snakes. Cambridge, MA: MIT Press, 1992, pp. 3–20. [23] D. Comaniciu and P. Meer, “Mean shift analysis and applications,” in Proc. Int. Conf. Comput. Vision, vol. 2. 1999, pp. 1197–1203. [24] M. Bertalmio, G. Sapio, and G. Randall, “Morphing active contours,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 22, no. 7, pp. 733–737, Jul. 2000. [25] M. Isard and A. Blake, “Condensation-conditional density propagation for visual tracking,” Int. J. Comput. Vision, vol. 29, no. 1, pp. 5–28, 1998. Jing-Ming Guo (M’06–SM’10) was born in Kaohsiung, Taiwan, on November 19, 1972. He received the B.S.E.E. and M.S.E.E. degrees from National Central University, Taoyuan, Taiwan, in 1995 and 1997, respectively, and the Ph.D. degree from the Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan, in 2004. From 1998 to 1999, he was an Information Technique Officer with the Chinese Army. He is currently a Professor with the Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei. His current research interests include multimedia signal processing, multimedia security, computer vision, and digital halftoning. Dr. Guo was granted the National Science Council Scholarship for Advanced Research from the Department of Electrical and Computer Engineering, University of California at Santa Barbara, Santa Barbara, from 2003 to 2004. He received the Outstanding Youth Electrical Engineer Award from the Chinese Institute of Electrical Engineering in 2011, the Outstanding Young Investigator Award from the Institute of System Engineering in 2011, the Best Paper Award from the IEEE International Conference on System Science and Engineering in 2011, the Excellence Teaching Award in 2009, the Research Excellence Award in 2008, the Acer Dragon Thesis Award in 2005, the Outstanding Paper Award from the Institute for Public Policy Research in 2005, and from Computer Vision and Graphic Image Processing in 2006, and the Outstanding Faculty Award in 2002 and 2003.

701

Yun-Fu Liu (S’09) was born in Hualien, Taiwan, on October 30, 1984. He received the M.S.E.E. degree from the Department of Electrical Engineering, Chang Gung University, Taoyuan, Taiwan, in 2009. He is currently pursuing the Ph.D. degree from the Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan. His current research interests include digital halftoning, steganography, image compression, object tracking, and pattern recognition. Mr. Liu received the Special Jury Award from Chimei Innolux Corporation in 2009, and the Third Masters Thesis Award from the Fuzzy Society, Taiwan, in 2009.

Che-Hao Chang was born in Taipei, Taiwan, on October 26, 1987. He received the B.S. degree in communication engineering from Yuan Ze University, Taoyuan, Taiwan, in 2010. Currently, he is pursuing the Masters degree with the Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei. His current research interests include intelligent surveillance systems.

Hoang-Son Nguyen was born in Vietnam. He received the M.S.E.E. degree from the Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, in 2010. His current research interests include intelligent surveillance systems.

Improved Hand Tracking System - IEEE Xplore

May 1, 2012 - training time by a factor of at least 1440 compared to the ... Taiwan University of Science and Technology, Taipei 106, Taiwan (e-mail:.

914KB Sizes 4 Downloads 331 Views

Recommend Documents

Improved Construction of Irregular Progressive Edge ... - IEEE Xplore
Aug 1, 2010 - Abstract—The progressive edge-growth algorithm is a well- known procedure to construct regular and irregular low-density parity-check codes.

Tag-Based Image Retrieval Improved by Augmented ... - IEEE Xplore
period of an event by leveraging millions of training web images and their associated rich textual descriptions. For any given query tag (e.g., “car”), the inverted ...

Egocentric Hand Pose Estimation and Distance ... - IEEE Xplore
Nanyang Technological University, 639798 Singapore. [email protected] [email protected] [email protected]. ABSTRACT. Articulated hand ...

Speckle Tracking in 3D Echocardiography with Motion ... - IEEE Xplore
tracking in 3D echocardiography. Instead of tracking each speckle independently, we enforce a motion coherence con- straint, in conjunction with a dynamic ...

Tag-Based Image Retrieval Improved by Augmented ... - IEEE Xplore
IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 14, NO. 4, AUGUST 2012. 1057. Tag-Based Image Retrieval Improved by Augmented. Features and ...

Efficient Multiple Hypothesis Tracking by Track Segment ... - IEEE Xplore
Burlington, MA, USA. {chee.chong, greg.castanon, nathan.cooprider, shozo.mori balasubramaniam.ravichandran}@baesystems.com. Robert Macior. Air Force ...

cooperative target tracking in vehicular sensor networks - IEEE Xplore
This subject has been studied in fields such as airborne traffic, computer vision, and wireless sensor networks. A VANET brings out new challenges that should ...

Cell Tracking in Video Microscopy Using Bipartite Graph ... - IEEE Xplore
Automated visual tracking of cells from video microscopy has many important biomedical applications. In this paper, we model the problem of cell tracking over pairs of video microscopy image frames as a minimum weight matching problem in bipartite gr

segmentation and tracking of static and moving objects ... - IEEE Xplore
ABSTRACT. In this paper we present a real-time object tracking system for monocular video sequences with static camera. The work flow is based on a pixel-based foreground detection system followed by foreground object tracking. The foreground detecti

Posterior Cramer-Rao Bounds for Multi-Target Tracking - IEEE Xplore
Jan 1, 2006 - Though basically defined for estimation of deterministic parameters, it has been extended to stochastic ones in a Bayesian setting. In the target ...

Adaptive Output-Feedback Fuzzy Tracking Control for a ... - IEEE Xplore
Oct 10, 2011 - Adaptive Output-Feedback Fuzzy Tracking Control for a Class of Nonlinear Systems. Qi Zhou, Peng Shi, Senior Member, IEEE, Jinjun Lu, and ...

Robust Multitask Multiview Tracking in Videos - IEEE Xplore
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS. 1. Robust Multitask Multiview Tracking in Videos. Xue Mei*, Senior Member, ...

IEEE Photonics Technology - IEEE Xplore
Abstract—Due to the high beam divergence of standard laser diodes (LDs), these are not suitable for wavelength-selective feed- back without extra optical ...

wright layout - IEEE Xplore
tive specifications for voice over asynchronous transfer mode (VoATM) [2], voice over IP. (VoIP), and voice over frame relay (VoFR) [3]. Much has been written ...

Device Ensembles - IEEE Xplore
Dec 2, 2004 - time, the computer and consumer electronics indus- tries are defining ... tered on data synchronization between desktops and personal digital ...

wright layout - IEEE Xplore
ACCEPTED FROM OPEN CALL. INTRODUCTION. Two trends motivate this article: first, the growth of telecommunications industry interest in the implementation ...

Evolutionary Computation, IEEE Transactions on - IEEE Xplore
search strategy to a great number of habitats and prey distributions. We propose to synthesize a similar search strategy for the massively multimodal problems of ...

Based Reasoning: High-Level System Design - IEEE Xplore
Page 1. Generic Tasks in Knowledge-. Based Reasoning: High-Level. Building Blocks for Expert .... building blocks forthe construction (and understanding) of.

An Ambient Robot System Based on Sensor Network ... - IEEE Xplore
In this paper, we demonstrate the mobile robot application associated with ubiquitous sensor network. The sensor network systems embedded in environment.

Copula-Based Statistical Health Grade System Against ... - IEEE Xplore
Abstract—A health grade system against mechanical faults of power transformers has been little investigated compared to those for chemical and electrical faults ...

Adaptive Air-to-Ground Secure Communication System ... - IEEE Xplore
Corresponding author, e-mail: [email protected]. Abstract—A novel ... hardware setup for the ADS-B based ATG system is analytically established and ...

SROS: Sensor-Based Real-Time Observing System for ... - IEEE Xplore
field ecological data transportation and visualization. The system is currently used for observation by ecological research scientists at the Institute of Geographic ...