TALa: Theft Alert for Laptops Shelah Castro, Karen Camae Chua, Melany Rosales Christine Clarin, Prospero Naval, Jr. Computer Vision & Machine Intelligence Group Department of Computer Science College of Engineering University of the Philippines Diliman

[email protected] ABSTRACT In recent times, the use of laptops is widespread. Due to their high market value, this increase in use poses threats to the security of laptops, most notably theft. Security cameras provide the convenience of monitoring multiple places simultaneously for cases of theft. However, as the number of cameras to monitor and monitoring time increases, manual monitoring of video outputs becomes prone to more errors. A possible solution to this problem is to automate the surveillance system through the use of computer vision techniques. Such will enable people to leave their laptops unattended without the need for additional security personnel. Users will have the option of not having to bring their laptops with them in the most inconvenient situations. The aim of this project is to develop a real-time smart surveillance system that detects laptop theft in workplaces and business establishments. The system will process real-time video sequences. The system is divided into three modules: (1) laptop detection and tracking; (2) human detection and tracking; and (3) theft detection. In the laptop detection and tracking module, a model-based approach based from a study by Lowe [10] is used. For human detection, functions from OpenCV that perform background subtraction by using a Mixture of Gaussians (MOG) to model the background; human tracking utilizes the Blob Tracking Auto module from the same library. The theft detection module consists of owner identification, extraction and storage of the owner’s features, and recognition of theft events.

Keywords Human tracking, object detection, association, theft detection, background subtraction

1.

INTRODUCTION

Research interest in the field of intelligent visual surveillance is growing as the demand for security increases, particularly in public transportation systems, business establishments, military operations and other places for human activity [1]. This is apparent in the formation of specialized workshops for visual surveillance and in various publications. The development of new real-time robust computer vision algorithms contributes much to the improvement of second generation techniques used in automated surveillance systems. In general, these technologies focus on tracking humans and objects of interest in various environments for behavioral analysis - from people counting and identification to threat assessment systems. However, there are relatively few systems that automate the detection of theft using closedcircuit television (CCTV) cameras, some of which include a bicycle theft detection system (BMVC07) and some implementations of abandoned baggage detection (PETS2006). TALa aims to contribute to this area of automated visual surveillance. Laptops already rank as one of the most commonly used electronic devices. Mobility, compactness and increasing computing power make these gadgets indispensable for businesses, the academia and personal use. But because of their high monetary values and portability, it is not surprising that laptops are increasingly attractive targets for theft. Many commercial security systems created to reduce laptop theft incidents are embedded in the devices themselves. For these systems to work, installation of additional software and hardware is needed. The proposed system, TALa, is not only concerned with the reduction of theft but also spares the users from the effort of installing other software or devices.

2. RELATED RESEARCH 2.1 OBJECT DETECTION Model-based object recognition aims to locate an object of a predefined class [2]. This priori knowledge can be used to construct a model for detecting and tracking objects in real world conditions [9]. Most studies using model-based approach are focused on rigid objects [7]. The process starts with the identification of hypotheses or groups of lines that are possibly part of the object being identified. The obtained hypotheses are then ranked according to their level of confidence. Starting from the highest, each hypothesis is fitted into the model until a match is found [10]. However, this is only applicable to rigid objects. As a preliminary step

in identifying an articulated object, find the possible rigid parts of the object. The parts are grouped according to proximity and constraints that define the object are checked [3]. If a group passes the constraints, it is identified as the object.

2.2

HUMAN DETECTION AND TRACKING

One of the essential tasks in computer vision applications is tracking people by using video sequences obtained from surveillance cameras. Human tracking is difficult because of changes in appearance caused by their motion, the necessity of low resolution in broad monitoring, changes in color that come from varying light conditions and clutters observed in the background [8]. Most approaches involve the use of blobs for tracking [5, 1, 13, 2, 14, 15]. This approach is computationally efficient which makes it suitable for realtime applications [15].

Another problem that the development of TALa poses is the choice of sufficient features to associate the owner to the laptop. The study will focus on the use of color of the owner’s clothes for owner identification and distance between laptop and human blobs for establishing ownership. The accuracy of the system will depend on the assumption that the owner will not change clothes and that no two persons are wearing the same clothes. Finally, the study will present a definition for theft. Theft is when the owner is not in the scene and his laptop disappears. There is a possibility that a laptop cannot be tracked because of occlusion. In this situation, the system will not raise an alarm. Instead, it will wait for the occluding blob to go.

4.

METHODOLOGY

Human detection is commonly done by extracting the foreground in a video sequence. Given a frame sequence from a fixed camera, foreground objects are detected by subtracting the current frame from a reference image. The resulting pixels that pass a certain threshold are identified as parts of the foreground. However, this simple frame difference is very sensitive to the threshold and assumes a static background model (e.g. no variations in lighting, no introduction of objects in the scene). As a response, many adaptive background modeling methods have been proposed. One of these is the adaptive nonparametric Gaussian mixture model (MOG) developed by Grimson et al [6]. Each pixel in the background model is modeled by a mixture of K Gaussians. Distributions that are higher or more compact are most probably part of the background.

2.3

OBJECT-HUMAN ASSOCIATION

Despite of the advances in object detection and tracking, there are only few applications that use these approaches to associate tracked objects with tracked humans [4]. In the study by Damen, owners of bicycles that are left in the scene are detected. To identify the owners, sets of person hypotheses and object hypotheses are generated. Given a situation, the association phase aims on determining what type of event is happening, like picking up and dropping of the bicycle, or neither. To detect theft, color information from an individual that is picking up a bicycle is compared to the owner’s color histogram. However, there are difficulties in using color as sole feature of the owner because of shadows and other factors that could introduce wide variations in color.

3.

PROBLEM DEFINITION

TALa is a smart surveillance system that establishes ownerlaptop relations which are used for detecting theft. As a preliminary step in the development of such a system, this study aims to determine what methods could be used to detect laptops and humans in video. The processes mentioned in the previous section will be implemented. Limitations of these processes need also be defined. For human detection, all huge, moving blobs are assumed to be people. Laptop detection will also require that a laptop be opened and not completely occluded; laptops that remain closed throughout the video will not be considered.

Figure 1: TALa system flowchart. TALa consists of three modules: (1) laptop detection, (2) human detection and tracking and (3) theft detection. Laptops present in the scene will be detected by matching a predefined model to a group of lines that could be parts of a laptop. After detection, small blob outputs from the background subtraction process are filtered for noise reduction. The remaining blobs are considered as people in the scene. Distances between the detected laptops and the blobs are obtained; the blob nearest a laptop is marked as the owner. The system monitors the owners’ movement. When a laptop disappears from the camera’s scope, theft detection is activated. If an event matches the theft definition, an alarm will be raised.

4.1 4.1.1

LAPTOP DETECTION AND TRACKING Detection

Lines are extracted from the laptop candidate through the use of Canny edge detector and Hough transform from OpenCV library. Hypotheses or groups of lines that are possibly part of a laptop’s screen or keyboard are formulated based on the proximity between endpoints of line segments that could represent the edges of a rectangle [10].

P (Xt ) =

K X

ωi,t η (Xt , µi,t , Σi,t )

i=1

covariances of the three channels are the same, S is reduced to a diagonal. The Gaussian components are arranged according to a fitness criterion w/s. The background model is estimated by the first B distributions that satisfy:

Each hypothesis is then fitted into a rectangular model to verify that each of them is a rectangle. The fitting is done through the Levenberg-Marquardt Algorithm [11]. LevenbergMarquardt algorithm searches for the viewpoint and the model parameters by iteratively solving:

B = argminb

b X

! wj > T

j=1

J T Jx = J T e where x is the vector of unknown corrections, J is the Jacobian matrix containing the partial derivatives, and e is the vector of errors measured. Manolis Lourakis’ LevenbergMarquardt implementation which approximates the Jacobians was utilized by the system. The identified rectangles are once more grouped to form another set of hypotheses. Each hypothesis consists of two rectangles representing the two rigid parts of the laptop: screen and the keyboard. Constraints, such as the two rectangles should be connected, are verified for the laptop candidate to be declared as a laptop.

4.1.2

If a laptop failed to be reidentified in five consecutive tries, then it is checked for possible occlusion. If there is none, the laptop is identified as not there anymore. Otherwise, the system continually checks until there is no more intersection between the laptop and the occluding blob. After that, the system checks again if the laptop is still there.

4.2.1

To further refine the segmented foreground, shadow detection has also been employed to prevent classifying shadow pixels as part of the scene. Pixels that have significantly lower values than their corresponding background values are classified as shadows and are then omitted from the foreground mask. Through this, connected components that had been merged by shadows can be separated and blob tracking will be more accurate.

Tracking

Laptop tracking is about reidentifying the laptop in the frames succeeding its initial identification. In reidentification, the color of the area where the laptop is supposed to be is compared to the image of the area where the laptop was initially found. The comparison is done through color histogram intersection. If the comparison succeeds, the laptop is successfully reidentified. Otherwise, the area is processed using the laptop detection module. Both processes are used in reidentification because the laptop detection module is sensitive to occlusion. Color comparison is used first because it is computationally cheaper.

4.2

The implementation used for background subtraction is based from [16]. The original MOG scheme is improved so that the new algorithm can automatically select the number of Gaussian components for each pixel. Automating such selection gives the algorithm the capability to fully adapt to the scene, thus, producing a better foreground mask and less processing time.

HUMAN DETECTION AND TRACKING Background Subtraction

Background subtraction, using the Mixture of Gaussian (MOG) algorithm is performed to identify people in the video. The method introduced in [6] makes use of mixture models to represent the statistics of the scene. Considering K Gaussian distributions, the probability of observing a pixel value x at time t is given by:

Because of the adaptive nature of the Mixture of Gaussians method, people that remain stationary for a long period of time are absorbed into the background model. For the system to be accurate, this problem needs to be resolved through tracking.

4.2.2

Tracking

Tracking is done with the use of OpenCV’s Blob Tracking Auto which has five modules. Only three of those modules, FG/BG detection module, Blob Entering detection module and Blob Tracking module, are utilized. The modules are interfaces - making them flexible and capable of trying different algorithms. There are also available implementations of different algorithms in OpenCV. The algorithm used for FG/BG detection module is discussed in the previous section on background subtraction. For Blob Entering detection module, the algorithm used is OpenCV’s algorithm that detects new blob through the uniform movement of connected components in the foreground. The said algorithm was altered such that only large foreground blobs that pass a certain minimum and maximum area threshold are identified as a new entering blob. These identified blobs are automatically classified as people. For Blob Tracking, OpenCV’s implementation of mean shift algorithm with foreground mask was used. Blobs are considered to have exited the scene when they pass through the boundaries of the camera.

Table 1: Theft with TP TN Laptop 1 N/A Human 5 N/A Theft 1 0

occlusion FP FN 0 0 17 0 0 0

Table 2: No theft with minimal TP TN FP Laptop 1 N/A 1 Human 6 N/A 17 Theft 0 1 0

4.2.3

occlusion FN 0 0 0

Histogram intersection is used to determine if a person is the owner of a laptop. It is advantageous because it is robust when comparing objects at different view points, occlusion, and varying image resolution. Also, there is no need to substract the background of the owner’s image [12]. In owner reidentification, a color constancy algorithm implemented by Wang Liang, Retinex, is applied to the images of the owner and an unidentified blob in the scene, then the color histograms (red, green, blue) of the two are computed. These are compared to each other using histogram intersection. If the average of the intersection is high, then that blob is reidentified as the owner of the laptop.

Table 4: No theft with occlusion TP TN FP FN Laptop 1 N/A 0 0 Human 7 N/A 19 0 Theft 0 0 1 0

Table 5: No theft with TP TN Laptop 1 N/A Human 4 N/A Theft 0 1

occlusion 2 FP FN 0 0 41 0 0 0

THEFT EVENT DETECTION

The theft detection module of the system is activated when a tracked laptop disappears. The system searches for the owner in the scene. If there is no blob with the ID of the owner, the system compares the histograms of each blob in the scene to the histograms of the owner. If there is no blob that passes the histogram comparison, then the alarm is raised.

5.

occlusion FN 0 0 0

Feature Extraction and Association

When a human blob is identified to be the owner of a laptop, the system associates the laptop and the owner by mapping the laptop ID to the stored image of the owner.

4.3

Table 3: Theft with minimal TP TN FP Laptop 1 N/A 0 Human 4 N/A 9 Theft 1 0 0

EXPERIMENTAL RESULTS

Figure 2: TALa’s sample theft detection Overall, the precision values are as follows: 0.56 for laptop detection, 0.21 for human detection and 0.50 for theft detection. Values for recall are 1.00 on all modules. Precision is affected by the presence of false positives, particularly in human detection, where background noise is sometimes considered as a human blob. The same is true in laptop de-

Table 6: TP Laptop 0 Human 3 Theft 0

No laptop TN FP N/A 3 N/A 5 0 1

FN 0 0 0

Table 7: Overall system performance TP TN FP FN Laptop 5 N/A 4 0 Human 29 N/A 108 0 Theft 2 2 2 0

[7]

[8]

Figure 3: TALa’s occlusion handling output [9] tection and theft detection as some objects are mistakenly identified as laptops. The system has been tested using video sequences that depict scenarios under perfect conditions. Only one laptop or none appears in each scene. Also, the environment is not cluttered and there are minimal changes in the background. Videos used span a maximum of five minutes.

[10]

[11]

The system needs optimization as it still cannot process realtime data. Reducing the number of frames to be processed will improve the speed, however, the performance may be affected. Implementing a faster tracking algorithm will considerably reduce processing time. Using a computer with a high-end processor will also boost the speed of the system.

[12]

6.

[14]

CONCLUSION

This paper presented a system that uses object recognition and human tracking to detect laptop theft from surveillance camera outputs. TALa is an innovative idea since it merges classic problems in digital image processing: association of detected objects with tracked humans. Also, it introduces an easier way of monitoring laptops, providing aid to security personnel. Since TALa uses model-based approach in object recogniton, it can later be extended to detect other objects aside from laptops by defining other models. Thus, TALa can cater to different purposes.

7.

REFERENCES

[1] M. Ali, S. Indupalli, and B. Boufama. Tracking multiple people for video surveillance. In International Workshop on Video Processing for Security, 2006. [2] B. Bose, X. Wang, and E. Grimson. Multi-class object tracking algorithm that handles fragmentation and grouping. In IEEE Conference on Computer Vision and Pattern Recognition, pages 17–22, June 2007. [3] L. Cinque, E. Sangineto, and S. Tanimoto. Articulated object recognition:a general framework and a case study. In Proc. IEEE Conf. Video and Signal Based Surveillance, 2006. [4] D. Damen and D. Hogg. Associating people dropping off and picking up objects. In British Machine Vision Conference, September 2007. [5] A. Francois. Real-time multi-resolution blob tracking. Technical report, Institute for Robotics and Intelligent Systems, April 2004. [6] W. E. L. Grimson and C. Stauffer. Adaptive background mixture models for real-time tracking. In IEEE Computer Society Conference on Computer

[13]

[15]

[16]

Vision and Pattern Recognition. IEEE Computer Society, 1999. Y. Hel-Or and M. Werman. Model based pose estimation of articulated and constrained objects. In 3rd European conference on Computer Vision, pages 262–273, 1994. Y. Kobayashi, D. Sugimura, Y. Sato, K. Hirasawa, N. Suzuki, H. Kage, and A. Sugimoto. 3d head tracking using the particle filter with cascaded classifiers. In Proc. of British Machine Vision Conference, 2006. D. Koller, K. Daniilidis, and H. Nagel. Model-based object tracking monocular image sequences of road traffic scenes. International Journal of Computer Vision, 10(3), June 1993. D. G. Lowe. Three-dimensional object recognition from single two-dimensional images. Artificial Intelligence, 31:355–395, 1987. D. G. Lowe. Fitting parameterized 3-d models to images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13:441–450, 1991. M. J. Swain and D. H. Ballard. Color indexing. International Journal of Computer Vision, 7(1):11–32, 1991. C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland. Pfinder: Real-time tracking of the human body. IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(7):780–785, July 1997. T. Yang, S. Li, Q. Pan, and J. Li. Real-time multiple objects tracking with occlusion handling in dynamic scenes. In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. T. Zhao and R. Nevatia. Tracking multiple humans in complex situations. IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(9):1208–1221, September 2004. Z.Zivkovic. Improved adaptive gausian mixture model for background subtraction. Proceedings of the 17th International Conference Pattern Recognition, 26(9):1208–1221, September 2004.