Support Vectors a Way to Adapt for Lane Marker Tracking: A Step towards Intelligent Transportation Systems Salman Afghani* Asad Ali+ [email protected] [email protected] Ghulam Ishaq Khan Institute of Engineering Sciences and Technology Abstract The paper describes a novel approach for tracking white lane markers with the view of driving assistance. The presented technique detects the lane markers using a raster scan approach. The detected data points are then converted to functional support vectors using a kernel function derived from the data and are compared with a trained model of similar vectors stored in a d-dimensional tree using a k-nearest neighbor classifier. Experimental results confirm the validity of the presented approach in different lightening conditions and scenarios. The presented technique is capable of detecting vehicles at fourteen frames per sec which makes it ideal for real time pre-crash sensing.

1. Introduction This paper deals with the detection and tracking of white lane markers present on the road surface in structured environments from a camera mounted on a vehicle with the view of driving assistance. Lane marker detection and tracking is an active research area in the building of intelligent transportation systems and over the past decade several approaches have been developed for tracking the lane markers in out door environments using imaging based techniques. The importance of tracking white lane markers and guiding drivers in navigation is highlighted by the fact that during the year 20032004 the total number of reported accidents on highways in Pakistan were 10,308 of which 4184 were fatal, resulting in 5199 causalities and about 65% of those accidents occurred in day light. A more detailed data on traffic accidents can be found in [4]. It is also estimated that around 30% of fatal car crashes can be attributed to driver inattention and fatigue [8, 9]. So building automatic or semi-automatic systems for navigational assistance can help significantly reduce that figure. In this paper we present a novel approach for pre-crash sensing which is essentially a lane departure warning system using a support vector

machine based trained model for classifying detected lane markers by indexing into a ddimensional tree and comparing the possible lane marker points with trained model using a knearest neighbor classifier. Before we move ahead let me briefly describe support vectors and d-dimensional trees.

1.1 Support Vector Machines Support vector machines (SVM’s) [12], were invented by Vladimir Vapnik. They are method of creating classification functions (kernel functions) from a set of labeled training data. For classification SVM’s operate by finding a hyper surface in the space of possible inputs. The hyper surface attempts to split the positive examples from the negative examples and the split is chosen to have the largest distance from the hyper surface to the nearest of the positive and negative examples. Intuitively, this makes the classification correct for testing data that is near, or identical to the training data [6]. The SVM problem is ill posed: finding one hyper plane (hyper surface) that separates the data but many such hyper planes might exist. So we need a principled way to choose the best possible hyper plane. Also the, margins (geometric and functional) between the separating hyper planes depend on the chosen kernel function. The points closest to the margins are called support vectors. [12] Hence the basic idea is to construct a hyper plane as the decision surface such that the margin of separation between positive and negative examples is maximized.

1.2 D-Dimensional Trees D-Dimensional trees [7] are used for indexing multidimensional data and support partial match retrieval and orthogonal range queries with rootic search time. They are a straight forward yet powerful extension of onedimensional trees. At every level of a dd-tree we

Figure 1 shows the complete system diagram of the presented algorithm for Lane Marker Tracking split the set of coordinates according to one of the coordinate values. Every edge of a dd-tree reduces the complexity of the set represented either in dimension or in size. Exact and partial match queries in dd-trees are performed by starting at the root node and comparing the search key with the value stored in the node and following the correct pointer. We have used ddtrees to store the set of support vectors identified by the learning phase as hyper-plane separating the data.

2. Related Work Considering the importance of intelligent transportation systems research is being conducted by many in various aspects of such systems, and lane marker tracking is one of them. In the context below we review some of the works that are most related to ours. Geometric transform based methods used by Broggi et al [10] in the construction of GOLD (Generic object and lane detection) use Inverse perspective mapping for lane marker detection and tracking. The input stream of images is transformed into a bird’s eye view which also transforms the lane markers to be quasi-vertical. The task of detection then becomes simple. Obtained results demonstrate the soundness of there technique but the assumption of planar road model severely limits its practical usage independently as roads in real world along a terrain are rarely flat. Hence another technique needs to be coupled with it to cater for non-planar surfaces. Template based approaches such as developed by Kreucher [2] in the construction of LOIS (Likelihood of Image Shape) developed at the University of Michigan Artificial Intelligence Lab, use a deformable template approach. A parametric family of shapes describes the set of all possible ways that the lane edges could

appear in the image. Lane detection is then performed by finding the lane shape parameters that maximize the function for the current image. However, this algorithm can not guarantee a global optimum and accuracy without consuming huge computational resources. Therefore, it is not suitable for the real time applications like the one under study. Neural Network based approaches like the one developed by Pomerlaeu [18] for the navigation of CMU’s Navlab make use of back propagation for the training of the network that keeps the vehicle on the road. The network is first presented with an input image and activation is propagated forward through the network to determine the networks response. The network response is then compared with the known correct response, if it does not match then the weights between connections in the network are altered to produce a response closer to the correct response. Experimental results demonstrate the soundness of there work but the vehicle can only be driven at 55mph and the system is not capable of adapting to changing lightening conditions along a terrain. Filter based methods such as the one developed by Joel et al [1] make use of steerable filters, which are the second derivative of two dimensional Gaussian, are convolved with an input image and provide features that allow them to be used to detect both bots dots[17] and solid lines while providing robustness to cluttering and lightening changes have been shown to perform well but these filters only work well with highly structured road textures because second derivative exaggerates the effect of noise (road patches) twice which limits there usage in many practical scenarios. Also no information regarding the processing complexity has been reported. We improve upon many of the short comings mentioned above.

3. Proposed Technique

3.3 Step3: Noise Removal

The algorithm is provided an input stream of images from a camera. The dimensions of each of the extracted video frames are 352 X 288. A step wise approach is described below which explains the lane marker detection and tracking algorithm.

The obtained points are cross verified to remove any noise from the gathered data that might result in wrong predictions later. For this purpose road is extracted from the image by performing segmentation based on region splitting and merging on the lower 2/3rd portion of the incoming stream of images. Segmentation is performed by dividing the image into 2 X 2 blocks and comparing the blue ratio [11], standard deviation and mean of the individual blocks against similar parameters of eight precomputed road texture models. The output obtained from the above step is shown below:

3.1 Step 1: Preprocessing As a first step the lower 1/3rd portion of the image after converting into gray scale is raster scanned for acmes above a certain threshold. This helps to determine the lightning conditions of the environment in which the vehicle is being driven as it observers the gradient of white lane markers. If the acmes are found and amount a suitable number, then it can be said that the image has been acquired in good lightning conditions. Otherwise a contrast and a brightness stretch are applied on the 2/3rd portion of the image using the following formulas on the three channels of the RGB color model: X = max ( 0 , min (M , ∑ ( α + β + γ ))) (1) Where α = ( Channel – β ) x Contrast β = 128 γ = Brightness M = 255 Channel = R , G , B

Figure 3 Shows an image before and after the road detection step. The grayed portion in the right image highlights the road from which road boundaries are extracted and recorded.

All points that were gathered in the raster scan phase are verified to be lying within the road boundaries and in clusters. Points that do not satisfy these properties are removed from the data set hence reducing the error only to the points within the computed road boundaries.

3.4 Step 4: Training Using Support vectors

Figure 2 Shows an image acquired under poor lightning conditions (left image) and after contrast and brightness stretch has been applied(right image).

3.2 Step2: Raster Scanning The 2/3rd portion of the incoming stream of images are raster scanned from left to right in-order to detect lane markers. The scanning algorithm detects the mass of hill above a certain threshold also referred to as the localized acme estimation. All points that satisfy the above condition are recorded for further processing.

In the training phase 30 images (10 for every lane model i.e three lanes) of different road scenarios including turns at different angles are provided as input to the classification algorithm. After applying steps from 3.1 to 3.3 as mentioned above we are left with lane marker data points only. Kernel Function: To classify the data points using SVM the kernel function was derived from the data set as a set of seed points whose norm from the left and the right boundaries of the input image is minimum over 1/3rd height of the image. The selected points using the said kernel form the Kernel Induced Vector (KIV) and are used later for classifying the data set.

the lane markers are separated in blue and red colors.

Figure 4 Shows the encircled points that were selected by the kernel function for KIV

Establishment of hyper plane: Then norm is computed between the KIV and the data set, which results in the establishment of two separating geometric hyper planes and separation of the data in the form of two sets, A and B where A is the first set from the left and B is the second. The discriminant hyper plane is defined as: N

f ( Xq ) = ∑ Yjβ jK ( Xq, Xj ) + b

(2)

j =1

Where N is the number of training samples (Xj,Yj) and Xj ε Rn and Yj ε {-1, 1}. K is the kernel function and the sign of f (Xq) determines the membership of query sample Xq. An optimal hyper-plane is determined by all non-zero βjs ,which correspond to the support vectors and the bias b. So the maximum values of set (A) w.r.t x form the geometric margin A and minimum value of set (B) form the geometric margin B. The points whose norm is minimum with certain applicable threshold from the hypothetical margin A and B form the geometric support vectors (GSV). The functional margin is then computed to be the center of the GSV resulting in the formation of a single functional support vectors (FSV).

We obtain three FSV’s by repeating the training process for all the three lane models provided as input. The obtained FSV’s are then indexed (stored) into a d-dimensional tree which effectively becomes 2d-tree because each value of the FSV’s is represented in two dimensions. It is important to note here that indexing into ddtree significantly reduces the number of comparisons required each time the current lane of a vehicle needs to be verified. With this the training phase comes to an end.

3.5 Step5: Tracking of Lane Markers In order to detect lane departure and generate warnings we need to track the movements of a vehicle in accordance with the trained support vectors. So step 3.1 to 3.4 are repeated (but no dd-tree is created) for the incoming stream of images and the computed FSV’s are compared with the trained support vectors by querying (exact match or partial match) into the 2d-tree created in the training phase. For comparison k-nearest neighbor classifier is used which assigns an unlabeled pattern X to the class of its nearest neighbor if the distance:

D (m i , X) = min j{ D(m j , X) }

(3)

Where mi ε {trained support vector data of three lane models stored in 2d-trees} and j = 1, 2, 3 If 80% of the input data is successfully classified then the vehicles current lane is highlighted on the screen other wise a lane departure warning is generated indicating the direction of transition.

3.6 Confidence Level of Decision

Figure 5 (On the left) Shows the geometric margins formed by the algorithm in green discrete lines and the functional margin or the separating hyper plane in the yellow color formed when processing road in lane 2. (On the Right). Output after hyper-plane is established in which

The number of points that are initially detected and the ones that survive to exist in the functional support vectors dictate the confidence level of the decisions generated by the algorithm. For this purpose a specialized slider trapped in three levels namely poor, medium and good outputs the line visibility, which indicates the precision of the decision generated by the algorithm. Beside this the algorithm provides stereo-phonic acoustic assistance in the frequency range of 18000Hz to 21000Hz for the

vehicular movements within a given lane. This helps the driver to reposition the steering as per the deviation observed and feedback provided to him by the system.

4. Results The algorithm was tested on a 2 GHz Pentium 4 machine with Windows XP and Visual C++ as the development tool. An original movie of the Islamabad – Lahore Motorway was recorded in the summer and fall of 2004. The algorithm was tested on continuous runs of the movies which depicted actual scenario encountered by the drivers while driving on highways. Lightening conditions also varied significantly along the terrain. All 35 occurrences of lane transition were detected successfully by the algorithm, which included empty roads and semi occluded road scenarios by obstacles such as vehicles. The response time of the algorithm was measured to be 2 seconds on average for the generated message of lane change. After successfully testing the algorithm on recorded movies it was tested in real time using a Compaq 800 MHz laptop with Windows XP as the operating system. All occurrences of lane transition were successfully detected by the algorithm. Shown below are some of the scenarios where lane departures were successfully detected.

(a)

Figure 7 Shows two images from the video sequence when the lane transition was detected by the algorithm; from lane 3 to lane 2.

Figure 8 Shows two images when the lane transition was detected from lane 2 to lane 1 and from lane 1 back to lane 2.

Figure 9 Shows two images in which the lane transition was detected from lane 2 to lane 3 and from lane 3 to lane 2 under poor lightning conditions.

Beside this the algorithm is capable of processing 14 frames per second, which makes it ideal for real time usage. The technique has been tested to work well at a maximum vehicular speed of 130Km/h.

5. Conclusion

(b) Figure 6a Shows transition from lane 2 to lane 3 when it was detected by the algorithm. Fig 6b indicates the position of a vehicle in a given lane and the associated transition.

In this paper a novel approach for lane marker tracking using a trained model of support vectors was proposed. Experimental results demonstrate that our approach is robust to widely different scenarios encountered by the driver along a terrain. Step 1, 2, 3 laid the foundation for the detection of white lane markers where as step 4 helped in the building a trained model for each of the three lanes and established a set of support vectors which were used in step 5 for verifying vehicles position

w.r.t the trained model. The focus of this work was highways with three lanes but the work can be extended to 2 lane highways easily. Presently work is in progress to extend this work to be applicable during night driving. After its completion the algorithms will be embedded into an imaging DSP for actual deployment and use. By incorporating such systems into vehicles the number of accidents and related fatalities will definitely fall significantly.

6. References [1]

Joel C. McCall and Mohan M. Trivedi “An Integrated, Robust Approach to Lane Marking Detection and Lane Tracking” IEEE Intelligent vehicle Symposium June 2004. [2] Kreucher C., Lakshmanan S. and Kluge K., "A Driver Warning System Based on the LOIS Lane Detection Algorithm", Proceeding of IEEE International Conference on Intelligent Vehicles, pp.17-22, 1998. [3] Erdem Bala, A.Enis Cetin “Computationally Efficient wavelet Affine Invariant Functions for Shape Recognition” IEEE Transactions on Pattern Analysis and Machine Intelligence page 1095-1098 August 2004 [4] Statistical Survey of Pakistan 2003-2004. [5] An Integrated Line Tracking and Vectorization Algorithm by Peter R. van Nieuwenhuizen, Olaf Kiewiet, Willem F. Bronsvoort Euro Graphics 94. [6] Support vector machines for pattern recognition by Christopher J.C Burges, Bell laboratories. [7] “Multi-dimensional Searching and computational geometry” Kurt Mehlhorn Monographs on theoretical computer science Spinger-Verlag. [8] HoRSCoCTA Inquiry into Managing Fatigue in Transport. The Parliament of the Commonwealth of Australia, 2000. [9] Trent Victor, Olle Blomberg, and Alexander Zelinsky. Automating driver visual behavior measurement. In Proceedings of Vision in Vehicles 9. [10] M. Bertozzi and A. Broggi. “GOLD: a Parallel Real-Time Stereo Vision System for Generic Obstacle and Lane Detection.” IEEE Transactions on Image Processing, 1997. [11] Sohail Nadimi, Bir Bhanu “Physical Models for Moving Shadows and object detection in video”, IEEE transactions on pattern analysis and machine intelligence page 1079-1087 August 2004. [12] Support Vector and kernel machines by Nello Christianini, BIOwulf technologies. [13] J.McDonald. “Detecting and tracking road markings using the Hough transform.” Proceedings of the Irish Machine Vision and Image Processing Conference, 2001.

[14] J. Kosecka, R. Blasi, C. Taylor and J. Malik. "A Comparative Study of Vision-Based Lateral Control Strategies for Autonomous Highway Driving". In IEEE International Conference on Robotics and Automation, pp 1903-1908, May 1998. [15] K. Huang, M. M. Trivedi, T. Gandhi, "Driver's View and Vehicle Surround Estimation using Omni-directional Video Stream," Proceedings of IEEE Intelligent Vehicles Symposium, Columbus, OH, pp. 444-449, June 9-11, 2003 [16] H. Furusho, R Shirato, M. Shimakage. “A Lane Recognition Method Using the Tangent Vectors of White Lane Markers,” 6th International Symposium on Advanced Vehicle Control, Sept. 9-13, 2002. [17] Stefan Gehrig, Axel Gern, Stefan Heinrich and Bernd Woltermann , “Lane recognition on poorly structured roads – The Bot Dot problem in California”, Proceedings of the 5th Conference on Intelligent Transportation Systems, 2002. [18] Dean Pomerleau, Todd Jochem and Charles Thorpe, “Next Generation Neurally Based Autonomous Road Follower” Proceedings of the International Conference on Intelligent Autonomous Systems: {IAS}—3, 1993.

+ The author is a fellow of National Engineering and Scientific Commission and presently doing his MS in Computer System Engineering from GIKI. * The author is presently the Dean of the Department of Computer Engineering Air University and Dean Research a Development APCOMS.