VEHICLE LICENCE PLATE RECOGNITION BY FUZZY ARTMAP NEURAL NETWORK Yap Keem Siah, Tay Yong Haur, *Marzuki Khalid, Tahir Ahmad Centre for Artificial Intelligence and Robotics (CAIRO) Faculty of Electrical Engineering, Universiti Teknologi Malaysia, Jalan Semarak, 54100 Kuala Lumpur E-mail: [email protected] (* All correspondence should be sent to)

Abstract Vehicle license plate recognition is one of the techniques that can be used for the identification of vehicles. It is useful to be applied to many applications such as entrance admission, security, parking control, airport or harbour cargo control, road traffic control, speed control and so on. In this paper, we present an approach of recognising vehicle license plate using the Fuzzy ARTMAP neural network, a relatively new architecture of the neural network family, which has the capability to learn incrementally unlike the conventional BP algorithm. The image of the vehicles are captured by a monochrome CCD camera and processed digitally through a frame grabber. A program is written using Visual C++ and the Matrox Image Library to process the captured image of the vehicle. The image processing technique includes noise reduction and image enhancement, which then finds the region of interest which is the licence plate of the vehicle. Once the ROI is identified, each character or digit in the license plate is isolated and sequentially recognised by the trained neural network. A prototype of the vehicle license plate recognition system, code-named ‘Apple-1’, is currently under development at the Centre for Artificial Intelligence and Robotics (CAIRO), Universiti Teknologi Malaysia.

Keywords Image Processing, Blob Analysis, Edge Detection, Fuzzy ARTMAP, Character Recognition. 1. INTRODUCTION With the rapid growth in the number of vehicles in Malaysia, there is a need to improve existing systems for identification of vehicles. A fully automated system that can identify vehicles is in demand in order to reduce the dependency on labour. One method to identify vehicles would be to identify the vehicles' license plate number as such numbers uniquely distinguish each vehicle. There are a number of useful applications for vehicle license plate recognition system for example electronic toll collection, security, parking control, speed detection, law enforcement and so on. There has been a number of commercialised software that can be used for identification of licence plates which are mainly from USA and Europe. However, they cannot be readily used in Malaysia as Malaysian vehicles differ in styles and formats. The licence plates of vehicles in Malaysia are divided into several different colours and formats. The license plate for normal private vehicles has black background and white characters as shown in Fig. 1. The license plate for taxis has white background and black characters. Diplomatic license plates have red background and yellow or white characters. Another types of license plates, which are occasionally introduced, are special plates such as Proton, Satria, Sukom, Tiara and several others. These plates are produced numbering about a 1,000 each time

usually during the introduction of such vehicles in the market.

Fig. 1. Typical formats of Malaysian vehicle license plates

At the Centre for Artificial Intelligence and Robotics (CAIRO) of Universiti Teknologi Malaysia, a vision system for identification of vehicle licence plates is currently being developed. It consists of three main modules as shown in Fig. 2 and are described as follows: • Image Capturing Device (ICD) which consists of a CCD camera, a frame grabber and a sensor. • Vehicle License Plate Recognition Module (LPR) which processes the raw image, identifies the license plate's region of interest, and recognises the license plate characters using a relatively new neural network algorithm referred to as the Fuzzy Artmap. • Main Module (MM) which consists of the graphical user interface. It is able lookup to the database of license plate numbers.

PC with Windows 95 License Plate Recognition Module

Main Module

Image Capturing Device Fig. 2.

Overall system configuration of the vision system.

In this paper we describe the development of this vision system and present some of its initial results. The paper has been organized as follows. The next section describes the technique that we use in identifying the region of interest of the captured image, which is the license plate. The next section explains the character recognition using the Fuzzy ARTMAP neural network. This is followed with discussions on the experimental results and conclusion. 2. LICENSE PLATE IDENTIFICATION There have been a number of studies in licence plate recognition, which used Hough Transform (HT) for line detection. V. Kamat and S. Ganesan [1], use HT to locate the boundary line of the vehicle license plate. They first applied a modified Sobel operator for edge detection and image thresh-holding is done depending on the average brightness of the image. HT is applied to the entire image, and peaks are searched in the Hough space in which the lines are identified in real space. However, boundary line detection is not suitable in our case, as most of vehicle’s license plates in Malaysia do not have boundary line. Also even if they exist, it may not be clearly preserved in the image due to noise and uneven brightness. Furthermore, HT is inherently a time-consuming process. The colour and texture of the license plate have also been used to identify the license plates. J.A.G. Nijhuis, et al. [2], incorporated fuzzy features of the license plate using yellow rectangle area and black characters. They use discrete-time cellular neural networks (DTCNN) as the feature extractor and multi-layered Perceptron (MLP) as the recogniser.

between the black color background and with color characters of the license plate. The algorithm to locate the license plate is given below: • Step 0: Initialize the threshold value (T) to a small positive value and set the step-up of threshold value, (Ts) to a small value too. Typically less than 5. • Step 1: Apply thresholding operation with threshold value T. • Step 2: Apply blobs analysis and calculates the basic properties of each blob. • Step 3: Cluster the blobs by cascade clustering network. • Step 4: If all characters of license plate are found then stop searching process. • Step 5: If T < 256, then T = T + Ts then go to step 1, else failed to located the license plate. The image captured through the CCD camera is in gray scale and the raw image before pre-processing usually contains a lot of noise. Binary thresholding can be done on the image to remove the noise, which makes the task of locating the license plate becomes easier. Figure 3 shows the image before thresholding and after thresholding.

height

Fig. 3. (a) A gray level scale image with noise. (b) After thresholding with noise removed.

We used the blob analysis method to identify the connected regions of pixels in the image and then the basic feature of the region is calculated. The regions are commonly know as blobs, which mean areas of touching pixels which are in the same logical pixel. The features that could be extracted from the blobs are the height, the width, the top and the left. The blob analysis is also used to segmentize the character. Figure 4 shows a blobbed character with some basic properties. width ( top,left ) height

In license plate recognition, one of the most difficult tasks is to locate the license plate, which could be at anywhere in the image. This task becomes more challenging if the illumination of the image varies from one plate to another. Our license plate identification system does not assume the presence of boundary line at the license plate. Instead, we use an iterative thresholding operation to fully exploit the large contrast

( bottom,right ) Fig. 4. A blobbed character and some of its features.

The characters of the license plate can be found by clustering the original attribution blobs, which can be identified as follows: • Total pixels in a blob • Height of the blobs • Width of the blobs • Distance between blobs • Position of the blobs We used a cascade-clustering network to cluster these attributions into different classes. The clustering produced blobs that satisfy the above attributions, which are considered as the characters of the license plate. In order to obtain the correct location of the number plates, the blob analysis and the clustering processes are repeated several times with different values. However, if the thresholding value does not give satisfactory results on the clustering, the algorithm will automatically increase the threshold value and repeat the whole process again. There are three processes before the recognition module, which are described as follows: • Image normalization - to normalize the characters into the same size, for example into a 32x32-pixel image. • Feature extraction – to extract the significant features of characters belonging to different classes. • Compression - to reduce the number of input nodes of the recogniser in order to increase the training and recognition speed.

We used Kirsch edge detection [5] for our feature extraction. The algorithms of the Kirsch nonlinear edge enhancement is described briefly as follows: F (i,j) = max{1,max[|5Sk - 3T k |]}

(1)

K = 0, 1, 2, 3, 4, 5, 6, 7

(2)

Sk = A k + A k+1 + A k+2

(3)

F(i,j)H = max( | 5S0 - 3T 0 | , | 5S4 - 3T 0 | )

(5)

F(i,j)V = max( | 5S0 - 3T 0 | , | 5S4 - 3T 0 | )

(6)

F(i,j)R = max( | 5S0 - 3T 0 | , | 5S4 - 3T 0 | )

(7)

F(i,j)L = max( | 5S0 - 3T 0 | , | 5S4 - 3T 0 | )

(8)

The above calculations can be replaced by simple mask operation as shown in the example in Fig. 6.

Fig. 6. Kirsch masks for edge detection (a) horizontal, (b) vertical,(c) right diagonal, (d) left diagonal

Horizontal Vertical

Fig. 7.

where

Tk = A k+3 + A k+4 + A k+5 + A k+6 + A k+7 (4) The F (i,j) is the gradient of pixel (i,j) and an example is shown as in Fig. 5.

Right Diagonal

Left Diagonal

Image before and after detection by Kirsch Edge Feature extraction.

Figure 7 show fives images (one original and four after edge detection) with a resolution of 32x32 pixels of gray scale. A binary image will be transformed into a gray scale image after the mask operation. Each of the images will be compressed to an 8x8 floating point array. The compression equation is given as follows: 4

V = 1 / 16∑ x =1

4

∑

f ( x, y)

(9)

y =1

where V is a floating-point value between 0 and 1. Fig. 5.

Example showing the eight neighbors of pixel (i,j)

Here, the four direct directional feature vectors for horizontal (H), vertical (V), right diagonal (R) and left diagonal (L) directions can be defined as follow:

3. THE FUZZY ARTMAP NEURAL NETWORK ARTMAP (Adaptive Resonance Theory - mapping) is a class of neural networks that can perform incremental supervised learning. It also performs multidimensional

mapping in response to binary input vectors presented arbitrarily [3]. The Fuzzy ARTMAP extends the ARTMAP by integrating fuzzy logic into the system, which allows it to accept input vector values between 0 and 1. Fuzzy ARTMAP consists of two fuzzy ART modules (ART a and ARTb ) which are linked together via an inter-ART module, Fab . During the learning phase, the input vector I0 , is presented to ART a and the desired output vector O0 , is presented to ART b . The ART a and ART b modules classify the input and the desired output vector into categories, then the map field (inter-ART module) makes associations from the ARTa category to the ART b category. Inter-ART module ARTa module

F a

ARTb module

ab

ρ

F2

F2

b

a

F1

b

a

ρa

x ab = ( x1ab ,..., x ab N b ) , and the weight vector from the jth ab F2 a node to Fab is denoted by wab = ( w ab j1 ,..., w jNb ) .

Vectors xa , ya , xb , yb , xab are initialised to zero between input presentations. Initially, the weight vectors wab are set to 1 (uncommitted). The inter-ART module Fab is activated whenever any of the ART a or ART b category is active. Hence,

x

ab

 y b ∧ wab J  ab  wJ  b y 0 

both the ARTa and ARTb is active ARTa is active, ARTb is inactive ARTa is inactive, and ARTb is active (12) both the ARTa and ARTb is inactive

During the learning phase, the ART a vigilance parameter ρa , equals the baseline vigilance, ρa at the beginning of

O = ( O 0, O 0c)

I = (I0, I0c)

F0

inter-ART module, Fab output vector is denoted by

reset

reset F1

y b = ( y1b ,..., y bN b ) . Mb and Nb are also arbitrary. For the

match tracking

F0

b

ρb

O0

I0

Fig. 8. A simplified Fuzzy ARTMAP architecture. If I0 predicts an incorrect O0 , then a mechanism called match tracking is triggered. This mechanism increases the vigilance parameter of ARTa, ρa , by a minimum value and, hence, forces the ART a module to search for another category suitable to be associated with the desired output vector. ρa is then set back to the baseline vigilance parameter, ρ, for every step of learning trial. The inputs to ART a and ART b are in complement code form, which is necessary for the successful operation of Fuzzy ARTMAP [3]. For ARTa, if the input is I o = (a1 , ... , a M a ) , then

every input presentation. When Fuzzy ARTMAP receives an input/output pair (I0 /O0 ), ART a chooses the Jth node of F2a and ART b chooses the Kth node of F2b . When both ART b and ART b are active and xab ≠ 0, then the input/output pairs are associated with the equation:

1 w ab jk =  0

j = J and k = K otherwise (13)

If xab = 0, then there is a mismatch. The Inter-ART module triggers a match tracking mechanism which increases ρa , by a minimum value and, hence, forces the ART a module to search for another category suitable to be associated with the desired output vector.

4. EXPERIMENTAL RESULTS In this section, we show some of the experimental results that have been obtained from the vision system that we have designed.

I = (a, a c ) = (a1 ,. .. , a M , a1c , ... , a cM ) (10) where

a ic = 1 − ai The

F1 a

1 ≤ i ≥ Ma

activity

vector

(11) is

denoted

by

x = ( x1a , .. ., x 2a M a ) and the F2a activity vector with Na a

number of category is denoted by y a = ( y 1a , .. ., y aN a ) . Ma and Na are arbitrary. For ART b , Oo = (b1 ,. .., b M b ) , O = (b , b c ) . The F1 b activity vector is denoted by

x b = ( x1b ,..., x 2b M b ) and the F2b activity vector with Nb

number

of

category

is

denoted

by Fig. 9(a) Recognition result from a license plate with 4 characters.

the distances of the vehicle from the camera. However, this can be solved through a subroutine.

Fig. 9(b) Recognition result from a license plate with 7 characters.

The system was also designed to solve the recognition problem of vehicles having licence plates with two-layer of characters such as that shown in Fig. 9(c). This format are normally used for the rear licence plates of many Malaysian vehicles. In another example the vision system can even recognize the licence plate characters of a slanted image. We also tested a number of images that are slightly blurred and found that the system was able to recognize blurred images up to a certain degree. However, research are being carried out to improve the recognition ability of the vision system for blurred images. 5. CONCLUSION During the development stage of the first proto-type of the vision licence plate recognition system, a number of problems were encountered. Here, we discuss the problems encountered and provide some suggestions as follows: •

Fig. 9(c) Recognition result from a license plate with 2-layer of characters.

•

Image acquisition module fails to locate the location of the license plate. Image acquisition module must deliver a clean image of license plate with minimum blur and reasonable resolution. To solve this problem, we need to use a higher quality CCD camera and frame grabber and also we need to control the illumination of the environment, for example at the indoor car park. Furthermore, by using the Otsu Thresholding operation also helps to improve this problem. Overlapping between 2 characters, as shown in Fig. 8. Using local blob analysis can solve this problem.

Local blob analysis Figure 9(d) Recognition result from a slanted image. This vision system is tested for a number of conditions. Figure 9(a) shows the result of a license plate with only 4 characters and the recognition processing speed was about 0.8 second. Figure 9(b) shows the recognition result of a vehicle with a licence plate consisting of 7 characters and the processing time was about 1.4 second. It can be observed that with more characters it takes longer to process and recognise the character. Figure 9(b) also shows that the vision system is able to solve the problem even with the vehicle located further way from the camera. We found that there is a limit to

Figure 10. Solving overlapping of 2 characters by blob analysis. •

Misclassification, for example between 'B' and '8' or 'S' and '5'. This could be overcome by using suitable other feature extraction techniques such as Sobel edge detection or moment invariant to extract more significant features from these characters. We can also train a neural network over a larger sample of training sets. We may also use some syntax checking method; such as the first character shall be a 'B' rather than an '8'.

•

License plate with rare font styles. This can be overcome by collecting the images of these rare fonts style and train the neural network with more samples.

[6] Rafael, A, Barroso, J., et al, "Number Plate Reading Using Computer Vision," Industrial Electronic, 1997 ISIE'97, Proceedings of IEEE International Symposium on Publish, vol.3, 1997, pp 761-766

The vision system that we have developed and discussed in this paper uses the Fuzzy ARTMAP neural network for the recognition of the licence plate characters, which is a relatively new neural network paradigm. Unlike the popular back-propagation algorithm which is used for training multilayer Perceptrons, the Fuzzy ARTMAP has the advantage of incremental learning and fast convergence [7].

[7] Tay, Y.H., Khalid, M., “Comparison of Fuzzy ARTMAP and MLP Neural Networks in Handwritten Character Recognition,” Pre-prints of International Federation of Automatic Control (IFAC) Symposium on Artificial Intelligent in Real -Time Control 1997 (AIRTC'97), 1997, pp.363-37.

The techniques designed in the vision system are able to detect and recognise normal vehicle license plates in Malaysia. Our techniques do not assume the presence of boundary line on the vehicle license plate. The average speed of identifying license plate is about 1.2s per plate. However, as the system has been developed locally it can be further improved. Further research is currently being carried out to solve several special vehicle’s license plates and also those from neighbouring countries such as Thailand and Singapore which make use of Malaysian roads. 7. REFERENCES [1] Kamat, V. and Ganesan, S, "An Efficient Implementation of the Hough Transform for Detecting Vehicle License Plates Using DSP'S," Cellular Neural Networks and Their Applications. Proceedings The IEEE 31st Annual 1997 International Carnahan Conference, 1997, pp 209218.

[2] Nijhuis, J.A.G, Brugge, M.H., Helmholt K.A., "License Plate Recognition Using DTCNNs," Security Technology, 1997. Proceedings 1998 Fifth IEEE International Workshop on Publish, 1998, pp 212-217.

[3] Carpenter, G.A., Grossberg, S., et al., “Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analogue Multidimensional Map,” Neural Networks, vol. 3, 1992, pp. 698-713.

[4] Carpenter, G.A. and Grossberg, S., “A Massively Parallel Architecture for a Self-Organising Neural Pattern Recognition Machine,” Computer Vision, Graphs, and Image Processing, vol. 37, 1987, pp. 54-115

[5] W.K Pratt, Digital Image Processing. New York: Wiley 1978.