COMBINING WAVELETS WITH COLOR INFORMATION FOR CONTENT-BASED IMAGE RETRIEVAL Mohamed A. Tahoun1, Khaled A. Nagaty2, Taha I. El-Arief2, Mohammed A-Megeed2 1. Faculty of Computers and Informatics, Suez Canal University, Egypt 2. Faculty of Computer and Information Science, Ain Shams University, Egypt. E-mails: [email protected], [email protected], [email protected] , and [email protected] Abstract. Content Based Image Retrieval (CBIR) is becoming an effective approach that used to retrieve images based on their visual features like color and texture. In this paper, we compared between the combination of wavelet-based representations of the texture feature and the color feature with and without using the color layout feature. To represent the color information, we used Global Color Histogram (GCH) beside the color layout feature and with respect to the texture information, we used Haar and Daubechies wavelets. Different categories of images have been tested by extracting color, texture and color layout features from them and used as the basis for a similarity test between a query image and the database images. The combination of GCH and 2-D Haar wavelet transform together with color layout feature gives the best retrieval accuracy and the results reflect the importance of using the spatial information beside the color feature itself for efficient content-based image retrieval systems. Keywords: Content-Based Image Retrieval, Global Color Histogram, Haar Wavelets, Daubechies Wavelets, Color layout, Euclidean Distance Measure.

1. Introduction The last few years have witnessed many advanced techniques evolving in Content-Based Image Retrieval (CBIR) systems. CBIR is considered as the process of retrieving desired images from huge databases based on extracted features from the image themselves without resorting to a keyword [1]. Features are derived directly from the images and they are extracted and analyzed automatically by means of computer processing [2]. Many commercial and research content-based image retrieval systems have been built and developed (For example: QBIC, MARS, Virage, Netra, and Photobook [3]). CBIR aims at searching image libraries for specific image features like colors and textures and querying is performed by comparing feature vectors (e.g. color histograms) of a search image with the feature vectors of all images in the database. The visual features are classified into low and high level features according to their complexity and the use of semantics [1]. The use of simple features like color or shape is not efficient [4]. When retrieving images using combinations of these features there is a need for testing the accuracy of these combinations and comparing them with the single features based retrieval in order to find the combinations that give the best matches that enhance the performance of CBIR systems. In fact, some CBIR systems give good results for special cases of database images as till now no standard data set for testing the accuracy of CBIR Systems [5]. So one of the most important challenges facing the evaluation of CBIR systems performance is creating a common image collection and obtaining relevance judgments [6]. The paper is organized as follows: section 2 briefly covers the feature extraction process using Global Color Histogram, Haar and Daubechies wavelets, and the color layout algorithm, in addition to showing how we constructed the features vectors and measured the distances among them. Section 3 presents the experimental results for comparing wavelets, and the combination of color, texture, and color layout features, and finally with concluding remarks.

2. Color and Texture Representations One of the most important challenges when building image based retrieval systems is the choice and the representation of the visual features [7]. Color is the most intuitive and straight forward for the user while shape and texture are also important visual attributes but there is no standard way to use them compared to color for efficient image retrieval while many content-based image retrieval systems use color and texture features [8]. In order to extract the selected features and index the database images based on them, we used Global Color Histogram (GCH) and Haar and Daubechies wavelets to extract color and texture features respectively then we constructed the color and texture features vectors and finally, the color layout feature is extracted from the database images and they are indexed based on the color layout algorithm.

2.1 Global Color Histogram Global Color Histogram (GCH) is the most traditional way of describing the color attribute of an image. It is constructed by computing the normalized percentage of the color pixels in an arrange corresponding to each color element [7]. To construct the color feature vector (its length is 256×3) for both the query image and all images in the database, we identified the three-color components (R, G, and B) and compute the corresponding histograms of these

components. The histograms for all database images will be saved in the feature vector database. To test the similarity between each two feature vectors one for the query image and the other for each image in the database, we used Manhattan Distance (1) as the histogram comparison distance measure: G

Di ,k = ∑ j =1

H i ( j) H k ( j) − M i * Ni M k * Nk

(1)

Where Hi(j) denote the histogram value for the ith image, j is one of the G possible gray levels, M i * N i is the number of pixels in an image i, M k * N k is the number of pixels in image k, and M is the number of rows and N is the number of columns. We calculated the difference between each two corresponding histograms of the three components Red, Green, and Blue and then we used the following transformation (2) to convert the three distances into one distance that will be sorted and used as the basis for displaying the results:

DC = 0.299* DC (R) + 0.587* DC (G) + 0.114* DC (B)

(2) Where D is the distance between the two color feature vectors, and D (R) , D (G) , and D (B) are the distances between each two corresponding components for Red, Green, and Blue respectively [9]. The obtained distances after comparing the query image with all the database images will be sorted and the corresponding images are displayed during the retrieval process. C

C

C

C

2.2 Haar and Daubechies Wavelets Wavelet transform can be used to characterize textures using statistical properties of the gray levels of the points/pixels comprising a surface image [10]. The wavelet transform is a tool that cuts up data or functions or operators into different frequency components and then studies each component with a resolution matched to its scale. There are different types of wavelet families whose qualities vary according to several criteria. Daubechies is one of the brightest stars in the world of wavelet research invented what are called compactly supported orthonormal wavelets thus making discrete wavelet analysis practicable. Daubechies family includes the Haar wavelet, written as ‘DB1, the simplest wavelet imaginable and certainly the earliest. Formulas (3) and (4) illustrate the mother wavelets for the Haar wavelet: (3)

(4)

Fig . 1 The approximation (on the left) and the details of an image (on the right)

Where φ is called the scale of the Haar wavelet and ψ is the actual wavelet [10]. The approximation A represents the image at a coarser resolution, it results from averaging in both directions of the image, x and y directions. The horizontal details H obtained by averaging in x-direction and differencing in the y-direction. The vertical details V obtained by averaging in y-direction and differencing in the x-direction. The diagonal detail D obtained by differencing in both directions then averaging. Fig. 1 illustrates that the horizontal edges appear in H (top right), Vertical edges in V (down left), and all other edges in D (down right). The mother wavelets of Haar and the other Daubechies wavelets are displayed in Fig. 2.

DB1

DB2

DB4

DB8

Fig. 2 From left to right: The mother wavelets of Haar (DB1) and Daubechies DB2, DB4, and DB8

The wavelet analysis of an image gives four outputs at each level of analysis l (we used 3-levels), one approximation and three details: the approximation Al, horizontal details Hl, vertical details Vl, and diagonal details Dl (Fig. 3). (b)

(c)

(a)

H3 D3

A3 V3

H2

V2

Horizontal Details

H1

D2

Vertical Details

Diagonal Details

V1

D1

Fig. 3 (a) A three-level wavelet analysis (an approximation and three details (Horizontal (H), Vertical (V), and Diagonal (D)), (b) An original image and in (c) the three levels applied to the original image.

Wavelet decomposition allows for a good image approximation with some few coefficients which provide information that is independent of the original image resolution [11]. We created the texture feature vector using two different ways: the first way is to calculate the energy of each subband and then use the Euclidean distance to measure the difference between each two texture feature vectors, while in the other way, we used the details of the wavelet analysis to construct the texture feature vector without calculating the energy and measure the distance between the feature vectors using a similarity measurement called point to point similarity. In the first case: The energy of each sub-band image is calculated using the following relation (5):

1 E= MN

m

n

∑ ∑ X (i , j )

(5)

i =1 j =1

Where M and N are the dimensions of the image, and X is the intensity of the pixel located at row i and column j. The texture feature vector will consist of the energies of horizontal, vertical and diagonal details for the three levels of analysis (the texture feature vector length is 9). We used the Euclidean distance (6) to measure the distance between each two sets of energies (texture feature vectors) one for the query image and the other for every image in the database and this process is repeated until comparing all images in the database with the query image [7].

D = T i

2

K

∑ (x k =1

k

− y k ,i )

(6)

Where K is the length of the texture feature vector, i represent the ith image in the database, and DiT is the Euclidean distance between the query image feature vector x and the feature vector of the ith image in the database yi . In the other way, we used a similarity measurement called point to point similarity which is related to the wavelet decomposition approach where the texture feature vectors are constructed based on the sub-bands or the details themselves resulting from the 3-levels wavelet of analysis and in this case, we did not compute the energies of these sub-bands (the texture feature vector length is also 9). By using wavelet analysis, we can define a similarity distance S (( x, y ), ( x′, y′)) for any pair of image points on the reference image f ( x, y ) and the matched image f ′( x′, y′) on any given level [12]. If we consider single point to point match on the jth level, the similarity distance can be defined using three differential components Dj,p f (x, y), p = 1, 2, 3. By using the following feature vector (Bj) in (7): (7) Where , p = 1,2, 3 Where

denotes to L2 norm and A j ( x, y ) denotes to the approximation, we can define a normalized similarity

distance as formulated in (8) [12]:

SB j (( x, y ), ( x′, y′)) = B j ( x, y ) − B′j ( x′, y′)

(8)

In order to calculate the similarity distance for the whole image, we calculated the arithmetic mean of all similarity distances which defined as the sum of their values divided by the number of pixels then we can get a single distance value between each query image and all images database.

2.3 Color Layout Color histogram does not include any spatial information about an image and for this reason; many research results suggested that using color layout (both color feature and spatial relations) is a better solution in image retrieval [5]. In traditional color layout image indexing, the image is divided into equal-sized blocks and then the average color is computed on the pixels in each block [13]. These values are stored for image matching using similarity measures. In our experiments, the steps for creating the color layout feature vector from an image are: • Divide the image into 16x16 sub-blocks. • Extract the color feature components for each sub-block (Identifying the three components R, G, and B for each block). • Calculate the average for each of the three components in each sub-block. • Then construct the color layout feature vector (16x16x3) that will represent the color layout feature. In order to test the similarity between each two color layout feature vectors one for the query image and the other for each image in the database, we used the Euclidean distance measure (9) [7]:

Di

CL

=

2

S

∑ (M s =1

s

− N s ,i )

(9)

where S is the length of the color layout feature vector, Di CL is the Euclidean distance between the query image feature vector M and the feature vector of the ith database image Ni . After calculating the distances between each two color components, we transformed them into one distance value (10) [9]: (10) D CL = 0.299 * DCL ( R) + 0.587 * D CL (G ) + 0.114 * D CL ( B) where D CL is the final Euclidean distance between the two color layout feature vectors and DCL (R) , D CL (G ) , and D CL (B) are the Euclidean distances between each two corresponding components for Red, Green, and Blue respectively.

3. Experimental Results The images database contains 300 compressed colored (RGB) images downloaded from the internet [14]. The images collection is classified into eight categories (Buses, Horses, Roses, Buildings, Elephants, Beach, Food, and People) and all images are in size 384x256 pixels in jpg format. The experiments were run on: Pentium IV, 2.4 GHz Processor, and 256 MB RAM using Matlab version 6. The experiments started with the features extraction process (based on GCH, wavelets, and the color layout feature algorithm) that is used to create the features vectors for each image in the database that will be stored to be ready for the matching process (offline processing). When starting with a query image the same process will be done for each query image (online processing). The comparison between each two feature vectors (one for the query image and the other for each database image) is performed using Euclidean distance and point to point similarity measures explained in the previous section. The resulted distances are normalized and sorted respectively then used as the basis for retrieving database images that are similar to the query [7]. Our experiments will have some main comparisons include: comparing Haar and Daubechies wavelets based on the Euclidean distance measure, and then using the best wavelet to test the combination of color and texture with and without adding the color layout feature. In order to test the performance of our CBIR system, the accuracy test was done to find the best results when comparing wavelets and testing their combination with color and color layout features. The retrieval accuracy (11) is defined as the ratio between the number of relevant (belongs to the same category) retrieved images and the total number of retrieved image (known as a single precision) [6]. Retrieval Accuracy =

No. of relevant retrieved images Total No. of retrieved images

(11)

Haar and Daubechies wavelets are compared together based on Euclidean distance. Fig. 4 shows that the 2-D Haar wavelet transform applied to grayscale images gives the best retrieval accuracy among the other used wavelets include Haar based on color components, DB2, DB4, and DB8. On the other hand DB2 gives the best results when using the point to point similarity measurement (in this case all wavelets are directly applied to colored images without changing

them into a grayscale, but the general retrieval accuracy is still less than the one obtained when using the Euclidean distance, Fig. 5). 57%

76%

Retrieval Accuracy

74%

73%

72% 70%

69%

68%

68% 66%

66% 64% 62%

Retrieval Accuracy

74%

56.13%

56%

55.38%

55%

54.38%

54% 53% 52%

51.75%

51% 50% 49% Haar

60% Haar (gray)

Haar(color)

DB2

DB4

DB8

Haar and Daubechies Wavelets

Fig. 4 A comparison of Haar and Daubechies wavelets using Euclidean distance measure

DB2

DB4

DB8

Haar and Daubechies

Fig. 5 A comparison of Haar and Daubechies wavelets using point to point similarity distance (all based on colored images)

The combination of Haar and Daubechies wavelets with Global Color Histogram (GCH) and the color layout feature is also tested using the Euclidean distance. Fig. 6 illustrates that the 2-D Haar wavelet transform gives better retrieval accuracy than the other used wavelets when combined with GCH and the color layout feature using the Euclidean distance measure. On the other hand, when comparing this combination with the effect on using GCH alone or the combination of GCH and 2-D Haar wavelet without using the color layout feature, we remarked that it gives the best retrieval accuracy and considered as the best obtained combination.

Retrieval Accuracy

100% 95%

94%

88%

90% 84%

85%

86%

85%

80% 75% Haar (gray)

Haar (color)

DB2

DB4

Haar and Daubechies Wavelets

DB8

Fig. 6 The combination of Haar wavelets with color and color layout using the Euclidean distance Fig. 7 (a) Global Color Histogram alone

Fig. 7 (b) The combination of GCH and Haar wavelets

Fig. 7 (c) The combination of GCH and Haar wavelets with the color layout feature

Fig. 7 is an example demonstrates that the retrieval accuracy when combining Haar wavelet with color and color layout features is better than using GCH alone or the combination of GCH and Haar wavelet without using the color layout feature as in (a) there are only 2 relevant images, in (b) 8 relevant images, and in (c) 12 relevant ones. Fig. 8 depicts the general retrieval accuracy of GCH alone and the combination of GCH and 2-D Haar wavelet with and without adding the color layout feature after testing the eight categories of the database images. In order to show the change in the retrieval accuracy as noise is added, a normal noise with different variances is added to the images. The noisy images are used as the query images and the rank of the original images is observed. We added six levels of noise (from 10% to 60% - named N10 to N60) to each query image then we compared between the results obtained when using the noisy query images and the ones obtained using the original images.

90%

80%

77%

81%

64%

60% 40% 20%

Retrieval Accuracy

Retrieval Accuracy

100%

80% 70% GCH

60% 50%

GCH+2D Haar

40% 30%

All Features

20% 10% 0%

0%

N10 GCH

GCH + Haar

Techniques

GCH +Haar + Color layout

Fig. 8 The general retrieval accuracy of the used techniques

N20

N30

N40

N50

N60

Six Levels of Noise (from 10% to 60%)

Fig. 9 The effect of noise on the retrieval accuracy with respect to original images

In Fig 9, the retrieval accuracy (with respect to original images) when applying the six levels of noise which shows that combining GCH and 2-D Haar wavelet transform in addition to the color layout feature gives the best retrieval accuracy with respect to the original images.

4. Conclusions The need for efficient content-based image retrieval systems becomes a must and the choice and the representation of the visual features when building CBIR systems are important tasks. In this paper, a comparison has been done between Global Color Histogram (GCH) alone and the combination of GCH and Haar and Daubechies wavelets with and without adding the color layout feature. Different categories of images have been tested based on these techniques and the results show that using GCH and 2-D Haar wavelet transform gives good results and when combining them with the spatial information the retrieval accuracy increases. Finding and testing the combinations of the features extractions techniques increase the retrieval accuracy and help enhance the performance of CBIR systems.

5. References [1] John Eakins and Margaret Graham, “Content-based Image Retrieval”, JISC Technology Applications Programme. University of Northumbria at Newcastle. January 1999. http://www.unn.ac.uk/iidr/report.html [2] Christopher C. Yang, “Content-based image retrieval: a comparison between query by example and image browsing map approaches “, Journal of Information Science, pp. 254–267, 30 (3) 2004. [3] Rui Y. & Huang T. S., Chang S. F. “Image retrieval: current techniques, promising directions, and open issues”. Journal of Visual Communication and Image Representation, 10, 39-62, 1999. [4] Karin Kailing, Hans-Peter Kriegel and Stefan Schönauer, “ Content-Based Image Retrieval Using Multiple Representations”. Proc. 8th Int. Conf. on Knowledge-Based Intelligent Information and Engineering Systems (KES'2004), Wellington, New Zealand, 2004, pp. 982-988. [5] Ahmed M.Ghanem, Emad M. Rasmy, and Yasser M. Kadah, “Content-Based Image Retrieval Strategies for Medical Image Libraries,” Proc. SPIE Med. Imag., San Diego, Feb. 2001. [6] Henning Muller, Wolfgang Muller, David McG. Squire and Thierry Pun, “Performance Evaluation in ContentBased Image Retrieval: Overview and Proposals”. Computing Science Center, University of Geneva, Switzerland, 2000. [7] Vishal Chitkara, “Color-Based image Retrieval Using Binary Signatures“. Technical Report TR 01-08, University of Alberta, Canada, May 2001. [8] Qasim Iqbal and J. K. Aggarwal, “Combining Structure, Color, and Texture for Image Retrieval: A performance Evaluation”. 16th International Conference on Pattern Recognition (ICPR), Quebec City, QC, Canada, August 1115. 2002, vol. 2, pp. 438-443. [9] Web Sites: http://en.wikipedia.org/wiki/YIQ and http://v4l2spec.bytesex.org/spec/colorspaces.html [10] M.G. Mostafa, M.F. Tolba, T.F. Gharib, M.A. Megeed, ” Medical Image Segmentation Using Wavelet Based Multiresolution EM Algorithm”. IEEE International Conference on Industrial Electronics, Technology, & Automation., Cairo IETA’2001. [11] Charles E. Jacobs, Adam Finkelstein, and David H. Salesin, “ Fast multiresolution image querying”. In Proceedings of SIGGRAPH 95, Los Angeles, August 6-11 1995. ACM SIGGRAPH, New York. [12] Jiri Walder, “Using 2-D wavelet analysis for matching two images”, Technical. University of Ostrava.2000. http://www.cg.tuwien.ac.at/studentwork/CESCG-2000/JWalder/ [13] James Ze Wang, Gio Wiederhold, Oscar Firschein, and Sha Xin Wei, ”Content-based image indexing and searching using Daubechies' wavelets”. Intl. Journal of Digital Libraries (IJODL), 1 (4):311-328, 1998 [14] Web Site: http://wang.ist.psu.edu/~jwang/test1.tar.