IMAGINATION-BASED IMAGE SEARCH SYSTEM WITH DYNAMIC QUERY CREATION AND ITS APPLICATION Diep NGUYEN-THI-NGOC1, Shiori SASAKI2 and Yasushi KIYOKI1 1 Faculty of Environment and Information Studies, Keio University 2 Graduate School of Media and Governance, Keio University 5322 Endo, Fujisawa, Kanagawa 252-0882 JAPAN {t09509tn, sashiori, kiyoki}@sfc.keio.ac.jp ABSTRACT An imagination-based image search system is a new environment to acquire unknown but desired images by queries, which reflects user’s dynamic imagination process. It leads to a new computation environment for searching images data resources in a contextual way. This paper presents a dynamic image-query creation method for imagination-based image search system and its application for travel information associated with scenery images. The main feature of our system is to extend analytical functions for image search, not only in retrieval processing, but also in query manipulation, according to the color-based combination of images with common features. A query creation method in this system is a novel approach to represent a user’s imagination process. This method enables a user to create dynamically a query, which reflects the user’s intention, impression and memory as his/her own context existing only his/her mind by color-based combinations of existent images in the real world. The proposed method consists of five operations for creating image-query vector from combinations of images, which are “plus”, “intersection”, “accumulation”, as local operations and “minus” and “difference” as global operations. Using the imagination-based image search system for travel information associated with scenery images, users can easily discover images and information of places where they never been. The system performs a series of qualitative and quantitative experiments to examine the feasibility, effectiveness of proposed method and scalability of the system. KEY WORDS Multimedia system, Image retrieval, Image-query creation, Context-dependent retrieval, Color feature extraction

1. Introduction As rapidly growing multimedia technology, a large number of various types of multimedia data such as images, audio and motion pictures are created and distributed widely on WWW. Especially image database is increasing in quality by the development of picture captured devices and storage or sharing such as community-based multimedia sharing sites Flickr [1], Picasa [2], Webshots [3], Photobucket [4], etc.

To increase opportunities to access to enormous amounts of image data efficiently and appropriately, there are two major approaches: text-based image retrieval and content-based image retrieval (CBIR). The text-based approach is to implement image retrieval by attaching keywords to images or retrieving text around images. Image search systems such as Imagery [5], Google Images Search [6], Yahoo! Image Search [7] adopt this approach. The other approach, content-based image retrieval is increasing in the very wide domains [8] and a lot of systems have been developed in the academia and the industry. This approach has two significant query techniques: query-by-an-image and query-by-sketch. Query-by-an-image search systems such as PicToSeek [9], SIMPLIcity [10], TinEye [11] and GazoPa [12] have been implemented by extracting low-level visual features such as color histograms, shapes, textures and structure to calculate the “similar” images. Query-by-sketch systems have been constructed to express the user’s intentions by sketching features of the image, which the user wants to retrieve [13]. Some recent online systems such as Multicolr Search Lab [14] and GazoPa [12] demonstrate this approach. However, a system to express the user’s intentions directly and fully has not been established even in these conventional query methods. From the point of view expressing intentions by using images, query-by-an-image is limited to a context shown by one sample input, and query-by-sketch requires detailed drawings to users for expressing their precise intentions. Thus, an intelligent multimedia search system, which responds to human’s imagination process as adequately as possible, has been required. In this paper, we propose a novel method to create an image-based context query by color-based combination of image in the real world. Our system extends analytical functions for image search, not only in retrieval processing but also in query manipulation. In this system, user’s intention, impression and memory are represented as context query by the combinations of colors. Though a color palette is available to generate simple color combinations [14], image data are more useful for creating complex color combinations. Moreover, image data in the real world such as photos of scenery are more effective to represent user’s context because those are highly connected to human’s impressions and memories. A user of this system creates a context query that represents his/her own inner intention, impression and

memory for specific things or places, and retrieve unknown but desired images with the associated information according to his/her own imagination by using the operations in our method. In this system, two steps define user’s imagination process. The first step is to select multiple images. In this step, a user selects a set of images, which includes desired colors (“With” image data set) and another set of images, which includes undesired colors (“Without” image data set) to represent their own intentions, impressions and memories. The second step is to create the combinations of colors. In this step, a user creates a query by using operations equipped to this system. To create image-query, our system provides several operations, which are plus, intersection, accumulation, difference and minus. Plus, intersection, accumulation operations can be used for “With” image data set and difference and minus operations can be used for “Without” image data set. By the combinations of these operations and image data sets, a user can create a query to represent his/her complex context dynamically. As related work, we refer to several researches on the relationships between color combinations and impressions. A psychological research on color combinations and human impressions [11], a semantic image retrieval method using the knowledge on colors and impressions [12], an impression metadata extraction method from image data [13] and an culture-based image-query creation method [14] have been proposed already. Based on these previous researches, we developed our dynamic image-query creation method for an imagination-based image search system. This paper also presents an easy-to-use web application for travel information to verify the feasibility of this image-query creation method and imaginationbased travel information search system. Users of this imagination-based travel information search system can easily discover images and information of places where they never been. We examine the effectiveness of our system as a web application in community-based multimedia sharing sites by implementing Query Saver function to share the created image-queries with imageURL and other information as knowledge memory.

2. Basic Method 2.1 Features of our method In this section, we present the main functionality of our image-query creation method and image retrieval environment especially for color-based imagination representation reflecting a user’s context. Our image-query creation method is based on the following observation results: (1) user’s context is too complex to be represented easily by a single image; (2) a part of user’s context can be represented by a set of multiple images in the real world such as photos of scenery because those images are highly connected to

human’s impressions and memories, and (3) color feature of images is effective to create user’s context query because it evokes human’s imagination. The main feature of our method is to create dynamically a query, which reflects the user’s intention, impression and memory as his/her own context existing only his/her mind by color-based combinations of existent images in the real world. First, the system extracts color histograms as perceptual features of a set of images, which a user input as his/her own context. Second, the system generates an image-query according to the color-based combinations of images with common features, which cannot be extracted from a single image. Third, the system retrieve target images from collected image database according to the user’s complex context, and provide associated information such as desired images and information of places where he/she never been to. 2.2 Query creation operations After calculating the area ratio of color distribution of each sample image data, the image-query is created by the combination of multiple image sets and the following operations: Operation 1 to create vector Qplus by the sum of each color bin from all the sample images in a set to increase color-features, Operation 2 to create vector Qintersection by the commonly-used color from all the sample images in a set, Operation 3 to create vector Qaccumulation by taking the dominant colors among all images in a set, Operation 4 to create vector Qminus by decreasing color-features of a single sample image to any other sample images in a set, Operation 5 to create Qdifference by the colors less frequently used in a single sample image compared with any other sample images in a set. For given n sample images (sl1,sl2,…,sln: l is a set identifier) which represent p sets of images (l1,l2,…,lp), color histograms are generated. The generated m-bin histogram consists of m basic color (c1,c2,…,cm). The relationships between each sample image and sample image sets are not necessarily prepared as mutually exclusive relationships. It is possible that an image is included in multiple image sets, and a set of images is included in other sets of images. The m-bin histogram for each sample image representing each image set is defined as color-image set vector sk={slk1, slk2,…, slkm: l is a set identifier}, and n by m matrix consisting of the color-image set vectors as row vectors is defined as color-image set matrix C as shown in Figure 1. In other words, the color-image set matrix C represents the color features of each image set as numerical values (q11,q12,…,qnm) of color histograms of n sample images data.

2.3 Color-based combination settings for image data

Figure 1. Color-Image set Matrix C In this image-query creation method, a query vector is constructed from the color features of image sets by using the column vectors of the color-image set matrix C. 

Operation 1: Plus To create a sub-query vector Qplus, the sum values of color elements are calculated from all the colors used in a set of sample images. The sum values are calculated by every color element (c1,c2,…,cm), that is, by each column vector of matrix C with following formula:

In our system, we set two kinds of image data set: “With” set and “Without” set. After calculating the area ratio of color distribution of each sample image data, an imagequery is generated by the combination of two kinds of operations: “Local” operation and “Global” operation. The “With” image set is composed of desired images which user input. The colors used in this image set represent a user wants to use for creation of his/her imagination about things or places. The “Without” image set is composed of undesired images which user input. The colors used in the image set represent the user want to delete for creation of his/her imagination. Each image set consists of multiple images. By each of the “With” and “Without” image sets, the “Local” operation generates a “sub-query”, respectively. Then, the “Global” operation generates an integrated query by two sub-queries. The relations between dataset and operations are represented in Figure 2, and the proposed operations are listed in Table 1.



Operation 2: Intersection To create a sub-query vector Qintersection, the minimum values of color elements are calculated from all the colors used in a set of sample images. The minimum values are calculated by every color elements (c1,c2,…,cm), that is, by each column vector of matrix C with following formula Qintersection= 

Operation 3: Accumulation To create a sub-query vector Qaccumulation, the maximum values of color elements are calculated from all the colors used in a set of sample images. The maximum values are calculated by every color elements (c1,c2,…,cm), that is, by each column vector of matrix C with following formula:

Qaccumulation = (max(q11,...,qn1),...,max(q1m ,...,qnm )) 

Operation 4: Minus To create an integrated query vector Qminus, the minus values of color elements are calculated from a single image to any other image in image set. The formula of operation: Qminus= (q11-q21-…-qn1,q12-q22-…-qn2,…, q1m-q2m-…-qnm) if (q1k-q2k-…-qnk<0) then q1k-q2k-…-qnk =0 (k=1..m)



Operation 5: Difference To create an integrated query vector Qdifference, the difference values of color elements are calculated from the colors only used in a single image to any other images in image set. The proposed formula (1): Qdifference = (q1,q2,…,qm) if (q1k>Hdif.max(q2k,…,qnk) then qk= q1k else qk=0 (k=1..m) while Hdif is a constant (Hdif >=1)

Figure 2. Relations between dataset and operations Table 1. Operations of our method “Local” operation

“Global” operation

Qplus(Set)

Qminus(Sub-query1, Sub-query2)

Qintersection(Set)

Qdifference(Sub-query1, Sub-query2)

Qaccumulation(Set)

Qintersection(Sub-query1, Sub-query2)

The “local” operation can be applied to color-image vector matrix C to create sub-query vector. The “global” operation can be only applied to two sub-query vectors s1(q11,q12,…,q1m) and s2(q21,q22,…,q2m). After creating image-query histogram q1,q2,…,qm, normalization process will be applied to it to create normalized query histogram. qj 

qj m

 qi i 1

( j  1..m)

3. Implementation of Image Search System

Imagination-based

3.1 System Architecture and Functions

names of places are extracted from the header and path of the website that includes the target image. The Correlation Calculator is a function to calculate correlations between a created image-query vector and target image vectors. We have implemented several distance calculation methods between color histograms such as Cosine Measure, Manhattan Distance, Euclidean Distance, Inner Product and Histogram Intersection. The Query Saver is a function to save created imagequeries as knowledge memory with image-URL and other information. The users can reuse the image-queries created by other people for image retrieval of other domains. 3.2. Web Application Interface

Figure 3. System Architecture of Imagination-based Image Search System We implemented an imagination-based image search system with our image-query creation method as a Web application. The system architecture is shown in Figure 3. The system is constructed by the folowing 7 functions: Context Interpreter, Color Analyzer, Combined Query Creator, Metadata Extractor, Correlation Calculator , Web Crawler and Query Sever.Context Interpreter is a query processing function to specify images from “With” image set and “Without” image set, send these image data to Color Analyzer function, and display these images to retrieval result windows. The Color Analyzer is a function to extract color features of input images and target images, and generate color histograms of them. In this function, we use a method proposed by A. Vadivel et al in [19] to extract color histogram with 284 bins (252 chromatic colors and 32 monochrome colors). The Combined Query Creator is a function to create an image-query vector dynamically by input image data sets and operations selected by a user as his/her context from color histograms generated by the Color Analyzer function. The Web Crawler is a function to collect websites and images to create target database from WWW automatically. Users can increase this number of indexes in target database by inputting a specific travel site for crawling, or inputting specific information about one image. For experiments, we selected a travel site [21] and collected 3769 images to construct target image database. The Metadata Extractor is a function to create metadata database of target images. The generated color histograms of target images by Color Analyzer are stored in the metadata database with other information such as image-URL, website-URL and names of places. WebsiteURL is a site URL which includes a target image, and

We implemented a user interface of our imaginationbased image search system as an easy-to-use web application. Figure 3 shows a screen shot of the user interface. Users can easily use this system by inputting images in box 1 and box 2. A user inputs one or several images, which include desired colors in box 1, and one or several images, which include undesired colors in box 2. Then, a user selects a type of distance measure by pull-down menu 7, and a combination operation by pull-down menu 8, 9, 10. Pull-down menu 8 is for combining images in box 1 and pull-down menu 9 is for combining images in box 2. Pull-down menu 10 is for combining images in box 1 and box 2 if there exist inputted images in both. The optional menu box 3 is used to specify a specific domain or region of result by the name of places. Button 4 allows users to insert more images into target database. Button 5 allows users to view the created queries in history and share them among community. After inputting images and selecting an operation as calculator, a user presses button 6 for searching.

Figure 4. Web application interface

4. Experiments 4.1 Experiment 1: examination on the feasibility of query creation method

In this experiment, to verify the feasibility of our proposed method of query creation and compare the different meaning of each operation, we performed qualitative experiments with the image data of scene and flower collected from the Internet. The number of target image data is 225. The types of image and included perceptual colors are: flower (yellow, orange, pink), sky (blue), sea (blue), sunset (orange), field (yellow), mountain and forest (green). With queries to which we applied each operation, we performed image retrieval and evaluated the correctness of the results by visual judgment. We calculated precision rate, recall rate and F-measure in the top ten result images. For this experiment, we selected Cosine Distance for distance calculation between color histograms. The search results by each operation are shown in Table 2, Table 3, Table 4, Table 5 and Table 6. Table 2 shows the image search results by using Plus operation for query creation. The input images are an image of blue sky with white clouds and an image of yellow flower with green background for “With” (desired) images. The created and combined image-query is shown in the visualization of color histogram. The histogram is created by addition operation on mathematics (Arithmetic operation), which means the combination of values of color bins in two histograms. The search results show that nine images of top 10 are correct because these images are including blue, white, yellow and green in the reasonable proportion.

an image of yellow flower with green background for “With” (desired) images. The created and combined image-query is shown in the visualization of color histogram. The histogram is created by intersection operation on set operations, which means the combination of the same (common) color bins in two histograms. The search results show that seven images of top 10 are correct because these images are including yellow, a little green in the reasonable proportion. Table 3. Results by Intersection operation: Images for query creation, image histograms, a combined query histogram and the search results

Intersection

Combined query histogram Yellow is extracted for “only yellow flower”

Result precision = 7/10 Recall = 7/14 F-measure = 0.58

Table 2. Results by Plus operation: Images for query creation, image histograms, a combined query histogram and the search results

Plus

Combined query histogram Blue and yellow are added for “yellow flower on blue sky”

Result precision = 9/10 (90%) Recall = 9/27 (33%) F-measure = 0.49

Table 4 shows the image search results by using Accumulation operation for query creation. The input images are an image of sunset sky and an image of red leaf in blue sky for “With” (desired) images. The created and combined image-query is shown in the visualization of color histogram. The histogram is created by union operation on set operations, which means the combination of the dominant values of color bins in two histograms. The search results show that ten images of top 10 are correct because these images are including red, orange, and dark brown in the reasonable proportion and red leaf and sunset in sense. Table 4. Results by Accumulation operation: Images for query creation, image histograms, a combined query histogram and the search results

Accumulation

Table 3 shows the image search results by using Intersection operation for query creation. The input images are an image of yellow flower with blue sky and

Combined query histogram Many red-orange is mixed for “sunset sky and red leaves”

Result precision = 10/10 (100%) Recall = 10/21(48%) F-measure = 0.65

sky for “With” (desired) image and an image of sunset sky for “Without” (undesired) image. The created and combined image-query is shown in the visualization of color histogram. The histogram is created by complement operation on set operations, which means the combination of color bins existing only in “With” image histogram. The search results show that eight images of top 10 are reasonable because these images are including blue color in the reasonable proportion. Table 6. Results by Difference operation: Images for query creation, image histograms, a combined query histogram and the search results

Table 5 shows the image search results by using Minus operation for query creation. The input images are an image of an image of yellow flower with green leave background for “With” (desired) image and an image of green leaf for “Without” (undesired) images. The created and combined image-query is shown in the visualization of color histogram. The histogram is created by subtraction operation on mathematics (Arithmetic operation), which means the combination of subtracted values of color bins of “without” histogram from “with” histogram, and setting minus values as 0 values. The search results show that eight images of top 10 are correct because these images are including yellow and green in the reasonable proportion.

Difference

Combined query histogram Only blue is remained for “blue sky or sea”

Result precision = 8/10 (80%) Recall = 8/25 (32%) F-measure = 0.46

Table 5. Results by Minus operation: Images for query creation, image histograms, combined query histogram and the search results

Minus

The evaluation of F-measure in this experiment is shown in Figure 5. The figure show that the mean of Fmeasure is 0.57 and the query by Accumulation operation and Minus operation leaded good results at least in this experiment.

Combined query histogram Green is subtracted for “less green”

Result precision = 8/10 (80%) Recall = 8/14 (57%) F-measure = 0.67

Table 6 shows the image search results by using Difference operation for query creation. The input images are an image of an image of sunset on the sea with blue

Figure 5. F measure evaluation on Experiment 1

4.2 Experiment 2: examination on the effectiveness as a web application In this experiment, to verify the effectiveness and scalability of real web application, we applied more complex context to image retrieval using the travel website and the place information. Top 20 result images will be listed first to evaluate the effective (precision) of real application. For this experiment, we also selected Cosine Distance for distance calculation between color histograms. The search results are shown in Figure 6, Figure 7, and Figure 8. Figure 6 shows the image search results by using Accumulation operation for query creation. We set a supposed user’s imagination (context) for query as “buildings besides beach having green tree (forest or mountain).” To express this imagination, an image of city with white building, an image with blue sea wave and an image of green mountain were input for “With” (desired) image data set. The selected area of the world was “United States”. The created and combined image-query is shown in the visualization of color histogram in Result Page. The search results show that sixteen images of the top 20 images were correct (precision = 80%) because these images have reasonable imagination such as white building, green tree and blue sea. The figure shows that the system displays the images, the travel website link and the map from Google map [22] for place so that a user can easily discover images and information of places where he/she never been.

Figure 6. Travel-related image search results by a query created by Accumulation operation, an example of website page and a map for place Figure 7 shows the image search results by using both Difference and Accumulation operation for query creation. The supposed imagination for image-query was “a desert with green trees/mountains without sky”. To express this imagination, the input images were an image of green tree in blue-sky background, an image of desert for “With”

(desired) images and an image of blue sky with white clouds for “Without” (desired) images. The selected area of the world was “Africa”. The created and combined image-query is shown in the visualization of color histogram in Result Page. The search results show that thirteen images of the top 20 images are reasonable (precision = 65%) because these images have reasonable imagination: brown desert with green trees/plants.

Figure 7. Travel-related image search results by a query created by Accumulation and Difference operation Figure 8 shows the image search results by using both Difference and Plus operation for query creation. Compared with the search results shown in Figure 7, we set a supposed user’s imagination (context) for query as “a desert with more green trees without sky”. To express this imagination, an image of green tree in front of bluesky background was input twice. The created and combined image-query shown in the visualization of color histogram in Result Page in Figure 8 includes more green colors than in Figure 7. And the search results show that sixteen images of this top 20 are correct (precision = 80%) because these images have reasonable imagination: brown desert with more green trees/plants.

Figure 8. Travel-related image search results by a query created by Plus and Difference operation 4.3. Experiment 3: examination by user comparison In this experiment, we apply performance evaluation by users [20]. This method of performance evaluation is User Comparison, which is known as an interactive method. In this method, the users judge the success of a query

directly after the query. We requested the users to evaluate the search results by 50 queries and asked them about the “precision” as the success of a query at the top 20 result images. The users were able to use any query image and apply any operations, which represent their imagination. As a result, the mean of precision was 72%, which shows that our search system with proposed dynamic query creation method is reasonable.

experimental results showed that our proposed method is reasonable. As future work, we will perform quantitative experiments by increasing data for target databases and improve the performance of the system We have also a plan to use more features of image such like shape or structure to allow user to represent his/her imagination context more sufficient and easier. Additionally, by introducing timeline to support more information about travel, we will make the system more intelligent.

References [1] [2] [3] [4] [5] [6] [7] [8] [9]

Figure 8. User Comparison: Performance Evaluation

5. Conclusion and Future work In this paper, we have presented a dynamic image-query creation method for imagination-based image search system and its application for travel information associated with scenery images. Our imagination-based image search system leads to a new computation environment for searching images data resources in a contextual way. The main feature of our system is to extend analytical functions for image search, not only in retrieval processing, but also in query manipulation, according to the color-based combination of images with common features. A query creation method in this system is a novel approach to represent a user’s imagination process. This method enables a user to create dynamically a query, which reflects the user’s intention, impression and memory as his/her own context existing only his/her mind by color-based combinations of existent images in the real world. The proposed method consists of five operations for creating image-query vector from combinations of images, which are “plus”, “intersection”, “accumulation”, as local operations and “minus” and “difference” as global operations. We have implemented the web application for this system, with a large amount of image data that is crawled automatically from the Internet. We have showed the feasibility and effectiveness of the proposed method by experimental results when using this system. The

[10] [11] [12] [13] [14] [15] [16]

[17]

[18]

[19]

[20] [21]

[22]

Flickr Photo Sharing, www.flickr.com Picasa Web Albums, picasaweb.google.com Webshots, http://www.webshots.com/ Photobucket, http://photobucket.com/ Imagery, http://elzr.com/imagery Google Images Search, http://www.google.com/ Yahoo! Image Search, http://images.search.yahoo.com/. Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, ''Image Retrieval: Ideas, Influences, and Trends of the New Age,'' ACM Computing Surveys, vol. 40, no. 2, 2008. T. Gevers and A.W.M. Smeulders, PicToSeek: Combining color and shape invariant features for image retrieval, IEEE Trans. on IP, 9 (2000) pp. 102-119. J.Z. Wang, J. Li and G. Wiederhold, SIMPLIcity: Semanticssensitive integrated matching for picture libraries, IEEE Trans. on PAMI, 23 (2001) pp. 947-963. TinEye Reverse Image Search, Idee, 2008, http://www.tineye.com/ GazoPa Similar Image Search, http://www.gazopa.com/ W.H.Leung, T. Chen, “Trademark retrieval using contour-skeleton stroke classification,” IEEE Int. Conf. on Multimedia and Expo., vol. 2, 2002, pp. 517-520. Multicolr Search Lab, Idee, 2008, http://labs.ideeinc.com/multicolr/ Shigenobu Kobayashi, Color Image Scale, The Nippon Color & Design Research Institute ed., translated by Louella Matsunaga, Kodansha International, 1992. Yasushi Kiyoki, Takashi Kitagawa, Takanari Hayama: “A metadatabase system for semantic image search by a mathematical model of meaning,” ACM SIGMOD Record, Volume 23 Issue 4 , December 1994. T. Kitagawa, T. Nakanishi, Y. Kiyoki: “An Implementation Method of Automatic Metadata Extraction Method for Image Data and Its Application to Semantic Associative Search,” Information Processing Society of Japan Transactions on Databases, VOl.43,No.SIG12(TOD16), pp38-51, 2002. Shiori Sasaki, Yoshiko Itabashi, Yasushi Kiyoki, Xing Chen, “An Image-Query Creation Method for Representing Impression by Color-based Combination of Multiple Images,” Frontiers in Artificial Intelligence and Applications; Vol. 190 Proceeding of the 2009 conference on Information Modelling and Knowledge Bases XX, p. 105-112, 2009. A.Vadivel, A.K.Majumdar and Shamik Sural, "Perceptually Smooth Histogram Generation from the HSV Color Space for Content Based Image Retrieval", International Conference on Advances in Pattern Recognition (ICAPR), Calcutta, India, 2003, pp. 248-251. Performance evaluation in content-based image retrieval: overview and proposals, Henning Müller, Wolfgang Müller, David McG. Squire, Stéphane Marchand-Maillet and Thierry Pun, April 2001. Online Travel Guides of Travel Destinations, www.destination360.com Google map API: http://code.google.com/apis/maps/documentation /javascript/