1

Content-Based Histopathology Image Retrieval Using Latent-Semantic-Kernels Jose G. Moreno

Abstract—In this paper we propose an approach to exploit the visual semantic existing in visual features in a medical image retrieval system. We achieve this using Latent Semantic Kernels based on different kernel functions to generate a latent semantic space in which topics relate different visual contents. The proposed method is tested in a collection of histology images. Experimental results show an improvement of the visual latent approach with respect to single visual information. Previously image retrieval systems use latent semantic indexing, but these systems need to make explicit the construction of visual words. This work presents an approach is based on the query-byexample paradigm, so an image is used to search more images. The main problem is to identify relevant images exploiting the visual information. Different kernel functions are used to generate a latent semantic space with topics that relate several visual contents. Index Terms—Latent Semantic Kernels, Histology Images, CBIR System.

I. I NTRODUCTION nformation retrieval is area currently working for research community, such effort emerge new models focused on improving results. Importance of the work are obtained with the information retrieval of text documents has given a strong impetus for generating interest in other important areas, such as content based information retrieval. Content based information retrieval systems to extract features to establish the visual representation of the images, which is then used with different strategies to accomplish the retrieval task. In medicine, the amount of generated images is constantly growing, creating management and storage needs. Is interesting that can be found associate images with similar medical records. Image analysis is focused on different tasks like classification[6], visualization, annotation[9] and retrieval[10], among others, are being exploited for purposes that range from the academy in the medical environment to support activities related to medical diagnosis. In recent methods is a strong trend of motivated approaches developed for text information retrieval, these methods use a visual representation called visual words. This visual words are image subsets that frequently occurs in all image database. Such techniques are interesting for the possibility of applied textual developments in images, assuming that the words in visual images behave similarly to the words in a document. Regions based techniques[1], [3] are based on the notion of the photographic composition of an image, focus on the identification of relevant regions and the

I

Jose G. Moreno is student master in system engineer and computer science of the National University of Colombia. (e-mail: [email protected]). Fabio A. González is Associate Professor of System and Computer Engineering Departament of National University of Colombia and he is subdirector in Bioingenium Research Group. (e-mail: [email protected]).

extraction of features such as color and texture of this region identified. Several papers [1] propose to use characteristic vectors representing used the multiple features and other models, which are exploring techniques used kernel functions between images to perform the same tasks, among these the most widely used methods of support vector machines (SVM). Tendency to use strategies that come from similar areas of study is strongly influenced by strategies such as bag of words, or others such as the application of latent semantic analysis in visual systems. This paper proposes the use of latent semantic kernels for the construction of a retrieval system for content. This strategy allows merging different visual characteristics to construct a kernel function that is the basis for the construction of a latent space where visual information is related to visual topics. These topics allow the reduction of the original with no loss information and are making an analogy with the strategy textual, visual concepts groupings, i.e. a reduction in similar topics. The LSK is based on singular decomposition and the results show that it is possible a significant reduction of the original space. The main contributions of this work are: • Propose the application of the LSK to fuse visual information. • Verify the effectiveness of this approach using different measures. The paper is organized as follows: Section 2 presents the previous work on histopathology image retrieval. Section 3 describes the latent semantic kernel. Section 4 presents experimental results, and finally the concluding remarks are in Section 5. II. P REVIOUS W ORK Methods like bag-of-words [3]for documents are carried to images approach. More intuitive operation of this approach is: 1. Get the whole regions of images to be processed. 2. Extract the characteristics of these regions. 3. Apply a clustering algorithm for visual words. There are different approaches that depend on the techniques used for find number of visual words. In such strategies it is still considered adequate acquire an appropriate vocabulary for the representation of images. In [5], [4], global features are described. Image content is analyzed and the visual features are extracted for the all image. Similarity between the images is calculate using this global information. In the histogram based approach, an histogram is building whit this features and histograms moments. In general collections, works such as [7] using latent semantic indexing (LSI), which requires the construction of visual

2

words (visterms). Visterms are calculated as in the approaches bag-of-words, however, also uses the sampling keypoints and regions. The document-term matrix is built using the frequency of visterms on images and this matrix is applied LSI. LSI is based on the eigendecomposition of the term-document matrix. This approach is used for multimodal retrieval. Approaches such as LSK not need the construction of explicit visual words or document-term matrix. Methods using PCA has been applied in medical imaging, in [8] create a correlation matrix of mean centered data set, then this matrix is computed eigenvalues and eigenvectors. Eigenimages are the projection of images using eigenvectors. These eigenimages have a particular characteristic, it is possible to obtain the database of original images from linear combinations of eigenimages. Similarity of the queries regarding the database of images is found using the relationship between the weights that allow the query image is reconstructed and the weights of the images in the database. III. V ISUAL L ATENT S EMANTIC KERNELS Visual features combination is a non-trivial task. More intuitive approaches tend to concatenate the feature vectors constructed for each feature. LSK [2] is a technique based on SVD is similar to LSI, but we do not have a direct relationship between the visual terms and images. Whit LSK, we can use kernel functions for the image representation, or combinations thereof. With this representation is related semantic information explicit. LSK allows reducing the information to the k most relevant topics, allowing information represent different precision making it big or small the set of k selected topics. With a certain kernel function, it is possible to construct the kernel matrix of the image database. This kernel matrix is a nxn squared matrix defined as: Ki,j = k(di , dj ); j = 1, ..., n; i = 1, ..., n

(1)

 φ(d)Uk =

1 λi− 2

l X

k (vi )j km (dj , d)

j=1

,

(2)

i=1

where φ(d)Uk is the vector whose coordinates contain the presence degree of each topic in the document d. Images database and queries can be mapped to a semantic space with the equation 2. In this space it is possible to determine the similarity between images, and in the specific task of retrieving information may be used a measure of similarity to can be establish a ranking. Then the similarity between a query and an image of the database is given by: S(di , dq ) = S(φ(di )Uk , φ(dq )Uk )

(3)

IV. R ESULTS AND D ISCUSSION Experimental setup is defined as follows: • Kernel matrix construction and eigendescomposition: kernel function used is the histogram intersection (see IV-B). Builds the kernel matrix K using the equation 1, which will be of size 1426x1426. After performing the matrix eigendescomposition, obtaining 1426 eigenvectors and equal number of eigenvalues. • Calculating image database and queries projections in the latent space: for the collection, the values obtained with the kernel function are mapped to the latent space using equation 2. Similarly the mapping is done with the queries. • Calculation of the similarity between images in the latent space: in latent space similarity is calculated using the cosine similarity like in equation 3. • Evaluation of the rankings: evaluation is performed using 76 images. Several performance measures for retrieval systems are calculated. following describes in detail.

where k(di , dj ) is a kernel value between image i and image j.

A. Data Set

Kernel matrix K is decomposed whit SVD and U matrix is used for image database projection. K = U ΣV 0 Σ and U can be calculating whit the follow equation: vi KK 0 = λi KK 0 where all vi are the columns of U and Σ is a diagonal matrix whit corresponding λi in the diagonal. This is an eigendescomposition equation. LSK is applied to the corresponding kernel matrix to obtain a matrix U with the eigenvectors v of the latent semantic space. This new representation space correlates visual and textual features in common semantic topics. The new semantic space is defined by selecting k topics, that corresponds to the k eigenvectors v associated to the k largest eigenvalues λ. Then, all images are projected to the latent semantic space using the following equation[2]:

A collection of medical images consist of histopathology images used for experimentation. Histological images are images of body tissues, this collection is related to the study of skin cancer. The images were obtained previously by the Bioingenium group. This comprised of 1502 images annotated by experts in 18 categories. These entries correspond to concepts in the histopathology image. It pick 76 images as queries and the rest of the collection, 1426 images are used as the database of images. When you perform a query the resulting images are relevant if the concept of the histopathology image query is contained in the image result. Each image can have one or more annotations for each image it is queried with each of its. Finally, we have 151 different queries with the 76 images. B. Kernel Construction It has six kernels, constructed from the following visual features:

3

• • • • • •

Local Binary Partition (LBP). Color (RGB). Tamura (TAM). Sobel (SOB). Invariant (INV). Gray (GRA).

With each of these images are features extraction and build a histogram. For the kernel function was used kernel intersection. Histogram intersection is a similarity function devised to calculate the common area between histograms as follows: k(A, B) =

m X

min{ai , bi }

i=0

Where A and B are the histogram of the image A and the image B respectively. It has been originally used for image similarity search [9].

Figure 2.

Average rank for different k values.

Figure 3.

MAP for different k values

C. Rank1 Rank1 is a measure use for ranks and this is the average position of the first relevant image. The results show that LSK whit all features get a better results that use a single feature. Figure 1 show the results of rank1 values for different k values.

F. Selecting k value

Figure 1.

Rank1 for different k values

Calculation of k visual topics was conducted by calculating the map, rank1, precision at 1 and average rank varying the k values. Table I show the better measures values obtained and the respectively k value. The proposed method shows an improvement on the results obtained with the information in the histograms. V. C ONCLUSIONS AND FUTURE WORK

D. Average Rank This measure evaluate the average position of relevant images. Smaller values indicate better position in the ranking of the images. Results show that LSK improvement over the use of only one feature.

E. Mean Average Precision MAP is a measure used in information retrieval to assess the accuracy taking into account the location of the relevant images, sensitive to the rank of every relevant document. Whit the mixed kernel Map, the measurement of MAP is not improved. However, it is not significantly deteriorated.

The proposed model does not use an explicit representation of information from the images of the finished document is sufficient with a kernel function that takes a measure of similarity between images. But exploiting the spectral decomposition of this information. It is based on spectral decomposition that allows exploiting the visual semantic relationships between different representations of the images. The results show no significant decrease of MAP, however, achieves an increase in measures such as Rank1 and Average Rank, indicating that the LSK should be used in tasks focused on the use of these measures. As future work is the use of this approach on data sets with different cross-modal representations made and the improvement self-correlation of documents associated problem with spectral decomposition.

4

MAP Feature LBP RGB TAM SOB INV GRA LSK

k 150 100 100 200 50 200 100

P(n=1) Value 0.093 0.089 0.084 0.093 0.091 0.088 0.097

k 1150 450 100 700 150 150 450

g Rank

Rank1 Value 0.47 0.45 0.34 0.55 0.40 0.37 0.52

k 600 850 400 50 500 50 150

Value 18.03 11.42 9.56 11.32 12.85 12.78 7.80

k 900 850 700 600 300 750 800

Table I B EST VALUES FOR EACH MEASURE FOR DIFFERENT FEATURE . B EST SCORE ARE IN BOLD .

R EFERENCES [1] C. Carson, S. Belongie, H. Greenspan, and J. Malik. Region-based image querying. In IEEE Workshop on Content-Based Access of Image and Video Libraries, 1997. Proceedings, pages 42–49, 1997. [2] Nello Cristianini, John Shawe-Taylor, and Huma Lodhi. Latent semantic kernels. Journal of Intelligent Information Systems, 18(2):127–152, March 2002. [3] R. W. K. Lam, H. H. S. Ip, K. K. T. Cheung, L. H. Y. Tang, and R. Hanka. A multi-window approach to classify histological features. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, volume 2, 2000. [4] H. Müller, N. Michoux, D. Bandon, and A. Geissbuhler. A review of content-based image retrieval systems in medical applications, clinical benefits and future directions. International journal of medical informatics, 73(1):1–23, 2004. [5] A. Mojsilovic and J. Gomes. Semantic based categorization, browsing and retrieval in medical image databases. In Proc. of IEEE ICIP, 2002. [6] Jia-Yu Pan, Hyung-Jeong Yang, Christos Faloutsos, and Pinar Duygulu. Automatic multimedia cross-modal correlation discovery. In Proc. ACM SIGKDD, pages 653–658, Seattle, WA, USA, 2004. ACM. [7] Trong-Ton Pham, Nicolas Eric Maillot, Joo-Hwee Lim, and JeanPierre Chevallet. Latent semantic fusion model for image retrieval and annotation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 439–444, Lisbon, Portugal, 2007. ACM. [8] U. Sinha and H. Kangarloo. Principal Component Analysis for Contentbased Image Retrieval 1, volume 22. RSNA, 2002. [9] A. Vinokourov, D. R. Hardoon, and J. Shawe-Taylor. Learning the semantics of multimedia content with application to web image retrieval and classification. In Proc. of Fourth International Symposium on Independent Component Analysis and Blind Source Separation., 2003. [10] L. Zheng, A. W. Wetzel, J. Gilbertson, and M. J. Becich. Design and analysis of a content-based pathology image retrieval system. IEEE Transactions on Information Technology in Biomedicine, 7(4):249–255, 2003.

Value 542.16 519.52 543.88 498.29 510.17 512.96 485.29