An Extension of PLSA for Document Clustering Young-Min Kim, Jean-François François Pessiot, Massih R. Amini, Patrick Gallinari Computer Science Laboratory of Paris 6, University Pierre and Marie Curie 104, Avenue du President Kennedy, 75016 Paris, France

Objective

Experiments

• Find an accurate generative process which better explains the generation of documents, • Enhance document clustering in a more compact vector space, • To this end we propose an extension of the existing PLSA model by incorporating an additional topic latent variable.

Type 1 MM model and PLSA model in bag-of-word space Type 2 Above models in reduced concept space (needed another learning to find word concept space) Type 3 Extended PLSA model Type 4 Other clustering methods P- : concept space Type1 Type 2 Type3 Type4 found by PLSA MM, P-MM, C-MM, Ext-PLSA K-means C- : concept space PLSA P-PLSA, C-PLSA NMF found by CEM Data Collections

Methods

# classes # documents Reuters 7 4335 20NewsGroups 5 16010 WebKB 4 4196

• Comparison of three generative models

Results

#words 6990 38630 11170

10 subsets Average performance

Evaluation Criteria : averaged Precision/Recall and NMI

• Clustering performance of PLSA type methods Ext-PLSA Ext-PLSA

P-PLSA

C-PLSA

PLSA

NMF

P-PLSA

PLSA K-means

NMF

(a) Multinomial Mixture model (MM) (b) PLSA model (c) Extended PLSA model (α, β : latent variables)

C-PLSA

K-means

Word Cluster Examples

• Generation process of corpus in the Extended PLSA p(β) b1

p(d )

.

.

.

bL

. . .

aK

p(α|d)

C-PLSA

P-PLSA

a1

document collection

PLSA

Ext-PLSA

NMF

wrong christian words christians truth meaning paul john bible word

K-means

• Performance comparison of all methods w1

w2

w3 p(w | α, β)

suicide suicides deaths stat selfdefense risks statstic guns accidents homicides

From CEM algorithm. similar quality clusters found by p(α, β|w)

Average Precision

w4

- Pick a document d with probability p(d ), - Choose a document concept a with probability p(α|d), - Choose a word concept β with probability p(β), - Generate a word w with probability p(w| α, β). • Maximize log-likelihood function

L = åå n(d , w) log p(d ) p(a d ) p( b ) p ( w a , b ) d

w

• Expectation Maximization (EM) Algorithm

cluster(d) = argmax p(α|d)

Conclusion • Extended PLSA : Effective model for document clustering • Concept space quality affects document clustering performance. • Simultaneous learning of word and document concepts in extended PLSA improve also word cluster quality.