Web Mining Nguyen Tri Thanh Email:
[email protected] Department of Information Systems Faculty of Information Technology
College of Technology May 11, 2011
1
Web Content Mining Evaluating Clustering
Approaches to Evaluating Clustering
Similarity-Based Criterion Functions
Probabilistic Criterion Functions
MDL-Based Model and Feature Evaluation
Classes to Clusters Evaluation
Precision, Recall and F-measure
Entropy
May 11, 2011
2
Approaches to Evaluating Clustering Criterion functions evaluate clustering models
objectively, i.e. using only the document content
Similarity-based functions: intracluster similarity and sum
of squared errors
Probabilistic functions: log-likelihood and category utility
MDL-based evaluation
May 11, 2011
3
Approaches to Evaluating Clustering (cont’d)
Document labels (if available) may also be used for
evaluation of clustering models
If labeling is correct and reflects accurately the document content we can evaluate the quality of clustering If clustering reflects accurately the document content we can evaluate the quality of labeling Criterion function based on labeled data: classes to
clusters evaluation
May 11, 2011
4
Similarity-Based Criterion Functions (distance)
Basic idea: the cluster center ci (centroid or
mean in case
of numeric data) best represents cluster Di if it minimizes the sum of the lengths of the “error” vectors x -ci for all x ∈ Di k
J e = ∑ ∑ x − ci
2
i =1 x∈Di
1 ci = Di
Alternative formulation based on
∑x
x∈Di
pairwise distance
between cluster members 1 k 1 Je = ∑ 2 i =1 Di May 11, 2011
∑x
2 j
− xl
x j , xl ∈Di
5
Similarity-Based Criterion Functions (cosine similarity)
cosine similarity is used k ci • d j 1 ci = dj J s = ∑ ∑ sim(ci , d j ) sim(ci , d j ) = ∑ Di d j ∈Di ci d j i =1 d ∈D
For document clustering the (centroid)
j
i
Equivalent form based on
1 k 1 JS = ∑ 2 i =1 Di
pairwise similarity
∑ sim(d
j
, dk )
x j , xk ∈Di
Another formulation based on
intracluster similarity (used
to controls merging of clusters in hierarchical Average pair-wise similarity agglomerative clustering) 1 k 1 1 k JS = ∑ sim(d j , d l ) = ∑ Di sim( Di ) ∑ 2 i =1 Di x j , xl ∈Di 2 i =1 May 11, 2011
6
Similarity-Based Criterion Functions (example) Sum of centroid similarity evaluation of four clusterings
May 11, 2011
7
Finding the Number of Clusters
May 11, 2011
8
Probabilistic Criterion Functions Document is a
random event P(d ) = ∑ A P(d | A) P( A)
Probability of document
Probability of sample (assuming that documents are
independent events) P(d1 , d 2 ,..., d n ) = ∏i =1 ∑ A P(d i | A) P( A) n
Log-likelihood (log of probability of sample)
L = ∑i =1 log∑ A P (d i | A) P( A) n
May 11, 2011
9
Probabilistic Criterion Functions (cont’d) Category Utility (based on probability of attributes)
Take into account the probabilities of attributes having particular values within clusters and across clusters (Gluck & Corter)
CU measures both probabilities
two objects in the same category have attribute values in common objects from different categories have different attribute values
CU = ∑∑∑ P (a j = vij | Ck ) P(Ck | a j = vij ) P(a j = vij ) i
May 11, 2011
j
k
10
Category Utility (basic idea)
∑∑∑ P(a i
j
j
= vij | Ck ) P (Ck | a j = vij ) P(a j = vij )
k
P(a j = vij | Ck )
is the probability that an object has value vij for its attribute aj given that it belongs to category Ck. The higher this probability, the more likely two objects in a category share the same attribute values
P(Ck | a j = vij )
is the probability that an object belongs to category Ck, given that it has value vij for its attribute aj. The greater this probability, the less likely objects from different categories have attribute values in common
P(a j = vij )
is a weight coefficient assuring that frequent attribute values have stronger influence on the evaluation
May 11, 2011
11
Category Utility Function CU = ∑∑∑ P(a j = vij | Ck ) P (Ck | a j = vij ) P (a j = vij ) i
j
k
P(Ck | a j = vij ) =
P(a j = vij | Ck ) P(Ck ) P(a j = vij )
Bayesian rule
CU = ∑ P(Ck )∑∑ P(a j = vij | Ck ) 2 k
∑∑ P(a i
j
i
j
j
= vij | Ck ) 2
Expected number of attribute values of a member of Ck correctly guessed using a probability
matching strategy
May 11, 2011
12
Category Utility Function (cont’d) Assume we don’t know the categories (clusters) in our sample, then
∑∑ P(a j = vij ) 2 i
j
Expected number of correctly guessed attribute values without knowing the categories (clusters) in the sample
The final expression of the function is CU (C1 , C2 ,..., Cn ) =
May 11, 2011
1 2 2 P ( C ) ( P ( a = v | C ) − P ( a = v ) ) ∑ k ∑∑ j ij k j ij n k i j
13
Category Utility (example) A = {1,3,4,6,8,10,12,16,17,18,19} B = {2,5,7,9,11,13,14,15,20}
May 11, 2011
14
Category Utility Function for continous attributes The category utility function can be extended to continuous attributes by assuming normal distribution and replacing probabilities with densities
1 n CU (C1 , C2 ,..., Cn ) = ∑ P(Ck )∑ n k =1 i where
[∫ f (v
ik
)dvik − ∫ f (vi )dvi
]
f (.) is the probability density function for normal
distribution
After solving the integrals, we obtain
1 n 1 CU (C1 , C2 ,..., Cn ) = ∑ P(Ck ) n k =1 2 π May 11, 2011
1 1 ∑i σ − σ i ik 15
Data Example
May 11, 2011
16
Finding Regularities in Data Describe the cluster as a pattern of attribute values that
repeats in data Include attribute values that are the same for all members
of the cluster and omit attributes that have different values (generalization by dropping conditions) Hypothesis 1 (using attributes
science and research)
R1 : IF (science = 0) AND (research = 1) THEN Class = A R : IF (science = 1) AND (research = 0) THEN Class = A A = {1,3,4,6,8,10,12,16,17,18,19} H1 = 2 B = {2,5,7,9,11,13,14,15,20} R : IF (science = 1 ) AND (research = 1) THEN Class = A 3 R4 : IF (science = 0) AND (research = 0) THEN Class = B
May 11, 2011
17
Finding Regularities in Data (cont’d) Hypothesis 2 (using attribute
offers)
R : IF (offers = 0) THEN Class = A H2 = 1 R2 : IF (offers = 1) THEN Class = B
Which one is better,
May 11, 2011
A = {1,3,4,6,8,10,12,16,17,18,19} B = {2,5,7,9,11,13,14,15,20}
H1 or H2?
18
Occam’s Razor “Entities
are not to be multiplied beyond necessity”
(William of Occam, 14th century)
Among several alternatives the simplest one is usually the best choice H2 looks simpler (shorter) than H1, so H2 may be better than H1
How do we measure simplicity?
May 11, 2011
19
Measure complexity Dropping more conditions produces simpler (shorter)
hypotheses, the simplest one is the empty rule
The latter however is a single cluster including the whole dataset (overgeneralization)
The most complex (longest) hypothesis has 20 clusters
and is equivalent to the original dataset (over-fitting) How do we find the right balance? The answer to both questions is
Minimum Description
Length (MDL)
May 11, 2011
20
Minimum Description Length (MDL) D (e.g. our document collection) and a set of hypotheses H1,H2,…,Hn, each one describes D
Given a data set
Find the most likely hypothesis
H i = arg max i P( H i | D) Direct estimation of
P(H1|D) is difficult, so we apply Bayes
P( H i ) P( D | H i ) P( H i | D) = P( D) Take –log of both sides
− log 2 P ( H i | D) = − log 2 P( H i ) − log 2 P( D | H i ) + log 2 P( D)
May 11, 2011
21
Minimum Description Length (MDL) (cont’d)
Consider hypotheses and data as messages and apply
Shannon’s information theory
which defines information in a message as a negative logarithm of its probability
Then estimate the number of bits (L) needed to
encode the messages
L( H i | D ) = L( H i ) + L( D | H i ) − L( D )
May 11, 2011
22
Minimum Description Length Principle
L(Hi) and L(D) are the minimum number of bits needed to
encode the hypothesis and data
L(D |Hi) is the number of bits needed to encode D if we know Hi If we think of Hi as a pattern that repeats in D we don’t
have to encode all its occurrences, rather we encode only the pattern itself and the differences that identify each individual instance in D Thus the more regularity in data the shorter description
length L(D |Hi)
May 11, 2011
23
Minimum Description Length Principle (cont’d) We need a good balance between L(Hi) and L(D|Hi)
because if H describes the data exactly then L(D|Hi)=0 but L(Hi) will be large We can exclude L(D) because it does not depend on the
choice of hypotheses
Minimum Description Length (MDL) principle
H i = arg min i L( H i ) + L( D | H i )
May 11, 2011
24
MDL-based Model Evaluation (Basics) Choose a description language (e.g. rules) Use the same encoding scheme for both hypotheses and
data given the hypotheses Assume that hypotheses and data are uniformly distributed
Then probability of the occurrence of an item out of n alternatives is 1/n And the minimum code length of the message informing us that a particular item has occurred is -log21/n=log2n
What is the possibility of the occurrence of
k items among
n items?
May 11, 2011
25
Recall Data Example
May 11, 2011
26
Recall Hypotheses Hypothesis 1 (using attributes
science and research)
R1 : IF (science = 0) AND (research = 1) THEN Class = A R : IF (science = 1) AND (research = 0) THEN Class = A A = {1,3,4,6,8,10,12,16,17,18,19} H1 = 2 B = {2,5,7,9,11,13,14,15,20} R3 : IF (science = 1) AND (research = 1) THEN Class = A R4 : IF (science = 0) AND (research = 0) THEN Class = B
Hypothesis 2 (using attribute
offers)
R : IF (offers = 0) THEN Class = A H2 = 1 R2 : IF (offers = 1) THEN Class = B
A = {1,3,4,6,8,10,12,16,17,18,19} B = {2,5,7,9,11,13,14,15,20}
May 11, 2011
27
MDL-based Model Evaluation (Example) Description language: 12 attribute-value pairs (6 attributes each
with two possible values) Rule
R1 covers documents 8, 16, 18 and 19
There are 9 different attribute-value pairs that occur in these
documents {history=0}, {science=0}, {research=1}, {offers=0}, {offers=1}, {students=0}, {students=1}, {hall=0}, {hall=1} Specifying rule
R1 is equivalent to choosing 9 out of 12 attribute-
value pairs, which can be done in C912
May 11, 2011
No
History
Science
Research
Offers
Students
Hall
8
0
0
1
0
0
0
16
0
0
1
0
1
1
18
0
0
1
1
1
1
19
0
0
1
1
1
1
28
MDL-based Model Evaluation (Example) (cont’d) Since we use only 1 among C912 possibilities, log2C912 bits
are needed to encode the left-hand side of R1 In addition we need one bit (a choice of one out of two
cluster labels) to encode the choice of the class (log22) Thus the code length of
R1 is
L(R1)=log2C912 +1=log2 220+1 =8.78136
May 11, 2011
29
MDL-based Model Evaluation (L(H1)) Similarly we compute the code lengths of
R2, R3, and R4
and obtain L(R2)=log2C712 +1=log2 792+1 =10.6294 L(R3)=log2C712 +1=log2 792+1 =10.6294 L(R4)=log2C1012 +1=log2 66+1 =7.04439 Using the additivity of information to obtain the code
length of L(H) we add the code lengths of its constituent rules
L(H1)=37.0845 May 11, 2011
30
MDL-based Model Evaluation (L(D|R1)) Consider the message exchange setting, where the
hypothesis R1 has already been communicated This means that the recipient of that message already
knows the subset of 9 attribute-value pairs selected by rule
R1 Then to communicate each document of those covered by
R1 we need to choose 6 (the pairs occurring in each document) out of the 9 pairs This choice will take log2 C69 bits to encode
May 11, 2011
31
MDL-based Model Evaluation (L(D|R1)) (cont’d) As
R1 covers 4 documents (8, 16, 18, 19) the code length
needed for all of them will be
L({8,16,18,19}|R1)=4 x log2 C69 =4 x log2 84=25.5693
May 11, 2011
32
MDL-based Model Evaluation (MDL(H1)) Similarly we compute the code length of the subsets of
documents covered by the other rules
L({4,6,10}|R2)=3 x log2 C67 =3 x log2 7=8.4220 L({1,3,12,17}|R3)=4 x log2 C67 =4 x log2 7=11.2294 L({2,5,7,9,11,13,14,15,20}|R4)=9 x log2 C610 =9 x log2 210=69.4282 The code length needed to communicate all documents
given hypothesis H1 will be the sum of all these code lengths, i.e. L(D|H1)=114.649 Now adding this to the code length of the hypothesis we
obtain MDL(H1)=L(H1)+L(D|H1)=37.0845+114.649=151.733 May 11, 2011
33
MDL-based Model Evaluation (H1 or H2?) Similarly we compute
MDL(H2)
MDL(H2)=L(H2)+L(D |H2)=9.16992+177.035=186.205
MDL(H1) < MDL(H2) ⇒ H1 is better than H2
Also (very intuitively), L(H1)
> L(H2) and
L(D |H1) < L(D |H2) How about the most general and the most specific
hypotheses?
May 11, 2011
34
Information (Data) Compression The most general hypothesis (the empty rule {}) does not
restrict the choice of attribute-value pairs, so it selects 12 out of 12 pairs L({})=log2C1212+1=1 L(D|{})=L(D)=20 x log2C612=20 x (log2924+1)=197.035
The most specific hypothesis has 20 rules – one for each
document L(S)=20 x (log2C612+1)=20 x (log2924+1)=217.035 Both {} and S represent extreme cases, undesirable in learning
overgeneralization and overspecialization Good hypotheses should provide smaller MDL than {} and S, or
stated otherwise (principle of Information Compression) L(H)+L(D|H)< L(D) or Hi=arg maxi (L(D)-L(Hi)-L(D|Hi)) May 11, 2011
35
MDL-based Feature Evaluation An attribute can split the set of documents into subsets,
each one including the documents that share the same value of that attribute Consider this split as a clustering and evaluate its MDL
score Rank attributes by their MDL score and select attributes
that provide the lowest score
MDL(H1) was actually the MDL score of attribute offers from our 6-attribute document sample
May 11, 2011
36
MDL-based Feature Evaluation (Example)
May 11, 2011
37
Classes to Clusters Evaluation Assume that the classification of the documents in a
sample is known, i.e. each document has a class label Cluster the sample without using the class labels Assign to each cluster the class label of the majority of
documents in it Compute the
error as the proportion of documents with
different class and cluster label Or compute the
accuracy as the proportion of documents
with the same class and cluster label
May 11, 2011
38
Classes to Clusters Evaluation (Example)
May 11, 2011
39
Confusion matrix (contingency table)
TP (True Positive), FN (False Negative), FP (False Positive), TN (True Negative)
FP + FN Error = TP + FP + TN + FN
May 11, 2011
TP + TN Accuracy = TP + FP + TN + FN
40
Precision and Recall TP Precision = TP + FP TP Recall = TP + FN
May 11, 2011
41
F-Measure Generalized confusion matrix form classes and k clusters
Combining precision and recall P(i, j ) =
nij
∑i =1 nij m
R(i, j ) =
nij
∑
k
n
j =1 ij
2 P(i, j ) R(i, j ) P(i, j ) + R(i, j ) Evaluating the whole clustering m ni F = ∑i =1 max F (i, j ) n j =1,...,k F (i, j ) =
ni = ∑ j =1 nij k
n = ∑i =1 ∑ j =1 nij m
May 11, 2011
k
Total number of documents
42
F-Measure (Example)
May 11, 2011
43
Entropy Consider the class label as a random event and evaluate its
probability distribution in each cluster
i in cluster j is estimated by the proportion of occurrences of class label i in cluster j nij pij = m ∑ nij
The probability of class
i =1
The entropy is as a measure of “impurity” and accounts for
the average information in an arbitrary message about the class label m
H j = −∑ pij log pij i =1
May 11, 2011
44
Entropy (cont’d) To evaluate the whole clustering we sum up the entropies
of individual clusters weighted with the proportion of documents in each k
H =∑ i =1
May 11, 2011
nj n
Hj
45
Entropy (Examples) A “pure” cluster where all documents have a single class
label has entropy of 0 The highest entropy is achieved when all class labels have
the same probability For example, for a two class problem the 50-50 situation
has the highest entropy of (-0.5 log 0.5- 0.5 log 0.5)=1
May 11, 2011
46
Entropy (Examples) (cont’d) Compare the entropies of the previously discussed
clusterings for attributes offers and students 11 8 8 3 3 9 3 3 6 6 H (offers ) = − log − log + − log − log = 0.878176 20 11 11 11 11 20 9 9 9 9 H ( students) =
May 11, 2011
15 10 10 5 5 5 1 1 4 4 − log − log + − log − log = 0.869204 20 15 15 15 15 20 5 5 5 5
47
Summary Evaluating Clustering
Similarity-Based Criterion Functions
Probabilistic Criterion Functions
MDL-Based Model and Feature Evaluation
Classes to Clusters Evaluation
Entropy
May 11, 2011
48
