Conditional ML Estimation Using Rational Function Growth Transform Ciprian Chelba and Alex Acero Microsoft Research, Redmond, U.S.A.

Abstract

Smoothing and Initialization

Conditional Maximum Likelihood (CML) estimation of probability models by means of the rational function growth transform (RFGT) [GKNN91]. Model parameter values — probabilities — properly normalized at each iteration. Discriminatively train a Na¨ıve Bayes (NB) classifier; same procedure is at the basis of discriminative training of HMMs in speech recognition [Nor91].

Model parameters θ are initialized with maximum likelihood estimates smoothed using MAP:

vergence speed significantly over a straightforward implementation.

Conditional Likelihood: Training Data 0

#(y) λy = , #(y) + α

λ y = 1 − λy

(3) −500

LogL

#(k, y) 1 θky = λy · + λy · , #(y) 2

P (y|x; θ) = Z(x; θ)

−1

· θy

F


total (min=0.1) ML count (min=0.1) ML count (min=0.075)

−1000

Model in Eq. (1) is re-parameterized: 1 fk (x) 1 fk (x) (λy · θky + λy · ) (λy · θky + λy · ) 2 2

−1500

(4)

k=1

✔ conditional likelihood H(T ; θ) =

0

50

100 150 200 Conditional Likelihood: Held−Out Data

250

300

0

50

100 150 200 Class Error Rate: Held−Out Data

250

300

0

50

100

250

300

−400

T

j=1 P (yj |xj ) is still a ratio of polynomials

−600

LogL

✔ Simple modification of the reestimation eqns. increases con-

Experiments: Convergence Speed

−800

✔ Reduces class error rate over ML classifier by 40% relative.

−1000

Model Parameter Updates

10

The re-estimation equations take the form:

−1θ = N θ ky y ky

• Conditional probability P (y|x), y ∈ Y, x ∈ X

 ∂ log H(T ; θ) + Cθ ∂θky

CER

Conditional Na¨ıve Bayes Models

where Cθ > 0 satisfies:

Na¨ıve Bayes model for the feature vector and the predicted variable (f (x), y): P (y|x; θ) = Z(x; θ)

−1

· θy

fk (x) (x) f k θky θky

6

(5)

4

• Feature set F = {fk (x) : X → {0, 1}, fk (x) = 1 − f (x), k = 1 . . . F }

F


∂ log H(T ; θ) + Cθ > , ∀k and y ∂θky

(1)

k=1



where: fk (x) = 1 − f (x), θy ≥ 0, ∀y ∈ Y, y∈Y θy = 1; θky ≥ 0, θky ≥ 0, θky + θky =  −1 1, ∀k = 1 . . . F, y ∈ Y and Z(x; θ) = y P (f (x), y) is a normalization term.

Equivalence with Exponential Models

Experiments: Classification Performance Training Objective Function Smoothing Class Error Rate (%) ML smoothed 11.3 CML not smoothed 8.5 CML smoothed 6.7 MaxEnt smoothed 4.9

where:

F θky Setting: fk (x, y) = fk (x) · δ(y); λky = log( ); λ0y = log(θy · k=1 θky ) and f0(x, y) = f0(y) θky

☞ log-linear model — maximum entropy probability estimation [BPP96]: P (y|x; λ) = Z(x; λ)−1 · exp 

F 

• δ() Kronecker delta operator, δ(y, yi) = 1 f or y = yi, 0 otherwise

 λky fk (x, y)

(2)

Algorithmic Refinements

k=0

where λky are free real-valued parameters and Z(x; λ) the normalization term.

Rational Function Growth Transform for CML Estimation of Na¨ıve Bayes Models Estimate θ = {θy , θky , θky : y ∈ Y, k = 1 . . . F } ∈ Θ such that the conditional likelihood of training data T = {(x1, y1) . . . (xT , yT )} is maximized: θ∗ = arg max H(T ; θ), θ

H(T ; θ) =

T


P (yj |xj )

normalize

γθ βθ = T

☞ Dependency on Class Count:

make Eqn. (6) dimensionally correct ζθ (y) βθ = #(y)

– Considerable convergence speedup #(y) – Setting ζθ (y) = T · γθ results in identical updates under either γ or ζ.

✔ CML trained classifier reduces CER by 40% relative over ML counterpart ✔ although equivalent with MaxEnt, does not result in same performance – different smoothing (Gaussian prior for MaxEnt [CR00]) – objective function (conditional log-likelihood) is not convex in parameter values for Na¨ıve Bayes (NB) model whereas it is convex for MaxEnt model – one extra free parameter in NB versus MaxEnt

Acknowledgements Thanks to Milind Mahajan and Asela Gunawardana for useful comments and discussions.

References [BPP96]

A. L. Berger, S. A. Della Pietra, and V. J. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–72, March 1996.

[CA04]

Ciprian Chelba and Alex Acero. Conditional maximum likelihood estimation of Na¨ıve Bayes probability models. Technical Report to appear, Microsoft Research, Redmond, WA, 2004.

[CR00]

Stanley F. Chen and Ronald Rosenfeld. A survey of smoothing techniques for maximum entropy models. IEEE Transactions on Speech and Audio Processing, 8(1):37–50, 2000.

j=1

H(T ; θ) is a ratio of two polynomials with real coefficients, each defined over a set Θ of probability distributions:  θy = 1; θky ≥ 0, θky ≥ 0 and θky + θky = 1, ∀y ∈ Y, ∀k} Θ = {θ : θy ≥ 0, ∀y ∈ Y and y

✔ iteratively estimate model parameters using growth transform for rational functions on the domain Θ [GKNN91].

☞ Dependency on Training Data Size:

Experiments: Setup

✔ ATIS II + III type A utterances (can be interpreted w/o context) • type A utterances: TRAIN/DEV/TEST: 5,822/410/914 (74,442/5,326/10,673wds); • class label assigned from SQL query (14 classes); • word-vocabulary: 780, OOV 0.24%

200

10-fold speed-up as measured by increase in log-likelihood and decrease in class error rate (CER) on held-out data

with  > 0 suitably chosen, see [GKNN91] and [CA04] for details. Calculating the partial derivatives in Eq. (5) we obtain:   T    λ y −1 · θ · 1 + β θ = N fk (xi)[δ(y, yi) − p(y|xi; θ)] (6) ky ky  θ ky 1  λy · θky + λy · −1 normalization constant that ensures θ + θ = 1 • Nky ky ky • βθ = 1/Cθ

150 Iteration Number

✔ about

2 i=1



8

[GKNN91] P. S. Gopalakrishnan, Dimitri Kanevski, Arthur Nadas, and David Nahamoo. An inequality for rational functions with applications to some statistical estimation problems. IEEE Transactions on Information Theory, 37(1):107–113, January 1991. [Nor91]

Y. Normandin. Hidden Markov Models, Maximum Mutual Information Estimation and the Speech Recognition Problem. PhD thesis, McGill University, Montreal, 1991.