Gamma Family of Distributions G.H. Lathrom Department of Mathematics Missouri Southern State University
October 19, 2015
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
1 / 32
Introducing. . . the Gamma Function!
Definition For 0 < x < ∞, define ˆ Γ(x) =
∞
t x−1 e−t dt.
0
To see that the integral converges when x > 0 we sketch the argument. First suppose that x ∈ Z+ . Then in any compact interval [0, b] we have for x = 1 that ˆ
b
e−t dt = −e−b + 1 = 1 − e−b → 0
as b → ∞.
0
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
2 / 32
Convergence of the Definition Let us assume this holds up to a positive integer n, then for n + 1 we have from integration by parts that ˆ
b
n −t
t e 0
b n −t
ˆ
dt = −t e + n 0
b
t n−1 e−t dt < ∞
as b → ∞
0
by L’Hosptial’s Rule and by our induction hypothesis. Therefore, by mathematical induction we know that the Gamma Function converges for all positive integers. Now suppose that x > 1 is arbitrary. We now use the fact that t x−1 ≤ t bx−1c+1 for t ≥ 1 where bxc denotes the greatest integer less that x. Because of this and the fact that if 0 ≤ f (t) ≤ g(t) on an interval [a, b] then ˆ
ˆ
b
b
f (t) dt ≤ a G.H. Lathrom (MSSU)
g(t) dt a
Gamma Family
October 19, 2015
3 / 32
More on Convergence we may write ˆ
ˆ
b
t
x−1 −t
e
t
=
0
ˆ
1
x−1 −t
e
b
dt +
0
ˆ
t x−1 e−t dt
1
≤
ˆ
1
t
x−1 −t
e
dt +
0
b
t bx−1c+1 e−t dt.
1
The last integral on the right converges as b → ∞ by previous work. Therefore, the gamma function will converge for all x ≥ 1. If 0 < x < 1 then we need to check both 0 and ∞ for convergence. Again using integration by parts with u = e−t
and
dv = t x−1 dt
so that du = −e−t dt G.H. Lathrom (MSSU)
and v =
Gamma Family
1 x t x October 19, 2015
4 / 32
Finally Establishing Convergence
we find for 0 < a < b that ˆ ˆ b 1 x −t b 1 b x −t x−1 −t t e dt = − t e + t e dt x x a a a We can see that as a → 0 and b → ∞ the integral on the right side is the gamma function for x > 1 which we know converges. Likewise, by L’Hospital’s Rule the definite part will also converge. Thus, the Gamma function is well defined on the interval x > 0.
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
5 / 32
Complex Analysis Version of the Gamma Function Definition The gamma function, Γ(z), is a meromorphic function on C with simple poles at z = 0, −1, −2, . . . defined by Γ(z) =
∞ eγz Y z −1 z/n 1+ e z n n=1
where γ is a constant chosen so that Γ(1) = 1.
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
6 / 32
Properties of the Gamma Function Theorem The Gamma function satisfies the following properties: On (0, ∞) the functional equation holds: Γ(x + 1) = xΓ(x). Γ(n + 1) = n! for n = 1, 2, 3, . . . log(Γ) is convex on (0, ∞).
Theorem If f is a positive function on (0, ∞) such that f (x + 1) = xf (x),
f (1) = 1,
and
log(f (x)) is convex
then f (x) = Γ(x).
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
7 / 32
The Beta Function Theorem If x > 0 and y > 0, then ˆ B(x, y ) =
1
t x−1 (1 − t)y −1 dt =
0
Γ(x)Γ(y ) Γ(x + y )
Problem Use the Beta function and the substitution t = sin2 (θ) to see that ˆ 2
π/2
0
Take the special case x = y =
G.H. Lathrom (MSSU)
Γ(x)Γ(y ) . Γ(x + y ) to determine Γ 12 .
(sin θ)2x−1 (cos θ)2y −1 dθ = 1 2
Gamma Family
October 19, 2015
8 / 32
Another Exercise
Exercise Use the substitution t = s2 to see that ˆ ∞ 2 Γ(x) = 2 s2x−1 e−s ds. 0
Use this to evaluate the special case Γ
G.H. Lathrom (MSSU)
Gamma Family
1 2
.
October 19, 2015
9 / 32
The Gamma Distribution
Definition A continuous random variable X is said to have a gamma-type probability distribution or X ∼ Γ(κ, θ) where (κ, θ) ∈ (0, ∞) × (0, ∞) if and only if the density function of X is given by f : [0, ∞) → R
s.t. f (x) =
1 θκ Γ(κ)
x κ−1 e−x/θ .
Theorem If X ∼ Γ(κ, θ), then m(t) =
G.H. Lathrom (MSSU)
1 1 − θt
Gamma Family
κ .
October 19, 2015
10 / 32
Expected Value and Variance Theorem If X ∼ Γ(κ, θ) then E[X ] = θκ and
V(X ) = θ2 κ.
Proof. Taking the first derivative of the mgf gives us m0 (t) = −κ(1 − θt)−κ−1 (−θ)
=⇒
m0 (0) = κθ.
=⇒
m00 (0) = θ2 κ(κ + 1).
For the second moment, we have m00 (t) = κθ(−κ − 1)(1 − θt)−κ−2 (−θ) For the variance we then have V(X ) = θ2 κ(κ + 1) − κ2 θ2 = θ2 (κ2 + κ − κ2 ) = θ2 κ.
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
11 / 32
The Exponential Family of Distributions The Gamma family of distributions is important because it gives special subfamilies of distributions. One such subfamily is the family of exponential distributions.
Definition A continuous random variables X is from the exponential family of distributions if X ∼ Γ(θ, 1) and so has pdf given by f : [0, ∞) → R s.t. f (x) =
1 −x/θ e . θ
We denote this by X ∼ EXP(θ). The exponential distribution is an important probability model for life-times, and is characterized by the following no-memory property. One direction is illustrated in the following theorem. G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
12 / 32
Exponential Properties Theorem If X ∼ EXP(X ), then P(X > a + t | X > a) = P[X > t]. for all a > 0 and t > 0. Proof. We have P(X > a + t | X > a) = =
P(X > a + t and X > a) P(X > a) P(X > a + t) e−(t+a)/θ = P(X > a) e−a/θ
= e−t/θ = P(X > t).
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
13 / 32
Relationship to the Gaussian Family The Gamma family of distributions is also related to the Gaussian family in a natural way. This family of distributions is the first we shall encounter in our introduction to inferential statistics and is called a χ2 -distribution.
Theorem If Z has a standard normal distribution, then
Z2
1 . ∼ Γ 2, 2
Proof. Using moment generating functions we have ˆ ∞ 1 2 2 2 mZ 2 (t) = E[etZ ] = √ etz e−z /2 dz 2π −∞ ˆ ∞ 1 2 = √ e−(1−2t)z /2 dz 2π −∞
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
14 / 32
A New Distribution
after a u-substitution this gives us 1 mZ 2 (t) = √ 1 − 2t
1 √ 2π
ˆ
∞
e
−u 2 /2
−∞
du
=
1 1 − 2t
1/2
We can see that this is the same as themoment generating function 1 with a random variable distributed as Γ 2, . 2 Because moment generatingfunctions uniquely determine 1 distributions we have Z 2 ∼ Γ 2, . 2
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
15 / 32
χ2 -family Definition The chi-square family of distributions with ν degrees ν of freedom, 2 denoted χ (ν), if it has a disributions of the form Γ 2, . 2
Theorem If Xi ∼ χ2 (νi ) for i = 1, 2, . . . , n are independent chi-square variables, then ! n n X X V = X i ∼ χ2 νi i=1
i=1
Proof. Since each variable is χ2 (νi ) the moment generating function is given by mXi (t) = (1 − 2t)νi /2 G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
16 / 32
Sum of χ2 Random Variables so that P
mV (t) = (1 − 2t)ν1 /2 (1 − 2t)ν2 /2 · · · (1 − 2t)νn /2 = (1 − 2t)−(
νi )/2
which gives P the moment generating function for a distribution of the form χ2 ( νi ).
Theorem If X1 , X2 , . . . , Xn are iid random variables with Xi ∼ N(µ, σ 2 ), then n X Xi − µ 2 ∼ χ2 (n) and σ i=1
G.H. Lathrom (MSSU)
Gamma Family
X −µ σ 2 /n
!2 ∼ χ2 (1).
October 19, 2015
17 / 32
Gaussian and Chi-Square Connection Proof. Since 2
Xi ∼ N(µ, σ )
=⇒
X −µ σ
and so
n X Xi − µ 2 ∼ χ2 (n). σ
∼ N(0, 1).
Thus,
Xi − µ σ
2
2
∼ χ (1)
i=1
For the second conclusion we have for n 1X ¯ ¯ ∼ N(µ, σ 2 /n). X = Xi so that X n i=1
Hence, X −µ σ 2 /n G.H. Lathrom (MSSU)
! ∼ N(0, 1)
=⇒
Gamma Family
n(X − µ) ∼ χ2 (1). σ2 October 19, 2015
18 / 32
More on χ2 -distribution Theorem If X1 , X2 , . . . , Xn denotes an iid sample from N(µ, σ 2 ) then X is independent of Xi − X for i = 1, 2, . . . , n. Proof. Since the Xi are independent, we know that the joint pdf is a product of the marginal distributions, so that n 1 2 2 2 2 2 2 √ e−(x1 −µ) /2σ e−(x2 −µ) /2σ · · · e−(xn −µ) /2σ fX (x1 , . . . , xn ) = 2πσ ! n n 1 X 1 2 √ (xk − µ) exp − 2 = 2σ 2πσ k =1
Let us then define a transformation as follows, n 1X Y1 = X = Xi and Yj = Xj − X n
(1)
i=1
for j = 2, 3, . . . , n. G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
19 / 32
Independence Theorem In order to find the new pdf for the Yi ’s we need the inverse transformation for which we immediately see that for j = 2, 3, . . . , n that Yj = Xj − X
Yj = Xj − Y1
=⇒
For X1 , we note that nY1 = X1 +
=⇒
n X
Xj = Y1 + Yj .
Xj .
i=2
Summing the Xj we find that n X
n n X X Xi = (Y1 + Yi ) = (n − 1)Y1 + Yi .
i=2
i=2
i=2
This means for X1 we have X1 = nY1 −
(n − 1)Y1 +
n X i=2
G.H. Lathrom (MSSU)
Gamma Family
! Yi
= Y1 −
n X
Yi .
i=2 October 19, 2015
20 / 32
Independence Theorem Ctd. Our inverse transformation is then given by X1 = Y1 − Y2 − Y3 − · · · − Yn X2 = Y1 + Y2 X3 = Y1 + Y3 .. . Xn = Y1 + Yn Therefore, (x1 − µ)2 = (y1 − µ)2 − 2(y1 − µ)
n X
yk +
k =2
n X
!2 yk
.
k =2
(x2 − µ)2 = (y1 − µ)2 + 2(y1 − µ)y2 + y22 .. . (xn − µ)2 = (y1 − µ)2 + 2(y1 − µ)yn + yn2 G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
21 / 32
The Substitution and so n X
(xk − µ)2 = (y1 − µ)2 − 2(y1 − µ)
n X
yk +
k =2
k =1
n X
!2 yk
.
k =2
+ (n − 1)(y1 − µ)2 + 2(y1 − µ)
n X
yk +
k =2
n X
yk2
k =2
so that n X
2
2
(xk − µ) = n(y1 − µ) +
k =1
G.H. Lathrom (MSSU)
n X k =2
Gamma Family
!2 yk
+
n X
yk2 .
k =2
October 19, 2015
22 / 32
Playing with Exponentials
giving n 1 X (xk − µ)2 exp − 2 2σ
!
k =1
= exp −
n 1 (y1 − µ)2 · exp − 2 2 2σ 2σ
n X k =2
!2 yk
+
n X
yk2
k =2
which gives the substitution as a product of a function only of y1 and another function only of y2 , y3 , . . . , yn .
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
23 / 32
The Jacobian The Jacobian of this transformation is then given by 1 −1 −1 −1 1 1 0 0 ∂(X1 , X2 , . . . , Xn ) 1 0 1 0 J= = ∂(Y1 , Y2 , . . . , Yn ) .. .. .. .. . . . . 1
G.H. Lathrom (MSSU)
0
0
0
n=1
=⇒
det(J) = 1
n=2
=⇒
det(J) = 2
n=3
=⇒ .. .
det(J) = 3
arbitrary n
=⇒
det(J) = n.
Gamma Family
··· ··· ··· ··· ···
−1 0 0 .. . 1
October 19, 2015
24 / 32
Proof of the Jacobian We will need the determinant of J and so we recall the definition of the determinant. For an n × n matrix A = (Aij ) we have X det(A) = (−1)σ A1σ(1) A2σ(2) · · · Anσ(n) σ∈Sym(n)
where Sym(n) is the group of all possible permutations on n elements and (−1)σ denotes the sign of a permutation, that is 1 if σ can be written as a product of an even number of transpositions and −1 if σ can be written as an odd number of transpositions. We can see that if in our permutation σ(1) = 1 then for all other possible values, 2, 3, . . . , n, that the only way to choose a non-zero matrix entry is for σ(j) = j. Hence σ must the be identity permutation. If σ(1) = k 6= 1, then for the k -th row the only non-zero value to have σ(k ) = 1. Likewise, for any j 6= 1, k we must have σ(j) = j in order to have a non-zero value. G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
25 / 32
Proof of the Jacobian Ctd. Thus the only permutations giving a non-zero value in the sum must be transpositions σ = (1, k ). The terms in our summation are therefore, ( 1 if σ = id σ (−1) J1σ(1) · · · Jnσ(n) = (−1)(1) · · · (1)(−1)(1) · · · (1) if σ = (1, k ) Since we have a total of n − 1 transpositions of the form (1, k ) and one identity, we are adding a total of n 1’s and so det(J) = n. Our inverse transformation will then have the form n n Pn Y 1 2 2 2 2 e−(y1 − i=2 yi −µ) /2σ e−(y1 +yk −µ) /2σ fY (y1 , . . . , yn ) = n √ 2πσ k =2 G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
26 / 32
Okay We’re Getting There! Therefore, our pdf will be fY (y1 , . . . , yn ) = √
n 2πσ
e
−
n (y −µ)2 2σ 2 1
" √
n−1
1 2πσ
e
−
# P 2 1 Pn yj2 +( nj=2 yj ) 2 j=2 2σ .
Which shows we have written our pdf as a product of the marginals for y1 and the joint distribution for the y2 , y3 , . . . , yn . Thus, X is independent of the variables Xj − X for j = 2, . . . , n. Lastly, because we have n X 0= (Xi − X )
=⇒
n X X1 − X = − (Xi − X ).
i=1
G.H. Lathrom (MSSU)
i=2
Gamma Family
October 19, 2015
27 / 32
Finally, the end. . . no kidding! It means that X1 − X is a function of the Xj − X for j = 2, . . . , n and so X is independent of X1 − X as well.
Corollary If X1 , . . . , Xn are iid N(µ, σ 2 ) then the sample mean and the sample variance are independent. Proof. Since X is independent of the Xi − X and the sample variance, n
S2 =
1 X (Xi − X )2 n−1 i=1
is a function of only the Xi − X , and so X and S 2 are independent.
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
28 / 32
Algebra Lemma Lemma For random variables X1 , X2 , . . . , Xn , we have n X Xi − µ 2 (n − 1)S 2 + = σ σ2 i=1
Proof. We have n X Xi − µ 2 i=1
σ
=
X −µ √ σ/ n
!2 .
n 1 X (Xi − µ)2 σ2 i=1 n X
=
1 σ2
=
n i 1 Xh ¯ )2 + 2(Xi − X ¯ )(X ¯ − µ) + X ¯ −µ 2 (X − X i σ2
¯ +X ¯ −µ (Xi − X
2
i=1
i=1
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
29 / 32
Proof of our Algebraic Lemma
Since
Pn
=
=
=
i=1 (Xi
¯ ) = 0 we have −X
n n n ¯ − µ) X 2 2(X 1 X ¯ 1 X 2 ¯ ¯ (X − X ) + (X − X ) + X −µ i i 2 2 2 σ σ σ
1 σ2
i=1 n X
i=1
i=1
2 n ¯ X − µ σ2 i=1 ¯ 2 (n − 1)S 2 X −µ √ + σ2 σ/ n ¯ )2 + (Xi − X
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
30 / 32
A Very Useful Theorem Theorem If X1 , X2 , . . . , Xn denotes an iid random sample form the normal distribution from N(µ, σ 2 ). Then (n − 1)S 2 /σ 2 ∼ χ2 (n − 1). Proof. We have Xi ∼ N(µ, σ 2 )
=⇒
Xi − µ ∼ N(0, 1) σ
This means that Hence that
Xi − µ σ
2
and
¯ −µ X √ ∼ N(0, 1). σ/ n
∼ χ2 (1).
n X Xi − µ 2 ∼ χ2 (n). σ i=1
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
31 / 32
Proof of the Useful Theorem By the previous Lemma we have n X Xi − µ 2 (n − 1)S 2 = + σ σ2 i=1
X −µ √ σ/ n
!2 .
Because the two expressions on the right-hand side are independent, the moment generating function of the left-hand side is the product of the moment generating functions of the terms on the right-hand side. So we may have (1 − 2t)−n/2 = m(t)(1 − 2t)−1/2
=⇒
m(t) = (1 − 2t)−(n−1)/2 .
This means that the mgf for (n − 1)S 2 /σ 2 is the mgf for a χ2 -distribution with n − 1 degrees of freedom. Because mgf’s uniquely determine distributions, the result follows.
G.H. Lathrom (MSSU)
Gamma Family
October 19, 2015
32 / 32
