Introduction Properties Probability Estimates Numerical Experiments

Probability Estimate for Generalized Extremal Singular Values of Random Matrices

Louis Yang Liu

University of Georgia July 28, 2010

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Why do we study extremal sigular values?

(Candes-Tao'2005) For an integer s ≤ n, the restricted isometry constant δs (A) is the smallest number δ which satises

(1 − δ) kxk2 ≤ kAxk2 ≤ (1 + δ) kxk2 for any s-sparse vector x ∈ Rn .

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Why do we study extremal sigular values?

(Candes-Tao'2005) For an integer s ≤ n, the restricted isometry constant δs (A) is the smallest number δ which satises

(1 − δ) kxk2 ≤ kAxk2 ≤ (1 + δ) kxk2 for any s-sparse vector x ∈ Rn . Equivalently, the inequality √ √ 1 − δ ≤ smin (AS ) ≤ smax (AS ) ≤ 1 + δ holds for any m × s submatrix AS . Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Largest singular value in

`2

Geman'1980 and Yin, Bai, and Krishnaiah'1988 showed that the largest singular value of random matrices of size m × n with independent entries of mean 0 and variance 1 converges to √ √ m + n almost surely.

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Smallest singular value in

`2

Rudelson and Vershynin showed Theorem (RudelsonVershynin, 2008) If

A

n × n whose entries are independent random 1 and bounded fourth moment. Then for δ > 0, there exists  > 0 and integer n0 > 0 such that    P sn (A) ≤ √ ≤ δ, ∀n ≥ n0 . n is a matrix of size

variables with variance any

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Smallest singular value in

`2

Later, they proved the following result Theorem (RudelsonVershynin, 2008) Let

A

be an

n×n

matrix whose entries are i.i.d. centered random

variables with unit variance and fourth moment bounded by B. Then, for every

there exist

δ>0

(polynomially) only on

δ

K>0

and

n0

which depend

and B, and such that

  K ≤ δ, P sn (A) > √ n

Louis Yang Liu

∀n ≥ n0

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Largest

q -singular

value

Denition The largest q -singular value (p)

s1 (A) : =

sup x∈RN , kxkp =1

kAxkp

for an m × N matrix A.

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Smallest

p-singular

value

Denition The smallest p-singular value of an n × n matrix

s(p) n (A) : =

Louis Yang Liu

inf

x∈Rn , kxkp =1

kAxkp .

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

A property of largest

p-singular

value

Lemma For any

p > 1,

the largest

p-singular

value is a norm on the space

of matrices, and (p)

max kaj kp ≤ s1 (A) ≤ N j

aj , j = 1, 2, · · · , N , m × N.

where

p−1 p

max kaj kp , j

are the column vectors of

Louis Yang Liu

A

of size

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Proof of the lemma

Proof. For any x = (x1 , x2 , , · · · , xN )T ∈ RN , by the Minkowski inequality

kAxkp ≤

N X

|xj | · kaj kp ≤ kxk1 max kaj kp , 1≤j≤N

j=1

and then by the discrete Hölder inequality,

kAxkp ≤ kxkp N

1− p1

max kaj kp .

1≤j≤N

On the other hand, choosing x to be the standard basis vectors of (q) RN yields maxj ||aj ||q ≤ s1 (A). Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Duality Property

Lemma (Duality) For any

p≥1

and

m×N

matrix

(p)

A, (q)

s1 (A) = s1 1

where p

+

1 q

AT




= 1.

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Sum of random variables

Lemma (Linear bound for partial binomial expansion) For every positive integer n, n X



+1 k=b n 2c

n k



xk (1 − x)n−k ≤ 8x

for all x ∈ [0, 1].

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Sum of random variables

Lemma Suppose ξ1 , ξ2 , · · · , ξn are i.i.d. copies of a random variable then for any ε > 0,

P

n X i=1

nε |ξi | ≤ 2

Louis Yang Liu

ξ,

! ≤ 8P (|ξ| ≤ ε) .

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Lower tail probability of largest

1-singular

value

Theorem (Lower tail probability of the largest 1-singular value) Let

A

ξ

be a pregaussian variable normalized to have variance

m × N matrix with i.i.d. copies of ξ in ε > 0, there exists K > 0 such that   (1) P s1 (A) ≤ Km ≤ ε

be an

any

where

K

only depends on

ε

and the pregaussian variable

Louis Yang Liu

1

and

its entries, then for

ξ.

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Lower tail probability of largest

1-singular

value

Proof. Since aij is pregaussian with variance 1, then for any ε > 0, there exists some δ > 0 such that

P (|aij | ≤ δ) ≤

ε . 8

P (1) Since s1 (A) = m i=1 |aij0 | for some j0 , by the previous lemma, we have !   m X mδ mδ (1) P s1 (A) ≤ ≤ P ≤ 8P (|aij | ≤ δ) ≤ ε. |aij0 | ≤ 2 2 i=1

Finally let K = 2δ , then the claim follows. Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Lower tail probability of largest

p-singular

value

Theorem (Lower tail probability of the largest -singular value, p > 1) Let

1

ξ

and

be a pregaussian random variable normalized to have variance

A

be an

then for every

where

γ

m × N matrix with i.i.d. copies of ξ in its entries, p > 1 and any ε > 0, there exists γ > 0 such that   1 (p) ≤ ε P s1 (A) ≤ γm p

only depends on

p, ε

and the pregaussian random variable

ξ.

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

Upper tail probability of largest

p-singular

value

Theorem (Upper tail probability of the largest p-singular value of rectangular matrices, 1 < p ≤ 2 ) Let

A

ξ

be a pregaussian variable normalized to have variance

1 and m × N matrix with i.i.d. copies of ξ in its entries, then for 1 < p ≤ 2 and any ε > 0, there exists K > 0 such that   1  1 1 (p) −1 P s1 (A) ≥ K m p + m p 2 N 2 ≤ ε

is an

every

where

K

only depends on

p, ε

and the pregaussian variable

Louis Yang Liu

ξ.

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

A Remark

Remark By the duality lemma, we can obtain the corresponding probabilty (p) estimates on s1 (A) for p > 2.

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

2-singular

value

For p = 2, we plot the largest 2-singular value of Gaussian random matrices of size n × n, where n runs from 1 through 100.

Figure: Largest

2-singular

value of Gaussian random matrices

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

1-singular

value

In the second numerical experiment for p = 1, we plot the largest 1-singular value of Gaussian random matrices of size n × n, where n runs from 1 through 200.

Figure: Largest

1-singular

value of Gaussian random matrices:

Experiment 1

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

1-singular

value

In the third experiment for p = 1, we plot the largest 1-singular value of Gaussian random matrices of size n × n, where n runs from 1 through 400.

Figure: Largest

1-singular

value of Gaussian random matrices:

Experiment 2

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

∞-singular

value

For p = ∞, we plot the largest ∞-singular value of Gaussian random matrices of size n × n, where n runs from 1 through 500.

Figure: Largest

∞-singular

value of Gaussian random matrices

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

4 3 -singular value

For p = 34 , using the pnorm algorithm, we plot the largest 4-singular value of Gaussian random matrices of size n × n, where n runs from 1 through 44 (= 256).

Figure: Largest

4 3 -singular value of Gaussian random matrices

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

4-singular

value

For rectangular matrices, we plot the largest 4-singular value of Gaussian random matrices and Bernoulli random matrices of size m × n, where m and n run from 1 through 34 (= 81).

Figure: Largest

4-singular

value of Gaussian random matrices

Louis Yang Liu

Generalized Singular Values of Random Matrices

Introduction Properties Probability Estimates Numerical Experiments

4-singuar

value of Bernoulli matrices

Also, we plot the largest 4-singular value of Bernoulli random matrices of size m × n, where m and n run from 1 through 34 (= 81).

Figure: Largest

4-singular

value of Bernoulli random matrices

Louis Yang Liu

Generalized Singular Values of Random Matrices