Optimization methods for the length and growth problems
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions
Christos Kravvaritis joint work with Marilena Mitrouli
Optimization formulation Optimization tools Numerical Results
The growth problem
University of Athens Department of Mathematics
Numerical Analysis 2008 - Kalamata
Definitions Optimization formulation Numerical Results
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis The length problem
A matrix A is called
Definitions Optimization formulation Optimization tools
I I
normalized if maxi,j |aij | = 1 normalized orthogonal (NO) if A is normalized and satisfies additionally AAT = c(A)In for some constant c(A).
Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis The length problem
A matrix A is called
Definitions Optimization formulation Optimization tools
I I
normalized if maxi,j |aij | = 1 normalized orthogonal (NO) if A is normalized and satisfies additionally AAT = c(A)In for some constant c(A).
Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
Example (NO matrices) I
Hadamard matrices H of order n, with entries ±1, satisfying HH T = nIn
C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools
I
weighing matrices W of order n and weight n − k , with entries (0, ±1), satisfying
Numerical Results
The growth problem Definitions
WW I
T
= (n − k )In
orthogonal designs D of order n and type (u1 , u2 , . . . , ut ), ui positive integers, with entries from the set {0, ±x1 , ±x2 , . . . , ±xt }, satisfying ! t X T 2 DD = ui xi In i=1
Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
The problem of determining
The length problem Definitions Optimization formulation
c(n) = sup{c(A)|A ∈ Rn×n , NO} is called the length problem. (Day & Peterson, 1988) p "length" → c(A): usual Euclidean length of every row of a NO matrix A
Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
The problem of determining
The length problem Definitions Optimization formulation
c(n) = sup{c(A)|A ∈ Rn×n , NO} is called the length problem. (Day & Peterson, 1988) p "length" → c(A): usual Euclidean length of every row of a NO matrix A
Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
Main idea: Represent the n2 unknown entries of an n × n matrix X = (xij ), 1 ≤ i, j ≤ n, as a vector x¯ with elements x1 , . . . , xn2
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem
x11 , . . . , x1n ,
x21 ,
...,
xn1 ,
. . . , xnn
Definitions Optimization formulation Numerical Results
m x1 ,
...,
xn ,
xn+1 , . . . , x(n−1)n+1 , . . . , xn2
Optimization methods for the length and growth problems C. Kravvaritis
Minimize the objective function −
n X
The length problem
x1j2 (= −c(X )),
j=1
Definitions Optimization formulation Optimization tools Numerical Results
or equivalently, in terms of the vector formulation,
The growth problem Definitions Optimization formulation
f (x¯ ) = −
n X i=1
This is not enough . . . constraints
Numerical Results
xi2
Optimization methods for the length and growth problems C. Kravvaritis
Minimize the objective function −
n X
The length problem
x1j2 (= −c(X )),
j=1
Definitions Optimization formulation Optimization tools Numerical Results
or equivalently, in terms of the vector formulation,
The growth problem Definitions Optimization formulation
f (x¯ ) = −
n X i=1
This is not enough . . . constraints
Numerical Results
xi2
Optimization methods for the length and growth problems C. Kravvaritis
1. The equality of the usual Euclidean lengths of every two distinct rows (equal to c(X ))
The length problem Definitions Optimization formulation Optimization tools
n X
xij2 −
n X
j=1
Numerical Results
2 xi+1,j = 0,
1≤i ≤n−1
j=1
The growth problem Definitions Optimization formulation Numerical Results
n X k =1
2 x(i−1)n+k
−
n X k =1
2 xin+k = 0,
1≤i ≤n−1
Optimization methods for the length and growth problems
2. The orthogonality of every two distinct rows of X n X
xik xjk = 0,
1≤i
C. Kravvaritis The length problem Definitions Optimization formulation
k =1
Optimization tools Numerical Results
n X
The growth problem
x(i−1)n+k x(j−1)n+k = 0,
1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n
k =1
3. Normalized matrix −1 ≤ xi ≤ 1, i = 1, . . . , n2
Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
2. The orthogonality of every two distinct rows of X n X
xik xjk = 0,
1≤i
C. Kravvaritis The length problem Definitions Optimization formulation
k =1
Optimization tools Numerical Results
n X
The growth problem
x(i−1)n+k x(j−1)n+k = 0,
1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n
k =1
3. Normalized matrix −1 ≤ xi ≤ 1, i = 1, . . . , n2
Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
Summarizing:
C. Kravvaritis
minx¯∈Rn2 f (x¯ ) = −
Pn
2 i=1 xi
The length problem Definitions Optimization formulation
Pn
2 k =1 x(i−1)n+k
Pn
subject to Pn 2 − k =1 xin+k = 0,
k =1 x(i−1)n+k x(j−1)n+k
Optimization tools Numerical Results
1 ≤ i ≤ n − 1,
= 0, 1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n,
The growth problem Definitions Optimization formulation Numerical Results
xi + 1 ≥ 0,
i = 1, . . . , n2
1 − xi ≥ 0,
i = 1, . . . , n2 .
Totally: n − 1 + n(n−1) equality constraints 2 2n2 inequality constraints
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
I
most appropriate method: the Sequential Quadratic Programming (SQP) algorithm
The length problem Definitions Optimization formulation Optimization tools
I
SQP: one of the most effective methods for nonlinearly constrained optimization problems
I
equally suitable for both small and large problems
I
based on generation of steps by solving sequentially appropriately formulated quadratic subproblems
Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools
minx∈Rn f (x)
(1)
subject to
ci (x) = 0,
i ∈ E,
(2)
and
ci (x) ≥ 0,
i ∈ I,
(3)
where f , ci : Rn → R are smooth.
Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
I
Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?
I
Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .
I
I
Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?
I
Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .
I
I
Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?
I
Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .
I
I
Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?
I
Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .
I
The quadratic programming subproblem is obtained by linearizing the inequality and equality constraints: 1 minn pT Bk p + ∇fkT p p∈R 2 subject to ∇ci (xk )T p + ci (xk ) = 0,
i ∈ E,
and ∇ci (xk )T p + ci (xk ) ≥ 0,
i ∈ I.
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions
Bk stands for B(xk , λk ), where B(x, λ) = is the Hessian matrix of the Lagrangian. I
∇2xx L(x, λ)
The solution of the quadratic subproblem produces a vector pk , and xk +1 = xk + ak pk
Optimization formulation Numerical Results
I
The quadratic programming subproblem is obtained by linearizing the inequality and equality constraints: 1 minn pT Bk p + ∇fkT p p∈R 2 subject to ∇ci (xk )T p + ci (xk ) = 0,
i ∈ E,
and ∇ci (xk )T p + ci (xk ) ≥ 0,
i ∈ I.
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions
Bk stands for B(xk , λk ), where B(x, λ) = is the Hessian matrix of the Lagrangian. I
∇2xx L(x, λ)
The solution of the quadratic subproblem produces a vector pk , and xk +1 = xk + ak pk
Optimization formulation Numerical Results
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Table: Numerical results for the length problem
n
c(n)
3 4 5 6 7 8 9 10 11 12 13 14 15 16
2.25 4 3.3611 5 5.0777 8 6.4308 9 8.4495 12 10.3934 13 11.8511 16
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
I
For n ≡ 0 mod 4: c(n) is given by a Hadamard matrix of order n Can be proved theoretically, resulting to:
C. Kravvaritis The length problem Definitions
c(n) = n iff there exists a Hadamard matrix of order n
Optimization formulation Optimization tools Numerical Results
I
For n ≡ 2 mod 4: c(n) is given by a weighing matrix of order n and weight n − 1 Theoretical proof . . . open problem
I
The rest of the values do not seem to follow a specific, predictable pattern.
I
Are the values for n odd true or not? Subject to any particular formula?
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
I
Very stable behavior of the algorithm
The length problem Definitions
I
Is c(5) < (4) , c(11) < c(10) , c(13) < c(12) etc.?
Optimization formulation Optimization tools Numerical Results
I
The growth problem
The functions
Definitions
c(n), n odd c(n), n even are increasing. Monotonicity of c(n)?
Optimization formulation Numerical Results
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
I
linear system Ax = b, A = [aij ] ∈ Rn×n
I
Gaussian Elimination (GE): a11 a12 · · · a1n a21 a22 · · · a2n A= .. .. .. −→ . . ··· .
A(n−1)
an1 an2 · · · ann a11 a12 · · · 0 a(1) · · · 22 (2) 0 0 a33 = .. .. .. . . . 0 0 0
C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation
··· ··· ··· .. . ···
a1n (1) a2n (2) a3n .. . (n−1)
ann
Numerical Results
Backward error analysis for GE −→ growth factor (k )
g(n, A) =
maxi,j,k |aij | maxi,j |aij |
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
Theorem (Wilkinson) The computed solution xˆ to the linear system Ax = b using GE with partial pivoting satisfies (A + ∆A)xˆ = b with k∆Ak∞ ≤ cn3 g(n, A)kAk∞ u.
The growth problem Definitions Optimization formulation Numerical Results
Backward error analysis for GE −→ growth factor (k )
g(n, A) =
maxi,j,k |aij | maxi,j |aij |
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
Theorem (Wilkinson) The computed solution xˆ to the linear system Ax = b using GE with partial pivoting satisfies (A + ∆A)xˆ = b with k∆Ak∞ ≤ cn3 g(n, A)kAk∞ u.
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
The problem of determining
The length problem Definitions
g(n) = sup{g(n, A)|A ∈ Rn×n }
Optimization formulation Optimization tools Numerical Results
is called the growth problem. (Cryer, 1968)
The growth problem Definitions Optimization formulation Numerical Results
g(n) ≤ [n 2 31/2 . . . n1/n−1 ]1/2 (Wilkinson, 1961)
Optimization methods for the length and growth problems C. Kravvaritis
I
g(n) = h(n), where
The length problem Definitions Optimization formulation
h(n) = sup{h(A)|A completely pivoted normalized} and h(A) =
(n−1) |ann |
Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
I
Matrices with the property that no row and column exchanges are needed during GECP are called completely pivoted (CP)
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
1. If a matrix is divided by its maximum element, it yields the same growth as the initial matrix. Hence, one can deal without loss of generality with normalized matrices.
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation
2. For discussing the growth factor arising during GECP we can restrict ourselves without any loss of generality to CP matrices.
Numerical Results
Optimization methods for the length and growth problems
Summarizing:
C. Kravvaritis The length problem
minX ∈Rn×n f (X ) =
(n−1) −(xnn )2
Definitions Optimization formulation Optimization tools Numerical Results
The growth problem
subject to
Definitions
x11 = 1,
Optimization formulation Numerical Results
xij + 1 ≥ 0,
1 ≤ i, j ≤ n,
1 − xij ≥ 0,
1 ≤ i, j ≤ n,
(k −1) (xkk )2
(k −1) 2 )
− (xij
≥ 0,
2 ≤ k ≤ n − 1, k ≤ i, j ≤ n, (i, j) 6= (k , k ).
Optimization methods for the length and growth problems
Proposition (Cryer, 1968) Rn×n
Let A ∈ on A. Then
C. Kravvaritis
be a CP, invertible matrix. We apply GECP
The length problem Definitions
A (k )
aij
=
1 2 ... k i 1 2 ... k j A(1 2 . . . k )
Optimization formulation
Optimization tools Numerical Results
,
1 ≤ k ≤ n − 1,
The growth problem Definitions Optimization formulation Numerical Results
where A
i1 i2 . . . ip j1 j2 . . . jp
A(i1 i2 . . . ip ) = A
ai j . . . ai j p 1 11 .. . . .. , = . . . ai j . . . ai j p 1 p p
i1 i2 . . . ip i1 i2 . . . ip
, p ≥ 1.
Optimization methods for the length and growth problems C. Kravvaritis The length problem
Lemma
Definitions
(Gantmacher, 1959)
Optimization formulation Optimization tools
Let A be a CP matrix. The magnitude of the pivots appearing after application of GE operations on A is given by
Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
(j−1)
ajj
=
A(1 2 . . . j) , A(1 2 . . . j − 1)
j = 1, 2, . . . , n,
A(0) = 1.
Outline
Optimization methods for the length and growth problems C. Kravvaritis
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Table: Gould’s results (1991) for the growth problem
n
g(n)
3 4 5 6 7 8 9 10 11 12 13 14 15 16
2.25 4 4.1325 5 6 8 8.4305 9.5254 10.4627 12 13.0205 14.5949 16.1078 18.0596
Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems
I
Cryer, 1968 Conjecture: g(n, A) ≤ n, with equality iff A is a Hadamard matrix
C. Kravvaritis The length problem Definitions
The first part of Cryer’s conjecture is false. The second part still remains open I
Matrices very sensitive to small perturbations
Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation
I
For what values of n are all local extrema for the growth problem in fact global extrema?
I
Is h(A) = c(A) = g(A) when A is a normalized orthogonal matrix?
I
For what values of n does g(n) = c(n)?
Numerical Results
References
Optimization methods for the length and growth problems C. Kravvaritis
C. W. Cryer, Pivot size in Gaussian elimination, Numer. Math. 12, 335–345 (1968)
The length problem Definitions
N. Gould, On growth in Gaussian elimination with pivoting, SIAM J. Matrix Anal. Appl. 12, 354–361 (1991) A. Edelman, The Complete Pivoting Conjecture for Gaussian Elimination is false, The Mathematica Journal 2, 58–61 (1992) J. H. Wilkinson, Error analysis of direct methods of matrix inversion, J. Assoc. Comput. Mach. 8, 281–330 (1961) T. A. Driscoll and K. L. Maki, Searching for rare growth factors using Multicanonical Monte Carlo Methods, SIAM Review 49(4) (2007), 673–692
Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis
C. Kravvaritis and M. Mitrouli, Evaluation of Minors associated to weighing matrices, Linear Algebra Appl., 426 (2007), 774-809 M. S. Bazaraa, C. M. Shetty, Nonlinear Programming, theory and algorithms, Wiley, Toronto, 1979 P. E. Gill, W. Murray, M. H.Wright, Practical Optimization, Academic Press, London, 1981
The length problem Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
Optimization methods for the length and growth problems C. Kravvaritis The length problem
Thank you very much for your attention!
Definitions Optimization formulation Optimization tools Numerical Results
The growth problem Definitions Optimization formulation Numerical Results
http://ckravv.googlepages.com
