Optimization methods for the length and growth problems

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions

Christos Kravvaritis joint work with Marilena Mitrouli

Optimization formulation Optimization tools Numerical Results

The growth problem

University of Athens Department of Mathematics

Numerical Analysis 2008 - Kalamata

Definitions Optimization formulation Numerical Results

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis The length problem

A matrix A is called

Definitions Optimization formulation Optimization tools

I I

normalized if maxi,j |aij | = 1 normalized orthogonal (NO) if A is normalized and satisfies additionally AAT = c(A)In for some constant c(A).

Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis The length problem

A matrix A is called

Definitions Optimization formulation Optimization tools

I I

normalized if maxi,j |aij | = 1 normalized orthogonal (NO) if A is normalized and satisfies additionally AAT = c(A)In for some constant c(A).

Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

Example (NO matrices) I

Hadamard matrices H of order n, with entries ±1, satisfying HH T = nIn

C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools

I

weighing matrices W of order n and weight n − k , with entries (0, ±1), satisfying

Numerical Results

The growth problem Definitions

WW I

T

= (n − k )In

orthogonal designs D of order n and type (u1 , u2 , . . . , ut ), ui positive integers, with entries from the set {0, ±x1 , ±x2 , . . . , ±xt }, satisfying ! t X T 2 DD = ui xi In i=1

Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

The problem of determining

The length problem Definitions Optimization formulation

c(n) = sup{c(A)|A ∈ Rn×n , NO} is called the length problem. (Day & Peterson, 1988) p "length" → c(A): usual Euclidean length of every row of a NO matrix A

Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

The problem of determining

The length problem Definitions Optimization formulation

c(n) = sup{c(A)|A ∈ Rn×n , NO} is called the length problem. (Day & Peterson, 1988) p "length" → c(A): usual Euclidean length of every row of a NO matrix A

Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

Main idea: Represent the n2 unknown entries of an n × n matrix X = (xij ), 1 ≤ i, j ≤ n, as a vector x¯ with elements x1 , . . . , xn2

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem

x11 , . . . , x1n ,

x21 ,

...,

xn1 ,

. . . , xnn

Definitions Optimization formulation Numerical Results

m x1 ,

...,

xn ,

xn+1 , . . . , x(n−1)n+1 , . . . , xn2

Optimization methods for the length and growth problems C. Kravvaritis

Minimize the objective function −

n X

The length problem

x1j2 (= −c(X )),

j=1

Definitions Optimization formulation Optimization tools Numerical Results

or equivalently, in terms of the vector formulation,

The growth problem Definitions Optimization formulation

f (x¯ ) = −

n X i=1

This is not enough . . . constraints

Numerical Results

xi2

Optimization methods for the length and growth problems C. Kravvaritis

Minimize the objective function −

n X

The length problem

x1j2 (= −c(X )),

j=1

Definitions Optimization formulation Optimization tools Numerical Results

or equivalently, in terms of the vector formulation,

The growth problem Definitions Optimization formulation

f (x¯ ) = −

n X i=1

This is not enough . . . constraints

Numerical Results

xi2

Optimization methods for the length and growth problems C. Kravvaritis

1. The equality of the usual Euclidean lengths of every two distinct rows (equal to c(X ))

The length problem Definitions Optimization formulation Optimization tools

n X

xij2 −

n X

j=1

Numerical Results

2 xi+1,j = 0,

1≤i ≤n−1

j=1

The growth problem Definitions Optimization formulation Numerical Results

n X k =1

2 x(i−1)n+k

−

n X k =1

2 xin+k = 0,

1≤i ≤n−1

Optimization methods for the length and growth problems

2. The orthogonality of every two distinct rows of X n X

xik xjk = 0,

1≤i
C. Kravvaritis The length problem Definitions Optimization formulation

k =1

Optimization tools Numerical Results

n X

The growth problem

x(i−1)n+k x(j−1)n+k = 0,

1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n

k =1

3. Normalized matrix −1 ≤ xi ≤ 1, i = 1, . . . , n2

Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

2. The orthogonality of every two distinct rows of X n X

xik xjk = 0,

1≤i
C. Kravvaritis The length problem Definitions Optimization formulation

k =1

Optimization tools Numerical Results

n X

The growth problem

x(i−1)n+k x(j−1)n+k = 0,

1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n

k =1

3. Normalized matrix −1 ≤ xi ≤ 1, i = 1, . . . , n2

Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

Summarizing:

C. Kravvaritis

minx¯∈Rn2 f (x¯ ) = −

Pn

2 i=1 xi

The length problem Definitions Optimization formulation

Pn

2 k =1 x(i−1)n+k

Pn

subject to Pn 2 − k =1 xin+k = 0,

k =1 x(i−1)n+k x(j−1)n+k

Optimization tools Numerical Results

1 ≤ i ≤ n − 1,

= 0, 1 ≤ i ≤ n − 1, i + 1 ≤ j ≤ n,

The growth problem Definitions Optimization formulation Numerical Results

xi + 1 ≥ 0,

i = 1, . . . , n2

1 − xi ≥ 0,

i = 1, . . . , n2 .

Totally: n − 1 + n(n−1) equality constraints 2 2n2 inequality constraints

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

I

most appropriate method: the Sequential Quadratic Programming (SQP) algorithm

The length problem Definitions Optimization formulation Optimization tools

I

SQP: one of the most effective methods for nonlinearly constrained optimization problems

I

equally suitable for both small and large problems

I

based on generation of steps by solving sequentially appropriately formulated quadratic subproblems

Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools

minx∈Rn f (x)

(1)

subject to

ci (x) = 0,

i ∈ E,

(2)

and

ci (x) ≥ 0,

i ∈ I,

(3)

where f , ci : Rn → R are smooth.

Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

I

Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?

I

Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .

I

I

Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?

I

Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .

I

I

Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?

I

Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .

I

I

Line search strategy: choose a direction pk and search along this direction from the current iterate xk for a new iterate with a lower function value. The distance ak to move along pk is the approximate solution of: min f (xk + apk ). a>0

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

Then the new iterate is updated as xk +1 = xk + ak pk . How do we find pk ?

I

Main idea of SQP: model (1)-(3) at the current xk by a quadratic programming subproblem and use its minimizer for a new xk +1 .

I

The quadratic programming subproblem is obtained by linearizing the inequality and equality constraints: 1 minn pT Bk p + ∇fkT p p∈R 2 subject to ∇ci (xk )T p + ci (xk ) = 0,

i ∈ E,

and ∇ci (xk )T p + ci (xk ) ≥ 0,

i ∈ I.

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions

Bk stands for B(xk , λk ), where B(x, λ) = is the Hessian matrix of the Lagrangian. I

∇2xx L(x, λ)

The solution of the quadratic subproblem produces a vector pk , and xk +1 = xk + ak pk

Optimization formulation Numerical Results

I

The quadratic programming subproblem is obtained by linearizing the inequality and equality constraints: 1 minn pT Bk p + ∇fkT p p∈R 2 subject to ∇ci (xk )T p + ci (xk ) = 0,

i ∈ E,

and ∇ci (xk )T p + ci (xk ) ≥ 0,

i ∈ I.

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions

Bk stands for B(xk , λk ), where B(x, λ) = is the Hessian matrix of the Lagrangian. I

∇2xx L(x, λ)

The solution of the quadratic subproblem produces a vector pk , and xk +1 = xk + ak pk

Optimization formulation Numerical Results

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Table: Numerical results for the length problem

n

c(n)

3 4 5 6 7 8 9 10 11 12 13 14 15 16

2.25 4 3.3611 5 5.0777 8 6.4308 9 8.4495 12 10.3934 13 11.8511 16

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

I

For n ≡ 0 mod 4: c(n) is given by a Hadamard matrix of order n Can be proved theoretically, resulting to:

C. Kravvaritis The length problem Definitions

c(n) = n iff there exists a Hadamard matrix of order n

Optimization formulation Optimization tools Numerical Results

I

For n ≡ 2 mod 4: c(n) is given by a weighing matrix of order n and weight n − 1 Theoretical proof . . . open problem

I

The rest of the values do not seem to follow a specific, predictable pattern.

I

Are the values for n odd true or not? Subject to any particular formula?

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

I

Very stable behavior of the algorithm

The length problem Definitions

I

Is c(5) < (4) , c(11) < c(10) , c(13) < c(12) etc.?

Optimization formulation Optimization tools Numerical Results

I

The growth problem

The functions

Definitions

c(n), n odd c(n), n even are increasing. Monotonicity of c(n)?

Optimization formulation Numerical Results

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

I

linear system Ax = b, A = [aij ] ∈ Rn×n

I

Gaussian Elimination (GE):  a11 a12 · · · a1n  a21 a22 · · · a2n    A= .. .. ..  −→  . . ··· . 

A(n−1)

an1 an2 · · · ann  a11 a12 · · ·  0 a(1) · · ·  22  (2) 0 0 a33 =  .. .. ..   . . . 0 0 0

C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation

··· ··· ··· .. . ···

a1n (1) a2n (2) a3n .. . (n−1)

ann

       

Numerical Results

Backward error analysis for GE −→ growth factor (k )

g(n, A) =

maxi,j,k |aij | maxi,j |aij |

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

Theorem (Wilkinson) The computed solution xˆ to the linear system Ax = b using GE with partial pivoting satisfies (A + ∆A)xˆ = b with k∆Ak∞ ≤ cn3 g(n, A)kAk∞ u.

The growth problem Definitions Optimization formulation Numerical Results

Backward error analysis for GE −→ growth factor (k )

g(n, A) =

maxi,j,k |aij | maxi,j |aij |

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

Theorem (Wilkinson) The computed solution xˆ to the linear system Ax = b using GE with partial pivoting satisfies (A + ∆A)xˆ = b with k∆Ak∞ ≤ cn3 g(n, A)kAk∞ u.

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

The problem of determining

The length problem Definitions

g(n) = sup{g(n, A)|A ∈ Rn×n }

Optimization formulation Optimization tools Numerical Results

is called the growth problem. (Cryer, 1968)

The growth problem Definitions Optimization formulation Numerical Results

g(n) ≤ [n 2 31/2 . . . n1/n−1 ]1/2 (Wilkinson, 1961)

Optimization methods for the length and growth problems C. Kravvaritis

I

g(n) = h(n), where

The length problem Definitions Optimization formulation

h(n) = sup{h(A)|A completely pivoted normalized} and h(A) =

(n−1) |ann |

Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

I

Matrices with the property that no row and column exchanges are needed during GECP are called completely pivoted (CP)

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

1. If a matrix is divided by its maximum element, it yields the same growth as the initial matrix. Hence, one can deal without loss of generality with normalized matrices.

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation

2. For discussing the growth factor arising during GECP we can restrict ourselves without any loss of generality to CP matrices.

Numerical Results

Optimization methods for the length and growth problems

Summarizing:

C. Kravvaritis The length problem

minX ∈Rn×n f (X ) =

(n−1) −(xnn )2

Definitions Optimization formulation Optimization tools Numerical Results

The growth problem

subject to

Definitions

x11 = 1,

Optimization formulation Numerical Results

xij + 1 ≥ 0,

1 ≤ i, j ≤ n,

1 − xij ≥ 0,

1 ≤ i, j ≤ n,

(k −1) (xkk )2

(k −1) 2 )

− (xij

≥ 0,

2 ≤ k ≤ n − 1, k ≤ i, j ≤ n, (i, j) 6= (k , k ).

Optimization methods for the length and growth problems

Proposition (Cryer, 1968) Rn×n

Let A ∈ on A. Then

C. Kravvaritis

be a CP, invertible matrix. We apply GECP

The length problem Definitions

 A (k )

aij

=

1 2 ... k i 1 2 ... k j A(1 2 . . . k )

Optimization formulation



Optimization tools Numerical Results

,

1 ≤ k ≤ n − 1,

The growth problem Definitions Optimization formulation Numerical Results

where  A

i1 i2 . . . ip j1 j2 . . . jp 

A(i1 i2 . . . ip ) = A



ai j . . . ai j p 1 11 .. . . .. , = . . . ai j . . . ai j p 1 p p

i1 i2 . . . ip i1 i2 . . . ip

 , p ≥ 1.

Optimization methods for the length and growth problems C. Kravvaritis The length problem

Lemma

Definitions

(Gantmacher, 1959)

Optimization formulation Optimization tools

Let A be a CP matrix. The magnitude of the pivots appearing after application of GE operations on A is given by

Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

(j−1)

ajj

=

A(1 2 . . . j) , A(1 2 . . . j − 1)

j = 1, 2, . . . , n,

A(0) = 1.

Outline

Optimization methods for the length and growth problems C. Kravvaritis

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Table: Gould’s results (1991) for the growth problem

n

g(n)

3 4 5 6 7 8 9 10 11 12 13 14 15 16

2.25 4 4.1325 5 6 8 8.4305 9.5254 10.4627 12 13.0205 14.5949 16.1078 18.0596

Optimization methods for the length and growth problems C. Kravvaritis The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems

I

Cryer, 1968 Conjecture: g(n, A) ≤ n, with equality iff A is a Hadamard matrix

C. Kravvaritis The length problem Definitions

The first part of Cryer’s conjecture is false. The second part still remains open I

Matrices very sensitive to small perturbations

Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation

I

For what values of n are all local extrema for the growth problem in fact global extrema?

I

Is h(A) = c(A) = g(A) when A is a normalized orthogonal matrix?

I

For what values of n does g(n) = c(n)?

Numerical Results

References

Optimization methods for the length and growth problems C. Kravvaritis

C. W. Cryer, Pivot size in Gaussian elimination, Numer. Math. 12, 335–345 (1968)

The length problem Definitions

N. Gould, On growth in Gaussian elimination with pivoting, SIAM J. Matrix Anal. Appl. 12, 354–361 (1991) A. Edelman, The Complete Pivoting Conjecture for Gaussian Elimination is false, The Mathematica Journal 2, 58–61 (1992) J. H. Wilkinson, Error analysis of direct methods of matrix inversion, J. Assoc. Comput. Mach. 8, 281–330 (1961) T. A. Driscoll and K. L. Maki, Searching for rare growth factors using Multicanonical Monte Carlo Methods, SIAM Review 49(4) (2007), 673–692

Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis

C. Kravvaritis and M. Mitrouli, Evaluation of Minors associated to weighing matrices, Linear Algebra Appl., 426 (2007), 774-809 M. S. Bazaraa, C. M. Shetty, Nonlinear Programming, theory and algorithms, Wiley, Toronto, 1979 P. E. Gill, W. Murray, M. H.Wright, Practical Optimization, Academic Press, London, 1981

The length problem Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

Optimization methods for the length and growth problems C. Kravvaritis The length problem

Thank you very much for your attention!

Definitions Optimization formulation Optimization tools Numerical Results

The growth problem Definitions Optimization formulation Numerical Results

http://ckravv.googlepages.com