Integer quadratic programming in the plane Alberto Del Pia

∗

Abstract We show that the problem of minimizing a quadratic polynomial with integer coefficients over the integer points in a general two-dimensional rational polyhedron is solvable in time bounded by a polynomial in the input size.

1 Introduction Among the most important theorems in the theory of fixed-dimension optimization is Lenstra’s result [6]. It states that it is possible to minimize a linear function with integer coefficients over the integer points in a rational polyhedron in time bounded by a polynomial in the size of the input, provided that the number of integer variables is a constant. Note that the polyhedron can be given either by an external or internal description, since in fixed dimension one can pass from one to the other in polynomial time. This polynomiality result does not extend if one replaces a linear objective function with a general polynomial as shown in [2], not even when the dimension is equal to two. In fact, Manders and Adleman [8] proved that the following decision problem is NPcomplete. (See problem AN1 on page 249 of [3].) AN1 Given positive integers a, b, and c, is there a positive integer x < c such that x2 is congruent to a modulo b? It is an easy exercise to reformulate problem AN1 as a minimization problem with a degree four polynomial with integer coefficients as an objective function, and a feasible domain consisting of all integer points in a two-dimensional box. This shows that minimizing a polynomial of degree four with integer coefficients over the integer points in a rational polyhedron is NP-hard already in dimension two. This naturally leads to the question about the complexity status of minimizing a polynomial of degree two or three over the integer points in a rational polyhedron in dimension two. Let us formally introduce the problem (1.1)

min{f k (z) : z ∈ P ∩ Zn },

∗ Goldstine Fellow, Business Analytics and Mathematical Sciences department, IBM Watson Research Center, United States † Institute for Operations Research, Department of Mathematics, ETH Z¨ urich, Switzerland

Robert Weismantel

†

where f k is a polynomial function of degree at most k with integer coefficients, and P is a rational polyhedron in Rn . We recall that a rational polyhedron is the set of points that satisfy a system of linear inequalities with rational data. Problem (1.1) is bounded if there exists γ ∈ Z such that f k (z) ≥ γ for every z ∈ P ∩ Zn , and it is unbounded otherwise. We say that we can solve problem (1.1) if we can either show that P ∩ Zn = ∅, or show that problem (1.1) is unbounded, or find an integer point in P that minimizes f . The complexity of problem (1.1) in fixed dimension is summarized in Table 1.

k=1 k=2 k=3 k=4 k general

n=1 P P P P P

n=2 P ? ? NPH NPH

n fixed P ? ? NPH NPH

Table 1: Complexity of problem (1.1) in fixed dimension. “P” stands for polynomial time solvable, and “NPH” stands for NP-hard.

For n = 1 problem (1.1) is easily solvable in polynomial time as follows. It is easy to check that problem (1.1) is unbounded if and only if there exists an integer vector v in the recession cone of P such that av k ≤ −1, where a is the leading coefficient of f . If problem (1.1) is bounded, then compute all the roots of the derivative f 0 of f (this can be done by finding the eigenvalues of the companion matrix of f 0 ). Then evaluate f only on the integer boundary of P , and on the integer points at distance at most one from such zeroes of f 0 . Therefore the first non-trivial case is when n = k = 2. Our main result is the following. Theorem 1.1. If n = k = 2, problem (1.1) can be solved in polynomial time. Note that Theorem 1.1 is best possible in the sense that if we add only one quadratic constraint to problem (1.1) with n = k = 2, then the problem becomes

NP-hard. To see this, we use again the the reduction from the NP-complete problem AN1. Problem AN1 is equivalent to asking whether the minimum of the quadratic polynomial function x2 − a − by over the integer points (x, y) in the bounded set defined by in2 ≤ y ≤ (c−1)b −a , and equalities 1 ≤ x ≤ c − 1, 1−a b x2 − a − by ≥ 0, is zero or not.

where f : R2 → R is a quadratic polynomial with integer coefficients

Let us finally mention that Theorem 1.1 is not only of relevance from the optimization point of view. Rather there are number theory type questions that can be transformed into the problem of minimizing a degree two polynomial over the integer points in a rational polyhedron in dimension two. As one such example let us mention the Pell equation. The Pell equation is the equation

(2.4)

x2 = dy 2 + 1, to be solved in positive integers x, y for a given positive and non-square integer d. We refer to [7] for a treatment of history and solution methods for this equation. The Pell equation always has a solution, but all its solutions can be of size exponential in the size of d. In turn, Theorem 1.1 cannot be directly applicable to find one. Theorem 1.1 can be applied, however, to a bounded version of the Pell equation:

(2.3)

f (x, y) = ax2 + bxy + cy 2 + dx + ey,

and where P is a rational polyhedron in R2 . The graph of the quadratic equation f (x, y) = γ

is always a conic section, though it may be degenerate. The conic sections described by equation (2.4) can be classified with the discriminant ∆ = b2 − 4ac. If the conic section is non-degenerate, then if ∆ < 0 the equation represents an ellipse, if ∆ = 0 the equation represents a parabola, and if ∆ > 0 the equation represents a hyperbola. (See Figure 1.) To distinguish the degenerate cases from the nondegenerate cases, let Γ = ∆γ + bde − ae2 − cd2 . Then the conic section is non-degenerate if and only if Γ 6= 0. If Γ = 0 we have a point in the case of an ellipse, two parallel lines (possibly coinciding with each other) in the case of a parabola, or two intersecting lines in the case of a hyperbola.

given a positive and non-square integer d, and a positive integer u, do positive integers x, y exist The following Lemma 2.1 gives an important necwith x ≤ u and x2 = dy 2 + 1? essary condition of the boundedness of problem (2.2). Later, in Section 4, we will see that such condition is In order to solve this question one can proceed as also sufficient, and we will see how to use it in order to follows. First compute a rational approximation d˜ of understand if problem (2.2) is bounded or not in poly√ d such that nomial time. We denote by “rec” the recession cone (see for example [10]). 2 ˜ x ≤ u} = {(x, y) ∈ Z+ : x ≥ dy, = {(x, y) ∈ Z2+ : x2 ≥ dy 2 , x ≤ u}. This can be accomplished in polynomial time, for example by means of continued fractions. Consider now the problem ˜ 1 ≤ x ≤ u, (x, y) ∈ Z2 } min{x2 − dy 2 : x ≥ dy, + that we can solve in polynomial time by means of Theorem 1.1. The minimum of such problem is 1 if and only if there exists a solution of the Pell equation with the first coordinate smaller than u. 2

Ingredients for the proof of Theorem 1.1

From now on we will study the problem (2.2)

min{f (x, y) : (x, y) ∈ P ∩ Z2 },

Lemma 2.1. Assume that P contains an integer point. If problem (2.2) is bounded, then for every nonzero v in Z2 ∩ rec P , either avx2 + bvx vy + cvy2 ≥ 1, or avx2 + bvx vy + cvy2 = 0 and for every (x, y) ∈ Z2 ∩ P , (2ax + by + d)vx + (bx + 2cy + e)vy ≥ 0. Proof. Let z¯ = (¯ x, y¯) ∈ P ∩ Z2 , and let v be a nonzero 2 vector in Z ∩ rec P . Then f (¯ z + λv) − f (¯ z ) = λ2 (avx2 + bvx vy + cvy2 )+ + λ((2a¯ x + b¯ y + d)vx + (b¯ x + 2c¯ y + e)vy ). If avx2 + bvx vy + cvy2 < 0, then limλ→∞ f (¯ z + λv) = −∞, contradicting that problem (2.2) is bounded. Thus avx2 + bvx vy + cvy2 ≥ 0. If avx2 + bvx vy + cvy2 > 0, then by the integrality of a, b, c, v it follows that avx2 + bvx vy + cvy2 ≥ 1 and we are done.

Given a polyhedron Q, we will denote by QI the integer hull of Q, i.e. QI = conv(Q ∩ Z2 ). Meyer [9] showed that if Q is a rational polyhedron, then QI is again a (rational) polyhedron. Given a convex set S, we say that f : S → R is quasi-convex if each set {(x, y) ∈ S : f (x, y) ≤ γ} is convex. f is quasi-concave if −f is quasi-convex. It is well-known that a convex function is quasi-convex, and that a concave function is quasi-concave. (a) An ellipse

(b) A parabola

Lemma 2.2. If P is rational and f is quasi-concave on PI , then problem (2.2) can be solved in polynomial time. Proof. As P is rational, it follows by Section 4.2 in Hartmann [4] (see also remark (2.4) in Cook, Hartmann, Kannan, and McDiarmid [1]) that we can describe the integer hull PI of P in polynomial time. It is wellknown that a quasi-concave function takes on only higher values on convex combinations than it takes on the defining point with the smallest function value. As any integer point in P can be written as a convex combination of integer points in the boundary of PI , it follows that the minimum is achieved at an integer point in the boundary of PI . Since it is possible to optimize in polynomial time the polynomial f over the integer points of every (at most) one-dimensional face of PI , the statement follows.  We are now prepared to give the proof of Theorem 1.1 under the assumption that ∆ ≤ 0. 3

Proof of Theorem 1.1 for ∆ ≤ 0

First assume a = c = 0. As b2 ≤ 4ac, it follows that b = 0. Then f is linear, and by Lenstra [6] we can solve Figure 1: Classification of conic sections problem (2.2) in polynomial time. We now assume that a and c are not both zero, wlog a 6= 0. Then (2.3) can be rewritten as follows: Otherwise avx2 + bvx vy + cvy2 = 0. Then  b 2 −∆ 2 f (¯ z +λv)−f (¯ z ) = λ((2a¯ x +b¯ y +d)vx +(b¯ x +2c¯ y +e)vy ). (3.5) y + dx + ey. f (x, y) = a x + y + 2a 4a If (2a¯ x + b¯ y + d)vx + (b¯ x + 2c¯ y + e)vy < 0, then limλ→∞ f (¯ z + λv) = −∞, again contradicting that If a > 0, it follows that −∆/4a ≥ 0. From the problem (2.2) is bounded. Hence (2a¯ x + b¯ y + d)vx + representation of f by means of equation (3.5) it follows (b¯ x + 2c¯ y + e)vy ≥ 0.  that f is a convex function. Thus by Khachiyan and Porkolab [5] we can solve problem (2.2) in polynomial Important ingredients of the proof of Theorem 1.1 time. Otherwise, a < 0. Then −∆/4a ≤ 0 and so the are the special cases where f is linear, convex or representation of f by means of equation (3.5) proves concave. In the case where f is a convex function, that f is a concave function. Hence by Lemma 2.2 we then problem (2.2) can be solved in polynomial time, for can solve problem (2.2) in polynomial time. (c) A hyperbola

example by Theorem 1.2 in Khachiyan and Porkolab [5]. Note that their result is actually much more general, and later on we will make use of such generality. In the case where f is a concave function, then the next lemma provides us with an algorithm to solve the optimization problem (2.2).

4

Proof of Theorem 1.1 for ∆ > 0

In the remainder of the paper we assume ∆ > 0. As a first step we establish a canonic representation of the quadratic function f based on its degenerate section. From this representation several properties follow.

A canonic representation of f . We define the value γ¯ as follows: ( if a = c = 0, − de (4.6) γ¯ = ae2b+cd2 −bde otherwise. ∆

The following three lemmas give structural properties that will be main tools to solve a quadratic integer optimization problem in polynomial time. We illustrate Lemma 4.1 and Lemma 4.2 in Figure 2.

First assume a = c = 0. As b2 > 4ac, it follows that b 6= 0. In this case (2.3) can be rewritten in the form:  e  d (4.7) b x+ y+ + γ¯ . b b Otherwise, a and c are not both zero, wlog a 6= 0. In this case (2.3) can be rewritten in the form:  bd − 2ae 2 b d 2 ∆  y+ (4.8) a x + y + − + γ¯ . 2a 2a 4a ∆ The important property of the number γ¯ is that the conic section f (x, y) = γ¯ is degenerate. This allows us to decompose R2 into four subregions each of which will be studied separately. The cases a > 0 and a < 0 are analogous, thus we will from now on assume a ≥ 0 in order to streamline the presentation. The analogy of the two cases a > 0 and a < 0 follows from the representation of f by means of (4.8) where either a > 0 ∆ and − 4a < 0, or conversely. Next we define linear functions `1 , `2 follows. ( by + d 1 √ ` (x, y) = √ 2ax + (b + ∆)y + d + bd−2ae ∆ ( x + e/b √ `2 (x, y) = √ 2ax + (b − ∆)y + d − bd−2ae ∆

: R2 → R as (b) Lines L1 and L2 with some level sets

if a = c = 0 if a > 0, if a = c = 0 if a > 0.

Consider the two distinct lines L1 = {(x, y) ∈ R2 : `1 (x, y) = 0}, 2

2

(a) Lines L1 and L2

2

L = {(x, y) ∈ R : ` (x, y) = 0}, and the four translated cones C 1 = {(x, y) ∈ R2 : `1 (x, y) ≥ 0, `2 (x, y) ≤ 0},

Figure 2: Illustration of Lemma 4.1 and Lemma 4.2 Lemma 4.1. f (x, y) = γ¯ for the points in L1 ∪ L2 , f (x, y) < γ¯ for the points in (int C 1 ) ∪ (int C 2 ), and f (x, y) > γ¯ for the points in (int C 3 ) ∪ (int C 4 ). Proof. For a = c = 0, this follows easily from (4.7), thus we now assume a > 0. It follows from (4.8) that, for ◦ ∈ {=, <, >}, the points satisfying f (x, y)◦¯ γ are the points that satisfy the following equivalent conditions

 b d 2 ∆  bd − 2ae 2 a x + y + ◦ y + , 2a 2a 4a ∆ C 2 = {(x, y) ∈ R2 : `1 (x, y) ≤ 0, `2 (x, y) ≥ 0}, r √  C 3 = {(x, y) ∈ R2 : `1 (x, y) ≥ 0, `2 (x, y) ≥ 0}, b d  ∆  bd − 2ae  y+ a x+ y+ ◦ , 4 2 1 2 2a 2a 4a ∆ C = {(x, y) ∈ R : ` (x, y) ≤ 0, ` (x, y) ≤ 0}. √  bd − 2ae  Note that, if a = 0 all such regions are rational, 2ax + by + d ◦ ∆ y + . ∆ while if a > 0 they might be not rational. However, if 1 2 a > 0, then the following rational inequalities will be Thus f (x, y) = γ¯ for the points in L ∪ L , f (x, y) < γ¯ 1 2 1 2 for the points in C ∪ C \ (L ∪ L ), and f (x, y) > γ¯ useful later. for the points in C 3 ∪ C 4 \ (L1 ∪ L2 ).  Remark 4.1. If a > 0, then ∆y + bd − 2ae ≥ 0 is valid for C 1 , ∆y+bd−2ae ≤ 0 is valid for C 2 , 2ax+by+d ≥ 0 Lemma 4.2. For every γ ≤ γ¯ , the set of points that is valid for C 3 , and 2ax + by + d ≤ 0 is valid for C 4 . satisfy f (x, y) ≤ γ is the union of two convex sets, one

contained in C 1 , and the other contained in C 2 . For Lemma 4.3. For γ ∈ Q with γ ≤ γ¯ , and for i = 1, 2, every γ ≥ γ¯ , the set of points that satisfy f (x, y) ≥ γ is the set {(x, y) ∈ C i : f (x, y) ≤ γ}, can be described by the union of two convex sets, one contained in C 3 , and means of polynomial rational inequalities. the other contained in C 4 . Proof. We prove the statement for i = 1, the case i = 2 1 Proof. For γ = γ¯ this follows from Lemma 4.1, thus we being symmetric. Let γ ≤ γ¯ , and F = {(x, y) ∈ C : 1 now assume γ 6= γ¯ . Let γ > γ¯ . By Lemma 4.1, the f (x, y) ≤ γ}. If a = 0, then the polyhedron C is set of points that satisfy f (x, y) ≥ γ is contained in described by rational linear inequalities, therefore the set F is defined by polynomial rational inequalities. (int C 3 ) ∪ (int C 4 ). We show that the set of points If a > 0, then the difficulty arises that the polyheW = {(x, y) ∈ int C 3 : f (x, y) ≥ γ} dron C 1 might be not rational. In this case we make use of Remark 4.1. The inequality ∆y + bd − 2ae ≥ 0 is is convex. The other cases follow in a similar way. valid for C 1 , while inequality ∆y + bd − 2ae ≤ 0 is valid 3 First assume a = c = 0. Since int C = {(x, y) ∈ for C 2 . It follows that 2 R : by + d > 0, x + e/b > 0}, by (4.7) we have F = {(x, y) ∈ R2 : ∆y + bd − 2ae ≥ 0, f (x, y) ≤ γ}.   o n  e by + d ≥ γ − γ¯ W = (x, y) ∈ int C 3 : x + This trick allows us to again describe the convex set F b n o by means of polynomial rational inequalities.  3 = (x, y) ∈ int C : g(x) − h(y) ≥ 0 , 4.1 Bounded case We are now prepared to complete the proof of Theorem 1.1 under the assumption that problem (2.2) is bounded. Later in Section 4.2 ∂2g 1 e −3/2 we will show how to check in polynomial time if prob(x) = − x + < 0, 2 lem (2.2) is bounded or not. ∂x 4 b   2 −5/2 Note that by Lenstra [6] we can check in polynomial √ ∂ h 3 (y) = b2 γ − γ¯ by + d ≥ 0, time if P contains integer points, thus from now on we 2 ∂y 4 will assume that P contains an integer point, i.e. PI 6= ∅. the function g(x) is concave, and the function h(y) is Let convex, thus the function g(x) − h(y) is concave, which P i = P ∩ C i , for i = 1, . . . , 4. shows that W is convex. We begin investigating the two polyhedra P 1 and P 2 . Now assume a > 0. By (4.8), W is the set of points More precisely, for i = 1, 2, we will solve the problem (x, y) in C 3 that satisfy min{f (x, y) : (x, y) ∈ P i ∩ Z2 }, i.e. we will either show  that P i ∩ Z2 = ∅, or we will find an integer point in P i b d 2 ∆ bd − 2ae 2 a x+ y+ ≥ y+ + γ − γ¯ . that minimizes f . The following observation is crucial. 2a 2a 4a ∆ If there is at least an integer point in P 1 ∪ P 2 , then b d Since by Remark 4.1 the inequality x + 2a y + 2a ≥ 0 is we do not need to consider integer points in P 3 ∪ P 4 , since by Lemma 4.1 they have higher objective value. It valid for C 3 , we have will remain then to analyze the situation if there are no n o √  b d 1 2 i 3 W = (x, y) ∈ C : a x + y + − g(y) ≥ 0 , integer points in P ∪ P . Then for each polyhedron P , i 2a 2a i = 3, 4, we will first construct a relaxation of P in form r  of a rational polyhedron using Remark 4.1. This will 2 ∆ bd−2ae allow us to solve the problems min{f (x, y) : (x, y) ∈ where g(y) = 4a y + ∆ + γ − γ¯ . Since P i ∩ Z2 }, for i = 3, 4. Polyhedra P 1 and P 2 . Let γ ∈ Q with γ ≤ γ¯ . γ − γ¯ ∂2g (y) = q  By Lemma 4.2, the set {(x, y) ∈ P 1 : f (x, y) ≤ γ} is 2 3/2 > 0, ∂y 2 4a(γ−¯ γ) ∆ bd−2ae convex. Since P 1 = P ∩ C 1 and P is rational, it follows y+ ∆ + 4a ∆ by Lemma 4.3 that the set {(x, y) ∈ P 1 : f (x, y) ≤ √ the function g(y) is convex, thus the function a(x + γ} can be described by means of polynomial rational d b inequalities. Hence, by Khachiyan and Porkolab [5], 2a y + 2a ) − g(y) is concave, which shows that the set W is convex.  we can solve in polynomial time the integer feasibility problem for the set {(x, y) ∈ P 1 : f (x, y) ≤ γ}, i.e. we Note that Lemma 4.2 implies that f is quasi-convex on can test in polynomial time whether such set contains C 1 and on C 2 , and is quasi-concave on C 3 and on C 4 . or not integer points. where g(x) =

p

x + eb , and h(y) =

q

γ−¯ γ by+d .

Since

Using the algorithm of Khachiyan and Porkolab as described above in combination with a binary search allows us then to solve the problem min{f (x, y) : (x, y) ∈ P 1 ∩ Z2 }. For this to work we derive an upper bound and a lower bound on the range of values {f (x, y) : (x, y) ∈ P 1 ∩ Z2 }. The upper bound follows from Lemma 4.1, since for every point (x, y) ∈ P 1 , f (x, y) ≤ γ¯ . We now explain how to calculate a lower bound. First we show that if the polyhedron P 1 is not bounded, then f (x, y) ≥ γ¯ for every (x, y) ∈ P 1 ∩ Z2 . Since P 1 is not bounded, rec P 1 contains a nonzero vector. rec P 1 is the intersection of rational cone rec P with full-dimensional cone rec C 1 , therefore rec P 1 contains a nonzero integer vector. Since P contains at least an integer point and problem (2.2) is bounded, Lemma 2.1 implies that for every nonzero integer vector v in rec P , either avx2 + bvx vy + cvy2 ≥ 1, or avx2 + bvx vy + cvy2 = 0 and for every (x, y) ∈ Z2 ∩ P , (2ax + by + d)vx + (bx + 2cy + e)vy ≥ 0. By Lemma 4.1 applied to g(x, y) = ax2 + bxy + cy 2 , this means that for every nonzero integer v in rec P 1 , either v ∈ rec C 3 and inequality `2 (x, y) ≥ 0 is valid for P ∩ Z2 , or v ∈ rec C 4 and inequality `1 (x, y) ≤ 0 is valid for P ∩Z2 . Since rec P 1 contains a nonzero integer vector, such vector can be either in rec C 3 or in rec C 4 . Assume that it is in rec C 3 (if it is in rec C 4 the argument is symmetric). Then `2 (x, y) ≥ 0 is valid for P ∩ Z2 , therefore f (x, y) ≥ γ¯ for every (x, y) ∈ P 1 ∩ Z2 . We now explain how to get a lower bound on {f (x, y) : (x, y) ∈ P 1 ∩ Z2 } in the case where P 1 is bounded. Let lx = min{x : (x, y) ∈ P 1 ∩ Z2 }, ux = max{x : (x, y) ∈ P 1 ∩ Z2 }, ly = min{y : (x, y) ∈ P 1 ∩ Z2 }, uy = max{y : (x, y) ∈ P 1 ∩ Z2 }. All these optimization problems have a finite optimum since P 1 is bounded. Moreover they can be solved in polynomial time using the algorithm by Khachiyan and Porkolab since we are optimizing linear functions over the integer points in the set P 1 , which is convex and can be described by means of polynomial rational inequalities by Lemma 4.3. In order to get a lower bound on the objective function value, then one can bound separately the monomials in (2.3): for every (x, y) ∈ P 1 ∩ Z2 , ax2 ≥ min{0, alx2 , au2x }, bxy ≥ min{blx ly , blx uy , bux ly , bux uy }, cy 2 ≥ min{0, cly2 , cu2y }, dx ≥ min{dlx , dux }, ey ≥ min{ely , euy }. Therefore, by using the polynomial time algorithm by Khachiyan and Porkolab in a binary search setting, we can solve the problem min{f (x, y) : (x, y) ∈ P 1 ∩ Z2 }. The problem min{f (x, y) : (x, y) ∈ P 2 ∩ Z2 } is symmetric, thus it can be solved in the same way. If there is an integer point in P 1 ∪ P 2 , then by Lemma 4.1 we have solved problem (2.2), and we are

done. Thus we now assume that there is no integer point in P 1 ∪ P 2 . Polyhedra P 3 and P 4 . We show how to solve the problem min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in polynomial time. By Lemma 4.2, f is quasi-concave on C 3 , and so also on P 3 and PI3 . If a = 0, the polyhedron P 3 is rational, thus by applying Lemma 2.2 to P 3 we can solve the problem min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in polynomial time. If a > 0, then P 3 might be not rational. In this case we construct a rational polyhedron P˜ 3 such that P˜I3 = PI3 . This can be achieved with the help of Remark 4.1. Set P˜ 3 = {(x, y) ∈ P : 2ax + by + d ≥ 0}. Since 2ax + by + d ≤ 0 is valid for P 4 and the polyhedra P 1 and P 2 contain no integer points, we have the property that P˜I3 = PI3 . Since the function f is quasiconcave on P˜I3 , and since P˜ 3 is a rational polyhedron, by applying Lemma 2.2 to P˜ 3 we can solve the problem min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in polynomial time. The same argument applies to solve the problem min{f (x, y) : (x, y) ∈ P 4 ∩ Z2 } in polynomial time. This completes the proof of Theorem 1.1 under the assumption that problem (2.2) is bounded. 4.2 Unboundedness We now drop the assumption that problem (2.2) is bounded, and we show how to check in polynomial time if it is bounded or not. Note that by Lenstra we can solve the integer feasibility problem for P , thus we now assume that P contains an integer point. The following lemma characterizes when problem (2.2) is unbounded. Lemma 4.4. Assume that P contains an integer point and that ∆ > 0. Then problem (2.2) is unbounded if and only if there exists a nonzero vector v¯ in Z2 ∩ rec P vx2 + vx v¯y + c¯ vy2 ≤ −1, or a¯ such that either a¯ vx2 + b¯ 2 2 b¯ vx v¯y + c¯ vy = 0 and there exists (¯ x, y¯) ∈ Z ∩ P such that (2a¯ x + b¯ y + d)¯ vx + (b¯ x + 2c¯ y + e)¯ vy ≤ −1. Proof. The sufficiency of the condition follows by Lemma 2.1. To show the necessity of the condition, assume that problem (2.2) is unbounded. Therefore there exist infinitely many distinct integer points in P with objective value strictly smaller than γ¯ . Since by Lemma 4.1 all the integer points with objective value strictly smaller than γ¯ are contained either in int P 1 or in int P 2 , then at least one among int P 1 and int P 2 contains infinitely many integer points. Wlog int P 1 contains infinitely many integer points. Therefore PI1 and P 1 are unbounded polyhedra. Since P 1 is unbounded, rec P 1 6= {0}. rec P 1 is the intersection of rational cone rec P with fulldimensional cone rec C 1 , therefore rec P 1 contains a nonzero integer vector v¯. Since v¯ ∈ rec P 1 , it follows by

Lemma 4.1 (applied to g(x, y) = ax2 + bxy + cy 2 ) that a¯ vx2 + b¯ vx v¯y + c¯ vy2 ≤ 0. If a¯ vx2 + b¯ vx v¯y + c¯ vy2 < 0, then the 2 integrality of a, b, c, v¯ implies a¯ vx + b¯ vx v¯y + c¯ vy2 ≤ −1 2 vx v¯y + c¯ vy2 = 0. and we are done, thus assume a¯ vx + b¯ Again by Lemma 4.1 this implies that v¯ lies on the boundary of rec P 1 , therefore it lies in either rec L1 or in rec L2 , wlog in rec L1 . Since there exists an integer point (¯ x, y¯) in int P 1 ⊆ int C 1 , we have that (2a¯ x + b¯ y + d)¯ vx + (b¯ x + 2c¯ y + e)¯ vy < 0, and by the integrality of a, b, c, d, e, v¯, x ¯, y¯, we have (2a¯ x+b¯ y +d)¯ vx + (b¯ x + 2c¯ y + e)¯ vy ≤ −1.  It can be shown that Lemma 4.4 is valid also if the assumption ∆ > 0 is dropped. However we do not need such statement since we have already fully solved the case ∆ ≤ 0 with other methods in Section 3. We now show how to use Lemma 4.4 to deduce in polynomial time if problem (2.2) is bounded or unbounded. Fist we need to check in polynomial time if there exists a nonzero vector v¯ in Z2 ∩ rec P such that vx v¯y + c¯ vy2 ≤ −1. By Lemma 4.2 applied a¯ vx2 + b¯ 2 to g(x, y) = ax + bxy + cy 2 , this amounts to solve the integer feasibility problem for the two convex sets (rec P )∩{(x, y) ∈ rec C i : ax2 +bxy+cy 2 ≤ −1}, for i = 1, 2. Since rec P is rational, it follows by Lemma 4.3 that each set (rec P ) ∩ {(x, y) ∈ rec C i : ax2 + bxy + cy 2 ≤ −1} can be described by means of polynomial rational inequalities. Hence, by Khachiyan and Porkolab, we can solve the integer feasibility problem for each set. Now we show how to find all nonzero vectors v in Z2 ∩ rec P that satisfy avx2 + bvx vy + cvy2 = 0. By Lemma 4.1 applied to g(x, y) = ax2 + bxy + cy 2 , g(x, y) = 0 for the points in (rec L1 ) ∪ (rec L2 ). The lines rec L1 and rec L2 contain an integer nonzero vector if and only if they are rational, and this happens if a = c = 0, or if a > 0 and ∆ is a square. (We again omit the case a < 0 since it is analogous to the case a > 0.) If the lines rec L1 and rec L2 are not rational, then there are no nonzero integer vectors that satisfy g(x, y) = 0. Otherwise, if rec L1 and rec L2 are rational, then there are only four nonzero integer vectors (up to multiplication by a scalar) that satisfy g(x, y) = 0 and we can find them in polynomial time. Let (¯ vx , v¯y ) be a vector such that a¯ vx2 +b¯ vx v¯y +c¯ vy2 = 0. By Lenstra [6], we can now check in polynomial time if there exists an integer point in Z2 ∩P that satisfies the linear inequality (2ax+by+d)¯ vx +(bx+2cy+e)¯ vy ≤ −1. References [1] W. Cook, M. Hartman, R. Kannan, and C. McDiarmid. On integer points in polyhedra. Combinatorica, 12(1):27–37, 1992.

[2] J.A. De Loera, R. Hemmecke, M. K¨ oppe, and R. Weismantel. Integer polynomial optimization in fixed dimension. Mathematics Of Operations Research, 31(1):147–153, 2006. [3] M.R. Garey and D.S. Johnson. Computers and Intractability: A Guide to the Theory of NPCompleteness. Freeman, San Francisco, 1979. [4] M. Hartmann. Cutting planes and the complexity of the integer hull. Technical Report 819, School of Operations Research and Industrial Engineering, Cornell University, 1989. [5] L. Khachiyan and L. Porkolab. Integer optimization on convex semialgebraic sets. Discrete & Computational Geometry, 23(2):207–224, 2000. [6] H.W. Jr. Lenstra. Integer programming with a fixed number of variables. Mathematics of Operations Research, 8(4):538–548, 1983. [7] H.W. Jr. Lenstra. Solving the Pell equation. Notices of the AMS, 49(2):182–192, 2002. [8] K. Manders and L. Adleman. NP-complete decision problems for binary quadratics. Journal of Computer and System Sciences, 16:168–184, 1978. [9] R.R. Meyer. On the existence of optimal solutions to integer and mixed-integer programming problems. Mathematical Programming, 7(1):223–235, 1974. [10] A. Schrijver. Theory of Linear and Integer Programming. Wiley, Chichester, 1986.