Criticality of ergodic type HJB equations and stochastic ergodic control Naoyuki Ichihara∗ Graduate School of Engineering Hiroshima University

Abstract The aim of this note is to give a summary of [5]. We study HamiltonJacobi-Bellman (HJB) equations of ergodic type associated with some stochastic ergodic control problems. We prove that the optimal value of the stochastic control problem coincides with the generalized principal eigenvalue of the corresponding HJB equation. The results can be regarded as a nonlinear extension of the criticality theory for linear Schr¨odinger operators with decaying potentials.

1

Introduction and Main results

In this note we consider the following minimization problem with real parameter β: [∫ T { } ] 1 |ξt |2 ξ Minimize Jβ (ξ) := lim sup E − βV (Xt ) dt , T →∞ T 2c(Xtξ ) 0 (1) ∫ t ξ subject to Xt = x − ξs ds + Wt , t ≥ 0, 0

where W = (Wt ) is an N -dimensional standard Brownian motion defined on some filtered probability space (Ω, F, P ; (Ft )), and ξ = (ξt ) stands for an RN -valued (Ft )progressively measurable process belonging to the admissible class A defined by A := {ξ : [0, ∞) × Ω → RN | ess-sup[0,T ]×Ω |ξt | < ∞ for all T > 0}. ∗

E-mail: [email protected]. Supported in part by JSPS KAKENHI Grant Number 24740089.

1

We assume throughout this note that c and V satisfy the following properties: (H1) c ∈ Cb2 (RN ) and κ ≤ c ≤ κ−1 in RN for some κ > 0. (H2) V ∈ Cb2 (RN ), V ≥ 0 in RN , V 6≡ 0, and |x|2 V (x) → 0 as |x| → ∞. Here, Cb2 (RN ) denotes the set of C 2 -functions f on RN such that f and its first and second derivatives are bounded on RN . We are interested in characterizing the optimal value Λ(β) := inf ξ∈A Jβ (ξ) as well as the optimal control of (1) in terms of the associated partial differential equation. More specifically, we consider the following HJB equation of ergodic type: 1 1 λ − ∆φ + c(x)|Dφ|2 + βV (x) = 0 in RN . 2 2 The unknown of (EP) is the pair (λ, φ) ∈ R × C 2 (RN ). We now set λ∗ := sup{λ | (EP) has a C 2 -subsolution φ}.

(EP)

(2)

Then the following theorem holds. Theorem 1.1 (Theorem 2.1 of [5]). Let (H1) and (H2) hold. Then λ∗ is well-defined and finite. Moreover, (EP) has a solution φ ∈ C 2 (RN ) if and only if λ ≤ λ∗ . We call λ∗ the generalized principal eigenvalue of (EP). Note that the value of λ∗ depends on β. The next theorem concerns qualitative properties of λ∗ (β) with respect to β. Theorem 1.2. Let (H1) and (H2) hold. Let λ∗ = λ∗ (β) be the generalized principal eigenvalue of (EP). (i) The mapping β 7→ λ∗ (β) is non-positive, non-increasing, and concave. (ii) There exists a βc ≥ 0 such that λ∗ (β) = 0 for β ≤ βc and λ∗ (β) < 0 for β > βc . (iii) βc = 0 for N ≤ 2 and βc > 0 for N ≥ 3. (iv) λ∗ (β) = Λ(β) for all β. We next consider the “ground state” of (EP), namely, a solution φ of the equation 1 1 λ∗ − ∆φ + c(x)|Dφ|2 + βV (x) = 0 in RN . 2 2

(EP∗ )

Theorem 1.3. Let (H1) and (H2) hold. Let βc be the constant given in Theorem 1.2. (i) For any β ≥ βc , there exists at most one solution φ ∈ C 2 (RN ) of (EP∗ ) up to an additive constant. 2

(ii) Suppose that β > βc . Then, there exists a C > 0 such that the solution φ of (EP∗ ) satisfies C −1 |x| − C ≤ φ(x) ≤ C(1 + |x|), x ∈ RN . (iii) Suppose that β = βc . Then, there exists a C > 0 such that the solution φ of (EP∗ ) satisfies C −1 log(1 + |x|) − C ≤ φ(x) ≤ C log(1 + |x|) + C,

x ∈ RN .

Theorem 1.3 plays a key role in constructing the optimal control of the stochastic ergodic control (1). Theorem 1.4. Assume (H1) and (H2). Let φ = φ(x) be a solution of (EP∗ ), and let X = (Xt ) be the diffusion process governed by the stochastic differential equation dXt = −c(Xt )Dφ(Xt ) dt + dWt ,

X0 = x.

(3)

(i) X is transient for β < βc , positive recurrent for β > βc , and recurrent for β = βc . (ii) Set ξt∗ := c(Xt )Dφ(Xt ). Then λ∗ (β) = Jβ (ξ ∗ ) for all β ≥ βc , and λ∗ (β) = Jβ (0) for all β ≤ βc . In other words, ξ ∗ is an optimal control provided β ≥ βc . The proof of Theorem 1.4 relies on the so-called Lyapunov method, which allows one to link the recurrence and transience of X to the asymptotic behavior as |x| → ∞ of the solution φ of (EP∗ ). We refer to Section 4 of [5] for details (see also [1, 3, 4, 6, 7, 12]).

2

Criticality

In this section we discuss a relationship between Theorem 1.4 and the criticality theory for linear Schr¨odinger operators. Throughout this section, we assume that c ≡ 1. In such a special case, (EP) can be written as 1 1 λ − ∆φ + |Dφ|2 + βV (x) = 0 in RN . 2 2

(4)

Let (λ, φ) be a solution of (4), and set h := e−φ (this transformation is called the Cole-Hopf transform). Then h is a positive solution of the stationary problem −Lh = λh in RN ,

3

1 L := ∆ + βV. 2

(5)

Let σ(−L) denote the spectrum of the self-adjoint extension of −L in L2 (RN ). Then we have λ∗ = sup{λ | (4) has a solution φ} = sup{λ | (5) has a positive solution h} = inf{z | z ∈ σ(−L)}. This observation allows one to extend the notion of principal eigenvalue to the nonlinear equation (EP). We now explain the connection between Theorem 1.4 and the classical criticality theory for Schr¨odinger operators. Let us consider the elliptic equation (L + λ∗ )h = 0 in RN ,

1 L := ∆ + βV. 2

(6)

Then, in view of the criticality theory for linear operators (see [2, 8, 9, 10, 11, 12, 13]), we see that L + λ∗ is critical for β ≥ βc and subcritical for β < βc . Recall that L + λ∗ is called subcritical if there exists a Green function of L + λ∗ , and called critical if there is no Green function of L + λ∗ but (6) has a positive solution. From the probabilistic point of view, the notions of criticality and subcriticality are equivalent to the recurrence and transience of Doob’s h-transformed process, respectively. Here, Doob’s h-transformed process is defined as a diffusion process whose infinitesimal generator is given by Lh + λ∗ , where Lh denotes the h-transform of L: 1 1 Dh Lh Lh f := L(hf ) = ∆f + · Df + f, f ∈ C 2 (RN ). h 2 h h We point out that Doob’s h-transformed process coincides with the feedback diffusion X governed by (3) provided c ≡ 1. Indeed, set φ := − log h. Then, by the definitions of h and Lh , we see that 1 Dh 1 L h + λ∗ = ∆ + · D = ∆ − Dφ · D, 2 h 2 which coincides with the infinitesimal generator of the feedback diffusion (3) with c ≡ 1. In this sense, Theorem 1.4 can be regarded as a nonlinear extension of the criticality theory in terms of the stochastic optimal control. We close this section by mentioning a connection between the stochastic ergodic conrtol (1) and the finite time horizon problem. Let us consider the minimizing problem [∫ T { } ] 1 2 ξ |ξt | − βV (Xt ) dt , Minimize Jβ (ξ; T, x) := E 2 0 (7) ∫ t ξ subject to Xt = x − ξs ds + Wt , t ≥ 0. 0

4

Then the value function uβ (T, x) := inf ξ∈A Jβ (ξ; T, x) of (7) turns out to be the unique classical solution to the Cauchy problem   ∂u − 1 ∆u + 1 |Dφ|2 + βV = 0 in (0, ∞) × RN , ∂t 2 2 (CP) u(0, · ) = 0 in RN . We now take the Cole-Hopf transform v := e−u . Then v satisfies the linear equation   ∂v − 1 ∆v − βV v = 0 in (0, ∞) × RN , ∂t 2 v(0, · ) = 1 in RN . In order to guess the long-time behavior of v, and therefore u, we apply the formal eigenfunction expansion: v(T, · ) =

∞ ∑

e−λi T (1, hi )hi ,

λi ∈ R,

hi ∈ L2 (RN ),

(8)

i=1

∫ where (1, h) := RN h(x)dx, and (λi , hi ) (i = 1, 2, . . . ) denote the pairs of eigenvalues and eigenfunctions of −L. Suppose furthermore that λ1 < λ2 ≤ λ3 ≤ · · · . Then we have (∞ ) ∑ u(T, · ) 1 = − log e−λi T (1, hi )hi −→ λ1 as T → ∞. T T i=1 On the other hand, we also see that Λ(β) = inf lim sup ξ∈A

T →∞

Jβ (ξ; T, x) Jβ (ξ; T, x) uβ (T, x) ≥ lim sup inf = lim sup . T T T T →∞ ξ∈A T →∞

Hence, if the inequality above can be replace by an equality, we obtain uβ (T, x) = λ∗ (β). T →∞ T

Λ(β) = lim

(9)

Although the formal expansion (8) is not valid in our setting, the equalities (9) hold true under (H1) and (H2). See Section 7 of [5] for details.

References [1] R. N. Bhattacharya, Criteria for recurrence and existence of invariant measures for multidimensional diffusions, Ann. Probab. 6 (1978) 541-553.

5

[2] E. B. Davies, Spectral theory and differential operators, Cambridge Studies in Advanced Mathematics, 42. Cambridge, 1995. [3] A. Friedman, Stochastic Differential Equations and Applications, Vol.1, Academic Press, 1975. [4] N. Ichihara, Recurrence and transience of optimal feedback processes associated with Bellman equations of ergodic type, SIAM J. Control Optim. 49 (2011) 1938-1960. [5] N. Ichihara, Criticality of viscous Hamilton-Jacobi equations and stochastic ergodic control, to appear in J. Math. Pures Appl. (DOI:10.1016/j.matpur.2013.01.005). [6] R.Z. Khasminskii, Ergodic properties of recurrent diffusion processes and stabilization of the problem to the Cauchy problem for parabolic equations, Theor. Probab. Appl. 5 (1960) 179-196. [7] R.Z. Khasminskii, Stochastic Stability of Differential Equations, 2nd edition, Stochastic Modelling and Applied Probability 66, Springer, 2012. [8] M. Murata, Structure of positive solutions to (−∆ + V )u = 0 in Rn , Duke Math. J. 53 (1986) 869-943. [9] Y. Pinchover, On positive solutions of second order elliptic equations, stability results and classification, Duke Math. J. 57 (1988) 955-980. [10] Y. Pinchover, Criticality and ground states for second order elliptic equations, J. Differential Equations 80 (1989) 237-250. [11] Y. Pinchover, On criticality and ground states for second order elliptic equations, II, J. Differential Equations 87 (1990) 353-364. [12] R.G. Pinsky, Positive Harmonic Functions and Diffusion, Cambridge studies in advanced mathematics 45, 1995. [13] B. Simon, Large time behavior of the Lp norm of Schr¨odinger semigroups, J. Func. Anal. 40 (1981) 66-83.

6