PHYSICAL REVIEW E, VOLUME 63, 011107

General measures for signal-noise separation in nonlinear dynamical systems J. W. C. Robinson* Defence Research Establishment, SE 172 90 Stockholm, Sweden

J. Rung† Department of Quantum Chemistry, Uppsala University, SE 751 20 Uppsala, Sweden

A. R. Bulsara‡ and M. E. Inchiosa§ Space and Naval Warfare Systems Center, Code D364, San Diego, California 92152-5001 共Received 2 June 2000; published 22 December 2000兲 We propose the ␾ divergences from statistics and information theory 共IT兲 as a set of separation indices between signal and noise in stochastic nonlinear dynamical systems 共SNDS兲. The ␾ divergences provide a more informative alternative to the signal-to-noise ratio 共SNR兲 and have the advantage of being applicable to virtually any kind of stochastic system. Moreover, ␾ divergences are intimately connected to various fundamental limits in IT. Using the properties of ␾ divergences, we show that the classical stochastic resonance 共SR兲 curve can be interpreted as the performance of a nonoptimal, or mismatched, detector applied to the output of a SNDS. Indeed, for a prototype double-well system with forcing in the form of white Gaussian noise plus a possible embedded signal, the whole information loss can be attributed to this mismatch; an optimal detection procedure 共for the signal兲 gives the same performance when based on the output as when based on the input of the system. More generally, it follows that, when characterizing signal-noise separation 共or system performance兲 of SNDS in terms of criteria that do not correspond to IT limits, the choice of criterion can be crucial. The indicated figure of merit will then not be universal and will be relevant only to some family of applications, such as the classical 共narrow-band SNR兲 SR criterion, which is relevant for narrow-band post processing. We illustrate the theory using simple SNDS excited by both wide- and narrow-band signals; however, we stress that the results are applicable to a much larger class of signals and systems. DOI: 10.1103/PhysRevE.63.011107

PACS number共s兲: 05.40.⫺a, 02.50.Ey, 47.20.Ky, 85.25.Dq

I. INTRODUCTION

One of the most common indices of signal-to-noise separation for narrow-band signals in noise is the signal-to-noise ratio 共SNR兲 expressed in the 共Fourier兲 spectral domain. There are several reasons for this, one being the simplicity in definition and computation, another the fact that, for the canonical case of a time-sinusoidal signal with random initial phase in Gaussian noise, the SNR immediately gives the optimal performance figures for several standard detection/ estimation problems 关1兴. For example, the maximal achievable probability of detection of the signal can in this case, for any fixed false alarm probability, be written as a function of SNR 共Marcum’s Q function兲. This intimate connection between the SNR and fundamental performance bounds can be attributed to the fact that the whole statistical structure of the process in this case is captured by the power spectrum 共Fourier transform of the autocovariance function of the process兲 关2兴. However, if a process of this type is passed through a nonlinear system, the output is no longer Gaussian and the spectrum of the output process will no longer represent the entire statistical structure of the process. Thus, for the output there is a choice between using as a signal-noise separation

*Email address: [email protected] †

Email address: [email protected] Email address: [email protected] § Email address: [email protected] ‡

1063-651X/2000/63共1兲/011107共11兲/$15.00

index 共SI兲 the SNR, which is simple to compute but which discards some statistical information, or turning to other SIs that retain the relevant information but might be more difficult to compute 关3兴. A similar tradeoff situation arises if one considers, instead of the SNR, other output SIs which, like the SNR, might be blind to certain parts of the statistical structure of the process but still are easy to compute, such as the deflections described below. Regardless of what type of index of separation between signal and noise one chooses, it will always reflect 共well兲 only one or a few aspects of the total behavior of the observed process. This is true even if one considers SIs that correspond to limits 共bounds兲 in statistics and information theory 共IT兲, such as the ␾ divergences employed below. In other words, no SI can serve all purposes and it is therefore imperative that one, in a given situation, clarify exactly what performance aspect or intended use one is interested in. Examples of objectives inherent in many applications include detection/hypothesis testing, classification, estimation, and communication, but others of a more phenomenological nature, such as similarity 共e.g., various form of correlation兲 between in/output 共signals兲 are also common. A field where questions of this type have recently elicited considerable interest is stochastic resonance 共SR兲 关4兴. In SR, the most commonly used SIs have traditionally been the output SNR and the spectral amplification 共change in spectral power兲, both usually measured in the output power spectrum at the input signal frequency 共or a harmonic thereof兲. The hallmark of SR has been, conventionally, the existence of a

63 011107-1

©2000 The American Physical Society

ROBINSON, RUNG, BULSARA, AND INCHIOSA

PHYSICAL REVIEW E 63 011107

local maximum in the output SNR at some optimal 共input兲 noise strength 共predicated on the system and signal characteristics兲. The prevalence of 共narrow-band兲 SNR-type SIs can perhaps best be explained by historical example, since the first applications of SR involved enhancement of a sinusoidal signal by passage through a stochastic nonlinear dynamical system 共SNDS兲. In this setting it is natural to quantify performance in terms of a spectrum-based SI 共focusing on the presence of a component in the output with the same frequency as the exciting signal兲. Not surprisingly, since the inception of SR, investigations have been carried out to determine whether or not the effect 共or some variant of it兲 could be used to facilitate detection 关5兴 or information transfer 关6兴. This led naturally to consideration of other SIs that 共also for more general distributions of signal and noise兲 are more closely related to IT limits, such as probability of detection, false alarm, and error in detection settings 关5,7,8兴 and mutual information and channel capacity 关6,9兴 in communication settings. It has been shown that the channel capacity of simple binary channels can be enhanced by adding noise to the input. An intuitive way of explaining this is that, unlike in the case of a linear channel, adding noise changes the structure of the equivalent channel 共in a nontrivial way兲. The communication problem thus gets an additional dimension; that of optimizing not only the channel coding but also the channel itself. In the present work we generalize the formalism introduced in 关7兴 to SNDS with the focus on output-based SIs and the problems of detection/hypothesis testing. We introduce the ␾ divergences of Csisza´r- Ali-Silvey 关10,11兴 as a canonical class of SIs and give a general formula for the computation of ␾ divergences between the probability measures induced by the output of a SNDS over a time interval 关 0,T 兴 . Using this formula and basic properties of ␾ divergences, we present a bound for SNR in terms of one member of this family, the ␹ 2 divergence, and show why a large class of SR phenomena can be associated with the performance of suboptimal detectors. The optimal detectors for these cases would give a monotonically 共with input noise strength兲 decreasing performance, but always at least as good as the suboptimal ones. This can be used to qualitatively explain a number of observations previously made in the literature 共such as various forms of resonances兲 for other SIs as well, such as those related to Neyman-Pearson detection 共see, e.g., 关8兴兲. A main conclusion of this paper is therefore the following: from a 共mathematical兲 systems-theoretic perspective, 共classical兲 SR can in many instances be explained simply as the result of a mismatching of the detector to the particular shape the output distributions take for a certain input noise level, or, equivalently, as deficiencies in the SI used. 共It should be pointed out, though, that if a measurement noise floor is present, resonances can occur in the classical SR setting also for more fundamental SIs, such as ␾ divergences 关12兴.兲 This insight facilitates the use of much more general characterizations of the stochastic resonance effect that can be introduced and explained without reference to any of the internal properties of the system, e.g., the matching of time scales 共and the concomitant connection to a bona fide resonance 关13兴兲 in a periodically rocked potential, even though

such explanations can offer insights into mechanism of the occurrence of the resonance in specific signal-SNDS combinations. Generalized resonances of this type 共in the sense of local maximization of a SI兲 are known to occur also for other SIs and signals/systems and, since they can usually be realized at a critical value of the noise background, they bear a resemblance to conventional SR 关4兴. In the next section we define the type of SNDS and signals we will be working with, and we outline the scope of the results to follow. The main material is presented in Sec. III, where we address the problem of characterizing system performance in terms of general SIs. First, in Sec. III A, we review the concepts of likelihood ratio 共LR兲 and sufficient statistic, since these are central to the subsequent developments. 共The impatient reader can skip this section and proceed directly to Sec. III B.兲 The LR will play the role of an information-preserving data reduction of an observable related to a SNDS, provided information preserving is interpreted in a certain statistical sense which we clarify. Of particular importance is the formula for the LR based on observations of the whole 共state兲 trajectory of an SNDS represented by a stochastic differential equation 共SDE兲, which we recall and discuss. Then, in Sec. III B, we introduce the ␾ divergences as a general class of SIs for SNDS that are calculated as functionals of LRs and describe a few of their properties. The most important property of ␾ divergences that we single out can be interpreted 共loosely兲 as an analog of the second law of thermodynamics for closed systems: deterministic transformations of a noise-contaminated signal should not be able to increase the 共statistical兲 visibility of the signal in the noise. We also give a concrete formula for computation of ␾ divergences generated by SNDS described by SDEs in terms of the representation of the SDE. This formula is very important for the practical applications of the theory, in particular for numerical studies. In Sec. III C we then proceed to discuss some of the intimate relations between ␾ divergences and limits in statistical inference that exist and, with this material at hand, we explain in Sec. III D why classical SR can be interpreted as the performance of a suboptimal detector. In Sec. IV we illustrate the theoretical developments in the preceding sections with numerical simulations, using a double-well-type SNDS for a number of different signals and SIs, and discuss the results in Sec. V. II. PRELIMINARIES

Many physical and biological dynamical systems operating in noisy environments can be described by stochastic differential equations of the Itoˆ/Stratonovich type 关14–16兴, a common example being the SNDSs of the double-well potential type most often encountered in the SR literature. We will also consider here systems of this kind, and for simplicity we will restrict ourselves to the case of a scalar-state variable and additive noise. It should be noted, however, that generalizations within the framework to more general dynamics 共e.g., higher order systems兲 and colored and/or statedependent noise can be carried out, several of which are straightforward. We shall consider SNDSs that can be described by a 共one-

011107-2

GENERAL MEASURES FOR SIGNAL-NOISE . . .

PHYSICAL REVIEW E 63 011107

dimensional, Itoˆ兲 SDE of the form dX t ⫽ f 共 X t 兲 dt⫹s t dt⫹ ␴ dW t ,

t苸 关 0,T 兴 ,

X 0⫽ ␰ ,

共1兲

where the function f represents the negative gradient of a potential, s t is a stochastic process representing a signal, and W t is a standard Wiener process 共independent of ␰ ) scaled by the noise strength parameter ␴ ⬎0. The function f, the process s t , and the initial variable ␰ must satisfy some technical conditions in order to suit the theory developed below. For example, these quantities must fulfill conditions that ensure the existence and uniqueness of a solution to the SDE 共strong solutions will be of particular interest to us兲 关17兴, conditions for the measure transformations 共infinite-dimensional probability density transformations兲 used below to work 共one such condition will be mentioned兲, as well as certain other measurability/integrability conditions 关15,16兴. In all our examples, these 共from an applications point of view not very strict兲 conditions are fulfilled. For later use we note that if the associated Fokker-Planck equation has a stationary solution, or cyclostationary 关18兴 in the case of a periodic signal s t , and ␰ has the corresponding one-dimensional probability distribution, the solution X t to Eq. 共1兲 will be a stationary, respectively cyclostationary, Markov process 关19兴. As a generic example of a potential, we will consider a soft double well for which f in Eq. 共1兲 is given by f 共 x 兲 ⫽⫺ax⫹b tanh共 x 兲 ,

a,b⬎0,

共2兲

and as examples of signal processes we will employ a sinusoid s t ⫽A sin共 ␻ 0 t⫹ ␸ 兲 ,

␻ 0 ⬎0,

共3兲

with constant amplitude A⭓0 and phase ␸ 苸 关 ⫺ ␲ , ␲ ), as well as a Gaussian pulse s t ⫽A exp

冉

⫺ 共 t⫺t 0 兲 2 2␦2

冊

共4兲

centered at t 0 苸 关 0,T 兴 with amplitude A⭓0 and standard deviation ␦ ⬎0. Although these signals are deterministic, there is in principal no difficulty in applying the methodology of this paper to random signals, e.g., the sinusoids with random phase or wide-band noise. An obstacle that arises, however, is that certain quantities will then no longer be exactly expressable by simple formulas. III. SEPARATION INDICES AND SNDS

One of the most basic objectives with measurements of a physical system is to determine if it is in one of two possible conditions 共or modes of operation兲. In a statistical setting 共with noise present兲 this corresponds to determining which of two possible probability measures is active on the space of all behaviors, which is an inference problem of the hypothesis testing type. If one of the two possible conditions corre-

sponds to the presence of a certain type of signal on the input 共or output兲 of the system, and the other condition corresponds to the absence of it, the decision problem is often referred to as a detection problem. For example, in the system 共1兲 with signal of the form 共3兲 or 共4兲, the canonical detection problem is to determine if A⫽0 or A⫽A 0 , for some fixed A 0 ⬎0. Thus, the simplest form of hypothesis testing can be described as any procedure that aims at deciding which of two possible probability measures 共distributions兲 is the correct one for some observed data. The two hypotheses about the distribution of data, or the condition the system is in, are usually denoted H 0 and H 1 respectively, and probability density functions 共PDFs兲 corresponding to the probability measures are, accordingly, denoted p 0 ,p 1 . It would appear that a very basic candidate for an SI in this setting is the performance of a given detector applied to the system’s output for the detection of a certain signal on the input. However, we argue that this is not generally a good choice unless the detector is optimal in some sense 共or one is interested only in one particular aspect of system performance兲.

A. Observables, likelihood ratios, and sufficient statistics

The optimal decision strategy 共detector兲 in all of the basic decision problem formulations 共e.g., Neyman-Pearson, Bayes, minimax兲 in statistics is based on one and the same central quantity, the likelihood ratio 关1兴. The LR is the ratio p 1 /p 0 and expresses how much more probable a given event is under H 1 relative to H 0 . Turning to the system 共1兲, we assume the existence of an underlying abstract probability space ⍀, equipped with a probability measure P, on which the initial variable ␰ , the signal process s t , and the Wiener process W t in Eq. 共1兲 are all defined 关20兴. Unless otherwise stated, the initial variable ␰ is henceforth taken to be zero. We assume further that Eq. 共1兲 has a strong solution for all choices of f and s t that we consider. Since the trajectories X t take values in the space of continuous functions C( 关 0,T 兴 ), we obtain also on C( 关 0,T 兴 ) probability measures induced by X t 关21兴, and these are different for different choices of f ,s t , and ␴ . The measure induced by X t for f ⫽0,s t ⬅0, and ␴ ⬎0 is known as the 共scaled兲 Wiener measure, denoted P␴ . It is well known that 共for fixed ␴ ⬎0兲 the various probability measures on C( 关 0,T 兴 ) induced by X t for different choices of f and s t in Eq. 共1兲 have 共under certain integrability conditions imposed on f and s t ) PDFs with respect to P␴ 关22兴. We denote by H 0 the hypothesis that the PDF in question is p 0 , the one obtained for s t ⬅0, and by H 1 the hypothesis that the PDF is p 1 , the one obtained when s t ⫽0, for fixed common f , ␴ and some given signal s t . In the simplest case, where f ⫽0 and s t is of the form 共3兲 or 共4兲, the process X t will be Gaussian 关23兴 under both H 0 and H 1 , and the LR L(X) ⫽p 1 (X)/p 0 (X) evaluated for the trajectory X t is given by the well known relation 关1兴 共the LR for deterministic signals in Gaussian white noise兲

011107-3

ln L 共 X 兲 ⫽

1

␴

2

冉冕

T

0

s t dX t ⫺

1 2

冕

T

0

冊

s 2t dt .

共5兲

ROBINSON, RUNG, BULSARA, AND INCHIOSA

PHYSICAL REVIEW E 63 011107

冋 冉冕

An important point to note about Eq. 共5兲 is that L(X) can be recovered by a simple deterministic transformation once the value of the stochastic functional 共6兲

S i共 X 兲 is known, where S i is defined by S i共 Y 兲 ⫽

冕

T

0

⌳ (1) 共 X 兲 ⌳ (0) 共 X 兲

共8兲

,

where ⌳ (k) (X) for k⫽0,1 is given by ln ⌳ (k) 共 X 兲 ⫽

1

␴

2

冉冕

T

0

ˆf (k) t 共 X 兲 dX t ⫺

1 2

冕

T

0

冊

2 关 ˆf (k) t 共 X 兲兴 dt .

共9兲

ˆ (1) For the system 共1兲, ˆf (0) t (X)⫽ f (X t ) and f t (X)⫽ f (X t ) ⫹sˆ t (X), where sˆ t (X) is the conditional expectation 共optimal mean square estimate兲 of s t given observations of X ␶ over 关 0,t 兴 , computed under the probability measure P. If s t is deterministic, we have sˆ t (X)⫽s t and the LR becomes particularly easy to compute since we can dispense with the nonlinear filtering operation 共in the statistical sense兲 关15兴, which is otherwise implicit in the computation of sˆ t (X). By dividing out terms in Eq. 共8兲, it is easy to see that a sufficient statistic for ⌳(X) in this case is given by 共10兲

S o共 X 兲 , where S o is defined by S o共 Y 兲 ⫽

冕

T

0

s t dY t ⫺

冕

T

0

1 2

T

0

关 f 共 X t 兲 ⫹s t 兴 2

冊册

⬍⬁,

共12兲

where the expectation E is with respect to P, then ⌳ (1) (X) in Eq. 共9兲 is well defined, as is ⌳ (0) (X) in Eq. 共9兲 if s t is set to 0 in Eq. 共12兲 关26兴.

共7兲

s t dY t ,

for processes Y such that the stochastic integral in Eq. 共7兲 is well defined. This leads us to the concept of sufficient statistic. A sufficient statistic for the LR is a function which maps data, here the trajectories X t , to some intermediate space such that the LR can be obtained from it by a subsequent deterministic transformation 关24兴. Hence, a sufficient statistic carries all the information needed for optimal decision making regarding the condition of the system (H 0 or H 1 兲. Therefore, as an observable to be used for decision making, the LR 共or a sufficient statistic for it兲 is as good as the whole trajectory X t , thereby providing lossless coding of the trajectory in this respect 关25兴. Thus, for inference, the LR deserves to be called a most compact representation of (all) the information in an observable. In the general case, with a nonzero f and possibly random signal s t , the LR ⌳(X) ⫽p 1 (X)/ p 0 (X) takes the form 关15兴 ⌳共 X 兲⫽

E exp

f 共 Y t 兲 s t dt,

共11兲

for processes Y such that the integrals in Eq. 共11兲 are well defined. We note in passing that a sufficient condition for the representation 共8兲,共9兲 to be valid is Novikov’s condition: If

B. ␾ Divergences: Definition, properties and computation

A number of fundamental limits for statistical inference can be expressed in terms of quantities known as ␾ divergences or Ali-Silvey distances 关10,11兴. Examples are the Fisher information 共Crame´r-Rao bound兲 for small parameter deviations, the bound in Stein’s lemma, the Chernoff bound, Wald’s inequalities, and the bound on minimal achievable probability of error in Bayesian hypothesis testing 关1,27,28兴. These bounds limit how well one can perform certain tasks based on measurements on a stochastic system, such as the detection of signals present on the input/output or estimation of parameters in the system. However, the bounds are all achievable 共at least asymptotically兲, i.e., there exist strategies for inference that yield a performance that approaches the bound. Thus, for physical systems these bounds effectively tell us how much information 共for various forms of inference兲 about the system different observables can provide 关29兴, and the ␾ divergences offer alternative 共compact兲 representations of it. The ␾ divergences have properties reminiscent of directed distances between probability measures 共PDFs兲 and are defined as convex functionals of the LR in the following way. Let p 0 ,p 1 be two PDFs with respect to a reference measure ␭ on some space X 共considering Lebesgue measure d␭⫽dx on X⫽R makes the picture clear兲 and let ␾ be a 共real-valued兲 continuous convex function on 关 0,⬁). The ␾ divergence d ␾ (p 0 ,p 1 ) between p 0 and p 1 is then given by 关10兴 d ␾ 共 p 0 ,p 1 兲 ⫽

冕 ␾冉 冊 X

p1 p d␭ p0 0

共13兲

共where we assume that p 1 is zero where p 0 is; however, in our examples p 0 is positive-␭ almost everywhere兲. In particular, for ␾ (x)⫽⫺ln(x) we obtain the Kullback-Liebler divergence, or information divergence d I 关30兴, also known as the relative entropy; for ␾ (x)⫽ 兩 (1⫺ ␣ )x⫺ ␣ 兩 , where ␣ 苸 关 0,1兴 , we obtain the 共weighted兲 Kolmogorov divergence, or error divergence d (␧␣ ) ; and for ␾ (x)⫽(x⫺1) 2 we obtain the ␹ 2 divergence d ␹ 2 关31兴. By definition 共13兲 the ␾ divergences contain several attractive features as statistical measures of dissimilarity, or separation, between p 0 ,p 1 . In particular, any given divergence d ␾ (p 0 ,p 1 ) is always maximized if p 0 p 1 ⫽0 共almost everywhere兲, and conversely d ␾ (p 0 ,p 1 ) is minimized if p 0 ⫽ p 1 共almost everywhere兲. For example, taking

011107-4

GENERAL MEASURES FOR SIGNAL-NOISE . . .

d (␧␣ ) 共 p 0 , p 1 兲 ⫽ ⫽

冕冏 冕 X

X

PHYSICAL REVIEW E 63 011107

冏

p1 ⫺ ␣ p 0 d␭ p0

d ␾共 X 兲 ⫽

兩 共 1⫺ ␣ 兲 p 1 ⫺ ␣ p 0 兩 d␭,

⫽

共 1⫺ ␣ 兲

Moreover, any transformation

␩ :X→Y of the underlying space X, which induces a new reference measure ␳ and corresponding PDFs q 0 and q 1 on Y, can never increase divergences, since we have the data processing inequality 关32兴 共14兲

Equality occurs if and only if the new LR q 1 /q 0 , when evaluated as q 1 关 ␩ (x) 兴 /q 0 关 ␩ (x) 兴 over X, is a sufficient statistic for the original LR p 1 (x)/ p 0 (x), and this makes the LR 共and its sufficient statistics兲 the most ‘‘informative’’ function of an observable for inference. For example, if p 0 ,p 1 are the PDFs with respect to P␴ on C( 关 0,T 兴 ) induced by the trajectories X t of the system 共1兲 for s t ⬅0 and s t ⫽0, respectively, ␩ is the functional on C( 关 0,T 兴 ) defined by the statistic 共10兲, and q 1 ,q 0 are the resulting two PDFs with respect to the Lebesgue measure on R of the values of this functional, then we trivially have equality in 共14兲. For future reference, we note also that if ␩ is invertible we will have equality in Eq. 共14兲 and no loss of information. In particular, systems such as 共1兲 are invertible in the following sense and thus are divergence preserving: each output trajectory X t in Eq. 共1兲 uniquely determines a trajectory defined by Z t ⫽X t ⫺ 兰 t0 f (X ␶ )d ␶ , and the map so defined is injective 关33兴. Since we can 共with probability one兲 identify Z t with the input trajectory

冕

t

0

s ␶d ␶ ⫹ ␴ W t

共15兲

共where s t can be zero in the case of no signal兲 it follows that the input and output trajectories are in one-to-one correspondence, and the system is invertible. Thus, for any ␾ divergence, the divergence between the two probability measures on C( 关 0,T 兴 ) induced by the input for s t ⬅0 and s t ⫽0, respectively, 共for which the LR is given by Eqs. 共8兲 and 共9兲 with f ⫽0) will coincide with that between the corresponding two measures on C( 关 0,T 兴 ) induced by the resulting output 关for which the LR is given by Eqs. 共8兲 and 共9兲兴. Further, for systems such as 共1兲, a concrete representation for ␾ divergences between probability measures on C( 关 0,T 兴 ) induced by X t has been given 关34兴 in terms of the LR in Eq. 共8兲. Let p 0 , p 1 be the densities with respect to P␴ induced by X t when s t ⬅0 and s t ⫽0, respectively. Then, the ␾ divergence d ␾ (X) between p 0 and p 1 can be written

p1 p dP p0 0 ␴

⌳ (1) 共 X 兲

⍀

⫽E ␾

兩 1⫺2 ␣ 兩 ⭐d (␧␣ ) 共 p 0 , p 1 兲 ⭐1.

F t⫽

C([0,T])

冉冉

it is clear that the extreme cases yield the bounds

d ␾ 共 p 0 , p 1 兲 ⭓d ␾ 共 q 0 ,q 1 兲 .

冕 ␾冉 冊 冕 ␾冉 冊 ⌳ (0) 共 X 兲

⌳ (1) 共 X 兲 ⌳ (0) 共 X 兲

冊

⌳ (0) 共 X 兲 d P

冊

⌳ (0) 共 X 兲 ,

共16兲

where ⌳ (0) (X),⌳ (1) (X) are given by Eq. 共9兲 and the expectation E is with respect to P. The importance of the representation 共16兲 lies in the fact that the divergence sought, which is somewhat abstractly defined by the first equality, admits a concrete representation in terms of the other two equalities 关where the dependence on the SDE 共1兲 is made explicit兴. In particular, the last two equalities provide us with a means to numerically compute the value of a divergence by Monte Carlo simulation. C. Relations to bounds for inference

Perhaps the most fundamental connection between ␾ divergences and limits for inference is the one furnished by the relation between the Kolmogorov divergence d (␧␣ ) and minimal achievable probability of error in hypothesis testing. Let p 0 and p 1 be two generic probability densities 共with respect to a measure ␭ as before兲 corresponding to two hypotheses H 0 and H 1 , symbolizing for example the absence/presence of a signal s t in the system 共1兲, and assume that parameter ␣ and its complementary value 1⫺ ␣ in the definition of d (␧␣ ) (p 0 ,p 1 ) represent two a priori probabilities for H 0 and H 1 , respectively, to occur 共the standard Bayesian setting in statistics兲. Then, it is straightforward to show that 关11兴 ˜P (e␣ ) 共 p 0 ,p 1 兲 ⫽ 21 关 1⫺d (␧␣ ) 共 p 0 ,p 1 兲兴 , where ˜P (e␣ ) (p 0 ,p 1 ) is the minimal achievable probability of error in hypothesis testing between H 0 and H 1 共for parameters ␣ and 1⫺ ␣ ) 关35兴. Thus, we see that an observable for which d (␧␣ ) (p 0 ,p 1 ) is large provides low ˜P (e␣ ) (p 0 ,p 1 ) and therefore much information for inference purposes. Optimal detection, such as minimizing the probability of error in the sense just described, requires full knowledge of the probability distributions involved, i.e., the LR, and this can be difficult to obtain in many applications. Therefore, an alternative type of SI known as the deflection ratio 共DR兲 is sometimes used. The DR depends only on the expectations and variances of an observable at hand and is most commonly defined as follows. Let h be some 共possibly兲 complexvalued observable of the data such that E 1 (h) and V 0 (h) both exist, where E 1 (h) is the expectation of h under H 1 and V 0 (h) is the variance of h under H 0 . The DR ⌬(h) of h is then defined as 关1,36兴

011107-5

⌬共 h 兲⫽

兩 E 1 共 h 兲 ⫺E 0 共 h 兲 兩 2 , V 0共 h 兲

共17兲

ROBINSON, RUNG, BULSARA, AND INCHIOSA

PHYSICAL REVIEW E 63 011107

where E 0 (h) is the expectation of h under H 0 . The DR is often viewed as a generalization of the concept of SNR. When used for detection, the decision that H 1 is true is made if h⬎ ␥ , where ␥ is some threshold; otherwise H 0 is chosen 关assuming h is real and E 1 (h)⬎E 0 (h); in general, h is compared with some decision boundary兴. By writing out ⌬(h) in terms of the integrals with respect to p 0 , p 1 and applying the Cauchy-Bunyakovsky-Schwarz inequality, one obtains the bounds 0⭐⌬ 共 h 兲 ⭐d ␹ 2 共 p 0 , p 1 兲 ,

共18兲

with equality on the left if and only if E 0 兵 关 h ⫺E 0 (h) 兴 (p 1 /p 0 ⫺1) 其 ⫽0 and equality to the right if and only if C 1 关 h⫺E 0 (h) 兴 ⫽C 2 (p 1 /p 0 ⫺1) with p 0 -probability one, for two 共complex兲 constants C 1 ,C 2 not both zero. Thus, in particular we have equality to the right in Eq. 共18兲 if h equals the LR p 1 / p 0 . D. Relations to SNR and detector optimality

Given the properties of ␾ divergences, it would be desirable to compare and relate these to those of the SNR, and this is indeed possible. It has been shown 关34兴 that the SNR used in SR can, under some mild technical conditions, be expressed as a limit 共as the observation time T goes to infinity兲 of deflections of Fourier transforms computed from the trajectories X t of the system 共1兲. Let p 0 and p 1 be the densities with respect to P␴ induced on C( 关 0,T 兴 ) by the trajectories X t when s t ⬅0; hypothesis H 0 , and s t ⫽0; hypothesis H 1 , respectively, as in Sec. III A. Further, let E 0 and E 1 denote the expectations computed under H 0 and H 1 , respectively, and assume that the system has a stationary solution X t under H 0 , a cyclostationary solution under H 1 , and that E 0 (X 2t )⬍⬁,E 1 (X 2t )⬍⬁. For the case of deterministic 共periodic兲 signals as in Eq. 共3兲 we can then define the SNR Sp as Sp ⫽

ap , g 0共 ␻ 0 兲

共19兲

where g 0 is the power spectral density of the Lorentzian process X t obtained under H 0 and a p ⫽ 兩 c 1 兩 2 /2␲ , where c 1 is the first coefficient in the Fourier expansion 兺 n苸Zc n e i ␻ 0 nt of the periodic function E 1 (X t ) 共this definition makes the most sense for weak signals, i.e., AⰆ1). Then, under some integrability conditions on the covariance and power spectral density functions of X t under H 0 , we have 关34兴 lim

⌬ 关 I T1/2共 ␻ 0 兲兴

T→⬁

T

共20兲

⫽Sp ,

where I T1/2 is a square root of the continuous-time periodogram defined as I T1/2共 ␻ 兲 ⫽

1

冑2 ␲ T

冕

T

0

X t e ⫺i ␻ t dt,

future use we note also that the SNR 共19兲 is invariant under transformation by a linear time-invariant system 共with finite nonzero Fourier transform near ␻ 0 ). The bounds 共18兲 provide us with a straightforward way of assessing the nonoptimality of a given detector 共i.e., statistic h). For example, I T1/2( ␻ 0 ) can be interpreted as a linear functional on C( 关 0,T 兴 ) 共where the trajectories X t take their values兲 so we can apply the bounds in Eq. 共18兲 to the statistic h⫽I T1/2( ␻ 0 ). The ratio N T 苸 关 0,1兴 defined by

␻ 苸R.

共21兲

As an aside, we note that a similar relation holds for the case of a random phase ␸ , for weak signals (AⰆ1) 关34兴. For

N T⫽

d ␹ 2 共 p 0 ,p 1 兲 ⫺⌬ 共 I T1/2共 ␻ 0 兲兲 d ␹ 2 共 p 0 ,p 1 兲

共22兲

关where the PDFs p 0 ,p 1 are the ones induced on C( 关 0,T 兴 ) by X t 兴 will then be an index of nonoptimality 关37兴 of the Fourier statistic I T1/2( ␻ 0 ) as a detection statistic. This can 共for large T) be expressed in terms of SNR if we divide both the numerator and denominator of the right hand side of Eq. 共22兲 by T and use Eq. 共20兲 to write ⌬ 关 I T1/2( ␻ 0 ) 兴 /T⫽Sp ⫹o(1) 关38兴. Thus, it follows that for signals and systems as in Sec. II the SNR is in general not to be equated with optimal detection performance but, rather, when compared to optimal detection performance, gives an index of the nonoptimality 关39兴 for detection of s t based on the trajectory X t using the statistic 共21兲 关40兴. IV. SIMULATIONS

We shall now illustrate the above findings with some numerical simulations involving the system 共1兲, for deterministic signal s t in the form of a sinusoid as in Eq. 共3兲 and a pulse as in Eq. 共4兲. In all the simulations the parameters used for the potential represented by f in Eqs. 共1兲,共2兲 are a⫽53.5,b ⫽216 and the SDE 共1兲 is solved using the Euler-Maruyama scheme. The 共integrated兲 input F t to the system 共1兲 is defined as in Eq. 共15兲, where s t ⬅0 under H 0 and is given by either Eq. 共3兲 or Eq. 共4兲 under H 1 . The output, finally, is given by X t in Eq. 共1兲. We compute the ␾ divergences for the statistics 共7兲 and 共11兲, and compare with the results computed from Eq. 共16兲, all evaluated both for the 共integrated兲 input F t and output X t . Note that, formally, X t ⫽F t for f ⫽0 so that, e.g., S i (F) is given by S i (X) in 共7兲 if f is set to 0 and analogously for the statistic in Eq. 共11兲 and a divergence as in Eq. 共16兲. Two distinctly different techniques were used to compute the various ␾ divergences depending on whether the divergence in question was one between PDFs on R or between PDFs on C( 关 0,T 兴 ). For PDFs on R, as encountered when evaluating divergences for the statistics S i (F),S i (X),S o (F),S o (X), the divergences were calculated using the basic formula 共13兲, where the PDFs p 0 ,p 1 were estimated using a simple histogram approach. For instance, when computing the divergences for the statistic S o (X) in Eq. 共10兲 the SDE 共1兲 was solved using both s t ⫽0 and s t ⫽0 共with the nonzero signal chosen according to the case under consideration兲 and two large sets of solution trajectories X t were created, representing the H 0 and H 1 hypotheses on C( 关 0,T 兴 ), respectively. These two sets of trajectories

011107-6

GENERAL MEASURES FOR SIGNAL-NOISE . . .

PHYSICAL REVIEW E 63 011107

FIG. 2. ␾ divergences 共Kolmogorov, left; information, middle; ␹ 2 , right兲 for the input to the system 共1兲, based on the statistic S o (F), which is not sufficient for the input with sinusoidal signal 共under H 1 ), plotted as functions of the noise intensity ␴ 2 . 共Curves are an enlargement of the dashed curves in the left column in Fig. 1.兲 These curves show a clear resonance.

FIG. 1. ␾ divergences 共Kolmogorov, top row; information, middle row; ␹ 2 , bottom row兲 for system 共1兲 with the sinusoidal signal 共under H 1 ) plotted as functions of the noise intensity ␴ 2 for the input 共left column兲 and output 共right column兲, all in dimensionless units. The divergences are computed based on the values of the statistic in Eq. 共7兲 共dash dotted lines兲, the statistic in Eq. 共11兲 共dashed lines兲, and formula 共16兲 共solid lines兲. For the input, it can be seen that the statistic S i (F) produces the same divergences as the ones obtained from d ␾ (F), which is to be expected since S i (F) is sufficient for the LR for the input process. The statistic S o (F) on the other hand 共with values at the bottom of the plots in the left column兲, which is not sufficient for the input, produces values far below the corresponding optimal ones obtained from S i (F) and d ␾ (F) 共cf. Fig. 2兲. For the output we analogously see that the statistic S o (X), which is sufficient for the LR for output, produces the same values as d ␾ (X), whereas the statistic S i (X) produces far lower values 共falling on the abscissa in the plots in the right column兲. Moreover, due to the invertibility the d ␾ (F) and d ␾ (X) curves coincide.

were then used to produce histogram estimates of the PDFs p 0 , p 1 for S o (X) under H 0 and H 1 , from which the divergences for this statistic were subsequently computed straightforwardly using Eq. 共13兲. The procedure employed for S i (F),S i (X),S o (F) was analogous, using the observations above about the relations between X t and F t . On the other hand, for PDFs on C( 关 0,T 兴 ), as encountered when evaluating the divergences d ␾ (F),d ␾ (X) in Eq. 共16兲, an entirely different approach was used based on directly estimating the integral on the right of the second equality in Eq. 共16兲. It utilizes the fact that if a process X t which is a Wiener process under the basic measure P is inserted into formula 共16兲 in all places where X t appears, then standard averaging will produce the expectation 共integral兲 on the right in Eq. 共16兲 关34兴. However, in order to achieve numerical convergence and efficiency, a number of numerical devices were needed, but

these will be described elsewhere. When computing deflection ratios, the expectations and variances in Eq. 共17兲 were computed directly by standard averaging, without first computing PDFs for I T1/2( ␻ 0 ). A. Harmonic signal

Our first example will illustrate the results of Secs. III B– III D for the case of sinusoidal signal s t as in Eq. 共3兲 under H 1 . The parameters for s t are A⫽1.3, ␻ 0 ⫽1.2252, ␸ ⫽0 共cf. 关41兴兲. The length of the time interval is T⫽153.8 共which corresponds to 30 periods of the sinusoid兲, the time step in the Euler-Maruyama scheme is 0.01, and a total of 10 000 trajectories has been used in the averaging. The value of the a priori probability in the Kolmogorov divergence is ␣ ⫽0.6. In Fig. 1 the Kolmogorov, information and ␹ 2 divergences are computed for the input and output processes, respectively, using the statistic in Eq. 共7兲, the statistic in Eq. 共11兲, and formula 共16兲. For example, the upper left panel in Fig. 1 shows d (␧␣ ) 兵 p 0 关 S i (F) 兴 ,p 1 关 S i (F) 兴 其 共dash-dotted line兲, d (␧␣ ) 兵 p 0 关 S o (F) 兴 ,p 1 关 S o (F) 兴 其 共dashed line兲, and d (␧␣ ) (F) 共solid line兲, where p k (S), k⫽0,1, is the PDF 共on R) obtained for statistic S 关as in Eq. 共7兲 or 共11兲兴 under hypothesis H k ; for the upper right panel, replace F with X. For the input, we see that the divergences for the statistic S i (F), which is a sufficient statistic for the input LR and for which the divergences are between PDFs on R 共where S i (F) takes its values兲, agree with the divergences d ␾ (F) obtained from formula 共16兲, which gives divergences between PDFs on C( 关 0,T 兴 ). This is in accordance with what we know about equality in the data processing inequality 共14兲, since here we can interpret p 0 ,p 1 in Eq. 共14兲 as the PDFs on C( 关 0,T 兴 ) induced by the input F t under H 0 and H 1 , respectively, and ␩ as the functional on C( 关 0,T 兴 ) defined by S i (F), which trivially yields equality in Eq. 共14兲, since S i (F) is a sufficient statistic for F t . For the statistic S o (F), which is not sufficient for the input, we obtain curves displaying a barely vis-

011107-7

ROBINSON, RUNG, BULSARA, AND INCHIOSA

PHYSICAL REVIEW E 63 011107

FIG. 3. Deflection ratio ⌬ 关 I T1/2( ␻ 0 ) 兴 for the output to the system 共1兲 with the sinusoidal signal 共under H 1 ) plotted as a function of noise intensity ␴ 2 共dimensionless units兲. The same definitions and parameters as in Fig. 1 have been used. A clear resonance can be observed.

ible resonant behavior with values far below the corresponding ones for the divergences d ␾ (F). The behavior of the S o (F) curves is more clearly seen when displayed separately as in Fig. 2. They illustrate that a nonoptimal detection statistic can give rise to performance curves that display typical SR behavior, and that the asymptotic behavior for small noise can be markedly different from the corresponding curves obtained for an optimal statistic. Finally we note that all divergences d ␾ (F) for the input decay monotonically with the input noise strength ␴ , which is consistent with intuition. For the output we make analogous observations. Here the statistic S o (X) is sufficient for the LR and produces the same divergences as the divergences d ␾ (X) obtained from formula 共16兲, whereas the statistic S i (X) produces far lower values. This is in accordance with the theory since if we interpret p 0 , p 1 in Eq. 共14兲 as output-induced PDFs on C( 关 0,T 兴 ) and ␩ as the functional defining the statistic S o (X), we have equality in Eq. 共14兲. Moreover, the divergences d ␾ (X) computed using formula 共16兲 coincide with the corresponding divergences d ␾ (F) for the input, since the system is invertible. The statistic S i (X) on the other hand, which is not sufficient for the output LR, produces curves that are 共far兲 below those obtained from statistic S o (X) and the divergences d ␾ (X) from formula 共16兲. The monotonic decay of the divergence curves computed from formula 共16兲 differs markedly from the behavior of the deflection of the Fourier statistic ⌬ 关 I T1/2( ␻ 0 ) 兴 based on the output, as defined in Sec. III D, which is shown in Fig. 3.

FIG. 4. Nonoptimality N T for the output as a function of noise intensity ␴ 2 共dimensionless units兲. The peak in the curve agrees well with the dip in the deflection curve in Fig. 3.

Here we see typical SR behavior with a clear resonance near ␴ 2 ⫽300. However, in view of Eq. 共20兲, this type of behavior is to be expected. It is also worth noting that the values, in particular after normalization with 1/T, are much lower than those for the output ␹ 2 divergence in Fig. 1. Another interesting observation about the curve can also be made which is not related to the resonance peak but to the dip immediately preceding it. Given the discussion in the preceding section about nonoptimality of the detection statistic I T1/2( ␻ 0 ), it is tempting to believe that the dip corresponds to maximal nonoptimality, in the sense of local maximization of the nonoptimality index N T defined in Eq. 共22兲, for the statistic I T1/2( ␻ 0 ). In Fig. 4 a plot of N T for the output is shown and indeed a peak appears at around ␴ 2 ⫽80, which matches the dip in Fig. 3 very well. In fact, the peak in the N T curve represents a global maximum, which means that the Fourier statistic 共21兲 is maximally poor as a 共threshold兲 detection statistic at this value of ␴ . Consequently, the peak in the deflection curve should therefore more aptly be thought of as representing a recovery behavior, where some of the performance lost in the region of values where the dip occurred is regained 关42兴. B. Pulse signal

To illustrate that the ␾ divergences are applicable also to nonperiodic forcing, we have computed the divergences for the same setup as in the example in Sec. IV A but with a Gaussian pulse signal as in Eq. 共4兲 under H 1 . The pulse is centered at t 0 ⫽T/2, and has a standard deviation of ␦ ⫽T/4 and an amplitude A⫽5. The length of the time interval is

011107-8

GENERAL MEASURES FOR SIGNAL-NOISE . . .

PHYSICAL REVIEW E 63 011107

FIG. 5. ␾ divergences 共Kolmogorov, top row; information, middle row; ␹ 2 , bottom row兲 for the system 共1兲 with pulse signal 共under H 1 ) plotted as functions of the noise intensity ␴ 2 for the input 共left column兲 and output 共right column兲, all in dimensionless units. The divergences are computed based on the values of the statistic 共7兲 共dash dotted lines兲, the statistic 共11兲 共dashed lines兲, and the formula 共16兲 共solid lines兲. For the input, the statistic S i (F), which is sufficient for the LR, produces the same divergences as the ones obtained from d ␾ (F). The statistic S o (F) on the other hand 共with values at the bottom of the plots兲, which is not sufficient for the input, produces values far below the corresponding optimal ones obtained from S i (F) and d ␾ (F). Analogously, for the output the statistic S o (X), which is sufficient for the LR, produces the same values as d ␾ (X), whereas the statistic S i (X) produces lower values. Here, the difference between the optimal and suboptimal values is not so great, however. Also, the d ␾ (F) curves coincide with the d ␾ (X) curves due to the invertibility.

T⫽10, the time step in the Euler-Maruyama scheme is 0.005, and a total of 20 000 trajectories has been used in the averaging. The value of the a priori probability in the Kolmogorov divergence is ␣ ⫽0.6. In Fig. 5 the divergences based on the input and output are shown. On the whole, the behavior is similar to that displayed in the example with sinusoidal signal; sufficient statistics always produce the same values as formula 共16兲, and the latter yields the same values for the input as for the output. For the output, the nonsufficient statistic does not produce much lower values than the optimal ones, though.

sibly random, wide-band兲 signal. If we make a loose analogy with 共thermodynamical兲 entropy in closed systems and interpret the similarity expressed by ␾ divergences as a figure of ‘‘mixed-up-ness’’ or overlap between two PDFs, the data processing inequality 共14兲 shows that ␾ divergences also behave consistently with intuition: deterministic transformations 共which do not involve any auxiliary random variables and thus are closed兲 cannot increase the separation 共i.e., decrease the mixed-up-ness兲 between two PDFs. In fact, the data processing property 共14兲 can be taken as a natural 共axiomatic兲 requirement of an SI between probability measures which guarantees that the output-input separation gain will always be between 0 and 1 关43兴. Further, as mentioned in Sec. III B, systems that represent invertible transformations preserve divergences, and therefore the separation between noise and signal is 共in this sense兲 the same whether it is measured on the input or output of the system. This also can be taken as a natural requirement of a separation index: for an invertible system, all the information about the signal is still present after passage through the system, it is just represented differently than at the input, and a good SI should be invariant under different 共equivalent兲 representations of data. The behavior of ␾ divergences is thus markedly different from other SIs that do not have the data processing property 共14兲 共with equality for invertible transformations兲. In particular, for a time-sinusoidal input such as Eq. 共3兲 embedded in a weakly stationary background 共noise兲 process, the SNR 共19兲 has only partially this property, since it is invariant under 共locally in the spectral domain, near ␻ 0 ) invertible linear filtering operations but not under the relatively simple nonlinear invertible transformations that correspond to a passage through a system such as 共1兲. This lack of invariance of the SNR is a consequence of the fact that the statistic 共21兲 implicit in the definition of the SNR is blind to certain parts of the statistical information about the process. Another way of quantifying the blindness to statistical information inherent in the statistic 共21兲 emerges naturally when considering its use in detection. When used for detection on the output 共to detect a time-sinusoidal signal present on the input兲 of the system 共1兲 the statistic 共21兲 always renders the detector suboptimal 共no matter how the statistic is used; it is not sufficient for the LR兲. More generally, any SI used to describe separation between signal present and absent on the output which is not a functional of the LR will necessarily be blind to certain parts of the statistical information and will suffer from similar inadequacies 共and may or may not produce resonances as, e.g., in Fig. 2兲. Still, SNR and similar SIs can be very relevant in those instances where only one specific aspect of system behavior is important, such as in narrowband processing where the signal power at a single frequency is the main concern.

V. CONCLUDING REMARKS

In view of what has been shown in the preceding sections, it is clear that the ␾ divergences represent a very general class of SIs that are applicable to almost any type of stochastic system, in particular systems like 共1兲 with a general 共pos-

ACKNOWLEDGMENTS

J.W.C.R. wishes to thank L. Gammaitoni for emphasizing the connection between closed systems in thermodynamics and the data processing property of ␾ divergences, and ac-

011107-9

ROBINSON, RUNG, BULSARA, AND INCHIOSA

PHYSICAL REVIEW E 63 011107

knowledges support from FOA Project No. E6022, Nonlinear Dynamics. J.R. acknowledges support from AIM Research School, FOA Project No. E6022, Nonlinear Dynamics, and the Gustafsson’s Foundation Grant No. 97:12.

A.R.B. and M.E.I. acknowledge support from the Office of Naval Research through Grant No. N00014-99-0592. Part of the computer simulations were carried out on the Cray T3E at NSC, Linko¨ping, Sweden.

关1兴 H.V. Poor, An Introduction to Signal Detection and Estimation 共Springer-Verlag, New York, 1994兲; H. van Trees, Detection, Estimation and Modulation Theory 共Wiley, New York, 1978兲. 关2兴 To be more precise, for a Gaussian process the set of finitedimensional probability distribution functions 共the law兲 is uniquely determined by the mean and autocovariance function, and the power spectrum is an equivalent representation of the autocovariance function. 关3兴 Here we think of a SI simply as some real-valued functional of data; a figure of merit. 关4兴 For good overviews see K. Wiesenfeld and F. Moss, Nature 共London兲 373, 33 共1995兲; A. Bulsara and L. Gammaitoni, Phys. Today 49共3兲, 39 共1996兲; L. Gammaitoni, P. Ha¨nggi, P. Jung, and F. Marchesoni, Rev. Mod. Phys. 70, 223 共1998兲. 关5兴 M.E. Inchiosa and A.R. Bulsara, Phys. Rev. E 53, R2021 共1996兲; ibid. 58, 115 共1998兲. 关6兴 J.J. Collins, C.C. Chow, and T.T. Imhoff, Phys. Rev. E 52, 3321 共1995兲; A. Bulsara and A. Zador, ibid. 54, R2185 共1996兲; C. Heneghan, C.C. Chow, J.J. Collins, T.T. Imhoff, S.B. Lowen, and M.C. Teich, ibid. 54, R2228 共1996兲; M. Stemmler, Network 7, 687 共1996兲; F. Chapeau-Blondeau, Phys. Rev. E 55, 2016 共1997兲; I. Goychuk and P. Ha¨nggi, ibid. 61, 4272 共2000兲. 关7兴 J.W.C. Robinson, D.E. Asraf, A.R. Bulsara, and M.E. Inchiosa, Phys. Rev. Lett. 81, 2850 共1998兲. 关8兴 V. Galdi, V. Pierro, and I.M. Pinto, Phys. Rev. E 57, 6470 共1998兲. 关9兴 A. Neiman, B. Shulgin, V. Anishchenko, W. Ebeling, L. Schimansky-Geier, and J. Freund, Phys. Rev. Lett. 76, 4299 共1996兲; M. Misono, T. Kohmoto, Y. Fukuda, and M. Kunitomo, Phys. Rev. E 58, 5602 共1998兲. 关10兴 F. Liese and I. Vajda, Convex Statistical Distances 共Teubner, Leipzig, 1987兲. 关11兴 S. Ali and D. Silvey, J. R. Stat. Soc. B 28, 131 共1966兲. 关12兴 M.E. Inchiosa, J.W.C. Robinson, and A.R. Bulsara, Phys. Rev. Lett. 85, 3369 共2000兲. 关13兴 L. Gammaitoni, F. Marchesoni, and S. Santucci, Phys. Rev. Lett. 74, 1052 共1995兲. 关14兴 See, e.g., A. Bharucha-Reid, Elements of the Theory of Markov Processes and their Applications 共McGraw-Hill, New York, 1960兲; N. van Kampen, Stochastic Processes in Physics and Chemistry 共North Holland, Amsterdam, 1992兲. 关15兴 R.S. Liptser and A.N. Shiryayev, Statistics of Random Processes I: General Theory 共Springer Verlag, New York, 1977兲. 关16兴 I. Karatzas and S.E. Shreve, Brownian Motion and Stochastic Calculus 共Springer-Verlag, New York, 1988兲. 关17兴 Cf., e.g., Theorem 4.8 in 关15兴. 关18兴 A stochastic process X t is cyclostationary with period T 0 if its finite-dimensional probability distributions F X (x 1 , . . . ,x n ;t 1 , . . . ,t n ) are invariant under time shifts by where m is an integer, i.e., if mT 0 ,

F X (x 1 , . . . ,x n ;t 1 , . . . ,t n )⫽F X (x 1 , . . . ,x n ;t 1 ⫹mT 0 , . . . ,t n ⫹mT 0 ). Conditions guaranteeing a cyclostationary Markov solution to 共1兲 can be found in Sec. III.5 of R.Z. Hasminskii, Stochastic Stability of Differential Equations 共Sijthoff and Noordhoff, Alphen an der Rijn, 1980兲. Here and in the following, a number of measure-theoretic details are omitted, in particular the various ␴ algebras 共and filtrations兲 involved, since they are not essential 共and a reader with the appropriate background can easily fill them in兲. Suffice it to say that there must exist a basic ␴ algebra F on ⍀ with respect to which P is defined and the various random variables and processes are measurable, and as basic ␴ algebra on C( 关 0,T 兴 ) we take the Borel ␴ algebra B关 C( 关 0,T 兴 ) 兴 . A good introduction to 共abstract兲 probability theory is given in D. Williams, Probability with Martingales 共Cambridge University Press, Cambridge, 1991兲 and the measure-theoretic details of the continuous-time stochastic processes encountered here are covered by, e.g., 关16兴. If ␭ is a measure on X and ␩ :X→Y, the map ␩ induces a measure ␳ on Y by ␳ (B)⫽␭( 兵 x苸X: ␩ (x)苸B 其 ) for B債Y 共again, details about ␴ algebras are omitted兲. A thorough treatment of the associated theory of such densities can be found in Chapter 7 of Ref. 关15兴. Also for f linear, X t will be Gaussian. Compare, e.g., Ref. 关30兴, Sec. 2.4. Indeed, it represents a tremendous coding, since the trajectory lives in an infinite-dimensional space, whereas the LR takes values on the real line. Note, however, that we use the word coding a little bit loosely here, and not in its strict informationtheoretic sense. See, e.g., Sec. 3.5.D of Ref. 关16兴 and Sec. 6.2 of Ref. 关15兴, where further conditions guaranteeing that 共12兲 is fulfilled can also be found. T.M. Cover and J.A. Thomas, Elements of Information Theory 共Wiley, New York, 1991兲. D. Siegmund, Sequential Analysis 共Springer-Verlag, New York, 1985兲. Here and in the following the word ‘‘information’’ is to be interpreted informally and not in its most common information-theoretic sense 共which applies to communication兲. It is noteworthy, however, that Kullback 关30兴 who mostly considered inference, quantified the word ‘‘information’’ by the value of the information divergence. S. Kullback, Information Theory and Statistics 共Dover, New York, 1997兲. See, e.g., 关36兴 for relations between these divergences. This terminology is borrowed from information theory, where a related inequality with the same name holds for the mutual information; see 关27兴. For the injectivity to hold, it is sufficient that f satisfy a global

关19兴

关20兴

关21兴

关22兴 关23兴 关24兴 关25兴

关26兴

关27兴 关28兴 关29兴

关30兴 关31兴 关32兴

关33兴

011107-10

GENERAL MEASURES FOR SIGNAL-NOISE . . .

关34兴

关35兴

关36兴 关37兴

关38兴

关39兴

PHYSICAL REVIEW E 63 011107

Lipschitz condition, but this is generally assumed in order to guarantee a strong solution to the SDE 共1兲. J. Rung and J.W.C. Robinson, in STOCHAOS: Stochastic and Chaotic Dynamics in the Lakes, edited by D.S. Broomhead, E.A. Luchinskaya, P.V.E. McClintock, and T. Mullin 共American Institute of Physics, Melville, NY, 2000兲. An error is said to occur if, after observation of data, such as the trajectory X t of 共1兲, one infers that H 0 is correct when in fact H 1 is, or vice versa. M. Basseville, Signal Proc. 18, 349 共1989兲. If we recall the conditions for equality in Eq. 共18兲, we see that the index N T in Eq. 共22兲 expresses nonalignment, or orthogonality, between h⫺E 0 (h) and p 1 /p 0 ⫺1. Strictly speaking, the limit 共20兲 is established under different conditions than for the divergence in 共22兲 since we have assumed ␰ ⫽0 for the latter. However, under mild conditions one can show that the limit in 共20兲 will exist and remain the same even if one instead starts 共1兲 with ␰ ⫽0 so that the solution to 共1兲 will be merely asymptotically cyclostationary under H 1 . Frequently, the quantity d ␹ 2 (p 0 ,p 1 )/T tends to infinity as T grows, with the consequence that the nonoptimality N T will tend to 1. In particular, for f ⫽0 it can be shown that d ␹ 2 (p 0 ,p 1 ) typically grows exponentially with T 关e.g., for the signal 共3兲兴, which implies that the two probability densities p 0 ,p 1 on C( 关 0,T 兴 ) eventually separate completely so that perfect 共zero error兲 detection becomes possible, yielding so-called

共asymptotically兲 singular detection 关1兴. 关40兴 In the special case of linear f and sinusoidal signal of the form 共3兲, a glance at Eqs. 共10兲,共11兲 reveals that for ␸ ⫽0 and T ⫽k ␲ / ␻ 0 (k being a positive integer兲 the real and imaginary parts of I T1/2( ␻ 0 ) together form a sufficient statistic for the LR. 关In the particular case where f ⫽0 and the integral in definition 共21兲 of I T1/2( ␻ ) is replaced by 兰 T0 e ⫺i ␻ t dX t the modified statistic I T1/2( ␻ 0 ) will always be sufficient for the LR in this case, for all ␸ ,T.兴 On the other hand, in the general case where f is nonlinear, it is clear from Eqs. 共10兲,共11兲 that the Fourier statistic I T1/2( ␻ 0 ) 共and its modification兲 is no longer a sufficient statistic for the LR 共8兲 共for any values of ␸ ,T). 关41兴 M.E. Inchiosa and A.R. Bulsara, Phys. Rev. E 52, 327 共1995兲. 关42兴 For the system 共1兲 with a potential such as that corresponding to Eq. 共2兲, which both locally near the two local minima of the potential and for large 兩 x 兩 is parabolic, there are, moreover, two asymptotes that are to be expected in the deflection curves, provided the signal is small and T is large: when the input noise strength ␴ is small the system acts essentially linearly, and hence will preserve not only divergences but also SNR, and it will also appear linear for very large ␴ , and the same preservation of both divergences and SNR will occur then also. 关43兴 It can be shown that functions of the LR that have the data processing property 共14兲 must 共under some technical conditions, cf., e.g., 关10兴兲 be of the form 共13兲, with a strictly convex ␾ .

011107-11