A Reality check for Data Snooping Halbert White (2000, Econometrica)
Pedro H. C. Sant’Anna Universidad Carlos III de Madrid
April 10th, 2012
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
General idea of the presentation
Data Snooping occurs when a given data seet is used more than once for purposes of inference or model selection “Good” results might be due to pure chance rather than to any merit inherent in the method Usual problem in Time Series Propose a method to test the null that the best model has no superior predictive power than a benchmark model
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Introduction
“Good” forecasting model is obtained by extensive specification search: might be just luck! Analogy: flip a sufficiently large number of coins –> a coin that always come up heads can emerge with positive probability Practical Problem: Investment advisory service that includes the past performance as part of their solicitation Is the past performance the result of skill or luck?
These are examples of “data snooping”, or data mining.
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Introduction Data snooping should be avoided, however for time series data, there is no other choice Only a single history of a given phenomenon of interest is available
Several studies have looked to the problem of data mining, specially for inference purposes. Problem of pre-testing, for example
However, no studies have addressed the problem of testing the null that the best model selected has no predictive superiority over a bench-mark model This would permit data snooping to be undertaken with some degree of confidence that one will not mistake results that could have been generated by chance for genuinely good results Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Introduction The Null hypothesis is formulated as a multiple hypothesis, the intersection of l one-sided hypothesis Bounds of p − value for tests of the null can be constructed using the Bonferroni inequality and its improvements via the union-intersection principle However, those methods are not designed to test a big number of models. The goal of the paper is a method that does not rely on such bounds, but delivers asymptotically valid p − values.
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
The Basic Framework
Predictions are to be made for n periods, indexed by R through T T = R +n−1
Forecast horizon = τ The first forecast is based on βˆ R ,using observations 1 through R. The next forecast uses βˆ R +1, and so on, until the final forecast is based on βˆ T
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
The Basic Framework Test a hypothesis about a l × 1 vector of moments, E (f ∗ ) f ∗ ≡ f (Z , β∗ ) is a l × 1 vector with elements fk∗ ≡ fk (Z , β∗ ), for a random vector Z and parameters β∗ ≡ p lim βˆ T Typically Z is a vector of dependent variables, say Y , and predict variables, say X Test is based on the l × 1 statistic 1 T f¯ ≡ ∑ fˆt +τ n t =R where fˆt +τ ≡ f (Zt +τ , βˆ t ) and the observed data generated by {Zt }, a stationary strong mixing sequence having marginal distributions identical to that of Z , with predictors variables of Zt +τ available at time t Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
The Basic Framework
For suitable choices of f , the null of our interest is H0 : E ( f ∗ ) ≤ 0 H0 is the null of no predictive superiority over a bench mark model Some Examples
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Example 2.1 - superior predictability (non-nested models)
Example Test whether a set of variables has predictive power superior to that of benchmark regression model in terms of MSE H0 = MSE (1) ≥ MSE (0) ∴ −MSE (1) + MSE (0) ≤ 0 0 ˆ fˆt +1 = − yt +1 − X1,t +1 β1,t
2
0 ˆ + yt +1 − X0,t +1 β0,t
2
0 where βˆ s are estimated using OLS based on different regressors.
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Example 2.2 - Financial Market trading strategy
Example Test whether a financial market trading strategy yields returns superior to a benchmark strategy, on average fˆt +1 = log [1 + yt +1 S1 (X1,t +1 , β1∗ )] − log [1 + yt +1 S0 (X0,t +1 , β0∗ )] . Here, yt +1 represents per period returns and S0 and S1 are “signal” functions that convert indicators (X0,t +1 and X1,t +1 ) and given parameters into market positions. You can think about long-neutral-short position.
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Example 2.3 - General performance of a model
Example Test generally whether a given model is superior to a benchmark fˆt +1 = log L1 (yt +1, X1,t +1 , βˆ 1,t ) − log L0 (yt +1, X0,t +1 , βˆ 0,t ). where, log Lk (yt +1, Xk,t +1 , βˆ 1,t ) is the predictive log-likelihood for a predictive model k, based on the quasi-maximum likelihood estimator βˆ k,t
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
More than one model Now, suppose you want to compare l models with the benchmark model. Then, the fˆt +1 will be l × 1 vector with components fˆk,t +1 = log Lk (yt +1, X1,t +1 , βˆ k,t ) − log L0 (yt +1, X0,t +1 , βˆ 0,t )., k = 1, . . . ,
We select the model with the best model selection criterion, so the appropriate null is that the best model is no better than the benchmark: H0 : max E (fk∗ ) ≤ 0 k =1...,l
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
The Basic Theory The method developed here uses West’s (1996) results and can be applied provided that d
n1/2 (f¯ − E (f ∗ )) → N (0.Ω), as T → ∞ and Ω(l × l ) is " Ω = lim var T →∞
1 T √ ∑ f ( Zt + τ , β ∗ ) n t =R
#
provided that either
Pedro Sant’Anna (UC3M)
∂ f (Z , β ∗ ) ∂β
F
≡ E
n R
→ 0 as T → ∞
A Reality check for Data Snooping
=0
April 10th, 2012
First result
The first result of the paper establishes that selecting the models with the best predictive model selection criterion does indeed identify the best model, when there is one
Theorem d
Suppose that n1/2 (f¯ − E (f ∗ )) → N (0.Ω), as T → ∞, for Ω positive semi-definite. (a) If E (f ∗ ) > 0 for some 1 ≤ k ≤ l, then, for any 0 ≤ c < E (f ∗ ), P (f¯k > c ) → 1. (b) if l > 1 and E (f1∗ ) > E (fk∗ ), ∀k = 2, . . . , l, then P (f¯1 > f¯k ) → 1.
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Asymptotic distribution of the test
A test of H0 for the predictive model selection criterion follows
Theorem d
Suppose that n1/2 (f¯ − E (f ∗ )) → N (0.Ω), as T → ∞, for Ω positive semi-definite. Then, as T → ∞ d
(a) maxk =1,...,l n1/2 (f¯ − E (f ∗ )) → Vl ≡ maxk =1,...,l Zk . d (b) mink =1,...,l n1/2 (f¯ − E (f ∗ )) → Wl ≡ mink =1,...,l Zk .
where Z is an l × 1 vector with components Zk , k = 1 . . . , l distributed as N (0.Ω)
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Asymptotic distribution of the test Given asymptotic normality, the conclusion holds regardless of whether the null is true or not. Enforce the null using the least favorable to the alternative E (fk∗ ) = 0 for all k. The behavior of the predictive model selection criterion for the best model, say V¯ l ≡ max n1/2 f¯k k =1,...,l
is thus known under the element of the null least favorable case, for T large enough, allowing us to compute asymptotic p − values. “Reality check”: any procedure which can construct valid p-values for the H0 : E (f ∗ ) ≤ 0 Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Procedure to compute the p-values The distribution of the extreme value of a vector of correlated normals is not known. White proposes 2 options: Monte Carlo 1
“Monte Carlo Reality check” : Compute a consistent estimator of Ω, (based on block resampling(or subsampling)) and then samples ˆ ) and obtain the desired p-values. from the N (0.Ω
2
“Bootstrap Reality Check”: For suitably chosen random indexes θ (t ), the resampled statistic is computed as 1 T ˆ∗ f¯ ∗ ≡ ft +τ , n t∑ =R
fˆt∗+τ ≡ f (Zθ (t )+τ , βˆ θ (t ) ),
t = R, . . . , T
Need to apply bootstrap procedures valid for strong mixing time series: Moving Blocks bootstrap, tapered block bootstrap, and stationary bootstrap (the one used for White) Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Bootstrap Reality Check The Bootstrap Reality Check p − value for the predictive model selection statistic, V¯ l , can be immediately obtained from the quantiles of V¯ l∗ ≡ max n1/2 (f¯k∗ − f¯k ) . k =1,...,l
White have proved the validity of the bootstrap. He also have shown that the same procedure can be applied to test the null: H0 : max g (E (hk∗ )) ≤ g (E (h0∗ )) k =1...,l
where g is continuously differentiable, such that the Jacobian is nonzero. White also provide details on how to implement the bootstrap. Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example
Illustrate the Reality check by applying it to forecasting daily returns of the S&P 500 index one day ahead. Sample: March 29, 1988 thorough May 31,1994 R=803 and T=1560 –> n=758 N period: June 3 ,1991 - May 31 1994
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example Check if excess returns are forecastable Consider a collection of linear models that use “technical” indicators Brock, Lakonishik and LeBaron (1992) suggest that these indicators had predictive ability
Use 29 technical indicators Construct forecasting using linear models including a constant, and exactly 3 predictors from the 29 available Total of 3,654 models Benchmark model: only a constant - simple efficient market hypothesis Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example
Technical indicators are based on : 1
Lagged returns
2
“momentum measures”: (pt −1 − pt −1−j )/pt −1−j
3
“local trend” measures: slopes from regressing the price on a constant and time trend for previous days
4
“relative strength indexes”: % of past days that returns are positive
5
“Moving average oscillators”: difference of 2 moving averages of closing prices from previous days.
OLS estimators
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example
Evaluate method: 1
Negative Mean square prediction error: 2 2 0 ∗0 ˆ ˆ β β + y − X fˆt +1 = − yt +1 − Xk,t t +1 0,t +1 0,t +1 k,t
2
Directional Accuracy h i h i ∗0 ∗0 ˆ ˆ β > 0 − 1 y X β > 0 fˆt +1 = 1 yt +1 Xk,t t +1 0,t +1 0,t +1 k,t
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
An Illustrative Example
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Outline
1
Introduction
2
Theory The Basic Framework The Basic Theory
3
An Illustrative Example
4
Conclusion
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
Conclusion
Data Snooping occurs when a given set of data is used more than once for purposes of inference or model selection When this happens, there is always the possibility that your “best” models is not really “best” “Reality check” provides a simple procedure for testing the null that the best model encountered in a specification search has no predictive superiority over a benchmark model
Pedro Sant’Anna (UC3M)
A Reality check for Data Snooping
April 10th, 2012
