Assignment 1 Can Yang Department of Mathematics Hong Kong Baptist University February 25, 2015 1. Suppose that we observe a quantitative response Y and p different predictors, X1 , X2 , . . . , Xp . We assume that there is a true relationship between Y and X = (X1 , X2 , . . . , Xp ), which can be written as Y = f (X) + ϵ,

(1)

where f is some fixed but unknown function of X1 , . . . , Xp , and ϵ is a random error term, which is independent of X with E(ϵ) = 0 and V ar(ϵ) = σϵ2 . Based on n training observations {(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn )}, we obtain the estimate fˆ and thus have the prediction fˆ(x1 ), fˆ(x2 ), . . . , fˆ(xn ). Let x0 be a testing observation. To characterize how well we will do on samples from outside the training data set, we define Err(x0 ) as the test error as Err(x0 ) = E[(Y − fˆ(x0 ))2 |X = x0 ], (2) where Y = f (x0 ) + ϵ and the expectation is taken with respect to all random variables (You need to tell where the randomness comes from). Show that Err(x0 ) = σϵ2 + (f (x0 ) − Efˆ(x0 ))2 + E[(Efˆ(x0 ) − fˆ(x0 ))2 ],

(3)

where (f (x0 ) − Efˆ(x0 ))2 is the square of model bias and E[(Efˆ(x0 ) − fˆ(x0 ))2 ] is the model variance. 2. Consider a linear model y = xβ + ϵ,

(4)

where x is an n × p matrix of observable random variables, β is a p × 1 vector of parameters (fixed), ϵ is an n × 1 random vector and Y is an n × 1 vector of observable random variables. Let yi and ϵi be the i-th component of y and ϵ, respectively. Model (4) can also be written as yi = xi β + ϵi ,

(5)

where xi is the i-th row of matrix x, ϵi is identically independent distributed with E(ϵi ) = 0 and V ar(ϵi ) = σϵ2 . ˆ = (xT x)−1 xT y. By assuming that ϵ is independent of x, please show that Let β ˆ ˆ (a) E(β|x) = β. ˆ ˆ (b) Cov(β|x) = σϵ2 (xT x)−1 . Hint: Note that A matrix x is “random” if some of the entries xij are random variables rather than constants. This is an additional complication. People often prefer to condition on x. Then x is fixed; expectations, variances, and covariances are conditional on x. 3. Apply Ridge regression and Lasso to the auto data and use cross-validation to evaluate the model performance. You are allowed to use available functions in Matlab or R to do this. Specifically, the response variable is “mpg” and the predictors are “cylinders”, “displacement”, “horsepower”, “weight”,“acceleration” and “year”. 4. Consider the Lotka-Volterra equations dx = αx − βxy dt dy = δxy − γy. dt 1

(6)

Use the Matlab function “ode45” to obtain the solution of Lotka-Volterra equation (6) for t ∈ [0, 20] with parameters α = 1, β = 0.05, δ = 0.02, γ = 0.5 and the initial condition x(0) = 20, y(0) = 10. 5. Implement the Standard Linear Regression (Ordinary least square estimate) in Matlab. You are not allowed to call linear regression functions, but only allowed to use basic Matlab functions, such as functions for linear algebra, distribution functions. Your function should be written as ˆ tval, pval] = LinearReg Y ourID(x, y), [β, where y ∈ Rn×1 is the response (n is the sample size), x ∈ Rn×M is the design matrix corresponding to M ˆ is the OLS estimate of regression coefficients, tval is a M -dim vector corresponding to the t values variables, β of M variables, pval is a M -dim vector corresponding to the p-values of M variables. Please try to make sure that your function works for n ≈ 2, 500 and M ≈ 2, 000. 6. Assigned reading on Big Data. Please refer to the course website for the following two: ˆ 1. The rise of Big Data. ˆ 2. “Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts”; please also check more comments from Michael Jordan.

Write down your own thoughts about Big Data in 200-250 words. 7. Suggested self-learning: properties of positive definite matrices, e.g., ˆ Triangular factorization: If A is a positive definite matrix, then A can be factored as LDLT , where L is lower triangular with 1’s along the diagonal and D is a diagonal matrix whose diagonal entries are all positive. ˆ Cholesky decomposition: If A is a positive definite matrix, then A can be factored as LLT , where L is lower triangular with positive diagonal elements. ˆ ...

The first five exercises in this assignment is due to March 18, 2015. The sixth one is due to April 5, 2015. You are NOT required to submit anything for the seventh exercise. The Rule for assignment: NO copy from others.

2