Question

linear statistics modeling and regression

2) Suppose you have multiple regression set up Ynxi XnxpBpxi Sxl and f ~ N(0nx1, σ21.). P Po X(X,X)-X, be the projection matrix on the column space of X. a) Show residual vector, e = (1,-P)Y. Here e is the vector of residuals ei S. b) Show that the variance of e, is 1 - Pi, where P is the i, j th entry of the matrix P c) Show that the sample covariance of Y of the fitted values and e, the residual vector is zero

Answer 1

Suppose you have multiple xegzesss p YA xp 2 entry af the matx p

c)

First, I will assume I have dataset of dependent variables Yi, and independent variables X1i, X2i, X3i,... Xki.

Then, I will fit a linear regression model to that dataset: Y=a + bX1 + Z + e, where Z is a linear combination of all the independent variables from X2 onwards: Z=cX2+dX3+... Z is therefore independent of a and b.

After the model is fitted, i.e. the parameters a, b, c, d... are determined, so that the sum of square of the errors s(a,b,c,d...) = Ʃei^2 = Ʃ(Yi-a-bX1i-Zi)^2 is minimized.

For this, I calculate the partial derivatives of s for a,b,c,d.... and set them to equal 0.

I find that ∂s/∂a = -2 Ʃ(Yi-a-bX1i-Zi). Therefore Ʃ(Yi-a-bX1i-Zi) = Ʃei = 0, and E[e]=e~= 0

∂s/∂b = -2 Ʃ X1i (Yi-a-bX1i-Zi). Therefore Ʃ X1i (Yi-a-bX1i-Zi) = Ʃ X1i ei= 0

Then, Ʃ (ei-e~)(X1i-X1~) = Ʃ (eiX1i - eiX1~ - e~X1i + e~X1~) = ƩeiX1i - ƩeiX1~ - Ʃe~X1i + Ʃe~X1~ = 0 - X1~Ʃei -Ʃ0 + Ʃ0 = -X1~0 = 0 Therefore Cov(e,X1) = 0, which is what I wanted to prove.

X1 is replacable with any of the other X:s that are all combined in Z, and repeat the above analysis. Because the regression function is symmetric for all the predictor variables, I would then find that cov(e,Xk)=0 for any k.