Question

Considering multiple linear regression models, we compute the regression of Y, an n x 1 vector, on an n x (p+1) full rank matrix X. As usual, H = X(XT X)-1 XT is the hat matrix with elements hij at the ith row and jth column. The residual is e; = yi - Ýi. (a) (7 points) Let Y be an n x 1 vector with 1 as its first element and Os elsewhere. Show that the elements of the vector of fitted values from the regression of Y on X are the hij, j = 1, 2,..., n. Show that the first element of the vector of residuals is 1 - hii, and the other elements are -hij, j> 1.

Answer 1

Answer:-

Given that:-

For a linear model where is full tank
Y = X +
least square estimator of $\beta$ is
Н.
where  
H
is
X(XTX)-?x"
which is symmetric.

For a singular matrix A we have
A-1A=I (A-1A) = 1 A(4-1) =

(1-P) = (AT)-1.

This result is in showing H is symmetric ,matrix.

$X_{n\times (p+1)}$ in forall rank matrix and,

H = X(X"X)-1XT

(a)$\underset{\sim}{y}=\begin{pmatrix} 1\\ 0\\ 0\\ \vdots \\ 0 \end{pmatrix}_{n\times 1}$

Now, $\underset{\sim}{\widehat{y}}=H\underset{\sim}{y}=X(X^{T}X)^{-1}X^{T}\underset{\sim}{y}$

So,$H\underset{\sim}{y}=\bigl(\begin{smallmatrix} h_{11} & h_{12}& \cdots & h_{1n}\\ h_{21}&h_{22} & \cdots &h_{2n} \\ \vdots & & & \\ h_{n1}& h_{n2} & \cdots & h_{nn} \end{smallmatrix}\bigr)\begin{pmatrix} 1\\ 0\\ 0\\ \vdots \\ 0 \end{pmatrix}$

$=\begin{pmatrix} h_{11}\\ h_{21}\\ \vdots \\ \vdots \\ h_{n1} \end{pmatrix}$

$=\begin{pmatrix} h_{11}\\ h_{12}\\ \vdots \\ \vdots \\ h_{1n} \end{pmatrix}$

So, $e_{i}=y_{i}-\widehat{y_{i}}=\begin{pmatrix} 1\\ 0\\ \vdots \\ \vdots \\ 0 \end{pmatrix}-\begin{pmatrix} h_{11}\\ h_{12}\\ \vdots \\ \vdots \\ h_{1n} \end{pmatrix}$

1-h11 -h12 = -hin

but we know that H is symmetric So, $h_{12}=h_{21}$

$H^{T}=(X(X^{T}X)^{-1}X^{T})^{T}$

$=(X^{T})^{T}((X^{T}X)^{-1})^{T}X^{T}$

$=X(X^{T}X)^{-1}X^{T}$

$A^{-1}A=I\Rightarrow (A^{-1}A)^{T}=I$

$\Rightarrow A^{T} (A^{-1})^{T}=I$

$\Rightarrow (A^{-1})^{T}=(A^{T} )^{-1}$

for sample invertible matrix A so, $((X^{T}X)^{-1})^{T}=((X^{T}X)^{T})^{-1}$

$=(X^{T}X)^{-1}$

(b)$\widehat{\beta }_{(i)}=\widehat{\beta }+\frac{(X^{T}X)^{-1}x_{i}e_{i}}{1-h_{ii}}$

Now, $\underset{\sim}{x_{i}}\widehat{\beta }_{(i)}=\underset{\sim}{x_{i}}(\widehat{\beta }+\frac{(X^{T}X)^{-1}\underset{\sim}{x_{i}^{T}}e_{i}}{1-h_{ii}})$

$=\underset{\sim}{x_{i}}\widehat{\beta }+\frac{\underset{\sim}{x_{i}}(X^{T}X)^{-1}\underset{\sim}{x_{i}^{T}}e_{i}}{1-h_{ii}})$

$H=X(X^{T}X)^{-1}X^{T}=\begin{pmatrix} \underset{\sim}{x_{1}}\\ \underset{\sim}{x_{2}}\\ \vdots \\ \underset{\sim}{x_{n}} \end{pmatrix}(X^{T}X)^{-1}(\underset{\sim}{x_{1}^{T}}........\underset{\sim}{x_{n}^{T}})$$=\bigl(\begin{smallmatrix} \underset{\sim}{x_{1}} & (X^{T}X)^{T} &\underset{\sim}{x_{1}^{T}} & \cdots & \underset{\sim}{x_{1}} &(X^{T}X) ^{-1} &\underset{\sim}{x_{n}^{T}} \\ \vdots & & & & & & \\ \underset{\sim}{x_{n}}& (X^{T}X) ^{-1}& \underset{\sim}{x_{1}}^{T} &\cdots & \underset{\sim}{x_{n}} &(X^{T}X)^{T} \underset{\sim}{x_{n}}^{T}& \end{smallmatrix}\bigr)$

$=\begin{pmatrix} \underset{\sim}{x_{1}}\\ \underset{\sim}{x_{2}}\\ \vdots \\ \underset{\sim}{x_{n}} \end{pmatrix}$ So, $h_{11}=\underset{\sim}{x_{1}}(X^{T}X)^{-1}\underset{\sim}{x_{1}^{T}}....h_{in}=\underset{\sim}{x_{1}}(X^{T}X)^{-1}\underset{\sim}{x_{n}^{T}}$

$y_{i}-x_{i}\widehat{\beta }_{(i)}=y_{i}-x_{i}\widehat{\beta }-\frac{\underset{\sim}{x_{i}}(X^{T}X)^{-1}\underset{\sim}{x_{i}^{T}}e_{i}}{1-h_{ii}}$$=e_{i}-\frac{h_{ii}e_{i}}{1-h_{ii}}$

So, there is problem in $\widehat{\beta }_{(i)},$ it has to be

$\widehat{\beta }_{(i)}=\widehat{\beta }-\frac{(X^{T}X)^{-1}\underset{\sim}{x_{i}^{T}}e_{i}}{1-h_{ii}}$

Then we have $y_{i}-x_{i}\widehat{\underset{\sim}{\beta }}_{(i)}=e_{i}+\frac{h_{ii}e_{i}}{1-h_{ii}}=\frac{e_{i}}{1-h_{ii}}$