Question

2. For this problem, follow the class notes (a) Obtain the Hessian matrix H(0) for the log likelihood function for a sample of size n from a normal distribution N(μ, σ2), as a function of the parameter θ σ2) (b) Obtain the Hessian H(0) at the specific parameter values θ, where θ is the MLE of θ. (c) Show that for any 2-vector xメ0 it holds that rtH(0)x < 0, where rt denotes the transpose of the vector x. What does this imply?

Answer 1

$\text{Let }\textbf{x}=(x_1,\ldots, x_n)\text{ be the samples chosen from Normal distribution}. \\ \text{Hence joint distribution of }\textbf{ x}\text{ is given by } \\ f_{\theta}(\textbf{x})=\left(\frac{1}{2\pi\sigma^2} \right )^{n/2}\exp\left[-\frac{1}{2\sigma^2}\sum_{j=1}^n(x_j-\mu)^2 \right ]\\ \text{Hence log likelihood is given by}\\ L(\theta)=\ln(f_{\theta(\textbf{x})})=-\frac{n}{2}\ln(2\pi)-\frac{n}{2}\sigma^2-\frac{1}{2\sigma^2}\sum_{j=1}^n(x_j-\mu)^2. \\ \text{Now by definition, Hessian is defined as}\\$

$H(\theta)=\begin{bmatrix} \frac{\partial ^2 L}{\partial x_1^2} & \frac{\partial ^2 L}{\partial x_1 \partial x_2} & \ldots & \frac{\partial ^2 L}{\partial x_1 \partial x_n}\\ \frac{\partial ^2 L}{\partial x_2 \partial x_1} & \frac{\partial ^2 L}{\partial x_1 \partial x_2} & \ldots & \frac{\partial ^2 L}{\partial x_2 \partial x_n} \\ \vdots & \vdots & \vdots & \vdots \\ \frac{\partial ^2 L}{\partial x_n \partial x_1} & \frac{\partial ^2 L}{\partial x_n \partial x_2} & \ldots & \frac{\partial ^2 L}{\partial x_n^2} \end{bmatrix}$

Hence diagonal terms are

$-\frac{1}{\sigma^2}\\ \text{and cross diagonal terms are equal to zero}. \\ \text{Hence Hessian matrix for log likehood of normal distribution is }\\ \begin{bmatrix} -\frac{1}{\sigma^2} & 0 & \ldots & 0 \\ 0 & -\frac{1}{\sigma^2} & \ldots & 0 \\ \vdots & \vdots & \vdots & \vdots\\ 0 & 0 &\ldots & -\frac{1}{\sigma^2} \end{bmatrix}$

(b)

Hessian is independent of samples, hence Hessian will be same as in part (a) even for MLE of theta.

(c)

$\text{For any two vector }\textbf{x}=(x_1,x_2),\text{ we have}\\ \textbf{x}^TH \textbf{x}=[x_1 ~x_2]^T\begin{bmatrix} -1/\sigma^2 &0 \\ 0& -1/\sigma^2 \end{bmatrix} [x_1~x_2]\\ =-\frac{x_1^2&plus;x_2^2}{\sigma^2}<0.\\ \text{This implies that MLE achives the minimum of log-likelihood function}$