6.1 Maximum Likelihood Estimation

<< 5.4 Extensions to Multivariate Distributions.md | 6.2 Rao-Cramér Lower Bound and Efficiency.md >>

Assumptions 6.1.1: Regularity conditions

• $(R0)$ : The cdfs are distinct; i.e., $\theta \ne {\theta }^{\prime }⟹F(x_i;\theta )\ne F(x_i;{\theta }^{\prime })$
• $(R1)$ : The pdfs have common support for all $\theta$
• $(R2)$ : The point ${\theta }_0$ is an interior point in $\Omega$

TODO: What is an interior point? #TODO: What is $\Omega$

Theorem 6.1.1

Assume

• ${\theta }_0$ : True parameter
• $E_{{\theta }_0}\left[\frac{f(X_i;\theta )}{f(X_i;{\theta }_0)}\right]$ exists

Then, under assumptions (R0) and (R1)

$$\underset{n\to \infty }{\lim }{P}_{{\theta }_0}[L({\theta }_0,\mathbf{X})>L(\theta ,\mathbf{X})]=1,\forall \theta \ne {\theta }_0$$

Definition 6.1.1: Maximum Likelihood Estimator

Let

• $\mathbf{X}$ : Observed data
• $\theta$ : Parameter
• $L(\theta ;\mathbf{X})$ : Likelihood function

If $\hat{\theta }=\mathrm{Argmax}L(\theta ;\mathbf{X})$

Then $\hat{\theta }$ is a maximum likelihood estimator (mle) of $\theta$

Link to original

Theorem 6.1.2

Let

• $X_1,\dots ,X_n$ : Random sample, with
  • pdf $f(x;\theta ),\theta \in \Omega$
• Let $\eta =g(\theta )$ : Parameter of interest

If $\hat{\theta }$ is the mle of $\theta$

Then $g(\hat{\theta })$ is the mle of $\eta =g(\theta )$

Theorem 6.1.3

Assume

• $X_1,\dots ,X_n$ satisfies all regularity conditions
• ${\theta }_0$ : True parameter
• $f(x;\theta )$ : differentiable with respect to $\theta$ in $\Omega$

Then the likelihood equation $\frac{\partial }{\partial \theta }L(\theta )=0$, or equivalently $\frac{\partial }{\partial \theta }l(\theta )=0$ has a solution ${\hat{\theta }}_n$ such that ${\hat{\theta }}_n\stackrel{P}{\to }{\theta }_0$

Corollary 6.1.1

Assume

• $X_1,\dots ,X_n$ satisfies all regularity conditions
• ${\theta }_0$ : True parameter
• $f(x;\theta )$ : differentiable with respect to $\theta$ in $\Omega$

If the likelihood equation has the unique solution $\widehat{{\theta }_0}$

Then ${\hat{\theta }}_0$ is a consistent estimator of ${\theta }_0$

Exercise

MLE on normal distribution

Let

• $X_1,X_2,\dots ,X_n$ : Random sample, with
  • Normal distribution
  • Unknown mean $\mu =\theta$
  • Known variance ${\sigma }^2=1$.

Find the maximum likelihood estimator (MLE) of $\theta$ ($\mu$).

Answer

The pdf of $X_n\sim N(\theta ,1)$ is: $f(x_i;\theta )=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x_i-\theta {)}^2}$

The likelihood function is:

$$\begin{array}{rl}L(\theta )& =\prod _{i=1}^{n}f(x_i;\theta )\\ & =\prod _{i=1}^n\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x_i-\theta {)}^2}\\ & =\frac{1}{(2\pi {)}^{n/2}}{e}^{-\frac{1}{2}{\sum }_{i=1}^n(x_i-\theta {)}^2}\end{array}$$

Solving $\ln L(\theta )$:

$\ln L(\theta )=-\frac{n}{2}\ln (2\pi )-\frac{1}{2}{\sum }_{i=1}^n(x_i-\theta {)}^2$

Solving $\frac{\partial }{\partial \theta }\ln L(\theta )=0$:

$$\begin{array}{rl}\frac{d}{d\theta }\ln L(\theta )& =\frac{d}{d\theta }\left[-\frac{n}{2}\ln (2\pi )-\frac{1}{2}\sum _{i=1}^n(x_i-\theta {)}^2\right]\\ 0& =0-\frac{1}{2}\sum _{i=1}^n\frac{d}{d\theta }(x_i-\theta {)}^2\\ 0& =-\frac{1}{2}\sum _{i=1}^{n}2(x_i-\theta )(-1)\\ 0& =\sum _{i=1}^n(x_i-\theta )\\ 0& =\sum _{i=1}^n{x}_i-n\theta \\ n\theta & =\sum _{i=1}^n{x}_i\\ \theta & =\frac{1}{n}\sum _{i=1}^n{x}_i\\ \theta & =\bar{x}\end{array}$$

$\therefore$ The maximum likelihood estimator of $\mu$ is:

$\theta =\frac{1}{n}{\sum }_{i=1}^n{X}_i=\bar{X}$

This shows that the sample mean is the MLE of the population mean for a normal distribution.

MLE on normal distribution with multiple parameters

Let

• $X_1,X_2,\dots ,X_n$ : Random sample, with
  • Normal distribution
  • Unknown mean $\mu ={\theta }_1$
  • Unknown variance ${\sigma }^2={\theta }_2$

Find the maximum likelihood estimators (MLE) of ${\theta }_1$ and ${\theta }_2$.

Answer

The pdf of $X_i\sim N({\theta }_1,{\theta }_2)$ is: $f(x_i;{\theta }_1,{\theta }_2)=\frac{1}{\sqrt{2\pi {\theta }_2}}{e}^{-\frac{1}{2{\theta }_2}(x_i-{\theta }_1{)}^2}$

The likelihood function is:

& = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\theta_2}} e^{-\frac{1}{2\theta_2}(x_i - \theta_1)^2} \\ & = \frac{1}{(2\pi\theta_2)^{n/2}} e^{-\frac{1}{2\theta_2}\sum_{i=1}^n(x_i - \theta_1)^2} \end{align}

Solving $\ln L({\theta }_1,{\theta }_2)$:

$\ln L({\theta }_1,{\theta }_2)=-\frac{n}{2}\ln (2\pi )-\frac{n}{2}\ln ({\theta }_2)-\frac{1}{2{\theta }_2}{\sum }_{i=1}^n(x_i-{\theta }_1{)}^2$

Finding MLE for ${\theta }_1$ (mean):

Solving $\frac{\partial }{\partial {\theta }_1}\ln L({\theta }_1,{\theta }_2)=0$:

0 & = 0 - 0 - \frac{1}{2\theta_2}\sum_{i=1}^n \frac{\partial}{\partial \theta_1}(x_i - \theta_1)^2 \\ 0 & = -\frac{1}{2\theta_2}\sum_{i=1}^n 2(x_i - \theta_1)(-1) \\ 0 & = \frac{1}{\theta_2}\sum_{i=1}^n (x_i - \theta_1) \\ 0 & = \sum_{i=1}^n x_i - n\theta_1 \\ n\theta_1 & = \sum_{i=1}^n x_i \\ \hat{\theta_1} & = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x} \end{align} $$ **Finding MLE for $\theta_2$ (variance):** Solving $\frac{\partial}{\partial \theta_2}\ln L(\theta_1, \theta_2)=0$: $$ \begin{align} \frac{\partial}{\partial \theta_2} \ln L(\theta_1, \theta_2) & = \frac{\partial}{\partial \theta_2}\left[-\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln(\theta_2) - \frac{1}{2\theta_2}\sum_{i=1}^n(x_i - \theta_1)^2\right] \\ 0 & = 0 - \frac{n}{2} \cdot \frac{1}{\theta_2} - \sum_{i=1}^n(x_i - \theta_1)^2 \cdot \frac{\partial}{\partial \theta_2}\left(\frac{1}{2\theta_2}\right) \\ 0 & = -\frac{n}{2\theta_2} - \sum_{i=1}^n(x_i - \theta_1)^2 \cdot \left(-\frac{1}{2\theta_2^2}\right) \\ 0 & = -\frac{n}{2\theta_2} + \frac{1}{2\theta_2^2}\sum_{i=1}^n(x_i - \theta_1)^2 \\ \frac{n}{2\theta_2} & = \frac{1}{2\theta_2^2}\sum_{i=1}^n(x_i - \theta_1)^2 \\ n\theta_2 & = \sum_{i=1}^n(x_i - \theta_1)^2 \\ \hat{\theta_2} & = \frac{1}{n}\sum_{i=1}^n(x_i - \theta_1)^2 \end{align} $$ Substituting $\hat{\theta_1} = \bar{x}$: $$\hat{\theta_2} = \frac{1}{n}\sum_{i=1}^n(x_i - \bar{x})^2$$ $\therefore$ The maximum likelihood estimators are:

\begin{align} \hat{\theta_1} & = \frac{1}{n}\sum_{i=1}^n X_i && = \bar{x} \ \hat{\theta_2} & = \frac{1}{n}\sum_{i=1}^n(X_i - \bar{X})^2 && =s^2 \end{align}

Note that $\hat{\theta_2}$ is the biased sample variance (dividing by $n$ instead of $n-1$).