Finding Maximum Likelihood Estimator (MLE)

Performing MLE

1. $L(\theta )$
2. $\ln [L(\theta )]$
3. $\frac{d}{d\theta }\ln [L(\theta )]$
4. Solve for $\theta$ in $\frac{d}{d\theta }\ln [L(\theta )]=0$

$$\begin{array}{rl}L(\theta )& =\prod _{i=1}^{n}f(x_i;\theta )\\ \frac{d}{d\theta }\ln [L(\theta )]& =0\end{array}$$

Note

MLE can be extended from 1 parameter to multiple parameters, as shown in this example.

Warning

Step $(2)$ is done because the logarithm turns a product of probabilities into a sum, making the derivative of the function usually easier to compute on step $(3)$.

Note that depending on the form of $L(\theta )$, applying step $(2)$ might overcomplicate the equation instead. In such cases, step $(2)$ may be skipped, and we would maximize $L(\theta )$ instead of $\ln [L(\theta )]$.

About MLE

Imagine you have a dataset, and you know it’s generated by some underlying process (distribution) that depends on an unknown parameter $\theta$. MLE helps you find the best estimate of $\theta$, denoted by $\hat{\theta }$, that maximizes the likelihood of observing the given data.

Read further: https://www.youtube.com/watch?v=XepXtl9YKwc

For example, Suppose you’re working in a factory that produces light bulbs. You want to estimate the probability $p$ that a randomly selected bulb is defective. You’ve observed that out of 100 randomly sampled bulbs, 5 are defective.

You recognize that the samples comes from binomial distribution, which has an MLE estimate $\hat{p}=k/n=5/100=0.05$

So, based on the data, the MLE estimate suggests that the probability of a bulb being defective is $5\%$

Example


Let $X_1,X_2,\dots ,X_n$ represent a random sample from the distribution having hte following probability density function. $f(x;\theta )=\frac{{\theta }^x{e}^{-\theta }}{x!}$, $x=0,1,2,\dots ;0\le \theta <\infty$, zero elsewhere, $f(0;0)=1$.

Find $\hat{\theta }$, the mle of $\theta$.

$$\begin{array}{rl}L(\theta )& =\prod \frac{{\theta }_i^x{e}^{-\theta }}{x_i!}\\ \ln [L(\theta )]& =\sum x_i\ln \theta -n\theta \ln e-\sum \ln x_i!\\ \frac{d}{d\theta }\ln [L(\theta )]& =\frac{\sum x_i}{\theta }-n=0\\ ⟺& \theta =\frac{\sum x_i}{n}\end{array}$$

So, the MLE of $\theta$, which we will denote $\hat{\theta }$, is $\frac{\sum x_i}{n}$