Method of Moments

The method of moments is a method for point estimation that involves equating population moments with sample moments to estimate unknown parameters.

Performing method of moments

To estimate $k$ unknown parameters ${\theta }_1,{\theta }_2,\dots ,{\theta }_k$:

1. Express population moments in terms of parameters Write the first $k$ population raw moments as functions of the parameters: ${\mu }_1^{\prime }=E(X)=g_1({\theta }_1,{\theta }_2,\dots ,{\theta }_k)$ ${\mu }_2^{\prime }=E(X^2)=g_2({\theta }_1,{\theta }_2,\dots ,{\theta }_k)$ $⋮$ ${\mu }_k^{\prime }=E(X^k)=g_k({\theta }_1,{\theta }_2,\dots ,{\theta }_k)$

2. Set population moments equal to sample moments $g_1({\theta }_1,{\theta }_2,\dots ,{\theta }_k)=m_1^{\prime }$ $g_2({\theta }_1,{\theta }_2,\dots ,{\theta }_k)=m_2^{\prime }$ $⋮$ $g_k({\theta }_1,{\theta }_2,\dots ,{\theta }_k)=m_k^{\prime }$

   where $m_j^{\prime }=\frac{1}{n}{\sum }_{i=1}^n{X}_i^j$ is the $j$-th sample raw moment.

3. Solve the system of equations: Solve for ${\theta }_1,{\theta }_2,\dots ,{\theta }_k$ to obtain the method of moments estimators ${\hat{\theta }}_1,{\hat{\theta }}_2,\dots ,{\hat{\theta }}_k$.

Common approach using raw and central moments

• First raw moment: ${\mu }_1^{\prime }=E(X)=\bar{X}$
• Central moments: ${\mu }_k=E[(X-\mu {)}^k]=\frac{1}{n}{\sum }_{i=1}^n(X_i-\bar{X}{)}^k$ for $k=2,3,\dots$

For many distributions, ${\mu }_1^{\prime }=\mu$ and ${\mu }_2={\sigma }^2$ are sufficient.

Properties of Method of Moments Estimators

• Consistency: Under regularity conditions, method of moments estimators are consistent
• Simplicity: Often easier to compute than maximum likelihood estimators
• Not always efficient: May not achieve the Rao-Cramer lower bound
• Existence: May not always exist or be unique

Comparison with Maximum Likelihood Estimation

Method of Moments	Maximum Likelihood
Equates sample and population moments	Maximizes likelihood function
Usually simpler calculations	Often more complex calculations
May be less efficient	Asymptotically efficient
Always consistent (under regularity conditions)	Consistent and asymptotically normal

Important formulas

• Mean
• Variance
• Moments
• Discrete Distributions
• Continuous Distributions

Examples

Example 1: Normal Distribution with Unknown Mean

Let

• $X_1,X_2,\dots ,X_n\sim N(\mu ,{\sigma }^2)$, with
  • $\mu$ unknown
  • ${\sigma }^2$ known

We know that

• Population moment is $E[X]=\mu$, and
• Sample moment is $m_1^{\prime }=\frac{1}{n}{\sum }_{i=1}^n{X}_i=\bar{X}$.

Step 1: Express population moment in terms of parameter: ${\mu }_1^{\prime }=E[X]=\mu$

Step 2: Set equal to sample moment: $E[X]=\mu =m_1^{\prime }=\bar{X}$

Step 3: Solve: $\hat{\mu }=\bar{X}$

Example 2: Normal Distribution with Two Unknown Parameters

Let

• $X_1,X_2,\dots ,X_n\sim N(\mu ,{\sigma }^2)$, with
  • $\mu$ unknown
  • ${\sigma }^2$ unknown

Step 1: Express population moments in terms of parameters:

$$\begin{array}{rl}{\mu }_1^{\prime }& =E[X]\\ & =\mu \\ & \\ {\mu }_2^{\prime }& =E[X^2]\\ & =\text{Var}(X)+[E(X){]}^2\\ & ={\sigma }^2+{\mu }^2\end{array}$$

Step 2: Set equal to sample moments:

$$\begin{array}{rl}\mu & =m_1^{\prime }=\bar{X}\\ {\sigma }^2+{\mu }^2& =m_2^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^2\end{array}$$

Step 3: Solve: From the first equation: $\hat{\mu }=\bar{X}$

Substituting into the second equation: ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n{X}_i^2-{\bar{X}}^2$

Using the identity ${\sum }_{i=1}^n{X}_i^2={\sum }_{i=1}^n(X_i-\bar{X}{)}^2+n{\bar{X}}^2$: ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\bar{X}{)}^2$

Using the identity ${\sum }_{i=1}^n{X}_i^2={\sum }_{i=1}^n(X_i-\bar{X}{)}^2+n{\bar{X}}^2$: ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\bar{X}{)}^2$

Example 3: Shifted Exponential Distribution

Let

• $X_1,X_2,\dots ,X_n$ be a random sample from a distribution with pdf:

f(x; \theta) =\begin{cases} e^{-(x-\theta)}, & \quad \theta \leq x < \infty, \quad -\infty < \theta < \infty \ 0 & \quad \text{otherwise} \end{cases}

- $\theta$ unknown **Step 1:** Express population moment in terms of parameter: For the shifted exponential distribution, let $Y = X - \theta$, so $Y \sim \text{Exponential}(1)$. Since $X = Y + \theta$: $$\mu_1' = E[X] = E[Y + \theta] = E[Y] + \theta = 1 + \theta$$ **Step 2:** Set equal to sample moment: $$\theta + 1 = m_1' = \frac{1}{n}\sum_{i=1}^n X_i = \bar{X}$$ **Step 3:** Solve: $$\hat{\theta} = \bar{X} - 1$$