Fisher Information

Definition

Let

• $X$ : Random variable, with
  • pdf $f(x;\theta )$, for $\theta \in \Omega$
• $S(X;\theta )$ : Score Function

Then the Fisher information is defined as:

$$\begin{array}{rl}I(\theta )& =E\left[S(X;\theta {)}^2\right]\\ & =E\left[{\left(\frac{\partial }{\partial \theta }\ln f(X;\theta )\right)}^2\right]\\ & =-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\ln f(X;\theta )\right]\end{array}$$

Remark

Important

The bigger the Fisher information $I(\theta )$, the better the information obtained about $\theta$.

This equation is derived under Regularity conditions: $I(\theta )=-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\ln f(X;\theta )\right]$

For a random sample $X_1,X_2,\dots ,X_n$, the Fisher information is:

I_n(\theta) = nI(\theta) = -nE\left[\frac{\partial^2}{\partial \theta^2} \ln f(X;\theta)\right] $$ > [!note] > > Fisher information measures the amount of information that the sample carries about the parameter $\theta$. It is the weighted mean of $\left(\frac{\partial}{\partial \theta} \ln f(x;\theta)\right)^2$, where the weights are given by the pdf $f(x;\theta)$. > > The greater these derivatives are on average, the more information we get about $\theta$. If the derivatives were equal to zero (so that $\theta$ would not be in $\ln f(x;\theta)$), there would be zero information about $\theta$.