5.4 Extensions to Multivariate Distributions

<< 5.3 Central Limit Theorem.md | 6.1 Maximum Likelihood Estimation.md >>

TODO: Link the theorems where the theorems below are equivalent to componentwise

Some notations

For a vector $\mathbf{v}\in R^p$, the Euclidean norm of $\mathbf{v}$ is

$$||\mathbf{v}||=\sqrt{\sum _{i=1}^p{v}_i^2}$$

This norm satisfies the usual properties:

1. $\forall \mathbf{v}\in R^p,||\mathbf{v}||\ge 0\wedge ||\mathbf{v}||=0⟺\mathbf{v}=0$
2. $\forall \mathbf{v}\in R^p\wedge a\in R,||a\mathbf{v}||=|a|||\mathbf{v}||$
3. $\forall \mathbf{v},\mathbf{u}\in R^p,||\mathbf{u}+\mathbf{v}||\le ||\mathbf{u}||+||\mathbf{v}||$

We denote the standard basis of $R^p$ by vectors ${\mathbf{e}}_1,\dots ,{\mathbf{e}}_p$, where all components of ${\mathbf{e}}_i$ are $0$ except for the $i$-th component, which is $1$. Then we can write any vector ${\mathbf{v}}^{\prime }=v_1,\dots ,v_p$ as

$$\mathbf{v}=\sum _{i=1}^p{v}_i{\mathbf{e}}_i$$

Lemma 5.4.1

Let ${\mathbf{v}}^{\prime }=(v_1,\dots ,v_p$) : Any vector in $R^p$

Then

$$\begin{array}{rl}|v_j|& \le ||\mathbf{v}||\\ & \le \sum _{i=1}^n|v_i|,\forall j=1,\dots ,p\end{array}$$

Definition 5.4.1: Multivariate converges in probability

Let

• $\mathbf{X}$ : Random vector
• $\{{\mathbf{X}}_n\}$ : Sequence of $p$-dimensional vectors
• $\{X_n\}$ and $\mathbf{X}$ defined on the same sample space

If

$$\underset{n\to \infty }{\lim }P\left[||{\mathbf{X}}_n-\mathbf{X}||\ge ϵ\right]=0,\forall ϵ>0$$

Then

• We say $\{{\mathbf{X}}_n\}$ converges in probability to $\mathbf{X}$
• We write ${\mathbf{X}}_n\stackrel{P}{\to }\mathbf{X}$

Link to original

Theorem 5.4.1

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of $p$-dimensional vectors
• $\mathbf{X}$ : Random vector
• $\{X_n\}$ and $\mathbf{X}$ defined on the same sample space

Then

$${\mathbf{X}}_n\stackrel{P}{\to }\mathbf{X}⟺X_{nj}\stackrel{P}{\to }{X}_j,\forall j=1,\dots ,p$$

Note

This theorem shows the convergence in probability of vectors is equivalent to componentwise convergence in probability.

Definition 5.4.2: Multivariate convergence in distribution

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of $p$-dimensional vectors, with
  • $F_n(\mathbf{x})$ : cdf of ${\mathbf{X}}_n$
• $\mathbf{X}$ : Random vector, with
  • $F(\mathbf{x})$ : cdf of $\mathbf{X}$
• $C(F)$ denote the set of all points where $F$ is continuous

If

$$\underset{n\to \infty }{\lim }{F}_n(\mathbf{x})=F(\mathbf{x}),\forall \mathbf{x}\in C(F)$$

Then

• We say $\{{\mathbf{X}}_n\}$ converges in distribution to $\mathbf{X}$
• We write ${\mathbf{X}}_n\stackrel{D}{\to }\mathbf{X}$

Link to original

Theorem 5.4.2

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of random vectors
• $\mathbf{X}$ : Random vector
• $g(\mathbf{X})$ : Function continuous on the support of $X$

If $\{{\mathbf{X}}_n\}\stackrel{D}{\to }\mathbf{X}$

Then

$$g({\mathbf{X}}_n)\stackrel{D}{\to }g(\mathbf{X})$$

Theorem 5.4.3

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of random vectors, with
  • $F_n(\mathbf{x})$ : cdf of ${\mathbf{X}}_n$
  • $M_n(\mathbf{t})$ : mgf of ${\mathbf{X}}_n$
• $\mathbf{X}$ : Random vector, with
  • $F(\mathbf{x})$ : cdf of $\mathbf{X}$
  • $M(\mathbf{t})$ : mgf of $\mathbf{X}$

Then

$$\{{\mathbf{X}}_n\}\stackrel{D}{\to }\mathbf{X}⟺\exists h>0\ni \underset{n\to \infty }{\lim }{M}_n(\mathbf{t})=M(\mathbf{t}),\forall \mathbf{t}:||\mathbf{t}||<h$$

Note

As in the univariate case, convergence in distribution is equivalent to convergence of moment generating functions.

Theorem 5.4.4: Multivariate central limit theorem

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of iid random vectors, with
  • $\mathbf{\mu }$ : Common mean vector
  • $\mathbf{\Sigma }$ : Variance-covariance matrix (which is positive definite)
  • $M(\mathbf{t})$ : Common mgf

If

• $M(\mathbf{t})$ exists in an open neighborhood of $0$

\begin{align} \mathbf{Y}{n} & = \frac{1}{\sqrt{ n }} \sum{i=1}^n (\mathbf{X}_{i}-\boldsymbol\mu) \ & = \sqrt{ n }(\bar{\mathbf{X}} - \boldsymbol \mu) \end{align}

$$Then$$

\mathbf{Y}{n} \xrightarrow{D} N{p}(\mathbf{0}, \mathbf{\Sigma})

Theorem 5.4.5

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of $p$-dimensional random vectors
• $\mathbf{A}$ : $m\times p$ matrix of constants
• $\mathbf{b}$ : $m$-dimensional vector of constants

If

$${\mathbf{X}}_n\stackrel{D}{\to }N(\mathbf{\mu },\mathbf{\Sigma })$$

Then

$$\mathbf{A}{\mathbf{X}}_n+\mathbf{b}\stackrel{D}{\to }N(\mathbf{A}\mathbf{\mu }+\mathbf{b},{\mathbf{A\Sigma A}}^{\prime })$$

Theorem 5.4.6

Let

• $\{{\mathbf{X}}_n\}$ : Sequence of $p$-dimensional random vectors
• $\sqrt{n}({\mathbf{X}}_n-{\mathbf{\mu }}_0)\stackrel{D}{\to }{N}_p(0,\mathbf{\Sigma })$
• $\mathbf{g}$ : Transformation $\mathbf{g}(\mathbf{x})=(g_1(\mathbf{x}),\dots ,g_k(\mathbf{x}){)}^{\prime }\ni 1\le k\le p$
• $\mathbf{B}$ : $k\times p$ matrix of partial derivatives are continuous and do not vanish in a neighborhood of ${\mathbf{\mu }}_0$

$$B=\left[\frac{\partial g_i}{\partial {\mu }_j}\right],i=1,\dots ,k;j=1,\dots ,p$$

• ${\mathbf{B}}_0=\mathbf{B}$ at ${\mathbf{\mu }}_0$

Then

$$\sqrt{n}(\mathbf{g}({\mathbf{X}}_n)-\mathbf{g}({\mathbf{\mu }}_0))\stackrel{D}{\to }{N}_k(0,{\mathbf{B}}_0{\mathbf{\Sigma B}}_0^{\prime })$$