Test for Comparing Nested Models

In regression analysis, we often want to determine which one among a set of candidate models best fits the data. This page discusses a test for comparing nested models.

Terms

• Nested model: Two models are nested if one model contains all the terms of the second model and at least one additional term.

• Complete/full model: The more complex of the nested models.

  $$E(y)={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_g{x}_g$$

• Reduced/restricted model: The simpler of the nested models.

  $$\begin{array}{rl}E(y)=& {\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_g{x}_g\\ & +{\beta }_{g+1}{x}_{g+1}+\cdots +{\beta }_k{x}_k\end{array}$$

Steps

1. Determine the complete and reduced models
2. Determine the decision rule
3. Compute the test statistic
4. Make a conclusion

Hypotheses

• $H_0:{\beta }_{g+1}+{\beta }_{g+2}=\cdots ={\beta }_k=0$
• $H_1:$ Atleast one of the $\beta$ coefficients being tested is not $0$

Test statistic

$$F=\frac{(SS{E}_R-SS{E}_C)/{\text{df}}_1}{SS{E}_C/{\text{df}}_2}$$

Where:

• ${\text{df}}_1:$ Amount of coefficients being compared/tested
• ${\text{df}}_2=n-(p+1)$
• $SS{E}_C=s_C^2\times {\text{df}}_C$
• $SS{E}_R=s_R^2\times {\text{df}}_R$
• $MS{E}_C=SS{E}_C/{\text{df}}_2$
• $n:$ Total sample size
• $p:$ Number of predictors
• $R:$ Reduced
• $C:$ Complete

Decision rule

Reject $H_0$, if at least one of

• $F>F_{\alpha }$, at degrees of freedoms ${\text{df}}_1$ and ${\text{df}}_2$ (See here)
• $\alpha >\text{p-value}$