Stepwise Regression Methods

Warning

• During whole procedure, a large amount of single parameter t-tests were conducted. This means that we have very probably included some unimportant independent variables in the model (Type I errors) and eliminated some important ones (Type II errors).
• Do not be deceived by the impressive-looking t-statistics, as stepwise regression in nature, is an algorithm biased toward large t-statistics.

Bidirectional elimination

1. Identifies the response $y$, and set of potentially important predictors $x_1,x_2,\dots ,x_k$ to use.
2. Test the parameter of all possible one-predictor models $y={\beta }_0+{\beta }_1{x}_i$.
3. Pick a predictor $x_1$ which yields the largest t-statistic out of all the tests on step $(2)$.
4. Test the remaining predictors to fit into a two-predictor model, and picks $x_2$ which yields largest t-statistics.
5. Test the first predictor to see if ${\hat{\beta }}_1$ is still significant.
   • If ${\hat{\beta }}_1$ is no longer significant, remove $x_1$. Search a predictor with largest t-statistic in the presence of the term ${\hat{\beta }}_2{x}_2$.
   • Continue searching for predictors, until no further predictors can be found that yield significant t-statistics (at $\alpha$ level).

Forward selection

Similar to bidirectional elimination. The difference is that this algorithm doesn’t test previously added predictors like on step $(4)$

Backward elimination

Instead of starting by fitting all possible one-predictor model like bidirectional elimination, this algorithm starts with all predictors included in the initial model.

This algorithm then tests for the least significant predictor, and removes them, the complete opposite of forward selection.

Software implementation

data(mtcars)
 
model_intercept <- lm(mpg ~ 1, data = mtcars)
model_full <- lm(mpg ~ ., data = mtcars)
 
# Bidirectional elimination
both <- step(model_intercept, direction = 'both', scope = formula(model_full))
 
# Forward selection
forward <- step(model_intercept, direction = 'forward', scope = formula(model_full))
 
# Backwards elimination
backward <- step(model_full, direction = 'backward')