Best Model Selection

About best model selection

Selecting appropriate predictor variables in regression modeling, especially for prediction, is crucial (Darlington, 2017).

Choosing the right predictors ensures the regression model effectively explains variable relationships. Variable selection methods help identify the best and most significant predictors for the model. The preferred regression model uses the simplest set of predictors.

The selection methods to be studied are backward selection, forward selection, and stepwise. Before applying these methods, understanding the partial F-test and sequential F-test is necessary. The partial F-test is used in backward selection, while the sequential F-test is used in forward selection.

Best model criteria

Criteria	Formula	Optimum?
$\mathrm{MSE}$	$\sum \frac{(Y-\hat{Y}{)}^2}{n-p-1}$	Minimum
${\text{R}}^2$	$R^2=\frac{b^T{X}^{T}Y-n{\bar{Y}}^2}{Y^{T}Y-n{\bar{Y}}^2}$	Maximum
$C_p\text{ Mallow}$	$C_p=\frac{JK{S}_p}{s^2}-(n-2p)$	Minimum

Mean squared error (MSE)

Mean squared error measures error in a model. If a model does not have any errors, then $MSE=0$. As the error goes up, MSE goes up.

$$MSE=\frac{{\sum }_i^n(y_i-{\hat{y}}_i{)}^2}{n}$$

Coefficient of determination

Coefficient of determination ($R^2$) is a statistic that measures proportion or total variation around the mean of the response variable that can be explained by the regression model.

$R^2\in [0,1]$ (Ranges from 0 to 1, inclusive).

$R^2=\frac{b^T{X}^{T}Y-n{\bar{Y}}^2}{Y^{T}Y-n{\bar{Y}}^2}$

Adjusted coefficient of determination

Adjusted coefficient of determination ($R_{\text{adj}}^2$) is a statistic, similar to Coefficient of determination but unaffected by degrees of freedom of the residual sum of squares or the total sum of squares.

Unlike Coefficient of determination, adding predictor variables to the model does not always increase $R_{\text{adj}}^2$.

$$R_{\text{adj}}^2=1-\left(\frac{n-1}{n-p}\right)(1-R^2)$$

CP Mallow statistic

$$CP=\frac{JKS}{s^2}-(n-2p)$$

Prediction Sum of Squares (PRESS)

A form of cross-validation used in regression analysis that measures the model’s fit to a sample of observations not used in estimating the model.

$$\text{PRESS}=\sum _i^n(y_i-{\hat{y}}_{i,-i}{)}^2$$