Question

What are the three things to remember when choosing additional independent variables for a multiple regression?

Answer 1

• First you need to check weather new independent variable is correlated or not with our dependent variable. If it is correlated , we may use this things to find weather to add or remove additional independent variable for a multiple regression model

For fitted model find Adjusted R-squared and Predicted R-squared :

Generally, you choose the models that have higher adjusted and predicted R-squared values. These statistics are designed to avoid a key problem with regular R-squared—it increases every time you add a predictor and can trick you into specifying an overly complex model.

The adjusted R squared increases only if the new term improves the model more than would be expected by chance and it can also decrease with poor quality predictors.

Compute t-test value and P-values for new independent variable added to model :

In regression, low p-values correspond to t-test for regression coefficients , indicate terms that are statistically significant. If it is so then we can add additional independent variable for a multiple regression model

Perform Stepwise regression and Best subsets regression :

Stepwise regression is a popular data-mining tool that uses statistical significance to select the explanatory variables (new variable) to be used in a multiple-regression model. Stepwise regression essentially does multiple regression a number of times, each time removing the weakest correlated variable . Hence if our new variable is not enough correlated to response variable or not enough significant to our model then it will get remove .

Additional terms will always improve the model whether the new term adds significant value to the model or not. As a matter of fact, adding new variables can actually make the model worse. Adding more and more variables makes it more and more likely that you will overfit your model to the training data.

So it is neccesary to perform Stepwise regression techniques. ( or forward and backward selection techniques ) to choose best fit in the model , as adding more variables may overfit your data and may give missleading outcomes .