Question

When doing linear regression in jupyter how does one determine what columns to drop?

Answer 1

When doing linear regression in jupyter how does one determine what columns to drop?

Which columns to drop:

● Those randomly generated field or column for only purpose is unique identification.

● Those who leaks the data or information from the future.

● Those who contain redundant data or information that means which are already included in the other column.

● Those who requires lot of processing and some other data and information to be elected as potentially useful column.

● Those who contains two many unique values which are occurring only once. Also drop the column which don’t add any information to the model like column containing only one unique value.

● Those who don’t contain cleaned values or contains missing values or information.

Drop the column using Z-score and IQR score:

● Z-score and IQR score both are used to detect the outlier.

● Z-score measures the data points which are above the mean value of what we are measuring using standard deviations.

● Interquartile range- IQR measures the variability that is amount of spread in the middle range around 50% of a dataset that is basically works on dividing a particular application dataset into quartiles like Q1, Q2 and Q3.

● Specify the threshold value. threshold = 3 and then Apply the following code to remove the outliers:

dataset_o = dataset_o [(z < 3).all(axis=1)]

dataset_out = dataset_o1[~((dataset _o1 < (Q1 - 1.5 * IQR)) |( dataset_o1 > (Q3 + 1.5 * IQR))).any(axis=1)]