Question

For kNN classifiers, explain the relationship between parameter k and the model’s tendency to overfitting.

Answer 1

The “K” is KNN algorithm is the nearest neighbors we wish to take vote from.

We should choose K in K - Nearest Neighbour Algorithms wisely. If we choose our K = 1 , then our algorithm behaves as over fitting and it gives a non - smooth decision surface. As K increases, our decision surface gets smoother. And,if we choose K = n, then our algorithm behaves as underfitting and it gives a smooth decision surface and everything becomes one class which is the majority class in our DataSet. So, we should choose K wisely such that it should neither be overfitting nor be underfitting .

Underfitting means the model does not fit, in other words, does not predict, the (training) data very well. On the other hand, overfitting means that the model predict the (training) data too well. It is too good to be true. If the new data point comes in, the prediction may be wrong.

The bias is an error from assumptions are made in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs. In other words, model with high bias pays very little attention to the training data and oversimplifies the model.

The variance is an error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended outputs. In other words, model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before.

Normally, underfitting implies high bias and low variance, and overfitting implies low bias but high variance.

So, when k=1 tendency for overfitting is more as k increases tendency for overfitting decreases.