Question

[True/False] The training error of KNN is always 0 .[Short answer] We discussed four different supervised models and algorithms so far - KNN, DT, NB, Perceptron. Circle all those that would get a 0 training error on a dataset labeled by the XOR function. Explain briefly. True/False] Naïve Bayes requires exactly the same number of parameters as models learned by Perceptron, since both learn linear models P(A,B,C) 0.125s 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0 0 conditionally independent of B given C? 0 0 1 0 0 0 0

Answer 1

False

The error rate at K=1 is always zero for the training sample. This is because the closest point to any training data point is itself.Hence the prediction is always accurate with K=1. If validation error curve would have been similar, our choice of K would have been 1. Following is the validation error curve with varying value of K:

but the training error of KNN is not always 0

60 Validation error 50 40 30 20 0 30 K-Value 50 0 10 20 40 60

This makes the story more clear. At K=1, we were overfitting the boundaries. Hence, error rate initially decreases and reaches a minima. After the minima point, it then increase with increasing K. To get the optimal value of K, you can segregate the training and validation from the initial dataset. Now plot the validation error curve to get the optimal value of K. This value of K should be used for all predictions.