Question

if coding is needed, please use python

A data set has 600 examples. To properly test the performance of the final hypothesis, you set aside a randomly selected subset of 200 examples which are never used in the training phase; these form a test set. You use a learning model with 1,000 hypotheses and select the final hypothesis g based on the 400 training examples. We wish to estimate Eout (g). We have access to two estimates: Ein(g), the in-sample error on the 400 training examples; and, Erest (g), the test error on the 200 test examples that were set aside. (a) Using a 5% error tolerance ( 0.05), which estimate has the higher 'error bar'? ere any reason why you shouldn't reserve even more examples for testing? (b) Is th

Answer 1

(a) E_in(g) is training error and it would be very less as training error is found by appling model on data in which it was trained. It is pretty obvios that when whe train our models in 400 training set , they will learn the features of all points in that dataset. But at the same time E_test(g) is errorfound when models run on test data, so this error would be larger. We can conclude that Test data will estimate higher error bar.

(b) No, as reserving more number of data points in test data would lead to underfitting of trained model. Therefore, the model will not be able to lear all the characterstic features of data and will not be able to provide accurate predictions.