Question

1. Decision trees As part of this question you will implement and compare the Information Gain, Gini Index and CART evaluation measures for splits in decision tree construction.Let D= (x,y), D = n be a dataset with n samples. The entropy of the dataset is defined as H(D)= P(c|D)log2P(c|D), where P(CD) is the fraction of samples in class i. A split on an attribute of the form X, <c partitions the dataset into two subsets Dy and Dn based on whether samples satisfy the split predicate or not respectively. The split Entropy is the weighted average Entropy of the resulting datasets Dy and Dx: TLY H(Dy, DN)= "* HDY :) + "N H(DN), where ny are the number of samples in Dy and nn are the number of samples in DN. The Information Gain (IG) of a split is defined as the the difference of the Entropy and the split entropy: IG(D, Dy, DN) = H(D) - H(Dy, Dn). (1) The higher the information gain the better. The Gini index of a data set is defined as G(D)=1- - P(C|D)2 and the Gini index of a split is defined as the weighted average of the Gini indices of the resulting partitions: G(Dy, Dn) = ny G(DY) + "N G(DN). The lower the Gini index the better. Finally, the CART measure of a split is defined as: ny IN CART(Dy, DN) = 2"Y |PC|DY) - P(GDN). ann" The higher the CART the better. You will need to fill in the implementation of the three measures in the provided Python code as part of the homework. Note: You are not allowed to use existing implementations of the measures. The homework includes two data files, train.tact and test.txt. The first consists of 100 observations to use to train your classifiers; the second has 10 to test. Each file is comma-separated, and each row contains 11 values - the first 10 are attributes (a mix of numeric and categorical translated to numeric, e.g. {T,F} = {0,1}), and the final being the true class of that observation. You will need to separate attributes and class in your load(filename) function.

Answer 1

a)

1. # importing libraries  
2. import numpy as nm  
3. import matplotlib.pyplot as mtp  
4. import pandas as pd  
5.   
6. #importing datasets  
7. data_set= pd.read_csv('user_data.csv')  
8.   
9. #Extracting Independent and dependent Variable  
10. x= data_set.iloc[:, [2,3]].values  
11. y= data_set.iloc[:, 4].values  
12.   
13. # Splitting the dataset into training and test set.  
14. from sklearn.model_selection import train_test_split  
15. x_train, x_test, y_train, y_test= train_test_split(x, y, test_size= 0.25, random_state=0)  
16.   
17. #feature Scaling  
18. from sklearn.preprocessing import StandardScaler    
19. st_x= StandardScaler()  
20. x_train= st_x.fit_transform(x_train)    
21. x_test= st_x.transform(x_test)
22. output:
23. 
    data_set - DataFrame - O X Index Gender Age Purchased User ID 15624510 EstimatedSalary 19000 Male 15810944 Male 35 20000 15668575 Female 43000 15603246 Female 57000 15884002 Male 76000 15728773 Male 58000 15598044 Female 84000 15694829 Female 150000 15600575 Male 33000 15727311 Female 35 65000 15570769 Female 80000 15606274 Female 52000 15746139 Male 86000 15704987 Male 32 18000 15628972 Male 18 82000 15697686 Male 29 80000 1 Resize Background color column min/max Save and Close Close
    

B)

1. #Visulaizing the trianing set result  
2. from matplotlib.colors import ListedColormap  
3. x_set, y_set = x_train, y_train  
4. x1, x2 = nm.meshgrid(nm.arange(start = x_set[:, 0].min() - 1, stop = x_set[:, 0].max() + 1, step  =0.01),  
5. nm.arange(start = x_set[:, 1].min() - 1, stop = x_set[:, 1].max() + 1, step = 0.01))  
6. mtp.contourf(x1, x2, classifier.predict(nm.array([x1.ravel(), x2.ravel()]).T).reshape(x1.shape),  
7. alpha = 0.75, cmap = ListedColormap(('purple','green' )))  
8. mtp.xlim(x1.min(), x1.max())  
9. mtp.ylim(x2.min(), x2.max())  
10. fori, j in enumerate(nm.unique(y_set)):  
11. mtp.scatter(x_set[y_set == j, 0], x_set[y_set == j, 1],  
12.         c = ListedColormap(('purple', 'green'))(i), label = j)  
13. mtp.title('Decision Tree Algorithm (Training set)')  
14. mtp.xlabel('Age')  
15. mtp.ylabel('Estimated Salary')  
16. mtp.legend()  
17. mtp.show()  

Output:

Decision Tree Algorithm (Training set) : 1.1, Estimated Salary -2 -1 0 Age