Question

Consider the following data set which will be used for a binary classification problem where the goal is to predict whether a house will sell within 6 months Sold Age Overrpriced Features Location 50 90 60 60 70 80 50 90 80 80 60 70 None Edinburgh None None Aberdeen GarageDundee Garage Edinburgh Pool Edinburgh] PoolInverness None Inverness Garage Edinburgh Garag PoolAberdeen Glagsow e Edinburgh Dundee 1. What is the initial entropy of the Sold variable? 2. If we classify this data using decision trees and the id5 algorithm, what attribute would be chosen as the root node? 3. What is the information gain associated with this root node? [Assessed 2 marks]

Answer 1

Initial entropy for variable sold is given by

count number of 0's and 1's in sold column

So 0's are = 7 and 1's are = 5 so total = 12

Formula for entropy = E( S) =-

Σ pilogpi

    

here log to the base 2 always SO here entropy is = -p1ogp1-p2logp2=-7/12log7/12-5/12log5/12 = 0.9798

Now we will form decision tree using id5 algorithm

before that we have choose which variable has highest possible information gain.

So let us start from sold and overpriced

Sp E ( S, overpriced)= E( S) - (sum of p(x)*H(x))

P(x) for overpriced=1 = 5/14 so h(x) = -5/5log5/5-0/5log0/5 = 0   and p(x) for overpriced=0 = 7/12 H (x) = -5/7log5/7-2/7log2/7 =0.8631

So E(S ovepriced )= 0.9798 - 0.5034 = 0.4764

Similarly we have to calculate for age , feature and location

So fo age E(S,age ) = 0.25022

E(S,features)= 0.2317

E(S, location) = 0.3580

So the hieghest entropy is for overpriced so the root node is overpriced which has two outcomes 0 an d 1

Information gain asociated with root node = 0.4764