Question

. Question 3 (5 pts]: Given the following six instances cach with five attributes (Outlook, Temperature, Humidity, Wind, Day) and one class label, calculate Entropy of the whole system [1 pt] • calculate Information gain for attribute "Outlook" [1 pt] • calculate Gini-index for attribute "Outlook" [1 pt] What is the information gain and Gini-index for attribute "Day" [pt] • Explain why "Day" is NOT a good feature being used as the root node of a decision tree. How to avoid using "Day" as the root node to create the tree [1 pt] ID Outlook Temperature Humidity Wind Day Class 1 Sunny Hot High Weak Monday No 2 Sunny Cool High Strong Tuesday No 3 Overcast Cool Normal Weak Wednesday No 4 Overcast Hot High Weak Thursday Yes 5 Sunny Mild High wak Friday Yes 6 Rain Cool Normal Weak Saturday Yes

Answer 1

1. Entropy of whole system

The entropy characterizes the (im)purity of an arbitrary collection of examples.

In Class, there are two values Yes and No

So, there are total = 6 instances : 3 Yes = 3/6 and 2 No = 3/6

Entropy(Class) = - [(3/6)log₂(3/6) + (3/6)log₂(3/6)]

= 0.5 x 1 + 0.5 x 1 = 1

2. Information Gain for attribute "Outlook"

Information Gain is the expected reduction in entropy caused by partitioning the examples according to a given attribute

Outlook has 3 values i.e Sunny, Rain and Overcast.

Sunny = 3/6

Rain = 1/6

Overcast = 2/6

Now, Let's find out entropy for each values

Entropy(Sunny) :

(Sunny, No) = 2 = 2/3 , (Sunny,Yes) = 1 = 1/3

Entropy = -(2/3 log₂ 2/3 + 1/3 log₂ 1/3) = 0.387 + 0.53 = 0.917

Similarly,

Entropy(Rain) = -(1 x log₂ 1 ) = 0

Entropy(Overcast) = 1

Hence, Entropy(Outlook) = 3/6 x 0.917 + 1/6 x 0 + 2/6 x 1 = 0.7885

Information Gain = G(Outlook) = Entropy(Class) - Entropy(Outlook) = 1 - 0.7885 = 0.2115

Hence, Information gained = 0.2115

3. Gini Index for Outlook :

Let represent Gini index by P()

So, in Outlook,

P(Outlook = Overcast): 2/6

P(Outlook = Rain): 1/6

P(Outlook = Sunny): 3/6

If (Outlook = Overcast and Class = yes), probability = 1/2

If (Outlook = Overcast and Class = No)), probability = 1/2

Gini index = 1 - ((1/2)^2 + (1/2)^2) = 0.5

If (Outlook = Rain and Class = yes), probability = 1/1

If (Outlook = Rain and Class = No)), probability = 0/1

Gini index = 1 - ((1/1)^2 + (0/1)^2) = 0

If (Outlook = Sunny and Class = yes), probability = 1/3

If (Outlook = Sunny and Class = No)), probability = 2/3

Gini index = 1 - ((1/3)^2 + (2/3)^2) = 0.46

Hence,

Gini index(Outlook) = 2/6 x 0.5 + 1/6 x 0 + 3/6 x 0.46 = 0.397

4. Information gain and Gini index for "Day" will be 0 as Day has all unique values and unique values always gives the result 0 in Information gain and Gini index.