Question

Suppose the joint probability distribution of two binary random variables X and Y are given as follows.

x/y	1	2
0	3/10	0
1	4/10	3/10

X goes along side as 0 and 1, Y goes along top as 1 and 2.

a) Show the marginal distribution of X.

b) Find entropy H(Y ).

c) Find conditional entropy H(X|Y ) and H(Y |X).

d) Find mutual information I(X; Y ).

e) Find joint entropy H(X, Y ).

f) Suppose X and Y are independent. Show that H(X|Y ) = H(X).

g) Suppose X and Y are independent. Show that H(X, Y ) = H(X) + H(Y ).

h) Show that I(X; X) = H(X).

Answer 1

x/y	1	2	totalP(X=x)
0	0.3	0	0.3
1	0.4	0.3	0.7
total=P(Y=y)	0.7	0.3	1

a)marginal distribution of X
X	P(X=x)
0	0.3
1	0.7
total	1

b)

Entropy: H(Y) = − Σ p(y).log p(y)

= -[0.7log₂0.7+0.3log₂0.3] = 0.88129

c)

Y	1	2
P(Y|X=0)=P(X=x,Y=y)/P(X=0)	0.3/0.3=1	0
P(Y|X=1)=P(X=x,Y=y)/P(X=1)	.4/.7=4/7	.3/.7=3/7

H(Y|X) = − p (x, y)log p(y|x) ] across all x ∈ X and y ∈ Y

= -[0.3log1+0log0+0.4log(4/7)+0.3log(3/7)]= 0.6897

H(X|Y) = − p (x, y)log p(x|y) ] across all x ∈ X and y ∈ Y

= -[0.3log(0.3/0.7)+0log(0/0.3)+0.4log(0.4/0.7)+0.3log(0.3/0.3)]= 0.6897

d)

mutual information I(X; Y )

I(X; Y) = H(X) - H(X|Y) = − Σ p(x).log p(x) - H(X|Y) = -[0.3log0.3+0.7log0.3]-0.6897 = 0.1916

---------------------------------------------------------------

please note that only first four subparts per post can be answered as per HOMEWORKLIB RULES