Question

Question 6 [14 points]: Consider a Markov chain (XnJn2o with state space S 11,2,3,4) and transition kernel: 1/9 0 0 8/9 1/16 1/8 13/16 0 (a) List all the communication classes, and indicate if they are recurrent or visit state 1 and then come back to state 4. Compute E(T). What is the 1/40 0 3/4ノ transient. (b) 4 (initial state) and T the number of transitions required to Let distribution of T?

Answer 1

a) The states of a Markov chain can be partitioned into communicating classes such that only members of the same class communicate with each other. That is, two states i and j belong to the same class if and only if 'i' can be visited from 'j' or 'j' can be visited from 'i'. Note that, every state can communicate with itself.

which is the transition graph, we can see that state 1 communicates with state 4 but no other state communicates.

2 4 7

Thus the communicating classes,

class 1= { state 1, state 4 }, class 2 = { state 2 }, class 3 = { state 3 }.

By definition, a recurrent state is defined as a state such that if at any time we leave that state, we return to that state in the future with probability 1. Here, class 1 is a recurrent class, since all its states are recurrent. While Class 2 and class 3 are transient.

b) Here, X₀= 4 (i.e. the initial state is 4) and T is the number of transitions required to visit state 1 and then come back to state 4. For any state i, let us define f_ii=P(X_n=i, for some n≥1|X₀=i), that is f_ii gives the probability of ever returning to state i given that the chain started in state i.

f₄₄ = 1 , since state 4 is recurrent

Counting over all time, the total number of visits to state i, given that X0 = i, is given by an infinite sequence of indicator random variables,

And it has a geometric distribution, P(T-t)-f44(t-1) (1-f44), i.e. T ~ Geometric(f44) The geometric distribution gives the probability that the first occurrence of success (here success is returning back to state 4 after starting from state 4) which requires 't' independent number of visits, each with success probability 'f44 We count the initial visit Xo 4 as the first visit.) The expected number of visits is thus given by E(T) 1-f44 and we conclude that since state 4 is recurrent (f44-1) then E(T) -