Question

3. (5 points) Since the long-run proportion of time that a Markov chain spends in a transient state is 0, there doesn't exist an irreducible Markov chain with all the states being transient. Is it true? If not, please give a counterexample.

Answer 1

Solution:

For a Markov chain, consider a pair of states (i, j).

We say that j is reachable from i, denoted by i to j, if there exists an integer n ≥ 0 such that P n ij > 0.

This means that starting in state i, there is a positive probability (but not necessarily equal to 1) that the chain will be in state j at time n (that is, n steps later); P(Xn = j|X0 = i) > 0.

If j is reachable from i, and i is reachable from j, then the states i and j are said to communicate.

The relation defined by the Markov chain satisfies the following conditions:

1. All states communicate with themselves: P 0 ii = 1 > 0.1

2. Symmetry: If i = j, then j = i.

3. Transitivity: If i = k and k = j, then i = j.

The above conditions imply that communication is an example of an equivalence relation, meaning that it shares the properties with the more familiar equality relation “ = ”: i = i. If i = j, then j = i. If i = k and k = j, then i = j.

Only condition 3 above needs some justification, so we now prove it for completeness:

Suppose there exists integers n, m such that P n ik > 0 and P m kj > 0.

Letting l = n + m, we conclude that P l ij ≥ P n ikP m kj > 0 where we have formally used the Chapman-Kolmogorov equations.

A little thought reveals that this kind of disjoint breaking can be done with any Markov chain: