Question

Consider a Markov chain with state space S = {1, 2, 3, 4} and transition matrix

P=$\begin{bmatrix} p1 &p2 &p3 &p4 \\ 1& 0 &0 &0 \\ 0& 1 &0 &0 \\ 0& 0 &1 &0 \end{bmatrix}$ where $\sum_{k=1}^{4} pk=1$

(a) Draw a directed graph that represents the transition matrix for this Markov chain.

(b) Compute the following probabilities:

P(starting from state 1, the process reaches state 3 in exactly three time steps);

P(starting from state 1, the process reaches state 3 in exactly four time steps);

P(starting from state 1, the process reaches states higher than state 1 in exactly two time steps).

(c) If the process starts from state 3, provide the states which are not attainable in exactly two time steps.

Answer 1