2.II.20D

Markov Chains | Part IB, 2005

Consider a Markov chain (Xn)n0\left(X_{n}\right)_{n \geqslant 0} with state space {0,1,2,}\{0,1,2, \ldots\} and transition probabilities given by

Pi,j=pqij+1,0<ji+1, and Pi,0=qi+1 for i0P_{i, j}=p q^{i-j+1}, \quad 0<j \leqslant i+1, \quad \text { and } \quad P_{i, 0}=q^{i+1} \quad \text { for } \quad i \geqslant 0

with Pi,j=0P_{i, j}=0, otherwise, where 0<p<10<p<1 and q=1pq=1-p.

For each i1i \geqslant 1, let

hi=P(Xn=0, for some n0X0=i),h_{i}=\mathbb{P}\left(X_{n}=0, \text { for some } n \geqslant 0 \mid X_{0}=i\right),

that is, the probability that the chain ever hits the state 0 given that it starts in state ii. Write down the equations satisfied by the probabilities {hi,i1}\left\{h_{i}, i \geqslant 1\right\} and hence, or otherwise, show that they satisfy a second-order recurrence relation with constant coefficients. Calculate hih_{i} for each i1i \geqslant 1.

Determine for each value of p,0<p<1p, 0<p<1, whether the chain is transient, null recurrent or positive recurrent and in the last case calculate the stationary distribution.

[Hint: When the chain is positive recurrent, the stationary distribution is geometric.]

Typos? Please submit corrections to this page on GitHub.