(a) Let $\left(X_{t}, t \geqslant 0\right)$ be a continuous-time Markov chain on a countable state space I. Explain what is meant by a stopping time for the chain $\left(X_{t}, t \geqslant 0\right)$. State the strong Markov property. What does it mean to say that $X$ is irreducible?

(b) Let $\left(X_{t}, t \geqslant 0\right)$ be a Markov chain on $I=\{0,1, \ldots\}$ with $Q$-matrix given by $Q=\left(q_{i, j}\right)_{i, j \in I}$ such that:

(1) $q_{i, 0}>0$ for all $i \geqslant 1$, but $q_{0, j}=0$ for all $j \in I$, and

(2) $q_{i, i+1}>0$ for all $i \geqslant 1$, but $q_{i, j}=0$ if $j>i+1$.

Is $\left(X_{t}, t \geqslant 0\right)$ irreducible? Fix $M \geqslant 1$, and assume that $X_{0}=i$, where $1 \leqslant i \leqslant M$. Show that if $J_{1}=\inf \left\{t \geqslant 0: X_{t} \neq X_{0}\right\}$ is the first jump time, then there exists $\delta>0$ such that $\mathbb{P}_{i}\left(X_{J_{1}}=0\right) \geqslant \delta$, uniformly over $1 \leqslant i \leqslant M$. Let $T_{0}=0$ and define recursively for $m \geqslant 0$,

$$T_{m+1}=\inf \left\{t \geqslant T_{m}: X_{t} \neq X_{T_{m}} \text { and } 1 \leqslant X_{t} \leqslant M\right\}$$

Let $A_{m}$ be the event $A_{m}=\left\{T_{m}<\infty\right\}$. Show that $\mathbb{P}_{i}\left(A_{m}\right) \leqslant(1-\delta)^{m}$, for $1 \leqslant i \leqslant M$.

(c) Let $\left(X_{t}, t \geqslant 0\right)$ be the Markov chain from (b). Define two events $E$ and $F$ by

$$E=\left\{X_{t}=0 \text { for all } t \text { large enough }\right\}, \quad F=\left\{\lim _{t \rightarrow \infty} X_{t}=+\infty\right\}$$

Show that $\mathbb{P}_{i}(E \cup F)=1$ for all $i \in I$.