A discrete-time controlled Markov process evolves according to

$$X_{t+1}=\lambda X_{t}+u_{t}+\varepsilon_{t}, \quad t=0,1, \ldots,$$

where the $\varepsilon$ are independent zero-mean random variables with common variance $\sigma^{2}$, and $\lambda$ is a known constant.

Consider the problem of minimizing

$$F_{t, T}(x)=\mathbb{E}\left[\sum_{j=t}^{T-1} \beta^{j-t} C\left(X_{j}, u_{j}\right)+\beta^{T-t} R\left(X_{T}\right)\right],$$

where $C(x, u)=\frac{1}{2}\left(u^{2}+a x^{2}\right), \beta \in(0,1)$ and $R(x)=\frac{1}{2} a_{0} x^{2}+b_{0}$. Show that the optimal control at time $j$ takes the form $u_{j}=k_{T-j} X_{j}$ for certain constants $k_{i}$. Show also that the minimized value for $F_{t, T}(x)$ is of the form

$$\frac{1}{2} a_{T-t} x^{2}+b_{T-t}$$

for certain constants $a_{j}, b_{j}$. Explain how these constants are to be calculated. Prove that the equation

$$f(z) \equiv a+\frac{\lambda^{2} \beta z}{1+\beta z}=z$$

has a unique positive solution $z=a_{*}$, and that the sequence $\left(a_{j}\right)_{j} \geqslant 0$ converges monotonically to $a_{*}$.

Prove that the sequence $\left(b_{j}\right)_{j \geqslant 0}$ converges, to the limit

$$b_{*} \equiv \frac{\beta \sigma^{2} a_{*}}{2(1-\beta)} .$$

Finally, prove that $k_{j} \rightarrow k_{*} \equiv-\beta a_{*} \lambda /\left(1+\beta a_{*}\right)$.