Let $Q$ be a positive-definite symmetric $m \times m$ matrix. Show that a non-negative quadratic form on $\mathbb{R}^{d} \times \mathbb{R}^{m}$ of the form

$$c(x, a)=x^{T} R x+x^{T} S^{T} a+a^{T} S x+a^{T} Q a, \quad x \in \mathbb{R}^{d}, \quad a \in \mathbb{R}^{m}$$

is minimized over $a$, for each $x$, with value $x^{T}\left(R-S^{T} Q^{-1} S\right) x$, by taking $a=K x$, where $K=-Q^{-1} S$.

Consider for $k \leqslant n$ the controllable stochastic linear system in $\mathbb{R}^{d}$

$$X_{j+1}=A X_{j}+B U_{j}+\varepsilon_{j+1}, \quad j=k, k+1, \ldots, n-1,$$

starting from $X_{k}=x$ at time $k$, where the control variables $U_{j}$ take values in $\mathbb{R}^{m}$, and where $\varepsilon_{k+1}, \ldots, \varepsilon_{n}$ are independent, zero-mean random variables, with $\operatorname{var}\left(\varepsilon_{j}\right)=N_{j}$. Here, $A, B$ and $N_{j}$ are, respectively, $d \times d, d \times m$ and $d \times d$ matrices. Assume that a cost $c\left(X_{j}, U_{j}\right)$ is incurred at each time $j=k, \ldots, n-1$ and that a final cost $C\left(X_{n}\right)=X_{n}^{T} \Pi_{0} X_{n}$ is incurred at time $n$. Here, $\Pi_{0}$ is a given non-negative-definite symmetric matrix. It is desired to minimize, over the set of all controls $u$, the total expected cost $V^{u}(k, x)$. Write down the optimality equation for the infimal cost function $V(k, x)$.

Hence, show that $V(k, x)$ has the form

$$V(k, x)=x^{T} \Pi_{n-k} x+\gamma_{k}$$

for some non-negative-definite symmetric matrix $\Pi_{n-k}$ and some real constant $\gamma_{k}$. Show how to compute the matrix $\Pi_{n-k}$ and constant $\gamma_{k}$ and how to determine an optimal control.