Consider the general linear model $Y=X \beta^{0}+\varepsilon$ where $X$ is a known $n \times p$ design matrix with $p \geqslant 2, \beta^{0} \in \mathbb{R}^{p}$ is an unknown vector of parameters, and $\varepsilon \in \mathbb{R}^{n}$ is a vector of stochastic errors with $\mathbb{E}\left(\varepsilon_{i}\right)=0, \operatorname{var}\left(\varepsilon_{i}\right)=\sigma^{2}>0$ and $\operatorname{cov}\left(\varepsilon_{i}, \varepsilon_{j}\right)=0$ for all $i, j=1, \ldots, n$ with $i \neq j$. Suppose $X$ has full column rank.

(a) Write down the least squares estimate $\hat{\beta}$ of $\beta^{0}$ and show that it minimises the least squares objective $S(\beta)=\|Y-X \beta\|^{2}$ over $\beta \in \mathbb{R}^{p}$.

(b) Write down the variance-covariance matrix $\operatorname{cov}(\hat{\beta})$.

(c) Let $\tilde{\beta} \in \mathbb{R}^{p}$ minimise $S(\beta)$ over $\beta \in \mathbb{R}^{p}$ subject to $\beta_{p}=0$. Let $Z$ be the $n \times(p-1)$ submatrix of $X$ that excludes the final column. Write $\operatorname{down} \operatorname{cov}(\tilde{\beta})$.

(d) Let $P$ and $P_{0}$ be $n \times n$ orthogonal projections onto the column spaces of $X$ and $Z$ respectively. Show that for all $u \in \mathbb{R}^{n}, u^{T} P u \geqslant u^{T} P_{0} u$.

(e) Show that for all $x \in \mathbb{R}^{p}$,

$$\operatorname{var}\left(x^{T} \tilde{\beta}\right) \leqslant \operatorname{var}\left(x^{T} \hat{\beta}\right) .$$

[Hint: Argue that $x=X^{T} u$ for some $u \in \mathbb{R}^{n}$.]