If $A=\begin{bmatrix}1&0&0\\0&1&0\\a&b&-1\end{bmatrix}$, then $A^2$ is equal to

a null matrix a unit matrix $-A$ $A$

a unit matrix

We have, $A^2=\begin{bmatrix}1&0&0\\0&1&0\\a&b&-1\end{bmatrix}\begin{bmatrix}1&0&0\\0&1&0\\a&b&-1\end{bmatrix}=\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}$ = a unit matrix.