If A= […] and f(x)=x^2-2x-3 , then f(A) is equal to

The correct answer is Option (2) → O (Null matrix)

In a matrix polynomial the constant term becomes a multiple of I: \( f(A)=A^2-2A-3I \).

    \( \displaystyle A^2=\begin{bmatrix}1 & 2\\2 & 1\end{bmatrix}\begin{bmatrix}1 & 2\\2 & 1\end{bmatrix} \)

    \( \displaystyle =\begin{bmatrix}5 & 4\\4 & 5\end{bmatrix} \)

    \( \displaystyle f(A)=\begin{bmatrix}5 & 4\\4 & 5\end{bmatrix}-\begin{bmatrix}2 & 4\\4 & 2\end{bmatrix}-\begin{bmatrix}3 & 0\\0 & 3\end{bmatrix} \)

    \( \displaystyle =\begin{bmatrix}0 & 0\\0 & 0\end{bmatrix} \)

    \( \displaystyle =O \)

Shortcut: By the Cayley-Hamilton theorem,

    \( \displaystyle A^2-(\mathrm{tr}\,A)A+|A|\,I=O \)

Here tr A = 2 and |A| = 1 − 4 = −3, so \( A^2-2A-3I=O \) directly.