If $A = \begin{bmatrix}1&-1\\2&-1\end{bmatrix}, B = \begin{bmatrix}a&1\\b&-1\end{bmatrix}$ and $(A + B)^2 = A^2 + B^2$ then 

$a = 1, b = 4$

The correct answer is Option (1) → $a = 1, b = 4$

$A = \begin{bmatrix} 1 & -1 \\ 2 & -1 \end{bmatrix},\quad B = \begin{bmatrix} a & 1 \\ b & -1 \end{bmatrix}$

Given: $(A + B)^2 = A^2 + B^2$

Expanding the left-hand side using the identity: $(A + B)^2 = A^2 + AB + BA + B^2$

So, $A^2 + AB + BA + B^2 = A^2 + B^2$

Subtracting $A^2 + B^2$ from both sides:

$AB + BA = 0$

This implies $AB = -BA$

Compute $AB$:

$AB = \begin{bmatrix} 1 & -1 \\ 2 & -1 \end{bmatrix} \begin{bmatrix} a & 1 \\ b & -1 \end{bmatrix}$

$= \begin{bmatrix} (1)(a) + (-1)(b) & (1)(1) + (-1)(-1) \\ (2)(a) + (-1)(b) & (2)(1) + (-1)(-1) \end{bmatrix}$

$= \begin{bmatrix} a - b & 1 + 1 \\ 2a - b & 2 + 1 \end{bmatrix}$

$= \begin{bmatrix} a - b & 2 \\ 2a - b & 3 \end{bmatrix}$

Compute $BA$:

$BA = \begin{bmatrix} a & 1 \\ b & -1 \end{bmatrix} \begin{bmatrix} 1 & -1 \\ 2 & -1 \end{bmatrix}$

$= \begin{bmatrix} (a)(1) + (1)(2) & (a)(-1) + (1)(-1) \\ (b)(1) + (-1)(2) & (b)(-1) + (-1)(-1) \end{bmatrix}$

$= \begin{bmatrix} a + 2 & -a -1 \\ b - 2 & -b + 1 \end{bmatrix}$

Now equating: $AB + BA = 0$

$\begin{bmatrix} a - b & 2 \\ 2a - b & 3 \end{bmatrix} + \begin{bmatrix} a + 2 & -a -1 \\ b - 2 & -b + 1 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix}$

Add the matrices:

$\begin{bmatrix} (a - b) + (a + 2) & 2 + (-a -1) \\ (2a - b) + (b - 2) & 3 + (-b + 1) \end{bmatrix}$

$= \begin{bmatrix} 2a - b + 2 & 1 - a \\ 2a - 2 & 4 - b \end{bmatrix}$

Now equate each element to 0:

$2a - b + 2 = 0$ → (1)

$1 - a = 0$ → (2)

$2a - 2 = 0$ → (3)

$4 - b = 0$ → (4)

From (2): $a = 1$

From (4): $b = 4$