If $A =\begin{bmatrix}1&-1\\2&-1\end{bmatrix},B=\begin{bmatrix}a&1\\b&-1\end{bmatrix}$ and $(A + B)^2 = A^2 + B^2$, then the values of a and b are

$a =1, b = 4$

We have,

$A +B=\begin{bmatrix}a+1&0\\b+2&-2\end{bmatrix}$

$A^2 =\begin{bmatrix}1&-1\\2&-1\end{bmatrix}\begin{bmatrix}1&-1\\2&-1\end{bmatrix}=\begin{bmatrix}-1&0\\0&-1\end{bmatrix}$

$B^2=\begin{bmatrix}a&1\\b&-1\end{bmatrix}\begin{bmatrix}a&1\\b&-1\end{bmatrix}=\begin{bmatrix}a^2+b&a-1\\ab-b&b+1\end{bmatrix}$

$(A + B)^2=\begin{bmatrix}a+1&0\\b+2&-2\end{bmatrix}\begin{bmatrix}a+1&0\\b+2&-2\end{bmatrix}=\begin{bmatrix}(a+1)^2&0\\(a-1)(b+2)&4\end{bmatrix}$

$∴(A + B)^2=A^2+B^2$

$⇒\begin{bmatrix}(a+1)^2&0\\(a-1)(b+2)&4\end{bmatrix}=\begin{bmatrix}-1&0\\0&-1\end{bmatrix}+\begin{bmatrix}a^2+b&a-1\\ab-b&b+1\end{bmatrix}$

$⇒\begin{bmatrix}(a+1)^2&0\\(a-1)(b+2)&4\end{bmatrix}=\begin{bmatrix}a^2+b-1&a-1\\ab-b&b\end{bmatrix}$

$⇒a-1=0, b = 4, (a+1)^2 = a^2+b-1, (a-1) (b + 2) = ab-b$

$⇒a =1$ and $b = 4$