If A and B are two square matrices of order ‘\( n \)’, where \( \mathrm{AB}\ne\mathrm{BA} \), then
Answer & explanation
Correct answer: option 3
The correct answer is Option (3) → \( \mathrm{(A+B)^2=A^2+AB+BA+B^2} \)
Matrix multiplication is not commutative, so the order of the factors must be kept:
\( \displaystyle (A+B)^2=(A+B)(A+B) \)
\( \displaystyle =A^2+AB+BA+B^2 \)
AB and BA cannot be combined into 2AB, because AB ≠ BA.
Why the other options are incorrect
Option (1) \( \mathrm{(A+B)^2=A^2+2AB+B^2} \) — Incorrect. This treats AB and BA as equal, which is true only when A and B commute.
Option (2) \( \mathrm{(A+B)^2=A^2+2BA+B^2} \) — Incorrect. This also treats AB and BA as equal, writing both as BA.
Option (4) \( \mathrm{A^2-B^2=(A+B)(A-B)} \) — Incorrect. Expanding the right side gives \( A^2-AB+BA-B^2 \), which equals \( A^2-B^2 \) only if AB = BA.