If A and B are square matrices such that $A^* = A$ and $B^* = B$, where $A^*$ denotes the conjugate transpose of A, then $(AB-BA)^* =$
Answer & explanation
Correct answer: option 3
We have,
$(AB-BA)^* = (\overline{AB-BA})^T= (\overline{A}\, \overline{B}- \overline{B}\,\overline{A})^T=(\overline{A}\, \overline{B})^T - (\overline{B}\,\overline{A})^T$
$⇒(AB-BA)^* = (\overline{B})^T (\overline{A})^T (\overline{A})^T (\overline{B})^T - = -B^* A^* A^* B^*$
$⇒(AB-BA)^* = BA-AB$