If A is a m × n matrix and B is n × p matrix and m ≠ p then,
Answer & explanation
Correct answer: option 1
The correct answer is Option (1) → (AB)′ = B′A′
A is m × n and B is n × p. The number of columns of A equals the number of rows of B, so AB is defined and is of order m × p.
By the reversal law for transposes, the transpose of a product is the product of the transposes in the reverse order:
\((AB)' = B'A'\)
Check of orders: B′ is p × n and A′ is n × m, so B′A′ is defined and is p × m, the same order as (AB)′.
Why the other options are incorrect
Option (2) (BA)′ = A′B′ — Incorrect. BA needs the p columns of B to match the m rows of A; since m ≠ p, BA is not defined.
Option (3) (AB)′ = A′B′ — Incorrect. A′ is n × m and B′ is p × n; since m ≠ p, A′B′ is not defined. The order must be reversed.
Option (4) (BA)′ = B′A′ — Incorrect. BA is not defined because m ≠ p.