Let m and M be respectively the minimum and maximum values of the determinant

$\begin{vmatrix}cos^2x & 1+sin^2x& sin 2x\\1+cos^2x& sin^2x& sin 2x\\cos^2x& sin^2x& 1+sin 2x\end{vmatrix}$

Then the ordered pair (m, M) is equal to:

(-3, -1)

The correct answer is option (3) : (-3, -1)

Let $Δ=\begin{vmatrix}cos^2x & 1+sin^2x& sin 2x\\1+cos^2x& sin^2x& sin 2x\\cos^2x& sin^2x& 1+sin 2x\end{vmatrix}$. Then

Applying $C_1→C_1+C_2, $ we obtain

$Δ=\begin{vmatrix}2 & 1+sin^2x& sin 2x\\2 & sin^2x & sin 2x\\2 & sin^2x& 1+sin 2x\end{vmatrix}$.

Applying $R_1→R_1-2R_3, R_2 → R_2-2R_3$, we obtain

$Δ=\begin{vmatrix}0 & cos^2x& -(2+sin 2x)\\0 & -sin^2x & -(2+sin 2x)\\1 & sin^2x& 1+sin 2x\end{vmatrix}$

Applying $R_1→R_1-R_2,R_3→R_3+R_2, $ we obtain

$⇒Δ= \begin{vmatrix}0 & 1 & 0\\0 & -sin^2x & -(2+sin 2x)\\1 &0 & -1\end{vmatrix}= -(2+sin2x)$

Now, $-1≤sin\, 2x ≤1$

$⇒-1≤sin\, 2x  ≤ 1 ⇒-3≤-2-sin\, 2x ≤ -1, ⇒ -3≤ Δ≤ -1$

Hence, $m=-3$ and $M=-1.$