The system of equations $x+ 2y - 3z=a, 2x+ 6y -11z=b, x- 2y + 7z = c $ has

infinite solution for 5a = 2b + c

The correct answer is option (3) : infinite solution for 5a = 2b + c

For the given system of equations, we have

$D=\begin{vmatrix}1 & 2 & -3\\2 & 6 & -11\\1 & -2 & 7\end{vmatrix}=20 -50 + 30 =0$

$D_1= \begin{vmatrix}a & 2 & -3\\b & 6 & -11\\c & -2 & 7\end{vmatrix}=20a -2(7b+11c) -3(-2b-6c)$

$D_2= \begin{vmatrix}1 & a & -3\\2 & b & -11\\1 & c & 7\end{vmatrix}=-5(5a-2b-c)$

$D_3=\begin{vmatrix}1 & 2 &a\\2 & 6 & b\\1 & -2 & c\end{vmatrix}=-2(5a-2b-c)$

If $5a-2b-c=0,$ then $D_1= D_2= D_3= 0$ and so the system of equations has infinitely many solutions.