Let a, b, c be the distinct non-negative. If the vectors $a\hat i + a\hat j + c\hat k$, $\hat i + \hat k$ and $c\hat i + c\hat j + b\hat k$ lie in a plane, then c is

G.M. of a and b

Since these vectors are coplaner, $\begin{vmatrix}a&a&c\\1&0&1\\c&c&b\end{vmatrix}=0$

$⇒ - ac – a(b –c) + c^2= 0 ⇒c^2 = ab$

Hence (B) is the correct answer.