If a, b and √3a + b are unit vectors, then the angle between a and b…

The correct answer is Option (4) → \(\displaystyle\frac{5\pi}{6}\)

Since \(|\vec{a}| = |\vec{b}| = 1\) and \(|\sqrt{3}\vec{a} + \vec{b}| = 1\), square the last one using \(|\vec{v}|^2 = \vec{v} \cdot \vec{v}\):

    \(|\sqrt{3}\vec{a} + \vec{b}|^2 = 3|\vec{a}|^2 + 2\sqrt{3}\,\vec{a} \cdot \vec{b} + |\vec{b}|^2\)

    \(1 = 3 + 2\sqrt{3}\,\vec{a} \cdot \vec{b} + 1\)

    \( \displaystyle \vec{a} \cdot \vec{b} = -\frac{3}{2\sqrt{3}} = -\frac{\sqrt{3}}{2} \)

If \(\theta\) is the angle between them, the dot product gives

    \( \displaystyle \cos\theta = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}|\,|\vec{b}|} = -\frac{\sqrt{3}}{2} \)

Since \(\cos\theta\) is negative, the angle is obtuse:

    \( \displaystyle \theta = \pi - \frac{\pi}{6} = \frac{5\pi}{6} \)