If A=1/3 […] is an orthogonal matrix, then

Since A is an orthogonal matrix.

$∴AA^T = I$

$⇒\frac{1}{3}\begin{bmatrix}1&2&2\\2&1&-2\\a&2&b\end{bmatrix}.\frac{1}{3}\begin{bmatrix}1&2&a\\2&1&2\\2&-2&b\end{bmatrix}=\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}$

$⇒\frac{1}{9}\begin{bmatrix}1&2&2\\2&1&-2\\a&2&b\end{bmatrix}\begin{bmatrix}1&2&a\\2&1&2\\2&-2&b\end{bmatrix}=\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}$

$⇒\begin{bmatrix}9&0&a+4+2b\\0&9&2a+2-2b\\a+4+2b&2a+2-2b&a^2+4+b^2\end{bmatrix}=\begin{bmatrix}9&0&0\\0&9&0\\0&0&9\end{bmatrix}$

$⇒a+4+2b=0$   ...(1)

$2a+2-2b=0$   ...(2)

$a^2+4+b^2 = 9$   ...(3)

$\text{Eq. (2) - 2 × Eq. (1)}$

$-2-4b=0$

$⇒b=-1$

and, putting value of b in equation (1),

$a+4+(2)(-1)=0$

$a=-2$

$⇒a=-2, b=-1$