The function $f(x)= \begin{cases}x^2 / a & , 0 \leq x<1 \\ a & , 1 \leq x<\sqrt{2} \\ \frac{2 b^2-4 b}{x^2}, & \sqrt{2} \leq x<\infty\end{cases}$ is continuous for $0 \leq x<\infty$, then the most suitable values of a and b are

a = -1, b = 1

It is given that f(x) is continuous for $0 \leq x<\infty$. So, it is continuous at x = 1 and $x=\sqrt{2}$.

∴   $\lim\limits_{x \rightarrow 1^{-}} f(x)=\lim\limits_{x \rightarrow 1^{+}} f(x)=f(1)$

and,  $\lim\limits_{x \rightarrow(\sqrt{2})} f(x)=\lim\limits_{x \rightarrow(\sqrt{2})^{+}} f(x)=f(\sqrt{2})$

$\Rightarrow \lim\limits_{x \rightarrow 1} \frac{x^2}{a}=\lim\limits_{x \rightarrow 1} a=a$

and, $\lim\limits_{x \rightarrow(\sqrt{2})^{-}} a=\lim\limits_{x \rightarrow(\sqrt{2})^{+}} \frac{2 b^2-4 b}{x^2}=\frac{2 b^2-4 b}{2}$

$\Rightarrow \frac{1}{a}=a$  and  $  a=\frac{2 b^2-4 b}{2}$

$\Rightarrow a= \pm 1$ and $b^2-2 b=a$

Now,  a = 1  and $b^2-2 b=a$

⇒ a = 1, b = $1 \pm \sqrt{2}$

⇒ a = -1  and $b^2-2 b=a$

⇒ a = -1, b = 1