The maximum area of the rectangle whose sides pass through the angular points of a given the rectangle is of sides a and b is

$(\frac{1}{2})(a+b)^2$

Let ABCD be the given rectangle of sides a and b and EFGH be any rectangle, whose sides pass through A, B, C, D.

A = area EFGH

$= (b\sin θ+ a\cos θ) (a\sin θ+b\cos θ)$

$= ab+ (a^2 + b^2)\sin θ\cos θ$

$\frac{dA}{dθ}=(a^2 + b^2)\cos 2θ$ so $\frac{dA}{dθ}=0⇒θ=\frac{π}{4}$

$⇒\frac{d^2A}{dθ^2}=-(a^2 + b^2)\sin 2θ$;

so $\frac{d^2A}{dθ^2}|_{θ=\frac{π}{4}}<0$

Hence $A_{max}=(\frac{1}{2})(a+b)^2$