Newton's method for 2D minimization

Click anywhere on the plot to set the starting point x₀.

Newton's method for 2D minimization

In two dimensions, Newton's method minimizes $f(\mathbf{x})$, $\mathbf{x} = (x_1, x_2)^\top$, by iteratively building a local quadratic (second-order Taylor) model and jumping to its minimizer.

The quadratic model at $\mathbf{x}_n$ is:

where $\nabla f$ is the gradient and $\mathbf{H}$ is the Hessian matrix. Setting $\nabla q = 0$ gives the update:

The ellipses drawn at each iterate are level curves of the local quadratic model $q(\mathbf{x}) = \text{const}$ centred at $\mathbf{x}_n$. Their shape is entirely determined by the Hessian: a well-conditioned Hessian gives nearly circular contours and fast convergence; an ill-conditioned one gives elongated ellipses and slower convergence.

Example functions

Elliptic quadratic — a simple convex bowl with different curvatures along each axis. The Hessian is constant, so Newton's method converges in a single step from any starting point. $$f(x_1, x_2) = 3(x_1-1)^2 + 8(x_2-0.5)^2, \qquad \mathbf{x}^* = (1,\; 0.5)$$

Rosenbrock function — the classic "banana valley" test problem. The minimum lies inside a narrow curved valley, producing a highly ill-conditioned Hessian and very elongated ellipses. Pure Newton's method still converges quickly once inside the valley. $$f(x_1, x_2) = 100(x_2 - x_1^2)^2 + (1 - x_1)^2, \qquad \mathbf{x}^* = (1,\; 1)$$

Himmelblau's function — a non-convex function with four minima. The starting point determines which minimum Newton's method converges to; near saddle points the Hessian is indefinite and the step may point away from any minimum. $$f(x_1, x_2) = (x_1^2 + x_2 - 11)^2 + (x_1 + x_2^2 - 7)^2$$ $$\mathbf{x}^* \in \{(3,2),\;(-2.805,\,3.131),\;(-3.779,\,-3.283),\;(3.584,\,-1.848)\}$$