Cognitive

Part 2 · Neural Networks

Gradient Descent

Training a model means searching for parameters that minimize a loss. Gradient descent does this by repeatedly moving against the local slope.

5 min readBuilds on The Artificial Neuron

Each hop is one gradient descent update on a real loss function.

The loss landscape

A model's behavior is determined by its parameters θ. The loss L(θ) is a single number that measures how poorly those parameters perform on the training data. Every setting of the parameters has a loss, so training can be pictured as moving across a surface whose height is the loss.

A real network has millions or billions of parameters, so this surface cannot be seen directly. The optimizer only has local information: the value and the slope at its current position.

The update rule

The gradient ∇L(θ) points in the direction of steepest increase. Gradient descent takes a step in the opposite direction, scaled by the learning rate η:

θt+1=θt−η∇L(θt)

The learning rate is usually the most important setting to tune. If it is too small, progress is slow. If it is too large, the iterates overshoot the minimum, oscillate, or diverge.

This interactive figure requires JavaScript.

Figure 7.1. Gradient descent on a one-dimensional loss, computed live. Change the learning rate to see slow convergence, oscillation and divergence. Start from the left and the iterates settle in the shallower local minimum.

Established. If L is convex and M-smooth, then with η≤1/M the suboptimality after t steps is bounded by:

L(θt)−L(θ*)≤‖θ0−θ*‖22ηt

Neural network losses are non-convex, so this guarantee does not carry over to them.

Training continues to reduce training error even after validation error begins to rise.

Stochastic gradients

Computing the gradient over an entire dataset is expensive. Stochastic gradient descent estimates it from a small random mini-batch B. With uniform sampling, the estimate is unbiased:

gt=1|B|∑i∈B∇ℓi(θt),𝔼[gt]=∇L(θt)

Each step is noisier, but many more steps can be taken for the same amount of computation.

Momentum and adaptive methods

Momentum accumulates a decaying sum of past gradients. This damps oscillation across narrow valleys and builds speed along consistent directions:

vt+1=βvt+gt,θt+1=θt−ηvt+1

Adam combines momentum with a separate adaptive step size for each parameter, based on running estimates of the first and second moments of the gradient.

This interactive figure requires JavaScript.

Figure 7.2. SGD, momentum and Adam on the same surface, computed live. Lighter regions have lower loss. Click the surface to choose a starting point. The ravine shows how momentum and adaptive steps handle poorly conditioned problems.

Open questions

Debated. Flat minima are often linked to better generalization, but sharpness depends on how a network is parameterized, so simple flatness measures can mislead.

Open problem. Why large non-convex networks train so reliably is not fully understood. In high dimensions, saddle points appear to be a more prominent obstacle than poor local minima.

Common misconception

Common belief

"Training finds the single best set of parameters."

In practice

Training finds a good low region of the loss surface. Different runs can reach different solutions that perform similarly well.

Key takeaways

  • Training repeatedly moves parameters against the gradient of the loss.
  • The learning rate controls the balance between speed and stability.
  • Mini-batches, momentum, and adaptive methods make training practical at scale.

Explore next