Cognitive

Part 2 · Neural Networks

Backpropagation

Backpropagation works out how much every connection contributed to a mistake, so each one knows how to change.

3 min readBuilds on Gradient Descent

Predictions flow forward, then gradients flow backward through every layer.

Key concepts

  • The forward pass makes a prediction. The loss measures the error.
  • The backward pass uses the chain rule to give each weight a gradient.
  • An optimizer then updates the weights.
  • Vanishing gradients make the signal fade in deep networks. Exploding gradients make it grow out of control.
  • Residual connections, normalization, and careful initialization keep the signal healthy.

A forward pass computes the loss, and a backward pass assigns gradients to every layer.

Figure 8.1. A forward pass computes the loss, and a backward pass assigns gradients to every layer.
Residual connections keep gradients strong in deep networks.

Formal view and evidence

  • Established. Backpropagation is reverse-mode automatic differentiation. For scalar loss L, it computes all partial derivatives at a small constant multiple of the cost of evaluating L. The chain rule gives ∂L∂𝐡k=(∂𝐡k+1∂𝐡k)⊤∂L∂𝐡k+1.
  • Debated. Whether brains implement anything functionally similar to backpropagation.
  • Open problem. Learning rules that approach backpropagation's performance while remaining local and biologically plausible.

Common misconception

Common belief

"Backpropagation is the learning algorithm."

In practice

It only computes the gradients. An optimizer such as SGD or Adam makes the updates.

Key takeaways

  • Backpropagation computes gradients using the chain rule in reverse.
  • An optimizer, not backpropagation, updates the weights.
  • Residual connections and normalization help gradients survive deep networks.

Explore next