Part 2 · Neural Networks
Backpropagation
Backpropagation works out how much every connection contributed to a mistake, so each one knows how to change.
Key concepts
- The forward pass makes a prediction. The loss measures the error.
- The backward pass uses the chain rule to give each weight a gradient.
- An optimizer then updates the weights.
- Vanishing gradients make the signal fade in deep networks. Exploding gradients make it grow out of control.
- Residual connections, normalization, and careful initialization keep the signal healthy.
A forward pass computes the loss, and a backward pass assigns gradients to every layer.
Formal view and evidence
- Established. Backpropagation is reverse-mode automatic differentiation. For scalar loss , it computes all partial derivatives at a small constant multiple of the cost of evaluating . The chain rule gives .
- Debated. Whether brains implement anything functionally similar to backpropagation.
- Open problem. Learning rules that approach backpropagation's performance while remaining local and biologically plausible.
Common misconception
Common belief
"Backpropagation is the learning algorithm."
In practice
It only computes the gradients. An optimizer such as SGD or Adam makes the updates.
Key takeaways
- Backpropagation computes gradients using the chain rule in reverse.
- An optimizer, not backpropagation, updates the weights.
- Residual connections and normalization help gradients survive deep networks.