Part 2 · Neural Networks
The Artificial Neuron
A neural network is many tiny decision-makers connected together, and each connection has a strength that can be learned.
Key concepts
- A neuron multiplies each input by a weight, adds a bias, and passes the result through an activation function.
- Neurons form layers. Layers between input and output are hidden layers.
- Without non-linear activations, stacked layers collapse into a single straight-line model.
- Networks learn useful features on their own, often building simple parts into complex ones.
Computation in one artificial neuron.
Formal view and evidence
- Established. A unit computes . A single linear threshold unit cannot represent XOR. The universal approximation theorem shows that one hidden layer with suitable non-linearity can approximate continuous functions on compact sets to any accuracy, given enough units.
- Established. Approximation is an existence result. It says nothing about whether training finds that network. Depth can represent some functions exponentially more efficiently than shallow networks.
- Open problem. A complete theory of which features networks learn and why.
Common misconception
Common belief
"Neural networks work just like the human brain."
In practice
They were loosely inspired by neurons, but biological neurons and brains work very differently.
Key takeaways
- A neuron applies weights, a bias, and a non-linear activation.
- Non-linearity is what gives stacked layers their expressive power.
- Being able to represent a function does not guarantee training will find it.