Cognitive

Part 2 · Neural Networks

The Artificial Neuron

A neural network is many tiny decision-makers connected together, and each connection has a strength that can be learned.

3 min readBuilds on Machine Learning Paradigms

Signals pass through weighted connections from layer to layer.

Key concepts

  • A neuron multiplies each input by a weight, adds a bias, and passes the result through an activation function.
  • Neurons form layers. Layers between input and output are hidden layers.
  • Without non-linear activations, stacked layers collapse into a single straight-line model.
  • Networks learn useful features on their own, often building simple parts into complex ones.

Computation in one artificial neuron.

Figure 6.1. Computation in one artificial neuron.
Non-linear activation functions let networks represent complex patterns.

Formal view and evidence

  • Established. A unit computes y=σ(𝐰⊤𝐱+b). A single linear threshold unit cannot represent XOR. The universal approximation theorem shows that one hidden layer with suitable non-linearity can approximate continuous functions on compact sets to any accuracy, given enough units.
  • Established. Approximation is an existence result. It says nothing about whether training finds that network. Depth can represent some functions exponentially more efficiently than shallow networks.
  • Open problem. A complete theory of which features networks learn and why.

Common misconception

Common belief

"Neural networks work just like the human brain."

In practice

They were loosely inspired by neurons, but biological neurons and brains work very differently.

Key takeaways

  • A neuron applies weights, a bias, and a non-linear activation.
  • Non-linearity is what gives stacked layers their expressive power.
  • Being able to represent a function does not guarantee training will find it.

Explore next