Cognitive

Part 3 · Language Models

Embeddings

Embeddings place words and ideas on a map where similar meanings sit close together.

3 min readBuilds on Tokenization

Embeddings place related ideas near each other in vector space.

Key concepts

  • An embedding is a list of numbers representing meaning.
  • Similar items get nearby vectors, commonly compared by cosine similarity.
  • Semantic search finds related meaning without matching exact words.
  • Sentences, images, and documents can be embedded too.
  • Embeddings absorb patterns from their training data, including social biases.

Semantic search compares meaning by comparing vectors.

Figure 14.1. Semantic search compares meaning by comparing vectors.
Words that appear in similar contexts form neighborhoods of meaning.

Formal view and evidence

  • Established. The distributional hypothesis says words in similar contexts tend to have similar meanings. Skip-gram with negative sampling implicitly factorizes a shifted PMI matrix under certain assumptions. Cosine similarity is cos⁡ฮธ=๐šยท๐›โ€–๐šโ€–โ€–๐›โ€–.
  • Debated. The famous king โˆ’ man + woman โ‰ˆ queen result is often overstated. Standard analogy evaluation excludes the input words from the answers, which flatters the method.
  • Open problem. Whether geometric proximity captures meaning or only usage, and how to remove bias without degrading useful information.

Common misconception

Common belief

"AI stores a dictionary definition for each word."

In practice

It stores positions and relationships learned from usage.

Key takeaways

  • Embeddings represent meaning as vectors.
  • Similarity between vectors enables search by meaning.
  • Embeddings also absorb patterns and biases from their training data.

Explore next