Part 3 · Language Models
Tokenization
AI does not read letters or whole words. It reads chunks called tokens.
Key concepts
- A tokenizer splits text into pieces and maps each to an ID from a fixed vocabulary.
- In English, a token averages roughly ¾ of a word, but this varies widely.
- Spaces, punctuation, capitals, and rare words change how text splits.
- Models see tokens rather than letters, which helps explain why counting letters can be difficult.
- Usage limits and pricing are usually measured in tokens.
How text becomes the integer IDs that a language model receives.
Formal view and evidence
- Established. Byte-pair encoding, adapted to neural translation by Sennrich et al., repeatedly merges frequent symbol pairs. Tokenization is a modeling decision that affects cost and performance.
- Established. Tokenizers can require substantially more tokens for the same content in some languages, creating unequal cost and latency.
- Open problem. Tokenization that serves all languages fairly, and models that operate well directly on bytes or characters.
Common misconception
Common belief
"One token equals one word."
In practice
A token can be a word, part of a word, punctuation, or a space-prefixed fragment.
Key takeaways
- Tokenizers convert text into integer IDs from a fixed vocabulary.
- Token count affects cost, speed, and how much fits in context.
- The same content can require more tokens in some languages.