Part 3 · Language Models
Instruction Tuning and RLHF
After learning language, a model is taught through examples and feedback to follow instructions and be helpful.
Key concepts
- A raw pre-trained model simply continues text.
- Instruction tuning trains on examples of requests and good responses.
- RLHF uses human preferences to train a reward model, which guides the assistant.
- Constitutional AI uses written principles and AI feedback to reduce reliance on human labels.
- Sycophancy can appear when a model learns that agreeable answers get rewarded.
Reinforcement learning from human feedback turns preference judgments into a training signal.
Formal view and evidence
- Established. Preference learning commonly fits a reward model by comparing response pairs, then optimizes a policy against it. Direct Preference Optimization avoids an explicit reinforcement-learning step by deriving a classification objective on preference pairs.
- Debated. How well human preference data captures truth and long-term user interests.
- Open problem. scalable oversight, meaning reliable supervision of systems on tasks that humans cannot easily evaluate. Reward models are also vulnerable to over-optimization.
Common misconception
Common belief
"Alignment training teaches a model the truth."
In practice
It shapes behavior based on feedback, and that feedback can be incomplete or biased.
Key takeaways
- Instruction tuning teaches a model to respond to requests.
- Preference training rewards the responses people judge to be better.
- Optimizing for approval can create sycophancy.