Cognitive

Part 3 · Language Models

Instruction Tuning and RLHF

After learning language, a model is taught through examples and feedback to follow instructions and be helpful.

3 min readBuilds on Reinforcement Learning, Large Language Models

Human preferences become a signal for improving model behavior.

Key concepts

  • A raw pre-trained model simply continues text.
  • Instruction tuning trains on examples of requests and good responses.
  • RLHF uses human preferences to train a reward model, which guides the assistant.
  • Constitutional AI uses written principles and AI feedback to reduce reliance on human labels.
  • Sycophancy can appear when a model learns that agreeable answers get rewarded.

Reinforcement learning from human feedback turns preference judgments into a training signal.

Figure 17.1. Reinforcement learning from human feedback turns preference judgments into a training signal.
Rewarding agreeable answers can accidentally encourage sycophancy.

Formal view and evidence

  • Established. Preference learning commonly fits a reward model by comparing response pairs, then optimizes a policy against it. Direct Preference Optimization avoids an explicit reinforcement-learning step by deriving a classification objective on preference pairs.
  • Debated. How well human preference data captures truth and long-term user interests.
  • Open problem. scalable oversight, meaning reliable supervision of systems on tasks that humans cannot easily evaluate. Reward models are also vulnerable to over-optimization.

Common misconception

Common belief

"Alignment training teaches a model the truth."

In practice

It shapes behavior based on feedback, and that feedback can be incomplete or biased.

Key takeaways

  • Instruction tuning teaches a model to respond to requests.
  • Preference training rewards the responses people judge to be better.
  • Optimizing for approval can create sycophancy.

Explore next