Part 4 · Generative and Multimodal Models
Model Compression and Edge AI
Clever compression lets powerful models run faster, cost less, and even fit on your phone.
Key concepts
- Quantization stores numbers with fewer bits.
- Pruning removes connections that contribute little.
- Distillation trains a small student model on a large teacher's behavior.
- LoRA adapts a model by training a small add-on.
- Edge AI runs on local devices, improving privacy and offline use at some cost in capability.
Compression methods trade a small amount of quality for large gains in size and speed.
Formal view and evidence
- Established. Knowledge distillation trains on softened teacher outputs. The lottery ticket hypothesis found sparse trainable subnetworks in many tested settings. LoRA parameterizes the weight update as a low-rank product, .
- Debated. How to measure AI's total energy and environmental cost, because results depend on hardware, location, utilization, and scope.
- Open problem. Compression methods that preserve safety behavior as well as benchmark accuracy.
Common misconception
Common belief
"Smaller models are always much worse."
In practice
A well-tuned small model can beat a larger one on a focused task.
Key takeaways
- Quantization, pruning, and distillation reduce model cost.
- On-device inference can improve privacy and latency.
- Compressed models need fresh accuracy and safety evaluation.