Cognitive

Part 4 · Generative and Multimodal Models

Model Compression and Edge AI

Clever compression lets powerful models run faster, cost less, and even fit on your phone.

3 min readBuilds on Large Language Models

Quantization stores weights with fewer distinct values.

Key concepts

  • Quantization stores numbers with fewer bits.
  • Pruning removes connections that contribute little.
  • Distillation trains a small student model on a large teacher's behavior.
  • LoRA adapts a model by training a small add-on.
  • Edge AI runs on local devices, improving privacy and offline use at some cost in capability.

Compression methods trade a small amount of quality for large gains in size and speed.

Figure 23.1. Compression methods trade a small amount of quality for large gains in size and speed.
Compression keeps the connections that matter most.

Formal view and evidence

  • Established. Knowledge distillation trains on softened teacher outputs. The lottery ticket hypothesis found sparse trainable subnetworks in many tested settings. LoRA parameterizes the weight update as a low-rank product, ΔW=BA.
  • Debated. How to measure AI's total energy and environmental cost, because results depend on hardware, location, utilization, and scope.
  • Open problem. Compression methods that preserve safety behavior as well as benchmark accuracy.

Common misconception

Common belief

"Smaller models are always much worse."

In practice

A well-tuned small model can beat a larger one on a focused task.

Key takeaways

  • Quantization, pruning, and distillation reduce model cost.
  • On-device inference can improve privacy and latency.
  • Compressed models need fresh accuracy and safety evaluation.