# Distillation

> Training a smaller model to imitate the behavior or outputs of a larger model.

- Category: Models & Training
- Stage: Established
- Also called: Knowledge distillation, Teacher-student training
- Canonical: https://www.joelcsaji.com/ai-dictionary/distillation

## Plain English

A big model teaches a smaller one.

## Example

A team uses a frontier model to generate labels, then trains a cheaper classifier on those outputs.

## Why it matters

Distillation helps reduce cost and latency while preserving enough task quality.

## Related

- [Small Language Model](https://www.joelcsaji.com/ai-dictionary/small-language-model)
- [Synthetic Data](https://www.joelcsaji.com/ai-dictionary/synthetic-data)
- [Fine-Tuning](https://www.joelcsaji.com/ai-dictionary/fine-tuning)

## Sources

- [Stanford HAI: 2026 AI Index](https://hai.stanford.edu/ai-index/2026-ai-index-report)
- [Google Cloud Generative AI Glossary](https://docs.cloud.google.com/docs/generative-ai/glossary)