# Speculative Decoding

> An inference technique where a smaller draft model proposes tokens that a larger model verifies in parallel to reduce latency.

- Category: Reasoning & Inference
- Stage: Rising
- Also called: Speculative sampling, Draft model decoding
- Canonical: https://www.joelcsaji.com/ai-dictionary/speculative-decoding

## Plain English

A fast understudy guesses the next words; the main model checks them in batches.

## Example

A serving stack uses a tiny draft model to propose eight tokens; the frontier model accepts six and regenerates the rest.

## Why it matters

It is one of the practical ways to cut time-to-first-token and tokens-per-second without shrinking the target model.

## Related

- [Latency](https://www.joelcsaji.com/ai-dictionary/latency)
- [Streaming](https://www.joelcsaji.com/ai-dictionary/streaming)
- [Quantization](https://www.joelcsaji.com/ai-dictionary/quantization)

## Sources

- [OpenAI: Speculative Decoding](https://platform.openai.com/docs/guides/latency-optimization)
- [Google Cloud Generative AI Glossary](https://docs.cloud.google.com/docs/generative-ai/glossary)