Speculative Decoding
An inference technique where a smaller draft model proposes tokens that a larger model verifies in parallel to reduce latency.
Plain English
A fast understudy guesses the next words; the main model checks them in batches.
Example
A serving stack uses a tiny draft model to propose eight tokens; the frontier model accepts six and regenerates the rest.
Why it matters
It is one of the practical ways to cut time-to-first-token and tokens-per-second without shrinking the target model.
Related