Speculative Decoding

Definition

Speculative decoding is an inference technique where a faster draft process proposes several future tokens, and the main target model verifies those tokens.

The plain-English version:

guess several next tokens cheaply
check them with the real model
accept the ones that match the target model's behavior

The goal is to reduce generation latency without changing the intended output distribution of the target model.

Why This Concept Exists

Autoregressive language models generate text one token at a time.

The basic loop is:

run model -> choose next token
run model again -> choose next token
run model again -> choose next token

If an answer has hundreds of tokens, this serial loop becomes a latency problem.

Speculative decoding exists because some future tokens are easier to guess than others. A smaller or cheaper drafter can propose likely continuations, and the larger target model can verify multiple proposed tokens in a more efficient step.

The Beginner Mental Model

A beginner may think:

Speculative decoding lets the small model write part of the answer.

That is the wrong product mental model.

The draft model proposes. The target model verifies. The final accepted output should still be governed by the target model's distribution under the decoding method.

A better picture:

draft possible tokens
target model checks them
accepted tokens move the generation forward
rejected token causes correction

The drafter is an accelerator, not the authority.

How Verification Helps

Suppose the current text is:

The request timed out because

A draft model may propose:

the database connection was slow

Instead of asking the target model for one token at a time, the serving system can ask the target model to evaluate the proposed continuation and accept a prefix of it if it agrees under the decoding rule.

If several drafted tokens are accepted, the system moves forward faster than ordinary one-token decoding.

If the target model rejects early, the system falls back to the target model's choice and continues.

Acceptance Rate

The speedup depends heavily on how often drafted tokens are accepted.

If the draft model is too weak, it guesses poorly and many tokens are rejected.

If the draft model is too expensive, the draft step may erase the benefit.

The serving question becomes:

is the draft path cheap enough and accurate enough to reduce total latency?

This is why speculative decoding is a system tradeoff, not only an algorithm label.

Product And Infrastructure Pressure

Speculative decoding matters when user experience depends on generation latency.

In a chat assistant, faster decoding makes long answers feel more responsive.

In a coding assistant, lower latency can make edits and completions feel interactive.

In a summarization product, speculative decoding can reduce the wall-clock time for long outputs.

In high-throughput serving, faster generation can improve hardware utilization and cost.

Failure Modes

Common failure modes include:

• draft model acceptance is too low
• draft model cost is too high
• serving complexity increases more than latency improves
• batching becomes harder across draft and target work
• quality checks focus only on speed and miss behavior regressions
• the product uses a speculative setup that does not match its decoding policy

The safe evaluation question is not only "did it get faster?" It is "did it get faster under the same output-quality contract the product needs?"

Common Confusions

Speculative decoding is not the same thing as distillation.

Distillation trains a smaller model from a larger model's behavior. Speculative decoding uses a draft-and-verify serving strategy.

Speculative decoding is not model routing.

The target model still verifies the accepted output path.

Speculative decoding does not make every request faster.

The benefit depends on acceptance rate, draft cost, target cost, sequence length, batching, and hardware.

Speculative decoding is not the same thing as beam search.

Beam search explores multiple candidate continuations for output quality or search behavior. Speculative decoding uses draft candidates to accelerate target-model generation.

What This Does Not Mean

Speculative decoding does not remove the need for good serving architecture.

The system still needs KV cache management, batching, scheduling, observability, and evals.

It also does not mean the product should hide quality risk behind a speed metric. Faster output is only useful when the behavior contract remains acceptable.

Related Topics

• LLM Inference Serving
• KV Cache
• Quantization
• LLM Evaluation