Does speculative decoding change model output quality?

No. The target model verifies every draft token in a single forward pass and recomputes any token it rejects, so the output distribution matches the target model. The optimization trades spare memory for latency, not accuracy for speed.

What acceptance rate should I look for in vLLM?

A high acceptance rate means most draft tokens were kept and the speedup is real. In the 2026 Blackwell demo, structured output showed a high rate and a 1.6x speedup, while creative writing showed a low rate. Watch the metric on your own workload rather than relying on published numbers.

Which tools support speculative decoding?

Kirui implemented her demo with vLLM, an open-source serving engine with documentation for getting started. More advanced draft architectures include EAGLE and Medusa, which change the drafting mechanism underneath rather than the verify-and-accept principle.

Why doesn't speculative decoding help long-context workloads?

It accelerates the decode phase only. RAG and document-analysis jobs spend much of their time in prefill, building the KV cache from large inputs, so drafting extra output tokens saves little when generation is short relative to the input.

Speculative Decoding vLLM: When It's Worth It

Speculative decoding vLLM users can now evaluate with real numbers: one Blackwell demo ran 1.6x faster on structured output. Learn when to turn it on.

What Is Speculative Decoding in vLLM?

Speculative decoding in vLLM is an inference optimization where a small draft model proposes several tokens and the large target model verifies them in a single forward pass, cutting generation latency without changing output accuracy. Sheilah Kirui, a developer advocate at Akamai, demonstrated the technique at the AI Engineer World's Fair 2026 in San Francisco using vLLM, an open-source LLM serving engine, on a single NVIDIA Blackwell GPU.

Her setup ran both a baseline and a speculative configuration side by side on one GPU. The headline result: a structured-output workload finished 1.6x faster with a high acceptance rate, while a creative-writing prompt saw a much lower acceptance rate and far smaller gains. Those numbers come from her live demo, not from an independent benchmark.

The technique itself is well established in the serving stack. vLLM's documentation covers it directly, and more advanced draft architectures such as EAGLE and Medusa build on the same verify-then-accept idea with extra heads or draft trees.

Prefill vs Decode: What Speculative Decoding Actually Speeds Up

LLM inference has two phases, and speculative decoding accelerates only one of them. In the prefill phase, the model reads your prompt, processes all input tokens once, and builds the KV cache, the model's working memory for the rest of the request. In the decode phase, the model writes the answer one token at a time, because each token depends on the tokens before it.

Generating a long answer from a large model like a Llama 70B can take hundreds of sequential forward passes, and that serial decode loop is where latency comes from. Speculative decoding attacks exactly this loop: a small model guesses three to five tokens per cycle, and the target model approves or rejects them all in one pass. Rejected tokens are recomputed by the target model, so output quality matches the target model.

Because prefill is untouched, workloads dominated by long inputs gain little. A RAG application feeding large amounts of context, or a document-analysis job where the model reads far more than it writes, spends most of its time building the KV cache rather than decoding. Kirui called this out as one of the clearest cases where speculative decoding is not worth enabling.

The Memory Cost of Hosting a Second Model

Speculative decoding is not free. Enabling it means hosting two models on the GPU, each with its own weights and its own KV cache allocation. The draft model is small, but the second KV cache is real memory that a baseline deployment would have given back to the target model.

In Kirui's demo on a single Blackwell GPU, the baseline target model needed about 16 GB of weight memory and the draft model about 2.5 GB, leaving the rest of the card for KV caching across both servers. Because she had to split one GPU between the baseline and speculative setups, she deliberately picked a draft model small enough to fit both.

The practical implication is a capacity trade. If your GPUs are already busy serving high-concurrency traffic, the extra memory and compute spent on speculative decoding compete with the requests you are already running. Spare VRAM is a precondition for the optimization, not a detail.

How to Choose a Draft Model

Draft model selection balances two questions: can the draft predict the target's tokens accurately, and is it fast enough that guessing beats waiting? Kirui's checklist for choosing one is short and concrete:

Size: pick a model roughly 10 to 50 times smaller than the target model.
Tokenizer: it must share the target's tokenizer, ideally from the same model family, or you will have to translate between token vocabularies manually.
Cost: the draft should be a small fraction of the target's per-token cost.
Accuracy: the higher the draft's match rate with the target, the higher the acceptance rate and the bigger the speedup.

The acceptance rate is the ratio of draft tokens the target accepts to the total tokens the draft generated. It is the single most useful number to watch when deciding whether speculative decoding is helping your workload.

Demo Results: Structured Output vs Creative Writing on Blackwell

Kirui ran two workloads side by side in vLLM on one NVIDIA Blackwell GPU, comparing baseline decoding against speculative decoding with the same target model. The results are from her live demonstration and reported as speaker-observed figures:

Workload	Speedup vs baseline	Acceptance rate	Why
Structured output (JSON / code-style prompts)	1.6x faster	High	Next tokens are predictable, so the draft matches the target often
Creative writing (high temperature)	Modest	Low	High token variety means most drafts get rejected

The structured case is where speculative decoding shines. When the model is writing JSON or SQL, the distribution of likely next tokens is narrow, so a small draft model guesses correctly most of the time and the target verifies batches of tokens in one pass. Throughput in tokens per second rose accordingly.

The creative case inverts that. With temperature set high and many plausible continuations for every position, the draft's guesses miss often, the target rejects them, and the overhead of drafting adds cost without adding speed. The measured acceptance rate dropped visibly in her demo, which is the expected behavior rather than a misconfiguration.

When Speculative Decoding Is Worth Enabling

Whether to enable speculative decoding depends on four properties of your workload, and Kirui's summary is a useful decision list:

You have spare GPU memory. The second model and its KV cache must fit alongside your target model and traffic.
Your batches are small. Low-concurrency deployments leave both memory and compute headroom; high-concurrency serving is usually better off without it.
Your output is structured. Coding, JSON generation, and SQL prompts have predictable token distributions and accept drafts at high rates.
Your inputs are short relative to outputs. Long-context workloads such as RAG or document analysis spend their time in prefill, which speculative decoding does not accelerate.

If your workload fails most of those checks, the sensible default is to skip the feature and keep the full GPU budget for the target model's KV cache. If it passes them, run an A/B comparison in vLLM the way the demo did and let your own acceptance rate decide.

Frequently Asked Questions

Does speculative decoding change model output quality? No. The target model verifies every draft token in a single forward pass and recomputes any token it rejects, so the output distribution matches the target model. The optimization trades spare memory for latency, not accuracy for speed.

What acceptance rate should I look for in vLLM? A high acceptance rate means most draft tokens were kept and the speedup is real. In the 2026 Blackwell demo, structured output showed a high rate and a 1.6x speedup, while creative writing showed a low rate. Watch the metric on your own workload rather than relying on published numbers.

Which tools support speculative decoding? Kirui implemented her demo with vLLM, an open-source serving engine with documentation for getting started. More advanced draft architectures include EAGLE and Medusa, which change the drafting mechanism underneath rather than the verify-and-accept principle.

Why doesn't speculative decoding help long-context workloads? It accelerates the decode phase only. RAG and document-analysis jobs spend much of their time in prefill, building the KV cache from large inputs, so drafting extra output tokens saves little when generation is short relative to the input.

Turn a Talk Like This Into a Written Reference

This article exists because a 15-minute conference talk contained a complete, decision-ready explanation of a technical tradeoff: the two inference phases, the memory cost, the demo numbers, and the workload checklist. That kind of knowledge usually stays locked inside a video timeline.

If you have talks, interviews, or screen-recordings with explanations worth keeping, Skala Blog turns a YouTube URL into a transcription and then into a structured written article. Paste the link, and the reasoning inside the video becomes a page people can search, skim, and cite.

Source video