Diffusion took over image generation years ago, and now it’s arriving in text. Instead of generating one token per forward pass from left to right, a block-diffusion model denoises a whole block of tokens at once.
Until recently, comparing text diffusion against autoregressive decoding meant comparing different models with different sizes, pretraining mixes, or backbones. Google’s DiffusionGemma (an experimental research release) and Gemma 4 (the production model) make a direct comparison possible. Both share the same 26B-parameter Mixture-of-Experts backbone with 4B active parameters from the Gemma 4 family, trained either for block diffusion or for next-token prediction.
In my previous post, I used DiffusionGemma as a single-pass classifier. This time I wanted to know what happens when you ask it to write. I went in with one intuition: a model that sees its whole response at once should handle global constraints better than a model forced to improvise from left to right.
Prefer video? Here’s the 5-minute summary:
https://medium.com/media/4dfeba3769e59e2c443c2ba9b25ba923/href
How block diffusion works
Unlike masked diffusion models that start from mask tokens and permanently freeze each word as it is unmasked, DiffusionGemma uses uniform state diffusion and can revise any slot on a later pass. It reads the prompt into the KV-cache once, initializes a 256-token output canvas with random tokens from the vocabulary, and refines that entire block in parallel over several passes before caching it and moving on to a fresh canvas for the next block. So responses don’t stream token by token; each 256-token block is emitted all at once.
Inside that active block, attention is bidirectional: every token being drafted can see the tokens before and after it. Three details of the DiffusionGemma generation loop explain most of the results.
Each pass, the model predicts every slot at once, keeps the ones it’s most confident about, and re-randomizes the rest. Slots picked in the same pass are chosen independently, so two neighbors can’t see each other’s choice until the next pass.
The canvas is always 256 slots. An autoregressive (AR) model stops the moment it emits a newline or end-of-turn token, but DiffusionGemma refines the whole block and fills everything after the answer with padding.
Finally, the loop stops once predictions settle and the average entropy across all 256 slots drops below a small threshold. That average includes the ~200 leftover slots after a 50-token answer ends. Those slots are near-certain, so they dilute the average and pull it under the threshold in 5 to 8 passes, even if a few content tokens are still settling on a global rule.
Results at a glance
I ran 200 paired prompts per task (400 for MMLU-Pro) across 10 tasks at batch size 1 on Google Cloud Run Jobs with NVIDIA RTX PRO 6000 Blackwell GPUs.
†Measured on the Structured JSON extraction outputs (how often span_count matched the model's own redactions).
‡Pass@1 after normalizing indentation and trailing lines (raw pass@1 before normalization was 0.475 vs. 0.335). See the Methodology appendix for decoding settings, confidence intervals, and timings.
When the input pins down the answer
In my runs, Gemma 4 AR tended to over-correct when fixing grammar in a draft, rewriting working clauses along the way. In copy-editing (CoEdIT), the input sentence is already mostly right, and you just want broken phrases patched in place without changing the author’s voice.
While both models read the full source sentence in the prompt, block diffusion lets the drafted output tokens attend to each other in both directions across the sentence. That may be why DiffusionGemma scored higher on SARI (0.651 vs. 0.629), which rewards keeping working words and only changing what’s broken, while running 2.7x faster.
- Source: “I also believe it would hard to get students to come to schools or stay out schools with session”
- DiffusionGemma: “I also believe it would be hard to get students to come to school or stay after school for sessions.”
- Gemma 4 AR: “I also believe it would be difficult to get students to attend school or stay after school for sessions.”
DiffusionGemma repairs the broken tokens and leaves the rest alone, matching the reference edit exactly. Gemma 4 AR also fixes the grammar, but rewrites “hard” to “difficult” and “come to” to “attend” along the way.
Tag-preserving translation from English to Spanish (FLORES+, with inline {user_name}, %s, and <b> markup tags) tied on quality (0.930 vs. 0.940) at twice the speed: the source sentence pins down the tag sequence, and both models copy the tags cleanly while translating the words around them.
On plain-text PII redaction (Nemotron-PII), both models preserved over 98% of the non-PII words, while Gemma 4 AR caught slightly more sensitive spans (0.865 vs. 0.891 recall).
Answer-first versus reasoning-first
In left-to-right generation, a standard rule of prompt and schema design is to use reasoning-first ordering: chain-of-thought steps before the final choice, or an array of extracted items before the summary count. If an AR model has to emit a total on line 3 before writing the body on line 5, it has to guess.
To test whether bidirectional denoising changes that rule, I asked both models to redact PII from the same 200 Nemotron-PII documents into a JSON object where span_count comes before redacted_text. On one 5-span document, Gemma 4 AR committed to the wrong count:
{
"has_pii": true,
"span_count": 4,
"categories": ["coordinate", "date", "date_time", "first_name", "url"],
"redacted_text": "On the date of [REDACTED], [REDACTED] captured a series of photographs..."
}
The full redacted_text contains five [REDACTED] tags; the DiffusionGemma output was identical except "span_count": 5. Because Gemma 4 AR commits to span_count before writing a single redaction, its count matched its own [REDACTED] tags 38% of the time, versus 51% for DiffusionGemma (the DiffusionGemma Technical Report shows the same pattern when an arithmetic prompt asks for the final number before the steps). Still, even 51% is too low to trust in production, so compute summary counts in application code rather than asking either model to count.
When the reasoning comes first, Gemma 4 AR pulled ahead. On 5-shot MMLU-Pro, where the model writes out step-by-step reasoning before picking an option letter, Gemma 4 AR led by 4 points (0.752 vs. 0.792), matching the technical report’s benchmark gap. In chain-of-thought reasoning, each step depends on the exact wording of the previous deduction, which plausibly suits left-to-right decoding.
Where parallel drafting trips up
IFEval looked like an easy win for diffusion. Its prompts stack rules that apply to the whole response, like “write three paragraphs, use no commas, end with a postscript,” and a model that drafts the whole block at once seemed built for that. Instead, the two models tied (0.860 vs. 0.895). On structural rules and keyword inclusion or exclusion they scored almost the same, while DiffusionGemma fell behind on counting rules that require exact token-level coordination, like hitting a 300-word target or using a keyword an exact number of times.
The sentence-ending probe isolates that coordination problem. Each prompt gives four words, and the model has to write four sentences of 6 to 12 words, each ending on one of them. On paper, this is diffusion’s best case. Instead, DiffusionGemma missed the target ending on 25 of the 800 generated sentences versus 7 for Gemma 4 AR. Given “oxidation”, “treat”, “juice”, and “prevent”, Gemma 4 AR landed all four, while DiffusionGemma wrote:
The iron metal underwent a process of rapid oxidation.
You must use a special chemical to treat wood.
She poured a cold glass of fresh orange juice.
Adding lemon to the fruit will help prevent browning.
In lines 2 and 4, the target word landed second-to-last inside a natural sentence. Every token looked locally fine, and the padding slots after the answer were near-certain, so the canvas met the stopping threshold before anything moved the target word to the end.
On other prompts in the same probe, neighboring slots sampled the same word in a single pass and produced adjacent duplicates like “The children children played happily in the back yard.” and “She reached out her hand to to catch.” (both still ended on their target words and passed the grader).
A similar miss showed up on concept coverage (CommonGen), where DiffusionGemma occasionally settled on a fluent sentence that left out one of the required words (given “serve”, “ball”, and “tennis”, it wrote “She delivered a powerful serve to win the tennis match”).
On structured JSON extraction, Gemma 4 AR produced valid JSON on all 200 documents. DiffusionGemma failed json.loads() on 14, mostly from unescaped newlines, quotes, or backslashes copied from the source, plus a few that kept writing after the closing } to fill the canvas. On the documents both models parsed, extraction quality was the same.
In production, using DiffusionGemma for structured output requires either a lightweight JSON repair step or schema-constrained decoding adapted to parallel denoising, since tools like Outlines mask each next token based on the prefix written so far, and a diffusion canvas has no fixed prefix.
Code autocomplete: right answer, wrong length
At first glance, filling in a missing line of Python (HumanEval-Infilling) looked like DiffusionGemma’s biggest win, beating Gemma 4 AR by 14 points on raw pass@1 (0.475 vs. 0.335). When I inspected the completions, the gap came from whitespace and stopping behavior rather than coding ability.
Take this function, where the model has to fill in the missing if n == 2: line:
def fibfib(n: int):
if n == 0:
return 0
if n == 1:
return 0
# <--- fill in this line (needs 4 spaces: " if n == 2:")
return 1
return fibfib(n - 1) + fibfib(n - 2) + fibfib(n - 3)
Both models wrote the right if statement, but splicing their raw responses directly into the file triggered an IndentationError for opposite reasons:
# Gemma 4 AR output (drops the 4-space indent):
if n == 2:
# DiffusionGemma output (keeps the 4-space indent, but repeats the rest of the function):
if n == 2:
return 1
return fibfib(n - 1) + fibfib(n - 2) + fibfib(n - 3)
Neither checkpoint was trained with fill-in-the-middle tokens, so both were prompted in chat format. Gemma 4 AR usually starts its reply at column 0, dropping the indentation the surrounding block requires. DiffusionGemma denoises against the surrounding code at once and preserves exact indentation almost every time, but because it fills a 256-token canvas instead of stopping at the first newline, it often continues by copying lines from the suffix. Above, its repeated return fibfib(…) closes the if block early, breaking the original return 1 below it.
Normalize both formatting habits and coding accuracy is a borderline tie (0.710 vs. 0.750), with AR nominally ahead. For single-line IDE autocomplete, Gemma 4 AR plus a syntax-aware re-indenter would be my pick: it was at least as accurate and 1.6x faster, because truncating DiffusionGemma’s output after the fact doesn’t save the compute spent denoising a 256-token canvas for a one-line completion.
Math: speed vs. verbosity
On competition math (MATH-500), the two models tied overall (0.930 vs. 0.915) while DiffusionGemma finished in 2.8 seconds on average versus 22.5 seconds for Gemma 4 AR.
Under a 1,536-token cap, the benchmark partly measured verbosity rather than math ability. DiffusionGemma wrote 18% shorter derivations, a behavior the technical report calls “emergent conciseness” from diffusion reinforcement learning, so AR’s longer derivations hit the cap far more often on the hardest problems. On problems where neither model ran out of tokens, AR led slightly; on the problems where only AR truncated, DiffusionGemma solved two-thirds. Raising the token cap would likely restore AR’s edge on hard math, at the cost of making its 22-second latency even longer.
How hardware and output length changed the speed
On my M4 Pro MacBook, DiffusionGemma ran 2.3x slower than Gemma 4 AR in a short 64-token test. On a Cloud Run Blackwell GPU, it was 1.5x to 8.1x faster on tasks whose outputs ran past about 20 tokens.
At batch size 1, an AR step spends most of its time loading model weights from memory just to compute a single token. A diffusion pass activates more experts per layer, but it computes 256 tokens for every weight read. On a datacenter Blackwell GPU, that turns memory-bound reads into dense Tensor Core work: denoising a full 256-token pass costs only about 3.6x a single AR token. Since an entire 256-slot canvas converges in 5 to 8 passes on structured tasks, break-even landed around 20 output tokens in my runs. Short outputs like concept coverage don’t produce enough tokens to pay for those passes, while long derivations like MATH-500 spread each pass across hundreds of tokens. Code autocomplete is the exception: it’s slower because DiffusionGemma echoes the suffix across its canvas.
My best explanation is that a laptop unified-memory GPU has a much lower compute-to-memory-bandwidth ratio than a server GPU, so the same 256-token pass hits the compute ceiling on Apple Silicon instead of running alongside the memory reads. Batching many concurrent AR requests together on a server GPU raises AR’s own efficiency: in the technical report’s H100 benchmarks, Gemma 4 AR overtakes DiffusionGemma in total throughput above about 32 concurrent requests.
What I’d reach for now
Gemma 4 is still the production default. DiffusionGemma is an experimental model worth trying in a few specific spots:
- Classification or routing: Don’t generate at all. Score the allowed label tokens in a single forward pass, as in my previous post.
- Low-concurrency GPU workloads (in-place text editing, or tasks where your eval shows a quality tie): DiffusionGemma cut single-stream GPU latency 2x to 8x in my runs.
- Everything else (laptops, high concurrency, outputs under ~20 tokens, strict JSON, step-by-step reasoning, or single-line code autocomplete): Gemma 4 AR.
I went in treating “sees the whole block at once” as an advantage on any task with global constraints. The picture turned out narrower: seeing the whole answer at once isn’t the same as planning it. In my tests, it helped when the input already pinned down the answer, and struggled when parallel tokens had to coordinate exact counts, endings, or escaping.
If you run a similar comparison on your own tasks, I’d love to compare notes. Find me on X, LinkedIn, or Bluesky.
Appendix: Methodology
Setup
These are personal results from my own evaluation harness, run in October 2026. For Google’s official full-dataset FP8 numbers, see the DiffusionGemma Technical Report.
The GPU runs used google/diffusiongemma-26B-A4B-it and google/gemma-4-26B-A4B-it in native bfloat16 on Cloud Run Jobs, one RTX PRO 6000 Blackwell (96 GB) per job, with Hugging Face transformers 5.18.0, PyTorch 2.14.1, and CUDA 13.0. Every generation ran at batch size 1 with thinking disabled. This reference transformers stack is slower in absolute terms than the optimized FP8 serving in the technical report, but the cost of one diffusion pass relative to one AR token is close (3.6x here vs. 3.2x in the report).
The laptop runs used 8-bit MLX ports of both models under mlx-vlm 0.6.7 on an Apple M4 Pro with 48 GB. The 2.3x laptop slowdown comes from a 40-item IFEval calibration sweep at 64 tokens (21.4 vs. 48.6 tok/s).
Decoding
The two models were not decoded the same way. Gemma 4 AR used greedy decoding (temperature=0.0), because a single run at its default temperature of 1.0 adds sampling noise unless you average several seeds. DiffusionGemma used its official entropy-bound sampler (temperature annealed from 0.8 to 0.4, entropy_bound=0.1, at most 48 denoising steps). Greedy decoding probably gives AR a small edge on exact-match and syntax-sensitive tasks.
Sampling and token budgets
I shuffled each dataset split and took the first 400 items for MMLU-Pro and the first 200 for the other nine tasks. Dataset shuffling and DiffusionGemma’s sampler both used a fixed seed, and the sampler was reseeded before every item so each result is reproducible on its own. The harness reproduced the technical report’s AR lead on 5-shot MMLU-Pro. I fixed the token budgets from a 10-item pilot before the full run; MATH-500 got 1,536 tokens after a Level 5 solution in the pilot ran to 1,232.
Timing
Latency is calculated with the full generate() call, including tokenization, prefill, and detokenization, and speedup is mean AR seconds divided by mean DiffusionGemma seconds. Token counts exclude the <pad> slots DiffusionGemma uses to fill out each 256-token canvas; counting them would inflate its tok/s by up to 12x. DiffusionGemma always denoises a full 256-token canvas, so CommonGen's 128-token budget doesn't shorten its work.
Checking the graders
Before the full run, I also tested the graders themselves. HumanEval-Infilling’s canonical solutions scored 0.0 under my first grader because str.strip() ate the first line's indentation, which would have made both models look unable to infill. Feeding stub outputs to the other graders showed that CommonGen and placeholder scoring gave full marks to a copy of the prompt. The fixed graders strip prompt echoes, and the placeholder grader also requires the text around the tags to be translated.
A decoding gotcha in transformers
If you run DiffusionGemma in transformers, watch out for how tokenizer.decode() handles thinking tokens. Even with thinking disabled, DiffusionGemma outputs an empty <|channel>thought\n<channel|> header before its response (Gemma 4 AR prefills that header in the prompt instead). Decoding with skip_special_tokens=True strips the channel tags but leaves thought\n at the start of every string, which breaks json.loads(), adds a stray line to code completions, and penalizes exact-match edits. Slice past <channel|> before decoding so skip_special_tokens=True can still strip <turn|> and trailing <pad> slots without eating leading indentation:
ids = output_ids[prompt_len:].tolist()
end = tokenizer.convert_tokens_to_ids("<channel|>")
if end in ids:
ids = ids[ids.index(end) + 1 :]
text = tokenizer.decode(ids, skip_special_tokens=True)
Per-task details and statistics
I counted a task as a quality win only when the paired 95% bootstrap confidence interval (10,000 resamples of item-level score differences) excluded zero. Across 10 tasks at the 95% level, one false positive is plausible by chance, and the two narrowest AR wins, concept coverage and sentence-ending lookahead, have upper bounds of −0.005 and −0.006.
On Structured JSON extraction, DiffusionGemma’s 14 json.loads() failures broke down into 4 items with unescaped literal newlines (recoverable with json.loads(…, strict=False)), 7 items with bare backslashes or unescaped double quotes inside markdown strings, and 3 items with trailing text after the closing }. On the 186 documents both models parsed, composite extraction quality tied (0.944 vs. 0.939).
Dataset attributions: Example items adapted from CoEdIT (Grammarly, Apache 2.0), Nemotron-PII (NVIDIA, CC BY 4.0), and HumanEval-Infilling (OpenAI, MIT).
When to use a diffusion LLM: I compared DiffusionGemma and Gemma 4 was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/when-to-use-a-diffusion-llm-i-compared-diffusiongemma-and-gemma-4-0e5704b67458?source=rss—-e52cf94d98af—4
About the Author
