NCM vs LLMs: Independent Speed Benchmark Results Explained
Independent DGX Spark tests show Stanford's NCM is 4x faster than matched LLMs solo, but slower once requests get batched.
Is NCM actually faster than a normal LLM?
It depends entirely on how you’re using it. Independent testing on a DGX Spark, reproducing and extending Stanford’s Next Concept Modeling (NCM) paper, found NCM runs about four times faster than a matched autoregressive model on single requests, and roughly twice as fast on long-document summarization. But flip the setup to a batch of 64 simultaneous requests, the kind a real server handles, and a standard autoregressive model wins instead. The architecture that looked like a clean win for single-user speed turns out to have a server-side weakness the original paper never tested.
TL;DR
- NCM writes answers in parallel passes rather than one word at a time, sketching a rough outline of the full response first and refining it over roughly nine to eleven passes depending on the task.
- Independent single-request tests confirmed the paper’s core speed claim, with NCM running about four times faster than a parameter-matched autoregressive model on translation and about two times faster on summarization.
- Batched load flips the result entirely, with a normal autoregressive model beating NCM once 64 requests are processed together, because NCM does more total compute per answer even though it needs fewer passes.
- The original paper only benchmarked NCM against diffusion models, never against standard autoregressive LLMs, which left the most practically relevant comparison untested until this reproduction.
- The headline Sudoku result (86% vs 1%) appears to be driven largely by decoding order, not the NCM architecture itself. A custom-trained diffusion model of matching size reached comparable accuracy once it filled in its most confident cells first instead of random ones.
- NCM has real limitations, including fixed-size output regardless of answer length, occasional errors swapping names and numbers, no public training code, and no testing yet past sub-1-billion-parameter models.
- Streaming isn’t possible with NCM in its current form, since the whole answer stays unreadable until the final refinement pass completes.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
How does NCM generate text differently from a normal LLM?
Standard large language models, including the ones behind most chatbots, are autoregressive: they generate one token, append it to the sequence, then generate the next token based on everything before it. A thousand-token answer takes a thousand sequential steps, and each step requires pulling the model’s full set of weights through GPU memory. That memory movement, not raw computation, is usually what limits local LLM speed.
NCM takes a different approach borrowed from image generation, where some models first produce a blurry low-resolution sketch and progressively add detail. NCM applies the same idea to text. It uses a compressor that encodes a passage into a stack of levels, each made of codes drawn from a fixed vocabulary of around 4,000 learned codes, which the paper calls “concepts.” The top level holds only a handful of codes representing a coarse outline of the whole passage. Each level below adds more codes and fills in finer detail, until the bottom level has one code per word. A standard transformer generator then predicts an entire level of codes in a single pass and moves down to the next level, rather than predicting one word and moving to the next position.
This means a summary that needs nine levels takes nine total passes, regardless of how long that summary is. In testing, a German-to-English translation took 11 passes to go from a mostly incoherent first draft to a clean final sentence, matching the model’s 11-level architecture.
How is this different from a diffusion model?
Diffusion language models, like Inception’s Mercury or Google’s Diffusion Gemma, also generate text in parallel rather than left to right. They start with a blank page, fill in a rough guess everywhere at once, then repeatedly refine the entire page. The catch is that every pass has to touch every token position, and getting good output quality typically takes 64 or more passes.
NCM avoids that cost by making its early passes genuinely small. Since the top level only contains a few codes, the first passes are cheap, and each subsequent pass only adds the detail assigned to that level. The original paper reported NCM matching the quality of the strongest diffusion model on news summarization while running 31 times faster, but that comparison was exclusively against diffusion models. It never measured NCM against a standard autoregressive model of equivalent size, which left an important gap.
What did the independent benchmark find on speed?
Filling that gap required building a matched autoregressive model of the same size and shape as the released NCM model, then testing both on identical prompts with identical output-length targets.
Sent one request at a time, NCM beat the autoregressive model by roughly 4x on translation and roughly 2x on summarizing long articles. That part lines up with the architecture’s promise: fewer total passes means less time waiting on memory bandwidth, which is the dominant cost when a GPU is serving a single user.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Sending 64 requests simultaneously reversed the outcome. In batched conditions, the GPU only needs to load the model’s weights once for the whole batch, so the bottleneck shifts from memory loading to raw computation. Here NCM loses its edge, because across its stacked levels it processes roughly four times more token positions than the words it ultimately produces. The released code also reprocesses the entire prompt on every single pass, which made the long-article summarization case especially slow in batch. For anyone planning to run multiple agents or handle concurrent users, this result matters more than the single-request numbers that dominate the paper.
Does NCM actually solve Sudoku better than LLMs, and why?
The paper’s most viral claim was that a small NCM model solved 86% of hard Sudoku puzzles that a same-size autoregressive model solved only 1% of the time, since autoregressive models can’t revise an early wrong digit.
Reproducing the setup on the same hardware confirmed the 86% figure holds up. Printing the board after every pass showed NCM doesn’t fill in cells sequentially. It redraws the entire board on every pass, even getting the given clues wrong in early passes before the whole grid snaps into place at the end. That also means NCM output can’t be streamed the way autoregressive text can, since nothing is readable until the final pass.
To test whether this result required the NCM architecture specifically, a diffusion model of matching size was trained from scratch on the same puzzles and evaluated on the same hard boards. Run the standard way, filling cells in random order, it solved about a quarter of the boards, consistent with the paper’s own diffusion baseline. But changing the decode order, filling in the cell the model was most confident about first instead of a random one, brought the diffusion model’s accuracy close to NCM’s. Allowing more refinement passes pushed it up to 95%.
That suggests the large Sudoku gap reported in the paper has more to do with decoding strategy than with NCM’s architecture being uniquely suited to constraint-solving. NCM still gets coarse-to-fine refinement built in by default, while the diffusion model needed an extra decoding trick to match it, so the comparison isn’t a wash. But it does mean the architecture alone doesn’t explain the original headline number.
Is NCM worth using right now?
For single-user, local inference, NCM offers a genuine speed advantage, up to roughly 4x in these tests, which matters for anyone running models on personal hardware and waiting for one answer at a time. For anything resembling production serving, batched workloads, or multi-agent pipelines, a standard autoregressive model currently performs better.
Other practical limitations stand out. NCM always generates into a fixed-size output structure, so even a short, one-line answer runs through every refinement level and pays the full computational cost. The compressor occasionally mishandles names and numbers when encoding and decoding text. The largest model the original authors trained is under 1 billion parameters, so it’s unproven at the scales people actually run at home, let alone at tens or hundreds of billions of parameters. The released code is evaluation-only, with no training code published, and the built-in prompt caching feature did not work correctly in testing.
Frequently Asked Questions
What does NCM stand for and who created it?
NCM stands for Next Concept Modeling, a text generation architecture from Stanford researchers that predicts hierarchical “concept” codes level by level instead of predicting words one at a time.
Is NCM a type of diffusion model?
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
No. NCM’s early passes are cheap because they only involve a small number of top-level codes, whereas diffusion models must process every token position on every single pass regardless of level.
Why is NCM slower on busy servers despite being faster for one user?
Single requests are bottlenecked by GPU memory bandwidth, where fewer passes help NCM. Batched requests are bottlenecked by raw computation, and NCM processes roughly four times more total positions than an equivalent autoregressive model, which erases its advantage.
Does the Sudoku benchmark prove NCM reasons better than LLMs?
Not clearly. A matched diffusion model reached similar accuracy once it was changed to fill in its most confident cells first rather than in random order, suggesting decoding strategy explains much of the gap rather than the NCM architecture itself.
Can NCM stream answers like a chatbot does?
No. Because NCM redraws the entire answer on every pass and only converges at the final pass, there’s no partial, readable output to stream before generation finishes.