Stanford Neural Cache Model presents a new method for generating text 31 times faster.
Stanford researchers introduced Next-Concept Modeling, a text-generation method that first sketches an answer at coarse abstraction levels and then adds detail. Instead of generating each token sequentially, it creates one level per pass, so 1,024 tokens can be produced in 32 passes. The headline speedup is 31 times faster than diffusion language models, not production autoregressive chat models. Tests on an NVIDIA GPU showed useful output and much of diffusion-model quality at lower cost, but same-size autoregressive models still produced better text in perplexity and translation scores. Next-Concept Modeling is therefore a promising third approach for parallel, level-wise generation, especially when speed matters more than peak quality.
Key Takeaways
- NCM generates text by first producing coarse “concept” codes, then refining them level by level, reducing sequential passes from one per token to one per abstraction level.
- The “31x faster” claim compares NCM with diffusion language models, not with mainstream production LLMs like ChatGPT.
- Same-size autoregressive models still generate better text, and NCM has limitations such as no streaming output, fixed generation depth, and unclear scaling to larger models.
- NCM’s strongest advantages so far are training efficiency and performance on structured tasks where the whole answer depends on all parts.
- Laptop-GPU runs showed that NCM outputs only become readable text near the final refinement levels.
Stanford researchers have released Next-Concept Modeling (NCM), a text generation method that sketches an entire answer roughly before adding detail level by level, cutting the number of sequential passes from one per token to one per abstraction level. On OpenWebText the method produces 1,024 tokens in 32 passes instead of 1,024, and the paper’s headline “31x faster” figure refers to a comparison with diffusion language models, not with production chatbots like ChatGPT.
This article is for anyone weighing whether NCM is a practical alternative to autoregressive LLMs or diffusion models for text generation, and for readers who want to understand what the released checkpoints actually do on consumer hardware. It explains how NCM works, what the speed claim really measures, and what happened when the released code and checkpoints were run on an NVIDIA RTX 4060 Laptop GPU.
Why current text generation waits in line
Every LLM in daily use — ChatGPT, Gemini, Llama — is autoregressive. It predicts one token at a time, and each new token depends on everything written before it, so step 501 cannot begin until step 500 finishes. A GPU can do a lot of work in parallel, but autoregressive decoding keeps it waiting in a queue, which is a major reason long answers feel slow.
Diffusion language models attack that queue directly. Think of a fill-in-the-blanks exam paper: the model starts with the whole answer as blanks and fills many blanks at once in every round, correcting earlier mistakes. Google DeepMind’s experimental Gemini Diffusion samples at about 1,479 tokens per second. The cost is that diffusion needs many rounds, and every round looks at the whole answer again; cutting rounds to save time drops quality. A run of MDLM (170M parameters, trained on OpenWebText) with the same prompt and seed on a 1,024-token canvas showed the trade-off plainly: 48 rounds took 5.38 seconds and produced sentences, while 8 rounds took 1.38 seconds and produced word salad. NCM is presented as a third route that avoids both the token-by-token queue and the many-round quality cliff.
Two models, one compressor and one writer
NCM is really two models working together. The compressor takes a piece of text and squeezes it into a list of numbers called codes — a code is just a position in a list, like ordering by item number in a restaurant with 4,096 dishes on the menu. The codes come in levels: the first level has only a few codes and carries the rough gist, and every level after that has more codes and stores only what the earlier levels missed. By the last level there is enough detail to recover the exact words, and the compressor rebuilds the original text exactly 98.8 to 100% of the time depending on the task.
That is what “concept” means in the name — not a human idea or a sentence, but the information sitting in the top, coarse levels. The name sounds a little grander than the thing.
The writer is the generative half. Like an LLM that only outputs token IDs, it only outputs codes. It writes all codes of level 1 in one go, then all codes of level 2, and so on; inside a level nothing waits in line, and the only waiting is between levels because each level corrects the one above it. After the last level, the compressor’s decoder turns codes into words — the first moment any text exists. The number of passes therefore equals the number of levels, not the number of tokens, which is the whole source of the speed.
What the 31x figure actually compares
The “31x faster” number is real but widely misread. On XSum summarization NCM matches the strongest diffusion model’s ROUGE-L (28.0 versus 26.6) while being 31x faster — one experimental fast method beating another. There is no comparison with any production LLM.
A same-size LLM still writes better text. On OpenWebText the normal autoregressive model scores a generation perplexity of 35.45 against NCM’s 83.12, where lower is better; on translation the gap is 25.2 BLEU versus 22.6. The biggest NCM trained so far is 725M parameters, and nobody knows yet whether it holds up at the size of the models people use daily. There is also no streaming: chat apps feel fast because tokens appear as they are written, but NCM’s early levels are not readable words, so a user sees nothing until the last level finishes. Fixed depth is another limitation — even a short reply goes through every level.
Where NCM does look promising is training efficiency and structured puzzles. It needed about 4x fewer training tokens on OpenWebText (127B versus 500B+) to reach its results, and it does very well on puzzles where every part of the answer depends on every other part.
What the released checkpoints do on a laptop GPU
The runs below used Stanford’s released code and checkpoints on an NVIDIA RTX 4060 Laptop GPU, with the same decoding settings as the repository’s official evaluation. Loading the XSum checkpoint — trained on BBC news to write a one-line summary from an article — means loading both halves, the compressor and the writer. The checkpoint reports codes per level of [4, 6, 10, 14, 20, 26, 32, 40, 48], nine passes, a summary length of 48 codes, and an 88M-parameter writer. Nine levels means nine passes for the whole summary: level 1 has 4 codes, the last level has 48.
For a BBC article about a fasting-style diet that helped the pancreas heal itself in mice, forcing the decoder to turn every level into words showed the sharpening process. Level 1 (4 codes) read “decades - eurozone” — nothing to do with the article, which is exactly why streaming does not work. By level 6 (26 codes) the topic was there: “Scientists discovered diet that causes pancreas possibly root causes 2.” The final level 9 (48 codes) read: “Scientists have discovered a vegan diet that causes in the pancreas - possibly cause the causes of type 2 diabetes.” The gist is in the neighbourhood and the details are off: the article describes a fasting-mimicking diet that resembles a vegan diet and helped regenerate pancreas cells, while NCM says a vegan diet that causes type 2 diabetes. The whole thing took 9 passes and 0.30 seconds in that notebook run, and between 0.18 and 0.39 seconds across other runs on the same laptop. No LLM was run on the same article, so no speed ratio is attached to that number.
Ten articles against BBC’s own summaries
One article is an anecdote, so the first ten articles of the XSum test set were compared against the summaries BBC actually wrote. Counting all ten by hand: 8 of 10 got the topic right — the right people, place and event — and 2 of 10 were garbled. Article 5, about the Essilor and Luxottica merger, came out as invented brand names; article 7 became a coffee shop. Even the good ones bend facts: NCM said rider Wendy Houvenaghel had been sacked by British Cycling, while the BBC summary quoted her saying a “medal at any cost” approach created a “culture of fear”; NCM had Celtic facing Queens Park Rangers, while the BBC summary had them meeting Rangers in the League Cup semi-final. Time per summary was 0.32 to 0.36 seconds across 9 passes each. For an 88M research model that is decent, and it is nowhere near what an LLM gives you — the paper does not claim otherwise.
Outside its training world, and where it shines
Two inputs from outside the training distribution exposed the limits. A short article about Voyager 1 (574 tokens, of which the model reads 512) produced “The latest launch of space Nasa agency - Nasa's astronaut spacecraft mission - has launched its launch launch from the Nasa space astronauts astronaut - and an unmanned space mission.” The NCM paper itself (30,810 tokens, of which the model reads 512) produced “acy-acy--Quible and analyticalocrographed, test, unique scenery, digital technology and tablets.” Two lessons follow: the model is a specialist, since a space article becomes “launch launch… astronaut astronaut” and a research paper becomes nonsense; and it only reads the first 512 tokens, so for the 30,810-token paper it saw the title page and part of the abstract — easy to miss if you only look at the output.
The picture changes on Sudoku, where every part of the answer depends on every other part and the level-by-level correction scheme has the most to work with. That is the setting the paper highlights, and it is the strongest argument for watching where NCM goes next rather than treating it as a drop-in replacement for the chatbot you use today.
What to watch
The open questions are whether the approach holds at parameter counts far above 725M, whether the missing streaming can be worked around for interactive use, and whether the training-token efficiency (127B versus 500B+ on OpenWebText) survives at production scale. For now, NCM is a research result with released checkpoints that run comfortably on a laptop GPU, a genuine speed advantage over diffusion models of similar quality, and a clear quality gap against same-size autoregressive models. Readers evaluating it should treat the 31x headline as a diffusion-versus-diffusion comparison and test the checkpoints on their own domain before drawing conclusions about production use.

Comments (0)