Recurrent Depth in LLMs Comes from the Arithmetic Behind Looped Transformers.
Dory and GLiDE show looped LLMs add depth by reusing layers, but gains cost compute and latency; cache choices decide real serving cost.
Key Points
- Looped transformers can execute more layers than they store by reusing core weights, so recurrent depth is a compute and latency trade-off rather than a free accuracy boost.
- Dory’s example shows 36 stored layers can execute 96 layer passes with four core loops, meaning weight memory follows stored layers while per-token compute follows executed layers.
- GLiDE’s measured loop benefit was modest: a 0.56-point held-out score gain came with about 35% higher median latency, showing the improvement must justify the serving cost.
- Cache behavior and token savings are implementation choices: looped models can keep cache at non-looped size, but fewer generated tokens do not necessarily mean lower total cost.
- More loops add computation, not information; they may help multi-step reasoning when facts are available, but missing facts, exact arithmetic, or slow responses may require retrieval, tools, or stopping earlier.
Published configurations for NVIDIA's Dory and a 10 November 2026 report from Fastino on its GLiDE model make a practical point about looped transformers: they can execute more layers than they store, but the extra depth is paid for in compute, latency, and serving behavior. The headline arithmetic is that Dory can store 36 layers while executing 96 layer passes per token when the core runs four times. That is not a typo. It is weight sharing, and it should be read as a cost model, not a free accuracy upgrade.
The Arithmetic Behind 36 Stored Layers and 96 Executed Layers
Dory's published layout uses 8 prelude layers, a 20-layer core, and 8 coda layers. A simplified recurrence is:
h0 = prelude(x)
h_{r+1} = core(h_r, x; theta) for r = 0 to R-1
logits = coda(h_R)
This recurrence is a simplification, not a spec for every model. The core weights are reused on every pass, and Dory reinjects the input x on each pass so the loop does not drift from the question.
Because the prelude and coda are stored once, the model stores 36 layers total. With four core passes, executed layer passes are 8 + 20 times 4 + 8 = 96. The layer counts are:
- R=1: stored=36, executed=36, ratio=1.00
- R=2: stored=36, executed=56, ratio=1.56
- R=3: stored=36, executed=76, ratio=2.11
- R=4: stored=36, executed=96, ratio=2.67
Weight memory follows the stored column. Compute per token follows the executed column to a first approximation; attention and key-value cache handling change exact computation, so these are layer numbers, not a latency model.
A toy example makes the trade clear: 8 blocks run 3 times execute 24 blocks while storing only 8. Weight sharing is appealing, but it does not promise the accuracy of a real 24-block network, because each reused block has to work on a different state every pass, and learning that is the hard part.
Fastino GLiDE Put a Measured Price on the Gain
Fastino's report on its 27B decision model GLiDE gives the clearest public price tag. Compared with one pass, looping two middle bands twice each raised the held-out score by 0.56 points, with a 95% confidence interval of 0.27 to 0.89. The interval excludes zero, so the measured gain is statistically positive in that evaluation.
On one RTX PRO 6000, median latency rose from 83 ms for one pass to 112 ms for the looped setting, about 35% slower. Whether that trade is worth it depends on what a decision is worth and what response time the application can afford.
The report also mentions about 110 candidate configurations tried, several failed designs, no clear benefit below 1B parameters, and a third pass that hurt a causal-reasoning benchmark.
Cache Choices, Token Savings, and Total Cost
Naively, each loop could keep its own attention cache, so cache memory would grow with R. Dory's paper reports a mode that keeps only the final loop's key-value cache. In that mode, memory stays at the non-looped footprint, and most of the accuracy gain is retained. The lesson is general: for looped models, cache growth is an implementation choice. Check the serving mode before sizing hardware.
Token savings can also look better than they are. Dory's authors report that 4 loops produce 1.3 to 2.5 times fewer generated tokens than 1 loop on agentic benchmarks. But per-token cost rises with executed layers, from 36 to 96 in the arithmetic above. Which effect wins is an empirical question per workload. The comparison that settles it is total cost per correct, checked task, including retries.
When More Thinking Stops Helping
Repeating a learned function has no built-in guarantee of improvement. Three trajectories are all possible: quality keeps rising with diminishing returns, quality plateaus, or quality peaks and then falls. Compute rises in all three cases.
A fixed loop count is predictable and easy to compare, so start there. An adaptive halting rule has to track real improvement, because confidence is not correctness. Check whether early exit actually saves wall-clock time in your serving stack; an exit rule on paper and a measured latency drop are different evidence.
A model trained at one loop count and served at another may drift, because its internal states can move outside the training distribution.
What Looping Cannot Fix
Loops add computation, not information. If the needed fact is not in the prompt or the weights, more passes over the same state cannot produce it. Retrieval or a tool call can. Diagnose the failure before picking the fix:
- Has the facts but fails multi-step reasoning: use more loops or a larger reasoning budget.
- Fact missing: use retrieval or a tool.
- Exact arithmetic or strings wrong: use a calculator or code tool.
- Right but too slow: use fewer loops.
- Worse the longer it thinks: stop earlier.
For readers deciding whether looped transformers deserve a place in production, track two depths: stored layers for memory and executed layers for compute. Cache growth is a design choice; Dory shows it can stay at one-loop size. Every pass has a latency price; GLiDE measured about 35% more median latency for a 0.56 point held-out gain. Fewer tokens do not imply lower cost. Choose loop count like any hyperparameter: per task type, against honest alternatives, and confirmed on fresh data.
The source article says all numbers come from public papers and model cards, that it was drafted with AI assistance, that every number was checked against sources linked at the end, and that the author did not benchmark the models. It expands arithmetic from the guide Looped Transformers Explained, originally published on piszczek.pl. The original adds the history from ACT and Universal Transformers to Huginn, a comparison with chain of thought and agent loops, and seven interactive labs where readers can change the loop count and watch these numbers move. Recurrent Depth in LLMs: The Arithmetic Behind Looped Transformers was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Meta description: NVIDIA Dory and Fastino GLiDE show looped transformers can execute more layers than they store, trading latency for accuracy and token savings.
Tags: looped transformers, Dory, GLiDE, transformer depth, AI inference
Featured image: abstract friendly illustration of a looped neural network passing tokens through shared layers, no real people, no logos, no text

Comments (0)