Why Setting Temperature to 0 Still Doesn’t Guarantee Determinism
Temperature 0 LLM outputs can drift from near-ties, floating-point changes, batching, and hardware, making exact repetition unreliable.
Key Takeaways
- Temperature 0 reduces sampling randomness but does not guarantee deterministic outputs, because tiny numerical differences can flip the top-token choice.
- Near-ties between token scores can compound over long completions, making identical outputs increasingly unlikely after about 100 tokens.
- Inference stack factors such as batch size, hardware, kernels, cache state, and quantization can slightly change logits and break reproducibility.
- Builders who need repeatability should cache outputs, pin model and runtime configurations, monitor score margins, and save prompts and outputs for audits.
Meta description: A Towards Data Science post uses a one-line formula to show why temperature 0 LLM outputs can drift after about 100 tokens, and what builders should watch.
Tags: LLM inference, temperature 0, deterministic sampling, AI reliability
Featured image: An abstract friendly illustration of branching token paths fading apart, no people, no logos, no text.
For developers, AI engineers, and analysts who assume temperature 0 makes completions repeatable, a post that first appeared on Towards Data Science—no date is given—argues that temperature 0 isn't deterministic. It highlights a one-line formula for how often an LLM's top token flips and why identical completions fall apart after about a hundred tokens. The exact formula is not shown in the source material, so the arithmetic below is a general way to reason about the claim. The core idea: near-ties can turn tiny runtime differences into different long outputs.
The core claim: small flips become large divergences
LLMs predict one token at a time. At each step, they assign scores, often called logits, to possible next tokens. Temperature 0 is commonly used to mean "choose the highest-scoring token." If that token is far ahead, output should be stable.
The warning targets cases where the top choice is not far ahead. If the first and second candidates are separated by a tiny amount, a small numerical change can make the second appear first. That is a top-token flip. Once one token changes, the context changes, and the model may follow a different path.
A simple line of arithmetic shows why this compounds. If the flip chance at any step is p, the chance of staying identical across n tokens is roughly (1 - p)^n. As n grows, even a small p can make divergence likely.
That is the logic behind the "about a hundred tokens" point. A 100-token completion gives many chances for a near-tie. If p is 1 percent per token, the chance of staying identical over 100 tokens is about (0.99)^100, or roughly 37 percent. The 1 percent figure is illustrative, not a universal model statistic. Long outputs amplify tiny uncertainties.
Why temperature 0 still leaves room for change
The issue is not that temperature 0 chooses randomly. It is that the underlying computation is not always identical. Several ordinary engineering factors can nudge logits by amounts too small for humans to notice.
Batch size is one common source. A request may run alone or grouped with other requests. Different batch shapes can change the order of floating-point additions. Floating-point arithmetic is not perfectly associative, so equivalent calculations can produce slightly different results.
Hardware and kernel behavior can matter too. GPU libraries, model parallelism, cache state, and concurrent workloads can affect numerical details. Quantization, storing weights in lower precision, can also make scores closer together. Deterministic output is therefore a property of the whole inference stack, not just the sampling parameter.
What builders should do when reproducibility matters
For chatbots, data pipelines, coding assistants, and evaluation harnesses, the question is not only whether temperature 0 produces a good answer, but whether the answer can be trusted to repeat.
Short tasks may show few flips. Longer tasks make them harder to ignore. A 100-token response is already long enough for one near-tie to send the completion down a different route. In agentic systems, where a token can trigger a different tool call, the effect can be larger than a wording change.
A reasonable response is to treat temperature 0 as a stability setting, not a guarantee. If exact repetition matters, store the completed output instead of relying on regeneration. If you must regenerate, pin the model version, runtime, hardware type, and batch configuration as closely as possible.
Monitoring helps. Many inference stacks can log the top candidate's score and the margin between the first and second candidates. If margins are often tiny, the system may be more flip-sensitive than expected. For audits, save the full prompt, model identifier, and output, because the same prompt may not reproduce byte-for-byte across environments.
What this changes for readers
The post does not suggest that temperature 0 is useless. It reduces sampling randomness and keeps outputs focused. The narrower point is that it does not erase the numerical conditions that can flip the top token.
For non-experts, the useful mental model is simple: the model is not always choosing from a wide gap between options. Sometimes two answers are almost tied, and a small technical wobble can tip the scale. A long completion gives many chances for that wobble to matter.
The follow-on impact is practical. Teams needing stable answers should care less about the exact flip formula and more about caching, version control, and observability. Teams running long completions, multi-step agents, or evaluations should expect some drift even when they set temperature to 0.
The article's one-line framing is a warning: identical completions can fall apart after about a hundred tokens because top-token flips compound. That does not mean every 100-token run will differ. It means builders should not assume they will not.
Comments (0)