Your Agent Nailed the Task. Can It Repeat the Success?

If you build or deploy LLM agents, you've probably trusted benchmark averages without realizing how misleading they can be. This article explains a reliability problem that most evaluations ignore: agents that succeed on average often fail unpredictably on individual requests. You'll learn what the "consistency gap" is, why it happens, and how a new open-source tool called the Consistency Analyzer — part of ALTK-Evolve — can cut that gap roughly in half without sacrificing accuracy.

Watching a Flaky Agent on Stage Is Painful. In Production, It's Worse.

When an agent nails a demo and then stumbles during the live repeat, the audience chuckles. But the same flakiness in a production system is a genuine reliability issue: a workflow that worked once can break the next time a user sends the identical request. For high-stakes tasks — reconciling a payment, checking a contract for a specific obligation — unpredictable behavior is a dealbreaker.

Most benchmarks bury this variability inside an average. On AppWorld, a ReAct agent powered by GPT-4.1 succeeded on 77.4% of runs, averaged over five repetitions. But it succeeded in all five runs on only 53.0% of tasks — a 24.4-point consistency gap.

Benchmarks almost always report the first number. A team has now built a way to measure the second — and improve it.

In earlier work, they introduced ALTK-Evolve, a system that distills an agent's own past trajectories into reusable guidelines that get injected back at inference time. That approach improves task success, but it only answered the average-case question. This post introduces consistency guidelines, a new guideline type in altk-evolve, built on a diagnostic tool called the Consistency Analyzer, which targets the reliability gap head-on.

The Short Version

• Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal) that averages 77.4% success passes all 5 repeated runs on only 53.0% of tasks — a 24.4-point consistency gap. On hard tasks, that gap reaches 30 points.
• They built a diagnostic for it. The Consistency Analyzer resamples an agent's recorded trajectory to find flip-prone decision points — spots where the model was one token away from choosing differently. It requires just one trace and no ground truth: instead of rerunning the task end-to-end, it resamples each decision point with a single call requesting k completions (k=5 by default).
• Turning diagnosis into guidelines halves the gap — from 24.4 to 12.0 percentage points (same-task Pass⁵ up +16.0pp, similar-task up +13.0pp), with no loss in average accuracy.

The full methodology and evaluations are in the technical report on arXiv.

The Metric Almost Nobody Reports

Standard agent evaluation reports Mean@k: run a benchmark k times and average the pass rate. Often k=3, sometimes just 1. It's the number on every leaderboard — it's what "77% accurate" actually means.

Mean@k answers: how good is this agent, on average? It doesn't answer the question real users care about: will it still work if I ask this exact question again? For that, you need Pass^k: the fraction of tasks where the agent succeeds on all k runs.

One warning: Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, which is the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. The inequality Pass^k ≤ Mean@k ≤ Pass@k always holds.

So the GPT-4.1 agent posts a Mean@5 of 77.4% — genuinely strong. But its Pass^5 is only 53.0%. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can't, with nothing changing between runs. That gap — Mean@k minus Pass^k — is the consistency gap.

This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.

Why Agents Flip: Sharp Decisions vs. Flat Ones

Every time an LLM agent makes a choice — which API to call, which argument to pass, whether to retry — that choice comes from a probability distribution over the next token. What matters is the shape of that distribution. A sharp distribution concentrates almost all its mass on one token: the runners-up are far behind, so the same choice appears run after run. A flat distribution spreads similar mass across several near-tied tokens — picking the winner is close to a coin flip.

The shape determines how much noise it takes to change the outcome. Sharp distributions are resilient: GPU floating-point non-associativity, request batching, and other platform-level effects nudge the numbers slightly, but nowhere near enough to overturn a clear winner. Flat distributions are vulnerable to exactly that nudge — near-ties can reorder under tiny perturbations. And because a trajectory chains dozens of decisions together, a small per-step chance of flipping compounds into a large chance that some run goes differently. That's where a 24-point gap comes from.

This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed control how a distribution becomes a token — they say nothing about the distribution itself. On a hosted endpoint, probabilities shift slightly between runs, so the same prompt to the same model at temperature zero can resolve a near-tie one way today and the other way tomorrow.

Notably, the ReAct agent in these experiments runs at temperature 0.0 — so none of this variance is ordinary sampling noise.

Diagnose, Then Fix

That reframes the problem as a search: which steps in a given trajectory were the flat ones, and what do you do about them once you know?

Consistency guidelines come from a two-stage pipeline that plugs into ALTK-Evolve's existing machinery, with a new source signal driving what gets written.

Step 1: Detect with the Consistency Analyzer. Given one recorded trajectory, the analyzer replays each decision step through controlled resampling, measuring how much the model's output actually varies at that point. Concretely, that's one extra model call per decision step, done once offline — issued with sampling parameters set to draw k completions at once (k=5 by default) — replayed against the already-recorded context. No new tool calls, no new environment interactions, no second end-to-end rollout. The result is a consistency score per decision step, written into a scorecard that pinpoints exactly which decisions risk flipping next time. Detection is fully black-box: no logits, no model internals, no instrumentation beyond the trace you already have.

Step 2: Generate targeted guidelines. Every flagged step becomes a candidate consistency guideline in the standard ALTK-Evolve format, so it slots into the existing storage and retrieval pipeline. Here's a real example generated by GPT-4.1 from an AppWorld trajectory ("How many activities are done in my bucket list as per my SimpleNote note?"):

[Guideline 1] When counting checkbox-style markers in note content, use a line-anchored regex match rather than a plain substring count — note titles often repeat the marker symbol in a legend line.

[Guideline 2] Always verify search results for note queries by checking for multiple matches and confirming the correct note before proceeding.

None of this is task-specific trivia. String-counting bugs and unverified search results are decision points that show high uncertainty across many AppWorld tasks. That's the point: the analyzer targets instability, not failure — so it catches steps the agent happened to get right this time but could easily get wrong next time.

A two-minute demo shows five parallel runs splitting 3–2 on this task due to uncertainty about the counting strategy; after adding these guidelines, all five runs agree.

Results: Shrinking the Gap Without Losing Accuracy

The evaluation used AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing on 5 fresh runs.

• The consistency gap is roughly halved. Aggregate Pass^5 rises from 53.0% to 69.0% while Mean@5 rises from 77.4% to 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp. Nearly a third of previously inconsistent tasks become tasks the agent passes on every single run.
• Middle and hard tiers gain the most. Medium tasks improve +22.9pp (+44% relative), Hard +14.3pp (+45% relative) — effectively tied in relative terms. Easy gains +12.2pp, having had the least room to improve.
• Mean@5 never drops. Preserving average accuracy was a hard requirement, not a nice-to-have. A system that raised Pass^5 by trading away Mean@5 would just move unreliability around rather than fix it. Mean accuracy holds or improves at every difficulty level.

The guidelines generalize — they aren't patching one trajectory. Applied to a different but related task in the same AppWorld scenario, consistency guidelines still lift Pass^5 by +13.0pp, only 3 points below the same-task result. A guideline derived from one run isn't just patching that run; it captures something transferable.

The sharper evidence comes from a weaker model, gpt-oss-120b. Same-task Pass^5 rose +6.0pp from a much lower baseline (10.1% to 16.1%) — and interestingly, the similar-task gain (+8.7pp) actually exceeded the same-task gain, suggesting the guidelines captured genuinely reusable failure patterns rather than memorizing one trajectory's specifics.

If You're Shipping an Agent

• Report Pass^k next to Mean@k. Averages can't distinguish a reliable agent from a lucky one; even k=3 will surface a gap you didn't know you had.
• Expect the gap to widen with difficulty. Your hardest tier is where a single averaged number is most misleading.
• Don't reach for a bigger model first. Consistency is orthogonal to capability. A stronger model raises Mean@k; it doesn't necessarily reduce the consistency gap.
• Diagnosis needs no grader and no live replay. One extra LLM call per decision step (sampling k=5 completions by default) is enough — no ground truth, no rerunning the task against the environment. That's what makes it usable on production traffic, where you often can't replay a task end-to-end even once.

Try It

The ALTK-Evolve open-source repo now includes the Consistency Analyzer and the consistency-guideline generation used in these experiments. The technical report on arXiv covers the complete methodology. If benchmark accuracy numbers you can't reproduce on your own tasks sound familiar, the team wants to hear about it — concrete examples of flip-prone behavior in your own agents directly shape what gets built next. Open an issue or a discussion.

Appendix: Understanding the Metrics

• Mean@k — Run a task k times, report the average pass rate. What most benchmarks call "accuracy."
• Pass^k — The fraction of tasks where the agent succeeds on all k independent runs. Always ≤ Mean@k. This is what a user experiences when they run the same query twice.
• Pass@k — At least one of k runs succeeds. The optimistic counterpart, common in code-generation papers.
• Consistency gap — Mean@k − Pass^k, in percentage points.

Linked artifacts:

• ALTK-Evolve open-source repo — github.com/AgentToolkit/altk-evolve
• Technical report — arXiv
• Consistency guideline demo — 2-minute video