Tool Reviews

ALTK-Evolve reveals how learned memory reaches the top of the τ²-bench leaderboard for customer service tasks without relying on a frontier model.

ALTK-Evolve improves GPT-4.1 customer-service agents by appending learned guidelines to the system prompt. On τ²-bench retail, Pass^4 reaches 67.5%, above leaderboard entries without fine-tuning or weight updates. The method extracts support-counted rules from training trajectories and tests reliability across domains, showing guidance amount matters: retail benefits from full guidelines, while airline reliability may prefer smaller sets.

ALTK-Evolve reveals how learned memory reaches the top of the τ²-bench leaderboard for customer service tasks without relying on a frontier model.

Key Takeaways

  • ALTK-Evolve improves a basic GPT-4.1 customer-service agent by appending learned guidelines to its system prompt, reaching 67.5% Pass^4 on τ²-bench retail without fine-tuning or weight updates.
  • τ²-bench evaluates strict, policy-bound service: agents must follow written policies, interact with simulated customers, and take correct tool actions, with Pass^4 measuring reliability across all four attempts.
  • Guidelines are extracted from the agent’s own training trajectories and ranked by support count; the optimal amount of guidance is domain-specific, with full rules helping retail and compact high-support rules improving airline reliability.
  • The gains come from better decisions rather than longer conversations: matched solved tasks use slightly fewer agent turns, reducing latency, while the main cost increase is prompt input tokens that can be largely mitigated by caching.

Meta description: ALTK-Evolve adds learned guidelines to GPT-4.1 and gpt-5.5 agents, lifting τ²-bench retail Pass^4 to 67.5% without weight updates or fine-tuning.

ALTK-Evolve reveals how learned memory reaches the top of the τ²-bench leaderboard for customer service tasks without re

Tags: customer-service agents, learned memory, τ²-bench, GPT-4.1, prompt guidelines

Featured image: Abstract friendly illustration of a customer-service chat interface and a compact playbook icon, no people, no logos, no text.

For teams building customer-service agents, learned memory has become a practical route to better reliability. ALTK-Evolve appended rules distilled from an agent’s own attempts to the system prompt of a basic GPT-4.1 agent, and that agent reached 67.5% Pass^4 on τ²-bench retail — meaning it solved the task on all four independent attempts for 67.5% of tasks. The result sits above every retail entry in the October 12, 2026 leaderboard snapshot. The setup kept the benchmark’s stock ReAct scaffold and only added learned guidelines to the system prompt.

Why policy-bound customer service is a strict test

τ²-bench, from Sierra, measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal, gives the agent domain tools such as look up an order, issue a refund, or rebook a flight, and supplies a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions. A task counts as solved only when the final database state matches the expected outcome, a strict binary reward. The work uses the benchmark’s retail and airline domains.

The runs used τ²-bench’s own implementation of a basic ReAct agent with a two-phase THINK → ACT loop. GPT-4.1 served as the agent model, gpt-5.5 as the customer-simulator model, and every task received 4 trials. The only change was a block of learned guidelines appended after the domain policy as a plain markdown list. ALTK-Evolve extracts those guidelines from the agent’s trajectories on τ²-bench train tasks using gpt-4.1, then applies them unchanged to the held-out test set. The same model powers guideline extraction and the agent, and no weight updates or human annotation are needed.

How learned guidelines are built and measured

This method follows earlier work on coding agents, now tested on customer service, where success depends on following domain policies, understanding a customer’s request, and executing the right tool actions.

Each guideline carries a support count: the number of distinct training tasks it was extracted from, so higher support means a rule generalized across more tasks rather than a one-off. The comparison uses three configurations: baseline with no guidelines, sup3 with only guidelines support ≥ 3, a small high-confidence core, and lossless with every guideline retained after consolidation, support ≥ 1.

ALTK-Evolve reveals how learned memory reaches the top of the τ²-bench leaderboard for customer service tasks without re

Two metrics matter. Mean Accuracy is the average success rate across the 4 trials, showing capability. Pass^4 is the percentage of tasks that passed across all 4 trials, showing reliability; τ²-bench’s leaderboard reports it. A support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers, so Pass^4 asks whether the agent can be trusted every time, not just on average.

More guidance helps, but the right amount depends on the domain

With gpt-4.1 as the agent model and gpt-5.5 as the user simulator, Mean Accuracy climbed in both domains: retail rose +4.4 points from 78.1 to 82.5, and airline rose +7.5 points from 57.5 to 65.0. Reliability moved further. The strict Pass^4 rose +10 points in retail from 57.5 to 67.5, and +10 points in airline from 35.0 to 45.0.

The best amount of guidance was not the same in both domains. In retail, more is better on every metric: the small sup3 core barely beats baseline on accuracy, and the full lossless set is needed for the real jump. In airline, the picture flips at the reliability end: lossless leads on accuracy at 65% versus 60%, but the compact sup3 core of 14 guidelines is more reliable, taking Pass^4 to 45% against lossless’s 40%. Support count is a knob worth tuning per domain, not a value to hard-code.

The gains come from better calls, not longer chats

Guidelines are not free, but the cost lands in prompt input tokens, not in a longer, more meandering conversation. The reported cost tables compare total LLM cost per conversation, covering both agent and user simulator, on the full base split across four trials, with lower values better.

To separate reliability from chat length, the team compared matched runs solved under both baseline and the guideline variant — identical scenario and seed — and counted assistant turns. The agent was not talking longer. On the shared set, lossless trims about 0.3 to 0.4 turns in both domains: retail from 10.59 to 10.16, and airline from 9.15 to 8.82. No variant lengthened conversations. Each turn is a sequential LLM round-trip, so fewer turns means lower end-to-end latency and a faster time-to-completion for the customer, not just a cheaper one.

The main price is on the input side. Retail’s 292-guideline prompt pushes cost from about $0.084 to about $0.139 per conversation, while airline’s smaller playbook is cheaper. In production, much of that is largely reclaimable with prompt caching, because the guideline block is identical across every call. The trade is favorable: for a slightly higher input cost, much of it cached away in practice, the agent lands higher accuracy, higher reliability, and lower time-to-completion at once.

ALTK-Evolve reveals how learned memory reaches the top of the τ²-bench leaderboard for customer service tasks without re

Frontier models and leaderboard results

A fair objection is that learned guidelines might only patch a weaker model. The team tested airline using gpt-5.5 for both the agent and guideline extraction on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent’s own runs. One detail changed: none of the guidelines reached support ≥ 3; they top out at support 2, so the high-support core is sup2 rather than sup3.

The learned guidelines improved this setup as well. On the same held-out test tasks, appending them lifted accuracy +10 points from 72.5 to 82.5 and Pass^4 +15 points from 55.0 to 70.0 — a larger reliability gain than with gpt-4.1 on these tasks, +10 points, and from a higher floor. Where baseline reliability swung between reruns, the lossless configuration reproduced exactly. The gain looks dependable, not a lucky draw.

Submissions are scored on the full base split, train ∪ test, as τ²-bench requires, and reported at Pass^4. Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers above. Prompt-level memory carries agents using each base model into, or past, current frontier company.

In retail, the agent with learned guidelines reached Pass^4 = 67.5% on the base split, ahead of every frontier model and ahead of RAFT-30B-A3B at 61.4%, a model fine-tuned on synthetic τ²-bench retail data. Using stock gpt-4.1 and a prompt-appended playbook, it outperformed that fine-tuned system by +6.1 points.

In airline, the compact sup3 core gave the gpt-4.1 agent Pass^4 = 44%, placing it 8th against the published board, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5. Swapping in gpt-5.5 as the stronger base took the same approach to 82.0%, clearing every frontier entry in the 66–72% range.

Fine-tuning a model on a benchmark is expensive, brittle, and does not necessarily transfer to the next domain. Here, ALTK-Evolve improved a stock agent on a stock model in both domains, and matched or beat a fine-tuned system on retail, using only prompt-level guidelines extracted from the agent’s own runs automatically, without human annotation and without test-set leakage.

For readers who want to inspect the work:

  • Thinking of ACE? We Can Do It with Fewer Tokens
  • How much memory does your agent actually need?
  • ALTK-Evolve library: https://github.com/AgentToolkit/altk-evolve
  • τ²-bench: https://github.com/sierra-research/tau2-bench
  • Retail leaderboard submission: τ²-bench PR #542
  • Airline leaderboard submission: τ²-bench PR #551

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.