Coding with AI

TokenEconomizer Helps AI Coding Agents Become More Efficient by Learning from Past Experience

TokenEconomizer learns from coding agent logs to cut SWE-Bench costs 18.4-30.9% via micro-skills, tool pruning, and failure-loop fixes.

TokenEconomizer Helps AI Coding Agents Become More Efficient by Learning from Past Experience

Meta description: TokenEconomizer, an open-source plugin, cuts AI coding agent costs by 18.4–30.9% across 277 SWE-Bench tasks without changing the underlying model.

TokenEconomizer Helps AI Coding Agents Become More Efficient by Learning from Past Experience

Tags: AI coding agents, TokenEconomizer, SWE-Bench, cost control

Featured image: An abstract, friendly illustration of repository session logs flowing into a lightweight controller that trims repeated tool calls and failure loops; no real people, no logos, no text.

Developers who run AI coding agents in the same codebase over many sessions can use TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research, and Microsoft Security. It turns completed session logs into repository-specific guidance for later sessions, cutting average costs by 18.4–30.9% across 277 real bug-fixing tasks while keeping task success broadly stable. It works beside an existing agent and does not modify the agent or its underlying model.

The waste shows up between tasks, not only inside one task

AI coding agents increasingly help developers fix bugs and build features with limited step-by-step input. Each session still sends instructions, tool descriptions, and session history to the model, so every turn has a cost. Most cost reduction work treats tasks as isolated jobs. Developers, however, return to the same repositories, which gives an agent a chance to learn from prior work. Without that learning, it may rediscover how to run tests, load tools it never uses, or spend dozens of turns revisiting the same error.

The research team first profiled Claude Code on 60 tasks from SWE-Bench, a benchmark of real bug fixes in open-source Python projects, running the tasks from each repository in sequence. Three patterns appeared again and again. Once a repository had at least five completed tasks, 30% of tool calls repeated an action from an earlier task in that same repository, while only 7% matched an action from another repository’s history. The agent received 23 tool descriptions on every turn, but 16 were never called. And the 17% of tasks that failed consumed 32% of spending.

The study also tested three popular open-source plugins built around fixed cost-saving strategies. Each reduced costs in some repositories but not others. That result pointed to a more practical conclusion: cost control needs to adapt to the repository instead of applying one strategy everywhere.

Three lightweight controls learn from finished sessions

After a session ends, TokenEconomizer reads the log and updates its repository knowledge. It then acts through three mechanisms, using only interfaces the agent already provides, so the agent remains unchanged.

The first mechanism is micro-skills. These are one-line notes that capture repository-specific facts. TokenEconomizer looks for recurring patterns across sessions and proposes a note when a pattern has appeared several times. The note is added to the agent’s starting context only after that confirmation, and it is retired if later evidence contradicts it.

The second is tool pruning. Repository history can show which tools the agent is unlikely to use. TokenEconomizer removes those tools from the starting context, while always retaining core tools for running commands and editing files. If a removed tool becomes necessary, it can be restored.

The third is failure-loop intervention. When an agent keeps hitting the same failure, a judge model decides whether it is truly stuck or still making progress. If it appears stuck, the agent receives a short note suggesting a different approach. In the open-source plugin, if the agent remains stuck after that note, TokenEconomizer alerts the developer instead of stopping the work itself.

The controller is designed to revise its decisions as new evidence arrives. The reported cost savings also include the cost of TokenEconomizer’s own model calls.

Test results: lower costs in every repository pair

The evaluation covered 277 SWE-Bench tasks from six repositories, using five AI coding agent and model combinations. For each repository, TokenEconomizer started with no prior session history. Across those settings, it lowered the average cost by 18.4–30.9%. It saved the most in all 30 repository-combination pairs without performance loss.

The comparison with reasoning effort is important. Lowering the model’s reasoning effort saved similar or larger amounts, but it led to substantial performance degradation. TokenEconomizer’s Pass@1 results rose in four of the five agent-model combinations, while costs fell in every repository tested.

The savings also changed over time. Early tasks can cost more while TokenEconomizer learns. In the Claude Code tests, cumulative costs fell below the default in every repository by the sixth task. Each of the three mechanisms contributed savings on its own, and combining all three produced the largest reduction.

The pattern did not stop at bug fixing. In sequential web development tasks on Web-Bench, costs were 16.0–21.2% lower. In security auditing, where each task happens in a new repository, costs were 6.4–19.6% lower.

Who should try it, and who should skip it

TokenEconomizer is most useful for developers and researchers who run AI coding agents repeatedly in the same repositories and want lower costs without replacing their agent or model. It requires no API key of its own, keeps what it learns on the developer’s machine, and is designed to run alongside existing agents. The paper “TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents,” along with a Microsoft Research Newsletter spotlight, describes the approach.

Teams should not expect instant savings in the first few tasks: early sessions can cost more while the plugin learns. The reported savings also vary by repository, agent, model, and task type. Still, the 277-task evaluation suggests a practical direction: coding agents can become more resourceful with experience, adapting how they work as well as what they know.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.