TokenCast: Predicting Token Usage During LLM Agent Execution

Meta description:
TokenCast forecasts LLM agent token use, cutting mean error by 14.5% and saving 21.3% tokens in budget replays, using 32.8 ms per run on SWE-bench Verified.

Tags: TokenCast, LLM agents, token forecasting, budget control, SWE-bench Verified

Featured image: A friendly abstract illustration of a token meter over a branching agent workflow, with soft geometric shapes, no text, no logos, no people.

TokenCast Forecasts LLM Agent Token Use and Cuts Errors by 14.5 Percent

For teams running LLM agents—language-model systems that plan, call tools, and keep working based on what those tools return—TokenCast is a forecasting method that predicts token consumption while an agent is still executing. The reported results are aimed at a common pain point: one agent task may cost only a little, while another run of the same task may consume over an order of magnitude more tokens. TokenCast reports a 14.5% average mean absolute error reduction against the strongest comparator across 96 evaluated combinations, and in offline budget-control replay it uses 21.3% fewer tokens than a fixed-budget policy at matched trace completion. The prediction overhead is reported as a mean cumulative prediction time of 32.8 ms per run on SWE-bench Verified.

Why agent token use is so variable

A plain language model call is easier to estimate because the prompt and the expected answer length are visible before generation starts. An agent is different. It may search, read, edit, test, or call an external tool. Each response changes the next prompt. If a tool returns a long error, a file diff, or a stack trace, the agent has to put that result back into context. The next call is then larger than the previous one. The next call after that may be larger still.

That feedback loop creates compounding cost. The same task can branch into many short paths or one long debugging path. A budget system that only looks at the original task description cannot know whether the run will finish in a few calls or repeatedly re-read the growing conversation. The problem description highlights this exact mechanism: the agent's next steps depend on tool feedback, and the growing context inflates the input size of every later call.

How TokenCast breaks the problem into parts

TokenCast does not treat a run as one opaque output. It learns a composable cost representation for each execution segment. The reported segment features are call count, net input-length change, and a cost residual. In practical terms, call count asks how many model calls are likely to happen, net input-length change asks whether the context is growing or shrinking between calls, and the cost residual captures remaining variation that the simpler segment features do not explain.

The method also includes an exact composition identity for the extra input cost of re-reading earlier context in every later call. That detail matters because context re-reading is not just a side effect of agent workflows; it is often the largest hidden cost. A later call may repeat much of the earlier conversation, tool output, or code state. If the model provider charges per input token, those repeated tokens become real spend. TokenCast tries to represent that repeated cost as part of the forecast rather than leaving it as a surprise at the end of the run.

A staged prefix-suffix predictor combines direct and compositional forecasts. It also refreshes its estimate from newly observed execution evidence. This is one of the most useful parts of the design: the forecast is not locked at the start. If an agent hits a tool error that adds hundreds or thousands of tokens to the context, TokenCast can update its expectation. Calibrated quantile models provide prediction intervals, so a planner can see a range rather than a single number. The method reports that no additional LLM calls are used for this refresh step, and the mean cumulative prediction time on SWE-bench Verified is 32.8 ms per run.

What the reported numbers mean for teams

Across 4 task suites and 6 agent models, the mean absolute error reduction against the strongest comparator averages 14.5% over 96 evaluated combinations. That is a broad result, not a single cherry-picked task. Still, it is an error-reduction number, not a guarantee that every deployment will save money. The offline budget-control replay result is more directly tied to cost: at matched trace completion, TokenCast uses 21.3% fewer tokens on average than a fixed-budget policy.

For a developer, that could mean fewer unnecessary stops when a run is still likely to finish, and fewer runaway calls when a task is becoming too expensive. For a product manager, it could make usage pricing and cost accounting easier to reason about. For an infrastructure team, it could make it simpler to set alerts, pause risky runs, or reserve capacity without relying on a single static token cap.

The practical value comes from the interval, not just the mean. A 21.3% token saving in a replay is useful, but a prediction interval tells teams how often they should plan for tail risk. If a forecast says a run may finish with low cost but has a wide upper tail, the system can choose a more cautious budget. If the forecast is tight, it can allow the agent to continue.

Who should use it / who should skip it

Teams should pay attention to TokenCast if they run multi-step LLM agents where tools can return long outputs, where context grows across calls, and where per-token cost affects budgeting or user experience. The reported 32.8 ms per run overhead on SWE-bench Verified makes the forecasting step look small compared with normal agent latency, and the lack of extra LLM calls is important for cost control.

Users may skip it if they only make single-turn model calls, use fixed-length prompts, or operate in a sandbox where token spend is not a meaningful constraint. They should also treat the 14.5% error reduction and 21.3% replay savings as benchmark results, not production promises. Real systems need their own logs, task mix, model provider pricing, and failure rates.

What comes next

The next question for readers is not only whether TokenCast forecasts better, but how a platform should act on the forecast. A prediction interval can drive early stopping, retry limits, tool-output truncation, or a switch to a cheaper model. The method's composable segments also suggest a path for cost dashboards: teams could see where the forecast came from—call count, context growth, or residual variation—rather than treating token spend as a black box.

For now, the core takeaway is concrete. Agent runs are expensive because later calls inherit earlier context. TokenCast tries to model that inheritance directly, updates as evidence arrives, and reports gains in both error reduction and replay-based token savings.