Tokens, Context Windows, and Caching: What You’ve Been Getting Wrong

Meta description: Production Claude agents often get expensive because token pricing, context growth, and prompt caching shape latency. Here is how to measure and fix them.

Tags: claude-opus-5, prompt caching, AI agents, token cost, latency

Featured image: Abstract illustration of token fragments, a growing conversation window, and a cache shelf; friendly flat style, no people, no logos, no text.

Production AI agents built with Anthropic's claude-opus-5 often fail on cost and speed without crashing. A tested agent can cost three times the budget, response times creep up as conversations go on, and a short demo may break after a user asks four questions in a row. Logs may show nothing useful because nothing actually crashed. This is for developers and technical product owners who need to predict real bills; it shows how token pricing, context growth, and prompt caching work. These are default failures when teams build without understanding the machine underneath.

Tokens are priced by fragments, not words

A token is a small text fragment selected statistically during tokenizer training, not a grammatical word. Common words such as "the" and "running" can be one token each. Rare strings may split: "idempotency" may appear as "idem," "pot," and "ency." API keys and other UUID-like values are especially costly. Because every UUID looks different, its character groups must be encoded individually. A single UUID can cost 12 to 14 tokens, and a list of ten UUIDs can exceed a short English paragraph.

The familiar rule of approximately four characters per token is an English-prose average. Agents rarely send clean prose. They send JSON, code, database records, timestamps, field names, and identifiers, which often need more tokens per character.

Anthropic's count_tokens method for claude-opus-5 can measure the real prompt. Call it with the system prompt, tool definitions, and conversation history. The returned input_tokens number is usually larger than expected.

The tool-definition problem catches almost everyone. The JSON schemas that describe each callable tool are sent on every request, not once. A dozen well-documented tools can run to 4,000 or 5,000 tokens. They are billed on turn 1, turn 2, turn 15, and turn 40, for the entire life of every conversation. If an agent handles 10,000 conversations a month and each conversation averages 8 turns, those schemas are paid for 80,000 times. Before adding an eighth tool with detailed parameter descriptions, run count_tokens with an empty message list to see tool definitions alone. That number does not disappear; it multiplies.

Output pricing is higher because hardware behavior is different. Input tokens are processed in parallel during prefill. A 40,000-token prompt can be handled in one pass because parallelism is what GPUs are built for. Output tokens are produced one at a time during decode, and each output token requires another forward pass. Each pass depends on the token before it, so nothing can be parallelized. That sequential work is memory-bandwidth-bound, the most expensive constraint modern hardware faces. This is why output tokens cost roughly 5 times what input tokens cost.

Two consequences follow. Latency is mainly controlled by output length, not input length. Doubling a system prompt may add about 100 milliseconds, while doubling response length roughly doubles the wait. If an agent feels slow, the fix is usually a shorter answer, not a shorter prompt. A verbose answer is also billed twice: first as output, then as part of history on every later turn. An early chatty reply can be charged as input a dozen times before a long conversation ends.

Context is a cost curve, not memory

The context window looks like memory. The conversation history is placed there, the agent appears to refer back, and continuity feels natural. It is not memory. The model stores nothing between calls. Every time you call the API, you reconstruct the entire history and send it again. The model reads it fresh. What looks like memory is payment to rebuild history on every turn.

This reframes the economics. A ten-turn conversation is not ten independent requests. Each request includes every prior turn, the new message, every tool result, the system prompt, and the tool definitions. The tenth request can be closer to fifty times the first request, not ten times, because the total tokens sent grow with every turn.

The growth can be steep. A first turn might send 6,000 tokens. A fifth turn might send 22,000. A twentieth turn might send 80,000. A fortieth turn might send 160,000 tokens, approaching window limits. Across a 40-turn conversation, total spend grows closer to quadratic than linear. That curve breaks agent economics in production.

In one mid-conversation example, the user typed 40 tokens while the request paid for roughly 46,000 tokens. The real question was under one tenth of a percent of the bill. Everything else is chosen architecture, so it can be measured, changed, and cut.

Three context failures look similar from the outside but have different causes and fixes.

Overflow happens when the window fills. The request fails or the provider silently compacts the history, and the agent loses something it needed. This is the failure everyone prepares for and the least dangerous because it announces itself. You get an error or an obviously confused response. It is fixable.

Dilution happens when attention degrades unevenly in long contexts. Information near the start and end is recalled more reliably than information in the middle. The effect grows with context length. A fact quoted correctly at 8,000 tokens can be ignored at 80,000 tokens with no error, no warning, and a confident answer that is simply wrong. "The model supports a million tokens" does not mean you can put a million tokens in it and expect consistent behavior. Capacity is not attention.

Poisoning happens when an incorrect intermediate value, hallucinated result, or misread tool output is written into the agent's own history. That error stays in context for every subsequent turn, and the agent treats its prior output as established fact. Poisoning compounds. An agent twenty turns into a poisoned conversation can be confidently reasoning from a premise that was never true, and the transcript reads as perfectly coherent.

Overflow announces itself. Dilution is silent. Poisoning is a feedback loop that gets worse the longer it runs.

Practical defenses are measurement and trimming. For long conversations, use a rolling summary when history exceeds a threshold such as 20,000 tokens. Summarize everything older than the last three turns into a single paragraph and replace the raw turns with it. This flattens the growth curve dramatically. In a claude-opus-5 call, use max_tokens=512 and a system line such as "You summarize conversation history accurately and briefly." Ask for 3-5 sentences that preserve facts, decisions, and values. If each turn has user and assistant messages, keeping the last three turns means keeping the last six messages.

Trim tool output aggressively. If the agent needs three records from a query that returned 100, 97 records' worth of tokens are wasted on every later turn. Return exactly what the agent needs and nothing more. Keep verified facts in typed state, not in the conversation transcript. If the agent confirmed an order ID, store it in state and retrieve it directly when needed. Do not let the agent reconstruct it from what it said three turns ago.

Prompt caching only works on an exact byte prefix

Every provider that offers prompt caching uses the same underlying mechanism: if the start of a new prompt matches the start of a prompt sent recently, the provider skips processing the matching portion and charges a fraction of the normal rate. The savings are real. With Anthropic, cache reads cost roughly 10 percent of normal input token rates. Cache writes cost about 125 percent of normal rates. A short-lived cache entry reaches break-even after two requests.

The mechanism is a prefix match on exact bytes. It is not semantic similarity and not "close enough." That one fact explains why most caching setups fail silently.

When a request is sent to Anthropic, content is rendered in a fixed order regardless of how the API call is structured: tool definitions, then system prompt, then conversation history. Tool definitions go first, always, and sit at position zero in the token sequence. If the tool list changes between requests, the cache is invalidated for the entire request: system prompt, all history, everything.

Dynamic tool lists are a common trap. A tool list that changes per user, a list built dynamically from user permissions, or one tool schema regenerated on each request with a slightly different property order can mean full-price billing for every request, for every user, every turn. Nothing in the API response will tell you it happened.

The invisible invalidators are often small. A timestamp in the system prompt using datetime.now() to tell the agent the current time changes the cache entry every minute. You never get a cache hit. A UUID or request ID near the front creates the same problem: every request is unique.

For production agents, the practical impact is that cost, latency, and reliability are architecture choices. Measure prompt size with count_tokens, trim tool output, keep cache prefixes stable, and watch context growth before scaling users.