Compaction Is Lossy, So You Should Measure What Your Summary Discarded.
Long-running AI agents use context compaction to fit finite windows and control costs, but summaries are lossy predictions about future needs. High compression ratios can hide rework caused by discarded identifiers, failed attempts, and open obligations. This article explains why operational fidelity, not token savings, should drive compaction, and recommends test harnesses that probe summaries and block compaction when critical memory is lost.
Key Points
- Compaction can look efficient by token savings but may cause costly rework if it discards operationally critical details.
- Summaries should preserve exact identifiers, failed attempts, open obligations, uncertainty, goals, quantities, and freshness markers.
- High compression ratios are misleading if the next model turn must re-fetch, re-diagnose, or repeat mistakes.
- Measure compaction by whether the agent can still act correctly, not by how similar the summary looks to the original text.
- A fidelity harness with recall, artifact, decision, and continuation probes can help gate compaction when important memory is lost.
Teams that operate long-running AI agents need a way to measure summarization fidelity before automatic context compaction turns useful memory into rework. Public compression evaluations have quantified the problem: a higher compression ratio can be misleading because omitted details force the next model turn to re-fetch, re-run diagnostics, or repeat a failed fix. In one audited debugging trace, a compaction step saved roughly 120,000 tokens, but the loss it created made the agent re-read a 400-character Redis connection string with a revealing typo, rerun a diagnostic from minute twenty, and retry fix number two, which had already failed. The savings looked clean on paper; the amnesia bill included multiple token costs and one human hour.
This is the core bet: every summary is a prediction about what future turns will need, written under a token budget by a model that cannot see those turns. Most teams optimize for length reduction, not for whether the successor can still act, and almost nobody checks whether the bet won. The practical answer is a test harness that probes compacted context for operational fidelity, then gates compaction when important memory is lost. The harness matters because compaction runs unsupervised, dozens of times a week, on every long-running agent.
Why Compression Ratios Mislead
Compaction is not optional because context windows are finite, cost scales with context, and memory rot can arrive before the window fills. The useful question is therefore not whether to compress, but what the compressor discards. Audits across teams show that the losses are not random. They cluster into seven recurring classes, and each class produces a downstream symptom.
The evaluation work pointed to the same inversion: a high compression ratio can be misleading because lost details can make the successor spend more tokens than the compaction saved. Teams often celebrate very high compression rates until they price the rework. The deeper mismatch is structural. A fluent summary is written to be read, but the only reader that matters is a model that needs to act. A human editor would remove many of the details that keep an agent moving, and a summarizer trained on that editorial instinct does the same.
The public evaluations also gave a numeric signal. Structured summaries outperformed narrative summaries in the functional tests. On a 0-5 scale, the best structured approach scored 3.70, while defaults from major providers scored 3.35-3.44. The advantage persisted even when the structured method compressed slightly less. That result points away from token savings and toward operational fidelity: the summary is useful when the next turn can do the right thing without re-deriving what the previous context already knew.
What Compaction Usually Erases
The seven loss classes are the inventory of agent amnesia.
Exact identifiers are error strings, file paths, ticket IDs, commit hashes, and API parameters. Narrative summarization tends to paraphrase them, but a paraphrased identifier is a destroyed identifier. The successor needs the 400-character Redis connection string byte-exact, not a short description such as a Redis connectivity issue. The visible symptom is re-fetching material that used to be in context.
Negative knowledge is the tried-and-failed list. It is often the most expensive loss because missing it does not only cause a lookup; it causes repetition. In the debugging trace, the agent retried fix number two with fresh confidence because the summary had reduced four failed attempts to a vague note that several approaches had been attempted. The symptom is the agent walking through a dead end that the trace shows it already walked.
Obligations are the open TODOs and tracked steps that make a plan real. A ten-step plan compacted twice can become an eight-step plan with unmarked graves. The symptom is a count that looks finished, such as 214-minus-9-equals-done, when work has silently disappeared.
Doubt and provenance tags are the hedges that say what is single-source, probable, or uncertain. When a probable, single-source claim becomes a plain fact, the summary has laundered confidence. Step-12 guesses can become step-37 premises. The symptom is downstream certainty that the upstream never earned.
The goal itself can drift through telephone-game compression. A precise objective such as migrating 214 services without behavior change can shrink to a softer phrase such as improving logging. The symptom is diligent work aimed at a drifted objective.
Quantities and specifics lose their force when numbers are rounded or thresholds vanish. The 3 failing tests in auth_test.py can become some test failures. The symptom is later decisions made on vaguer ground than the original run supported.
State-of-the-world markers include as-of tags, freshness stamps, and which-version qualifiers. Without them, stale beliefs look fresh. The symptom is decisions made on information that has no expiry metadata.
These categories have one thing in common: they are operationally important and narratively optional. That is why summaries feel polished while agents behave badly.
Probing the Summary for Operational Usefulness
A fidelity harness should not ask whether the summary looks similar to the original. Lexical overlap is the wrong question. It should ask whether the successor can still operate. The evaluation template uses four probe families against the post-compaction context.
Recall probes ask for verbatim survival, such as what the original error message was. Artifact probes ask which files have been modified, testing the working-set inventory. Decision probes ask what was decided and why, because a decision without its rationale invites relitigation. Continuation probes ask what should happen next, testing whether the compacted context still implies the same next action as the full context.
Answers can be scored on accuracy, artifact-trail completeness, continuity without re-fetching, and instruction retention. That produces a fidelity number with practical meaning. The teams that first run this audit often stop arguing about compression ratios once they see the cost of rework.
Two metrics should stay on the dashboard: post-compaction re-fetch rate and post-compaction repeated-action rate. They itemize the amnesia bill. Every post-compaction re-fetch in a trace is a labeled loss, because the successor is effectively reporting what the summary failed to keep. A probe suite mined from that log is better than an invented one, because the log records real demand and weights identifier recall above trivia.
A methodological caution matters here. Probes are a proxy. The strongest test is downstream performance: run the same continuation task from full context and compacted context, then compare the outcomes. This counterfactual method is closer to reality, but it is expensive. The practical arrangement is to run full downstream evaluations occasionally to check whether probe scores predict continuation quality, and to run probes routinely because they are cheap.
Probe difficulty should be calibrated against what successors actually needed, not against byte-for-byte completeness. A suite that demands recall of everything defeats the purpose of compaction. It should demand the facts whose absence caused re-fetches, repeated actions, or bad decisions.
Engineering a Safer Compaction Step
The highest-yield change is to stop asking for a summary and ask for a schema. The schema should mirror the loss inventory, but in the opposite direction. It should have sections for exact identifiers, each tried-and-failed item and its failure reason, open obligations carried forward as a tracked list without silent edits, decisions with rationale, current working state, and next steps. Identifiers should be placed verbatim in a protected block that the summarizer cannot rewrite. An empty tried-and-failed section is visible in a way that a missing nuance never is.
This matters because narrative compaction places one holistic bet, and holistic bets shed operationally important but narratively optional details. A schema places a separate bet for each category, making missing content obvious.
The protected set should be enforced mechanically. It should include the goal, open obligations, doubt and provenance tags, and exact identifiers. A simple diff test can require that protected strings present before compaction remain present after compaction. If the check fails, reject the compaction. This five-line test retires four of the seven loss classes outright.
The deeper engineering shift is to compress to pointers instead of prose. The choice is not summarize or keep; it is summarize or archive. Move the raw segment to storage, such as a filesystem or memory system, and leave in context a structured summary plus a pointer, for example an archive path to the full debugging transcript. Then a failed bet costs a lookup instead of a re-derivation. An agent that needs the original error message can retrieve it with one tool call rather than repeating twenty minutes of diagnosis. This makes compaction non-destructive, because the original remains addressable. The absolute rule is never to compact material that has not been archived. An unarchived original makes every summarization mistake permanent.
Finally, the compaction prompt should be written as a worker brief, not a careless sentence. The summarizer should know its only reader is the model that must act next. It should be told which categories are mandatory, which strings are protected, how to mark uncertainty, where to place pointers, and what not to smooth away.
It also closes the failure-shapes arc that began with The Harness Is the Product, where compaction loss has been the mechanism behind half of the arc’s stories: telephone games that erode goals, dropped TODOs that become omissions, and stripped hedges that launder confidence. A summarization fidelity harness turns the hidden bet into a measured, gated operation, so a long-running agent can keep its memory without pretending that less text means better memory.
Comments (0)