Uncovering Agent Behavior: Evidence-Based Oversight for Long-Horizon Agents


Meta description: Learn how evidence-grounded oversight helps users track long-horizon AI agent decisions with AgentMonBench, EBG, and Codex Harness checkpoints.

Tags: AI agent oversight, long-horizon agents, evidence-grounded evaluation, AgentMonBench, EBG

For anyone who uses or builds autonomous AI agents, a recent paper introduces evidence-grounded oversight as a way to answer a practical question: what did the agent actually do? The paper, "What Did the Agent Actually Do? Evidence-Grounded Oversight for Long-Horizon Agents," presents AgentMonBench, a benchmark for checking whether large language models, or LLMs, can spot important choices and find the logs that support them. It also presents the Evidence-Grounded Behavior Graph, or EBG, a training-free structure that connects scattered evidence to task requirements.

Why agent oversight is becoming a behavioral problem

The core issue is not that AI agents are simple. They are becoming more capable, moving beyond single questions or isolated tool calls into workflows that can last for hours or even days. In software development, data analysis, and scientific research, agents can plan on their own, invoke tools, modify code, and change strategy as results arrive.

That capability creates a supervision gap. Traditional AI interpretability research has focused on how a model makes internal decisions. For long-horizon agents, the harder question may be different: how can a user tell what the agent actually did across many steps? The problem is that a stated goal is not a complete contract. The agent may need to fill gaps, choose between options, or adapt when something fails.

The paper highlights three failure modes that can happen even without any obvious error. First, underspecified requirements: if a user instruction is incomplete, the agent may make assumptions that do not match the user's real intention. Second, silent behavioral deviations: the agent may change its plan or implementation without telling the user, and passing tests may not reveal the difference. Third, unverified consequential decisions: choices such as changing evaluation metrics, filtering data, or adjusting experimental settings can reshape the final outcome without being confirmed.

As execution traces grow longer, the evidence for these decisions is spread across conversation turns, tool calls, code changes, and intermediate artifacts. A user cannot realistically read every step. This creates a challenge for traditional human-in-the-loop oversight: a person can remain nominally in the loop while still missing what mattered.

What the new oversight system is designed to detect

The paper frames the task around two related questions. The first is alignment: does the agent's observed behavior match the user's requirements? This includes cases where the original request left important details unresolved, or where the agent drifted from the intended path. The oversight system should flag issues that may need clarification or verification.

The second question is awareness of consequential decisions: what choices did the agent make that could affect the result? The system should surface decisions even when they are not obviously wrong. A change to a dataset filter, a metric definition, or an experimental setting may be reasonable, but it still deserves a human check because it can determine the final answer.

This distinction matters for everyday agent users. A useful oversight tool should not only ask whether the agent produced a result. It should help the user see which decisions shaped that result and where the supporting evidence lives.

How AgentMonBench and EBG turn logs into a reviewable trail

To measure progress, the paper introduces AgentMonBench. The benchmark has three subsets: SpecGAP, SilentSwap, and FeedbackTrace. Together, they are meant to evaluate whether large language models can identify consequential decisions and locate supporting evidence in agent workflows.

The proposed method is the Evidence-Grounded Behavior Graph, EBG. Its basic idea is to organize scattered evidence from repositories and execution trajectories into a structured behavioral graph. Instead of forcing a human to search through raw logs, the graph links task requirements, agent actions, changes, and evidence sources in a way an oversight model can inspect.

EBG is described as training-free. It can be paired with existing models to surface decisions and trace them back to their original sources. The paper also integrates the approach into the Codex Harness, placing oversight checkpoints before execution, after consequential execution changes, and before final reporting. These checkpoints are intended to help users clarify requirements, review important behavioral changes, and verify outcomes with evidence.

What the experiments suggest, and what they do not claim

The experiments compare behavior across eight models. The paper reports that EBG improves consequential decision identification and evidence localization in most evaluation settings. The results point in a useful direction, but they do not mean that oversight is solved. The benchmark measures a specific skill: spotting important decisions and finding evidence. Real user workflows are messier, and not every risky decision will look dramatic in logs.

Case studies involving real research workflows are included to show how the system could support verification and guide later agent execution. Those examples illustrate the intended workflow: a human reviews highlighted decisions and evidence, then decides whether to accept, adjust, or intervene.

Why this may change how people use autonomous agents

The broader message is a shift in interpretability. For long-horizon agents, the question is not only "why did the model reason this way internally?" It is also "what did the agent actually do, which decisions shaped the outcome, and what evidence supports those decisions?" That reframing makes oversight more concrete for users.

For developers and researchers, the value is practical: a way to move from passive log review to targeted verification. For users of agents in coding, analysis, or research, the implication is that the next trust layer may not be a longer list of permissions or a bigger prompt. It may be a system that tells them, at the right moment, what changed, why it could matter, and where to check it.

The paper is available at arXiv:2610.06406. Code is available at zhk-lab/EBG, the benchmark at ZhaoHongKang/AgentMonBench, and the project page at zhk-lab.github.io/agentmonbench-ebg.