Opera: A Language-Based Critic Framework for Extended-Horizon Coding Agents

Meta description: Opera, a verbal critic for long-horizon coding agents, tracks corrections until issues are resolved and improves resolve rates on software benchmarks.

Tags: coding agents, test-time critics, software engineering, Qwen3.5-9B, Opera

Featured image: Abstract friendly illustration of a small agent checking a persistent note beside a terminal window and a repair path, with no real people, no logos, and no text.

Opera verbal critic is a test-time critic for long-horizon coding agents that is designed to keep following up on each correction until the underlying problem is resolved. In reported evaluations, it raised resolve rates by up to 12.4, 15.0, and 8.9 points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1 across four policy models. The result matters to anyone who has watched an agent accept feedback, make a superficial edit, and still fail the task: Opera shifts the value of the critic from giving advice to verifying that the advice worked.

From a one-time note to a tracked correction

Long-horizon coding tasks often fail in delayed ways. An agent may run many steps, edit files, install packages, or test code before the real problem becomes visible. A simple critic can interrupt with a natural-language instruction, but that instruction may be ignored, partially applied, or followed in a way that only appears compliant. Opera treats every correction as a persistent note. The critic does not stop after sending the message. It continues to monitor the agent's later actions and asks whether the diagnosed issue has actually been fixed.

This design addresses a common weakness in agent pipelines: feedback that is delivered but not enforced. For readers building or evaluating coding agents, the distinction between the agent changed something and the agent solved the reported problem is important. A visible diff is not the same as a successful task. Opera aims to close that gap by tying each correction to a follow-up check.

When to review, what to diagnose, and what to audit

Opera decides when to inspect the agent's work using two kinds of triggers. Periodic triggers let it step in at regular intervals, while event-driven triggers let it react when something notable happens in the task. This combination is useful because long-running coding sessions can fail both slowly and suddenly. A periodic check can catch drift; an event-driven check can respond to a failed test, a stuck loop, or another visible signal.

When it reviews, Opera uses typed operators to diagnose issues. In plain terms, typed operators are structured ways of naming what kind of problem has occurred, rather than leaving the critique as a vague natural-language comment. That structure can make the feedback more testable and easier to track. Before sending a correction, Opera audits the feedback against visible evidence. This step is intended to reduce critics that rely on assumptions or hallucinated errors. After the agent receives the correction, Opera tracks subsequent actions to separate real fixes from mere compliance.

Benchmark gains across coding tasks

As a test-time critic, Opera improves resolve rates by up to 12.4, 15.0, and 8.9 points on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1 across four policy models. Those are reported gains in the final task-resolution metric, not just intermediate edits or reviewer satisfaction. Opera also outperforms competitive critic baselines on all three benchmarks, suggesting the follow-through mechanism is not only a design detail but a practical advantage in evaluation.

The system also helps when a policy critiques itself. That is an interesting case because self-critique is often used to reduce the need for an external judge or to improve a model without adding another actor to the pipeline. If Opera's method still helps in self-critique settings, readers can infer that the value may come less from an outside model and more from the structure of the review process: trigger, typed diagnosis, evidence audit, and follow-up tracking.

Training data that carries over across harnesses

The reported benefits are not limited to inference-time use. Opera's guided rollouts can serve as near on-policy training data. Near on-policy means the data is close to what the policy would produce under its own behavior, rather than coming only from a distant teacher or human-written examples. That can matter for fine-tuning, because training on data that is too far from the model's own decisions may teach patterns that the model struggles to reuse.

In the reported result, fine-tuning Qwen3.5-9B on Opera's guided rollouts boosts held-out SWE-Bench Pro performance by 10.2 points without a critic at inference. The model gets better even after the critic is removed. The result also matches fine-tuning on a stronger model's rollouts, while, unlike that approach, preserving performance when switching harnesses from OpenHands to Terminus-2.

This last point is especially relevant for teams that change agent frameworks. A harness is the surrounding software that gives the agent access to tools, files, terminals, tests, or task environments. If a model trained under one harness drops sharply when moved to another, the learned behavior may be too tied to the framework. The reported robustness suggests Opera's rollouts may capture more transferable problem-solving behavior than the stronger-model rollout baseline in this comparison.

What readers should watch for

For developers, the practical takeaway is that Opera is not just another critic prompt. It adds a workflow: decide when to inspect, classify the issue, check the evidence, send a correction, then track whether the correction becomes a real fix. For benchmark users, the reported point gains on Terminal-Bench 2.1, a SWE-Bench Pro subset, and DeepSWE v1.1 across four policy models show that the method can improve end-task success. For model trainers, the Qwen3.5-9B result shows that guided rollouts can improve held-out SWE-Bench Pro performance by 10.2 points without using a critic at inference, while also holding up across OpenHands and Terminus-2.

The next questions are how these triggers and typed operators are chosen, how expensive the follow-up tracking is, and whether the gains hold outside the reported benchmarks. The reported results show the performance numbers and the transfer behavior, but readers evaluating Opera for their own agents should still check whether their coding tasks, harnesses, and policy models resemble the settings in which the method was tested.