Coding with AI

Use NVIDIA NeMo to convert agent failures into tests and harness fixes.

NemoClaw turns agent traces into prioritized issues, repeatable tests, and measurable harness fixes for domain-specific AI agents.

Key Takeaways

  • NVIDIA's NemoClaw blueprint turns recorded AI-agent traces into prioritized issues, custom evaluations, and targeted harness fixes.
  • It helps diagnose domain-specific agent failures that generic benchmarks miss, such as wrong tool calls, missing confirmations, or broken business rules.
  • The workflow uses NeMo Compass to analyze traces, NeMo Eval Author to build repeatable tests, and then updates one harness component at a time.
  • Fixes are measured with development and held-out evaluations to confirm improvement and avoid overfitting or regressions.
  • The loop can be repeated: after fixing one issue, Compass can surface the next failure to optimize.

Meta description: NVIDIA's NemoClaw blueprint shows how NeMo Compass and NeMo Eval Author turn agent traces into tests, fix prompt harness issues, and measure results.

Tags: NemoClaw, NVIDIA NeMo Compass, NeMo Eval Author, agent evaluation, AI agents

Featured image: abstract, friendly illustration style, no real people, no logos, no text, showing agent traces becoming prioritized issues and evaluations.

NVIDIA's NemoClaw blueprint for agent optimization, available on GitHub and connected to a video from NVIDIA GTC Berlin 2026, gives developers and AI teams an agent optimization loop that converts recorded traces into prioritized issues, custom evaluations, and measurable harness changes. It is aimed at builders of domain-specific agents, where ordinary benchmark scores cannot explain failures.

Why generic scores miss agent failures

A domain agent can pick an unsuitable tool, take the wrong workflow branch, overlook needed context, waste a costly model, or complete a task while breaking a business rule. Reliability in production is shaped by prompts, tools, routing, memory, skills, control flow, and the surrounding code, so model tuning alone is not enough. Traces show where a run went wrong, and evaluations show whether a change makes the behavior better.

The tutorial demonstrates this with Hermes Agent, used as an enterprise assistant for a fictional company. The assistant runs in an NVIDIA OpenShell sandbox and uses Model Context Protocol, or MCP, a standard for connecting agents to external tools, to reach chat, email, files, and tickets. When asked to draft a message with project updates, it wrote the draft but also called send_message without user confirmation. That failure points to both the harness instructions and the tool execution path.

Prerequisites for the NemoClaw loop

Before following the example, prepare a Linux machine with Docker running natively. The recommended setup is Ubuntu 24.04 with 8 vCPUs and 32 GB of RAM. You also need OpenShell 0.1.2 and Harbor 0.22.0, an NVIDIA API key from build.nvidia.com, and a coding agent that NeMo Eval Author supports, such as Codex or Claude Code.

The tutorial points to a GitHub walkthrough and an NVIDIA Brev launchable for faster setup. The NemoClaw blueprint combines three libraries: NeMo Compass, NeMo Eval Author, and NVIDIA NeMo Relay.

The complete optimization cycle

  1. Analyze traces with NeMo Compass. The example repository contains 42 Hermes traces, one JSON line per run, a machine-readable format for each run, plus ETHOS.md, a short file that describes expected assistant behavior. NeMo Compass examines stored traces and evaluation evidence to compress a large trace set into a ranked list of issues, examples, and suggested areas for work. Its findings draw on unusual behavior, patterns in tool use, departures from intended behavior, and trace IDs that connect signals to specific runs. In the example, Compass compared actions with ETHOS.md rules and identified that the agent sent messages without confirmation. The tutorial reports that internal benchmarks on thousands of traces found anomalies a coding agent given a similar task missed.
  1. Build an evaluation with NeMo Eval Author. Once the problem is visible, create a repeatable test. Install NeMo Eval Author skills in the coding agent and instruct it to use them. NeMo Eval Author supports two evaluation formats: Harbor evaluations and NVIDIA NeMo Gym evaluations. Provide the Compass findings, raw traces, ETHOS.md, and access to the agent repository as context. In the example, the coding agent generated four Harbor evaluations similar to tasks like draft a response. It also created a verifier that fails a run when the agent calls send_message without human confirmation. Review the generated evaluations to make sure they reflect the actual requirements. NeMo Eval Author can also build evaluations before production data exists, using requirements from ETHOS.md, tool definitions, and existing workflows. As more traces and known failures arrive, expand the suite so it tracks real usage.
  1. Change one part of the harness. The example updated Hermes SOUL.md, a file similar to a system prompt. The coding agent receives Compass outputs, ETHOS.md, and the implementation, which grounds its proposed change. Harness updates can be simple instruction edits or more complex middleware, plugin extensions, and other patterns. The blueprint's earlier posts also cover LangChain Deep Agents and Nous Research Hermes.
  1. Measure before and after. Rerun the evaluations created by NeMo Eval Author. The example withheld two tasks labeled Held out from the coding agent that proposed harness changes. Held-out tasks check whether the fix generalizes beyond the development tasks.
Evaluation setBaseline passes/runsUpdated-agent passes/runsTool calls, before/after
Development, 2 tasks x 3 runs3/66/610 to 6
Held out, 2 tasks x 3 runs4/66/628 to 15

The tutorial says production evaluation suites often contain hundreds to a few thousand examples. Accept a change only when it improves both the development tests and the held-out set. If results are unclear, or a fix creates a regression, run NeMo Compass on the new traces from the updated agent and use those findings to refine the change. Keep held-out cases hidden from the coding agent to avoid overfitting.

  1. Repeat the loop. After the update, NeMo Compass on the scored traces no longer reported the sending-without-confirmation issue. It surfaced a new problem: for some queries, the agent found an answer but did not ground the response in retrieved evidence. That becomes the next evaluation and fix target. The tutorial suggests starting with the NemoClaw blueprint, the workplace assistant example, or the CAD guide. Run the example agent, establish a baseline score, complete one cycle, then point the loop at your own agent.

Engineering agent example

The loop can also be used for engineering agents, where tests should measure both result quality and tool-use efficiency. The blueprint includes a car door design example using LEO, the engineering Virtual Companion from Dassault Systèmes. The workflow proposes design changes and tests them through simulation to make a stronger, safer car door. A video titled Super AI agents at NVIDIA GTC Berlin 2026 accompanies the example.

In the engineering scenario, NeMo Compass analyzes traces to identify areas for improvement. A coding agent uses those findings to make targeted prompt and configuration changes, and evaluations measure the updated agent in the engineering environment. Additional scenarios check whether those changes introduce regressions.

NeMo Compass found that the agent sometimes placed QueryModel arguments outside the expected options field. The tool returned an empty error message, so the agent could not recover. The issue appeared in multiple traces. A coding agent then added a validation step that catches misplaced arguments and names in tool calls. Compared with the baseline, malformed tool calls fell from 142 to zero. Accuracy remained unchanged, and no regressions appeared in the remaining scenarios.

The first pass ran overnight without human intervention. In approximately 12 hours, the reported results showed a 28% relative improvement in the door design workflow's overall evaluation score and 35% fewer tool calls per run across six tested scenarios, with no regressions reported in the other scenarios. The example connects the workflow to two measures: task quality and execution efficiency. It follows the same process described in the tutorial: use traces to identify what to change, then use custom evaluations to assess the targeted improvement and check for regressions before adopting the change.

Common mistakes

  • Relying only on generic benchmark scores. If the agent can call the wrong tool or skip a confirmation step, the trace and evaluation must show why.
  • Inspecting every trace manually. NeMo Compass is meant to convert large trace corpora into a prioritized backlog of issues, supporting examples, and suggested areas for work.
  • Letting the coding agent see held-out tasks. If the agent knows the held-out cases, the improvement may be overfitting rather than generalization.
  • Accepting a change that fixes only development tasks. The example requires improvement on both development and held-out tasks before adoption.
  • Stopping after one fix. The loop should continue: after a change, analyze the new traces, find the next issue, and create the next test.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.