Tool Reviews

NVIDIA NeMo Gym is used to improve agent reliability and tool-calling efficiency in chip design.

NVIDIA NeMo Gym 用于评估和改进 Siemens Fuse EDA AI Agent,在标准单元表征中将 Liberty 文件验证通过率从约 55% 提升至约 95%。该案例说明芯片设计团队应验证最终工程产物、跨多阶段工作流追踪错误,并优化 Agent 技能与工具接口。NeMo Gym 通过定义任务、工具、状态和验证器,在受控评估循环中定位工作流故障并追踪错误路径,帮助 Agent 更可靠地调用表征工具并生成 Liberty 文件。

NVIDIA NeMo Gym is used to improve agent reliability and tool-calling efficiency in chip design.

Key Takeaways

  • NVIDIA NeMo Gym helped improve the Siemens Fuse EDA AI Agent’s standard-cell characterization workflow, raising Liberty file verification reliability from about 55% to about 95%.
  • The case shows that chip design agents should be evaluated against final engineering artifacts, not isolated tool calls, because early setup errors can silently propagate through multistage workflows.
  • NeMo Gym provides controlled evaluation environments that capture task state, tool actions, reasoning traces, and verifier diagnostics to identify where workflows fail and why.
  • Engineers used the evaluation loop to refine agent skills and Model Context Protocol tool descriptions, then retest under the same conditions to improve both reliability and efficiency.
  • A composite verifier checks Liberty file coverage, validity, relative validation, accuracy, and integrity, making the evaluation repeatable and focused on production-quality outputs.

Meta description: NVIDIA NeMo Gym helped a Siemens Fuse EDA AI Agent improve Liberty file verification from about 55 percent to about 95 percent in standard-cell characterization.

NVIDIA NeMo Gym is used to improve agent reliability and tool-calling efficiency in chip design.

Tags: NVIDIA NeMo Gym, Siemens Fuse EDA AI Agent, Liberty file verification, standard-cell characterization, agentic evaluation

Featured image: abstract, friendly illustration style of an AI agent checking a chip library file, with no real people, no logos, and no text

Chip design teams and electronic design automation developers can use this case to see how NVIDIA NeMo Gym evaluated and improved the Siemens Fuse EDA AI Agent as it called characterization tools and generated Liberty (.lib) files for standard-cell characterization. The work is practical because it shows how to measure final engineering artifacts, trace errors across a multistage workflow, and refine agent skills and tool interfaces instead of trusting isolated tool calls.

Why final outputs matter for chip design agents

AI agents can deliver large productivity gains in complex chip design workflows, but they also can produce outputs that look correct while containing hidden mistakes. That reliability gap is one of the biggest obstacles to production use. In multistage engineering work, a setup error can remain silent, move through downstream steps, and affect the final artifact. The safer evaluation target is the verified final output, not each intermediate action.

Standard-cell characterization creates timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and must be repeated when a semiconductor foundry releases a new process design kit. Each kit carries transistor models, design rules, and process, voltage, and temperature definitions that link manufacturing to design tools. When the kit changes, older characterization data may no longer match the updated devices. Timing and power models then need to be regenerated and checked for every affected cell across combinations of input slew, output load, and process, voltage, and temperature conditions.

The Solido Characterization Suite agent automates that path. It coordinates tools from the broader Solido suite through the Siemens Fuse EDA AI Agent, which exposes Model Context Protocol servers and skills to common coding harnesses such as Claude Code, Codex, and Cursor. These interfaces let engineering agents use accelerated tools, while skills guide their use inside characterization workflows. Faster simulation helps agents finish iterations sooner, and NVIDIA NeMo Gym helps developers measure how reliably and efficiently the agent orchestrates those tools.

The agent must handle tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation. The workflow moves from SPICE inputs through pilot validation, scale-out, Liberty generation, and waveform comparison, and early setup errors can propagate into the final library. Problems often show up as believable results rather than clean failures, leaving incorrect thresholds, missing corners, or incomplete cells hidden. Because each stage depends on the one before it, an early setup mistake can travel all the way to the final library. Defining a rigorous evaluation protocol was therefore a key step before bringing the agent into production workflows.

How NeMo Gym turned failures into fixes

NeMo Gym is an open library that evaluates and improves models and agents through environments. It is part of the NVIDIA NeMo collection of open libraries for agent specialization and optimization, and it integrates with the standard-cell characterization flow inside a controlled evaluation loop. Its purpose is to identify where workflows break and to follow the path of errors through complex, multistage tasks. For each rollout, NeMo Gym defines the task, tools, state, and verifier, so attempts start from a clean state and can be compared fairly. It also records thinking traces, input and output messages, and tool actions.

NVIDIA NeMo Gym is used to improve agent reliability and tool-calling efficiency in chip design.

This setup gave engineers a repeatable optimization loop:

  1. Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment.
  2. Inspect trajectories, failed tool calls, and verifier diagnostics.
  3. Identify ambiguous instructions, unproductive actions, and incorrect tool selection.
  4. Refine the agent’s skill and Model Context Protocol tool descriptions.
  5. Repeat the same tasks under the same evaluation conditions.
  6. Compare verified outcomes and efficiency per successful result.

The Siemens Fuse team used NeMo Gym to run evaluations at larger scale, checking generated Liberty (.lib) files against known reference artifacts and following both common and rare failure paths through each rollout. NeMo Gym checks Liberty files produced by the Solido Characterization Suite agent and Solido tools, then feeds failure analysis into the next skill and tool-description revision. A coding agent drove the process, with a human in the loop.

End-to-end checks revealed problems that intermediate checks missed. Final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. Those findings supported a simple rule: engineering agents should be judged against final artifacts, not only against successful individual tool calls.

A verifier that rewards complete Liberty files

To make the evaluation repeatable, the team created verifiers that return a final composite reward based on five checks:

  • Coverage: whether the agent produced every expected cell and process, voltage, and temperature corner.
  • Validity: whether the delivered Liberty files are well-formed and usable by validation tools.
  • Relative validation: whether the output introduced failures beyond those already present in the reference.
  • Accuracy: whether the expected timing and power values fall within the defined tolerance.
  • Integrity: whether the characterization workflow produced a complete result rather than reusing, truncating, or reformatting existing values.

The reward is defined as:

R = 1[Coverage] × 1[Validity] × 1[Relative Validation] × 1[Integrity] × (X_tol / X_total)

Here, X_tol is the number of output values within tolerance, and X_total is the total number of output values. Each indicator equals 1 when its gate passes and 0 when it fails, so any failed gate makes the reward zero. This verifier helped the team systematically tune the agent across multiple iterations and trace complete rollout trajectories to find root causes behind subtle errors.

NVIDIA NeMo Gym is used to improve agent reliability and tool-calling efficiency in chip design.

Benchmark results after skill and tool fixes

The team used the open source GF180MCU PDK to benchmark the Solido Characterization Suite agent on Liberty-file generation. The dataset includes 229 standard cells across 36 process, voltage, and temperature corners, with Solido AFS used for SPICE simulation. A custom agentic environment was built using NeMo Gym. For each run, the characterization flow was limited to two cells across 36 process, voltage, and temperature corners because this smaller subset was sufficient for tuning the agentic workflow. Each run took approximately 1.5 to 2 hours, and the team conducted approximately 25 rollouts using Claude Opus 4.8 running in Claude Code as the fixed model and agent harness.

Across successive evaluation rounds, the team refined the agent’s characterization skills and Model Context Protocol tool descriptions. To improve tool selection, they simplified overlapping file and shell interfaces and defined tool inputs with explicit JavaScript Object Notation schemas, making characterization tools easier for the agent to call.

Following those refinements, the mean composite verification score across evaluated rollouts increased from approximately 55% to 95%. The improved workflow also required fewer agent turns and output tokens. Agent turns fell from 155 to 53, a reduction of approximately 66%, while output tokens fell from approximately 81,000 to approximately 29,000, a reduction of approximately 64%. These results cover representative standard-cell characterization tasks.

ModelBenchmark composite scoreTurnsAverage output tokens
Claude Opus 4.8about 55% to about 95%155 to 53about 81,000 to about 29,000

Who should use it / who should skip it

This case is most useful for chip design teams, electronic design automation developers, and engineering-agent builders who need measurable reliability for multistage workflows. The same method applies to engineering agents in electronic design automation, computer-aided engineering, and other technical domains: evaluate final engineering artifacts, not just individual tool calls; write a domain-specific grader or verifier that detects missing, truncated, reformatted, or reused results to prevent agents from gaming the score; and collect large numbers of rollouts to improve reliability through optimized skills and Model Context Protocol tool interfaces.

To get started, teams can use the NeMo Gym Quickstart guide to run an initial evaluation. Then they can apply the approach to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent evaluation conditions. More details about how Solido Fuse brings generative and agentic AI to the Solido Characterization Suite agent are available in the Self-Verifying, Long-Running Agentic Characterization Workflow Demo. Readers can also learn more about the collaboration between NVIDIA and Siemens.

Siemens’ work provides a practical example for chip design teams and electronic design automation developers on how to leverage the NVIDIA AI software stack and open libraries to improve agent reliability and chip design workflows.

The team credited Gunjan Shah of NVIDIA, along with Tim Wheler, Lakshminarayanan Ramesh, and Carl Hofmeister of Siemens, for key contributions to developing the agent and establishing the evaluation methodology.

Teams that cannot define a verified final artifact, or that only need lightweight assistance for isolated tool calls, may find the full evaluation loop unnecessary. For production-oriented characterization workflows, however, the case shows that final-output verification is the difference between plausible automation and dependable automation.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.