Inside the Omniverse: How Developers Use Frontier AI Agents to Turn Ideas into Simulations

Meta description: See how developers use GPT-6 Astra with NVIDIA Omniverse libraries to build robot, autonomous vehicle, digital twin, ISS, and room simulation apps.

Tags: NVIDIA Omniverse, GPT-6 Astra, simulation development, digital twins, robotics

Featured image: abstract friendly illustration of connected 3D warehouse, robot, camera, and simulation nodes; no real people, no logos, no text

NVIDIA teams and developers have shared examples of using frontier AI models such as GPT-6 Astra with NVIDIA Omniverse libraries to turn simulation ideas into working applications. The examples are aimed at developers building tools to explore scenarios, investigate failures, and refine designs. In these workflows, people give natural-language instructions to AI agents, review generated code and scene changes, and guide corrections while Omniverse libraries provide GPU-accelerated physics, rendering, and sensor simulation capabilities.

Warehouse humanoid control becomes a playable simulation

Frank DeLise, Omniverse product manager at NVIDIA, used Astra to turn a SimReady warehouse and humanoid robot into an interactive simulator with first-person and third-person views. The demo presents a gamified, physics-based control experience for a humanoid robot. Before warehouse tasks are automated, developers need a place to explore task behavior and evaluate how work gets done. DeLise directed Astra to connect Omniverse libraries for physics (ovphysx), scene updates (ovstage), rendering (ovrtx), and the user interface (ovui). He also used Astra with SimReady (simready-foundation) to create the physical scene in simulation. Astra generated animation and application code to bring those capabilities together.

Humanoid warehouse demo

Autonomous-driving tests move from scene changes to driving behavior

Doyub Kim, a manager on the simulation technology team at NVIDIA, asked Astra to build Zero to Alpamayo, a reusable simulation environment based on San Francisco's Market Street. Even small changes to a scene, sensor setup, or driving model can shift autonomous-vehicle simulation outcomes. Kim directed Astra to map out the workflow, then connect asset creation, traffic, Omniverse RTX sensor simulation, and Alpamayo driving in stages, checking each integration. The resulting prototype became a testing ground for comparing models and tracing how scene or sensor changes affect downstream driving behavior. A separate Cosmos3-Nano experiment varied weather and lighting in recorded simulation videos, allowing Kim to compare the driving model's responses to the same scenario under different conditions.

Zero to Alpamayo demo

Digital twins improve when real and simulated sensors disagree

When recorded camera and LiDAR data are compared with simulated outputs, developers receive measurable feedback that can guide scene refinements. Ashley Reid, who works on RTX sensor validation at NVIDIA, directed Astra and Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs with recorded data. The agents created two digital twins from scratch and improved two existing ones. Over about three days, Reid guided an iterative workflow in which agents measured differences, created or modified OpenUSD scenes, and checked the results. Changes addressed missing objects, geometry, and materials, with acceptance depending on camera and LiDAR metrics. This gives developers a way to use measured discrepancies to guide scene creation and improvement. A useful starting point is rendering an OpenUSD scene with the ovrtx minimal Python example, then defining a sensor measure to compare with recorded data.

Sensor comparison demo

Robot skills get tested through physics trials

Robot training for new motions needs to confirm that actions remain feasible under physical constraints. Tae Kim, who leads NVIDIA Omniverse engineering and product, used sports videos and natural-language instructions to guide Astra in building Robo Olympics, an experimental project that tests simulated Unitree G1 humanoids performing sports movements. Under Kim's direction, Astra built controllers and refined them through physics trials. Newton Physics Engine simulated behavior, the open source NVIDIA Warp framework accelerated calculations, and ovrtx rendered scenes and virtual-camera images. In one experiment, the robot cleared a single hurdle in 64 of 100 simulation trials. The trials gave Kim feedback for improving the robot's timing and control.

Disassembly testing links CAD, simulation, and robot tools

Product disassembly requires checking whether a robot's tools can access and detach each part. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA Isaac Sim. With Astra, Jebens explored computer-aided design and tooling revisions for robots informed by simulation. The agent measured the available space and designed a wrench the robot could use to reach the suspension's bolts. Jebens reported successful removal of a suspension component in simulation. This connects design and tooling decisions to disassembly results, offering a starting point for robot policy training. The Onshape importer guide can help developers begin similar CAD-to-simulation workflows.

The International Space Station becomes a browser application

A browser-based 3D application needs assets, live data, and an interface wired together. Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble NASA assets into an OpenUSD International Space Station model with telemetry. Johns built the application with a single prompt, then used a follow-up prompt to shift the scene to Earth's daytime side so the planet was visible. The workflow used Blender for asset preparation and Omniverse libraries for rendering (ovrtx), scene runtime (ovstage), and streaming (ovstream). The application brings 3D models and operational data into the browser, with Johns guiding development through prompts and corrections. The Omniverse Real-Time Viewer skill is a starting point for similar browser-based 3D experiences.

Captured rooms become editable testing environments

A reconstructed room becomes useful for interaction testing only when its objects are editable and physically plausible. Chirag Majithia, from the Isaac engineering applications team at NVIDIA, directed Astra to turn stereo camera captures into an editable OpenUSD studio. The workflow combined PyCuSFM, FoundationStereo, and nvblox for reconstruction, with user review guiding object selection and placement. Astra assembled generated and Blender-authored assets and used USD Content Agents to configure how objects move and interact in simulation. Isaac Sim tests guided collision and contact revisions for doors and drawers. The studio connects captured geometry to interaction testing, making gaps and object behavior easier to inspect. USD Content Agents can be explored for configuring object behavior in OpenUSD scenes.

What developers should do next

Developers building simulation apps should treat AI agents as executors of a plan, not as replacements for validation. The examples share a pattern: define the target scene, connect the right Omniverse libraries, ask the agent to wire assets and code, then measure outputs with physics, sensors, or task-specific tests. Those working on humanoid control, autonomous-driving comparisons, digital twins, robot skill testing, disassembly, browser visualization, or room reconstruction can use these projects as templates. Developers who need fully governed production pipelines should still add their own review gates, asset validation, and metric checks before trusting generated scenes. NVIDIA also invites readers to check back for new examples from teams and developers across the ecosystem.