Tool Reviews

NVIDIA has launched new open Cosmos 3 models for interactive world simulation.

NVIDIA adds four open-weight Cosmos 3 Nano checkpoints for driving, robotics, interactive streaming simulation, and 16B post-training workflows.

NVIDIA has launched new open Cosmos 3 models for interactive world simulation.

Key Takeaways

  • NVIDIA released four open-weight Cosmos 3 Nano checkpoints with 16B trainable parameters for driving, robotics, and streaming world simulation under OpenMDW1.1.
  • Cosmos3-Nano-Transfer-Auto generates offline synthetic driving data with synchronized video across up to 11 cameras and joint RGB-LiDAR generation controlled by World Scenario Maps and text prompts.
  • Cosmos3-Nano-Sim-Auto is a causally distilled streaming model that produces fast, successive camera and LiDAR chunks for interactive closed-loop driving rollouts.
  • Cosmos3-Nano-Sim-Transfer converts physics-simulator states and spatial controls such as depth, segmentation, edges, and blur into RGB observations for policies.
  • Cosmos3-Nano-Sim-Bimanual predicts visual environment changes from bimanual robot actions, supporting interactive two-arm simulation for YAM and AgiBot embodiments.

NVIDIA Releases Four Cosmos 3 Checkpoints for Driving and Robotics

NVIDIA has launched new open Cosmos 3 models for interactive world simulation.

Meta Description: NVIDIA adds four open-weight Cosmos 3 Nano checkpoints for driving, robotics, streaming world simulation, and 16B post-training workflows for developers.

Tags: NVIDIA Cosmos 3, World Model, Autonomous Driving, Robotics Simulation, Open Weights

Featured image: abstract, friendly illustration of a simulated road scene with camera views, a robot arm, and streaming video chunks, no real people, no logos, no text.

NVIDIA’s Cosmos world model development platform has added four NVIDIA Cosmos 3 checkpoints to the Cosmos 3 model family. The new open-weight checkpoints are designed to generate interactive world simulation and multi-sensor, multi-view driving scenes. They are Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer, and Cosmos3-Nano-Sim-Bimanual. Each is a post-trained example of Cosmos 3 Nano and has 16B trainable parameters. The checkpoints use OpenMDW1.1, which permits commercial and non-commercial use under its terms.

Four Checkpoints Address Different Simulation Needs

The release separates world-simulation work into driving, simulator-to-video rendering, and robot-action prediction. For robotics, the models need environments that can respond across changing scenes and viewpoints. Cosmos3-Nano-Sim-Bimanual predicts visual consequences from actions, while Cosmos3-Nano-Sim-Transfer turns a physics simulator’s controls into generated observations.

For autonomous driving, appearance and geometry must agree. Synchronized camera and LiDAR generation connects visual observations with 3D sensing. Scene and traffic-signal controls let researchers vary road layouts and signal states, then study how a driving policy responds.

Each observation can shape the next action. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together. The release also presents open weights and post-training recipes as a shared starting point for adapting models, comparing methods, and contributing improvements. NVIDIA’s stated vision is to help researchers and developers create richer, more responsive simulated worlds.

The work remains hard. The source notes that the next step is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short.

Transfer-Auto Builds Offline Driving Data

Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data generation. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB-LiDAR generation. Per-camera World Scenario Map controls, abbreviated as WSM, specify scene structure and traffic motion, while text prompts guide appearance.

Developers can vary environmental conditions to expand training and evaluation coverage. Optional initial RGB images can anchor scene appearance. Joint LiDAR generation additionally uses aligned HD-map range-map controls.

Transfer-Auto also serves as the foundation for adapting Cosmos to a different sensor rig and driving domain. The source recommends post-training it on local data to accommodate camera layout, LiDAR configuration, and target environment. The adapted model can then be used directly for offline data generation, or it can undergo causal training and few-step distillation for fast streaming simulation. That is the development path used to create Cosmos3-Nano-Sim-Auto.

NVIDIA provides an open-skill inference path for Cosmos3-Nano-Transfer-Auto, described as “Run inference with Cosmos3-Nano-Transfer-Auto.” A notebook path named Transfer-Auto is also listed for users who prefer notebooks.

Sim-Auto Streams Camera and LiDAR Observations

Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps. It uses generated history and updated driving controls to support fast streaming and long, interactive rollouts.

The model expects per-camera captions and synchronized WSM controls. Optional initial RGB images can be added. For LiDAR generation, users must include HD-Map range controls aligned with the camera controls. The distilled model generates successive chunks using its rolling history.

Closed-loop interaction requires a simulation runtime and WSM renderer to update controls from driving inputs. For a new sensor rig or domain, the source says to adapt Transfer-Auto first, then distill it into Sim-Auto.

The checkpoint also has an open-skill inference path, “Run inference with Cosmos3-Nano-Sim-Auto,” and a notebook path named Sim-Auto.

Sim-Transfer Turns Simulator State into RGB

Cosmos3-Nano-Sim-Transfer is built for cases where a simulator already computes motion, contacts, and physical state. It uses spatial controls and a text prompt to render color observations that a policy can consume.

The flow is: action, simulator update, spatial controls, generated RGB, then the next policy action. The control types include depth, which represents relative distance from the camera; segmentation, which defines regions and object boundaries; edges, which show scene contours; and blur, which gives coarse appearance and layout. Text guides appearance, while physics and ground-truth state stay with the simulator.

The source lists sample demonstrations such as depth control to video and segmentation control to video. NVIDIA provides an open-skill inference path, “Run inference with Cosmos3-Nano-Sim-Transfer,” and a notebook path named Transfer.

Sim-Bimanual Predicts Video from Robot Actions

Cosmos3-Nano-Sim-Bimanual simulates how an environment evolves in response to a bimanual robot’s actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks.

Users can start with an image, move the robot or camera, and generate the next observations. One control sequence can switch between viewpoint changes and manipulation. The mixed-action example uses 59 control values, padded to 64 during training. Its schema defines coordinate frames, rotations, gripper calibration, validity masks, and control domains. Users are advised to follow those conventions so each value describes the intended motion.

Timing alignment matters: T transitions need T+1 observations. Training transforms raw captures into the model’s action representation, while released inference actions are already normalized. The source distinguishes this checkpoint from Sim-Transfer: Sim-Transfer generates visuals from a physics simulator’s updated state, while Sim-Bimanual generates the next observation directly from the scene and robot actions. A data contract and action schema are also referenced.

The open-skill inference path is “Run inference with Cosmos3-Nano-Sim-Bimanual,” and the notebook path is named Bimanual.

Streaming Generation Uses Causal Chunks and Latents

Cosmos 3 uses a Mixture-of-Transformers backbone that connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls, and generated history. The loop is: controls and history, generate chunk, decode images, update history.

Denoising refines noisy, compressed video representations into a segment. Temporal-causal attention lets the model read available history and controls while blocking future target observations. A key/value cache retains attention information for the next prediction.

The model operates on compressed video representations called latents. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk, while the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often.

Requirements and Runtime Notes

The published setup requires Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The vLLM-Omni setup was tested on one GB200, and the source says this is not a minimum-memory specification. The server downloads weights, while clients fetch sample inputs.

Four-Stage Training Recipe and Adaptation Path

The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. Users are advised to keep full training checkpoints to resume, while exported inference weights omit optimizer and other training state.

Stage 0 is bidirectional adaptation, where the model learns the domain and its control-to-video relationships from complete recorded clips.

Stage 1 is teacher forcing, where the model predicts each chunk from earlier recorded observations and available controls, with the clean answer and future targets masked out.

Stage 2 is consistency distillation, where a frozen teacher trains a student that generates each chunk in four denoising steps.

Stage 3 is self-forcing, where the model trains on the student’s generated history. A frozen teacher and an auxiliary score model supply the distribution-matching signal.

The final stage matters because a slightly mispredicted object becomes context for later frames, so errors can compound. Self-forcing exposes training to that generated history. Distillation and caching reduce the cost of producing each observation.

The adaptation guidance splits by checkpoint. Transfer-Auto can be post-trained on driving data to adapt to a sensor rig and domain, then used for offline synthetic data generation. Sim-Auto is created by applying causal training and distillation to an adapted Transfer-Auto model for fast streaming simulation. Sim-Transfer can be post-trained for a specific environment with local data preparation, stage configurations, and resume support. Sim-Bimanual can be post-trained for a target embodiment with data validation, stage configurations, checkpoint handoffs, and export.

NVIDIA recommends giving a coding agent the skill and dataset location, following environment checks, validating a sample, and testing training and restart before a longer run. Quality should be evaluated on held-out episodes, because a successful short run verifies execution, not convergence.

Resources and Next Steps

The release points to downloadable checkpoints, a technical report, and demos. Additional resources include a Cosmos Labs livestream replay and a technical blog. The broader message is that these checkpoints give researchers and developers open, post-trained starting points for driving simulation, streaming world models, simulator rendering, and robot-action prediction, while leaving evaluation, limits, and improvement as the next work ahead.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.