RynnVLA-Latent scales robotic manipulation foundation models by grounding latent actions in three-dimensional space.
RynnLAM creates latent action targets from 30,100 hours of human, robot, and simulation video; RynnVLA-Latent transfers them into real robot control.
Key Takeaways
- RynnVLA-Latent learns robot manipulation policies from 30.1K hours of mixed video instead of requiring native robot-action labels.
- RynnLAM provides latent-action supervision by using residual feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling.
- The method follows a two-stage path: first learn shared 3D-grounded transition representations from video, then adapt them to specific robots through embodiment-specific post-training.
- Validation covers representation quality and policy performance, including frozen-representation probes and manipulation tests across simulation and real-world platforms.
- The approach scales with more video diversity and may reduce dependence on expensive, narrowly collected robot-action datasets.
Meta description: RynnLAM and RynnVLA-Latent learn latent actions from 30.1K hours of video, then transfer robot control across simulation and real-world platforms.
Tags: RynnLAM, RynnVLA-Latent, latent-action supervision, video pretraining, robot manipulation
Featured image: abstract friendly illustration of video frames, camera motion arrows, and robot arms connected by soft latent-action lines.
For readers following robot learning, the core fact is straightforward: RynnVLA-Latent is built to create a robot-control policy from 30.1K hours of mixed video, while RynnLAM provides the supervision that lets video stand in for missing robot actions. Instead of requiring native robot-action labels, the method constructs continuous transition representations that capture how visual states change over time. The contribution is a two-step path—first learn latent-action targets from human, robot, and simulation video, then adapt the pretrained policy to a specific robot through embodiment-specific post-training.
Why labeled robot actions are the bottleneck
Robot policies usually need more than pictures. To learn how a manipulation works, a model often needs the exact motor commands that caused the robot to move, grasp, or reorient an object. That requirement makes robot data expensive and narrow, because each clip is tied to a particular machine, sensor setup, and control interface.
Video offers a much broader source of motion, but raw video is not a clean substitute for robot actions. Human videos show rich manipulation without robot motor commands. Simulation clips can contain motion that looks realistic but may not match real-world physics or camera behavior. Robot videos are closer to deployment, yet they are still limited by the number of machines that can collect them. The contribution addresses this gap by learning an internal, action-like signal from video first, then connecting that signal to a robot’s own controls later.
RynnLAM builds the latent-action bridge
RynnLAM is the supervision layer that turns video into usable training targets. It combines residual feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling to construct continuous transition representations without native robot-action labels.
Residual feature prediction helps the model focus on what changes between visual states, rather than treating each frame as an isolated image. Camera-compensated 3D-flow supervision uses motion in the scene while adjusting for the fact that the camera itself may be moving. Camera-motion decoupling further separates camera movement from object and body movement.
Together, these components form a more stable internal representation of transitions. That matters because a robot does not need to know the original motor command from the video source. It needs a representation that predicts how the world changes and can later be mapped into its own control space.
RynnVLA-Latent scales across heterogeneous video
The second system, RynnVLA-Latent, uses those representations as shared latent-action targets. It learns from 30.1K hours of heterogeneous video, a training pool that includes human video, robot video, and simulation video. The method does not assume all three sources can be used as if they were identical robot datasets. Instead, it extracts common transition patterns that can be reused across different data sources.
This is where the pretraining becomes more scalable. Human video can teach broad object manipulation patterns. Robot video can anchor the learning to machine-specific movement. Simulation video can expand the variety of scenes and interactions. RynnVLA-Latent combines these sources into latent-action targets, then transfers the pretrained policy to native robot controls through embodiment-specific post-training.
Embodiment-specific post-training is the practical handoff. A policy trained on video cannot automatically drive a robot’s joints, grippers, wheels, or actuators without adaptation. The pretrained latent-action representation provides a shared starting point, while post-training aligns that representation with the sensors, command format, and mechanical constraints of a particular robot.
Validation checks representation and policy
The contribution is tested at two levels: representation quality and policy performance. Frozen-representation probes are used to examine control predictability. In this setup, the learned representation is kept fixed while a smaller probe is trained to see whether the representation already contains useful control-related information. This helps separate the value of the representation itself from the tuning that happens later.
The method is also evaluated through manipulation performance across simulation and real-world platforms. The reported result is competitive manipulation performance, which suggests the latent-action pathway is not only useful as an internal representation but also as a practical training bridge for control.
Another important signal is scaling behavior. The contribution points to improvements as latent-action pretraining data increases. That trend matters because it suggests the approach benefits from more video diversity rather than relying only on carefully matched robot-action labels.
What readers should watch next
The practical takeaway is that video-based robot training is getting closer to a reusable pretraining pipeline. If a robot can learn shared transition patterns from 30.1K hours of human, robot, and simulation video, teams may need less hand-labeled robot-action data before adapting a model to a new machine.
For readers building or evaluating robot systems, the next question is how far the recipe travels. How well does the latent-action representation hold up across different camera setups, different robot bodies, and different manipulation tasks? How much embodiment-specific post-training is still required? The reported validation across simulation and real-world platforms makes the approach worth following, but the result remains a research contribution rather than a finished product.
Comments (0)