Tool Reviews

Reflex enables visually grounded reactive control for humanoid agents.

Reflex splits humanoid box catching into state control, dynamics inference, and vision, reaching 85.7 percent in simulation and 65 percent on robots.

Reflex enables visually grounded reactive control for humanoid agents.

Key Takeaways

  • Reflex enables a humanoid robot to catch thrown boxes using only egocentric RGB-D vision, achieving 85.7% success in simulation and 65% on a real Unitree G1 at 2.5 m.
  • The framework uses three trainable stages: whole-body RL control, object-dynamics inference from delayed observations, and visual recovery of the same dynamics representation from RGB-D input.
  • Temporal visual history is critical: removing it causes large performance drops, showing that catching requires tracking motion over multiple frames rather than reacting to a single image.
  • The method transfers to physical hardware but remains weaker at longer distances, reaching 40% success at 5 m on the real robot.
  • The main contribution is a modular chain from vision to dynamics prediction to whole-body control, offering a scalable path toward visually grounded reactive humanoid behavior.

For readers following humanoid robotics, the key result is Reflex, a framework that lets a humanoid catch a thrown box using only egocentric RGB-D vision, achieving 85.7% success in simulation and 65% success on a real Unitree G1 at 2.5 m. The work targets a gap between human reflexes and robot control: people can return a tennis shot, catch a falling cup, step aside from a passing cart, or receive a box tossed during truck unloading by predicting motion and reacting with the whole body in about a second.

Reflex enables visually grounded reactive control for humanoid agents.

Why catching a box is a useful test

Catching thrown boxes is chosen as representative because it forces three abilities at once: whole-body coordination, real-time response to object dynamics, and perception grounded in egocentric vision. Existing systems cover only part of this stack, often relying on end-effectors, motion capture, or marked objects. By requiring the robot to see, predict, and move as one task, box catching exposes the weakness of approaches that treat perception and control as separate add-ons.

How Reflex turns a hard problem into three learnable stages

Reflex uses a three-stage framework. First, a state-based policy learns whole-body catching with reinforcement learning. At this stage, the controller is trained from direct state information, so the system can discover how to move the entire humanoid body rather than only the hand or end-effector.

Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. This stage teaches the model to reconstruct object motion even when sensory data does not arrive in a clean, full sequence.

Third, an egocentric RGB-D encoder learns to recover the same dynamics representation directly from visual observations while keeping the whole-body controller fixed. The result is a compact representation of object dynamics that links the stages: visual input, motion prediction, and whole-body control can be trained separately, then connected without retraining the controller from scratch.

Reflex enables visually grounded reactive control for humanoid agents.

What the simulation numbers show

In simulation, Reflex reaches 85.7% catch success when using egocentric RGB-D input. For comparison, the system reaches 87.9% when given privileged box observations, meaning access to the object state that a real robot would not normally have. The gap between 85.7% and 87.9% suggests the visual pipeline is close to the performance ceiling available from direct object state, at least in simulation.

Temporal visual history matters. Removing it reduces success by 35.6 points on centered throws and by 39.8 points on laterally offset throws. That drop is large because a moving box cannot be handled from a single frame; the robot needs to track how the object’s position and motion change over time.

Real Unitree G1 results are lower but still concrete

On a real Unitree G1, Reflex catches human-thrown boxes with 65% success at 2.5 m and 40% success at 5 m. The framework reaches 76% of human catchers’ success at 2.5 m and 59% of human catchers’ success at 5 m. These numbers show that the method transfers from simulation to physical hardware, while also marking a clear performance gap at longer distances.

Reflex enables visually grounded reactive control for humanoid agents.

Why this matters for humanoid developers

The practical value is in the training structure. By separating whole-body control, dynamics inference, and visual recovery, Reflex gives each capability a direct training signal and makes visual learning substantially more scalable. For developers of humanoid systems, the lesson is that reactive whole-body control may depend less on one monolithic model and more on a chain that converts egocentric RGB-D observations into a shared object-dynamics representation. The open question is how much the gap between 65% at 2.5 m and 40% at 5 m can be closed.

Meta Description
Reflex lets humanoids catch thrown boxes using egocentric RGB-D vision, reaching 85.7 percent in simulation and 65 percent on a Unitree G1 at 2.5 m.

Tags
Reflex, humanoid robots, visual control, reinforcement learning, RGB-D

Featured Image
A humanoid robot catching a thrown box, abstract friendly illustration, no real people, no logos, no text.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.