Coding with AI

Video-Coder Strengthens Agentic Coding for Generating Videos

Video-Coder trains code agents to build 3D draft scenes, improving video structure and motion with open data, models, and gains of 7 and 24 points.

Key Takeaways

  • Video-Coder is a code-driven video generation agent that writes and executes code to build dynamic 3D “Draft Video” scenes, giving downstream generators explicit spatial structure and motion signals.
  • It separates video creation into two stages: the coding agent handles scene construction and motion control, while the downstream generator produces the final visual result.
  • The agent is trained using dedicated datasets, the Video-Coder Harness, supervised fine-tuning, and agentic reinforcement learning with dual execution-based and video-based rewards.
  • Experiments show Video-Coder improves Minimax-H3 by about 7 points on PhyGenBench and 24 points on VBench-2.0.
  • The data, models, and code are fully open-sourced, offering an open foundation for coding-agent-based video generation.

Video-Coder Lifts Minimax-H3 by About 7 and 24 Points

Meta description: Video-Coder trains a code-driven video agent to build dynamic 3D draft scenes, improving Minimax-H3 by about 7 and 24 points on PhyGenBench and VBench-2.0.

Tags: Video-Coder, video generation, coding agents, 3D scenes, open models

Featured image: abstract friendly illustration of a code editor shaping a dynamic 3D video scene, no real people, no logos, no text

A paper introduces Video-Coder, a code-driven video generation agent designed to improve how generated videos handle spatial structure, motion dynamics, and controllable scene composition. The central idea is to let the agent autonomously write and execute code, then build a dynamic 3D scene called Draft Video. That scene gives downstream video generators explicit structure and motion signals, rather than leaving those signals implicit.

The paper describes Video-Coder as the first attempt to train such a code-driven agent for video generation. It says recent years have shown remarkable progress in visual quality and temporal consistency, while precise spatial structure, coherent motion dynamics, and controllable scene composition remain challenging. Video-Coder addresses that gap by turning part of video construction into code: the agent builds a 3D scene, renders it, and passes structure and motion signals to a downstream generator.

From Prompt to Draft Video

The workflow starts with prompts and uses an intermediate scene to guide final video creation. Video-Coder writes and executes code to construct a dynamic 3D scene, labeled Draft Video in the paper. The Draft Video is not described as the final video; it is a structured source of motion and spatial signals. Downstream video generators then use those signals.

This design separates two hard problems. The coding agent handles scene construction and motion control. The downstream generator handles the final visual result. By making the first stage explicit, the method gives the second stage a more structured input.

Datasets, Harness, and Two-Stage Training

To support the paradigm, the paper introduces a dedicated data pipeline and two datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k. They contain diverse prompts and ground-truth reference videos. The pipeline provides training material for both supervised and reinforcement learning inside the code-and-rendering setting.

The training environment is called the Video-Coder Harness. It is described as a unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. The agent can therefore practice in the same kind of setting used for inference.

Video-Coder is trained with SFT followed by harness-native agentic reinforcement learning. This reinforcement-learning stage uses dual reward feedback, combining execution-based and video-based rewards. The paper also introduces an annealing training strategy that gradually reduces the execution reward over time, allowing the model to rely more on the video reward in later training stages. The effect is to move the agent from producing executable code toward producing code that supports better final video results.

Measured Gains and Open Release

Experiments show that Video-Coder achieves significant gains, improving Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. These results connect the Draft Video pipeline to measurable improvements in a downstream video generator.

The paper hopes the work can serve as an open foundation for coding agents in video generation. The data, models, and code are fully open-sourced. For researchers, that makes the approach available to inspect, reproduce, and extend; for the field, it highlights code execution and 3D scene construction as possible control signals for future video generators.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.