Tool Reviews

VibeVideoBench evaluates multimodal agents on code-driven video generation.

VibeVideoBench tests 9 AI agents on 239 code-driven video tasks, requiring finished videos with audio in an hour; best systems still miss delivery.

VibeVideoBench evaluates multimodal agents on code-driven video generation.

Key Takeaways

  • VibeVideoBench is a new benchmark for testing general multimodal AI agents on code-driven video generation, requiring a finished MP4 with audio produced entirely from code within one hour across 239 open-domain tasks.
  • Agents must autonomously plan, render, synchronize, and deliver videos without image or video generation models, making production scheduling, file completion, and timing as important as visual quality.
  • GPT-6 Astra and Claude Opus 5.5 ranked highest among 9 frontier systems, but even the best agent fully met core requirements on only 76.3% of tasks, showing the field is promising but far from solved.
  • The evaluation uses delivery checks, around 13,000 task-specific rubric checks, and quality reviews by vision-language judge panels and blind human raters, separating technical completion from subjective quality.
  • A major bottleneck is pipeline management: many failures occur when agents start final renders too late, while inspecting their own frames does not reliably predict the quality of the finished video.

Meta description: VibeVideoBench tests 9 frontier AI agents on 239 open-domain tasks requiring finished MP4 videos with sound generated entirely from code within one hour.

VibeVideoBench evaluates multimodal agents on code-driven video generation.

Tags: AI agents, code-driven video, benchmark, multimodal, video rendering

Featured image: Abstract, friendly illustration of code blocks assembling into a video timeline with waveforms and charts, no real people, no logos, no text.

VibeVideoBench is a new benchmark that asks general multimodal AI agents to produce a finished video entirely from code, including audio, within one hour. It tests 9 frontier agent systems across 239 self-contained tasks written in English and Chinese. The central question is not whether an agent can write a script, but whether it can plan, render, synchronize, and deliver a playable MP4 file without relying on an image or video generation model.

The project describes it as, to its knowledge, the first benchmark of general multimodal agents on open-domain code-driven video generation with audio. The idea is simple: instead of asking a model to imagine moving pixels, the agent writes instructions that render every frame and synthesize every sound. In principle, this can make formulas, charts, and narration align exactly with timed on-screen events.

What VibeVideoBench asks agents to do

The tasks cover 12 categories, including data stories, music-driven pieces, classroom lessons, paper explainers, games, build-and-process videos, and other open-domain assignments. Each task is self-contained, so the agent must infer the deliverable and produce it from scratch.

Agents run fully autonomously in their native harness. They operate in a sandbox with no image or video generation model and only text-to-speech capability for audio. Their output must be a finished MP4 video with sound within the one-hour limit.

VibeVideoBench evaluates multimodal agents on code-driven video generation.

That constraint changes the nature of the evaluation. An agent cannot simply generate a visually pleasing clip and call it done. It must also manage file creation, audio timing, rendering time, and delivery format under a fixed deadline.

How the benchmark is scored

Evaluation is hierarchical and evidence-based. Level 1 program checks verify whether the delivery specification is met. Level 2 includes about 13,000 task-specific rubric checks, each traced to the requirement it enforces. Level 3 reviews five quality dimensions anchored to the task.

The project uses three independent vision-language model judge panels. Audio checks are given only to judges that can hear, helping ensure that audio requirements are evaluated with hearing-capable models.

This design separates technical completion from quality review. A video may exist and still fail because it misses a required chart, narration segment, timing rule, or output format. Conversely, a high-quality-looking result can lose points if the delivery specification is not satisfied.

What the 9 systems reveal

Across 2,151 scored rollouts from 9 frontier agent systems, GPT-6 Astra and Claude Opus 5.5 rank first and second when judged by both the judge panels and blind human raters. The agreement suggests the benchmark can distinguish system-level performance from subjective taste alone.

Still, the best system meets every core requirement on only 76.3% of tasks. That number is the benchmark's clearest signal: code-driven video generation is promising, but it is far from solved. An agent can be the leader and still fail a quarter of the basic delivery expectations.

The ranking also shows that delivery is a second axis of performance. When missing videos are counted as zero, the order changes. Gemini 4 Argon delivers 77% of its videos and drops from fourth to seventh. A system that can create good frames is not necessarily a system that can finish the task.

VibeVideoBench evaluates multimodal agents on code-driven video generation.

Planning the hour is part of the test

Of 224 runs with no playable video, 169 are final renders that started too late, ran too slowly, or never started. The benchmark results point to how agents plan their hour as the underlying problem.

This matters because video production has a different failure pattern from text generation. An agent can spend most of the available time writing code, debugging assets, or generating intermediate frames, then leave too little time for the final render. The one-hour limit turns scheduling into a core capability.

The results also show that looking at the work is not enough. Every system keeps its own rendering stack, and 95% of runs inspect their own frames. Yet how much they inspect does not predict quality, with a correlation coefficient of ρ = -0.07. The weak correlation suggests that visual inspection alone does not explain final quality.

Judge reliability and what readers should take away

The judge panels rank systems in a way that closely matches human raters, with ρ = 0.98. However, they agree on only 73% of individual video pairs. That makes the benchmark useful for comparing agent systems, but less reliable for declaring a single generated video objectively better than another.

For readers, the main takeaway is that AI agents are moving toward automated technical video assembly, especially for content with structured elements such as charts, formulas, lessons, and timed narration. The benchmark also highlights the gap between impressive code execution and dependable finished-product delivery. The practical bottleneck is not just visual quality; it is whether the agent can manage the whole production pipeline before the clock runs out.

The project page is available at https://vibevideobench.github.io/.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.