Tool Reviews

The second version of the Common Weakness Enumeration Benchmark offers broader coverage, improved quality, and greater transparency.

Collinear AI's CWE-bench v2 broadens coverage with 100 public tasks, programmatic scoring, a held-out set, and AI security results.

The second version of the Common Weakness Enumeration Benchmark offers broader coverage, improved quality, and greater transparency.

Key Takeaways

  • Collinear AI released CWE-bench v2, a public benchmark with 100 security tasks that test whether AI agents can find and fix vulnerabilities in real open-source code while preserving existing functionality.
  • The benchmark offers broader coverage across 91 CWEs, fully programmatic and reproducible scoring, curated task-quality checks, and a held-out set for tracking progress on unseen tasks.
  • Results for 11 models show Gemini 4 Argon leading with 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5, while 95 tasks are solved by at least one model but the top model still fails 19% of tasks on every attempt.
  • Cost is an important factor: DeepSeek V4.1 Flash reaches 57.0% pass@1 at only $0.04 per rollout, while stronger models such as Claude Opus 5.5 and Gemini 4 Argon require much higher rollout costs.
  • The tasks run through Harbor on Hugging Face with Dockerized environments and deterministic verifiers, and they can also support reinforcement-learning-style training for defensive AI coding agents.

Meta description: Collinear AI opens CWE-bench v2 with 100 public security tasks, programmatic scoring, held-out set, and results for 11 models including Gemini and Claude.

The second version of the Common Weakness Enumeration Benchmark offers broader coverage, improved quality, and greater t

Tags: CWE-bench v2, AI security benchmark, Collinear AI, Harbor, model evaluation

Featured image: Abstract friendly illustration of a digital lock and code puzzle representing CWE-bench v2 security tasks, with no people, no logos, and no text.

Collinear AI has opened CWE-bench v2, a public benchmark that lets researchers and model builders test whether AI agents can find and fix security weaknesses in real open-source code. The release gives anyone 100 tasks that can be run and scored programmatically on the Hub, with no extra cost to reproduce results.

What the benchmark asks models to do

CWE-bench v2 keeps the same task setup as before. An agent receives a checkout of a real open-source repository and a short instruction. Its job is to find and fix a security weakness without breaking existing functionality. The benchmark is meant to test the limits of frontier cyber capabilities, a phrase used here for advanced security-related model abilities.

The public set covers 91 different CWEs across a wide range of vulnerability classes. The goal is exhaustive coverage: a defender must anticipate the ways an attacker could get in and succeed every time. A model that does well here must understand multiple weakness surfaces, not just recognize a familiar pattern.

Scoring is fully programmatic, so results are deterministic and reproducible. The checks were curated with LLM judge panels to push them toward hardness and fairness. Collinear AI also maintains a held-out set of tasks that no model has seen, so it can continue tracking progress outside the public tasks. To evaluate a model on the held-out set, contact [email protected].

How the task quality is checked

For every task, Collinear AI checked that several conditions were true. The instruction must read like a realistic instruction, not an evaluation setting. Every graded behaviour must be inferable from the instruction. The verifier must check correct behaviour rather than one specific structure of a fix. The starter code must fail, while the reference fix must pass. The original functionality must still work after the new fix. The verifier must also fail memorized solutions, such as the known upstream patch.

Each task also received a manual review. Open-sourcing the set is part of the same effort to improve quality, and the company welcomes feedback on any task and contributions to the public set.

What the results show so far

Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open-weight models and updates on its own. Labs and the community can add a result by opening a pull request on the model's repository with a results file. For the full comparison, including API-only models such as Gemini, Claude and GPT, the CWE-bench leaderboard Space adds new models through a pull request on the Space.

Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5, which scores 67.3% pass@1 and 78.0% pass@4. Here, pass@1 means success on one attempt, while pass@4 means success within four attempts.

The second version of the Common Weakness Enumeration Benchmark offers broader coverage, improved quality, and greater t

95 tasks are solved by at least one model, which suggests the tasks are solvable. But even the leader fails 19% of the set on every attempt, so the long tail is far from solved. The public set, the grader and the harness are open, so anyone can run a model and see where it stands.

Cost is part of the picture. DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier, meaning they show the strongest reported trade-off between pass@1 performance and rollout cost. DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, where a rollout is one model run on a task. Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92.

How to run it from the Hub

Each task is a self-contained Harbor task, with a Dockerized environment, an instruction and a deterministic verifier. The dataset ships a registry.json file, so Harbor can run the whole benchmark straight from the Hugging Face repo without cloning anything yourself.

A command with -k 4 runs 4 attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated, so users must accept its access conditions on the dataset page first. Harbor fetches the tasks with git, so users also need to save their Hugging Face token to git's credential helper once.

The tasks are also tagged rl-environment. That means the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them. For teams building AI coding agents, CWE-bench v2 now offers a public, reproducible way to measure security-fixing ability while still leaving a hard long tail to solve.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.