🔍 Read the full analysis: CWE-bench V2: A Broader, Clearer Benchmark For AI on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks from open-source repositories, spanning 91 CWE categories. Its reported leader scored 67.8% pass@1, but no evaluated model passed every task on every attempt; the release does not establish how well benchmark scores predict production security work.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from real open-source repositories, along with its grader and evaluation harness, as detailed in the original analysis. The release reports that the leading evaluated model scored 67.8% pass@1 across tasks spanning 91 CWE weakness categories, offering researchers a shared test while showing that substantial gaps remain.
Each task gives an AI coding agent a checkout of a real repository and a short instruction to find and fix a security weakness without breaking existing functionality. Tasks are packaged as self-contained Harbor environments, and the dataset includes a registry file for running them through Hugging Face. The authors describe scoring as fully programmatic, using curated checks intended to make runs repeatable.
The release reports evaluations of 11 models. Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%, respectively. The authors say at least one evaluated model solved 95 of the 100 tasks; that figure does not mean a single model solved all 95. They also report that the leader failed 19% of the task set on every attempt.
The authors compare cost as well as scores. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout; Claude Opus 5.5 scored about 10 percentage points higher at $1.83, while Gemini 4 Argon scored slightly higher at $3.92. These are figures from the release, and the source notes that costs can vary with model pricing and evaluation setup.
What the Scores Say About Repair Agents
The benchmark gives developers and researchers a common set of tasks for comparing whether coding agents can identify weaknesses, make repairs and preserve expected behavior. Its spread across 91 CWE categories is meant to test more than one familiar vulnerability pattern, while the public checks provide a way to inspect and repeat evaluations on the released set.
The results also put limits on what a high score can imply. The reported leader remains below 70% on pass@1, and the release says it fails a share of tasks on every attempt. A score measures performance on this curated benchmark; it does not show that an agent can safely handle every repository, threat model or live security review. For teams weighing agents for security work, the results are a comparison point, not evidence of autonomous readiness.
Products Worth Considering
Bose QuietComfort Ultra Headphones (2nd Gen), Wireless Bluetooth Noise-Cancelling Over-Ear Headphones, Black
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
Anker SOLIX C1000 Gen 2 Portable Power Station
As an affiliate, we earn on qualifying purchases.
How Version 2 Builds Its Test Set
Version 2 expands the publicly available benchmark to 100 tasks while retaining the prior task format. According to the authors, they reviewed tasks by hand for realistic instructions and checks that assess correct behavior rather than a single code structure. They also report verifying that starter code fails, a reference fix passes, existing functionality is preserved, and checks reject memorized solutions such as known upstream patches.
The release says LLM judge panels helped shape programmatic checks for difficulty and fairness. It also describes a separate held-out set for tracking progress on tasks not seen by evaluated models. That set is not public; interested teams must contact Collinear AI for access. Public results are tracked through two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models. Contributors may submit results through pull requests, and users must accept dataset access conditions before running the tasks.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The supplied release details do not provide enough information to independently assess every comparison, including exact model configurations and the conditions used for each cost estimate. Results may change as new submissions arrive. The benchmark describes task review and verifier checks, but outside readers cannot inspect the private held-out tasks or reproduce evaluations on them.
It also remains unclear how strongly performance on the 100 public tasks predicts results in production security work. Curated tasks cannot represent every software environment, vulnerability or deployment condition. The release does not establish that a model with a higher benchmark score will perform better in live security reviews, or that the reported scores will remain stable across configurations and independent runs.
Public Runs and Further Submissions
Researchers and developers can run models against the public tasks using Harbor after accepting the dataset conditions and setting up Hugging Face access. Collinear AI invites task feedback and contributions, while leaderboard results can be submitted through pull requests. The release says teams interested in evaluating models on the held-out set can contact the company.
Further submissions may update the public comparisons and expose issues with task design or grading. Independent reruns, clearer reporting of model configurations and cost assumptions, and comparisons with real security-review outcomes would help readers judge how far benchmark scores generalize. The source gives no schedule for those developments.
Key Questions
What is CWE-bench v2?
It is a public benchmark from Collinear AI with 100 tasks that ask coding agents to fix security weaknesses in real open-source repositories while keeping existing functionality intact.
Which model had the highest reported score?
The release lists Gemini 4 Argon first, at 67.8% pass@1 and 81.0% pass@4. These are reported benchmark results, not proof of performance across all security work.
Did one model solve 95 of the 100 tasks?
The source says at least one evaluated model solved 95 tasks, but clarifies that this does not mean one model solved all 95. It also reports that the leading model failed 19% of the set on every attempt.
Can anyone reproduce the evaluations?
The public tasks, grader and evaluation harness are available for runs, but users must accept the dataset access conditions and set up Hugging Face access. The separate held-out tasks are private, so outside users cannot directly reproduce results on that set.
Do the scores show that AI agents are ready for production security work?
No. The release reports results on a curated benchmark, and it is unclear how well those results predict performance across production repositories, threat models and live security reviews.
Primary source: Hugging Face · via ThorstenMeyerAI.com
