🔍 Read the full analysis: What’s New In CWE-bench V2 For AI Coverage And Transparency? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Collinear AI has released CWE-bench v2, a benchmark of 100 security-fix tasks from open-source repositories, along with its grader and evaluation harness. The release reports a leading pass@1 score of 67.8% across 11 evaluated models, while the top model still failed some tasks on every attempt.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from real open-source repositories, together with its grader and evaluation harness, as detailed in the original analysis. The release reports results for 11 models: Gemini 4 Argon led at 67.8% pass@1, a measure of success on a single attempt, as researchers gain a shared way to test coding agents on security repairs.
The tasks cover 91 CWE weakness categories. In each one, an agent receives a checkout of a real repository and a short instruction to identify and fix a security weakness without breaking existing functionality. The authors describe the scoring as fully programmatic, using curated checks intended to make runs repeatable. Tasks are packaged as self-contained Harbor environments, with a registry file for running them from Hugging Face. Users must accept the dataset’s access conditions before downloading it.
In the reported results, Gemini 4 Argon scored 67.8% pass@1 and 81.0% pass@4; Claude Opus 5.5 scored 67.3% and 78.0%, respectively. Pass@4 measures whether a task is solved within four attempts under the evaluation setup. The release also says at least one model solved 95 of the 100 tasks; that figure does not mean a single model solved all 95. The top model failed 19% of tasks on every attempt, according to the announcement.
The release compares results with estimated costs. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout. Claude Opus 5.5 scored about ten percentage points higher at a reported $1.83, while Gemini 4 Argon scored slightly higher at $3.92. Those are the announcement’s figures; costs may vary with model pricing and evaluation conditions, which are not fully detailed in the supplied results.
What Security-Fix Scores Can Show
CWE-bench v2 gives researchers and developers a shared set of practical repair tasks for comparing coding agents. Because the work involves finding a weakness in a repository and fixing it while preserving behavior, the benchmark tests more than whether a system can produce code that resembles a known patch. Public tasks and a programmatic grader also give outside teams a way to inspect the test design and repeat evaluations, subject to the dataset’s access conditions.
The scores are evidence about performance on this particular benchmark, not proof that a model can safely conduct security work across production systems. The leading reported pass@1 result remains below 70%, and the release says the top model repeatedly failed a portion of the tasks. For teams considering these tools, that gap matters: benchmark performance may help guide comparisons, but it does not remove the need for review and testing of agent-generated changes.
Products Worth Considering
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
Anker SOLIX C1000 Gen 2 Portable Power Station
Bose SoundLink Plus Portable Bluetooth Speaker, Wireless Outdoor Speaker, Up to 20 Hours Battery Life, Waterproof and Dustproof, Black
As an affiliate, we earn on qualifying purchases.
How V2 Builds on the Task Set
Version 2 keeps the benchmark’s existing task format and makes the public set 100 tasks. The authors say they reviewed each task for realistic instructions, whether the expected behavior could be inferred from the prompt, and whether the verifier checked correctness rather than requiring one specific code structure. They also report testing that the starter code fails, the reference fix passes, and existing functionality remains intact. Verifiers were checked against memorized solutions, including known upstream patches.
The announcement says panels of language models helped shape programmatic checks for task difficulty and fairness. Collinear AI also maintains a separate held-out set for tracking performance on tasks not seen by evaluated models. That set is not publicly inspectable; the release says interested teams can contact the company for access. Public results are tracked through a Hugging Face dataset leaderboard for open-weight models and a leaderboard Space that also includes API-only models. Contributors can submit results through pull requests.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparisons
The supplied results do not give enough methodological detail to independently assess every comparison, including the exact model configurations and the conditions used for each cost estimate. Scores may change as new submissions are added. The release describes task review and verifier checks, but the information provided does not establish how well results on these 100 curated tasks predict production performance across different repositories, threat models, or deployment conditions.
The held-out set may support tracking on less familiar tasks, but it is not available for direct public inspection or independent reproduction. It is also unclear whether separate teams will reproduce the published results under matching configurations, or how closely held-out scores align with outcomes in real security reviews. The announcement provides no evidence that benchmark success alone establishes an agent’s readiness to make unsupervised security changes.
More Runs and Community Review
Researchers and developers can run models against the public tasks using Harbor after accepting the dataset conditions and setting up Hugging Face access. The release invites feedback on tasks and contributions to the public leaderboard through pull requests. New submissions could change the reported comparisons and help identify problems in the task set or grading checks.
Teams seeking to evaluate models on the held-out tasks can contact Collinear AI, according to the announcement. The next useful evidence would include independently repeated public evaluations with clearly reported model settings and cost assumptions, alongside further explanation of how private-set results are used. No schedule for such evaluations or for additional benchmark releases is specified in the supplied material.
Key Questions
What is CWE-bench v2?
CWE-bench v2 is a benchmark of 100 security-fix tasks from real open-source repositories, released by Collinear AI with an evaluation harness and grader.
Which model scored highest in the reported results?
Gemini 4 Argon led the reported evaluation at 67.8% pass@1 and 81.0% pass@4. These figures describe the benchmark results in the release, not performance across all security work.
Does the 95-task result mean one model solved 95 tasks?
No. The announcement says at least one evaluated model solved 95 of 100 tasks; it does not say that any one model solved all 95.
Can the results be independently checked?
The public tasks, grader, and harness allow runs on the published set, subject to dataset access conditions. The separate held-out set is not public, and the supplied results lack some details needed to assess every model and cost comparison independently.
Primary source: Hugging Face · via ThorstenMeyerAI.com
