🔍 Read the full analysis: FrontierCode: How Maintainers Weigh AI-Generated Changes on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis tested 23 AI models with FrontierCode, a benchmark that scores whether maintainers would accept a code change, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; results show substantial differences in cost and performance.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark built around a practical maintainer question: whether a proposed change is good enough to merge. Claude Opus 5.5 ranked first with a 52.2% score, but it passed every blocking check in just 57.7% of attempts, indicating that even the top result does not establish reliable autonomous pull requests.
FrontierCode evaluates changes against six kinds of checks, including hidden tests, regression and cleanliness commands, tests that must fail on the original code, adaptive tests, limits on file scope and diff size, and an AI review based on maintainers’ criteria. An attempt that fails any blocking check scores zero; passing all blocking checks earns a score from 70% to 100%, depending on quality checks. The benchmark also assigns zero to attempts flagged for looking up solutions online.
Artificial Analysis ran the models with the same Stirrup open-source agent harness, allowing up to 250 turns per attempt and three attempts per task. The review step used GPT-6 Luna (medium) as the judge. Across the 100-task Main set, nine tasks were not solved by any model on any attempt.
GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Xiaomi’s MiMo-V2.6-Pro scored 45.4%, the highest result among the open-weight models listed. The cost per task ranged from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. Artificial Analysis placed GPT-6 Luna, MiMo-V2.6-Pro, GPT-6.1 Sol and Claude Opus 5.5 on its cost frontier, where no other model is both cheaper and higher-scoring.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Passing Review
The results address a gap in coding evaluations: a patch can pass tests and still be difficult or inappropriate to merge. FrontierCode’s scope checks, test-quality criteria and maintainer-informed review attempt to measure some of that difference. Its 57.7% blocking-check pass rate for the top model is a more limited result than a claim that an agent can routinely complete and submit acceptable changes without human oversight.
For engineering teams, performance also has to be weighed against price. Artificial Analysis reports that moving from GPT-6.1 Sol at $0.81 per task to Opus at $10.15 buys 3.1 score points. That trade-off may matter when teams choose models for routine assistance versus more demanding work. The benchmark does not by itself show how either model performs across every repository, workflow or production review process.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Sets Its Bar
Cognition developed FrontierCode with open-source maintainers; the benchmark was announced in June 2026, with version 1.1 following in July and rules against searching online for solutions. More than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The complete benchmark contains 150 tasks, while Artificial Analysis evaluated models on its 100-task Main set.
The task set is private, according to the source material, to reduce the chance that models have already encountered the solutions in training. The benchmark’s design treats review quality as part of the result: oversized diffs, changes outside relevant files and unhelpful tests can reduce scores. Tomer Nosrati, a Celery maintainer, characterized that approach in Cognition’s announcement as grading “like a tech lead,” rather than like a continuous-integration system.
The rankings also differ from general model evaluations. In the supplied comparison, Claude Sonnet 5.5 falls from second on the Artificial Analysis Intelligence Index to fifth on FrontierCode, while DeepSeek V4.1 Flash places seventh on FrontierCode despite a more modest Intelligence Index score. Those comparisons suggest that broad intelligence rankings do not directly predict performance on this particular code-review benchmark.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Coverage and Real-World Limits
The results are limited to the models and benchmark setup reported as of October 13, 2026. Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result at that point, so the standings do not cover every model that might be relevant. The source says future results from some missing models are worth watching, but gives no test date or expected release.
It is also unclear how closely scores translate to acceptance rates in live software projects. The evaluation used a fixed task set, a particular agent harness and an AI reviewer; the source does not provide evidence that the same rankings would hold across different tools, repositories or human review teams. The reported pass rate and score are distinct measures: the 57.7% figure is the share of attempts clearing all blocking checks, while 52.2% is Opus 5.5’s overall FrontierCode score.
More Model Results to Watch
The immediate next development is whether additional models receive FrontierCode results, particularly those listed as missing in the October 13 report. Further evaluations could change both the rankings and the cost comparisons. Readers should check the benchmark date and model configuration when comparing any later results, since the reported standings cover specific versions and settings.
For maintainers and engineering teams, the reported data offers a point of comparison rather than a decision rule. Teams considering AI-generated changes would still need to assess their own review requirements, repositories and costs. The supplied material does not announce a new release schedule or say when Artificial Analysis will update its leaderboard.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates whether model-generated changes meet correctness, testing, scope and review-quality checks, rather than relying only on whether tests pass.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does the result mean AI can make merge-ready changes reliably?
No. The reported top model failed to clear every blocking check in about 42% of attempts. The benchmark measures performance on its task set and does not establish reliability across all repositories or production workflows.
Which model had the highest score among open-weight models?
MiMo-V2.6-Pro scored 45.4%, the highest open-weight result identified in the supplied report. DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6%.
How much did the leading models cost per task?
Artificial Analysis listed $0.81 per task for GPT-6.1 Sol and $10.15 for Claude Opus 5.5. The report says Opus scored 3.1 points higher; costs and results apply to the evaluated configurations.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
