🔍 Read the full analysis: FrontierCode: Would Your AI’s Contribution Fit The Project? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added the repository-focused FrontierCode benchmark to its evaluations on October 12, 2026, testing 23 AI models. Claude Opus 5.5 led with a 52.2% score, but passed every blocking check in 57.7% of attempts; results indicate that human review remains necessary under this benchmark.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed to judge whether a patch meets repository maintainers’ standards, not simply whether it passes tests. Claude Opus 5.5 led with a 52.2% score, but cleared all blocking checks in just 57.7% of attempts, according to the published results—leaving substantial room for failure even at the top.
FrontierCode scores patches against six categories: hidden tests, command checks, reverse tests, adaptive tests, scope limits and an LLM code review. The checks can be marked blocking or non-blocking. An attempt that fails any blocking check receives a score of zero; passing all blocking checks produces a score between 70% and 100%, depending on quality checks. The benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up solutions.
In Artificial Analysis’ results, GPT-6.1 Sol scored 49.1%, followed closely by Claude Fable 5.1 at 49.0%. GPT-6 Astra scored 47.0%, while Claude Sonnet 5.5 scored 45.8%. Xiaomi’s MiMo-V2.6-Pro, the highest-scoring open-weights model listed, scored 45.4%; DeepSeek V4.1 Flash and Kimi K3 scored 44.0% and 43.6%, respectively. The benchmark’s reported pass rate is a separate measure from its overall score: Opus’s pass rate was 57.7%.
Artificial Analysis ran each model through the same open-source Stirrup agent harness, with a limit of 250 turns per attempt and three attempts per task. Its evaluation used the 100-task Main set from FrontierCode. The source says nine of the 100 tasks were not solved by any model on any attempt. These figures describe performance under the benchmark’s specific setup, not a general success rate for all coding work.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode focuses on a practical concern for software teams: a change can pass tests and still be difficult or risky to merge. By scoring scope, test usefulness and review criteria alongside correctness, the benchmark attempts to capture qualities maintainers weigh when deciding whether to accept a contribution. Its results suggest that performance on conventional coding tests alone may not tell teams how much oversight an AI-generated patch needs.
The gap between the leading model’s overall score and its blocking-check pass rate also cautions against reading the leaderboard as evidence of dependable autonomous pull requests. Under this test, the top model did not clear every blocking check in 42.3% of attempts. That figure is the complement of the reported 57.7% pass rate, not a measured failure rate across real-world repositories. The benchmark does not establish how often human developers would accept or reject the same patches.
Cost adds another deployment trade-off. Artificial Analysis lists Opus at $10.15 per task and GPT-6.1 Sol at $0.81, with scores of 52.2% and 49.1%. The source characterizes the difference as 3.1 percentage points for about 12.5 times the cost. Teams considering these systems must weigh benchmark performance against expense, review workload and the consequences of a poor patch.
Products Worth Considering
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, and IPX4 Water Resistance
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against searching online for task solutions. According to the source, more than 20 maintainers created tasks based on 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The full benchmark contains 150 tasks, while Artificial Analysis evaluated its 100-task Main set. Tasks are private, a design intended to prevent models from having encountered their solutions in training.
The grading combines automated checks with a language-model review. Artificial Analysis used GPT-6 Luna (medium) as the review judge. That means the reported results reflect not only model-generated patches and test outcomes, but also the benchmark’s task selection, harness limits and judging process. A quote in Cognition’s announcement from Celery maintainer Tomer Nosrati summarized the intended distinction: “Where others grade like a CI, FrontierCode grades like a tech lead.”
The leaderboard also differs from general model rankings in the source material. Claude Sonnet 5.5, for example, is listed second among selected models on the Artificial Analysis Intelligence Index but fifth on FrontierCode. Grok 4.7 is listed at 18th on FrontierCode, while DeepSeek V4.1 Flash ranks seventh. Those comparisons suggest that general intelligence rankings and repository-patch results are not interchangeable; they do not establish why any individual model moved up or down.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s FrontierCode announcement
Limits of the Published Results
The results do not show how models perform across every repository, programming language or type of software task. FrontierCode uses a defined set of private tasks, a specific agent harness and a particular scoring system; real-world acceptance rates may differ. The source also does not provide enough detail here to independently compare the benchmark’s results with human developers completing the same tasks.
Several models were absent from the leaderboard as of October 13, according to the source, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published rankings are not a complete comparison of all prominent coding models. It is also unclear from the supplied material how later benchmark versions, additional tasks or changes to model configurations might affect standings.
Cost figures and model results are tied to the configurations tested and reported by Artificial Analysis. They should not be assumed to represent every provider’s pricing, a different model setting or the cost of a production coding workflow. The benchmark alone cannot establish whether a particular team should deploy an agent without review.
Models Still Awaiting Results
The immediate next development to watch is whether Artificial Analysis adds results for the models not listed as of October 13, including Gemini 4 Argon and Muse Wonder Full. New entries could change relative rankings, but no release date for those evaluations is given in the supplied material.
Further comparisons will also depend on whether the benchmark expands beyond its current 100-task Main set or publishes new results under later versions. For now, the available figures offer a snapshot of performance under one shared evaluation setup. Readers should distinguish the benchmark score from a guarantee of mergeable code and treat human review as part of the process unless other evidence supports a different approach.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates AI-generated repository changes against tests, scope rules and review criteria intended to reflect maintainers’ expectations.
Which model scored highest?
Claude Opus 5.5 (max with fallback) ranked first in the Artificial Analysis results with a 52.2% FrontierCode score. Its reported pass rate for all blocking checks was 57.7%.
Does a 52.2% score mean Opus passed 52.2% of tasks?
No. The source reports the 52.2% benchmark score separately from Opus’s 57.7% blocking-check pass rate. FrontierCode’s scoring also accounts for quality checks, so the two figures are not the same measure.
Which open-weights model performed best in the listed results?
MiMo-V2.6-Pro was the top open-weights model identified in the source, with a 45.4% score. DeepSeek V4.1 Flash scored 44.0%, and Kimi K3 scored 43.6%.
Does the benchmark prove AI coding agents are ready to merge changes autonomously?
No. The results measure performance on FrontierCode’s tasks and setup, not every real-world coding workflow. The leading model cleared all blocking checks in 57.7% of attempts, and the benchmark does not establish that autonomous merging is safe for any particular project.
Source: ThorstenMeyerAI.com
