🔍 Read the full analysis: FrontierCode: Should Maintainers Merge Pull Requests Written By AI? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis tested 23 models on FrontierCode, a benchmark that grades coding agents on whether maintainers might accept their patches, not just whether tests pass. Claude Opus 5.5 ranked first, but cleared all blocking checks in 57.7% of attempts; the results do not establish that AI-written pull requests are ready to merge without human review.
Artificial Analysis published results for 23 AI models on FrontierCode, a coding benchmark built to assess whether a patch meets repository maintainers’ review standards. Claude Opus 5.5 ranked first at 52.2%, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review before AI-generated changes are merged.
The evaluation’s top nine models, from six labs, scored between 43.2% and 52.2%. GPT-6.1 Sol placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Artificial Analysis ran the models using the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the judge for the code-review component.
FrontierCode grades patches across six areas: hidden tests, command checks, tests that must fail on the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. Failing any blocking check results in a zero for that attempt. Passing all blocking checks produces a score between 70% and 100%, based on quality checks. The benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up solutions online.
The reported cost per task ranged from $0.09 to $12. Artificial Analysis lists GPT-6 Luna (max) at 41.9% for $0.09, MiMo-V2.6-Pro at 45.4% for $0.18, GPT-6.1 Sol at 49.1% for $0.81, and Opus at 52.2% for $10.15 as points on the cost frontier. In that comparison, moving from Sol to Opus buys 3.1 percentage points at about 12.5 times the cost. These are benchmark-specific costs and scores, not guarantees of performance on other repositories.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Merge Readiness Remains Uneven
The result addresses a practical question for open-source maintainers: does a coding agent produce a patch that meets repository-specific standards, rather than merely one that passes visible tests? FrontierCode’s top score and blocking-check rate suggest that performance on this benchmark is not yet consistent enough to treat autonomous merging as a safe default. They do not, by themselves, establish how any particular project should handle AI-written contributions.
The cost figures also show a trade-off in the tested setup. The most expensive leading model scored higher than the cheaper frontier options, but its gain over GPT-6.1 Sol was limited to 3.1 points. Meanwhile, MiMo-V2.6-Pro, an open-weights model, scored 45.4%, close to Claude Sonnet 5.5’s 45.8%. That may matter to teams weighing performance against inference cost or deployment choices, though the benchmark does not measure those teams’ full operating expenses.
Artificial Analysis’ comparison with its Intelligence Index points to another limitation of general rankings. Claude Sonnet 5.5 was second among the listed models on that index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. DeepSeek V4.1 Flash and GPT-6 Luna performed relatively better on the coding benchmark than their Intelligence Index positions might suggest. General capability scores and repository-focused patch quality are different measures, and the comparison suggests model selection for coding work may depend on the task being evaluated.
Products Worth Considering
LG PuriCare AeroHit Smart Air Purifier for Large Rooms up to 1000 sq ft, True HEPA Filter with 360° Airflow, PM1.0 Sensor & LED Air Quality
Dell XPS 13 9350 Copilot+ Laptop, 13.4-inch 120Hz Display, Intel Core Ultra 7 256V, 16GB RAM, 512GB SSD, Ubuntu Linux 24.04 LTS
FLEXISPOT 60-Inch Executive Standing Desk with Drawers and CPU Cabinet
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers, according to the source material. It was announced in June 2026, followed by version 1.1 in July, which added rules against searching online for solutions. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task.
The full benchmark contains 150 private tasks, while Artificial Analysis evaluated models on the 100-task Main set. Keeping tasks private is intended to reduce the chance that models encountered solutions during training. The report says nine of the 100 tasks were not solved by any model on any attempt. That finding applies to this test set and evaluation setup, not to all software tasks.
The benchmark’s design differs from evaluations that mainly count passing tests. It penalizes issues such as excessively large diffs, edits outside relevant files and unhelpful tests. Tomer Nosrati, a Celery maintainer, summarized the approach in Cognition’s announcement as: “Where others grade like a CI, FrontierCode grades like a tech lead.”
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Limits of the Published Results
The results are benchmark scores, not observed merge rates. They do not show how often maintainers accept AI-written pull requests in day-to-day work, nor whether the ranking holds across repositories, languages or agent configurations outside the test. The source describes the scores and evaluation setup but does not provide enough information here to assess uncertainty ranges or the statistical significance of small gaps between models.
Several models had no result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published table is not a complete comparison of all models a reader might want to evaluate. It is also unclear from the source material when additional results will be added or whether future runs will use the same models and setup.
The cost figures likewise refer to the benchmark’s cost per task. The material does not specify whether all costs would match what organizations pay under different usage patterns, infrastructure arrangements or pricing terms. The scores should not be read as proof that a model will succeed or fail on an individual pull request.
Further Model Results to Watch
Artificial Analysis may add results for models that were missing from the October 13 comparison, but the source material gives no timetable for updates. Results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview would broaden the published comparison if they are released.
For now, the next useful evidence will be additional benchmark runs and real-world evaluations that show how model performance varies by repository and task. Maintainers considering coding agents can use FrontierCode as one reference point, while checking patches against their own project rules and reviewing changes before merge. The benchmark does not remove the need for that review, and its results do not predict the outcome of a particular contribution.
Key Questions
Which model ranked first on FrontierCode?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’ reported results, with a score of 52.2%.
Does the top score mean the model passed checks on 52.2% of attempts?
No. The reported 52.2% is its FrontierCode score. The model cleared every blocking check in 57.7% of attempts; the scoring system also accounts for quality checks after blocking checks pass.
Does FrontierCode show that AI pull requests can be merged without review?
No. It is a benchmark, not a record of real-world merge decisions. The leading model failed to clear all blocking checks in a substantial share of attempts, and the results do not establish that human review can be skipped.
How many tasks and models were included?
Artificial Analysis evaluated 23 models on the 100-task Main set. The full FrontierCode benchmark contains 150 tasks, according to the source material.
Were all major models included in the results?
No. As of October 13, 2026, the source listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
