AI Coding Benchmarks

One-line summary: The benchmark ecosystem has specialized by what it measures — no single leaderboard ranks "the best AI coding tool" because (a) benchmarks measure models, not tools, (b) multiple benchmarks are contamination-compromised, and (c) the agent scaffolding around a model matters as much as the model itself.

The insight

Reading benchmark leaderboards as tool rankings is a category error. The major benchmarks measure models, but in practice the agent framework around a model — how it handles planning, context, tool use, feedback loops — matters as much as or more than the underlying LLM. MorphLLM's practitioner test makes this point directly: "Same model can score differently in different agents."

The benchmark landscape

From 2026-04-21-autoresearch-best-ai-coding-tools (synthesized primarily from MorphLLM's benchmark taxonomy):

April 2026 leaders

From 2026-04-21-autoresearch-best-ai-coding-tools (via MorphLLM):

Tool-level figures from MorphLLM's 15-agent practitioner test:

• SWE-bench Verified (agent-level, not model-level): claude-code 80.9%, google-antigravity 76.2%, codex-cli 75.2%.
• Terminal-Bench 2.0: codex-cli 77.3%, claude-code 65.4%.
• devin PR merge rate: 67% on defined tasks — a rare output-oriented metric (measures what the tool ships, not what it solves in isolation).

Official Terminal-Bench 2.0 harness swing (2026-08-27 pass)

From 2026-08-27-best-agent-harnesses-for-programming. This is a harness-vs-harness table (Harbor; harbor run … -k 5; no timeout/resource overrides), not a replacement of the April MorphLLM rows above.

Same-model swing: Opus 4.6 moves ~16–18 points across harnesses; GPT-5.5 is 82.2% in Codex CLI vs 84.7% in NexAU-AHE. That is the official-table version of MorphLLM’s “same model, different agents” finding. Canonical: terminal-bench-harness-swing / harness-choice-to-terminal-bench-swing.

LangChain claims a harness-only jump of Deep Agents from top-30 to top-5; the fetched table has rank 27. Leave both. Transfer to daily coding is open — does-terminal-bench-harness-swing-transfer-to-daily-coding. Do not overwrite the April MorphLLM 77.3% / 65.4% rows; different vintage and model pairing.

Why benchmark rankings don't translate to tool rankings

MorphLLM's core empirical finding: "Same model can score differently in different agents, revealing that scaffolding architecture matters as much as underlying models."

This decouples three things that leaderboards conflate:

1. Raw model coding ability. Measured cleanest by Aider Polyglot (agent-neutral).
2. Agent scaffolding quality. Measured indirectly by comparing the same model across different agent frameworks.
3. Real-world productivity. Not measured by any of the above; closer to Devin's PR merge rate or the field data in ai-coding-productivity-paradox.

A tool that pairs a middling model with excellent scaffolding can beat a tool with a better model and weaker scaffolding. This is why claude-code benchmarks well even though Opus isn't always the top model across every benchmark — its scaffolding is load-bearing.

SkillsBench print (August 2026) — independent X post

From 2026-08-18-x-ai-news-18-aug-2026-gemini-3-7-flash-plus-lab-concentration (X post by @ValsAI, 2026-08-18; not a Google announcement): "Full results are in for Gemini 3.7 Flash, and it ranks #2 on SkillsBench by leaning on skill files. It scores 65.9, 0.1pts behind Grok 4.5 (66.0), a gap inside the error bars."

This is a single independent-bench X post. The clipping records no SkillsBench task list, error-bar width, or methodology beyond "leaning on skill files." The 0.1pt gap being "inside the error bars" is the post's own qualification — do not read 65.9 vs 66.0 as a settled rank. See gemini-3-7-flash and agi-definitions-and-benchmark-saturation.

Contamination and saturation

• SWE-bench Verified contamination: OpenAI confirmed training-data leakage on every frontier model and 59.4% of the hardest unsolved tasks had flawed tests. OpenAI stopped reporting Verified scores. SWE-bench Pro's SEAL leaderboard is the preferred variant.
• HumanEval / MBPP saturation: Frontier models score 90%+ without meaningful differentiation. No longer useful for comparing top models.
• LiveCodeBench remains contamination-resistant by sourcing problems after model training cutoffs.

Terminal-Bench 4.0 / Science 0.1 (issuer table, September 2026)

• From 2026-09-02-grok-com-ai-news-digest-2026-09-02-fable-5-1-astra-atlas-g20 (Anthropic announcement): Terminal-Bench 4.0 — Fable 5.1 55.8%, Mythos 5.1 60.9%, Fable 5 42.0%. Terminal-Bench-Science 0.1 — Fable 5.1 52.6% vs Fable 5 24.7%. Anthropic notes the Fable/Mythos gap on TB 4.0 reflects earlier, less precise cyber safeguards intervening. These are TB 4.0 / Science 0.1, not the official TB 2.0 harness-swing table. Do not fold into terminal-bench-harness-swing. Full treatment: claude-fable-5-1.
• From 2026-09-02-x-ai-overnight-fable-5-1-astra-critical-reward-seeker (official product page + @claudeai X): same TB 4.0 55.8% / Science 0.1 52.6% vs Fable 5 42.0% as issuer-claimed; CursorBench 3.2.0 at 73.4% for Fable 5.1 (not in the grok.com digest grain). Anthropic footnotes note safeguard interventions zeroing some tasks — how much of the bench jump is harness / effort-level / safeguard artifact remains open. Do not re-file Mythos 5.1 60.9% from this pass (that number is on the digest page, not restated here).

Code Arena WebDev + E-Commerce Bench (2026-09-02/03, issuer + board operator)

• From 2026-09-03-x-ai-pass-3-sep-2026-open-model-fleets-agentic-coding-price (@arena): qwen-3-8-max-0902 debuted at #1 overall in Code Arena: WebDev with 1,691 points — 3 pts above Claude Opus 5 (Max), 17 pts above Kimi K3 (Max), 22 pts above the previous Qwen3.8-Max. Carry as statistically tied at the top, not "beats." Arena.ai is the board operator; that is consistency with Qwen's own posts, not a second measurement.
• From the same source (@arena): blended $5/MToken "highest-scoring model on the Pareto frontier," above HY4 Preview (1,629 pts at $2.08/MToken). Price axis is the durable fact; see llm-as-commodity-thesis.
• From the same source (@Alibaba_Qwen, 2026-09-03): E-Commerce Bench — ¥100,000 start, 365 simulated days, terminal cash as the scalar, "real e-commerce data." Vendor-authored long-horizon business-outcome bench. Treat as a claim about what should be measured, not an independent result. Eval-unit drift also on agi-definitions-and-benchmark-saturation.

Gemini 3.8 Flash — Google-reported HLE-Verified (not a coding bench)

• From 2026-09-03-gemini-3-8-flash-and-3-8-flash-cyber-sep-2-2026 (Google blog, Sep 2, 2026): Flash 54.9% HLE-Verified. That is a Google-reported score, not a third-party rerun. HLE-Verified is not SWE-bench / Terminal-Bench / SkillsBench — do not fold it into the coding-bench tables above. Flash Cyber benches (internal 20-language vuln set, CWE-Bench 47.2% vs unnamed frontier 47.8%, Chrome 2.6×) are cyber evals, also Google-reported, and are not coding benches — do not fold them here. Cross-model harness comparability remains open. Full treatment: gemini-3-8-flash. Do not rank those Cyber numbers against openai-astra-critical-cyber as if they share a harness.

GPT-6 Astra — named boards, no scores (Sep 3)

• From 2026-09-03-gpt-6-astra-rolls-out-computer-use-flagship (official @OpenAI X text, 2026-09-03): the rollout thread names Agents' Last Exam, AutomationBench, ScreenSpot Pro, FrontierMath Tier 4, ARC-AGI 3, TerminalBench-4.0, Terminal-Bench Science 0.1, and HealthBench Pro. Charts are images; this pass did not OCR them and did not retrieve the marketing score table. Names only — no scores. Do not invent numbers. Do not fold named boards into the TB 2.0 / TB 4.0 / Code Arena tables above as if they were prints. Full treatment: gpt-6-astra.

AA Index v4.3 prints (2026-09-07) — effort-labeled; not a TB 2.0 rewrite

From 2026-09-07-artificial-analysis-intelligence-index-v4-3. Independent AA, not lab-issuer tables. Do not overwrite Harbor TB 2.0 or Anthropic’s Fable 55.8%.

• From the same source: artificial-analysis-intelligence-index v4.3 puts gpt-6-astra (max) and claude-fable-5-1 (max with fallback) at 53. Category-weight FAQ 25% vs thread 30/20/30/20 left open on that page.
• From the same source (TB4 thread + page): Astra (max) 59.1% / (xhigh) 59.6%; Fable 52.0% (max with fallback) vs 55.1% (xhigh). 66 tasks; mini-SWE-agent. Full treatment: terminal-bench-harness-swing.
• From the same source (automationbench-aa): Astra (max) Score 68.5%; Grok 4.6 high 66.7% vs xhigh 67.0%. Keep effort labels.
• From the same source (cost post): AA cost-per-Index-task — Astra $3.26 vs Fable $7.63 at Index 53. That is a composite-Index dollar figure, not MorphLLM’s missing coding-tool cost-per-task. The open question below stays open.

What benchmarks still don't measure

MorphLLM flags this explicitly: "No benchmark measures cost-per-task, latency, or real-world workflow integration including code reviews and team communication." The most important properties of a tool — what it costs to ship a feature, how long the loop takes with human review, how it plays with a team's existing process — aren't captured anywhere in the leaderboard ecosystem.

Design implications

• Prefer SWE-bench Pro (SEAL) + Aider Polyglot as a pair — one tests agent-scaffolding-plus-model, the other tests pure model ability. Together they triangulate.
• Prefer Terminal-Bench for tier-2 / terminal-agent selection specifically.
• Prefer LiveCodeBench when you want contamination-clean reasoning signal.
• Ignore HumanEval at the frontier. It's saturated.
• Don't let model benchmarks pick your tool. The scaffolding-vs-model point means the best model doesn't always live in the best tool.
• Watch for output-oriented metrics (like Devin's PR merge rate). These are rare but more informative than pass@k on isolated tasks.

Open questions

• What would a rigorous "cost-per-task" benchmark look like? MorphLLM flags the absence; nobody has filled the gap.
• Can scaffolding quality be benchmarked independently of the model? Decoupling would be hugely valuable for evaluating new agent frameworks.
• Do benchmark scores predict field productivity at all? ai-coding-productivity-paradox suggests not clearly.

Sources

• 2026-04-21-autoresearch-best-ai-coding-tools
• 2026-08-18-x-ai-news-18-aug-2026-gemini-3-7-flash-plus-lab-concentration — @ValsAI SkillsBench X post (65.9 vs Grok 4.5 66.0)
• 2026-08-27-best-agent-harnesses-for-programming — official TB 2.0 harness swing
• 2026-09-02-grok-com-ai-news-digest-2026-09-02-fable-5-1-astra-atlas-g20 — issuer TB 4.0 / Science 0.1 for Fable 5.1 / Mythos 5.1 (not TB 2.0)
• 2026-09-02-x-ai-overnight-fable-5-1-astra-critical-reward-seeker — same TB 4.0 / Science 0.1 via X; CursorBench 3.2.0 73.4%
• 2026-09-03-x-ai-pass-3-sep-2026-open-model-fleets-agentic-coding-price — Code Arena WebDev 1,691 (3-pt tie at the top); E-Commerce Bench introduced (vendor-authored)
• 2026-09-03-gemini-3-8-flash-and-3-8-flash-cyber-sep-2-2026 — Google-reported 54.9% HLE-Verified for 3.8 Flash (not a coding bench); Cyber benches not folded here
• 2026-09-03-gpt-6-astra-rolls-out-computer-use-flagship — issuer-named boards only; no scores; marketing table not retrieved
• 2026-09-07-artificial-analysis-intelligence-index-v4-3 — AA Index v4.3 + TB4 / AutomationBench-AA prints; effort labels kept; not a TB 2.0 rewrite

Related

• ai-coding-tool-landscape-2026
• ai-coding-productivity-paradox
• gemini-3-8-flash
• gemini-3-7-flash
• claude-fable-5-1
• claude-opus-4-5
• claude-code
• codex-cli
• devin
• google-antigravity
• agent-harness
• terminal-bench-harness-swing
• harness-choice-to-terminal-bench-swing
• openhands
• does-terminal-bench-harness-swing-transfer-to-daily-coding
• qwen-3-8-max-0902
• gpt-6-astra
• agi-definitions-and-benchmark-saturation
• artificial-analysis-intelligence-index
• automationbench-aa