Research-Verified · Primary Sources Only · AI Model Benchmarking
Software Engineering Model Comparison
August 2026
15 frontier AI models. 19 software engineering tasks. Only 2 categories have independently verified per-model data. This is the honest map — not the marketing one.
Bottom line
GPT-5.6 Sol leads the only fully independently-run board (Artificial Analysis Terminal-Bench v2.1, 89.5%). Claude Opus 4.8 leads the harder vendor-reported SWE-bench Pro (69.2%) and is the developer-preferred agentic coder. Most vendor "SWE-bench Pro" numbers use incompatible harnesses and are not comparable to each other — treat any single figure with skepticism unless the harness is named.
12 of the 19 task categories below have no independently verified per-model data and are marked as gaps rather than guessed.
Confidence Levels — What the Markers Mean
● Independently-run
Artificial Analysis, Scale AI SWE-bench Pro — neutral third party, fixed scaffold. Highest confidence.
● Vendor-reported
Lab's own harness (OpenAI/Anthropic/Google/xAI/Alibaba model cards). Treat as an upper bound.
● Vendor-claimed only
Single-source claim (e.g. one customer eval), not independently reproducible. Shown as hollow marker.
● No data
No benchmark maps to this task; inventing a score here was the exact failure mode being corrected.
Categories with Defensible Benchmark Evidence
Only two software engineering task categories have independently verified per-model data. Everything else is either vendor-claimed or simply unmapped.
| Task | OpenAIGPT-5.6 Sol | OpenAIGPT-5.6 Terra | OpenAIGPT-5.6 Luna | AnthropicOpus 4.8 | AnthropicSonnet 5 | AnthropicHaiku 4.5 | GoogleGemini 3 Deep Think | GoogleGemini 3.1 Pro | GoogleGemini 3.6 Flash | xAIGrok 4.5 | xAIGrok 4.3 | AlibabaQwen 3.7 Max | DeepSeekDeepSeek V4 | MistralMistral Large 3 | MetaLlama 4 Maverick |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agentic SWE CodingSWE-bench Pro / Terminal-Bench composite | — | ||||||||||||||
| Large-Codebase / Long-ContextContext window + vendor long-context evals | — |
Source: Artificial Analysis Terminal-Bench v2.1 (independent) · SWE-bench Pro / SWE-bench Verified (vendor-reported) · Scale AI public leaderboard
Categories with No Independently Verified Data
13 of 19 software engineering task categories have zero independently verified per-model benchmark data. The hollow markers below are single-source vendor claims — the rest is simply unmapped. This is the gap most benchmark roundups quietly fill with guesses.
| Task | OpenAIGPT-5.6 Sol | OpenAIGPT-5.6 Terra | OpenAIGPT-5.6 Luna | AnthropicOpus 4.8 | AnthropicSonnet 5 | AnthropicHaiku 4.5 | GoogleGemini 3 Deep Think | GoogleGemini 3.1 Pro | GoogleGemini 3.6 Flash | xAIGrok 4.5 | xAIGrok 4.3 | AlibabaQwen 3.7 Max | DeepSeekDeepSeek V4 | MistralMistral Large 3 | MetaLlama 4 Maverick |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Architecture / System Design | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Code Review | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Unit Test Generation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Playwright / Selenium | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Backend / API Development | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Frontend / UI | — | — | — | — | — | — | — | — | — | — | — | — | — | ||
| SQL | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| DevOps / CI-CD | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Kubernetes | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| AWS / Azure / GCP | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Security Review | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Performance Optimization | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Documentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
Hollow dots = vendor-claimed only (not independently reproducible). Frontend/UI: GPT-5.6 Sol (Triple Whale 4.4/5), Opus 4.8 (3.5/5). Security Review: GPT-5.6 Sol (ExploitBench 73.5%).
Pricing — $ per 1M Tokens
Vendor-listed pricing as of August 2026. Input / output per 1M tokens. The cost spread between the most expensive model (GPT-5.6 Sol at $5/$30) and the cheapest (DeepSeek V4 Flash at $0.14/$0.28) is 35× on input and 107× on output — the primary driver of the developer migration to open-weight coders.
| Model | Price (input / output per 1M) |
|---|---|
| GPT-5.6 Sol | $5 / $30 |
| GPT-5.6 Terra | $2.50 / $15 |
| GPT-5.6 Luna | $1 / $6 |
| Opus 4.8 | $5 / $25 |
| Sonnet 5 | $2 / $10 (intro thru Aug 31) → $3 / $15 |
| Haiku 4.5 | $1 / $5 |
| Gemini 3 Deep Think | mode of 3.1 Pro — no separate price |
| Gemini 3.1 Pro | $2 / $12 |
| Gemini 3.6 Flash | $1.50 / $7.50 |
| Grok 4.5 | $2 / $6 |
| Grok 4.3 | $1.25 / $2.50 |
| Qwen 3.7 Max | $1.65 / $4.95 (Alibaba docs; AA lists $2.50 / $7.50) |
| DeepSeek V4 | $0.435 / $0.87 (Pro) · $0.14 / $0.28 (Flash) |
| Mistral Large 3 | $0.50 / $1.50 |
| Llama 4 Maverick | self-hosted / provider-variable |
Key Findings
GPT-5.6 Sol: The independent benchmark leader
GPT-5.6 Sol leads the only fully independently-run benchmark board — Artificial Analysis Terminal-Bench v2.1 at 89.5%. It also shows Strong capability in long-context codebase handling. At $5/$30 per 1M tokens, it's the most expensive model in the comparison, but the only one with top-tier independent verification across both benched categories.
Claude Opus 4.8: Developer-preferred, vendor-reported leader
Opus 4.8 leads the harder vendor-reported SWE-bench Pro at 69.2% and is the developer-preferred agentic coder per HN/Reddit sentiment. However, OpenAI and Anthropic report different Terminal-Bench 2.1 scores for the same model (78.9% vs 74.6%) due to differing harnesses (Codex CLI vs Terminus-2) — cross-vendor tables are not apples-to-apples.
DeepSeek V4: The unreproduced claim
DeepSeek V4's 80.6% SWE-bench Verified is vendor-reported and unreproduced. An independent DeepSWE re-run reportedly measured ~8%. At $0.435/$0.87 (Pro) or $0.14/$0.28 (Flash), the cost advantage is real — but the performance claim should be treated with significant skepticism until independently verified.
The 12-of-19 gap: Most categories are unmapped
12 of 19 software engineering task categories — including Architecture, Code Review, Unit Test Generation, Backend/API, Frontend/UI, SQL, DevOps, Kubernetes, Cloud, Security, Performance, and Documentation — have no independently verified per-model data. Any benchmark roundup claiming per-model scores across these categories is either using vendor claims or inventing numbers.
Developer Sentiment — HN/Reddit (Qualitative)
Cost-driven migration to open-weight coders
Visible migration to Kimi K2.7 Code and GLM-5.2 — developers are choosing open-weight models that deliver 80% of frontier capability at 5–10% of the cost for production workloads.
Skepticism about agentic → assisted translation
Developers question whether agentic benchmark gains (fully autonomous coding) translate to assisted coding (human-in-the-loop). The tasks are structurally different — an agent that can autonomously solve a SWE-bench task may not be the best pair-programmer.
Agent-generated PR review burden
An empirical study of 567 Claude Code PRs found 45.1% needed human revision after merge. The concern: large, hard-to-review agent-generated PRs are creating a hidden maintenance cost that benchmark scores don't capture.
Excluded as Unreliable
The following aggregator/SEO sites were excluded from this comparison because they produced fabricated or mutually contradictory figures during triangulation — including a public benchmark score for the invitation-only Claude Mythos 5, which cannot legitimately have one:
The Honest Takeaway
The AI model comparison space is dominated by benchmark roundups that fill every cell with a number — regardless of whether that number is independently verified, vendor-claimed, or simply invented. This comparison does the opposite. Where data exists, we show it with its confidence level. Where it doesn't, we show a gap.
The result: 2 of 19 categories have defensible per-model data. The other 17 are either vendor-claimed or unmapped. If you're choosing a model for production software engineering work, the honest answer is that most of the comparison surface is unknown — and the few data points we do have suggest that cost, not capability, is the primary differentiator for most use cases.
Don't trust any single benchmark score unless the harness is named and independently run. The marketing tables are not the reality.
Start a Critical Discussion
These questions don't have consensus answers. Share one to LinkedIn or X and see what your network actually thinks.
"GPT-5.6 Sol leads the only independently-run benchmark (Terminal-Bench v2.1, 89.5%). But 12 of 19 software engineering task categories have zero independently verified per-model data. Is the AI model comparison industry mostly marketing dressed as data?"
"DeepSeek V4 claims 80.6% on SWE-bench Verified — but an independent re-run measured ~8%. At 1/100th the cost of GPT-5.6 Sol, does the cost advantage outweigh the credibility gap, or is this the benchmark equivalent of a Nigerian prince email?"
"An empirical study of 567 Claude Code PRs found 45.1% needed human revision after merge. Are agentic coding benchmarks measuring the wrong thing entirely — autonomous capability when the real workload is assisted?"
Share this analysis
If this changed how you think about something, share it. The AI workforce conversation needs more data and less hype.
Sources & Methodology
All sources accessed August 2026
Agentic SWE Coding row
Artificial Analysis Terminal-Bench v2.1 (independent) cross-checked against SWE-bench Pro / SWE-bench Verified as reported on OpenAI, Anthropic, Google, xAI, and Alibaba's own model pages, plus Scale AI's standardized SWE-bench Pro public leaderboard where a model has been run there.
Long-Context row
Published context-window specs plus vendor long-context eval claims (MRCR, GraphWalks) where available.
Known conflicts
OpenAI and Anthropic report different Terminal-Bench 2.1 scores for the same Claude Opus 4.8 model (78.9% vs 74.6%) due to differing harnesses (Codex CLI vs Terminus-2). DeepSeek V4's 80.6% SWE-bench Verified is vendor-reported and unreproduced; an independent DeepSWE re-run reportedly measured ~8%. Qwen 3.7 Max, Grok 4.5, Grok 4.3, DeepSeek V4, and Mistral Large 3 have no independently-run agentic coding score on Scale AI or Terminal-Bench as of this snapshot.
Developer sentiment
Hacker News and Reddit threads (qualitative), cross-referenced with the empirical study of 567 Claude Code PRs finding 45.1% needed human revision after merge.