@bastani/atomic 0.9.19-alpha.1 → 0.9.19-alpha.10

package/docs/models/artificial-analysis-index.md

@@ -0,0 +1,89 @@
+---
+title: "Benchmark Sources"
+description: "The external benchmarks that inform Atomic model selection — Artificial Analysis and DeepSWE — broken down per benchmark: what each measures and when to reference it."
+---
+
+# Benchmark Sources
+
+<Note>
+This is the historical benchmark snapshot retained from the September 3, 2026 documentation baseline. It is not current model-selection guidance. For the latest dated measurements, methodology, and task-specific recommendations, use [Evals](/models/evals) and [Model selection](/models/model-selection).
+</Note>
+
+Atomic's model-selection docs are keyed to two live external benchmark sources rather than a hand-maintained table of scores. This page lists each benchmark, what it measures, and **when to reference it** for a given workflow role — so the docs stay useful as new models ship without a manual rewrite every time.
+
+<Warning>
+No single benchmark is the source of truth. Use these as inputs and validate against Atomic's own workflow evals — public suites test different task distributions than real engineering loops. When Atomic's numbers disagree with a public index, Atomic's evals win. The DeepSWE snapshot used by the linked model-selection pages was updated September 3, 2026 and was re-read from the live source on **2026-09-03**. The Artificial Analysis sections below were **last reviewed 2026-09-01** and were not re-fetched in that pass.
+</Warning>
+
+## The two sources at a glance
+
+| Source | URL | What it is | Reference it for |
+| --- | --- | --- | --- |
+| DeepSWE | [deepswe.datacurve.ai](https://deepswe.datacurve.ai/) | Long-horizon, contamination-free software-engineering tasks (113 tasks, 91 repos, 5 languages), all run on `mini-swe-agent` for consistency | The primary signal for coding-agent routing: real `pass@1`, cost, output tokens, and agent steps on engineering-loop work |
+| Artificial Analysis | [artificialanalysis.ai](https://artificialanalysis.ai/) | Cross-provider intelligence, coding, and agentic indices plus per-capability breakdowns | Cross-domain intelligence, tool use, knowledge reliability, long context, and non-coding capabilities |
+
+## DeepSWE — coding-agent performance
+
+DeepSWE is the closest public proxy for what Atomic actually does. Tasks are written from scratch (not scraped from PRs), so no model has seen the solutions; solutions require substantially more code than SWE-bench-style suites; and verifiers test behavior rather than implementation.
+
+- **Current snapshot:** DeepSWE v1.1, 113 tasks across 91 repositories and 5 languages, updated September 3, 2026. The site reports 28 measured models and displays 21 leaderboard rows by default, out of 70 published model/effort configurations.
+- **Metric:** `pass@1`, plus average cost per task, output tokens, and agent steps.
+- **When to reference:** default weighting for debugger, worker, and any code-writing role. This is the table that drives [Model Selection](/models/model-selection) and [Pareto Efficiency](/models/pareto-efficiency).
+- **Watch:** cost and step count, not just score — a model that passes but takes 268 steps (e.g. sonnet-5) is a poor worker even at a good pass rate, and the two accuracy leaders sit at opposite ends of that axis: Gemini 3.8 Flash leads the highest-published-effort reading the linked pages use at 166 average steps, while the live default Best view's leader, GPT-6 Astra [xhigh], averages 29.
+
+## Artificial Analysis — intelligence and capability breakdown
+
+Artificial Analysis separates performance by benchmark, which lets a workflow pick the model that is strong at the *specific* thing a role needs. Reference the individual evaluations, not just the composite index.
+
+### Composite indices
+
+- **Intelligence Index (v4.1.1)** — composite of the nine evaluations below, weighted across four categories: Agents 34%, Coding 24%, Scientific Reasoning 24%, General 18%. Use as a first-pass filter when a new model appears.
+- **Coding Index** — coding-weighted sub-index. Cross-check against DeepSWE.
+- **Agentic Index** — tool use, planning, autonomy, complex problem solving. The best AA signal for orchestrator and reviewer roles.
+
+### Individual evaluations — what each measures and when to reference
+
+| Benchmark | Measures | Reference it for |
+| --- | --- | --- |
+| GDPval-AA v2 | Agentic real-world work tasks | Orchestrator / planner roles doing economically realistic work |
+| τ³-Banking | Agentic tool use | Tool-heavy workflows and function-calling reliability |
+| Terminal-Bench v2.1 | Agentic coding & terminal use | Debugger and shell-driven workers |
+| SciCode | Coding (scientific) | Code-writing roles in technical domains |
+| Humanity's Last Exam | Reasoning & knowledge | Hard planning / judgment gates |
+| GPQA Diamond | Scientific reasoning | Research roles in technical domains |
+| CritPt | Physics reasoning | Physics/engineering-heavy tasks |
+| AA-Omniscience | Knowledge accuracy & non-hallucination | Research and any role where a confident wrong answer is costly |
+| AA-LCR | Long-context reasoning | Large-codebase research and long-session work |
+
+### Capability indices
+
+Artificial Analysis also publishes per-domain capability indices — **Agentic, Coding, Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, Economics**. When a workflow is domain-specific, pick by the matching capability index rather than the general Intelligence Index.
+
+<a id="role-benchmark-map"></a>
+
+## Role → benchmark map
+
+A quick lookup for which benchmark to weight per role:
+
+| Role | Primary benchmark | Secondary |
+| --- | --- | --- |
+| Debugger | DeepSWE pass@1 | Terminal-Bench v2.1 |
+| Worker / cheap loop | DeepSWE cost & steps | — |
+| Reviewer / judgment gate | DeepSWE pass@1 | AA Agentic Index |
+| Planner / orchestrator | AA Agentic Index, GDPval-AA v2 | τ³-Banking (tool use) |
+| Research | AA-LCR (long context) | AA-Omniscience (reliability) |
+| Domain-specific work | Matching AA capability index | — |
+
+## Keeping the docs fresh
+
+flora131's guidance on this issue: point the model at the live benchmark URLs and describe what each measures and when to reference it, rather than hardcoding scores that go stale on every release.
+
+1. Treat the model-selection pages as timestamped snapshots that read *from* the live sources above.
+2. When a new model appears on DeepSWE or Artificial Analysis, add it by pulling its numbers from the source — the frontier may move (as the gpt-5.6 family did).
+3. Mark a model **unmeasured** only if it is absent from both sources; unmeasured models may still be operational defaults.
+4. Prefer generating the docs and any future routing policy from the same underlying data, so documentation and routing cannot drift apart.
+
+## Related
+
+- [Model Selection](/models/model-selection)
+- [Pareto Efficiency](/models/pareto-efficiency)