@bastani/atomic 0.9.18-alpha.4 → 0.9.18-alpha.6

package/docs/intercom.md

@@ -140,10 +140,11 @@ Name sessions with `/name` so they can target each other (for example `/name pla
 | Parameter | Type | Description |
 |-----------|------|-------------|
 | `action` | string | `"list"`, `"groups"`, `"join"`, `"leave"`, `"send"`, `"ask"`, `"reply"`, `"pending"`, or `"status"` |
-| `to` | string | Exact session name/full session ID, or `<runId>:<stageKey>` for `send` to a not-yet-started workflow stage (for send/ask, or targeted reply) |
+| `to` | string | Exact session name/full session ID, or `workflow:<rootRunId>/<segment>[/<segment>...]`; `*` matches one segment and `**` any depth. Sends support pending/future patterns and broadcast; `ask` requires a live target. |
 | `message` | string | Message text (for send/ask/reply) |
 | `attachments` | array | Optional `file`, `snippet`, or `context` attachments |
 | `replyTo` | string | Optional message ID for threading or replying to an `ask` |
+| `retryToken` | string | Opaque claim returned by a retryable `send`/`ask`/`reply` failure; valid only with the exact same caller arguments. Omit it for fresh operations. |
 | `group` | string | Group name for `join` or an optional targeted `leave`; read-only group filter for `list`/`status`. `send`/`ask` remain limited to shared memberships. |
 
 ### Actions
@@ -153,8 +154,8 @@ Name sessions with `/name` so they can target each other (for example `/name pla
 | `join` | Adds a trimmed named group membership and creates the group if needed. The action waits for broker acknowledgement and reports the complete resulting membership set. `default` is shared; `true` and `auto` are reserved for subagent auto-groups. |
 | `leave` | With `group`, removes only that membership and keeps all others. Without `group`, resets the session to its resolved startup home group. Both forms report the resulting membership set. |
 | `groups` | Lists every group represented by a connected session, with its session count and a marker for each group this session belongs to. Use it to discover names rather than guessing. |
-| `list` | Returns the current session, every active session sharing at least one membership, and discoverable workflow stages. Workflow rows are explicitly labeled `PENDING` or `RUNNING` and include the canonical `<runId>:<stageId>` target. Pass `group` for a read-only view of one group. |
-| `send` | Fire-and-forget delivery through ordinary Intercom. A live workflow-stage session receives the message immediately and returns `delivered`. A known workflow stage whose session has not initialized is addressed as `<runId>:<stageKey>`; Atomic persists the message and returns the distinct `queued` result with its FIFO position. Unknown stage identities retain the ordinary unknown-target failure. Requires `to` and `message`; cannot message the current session. |
+| `list` | Returns the current session, active sessions sharing a membership, materialized workflow stages labeled `PENDING` or `RUNNING` with canonical path targets, and possible future literals, globs, and child paths with queued counts. Pass `group` for a read-only view of one group. |
+| `send` | Fire-and-forget delivery through ordinary Intercom. A live workflow-stage match receives the message immediately. A pending, future, name, or pattern path is persisted as sticky delivery and returns `queued`; valid paths outside the known set also return `notInKnownSet`. Requires `to` and `message`; cannot message the current session. |
 | `ask` | Sends a message and blocks until a live recipient replies (10-minute timeout). An ask to a known workflow stage whose session has not initialized is refused with `pending_stage_ask_unsupported` and recommends ordinary `send`; holding a waiter until a stage eventually starts would be unbounded. A live recipient disconnect fails promptly. From a foreground child to its launching parent, the existing fresh-subagent handoff path remains unchanged. |
 | `reply` | Replies to the intercom-triggered message of the current turn; otherwise falls back to the single unresolved inbound ask. With multiple pending asks, pass `to` or inspect with `pending` first. |
 | `pending` | Lists unresolved inbound asks with sender, message ID, elapsed time, and a short preview. |
@@ -172,19 +173,21 @@ Sent and received messages are recorded in session history as `intercom_sent` /
 
 ### Targeting Sessions and Pending Workflow Stages
 
-Live-session lookup accepts only an exact full Intercom session ID or an exact case-insensitive session name. Workflow stages use the canonical exact `<runId>:<stageId>` target printed by `intercom list` and workflow status surfaces; this target works while the row is `PENDING` and after it becomes `RUNNING`. Status surfaces label pending stages whose pre-start delivery capability is unavailable without presenting a usable target, and never advertise a retained pending stage after its run terminates. The `sessionId` shown by `workflow status` belongs to the workflow SDK and is **not** an Intercom target.
+Live-session lookup accepts only an exact full Intercom session ID or an exact case-insensitive session name. Workflow stages use the canonical `workflow:<rootRunId>/<segment>[/<segment>...]` path printed by `intercom list` and workflow status surfaces; an exact target works while the row is `PENDING` and after it becomes `RUNNING`. Each segment may be a stage name, run id, or glob: `*` matches one segment and may be embedded, while `**` matches any depth. Status surfaces label pending stages whose pre-start delivery capability is unavailable without presenting a usable target and never advertise a retained pending stage after its run terminates. The `sessionId` shown by `workflow status` belongs to the workflow SDK and is **not** an Intercom target.
 
 Before steering a stage from the main chat, enter the workflow invocation context by joining `workflow:<rootRunId>` with `intercom({ action: "join", group: "workflow:<rootRunId>" })`; workflow-owned invocation sessions already start there. A member of that invocation group can list, `send` to, and live-`ask` exact stages in any invocation-owned subgroup (`workflow:<rootRunId>/<name>`), including intentionally isolated reviewer batches. This control is directional: a session registered as a subgroup stage cannot gain parent control by joining the invocation group, subgroup members cannot discover or reach sibling subgroups, and another workflow invocation remains refused. `PENDING` accepts queued `send` only; `RUNNING` accepts immediate `send` and correlated `ask`/`reply`.
 
 ### Deferred delivery to pending stages
 
-Send material updates through Intercom to every affected workflow stage, including stages that have not started. Atomic queues messages for known pending stages and delivers them when their sessions initialize. A deferred send returns `queued`, including its FIFO position, rather than claiming the message was `delivered`; live delivery remains immediate and returns `delivered`.
+Send material updates through Intercom to every affected workflow stage, including stages that have not started. Inside `workflow:<rootRunId>`, `intercom list` shows live sessions, materialized `PENDING`/`RUNNING` stages, and possible future literal, glob, and nested-child targets with queued counts. A deferred send returns `queued`, including its FIFO position, rather than claiming delivery; live matches receive the ordinary inbound message immediately.
 
-The workflows extension persists up to **50 queued messages per exact run/stage key** with workflow state. Messages survive resume/replay and broker restart, and logical message IDs prevent redelivery across stage-attempt restarts. When the stage session initializes, it receives the FIFO entries through the ordinary Intercom inbound path before its first model turn, under the heading **Messages received before you started**, with sender identity and `Sent:` timestamps visible separately from the task prompt.
+Name and pattern paths remain sticky for every future matching stage until the root terminates. When shared scope or acceptance criteria change, broadcast one authoritative update to `workflow:<rootRunId>/**` (or a narrower path pattern) rather than enumerating stages: `**` reaches every live stage now and every future descendant. A syntactically valid path outside the persisted known set still queues and returns `notInKnownSet`; if it never delivers, root-terminal settlement sends the correlated undeliverable notification. A sticky entry delivered at least once is not reported undeliverable.
+
+The workflows extension persists up to **50 queued messages per target** with workflow state. Messages survive resume/replay and broker restart, and logical message IDs prevent redelivery to the same materialized stage across stage-attempt restarts. When a matching stage session initializes, it receives the FIFO entries through the ordinary Intercom inbound path before its first model turn, under the heading **Messages received before you started**, with sender identity and `Sent:` timestamps visible separately from the task prompt.
 
 Only a workflow invocation member with eligible invocation-control authority can queue to its invocation-owned stages; this includes a main-chat session that explicitly joined `workflow:<rootRunId>`. Subgroup peers and another root run remain refused even if they add that membership. An explicit stage `group: "default"` is a shared-group escape, is not workflow-owned, and does not receive pending invocation delivery. An ineligible attempt is refused with `Target workflow run is in a different intercom group`. The 51st queued message is refused with `Pending stage message queue is full (limit 50)` rather than evicting an earlier entry.
 
-If the destination stage is skipped, the run terminates, or the stage becomes terminal before its session initializes, Atomic marks the queued message undeliverable and sends the correlated failure notification when acknowledgment was requested. Blocking `ask` is deliberately unsupported before initialization: use ordinary `send`, because a stage may start much later or never start.
+If the destination stage is skipped, the run terminates, or the stage becomes terminal before its session initializes, Atomic marks the queued message undeliverable and sends the correlated failure notification when acknowledgment was requested. Use `ask` only on a live, reply-capable exact target. Pending, future, and pattern asks return `pending_stage_ask_unsupported`; use ordinary `send`, because a stage may start much later or never start.
 
 ### Groups
 
@@ -472,7 +475,27 @@ graph TB
     B2 <-->|Local Socket/Pipe| B3
 ```
 
-The broker is a standalone process that manages session registration and message routing. It auto-spawns when the first session that invokes Intercom needs it and exits 5 seconds after it last has no registered sessions, including brokers that never received a connection and sockets that close before register; clients reconnect automatically if the broker restarts. A spawn lock keyed by PID and timestamp prevents duplicate brokers when multiple sessions start at once.
+The broker is a standalone process that manages session registration and message routing. It auto-spawns when the first session that invokes Intercom needs it and exits 5 seconds after it last has no registered sessions, including brokers that never received a connection and sockets that close before register; clients reconnect automatically if the broker restarts. A reconnect that fails schedules the next attempt on a bounded backoff (1s, 2s, 5s, 10s, then 30s) and keeps retrying until the session connects or shuts down, so recovery never waits for an explicit Intercom call. A failed explicit `intercom` or overlay connection surfaces its error to the caller and still leaves that background retry in place. A reconnect that fails after the broker already accepted it closes that connection first, so a session never appears twice in `intercom list`. A spawn lock keyed by PID and timestamp prevents duplicate brokers when multiple sessions start at once.
+
+A recoverable disconnect is only reported where someone is waiting on it. Work Intercom starts on its own — eager workflow-stage warm-up during `session_start`, the background subagent and pending-stage event relays, and the advisory supervisor-authorization request made before a child launches — does not surface such a disconnect as a stage error; the stage keeps running, and a launch proceeds with supervisor metadata omitted rather than aborting. Recovery still has an owner in every case. Once the heavy module exists, the bounded reconnect backoff above owns it. Warm-up is the one point where no heavy module exists yet to run that backoff, so the wrapper itself retries the warm-up on the same bounded schedule — which matters because a stage holding queued messages waits for that first successful delivery.
+
+When those warm-up attempts run out there is no owner left, and the stage decides its own outcome rather than the extension writing a diagnostic. The wrapper hands the stage's pending delivery a typed terminal reason through the `fail(reason)` member of `WorkflowPendingStageDelivery`; the workflow side turns it into a stage-scoped error naming the run, stage id, and stage name, so `pendingStageDelivery.ready()` settles exactly once instead of waiting forever and the stage ends `failed`. Nothing goes to the console, so no raw extension text reaches the root session's transcript. The queued messages are not consumed either: a delivery asked to drain after that point is a no-op, so the steering stays queued rather than being marked delivered to a stage that never read it. A stage with nothing queued is unaffected — `ready()` still short-circuits and the stage runs.
+
+`fail` is part of the delivery contract rather than an optional extra, because `ready()` has no timeout: a delivery nobody can settle is a stage parked forever. The stage lifecycle also refuses that failure as a model failure — no same-model retry is spent, no fallback candidate is walked, and no `[fallback]` warning blames a model — because a stage refused its queued instructions would be refused them by every candidate. That refusal is by error type, not by message text, so it holds even where the shared model-failure classifier would read the underlying transport error as a retryable network problem. The delivery owner's own reason is kept on the error's `reason` property rather than chained as `cause` for the same reason.
+
+Explicit `intercom` calls, `/intercom`, and the ALT+M overlay still fail visibly. Protocol, authentication, configuration, non-recoverable initialization, and terminal relay failures are reported on every path. An exhausted warm-up retry is terminal too, but it surfaces as the stage failure described above rather than as extension output. Classification is by the error type raised inside the broker client, not by message text, so an identically worded failure from anywhere else stays actionable. A drop that first surfaces as a socket error on an already-registered connection — `ECONNRESET`, `EPIPE`, and the rest — is a recoverable disconnect and enters the same bounded recovery, with the original transport error kept as the `cause` so the code is still there to read. A framing or protocol error keeps its own `Intercom protocol error: …` diagnosis even when a socket error follows it, and a failure before registration completes is never reclassified.
+
+When an explicit `send`, `ask`, or `reply` returns the typed `Client disconnected` failure, its tool result includes an opaque `retryToken`. Retry the exact action and arguments with that token, for at most three claimed attempts. A call without `retryToken` is always a fresh intentional operation, even when every other byte is identical; concurrent failures receive independent tokens. The token is process-local, unguessable, scoped to this tool registration and host session, and mapped to one canonical operation (caller-issued target verbatim, message, canonical attachment objects with significant omission versus explicit `[]`, array order/duplicates, and reply fields). An invalid, expired, foreign-session, mismatched, in-flight, exhausted, or settled token fails before sending. The client retains the original message ID for at most 11 minutes from the first attempt—the full 10-minute `ask` wait plus one minute—with no deadline extension and no more than three token claims.
+
+A token-claimed attempt settles only after `delivered: true`, queued terminal success, or another conclusive outcome. A resolved nondelivery such as `Session not found`, durable-authority uncertainty/capacity refusal, or another result that cannot prove the accepted operation settled returns the same token for the next bounded claim; another typed disconnect does the same. The retry budget is consumed by each claim while the original deadline stays fixed. Initial tokenless nondelivery and unrelated/non-recoverable failures retain their existing classification and do not create retry state.
+
+Accepted-operation authority is stored for 12 minutes in `delivered-messages.sqlite`, but canonical payload signatures are never persisted. The broker stores only a fixed 32-byte keyed SHA-256 HMAC (hex encoded) and keeps its random key in the paired `delivered-messages.key`; this remains stable across broker replacement without exposing message or attachment text or enabling offline guesses for low-entropy payloads by users who cannot read the key. The Intercom directory is corrected to owner-only mode (`0700`) and the database, WAL, SHM, and key artifacts to `0600` on POSIX; Windows keeps its platform permission semantics. A missing/malformed database-key pair, malformed digest record, or truncated authority fails closed instead of starting empty.
+
+The broker durably reserves identity before forwarding, then marks it accepted after the confirmed write and before acknowledging the sender. A crash after forwarding and acceptance can therefore return retained success without another delivery; a pre-forward reservation is refused as uncertain. After a deduplicated ask retry, public `reply` first uses the exact recorded sender ID while it remains live, even if another live session shares its name. Only after that ID departs may reconnect-oriented name/stable-route resolution run, and ambiguity, changed stable endpoint or groups, payload, or message ID is refused without sending. An implicit reply retry retains the original sender/question route inside its token reservation and validates caller-visible `to`/`replyTo` before reading current reply state, so a later ask cannot redirect or invalidate it. Explicit `to` remains caller-controlled and broker `requirePendingReply` authorization remains mandatory. Legacy frames without logical-target metadata keep their transport-target behavior.
+
+Both sides fail closed at memory or storage pressure instead of evicting authority that can still suppress a duplicate. A fresh client operation reserves one of 1,000 identity slots before consuming an ID, showing confirmation UI, resolving its target/reply route, or sending; valid token claims remain available at full capacity. Releasing or expiring an in-flight/retained identity reopens capacity, and settled tombstones are discarded. The broker holds at most 10,000 live records and 64 MiB of digest and routing authority; it refuses new delivery until TTL cleanup makes room. The local subagent result relay also reserves before its chat side effect and accepts before positive acknowledgement, refusing its 10,001st live ID and uncertain replays without repeating delivery. SQLite transactions serialize concurrent broker access and stale rows are removed by TTL.
+
+On the broker side, a session is retired as soon as its socket stops being able to accept a frame, rather than only when the connection finally closes. A peer that half-closes, or one whose connection the broker itself ended after refusing a registration, can hold its read side open indefinitely; leaving it in the routing table meant every later broadcast wrote into a socket whose writable side was gone, which destroys that socket and floods `broker.log`. Every broker write now checks writability as part of the write itself. Delivery-producing sends also wait for the socket write callback, so an immediate asynchronous reset cannot be recorded as a successful delivery. A write that fails is answered `Session not found`, the message id stays retryable rather than being recorded as delivered, and no reply authorization is opened for a message that was not sent.
 
 Transport is local IPC only — a Unix domain socket on macOS/Linux or a named pipe on Windows — using length-prefixed JSON (4-byte length + payload) with request correlation for session listing, explicit delivery failures, and validation of malformed or out-of-order messages. `ask` stays client-side: the broker routes plain messages, and the client waits for the matching reply before returning it as the tool result.
 
@@ -483,6 +506,8 @@ Runtime files live under the active agent directory — `~/.atomic/agent/interco
 - `broker.pid` — Broker process ID
 - `broker.spawn.lock` — Short-lived lock used to avoid duplicate auto-spawns
 - `broker.log` — Broker stderr, truncated on every spawn and capped at 8 KiB by the broker itself
+- `delivered-messages.sqlite` — bounded 12-minute accepted-operation authority containing fixed keyed digests, never message or attachment text
+- `delivered-messages.key` — random owner-only HMAC key paired with the authority database
 - `config.json` — User configuration
 
 The broker runs as a detached subprocess, so it does not share the host session's module graph: every module it loads resolves from Node built-ins and Intercom's own files only. Standalone Atomic binaries run it through the internal broker handoff of the same executable, with no external runtime package to resolve.

package/docs/keybindings.md

@@ -84,10 +84,12 @@ The dedicated history actions always change history entries, regardless of the c
 
 ### TUI Fullscreen Viewport
 
-Interactive sessions always use this fullscreen viewport for the primary transcript scroll region. Mouse-wheel input scrolls the region under the pointer, falling back to the transcript over the fixed editor/status/footer dock. Clicking an OSC 8 hyperlink opens it in the default handler; the jump-to-bottom indicator's internal OSC 8 link returns the focused transcript surface to its live end, including an attached workflow stage chat. Dragging with the primary mouse button selects text and, by default, copies it to the clipboard. Set `fullscreenCopyOnSelect` to `false` to retain selections for explicit Ctrl+X copying. See [Terminal setup](/terminal-setup) for terminal-specific mouse and trackpad behavior.
+Interactive sessions always use this fullscreen viewport for the primary transcript scroll region. Mouse-wheel input scrolls the region under the pointer, falling back to the transcript over the fixed editor/status/footer dock. While the main transcript is scrolled up, a clickable "Jump to latest message" label on its bottom row shows the `tui.altScreen.bottom` shortcut; clicking it returns that transcript to its live end. An attached workflow stage chat keeps its own "Jump to latest message" OSC 8 link with the same shortcut, which returns the stage chat to its live end. Clicking other OSC 8 hyperlinks opens them in the default handler. Dragging with the primary mouse button selects text and, by default, copies it to the clipboard. Set `fullscreenCopyOnSelect` to `false` to highlight text without copying. See [Terminal setup](/terminal-setup) for terminal-specific mouse and trackpad behavior.
 
+Hold Alt while using the mouse wheel to scroll five times as far in fullscreen mode. This requires the terminal to forward the Alt modifier in mouse events; focused components may still consume the wheel input themselves.
 
-Fullscreen text selection comes from the installed pi-tui 0.84.4 renderer. Drag with the primary button to select characters; double-click selects a word, including complete slash-delimited paths and kebab-case names, and triple-click selects a line. Focus changes and non-drag clicks clear transient selection state, preventing a stale highlight from appearing. A drag release reported with the generic SGR button code also ends the selection. The renderer also reduces mouse tracking in tmux, Zellij, and GNU Screen.
+
+Fullscreen text selection comes from the installed pi-tui 0.85.1 renderer. Drag with the primary button to select characters; double-click selects a word, including complete slash-delimited paths and kebab-case names, and triple-click selects a line. Focus changes and non-drag clicks clear transient selection state, preventing a stale highlight from appearing. A drag release reported with the generic SGR button code also ends the selection. The renderer also reduces mouse tracking in tmux, Zellij, and GNU Screen.
 Fullscreen transcript bindings take precedence over editor bindings while the main editor has focus. The default unmodified navigation keys therefore control the transcript, while their `ctrl` variants continue to control the editor. When a fullscreen overlay or inline custom component has focus, Atomic sends matching viewport bindings to that component first. Returning `true` keeps the key local. For an in-process component, returning `false`, `undefined`, or `void` lets transcript scrolling handle it. A remote component's correlated reply falls through on `false`, failure, or timeout; `undefined` after disposal is dropped because that component no longer owns focus.
 
 | Key | Editor action | Fullscreen action |
@@ -128,7 +130,6 @@ On Windows, pressing the secondary mouse button in fullscreen pastes text from t
 | `app.suspend` | `ctrl+z` (none on Windows) | Suspend to background |
 | `app.editor.external` | `ctrl+g` | Open in external editor (`$VISUAL` or `$EDITOR`) |
 | `app.clipboard.pasteImage` | `ctrl+v` (`alt+v` on Windows) | Paste image or text from clipboard |
-| `app.message.copy` | `ctrl+x` | When `fullscreenCopyOnSelect` is `false`, copy the active fullscreen selection; otherwise copy the last assistant message |
 
 When `app.clipboard.pasteImage` finds text rather than an image, Atomic inserts that clipboard text into the editor instead of reporting an image-paste failure.
 
@@ -138,7 +139,7 @@ Inside tmux on macOS, `Ctrl+V` is the reliable image-paste shortcut; native `Cmd
 
 When the clipboard has both text and an image, behavior depends on the terminal: empty-paste terminals may insert the text on `Cmd+V`, while Kitty-protocol terminals that deliver `super+v` go through the image path (same preference as `Ctrl+V`). `Ctrl+V` always prefers the image. Apple Terminal may send nothing for image-only paste; use Ghostty/iTerm/Kitty or `Ctrl+V` in that case.
 
-Ctrl+X keeps Atomic's hierarchy precedence. A workflow tool-detail view closes to its graph; the scoped-model selector clears its local selection; an attached workflow stage chat returns to its graph; and a workflow graph returns to main chat. Only the main editor then runs `app.message.copy`. Workflow surfaces recognize the physical Ctrl+X chord directly, including CSI variants, rather than the configurable application action. `/copy` is separate and always copies the last assistant message.
+Ctrl+X does not copy messages or selections. A workflow tool-detail view closes to its graph; the scoped-model selector clears its local selection; an attached workflow stage chat returns to its graph; and a workflow graph returns to main chat. Workflow surfaces recognize the physical Ctrl+X chord directly, including CSI variants. `/copy` always copies the last assistant message.
 
 A held paused queue by itself is idle for Ctrl+C handling. After an interruption settles, the next Ctrl+C clears the editor without releasing or dequeuing the hold, and a second quick idle press exits normally.
 
@@ -179,6 +180,8 @@ Ctrl+C is the host's escape hatch whenever an engine-owned `ctx.ui.custom()` com
 | `app.thinking.cycle` | `shift+tab` | Cycle thinking level |
 | `app.thinking.toggle` | `ctrl+t` | Collapse or expand thinking blocks |
 
+Interactive model and thinking choices automatically become startup defaults. There is no save-default shortcut.
+
 ### Display and Message Queue
 
 | Keybinding id | Default | Description |
@@ -205,11 +208,10 @@ Ctrl+C is the host's escape hatch whenever an engine-owned `ctx.ui.custom()` com
 
 ### Scoped Models Selector
 
-Used inside the scoped models selector (opened via `/scoped-models`).
+Used inside the scoped models selector (opened via `/scoped-models`). Changes are saved automatically.
 
 | Keybinding id | Default | Description |
 |--------|---------|-------------|
-| `app.models.save` | `ctrl+s` | Save current model selection to settings |
 | `app.models.enableAll` | `ctrl+a` | Enable all models (or all matching the current search) |
 | `app.models.clearAll` | `ctrl+x` | Clear all models (or all matching the current search) |
 | `app.models.toggleProvider` | `ctrl+p` | Toggle all models for the current provider |

package/docs/models/evals.md

@@ -0,0 +1,394 @@
+---
+title: "Evals"
+description: "The external evals that inform Atomic model selection — Artificial Analysis and DeepSWE — broken down per benchmark with the measured numbers: what each measures, who scores what, and when to reference it."
+---
+
+# Evals
+
+Atomic's model-selection docs are keyed to two live external eval sources rather than a hand-maintained table of scores. This page lists each eval, what it measures, the measured numbers for the models in Atomic's catalog, and **when to reference it** for a given workflow role — so an agent authoring a workflow can pick a model for a task type from evidence rather than from an aggregate rank.
+
+<Warning>
+No single benchmark is the source of truth. Validate these inputs against Atomic's own workflow evals, whose task distribution is closer to the work you intend to run. Artificial Analysis was re-fetched on **2026-09-05**, including its September 4 Intelligence Index revision; the per-evaluation scores, leaderboard rows and Coding Agent Index rows below were read from the rendered charts on that date. The DeepSWE leaderboard rows below retain their **2026-09-03** compilation date and were read from the live page on **2026-09-05**. Every number is a rounded value as displayed by the source; unrounded values and confidence intervals live on the linked pages.
+</Warning>
+
+## The two sources at a glance
+
+| Source | URL | What it is | Reference it for |
+| --- | --- | --- | --- |
+| DeepSWE | [deepswe.datacurve.ai](https://deepswe.datacurve.ai/) | Long-horizon, contamination-free software-engineering tasks (113 tasks, 91 repos, 5 languages), all run on `mini-swe-agent` for consistency | The primary signal for coding-agent routing: real `pass@1`, cost, output tokens, and agent steps on engineering-loop work |
+| Artificial Analysis | [artificialanalysis.ai](https://artificialanalysis.ai/) | Model intelligence and professional capability indices, individual evaluations, and a separate coding-agent leaderboard | Cross-domain intelligence, tool use, knowledge reliability, long context, and agent/model comparisons |
+
+## Pick by task type
+
+Start here when a stage needs a model. Each row names the benchmark that measures the task type, the top measured picks and the cheapest pick that stays close, using the tables further down. "Measured" means the exact configuration named; a different effort level or agent is a different row on the source.
+
+| Task type | Benchmark to read | Top measured picks | Budget pick that holds up |
+| --- | --- | --- | --- |
+| Implementing features and fixing bugs in a repo | DeepSWE `pass@1`, `$/task`, steps | gpt-6-astra [xhigh] 74% / $6.52 / 29 steps; gemini-3.8-flash [high] 74% / $2.36 / 166 steps; claude-opus-5 [max] 74% / $11.84 | gpt-5.6-luna [max] 67% / $0.61; glm-5.3-flash [max] 63% / $0.24 |
+| Terminal work, sysadmin, debugging in a shell | Terminal-Bench v2.1 | Fable 5.1 max 91%; Astra high and medium 90%; Sol xhigh 90%; Opus 5 max 89% | Terra max and Gemini 3.8 Flash high 88%; Gemini 3.7 Flash high 86%; GLM-5.3-Flash 84%; Luna max 81% |
+| Multi-file knowledge-work deliverables (spreadsheets, decks, memos) | AA-Briefcase, GDPval-AA v2 | Fable 5.1 max 58% / 63%; Opus 5 max 58% / 62%; Muse Spark 1.3 max 53% / 61% | GLM-5.3-Flash 48% / 59% at $0.18 per Index task |
+| Tool-calling against an API or knowledge base | 𝜏³-Banking | Muse Spark 1.3 max 52%; Grok 4.6 high 51%; GLM-5.3 max 50% | GLM-5.3-Flash 47%; Gemini 3.8 Flash high 45%. Luna max scores 31%: avoid it for tool-heavy loops |
+| Reasoning over long PDFs and reports | GDP.pdf All-pass | Astra max 33%; Astra xhigh 32%; Sol max 28%; Fable 5.1 max 26% | Astra low 30%; Luna max 23%. GLM-5.3 max 12% and Sonnet 5 max 13% are weak here |
+| Long-context extraction and synthesis | AA-LCR v1.1 | Kimi K3 max 89%; Fable 5.1 max 85%; Sol max, Luna max and Muse Spark 1.3 max 84% | Luna max 84% at $0.10 per Index task; nearly everything else sits at 79–83% |
+| Facts without citations, where a wrong answer is worse than "I don't know" | AA-Omniscience non-hallucination rate | GLM-5.3-Flash 72%; GLM-5.3 max 70%; Muse Spark 1.3 xhigh 69%; Grok 4.6 high 66% | GLM-5.3-Flash is already the cheapest row; Sonnet 5 max 61% is the Anthropic option. Sol max 8%, Luna max 7%, Terra max 12% and DeepSeek V4 5–8% answer wrong rather than abstain |
+| Facts where raw recall matters and the answer will be checked | AA-Omniscience accuracy | Fable 5.1 max 67%; Fable 5 65%; Astra max 63% | Gemini 3.8 Flash high and Gemini 3.7 Flash high 55% |
+| Scientific or numerical programming | SciCode | Fable 5.1 max 63%; Fable 5 61%; Muse Spark 1.3 xhigh 60%; Kimi K3 max and GLM-5.3 max 59% | Gemini 3.8 Flash high and Gemini 3.7 Flash high 57%; Luna max 54% |
+| Hard closed-form reasoning and research-level physics | Humanity's Last Exam, CritPt | HLE: Fable 5.1 max 59%; Opus 5 max and Astra max 55%. CritPt: Sol max and Astra max 32% | Astra medium 53% HLE / 29% CritPt |
+| Whole coding-agent product comparison (agent + model) | AA Coding Agent Index v1.4 | Claude Code + Fable 5.1 max 70; Claude Code + Opus 5 xhigh 68; Muse Code + Muse Spark 1.3 max 68 | Opencode + Gemini 3.8 Flash high 61 at $2.04 and 11.9 min; Codex + Luna max 57 at $0.29 and 8.0 min |
+
+Three cross-cutting reads from the numbers:
+
+- **Fable 5.1 (max, default fallback) is the broadest model** — first or tied-first on Terminal-Bench, AA-Briefcase, GDPval, SciCode, HLE and Omniscience accuracy — but it is the most expensive per AA task ($6.12) and its non-hallucination rate is 27%, below Opus 5 (39%) and Astra (49–55%). It is absent from the DeepSWE snapshot.
+- **GPT-6 Astra is the document and terminal specialist** — it leads GDP.pdf at every effort level, sits within a point of the Terminal-Bench leader, and its DeepSWE Best row solves 74% in 29 steps — but it trails the Anthropic rows by 5 points on AA-Briefcase and 8–9 on GDPval, where Muse Spark 1.3 max also leads it by 7, and it trails Meta, xAI and Z.AI by 9–11 points on 𝜏³-Banking.
+- **The OpenAI budget tier is accurate but overconfident.** Luna and Sol score 39–49% on HLE and 81–90% on Terminal-Bench, yet answer wrongly rather than abstain 91–93% of the time when they do not know. Pair them with verification tool nodes; do not use them for uncited research summaries.
+
+## DeepSWE — coding-agent performance
+
+DeepSWE is the closest public proxy for what Atomic actually does. Tasks are written from scratch (not scraped from PRs), so no model has seen the solutions; solutions require substantially more code than SWE-bench-style suites; and verifiers test behavior rather than implementation.
+
+- **Current snapshot:** DeepSWE v1.1, 113 tasks across 91 repositories and 5 languages, updated September 3, 2026. The site reports 28 measured models and displays 21 leaderboard rows by default, out of 70 published model/effort configurations.
+- **Metric:** `pass@1`, plus average cost per task, output tokens, and agent steps.
+- **When to reference:** default weighting for debugger, worker, and any code-writing role. This is the table that drives [Model Selection](/models/model-selection) and [Pareto Efficiency](/models/pareto-efficiency).
+- **Watch:** cost and step count, not just score — a model that passes but takes 268 steps (e.g. sonnet-5) is a poor worker even at a good pass rate, and the two accuracy leaders sit at opposite ends of that axis: Gemini 3.8 Flash leads the highest-published-effort reading the linked pages use at 166 average steps, while the live default Best view's leader, GPT-6 Astra [xhigh], averages 29.
+
+### Live leaderboard, Best view
+
+DeepSWE's default table is the **Best** view: the best-scoring effort configuration per model. These are the 21 rows it displayed on 2026-09-05 for the September 3, 2026 snapshot. [Model Selection](/models/model-selection) instead tabulates the highest published effort per model, so four rows differ there (`gpt-6-astra [max]`, `claude-fable-5 [max]`, `grok-4.6 [xhigh]`, `gemini-3.7-flash [high]`). Confidence intervals are DeepSWE's displayed ±.
+
+| Model [effort] | pass@1 | Avg $/task | Output tokens | Steps |
+| --- | --- | --- | --- | --- |
+| gpt-6-astra [xhigh] | 74% ±3 | $6.52 | 30k | 29 |
+| gemini-3.8-flash [high] | 74% ±1 | $2.36 | 143k | 166 |
+| claude-opus-5 [max] | 74% ±4 | $11.84 | 118k | 99 |
+| gpt-5.6-sol [max] | 73% ±3 | $6.46 | 60k | 61 |
+| claude-fable-5 [xhigh] | 70% ±3 | $13.41 | 80k | 68 |
+| glm-5.3 [max] | 69% ±3 | $3.99 | 80k | 124 |
+| kimi-k3 [max] | 69% ±5 | $4.65 | 81k | 98 |
+| grok-4.6 [medium] | 67% ±2 | $3.45 | 50k | 70 |
+| gpt-5.6-luna [max] | 67% ±4 | $0.61 | 73k | 102 |
+| gpt-5.5 [xhigh] | 67% ±6 | $7.23 | 46k | 82 |
+| gemini-3.7-flash [medium] | 65% ±3 | $2.03 | 94k | 117 |
+| glm-5.3-flash [max] | 63% ±4 | $0.24 | 73k | 123 |
+| deepseek-v4-pro [max] | 63% ±6 | $1.67 | 106k | 155 |
+| claude-opus-4.8 [max] | 59% ±2 | $13.22 | 135k | 120 |
+| qwen3.8-max [xhigh] | 57% ±3 | $3.73 | 95k | 111 |
+| muse-spark-1.2 [xhigh] | 55% ±2 | $3.70 | 99k | 101 |
+| claude-sonnet-5 [max] | 54% ±4 | $26.40 | 214k | 268 |
+| deepseek-v4-flash [max] | 53% ±4 | $0.46 | 108k | 153 |
+| gemini-3.6-flash [high] | 47% ±4 | $2.21 | 96k | 117 |
+| glm-5.2 [max] | 44% ±2 | $3.92 | 78k | 129 |
+| gemini-3.5-flash [high] | 36% ±4 | $3.45 | 76k | 105 |
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 380}}}%%
+xychart-beta
+    title "DeepSWE v1.1 pass@1 (%), Best view, September 3, 2026 snapshot"
+    x-axis ["astra xhigh", "g3.8-flash high", "opus-5 max", "sol max", "fable-5 xhigh", "glm-5.3 max", "kimi-k3 max", "grok-4.6 med", "luna max", "gpt-5.5 xhigh", "g3.7-flash med", "glm-5.3-flash", "dsv4-pro max", "opus-4.8 max", "qwen3.8-max", "muse-1.2 xhigh", "sonnet-5 max", "dsv4-flash max", "g3.6-flash high", "glm-5.2 max", "g3.5-flash high"]
+    y-axis "pass@1 (%)" 0 --> 100
+    bar [74, 74, 74, 73, 70, 69, 69, 67, 67, 67, 65, 63, 63, 59, 57, 55, 54, 53, 47, 44, 36]
+```
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 380}}}%%
+xychart-beta
+    title "DeepSWE average USD per task, same rows and order"
+    x-axis ["astra xhigh", "g3.8-flash high", "opus-5 max", "sol max", "fable-5 xhigh", "glm-5.3 max", "kimi-k3 max", "grok-4.6 med", "luna max", "gpt-5.5 xhigh", "g3.7-flash med", "glm-5.3-flash", "dsv4-pro max", "opus-4.8 max", "qwen3.8-max", "muse-1.2 xhigh", "sonnet-5 max", "dsv4-flash max", "g3.6-flash high", "glm-5.2 max", "g3.5-flash high"]
+    y-axis "USD per task" 0 --> 28
+    bar [6.52, 2.36, 11.84, 6.46, 13.41, 3.99, 4.65, 3.45, 0.61, 7.23, 2.03, 0.24, 1.67, 13.22, 3.73, 3.70, 26.40, 0.46, 2.21, 3.92, 3.45]
+```
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 380}}}%%
+xychart-beta
+    title "DeepSWE average agent steps per task, same rows and order"
+    x-axis ["astra xhigh", "g3.8-flash high", "opus-5 max", "sol max", "fable-5 xhigh", "glm-5.3 max", "kimi-k3 max", "grok-4.6 med", "luna max", "gpt-5.5 xhigh", "g3.7-flash med", "glm-5.3-flash", "dsv4-pro max", "opus-4.8 max", "qwen3.8-max", "muse-1.2 xhigh", "sonnet-5 max", "dsv4-flash max", "g3.6-flash high", "glm-5.2 max", "g3.5-flash high"]
+    y-axis "steps" 0 --> 280
+    bar [29, 166, 99, 61, 68, 124, 98, 70, 102, 82, 117, 123, 155, 120, 111, 101, 268, 153, 117, 129, 105]
+```
+
+What the three charts say together:
+
+- **Accuracy is flat at the top.** Three models display 74% and a fourth 73%, all inside each other's confidence intervals. Choose among them on cost and steps, not score.
+- **Cost spans two orders of magnitude at the same score.** Luna [max] and Gemini 3.8 Flash [high] reach 67% and 74% for $0.61 and $2.36; Opus 5 [max] and Fable 5 [xhigh] reach 74% and 70% for $11.84 and $13.41. Sonnet 5 [max] is the outlier to avoid: 54% for $26.40 and 268 steps.
+- **Steps predict wall time and tool-call load.** Astra [xhigh] (29) and Sol [max] (61) finish in a third of the steps that Gemini 3.8 Flash [high] (166) or DeepSeek V4 Pro [max] (155) need. For a worker loop that pays per tool call or that a reviewer must audit, prefer the low-step row at the same accuracy.
+- **The cheap tier is honest about its ceiling.** GLM-5.3-Flash [max] 63% at $0.24 and DeepSeek V4 Flash [max] 53% at $0.46 are the only rows under $1 besides Luna; they are budget workers, not judgment gates.
+
+## Artificial Analysis: current measures
+
+### Intelligence Index v4.2
+
+The [September 4, 2026 announcement](https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-2) and [current methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking), retrieved 2026-09-05, identify **Artificial Analysis Intelligence Index v4.2**. It adds AA-Briefcase and GDP.pdf, removes GPQA Diamond from this index, upgrades AA-LCR to v1.1, improves SciCode grading, and rebalances the weights. It also revises GDPval-AA v2 and AA-Briefcase Elo sampling and anchoring. Do not compare scores across index revisions as though only the models changed. The announcement describes v5 as upcoming, not current.
+
+The ten evaluations and their contributions are:
+
+| Category and total weight | Evaluation | Index weight | Use it for |
+| --- | --- | --- | --- |
+| Agents, 30% | AA-Briefcase | 15% | Multi-week knowledge-work projects and file deliverables |
+| Agents | GDPval-AA v2 | 10% | Economically realistic professional work |
+| Agents | 𝜏³-Banking | 5% | Tool use and customer interaction |
+| Coding, 20% | Terminal-Bench v2.1 | 10% | Terminal execution and debugging |
+| Coding | SciCode | 10% | Scientific programming |
+| Scientific Reasoning, 20% | Humanity's Last Exam | 10% | Hard reasoning and knowledge |
+| Scientific Reasoning | CritPt | 10% | Physics reasoning |
+| General, 30% | AA-Omniscience | 15% | Knowledge accuracy, 10%, and non-hallucination, 5%, as separate components |
+| General | GDP.pdf | 10% | Professional document reasoning; headline All-pass requires every criterion to pass |
+| General | AA-LCR v1.1 | 5% | Long-context reasoning |
+
+This is primarily a text-based, English-language suite, not a universal measure of multimodal or multilingual quality. The [additional evaluations](https://artificialanalysis.ai/methodology/intelligence-benchmarking#additional-evaluations), such as AutomationBench-AA, AA-AnalystAgent and ITBench-AA, can be better matches for SaaS workflows, spreadsheet analysis or incident diagnosis. Their presence on the site does not make them Intelligence Index components. GPQA Diamond also remains visible separately and in the Engineering capability index.
+
+### Headline leaderboard rows for catalog models
+
+From the [LLM leaderboard](https://artificialanalysis.ai/leaderboards/models), retrieved 2026-09-05. Cost is AA's weighted **cost per Intelligence Index task** (confirmed against the model-page label), not a token price. Speed is output tokens per second on the default 10k-input workload. Latency is AA's time to first token, which for a streaming reasoning model can be the first reasoning token; end-to-end is seconds to a 500-token answer including thinking. `—` means AA does not report the value.
+
+| AA configuration | Intelligence Index | $/Index task | Output tok/s | TTFT (s) | End-to-end 500 tok (s) |
+| --- | --- | --- | --- | --- | --- |
+| Claude Fable 5.1 (max with fallback) | 57 | $6.12 | 69 | 266.5 | 273.8 |
+| GPT-6 Astra (max) | 55 | $2.57 | 87 | 463.7 | 469.5 |
+| GPT-6 Astra (xhigh) | 54 | $1.85 | 81 | 309.5 | 315.7 |
+| Claude Opus 5 (max) | 54 | $4.21 | 59 | 91.2 | 99.7 |
+| Claude Opus 5 (xhigh) | 53 | $3.36 | 57 | 33.8 | 42.6 |
+| GPT-6 Astra (high) | 53 | $1.41 | 87 | 141.6 | 147.3 |
+| Claude Fable 5 (with fallback) | 53 | $5.62 | 70 | 111.4 | 118.5 |
+| Muse Spark 1.3 (max) | 53 | $0.96 | 190 | 18.7 | 31.8 |
+| GPT-6 Astra (medium) | 52 | $1.16 | 79 | 24.6 | 30.9 |
+| Claude Opus 5 (high) | 52 | $2.44 | 56 | 23.0 | 31.9 |
+| Muse Spark 1.3 (xhigh) | 52 | $0.84 | 135 | 42.6 | 61.0 |
+| GPT-5.6 Sol (max) | 51 | $1.25 | 85 | 163.0 | 168.8 |
+| Grok 4.6 (high) | 51 | $1.25 | 65 | 52.4 | 60.1 |
+| Kimi K3 (max) | 50 | $1.58 | 40 | 4.7 | 67.9 |
+| GPT-5.6 Sol (xhigh) | 50 | $0.89 | 82 | 76.0 | 82.1 |
+| GLM-5.3 (max) | 49 | $1.26 | 80 | 2.1 | 33.3 |
+| GPT-6 Astra (low) | 49 | $0.63 | 86 | 6.6 | 12.5 |
+| GPT-5.6 Sol (high) | 48 | $0.61 | 72 | 18.8 | 25.8 |
+| Gemini 3.8 Flash (high) | 47 | $0.74 | — | — | — |
+| Qwen3.8 Max | 47 | $1.19 | 39 | 2.4 | 66.5 |
+| Muse Spark 1.2 (xhigh) | 47 | $0.55 | 267 | 15.3 | 24.6 |
+| GPT-5.6 Terra (max) | 47 | $0.81 | 111 | 244.8 | 249.3 |
+| GLM-5.3-Flash | 46 | $0.18 | 48 | 1.6 | 54.2 |
+| Gemini 3.7 Flash (high) | 45 | $0.55 | 310 | 10.6 | 12.2 |
+| Claude Sonnet 5 (max) | 45 | $3.31 | 74 | 178.7 | 185.4 |
+| GPT-5.6 Luna (max) | 43 | $0.10 | 135 | 173.2 | 177.0 |
+| DeepSeek V4 Pro 0813 (max) | 42 | $0.33 | 70 | 1.6 | 37.2 |
+| GPT-5.6 Luna (xhigh) | 42 | $0.06 | 125 | 71.1 | 75.1 |
+| DeepSeek V4 Flash 0731 (max) | 41 | $0.14 | 135 | 1.1 | 19.6 |
+| Gemini 3.6 Flash | 40 | $0.36 | 215 | 15.5 | 17.9 |
+
+Two things the latency columns make obvious that the index hides: `max` effort on OpenAI and Anthropic models costs three to eight minutes before the first answer token (Astra max 464 s, Fable 5.1 max 267 s, Terra max 245 s, Luna max 173 s), and the fast interactive tier is Opus 5 high or xhigh (23–34 s), Astra medium or low (7–25 s), Muse Spark 1.3 (19–43 s) and the Flash models (1–16 s). Pick effort for an interactive session from this column, not from the index.
+
+### Per-evaluation scores for catalog models
+
+Read from the "Intelligence Evaluations" charts on the AA model pages ([GPT-6 Astra](https://artificialanalysis.ai/models/gpt-6-astra), [GLM-5.3-Flash](https://artificialanalysis.ai/models/glm-5-3-flash), [Gemini 3.7 Flash](https://artificialanalysis.ai/models/gemini-3-7-flash), [Claude Sonnet 5](https://artificialanalysis.ai/models/claude-sonnet-5), [GPT-5.6 Terra](https://artificialanalysis.ai/models/gpt-5-6-terra)) on 2026-09-05. AA-Briefcase and GDPval-AA v2 are Elo scales; the model pages display them as `(Elo − 500) / 2000`, so 58% is Elo 1666 and 63% is Elo 1769. Everything else is a pass rate. Bold marks the column leader among these rows.
+
+**Agentic and coding evaluations**
+
+| AA configuration | AA-Briefcase | GDPval-AA v2 | 𝜏³-Banking | Terminal-Bench v2.1 | SciCode |
+| --- | --- | --- | --- | --- | --- |
+| Claude Fable 5.1 (max with fallback) | **58%** | **63%** | 47% | **91%** | **63%** |
+| Claude Opus 5 (max) | **58%** | 62% | 42% | 89% | 56% |
+| Claude Opus 5 (xhigh) | 56% | 61% | 43% | 88% | 56% |
+| Claude Opus 5 (high) | 53% | 57% | 45% | 88% | 55% |
+| Claude Fable 5 (with fallback) | 52% | 57% | 38% | 85% | 61% |
+| GPT-6 Astra (max) | 53% | 54% | 41% | 88% | 56% |
+| GPT-6 Astra (xhigh) | 52% | 53% | 43% | 89% | 56% |
+| GPT-6 Astra (high) | 50% | 52% | 40% | 90% | 55% |
+| GPT-6 Astra (medium) | 48% | 50% | 35% | 90% | 54% |
+| GPT-6 Astra (low) | 38% | 46% | 32% | 88% | 54% |
+| GPT-5.6 Sol (max) | 49% | 56% | 44% | 88% | 57% |
+| GPT-5.6 Sol (xhigh) | 47% | 55% | 38% | 90% | 57% |
+| GPT-5.6 Sol (high) | 43% | 51% | 37% | 87% | 58% |
+| GPT-5.6 Terra (max) | 42% | 49% | 40% | 88% | 55% |
+| GPT-5.6 Luna (max) | 42% | 50% | 31% | 81% | 54% |
+| Muse Spark 1.3 (max) | 53% | 61% | **52%** | 86% | 58% |
+| Muse Spark 1.3 (xhigh) | 49% | 58% | 47% | 85% | 60% |
+| Grok 4.6 (high) | 52% | 57% | 51% | 88% | 56% |
+| Kimi K3 (max) | 50% | 54% | 46% | 85% | 59% |
+| GLM-5.3 (max) | 51% | 59% | 50% | 84% | 59% |
+| GLM-5.3-Flash | 48% | 59% | 47% | 84% | 52% |
+| Gemini 3.8 Flash (high) | 35% | 48% | 45% | 88% | 57% |
+| Gemini 3.7 Flash (high) | 31% | 47% | 33% | 86% | 57% |
+| Claude Sonnet 5 (max) | 43% | 50% | 37% | 81% | 54% |
+| DeepSeek V4 Pro 0813 (max) | 39% | 50% | 40% | 79% | 51% |
+| DeepSeek V4 Flash 0731 (max) | 38% | 49% | 39% | 79% | 50% |
+
+**Reasoning, knowledge and document evaluations**
+
+| AA configuration | Humanity's Last Exam | CritPt | GDP.pdf All-pass | AA-Omniscience accuracy | AA-Omniscience non-hallucination | AA-LCR v1.1 |
+| --- | --- | --- | --- | --- | --- | --- |
+| Claude Fable 5.1 (max with fallback) | **59%** | 30% | 26% | **67%** | 27% | 85% |
+| Claude Opus 5 (max) | 55% | 29% | 22% | 61% | 39% | 79% |
+| Claude Opus 5 (xhigh) | 54% | 28% | 21% | 60% | 40% | 80% |
+| Claude Opus 5 (high) | 53% | 28% | 20% | 59% | 39% | 79% |
+| Claude Fable 5 (with fallback) | 55% | 29% | 24% | 65% | 36% | 82% |
+| GPT-6 Astra (max) | 55% | **32%** | **33%** | 63% | 49% | 81% |
+| GPT-6 Astra (xhigh) | 55% | 31% | 32% | 62% | 52% | 80% |
+| GPT-6 Astra (high) | 53% | 29% | 31% | 61% | 55% | 80% |
+| GPT-6 Astra (medium) | 53% | 29% | 30% | 61% | 53% | 80% |
+| GPT-6 Astra (low) | 49% | 26% | 30% | 60% | 53% | 80% |
+| GPT-5.6 Sol (max) | 49% | **32%** | 28% | 59% | 8% | 84% |
+| GPT-5.6 Sol (xhigh) | 47% | 29% | 28% | 59% | 8% | 82% |
+| GPT-5.6 Sol (high) | 46% | 26% | 28% | 58% | 9% | 82% |
+| GPT-5.6 Terra (max) | 43% | 30% | 26% | 47% | 12% | 83% |
+| GPT-5.6 Luna (max) | 39% | 21% | 23% | 43% | 7% | 84% |
+| Muse Spark 1.3 (max) | 49% | 25% | 26% | 44% | 66% | 84% |
+| Muse Spark 1.3 (xhigh) | 47% | 26% | 23% | 42% | 69% | 83% |
+| Grok 4.6 (high) | 43% | 17% | 19% | 48% | 66% | 80% |
+| Kimi K3 (max) | 47% | 23% | 20% | 48% | 47% | **89%** |
+| GLM-5.3 (max) | 42% | 19% | 12% | 34% | 70% | 80% |
+| GLM-5.3-Flash | 40% | 15% | 13% | 28% | **72%** | 80% |
+| Gemini 3.8 Flash (high) | 48% | 18% | 19% | 55% | 45% | 81% |
+| Gemini 3.7 Flash (high) | 48% | 14% | 24% | 55% | 35% | 82% |
+| Claude Sonnet 5 (max) | 41% | 17% | 13% | 40% | 61% | 82% |
+| DeepSeek V4 Pro 0813 (max) | 41% | 18% | 13% | 49% | 5% | 80% |
+| DeepSeek V4 Flash 0731 (max) | 39% | 17% | 14% | 40% | 8% | 80% |
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 360}}}%%
+xychart-beta
+    title "Terminal-Bench v2.1 pass rate (%), top effort per model"
+    x-axis ["fable-5.1 max", "astra high", "sol xhigh", "opus-5 max", "terra max", "g3.8-flash high", "grok-4.6 high", "muse-1.3 max", "g3.7-flash high", "fable-5", "kimi-k3 max", "glm-5.3 max", "glm-5.3-flash", "luna max", "sonnet-5 max", "dsv4-pro max", "dsv4-flash max"]
+    y-axis "pass rate (%)" 60 --> 100
+    bar [91, 90, 90, 89, 88, 88, 88, 86, 86, 85, 85, 84, 84, 81, 81, 79, 79]
+```
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 360}}}%%
+xychart-beta
+    title "AA-Briefcase, agentic knowledge work, (Elo - 500) / 2000 as %"
+    x-axis ["fable-5.1 max", "opus-5 max", "astra max", "muse-1.3 max", "grok-4.6 high", "fable-5", "glm-5.3 max", "kimi-k3 max", "sol max", "glm-5.3-flash", "sonnet-5 max", "terra max", "luna max", "dsv4-pro max", "dsv4-flash max", "g3.8-flash high", "g3.7-flash high"]
+    y-axis "score (%)" 0 --> 70
+    bar [58, 58, 53, 53, 52, 52, 51, 50, 49, 48, 43, 42, 42, 39, 38, 35, 31]
+```
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 360}}}%%
+xychart-beta
+    title "GDP.pdf All-pass (%), professional document reasoning"
+    x-axis ["astra max", "sol max", "fable-5.1 max", "muse-1.3 max", "terra max", "fable-5", "g3.7-flash high", "luna max", "opus-5 max", "kimi-k3 max", "g3.8-flash high", "grok-4.6 high", "dsv4-flash max", "sonnet-5 max", "dsv4-pro max", "glm-5.3-flash", "glm-5.3 max"]
+    y-axis "all-pass rate (%)" 0 --> 40
+    bar [33, 28, 26, 26, 26, 24, 24, 23, 22, 20, 19, 19, 14, 13, 13, 13, 12]
+```
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 360}}}%%
+xychart-beta
+    title "AA-Omniscience non-hallucination rate (%), higher means it abstains instead of guessing"
+    x-axis ["glm-5.3-flash", "glm-5.3 max", "muse-1.3 max", "grok-4.6 high", "sonnet-5 max", "astra max", "kimi-k3 max", "g3.8-flash high", "opus-5 max", "fable-5", "g3.7-flash high", "fable-5.1 max", "terra max", "sol max", "dsv4-flash max", "luna max", "dsv4-pro max"]
+    y-axis "non-hallucination rate (%)" 0 --> 80
+    bar [72, 70, 66, 66, 61, 49, 47, 45, 39, 36, 35, 27, 12, 8, 8, 7, 5]
+```
+
+How to read the per-evaluation tables:
+
+- **Effort buys different things on different evaluations.** Raising Astra from `medium` to `max` moves AA-Briefcase from 48% to 53% and GDP.pdf from 30% to 33%, but Terminal-Bench is flat at 88–90% across every level, including `low`. Sol `high` beats Sol `max` on SciCode. Do not assume the top effort is the best row for a coding stage; check the column.
+- **Knowledge-work agents and coding agents are different skills.** Gemini 3.8 Flash scores 88% on Terminal-Bench and 74% on DeepSWE, yet 35% on AA-Briefcase — the lowest of these rows. GLM-5.3-Flash is the opposite shape: 59% on GDPval, at the level of Opus 5 high, for $0.18 per Index task. Route by the column that matches the stage.
+- **Non-hallucination is a family trait, not an intelligence signal.** The Z.AI, Meta and xAI models abstain at 66–72%; Anthropic models sit at 27–61%; OpenAI's Sol, Luna and Terra and both DeepSeek rows sit at 5–12%. For an uncited research summary or a "does this API exist" question, a 7% model needs a verification tool node behind it regardless of its index score.
+- **Long context is not a differentiator at the top.** Every row except Kimi K3 (89%) sits at 79–85% on AA-LCR v1.1. Choose long-context stages on cost and on the accuracy or document columns instead.
+
+### Additional evaluations for catalog models
+
+Not Intelligence Index components. AA measures only some models on each; a blank means AA had no result for that configuration on 2026-09-05, not a zero. GPQA Diamond and MMMU-Pro are included because they remain on the model pages; GPQA is saturated (89–96% across every row) and no longer discriminates.
+
+| AA configuration | AutomationBench-AA (SaaS workflows) | Harvey LAB-AA (legal) | EnterpriseOps-Gym-AA | AA-AnalystAgent (spreadsheets) | IFBench (instruction following) | ITBench-AA (K8s incidents) | GPQA Diamond | MMMU-Pro |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Claude Fable 5.1 (max with fallback) | | 93% | | | | | 94% | |
+| Claude Fable 5 (with fallback) | 49% | 94% | 51% | 49% | 63% | | 93% | |
+| Claude Opus 5 (max) | | 93% | 47% | 54% | | | 93% | 85% |
+| Claude Sonnet 5 (max) | 39% | 90% | 45% | 46% | | | 91% | 77% |
+| GPT-6 Astra (max) | | | | | | | 96% | 87% |
+| GPT-5.6 Sol (max) | 51% | 87% | 43% | 48% | 73% | 56% | 94% | 83% |
+| GPT-5.6 Terra (max) | 46% | 85% | 38% | | 71% | 51% | 93% | 81% |
+| GPT-5.6 Luna (max) | 42% | 88% | 41% | | | 40% | 91% | 79% |
+| Muse Spark 1.3 (xhigh) | | 95% | | | | | 94% | 82% |
+| Grok 4.6 (high) | | | 48% | 41% | | | 95% | |
+| Kimi K3 (max) | 53% | 95% | 45% | 39% | | 48% | 94% | 81% |
+| GLM-5.3 (max) | | | 36% | | | | 92% | |
+| GLM-5.3-Flash | | | 33% | | | | 91% | |
+| Gemini 3.8 Flash (high) | 51% | | | | | | 95% | 86% |
+| Gemini 3.7 Flash (high) | 63% | 91% | | 60% | | | 95% | 85% |
+| DeepSeek V4 Pro 0813 (max) | | | 50% | | | | 93% | |
+| DeepSeek V4 Flash 0731 (max) | | | | | | | 91% | |
+
+Two rows worth knowing: Gemini 3.7 Flash leads AutomationBench-AA (63%) and AA-AnalystAgent (60%) among measured rows, so it is the SaaS-automation and spreadsheet candidate despite its weak AA-Briefcase; and Sol max leads IFBench (73%) and ITBench-AA (56%), which makes it the strict-format and incident-diagnosis candidate in the OpenAI family.
+
+### Coding Agent Index v1.4 is a different comparison
+
+The [Artificial Analysis Coding Agent Index](https://artificialanalysis.ai/agents/coding-agents) evaluates named **agent + model + settings** combinations, not interchangeable base-model rows. Its [methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking), retrieved 2026-09-05, identifies **v1.4**, current since August 2026. It equally weights DeepSWE, Terminal-Bench v2.1 and SWE-Atlas-QnA. Those components contain 113, 89 and 124 tasks respectively, each with three attempts per task. Per-evaluation pass@1 averages attempts within a task, then tasks within an evaluation. Reward-hacked Terminal-Bench attempts receive zero.
+
+Cost and execution time instead pool task attempts across the suite. Cost uses pay-per-token API pricing, including supported cache charges, not subscription-plan prices. Execution time is measured wall time; missing telemetry is excluded from the relevant average, not treated as zero. Agent defaults apply unless the row specifies other settings.
+
+The fourteen rows on the rendered leaderboard, read 2026-09-05:
+
+| Agent + model (settings) | Coding Agent Index | DeepSWE | Terminal-Bench v2.1 | SWE-Atlas-QnA | $/task | Wall time/task |
+| --- | --- | --- | --- | --- | --- | --- |
+| Claude Code + Fable 5.1 (max, with fallback) | **70** | 66 | 89 | **56** | $9.18 | 24.0 min |
+| Claude Code + Opus 5 (xhigh) | 68 | 60 | **89** | 55 | $8.17 | 23.7 min |
+| Muse Code + Muse Spark 1.3 (max) | 68 | **68** | 84 | 52 | $1.58 | 24.6 min |
+| Codex + GPT-6 Astra (max) | 67 | 67 | 83 | 51 | $4.72 | 26.8 min |
+| Muse Code + Muse Spark 1.3 (xhigh) | 64 | 67 | 82 | 44 | $1.62 | 12.8 min |
+| Grok Build + Grok 4.5 (high) | 64 | 60 | 84 | 48 | $2.44 | 15.5 min |
+| Kimi Code CLI + Kimi K3 | 63 | 64 | 88 | 37 | $3.08 | 24.1 min |
+| Claude Code + Qwen3.8 Max | 61 | 52 | 84 | 48 | $3.23 | 29.9 min |
+| Opencode + Gemini 3.8 Flash (high) | 61 | 62 | 84 | 38 | $2.04 | 11.9 min |
+| Codex + GPT-5.6 Luna (max) | 57 | 63 | 75 | 33 | $0.29 | 8.0 min |
+| Devin CLI + SWE-1.7 Lightning Max | 52 | 40 | 79 | 37 | $8.52 | 10.6 min |
+| Codex + DeepSeek V4 Flash 0731 (max) | 50 | 43 | 68 | 39 | $0.06 | 14.5 min |
+| Claude Code + GLM-5.2 | 43 | 29 | 72 | 29 | $1.91 | 25.1 min |
+| Cursor CLI + Composer 2.5 Fast | 38 | 16 | 68 | 31 | $0.56 | 7.9 min |
+
+```mermaid
+%%{init: {"xyChart": {"width": 1100, "height": 360}}}%%
+xychart-beta
+    title "AA Coding Agent Index v1.4 (agent + model)"
+    x-axis ["CC fable-5.1", "CC opus-5", "Muse spark-1.3 max", "Codex astra", "Muse spark-1.3 xhigh", "Grok Build 4.5", "Kimi CLI k3", "CC qwen3.8", "Opencode g3.8f", "Codex luna", "Devin swe-1.7", "Codex dsv4-flash", "CC glm-5.2", "Cursor composer"]
+    y-axis "index" 0 --> 80
+    bar [70, 68, 68, 67, 64, 64, 63, 61, 61, 57, 52, 50, 43, 38]
+```
+
+Three reads from the agent table:
+
+- **SWE-Atlas-QnA is where the Anthropic rows separate.** Fable 5.1 and Opus 5 score 55–56% on repository-understanding questions against 33–52% for every other row; on DeepSWE and Terminal-Bench they are inside the pack. If a stage is mostly reading and explaining code rather than patching it, that column is the one to weight.
+- **Muse Spark 1.3 is the value row.** Muse Code + Muse Spark 1.3 (max) ties Opus 5 on the index for $1.58 per task, and leads DeepSWE inside AA's harness at 68. Its `xhigh` row halves wall time to 12.8 minutes for four index points.
+- **Cheap and fast is a real trade.** Codex + Luna (max) at 57 costs $0.29 and finishes in 8.0 minutes; it sits three points behind Fable 5.1 on DeepSWE and loses its gap on SWE-Atlas-QnA and Terminal-Bench instead. Codex + DeepSeek V4 Flash at 50 costs $0.06 but trails on all three components.
+
+AA's DeepSWE component uses the DeepSWE dataset with the named agent. It is not the same experiment as Datacurve's `mini-swe-agent` leaderboard, and the two disagree: inside AA's harness Muse Code + Muse Spark 1.3 (68) edges Codex + Astra (67), while Datacurve's Best view has Astra [xhigh] at 74% and has not published Muse Spark 1.3 at all (its Muse Spark 1.2 [xhigh] row sits at 55%). Neither its component score nor its composite belongs in the [DeepSWE frontier](/models/pareto-efficiency).
+
+Earlier versions of these docs referred to a base-model **Coding Index** and **Agentic Index**. Neither is listed in the current [capability directory](https://artificialanalysis.ai/models/capabilities) or [capability methodology](https://artificialanalysis.ai/methodology/capability-indices) inspected on 2026-09-05. We therefore do not assign them a current version or silently rename either to Coding Agent Index. Use the named coding and agentic evaluations above instead.
+
+### Professional capability indices
+
+The current directory lists Finance & Accounting, Strategy & Ops, Legal, Healthcare & Medical, Engineering, and Economics. The [capability methodology](https://artificialanalysis.ai/methodology/capability-indices) specifies domain-dependent components and weights, rather than a single shared formula. It displays no version identifier. Use the matching domain index when its task mix fits your work.
+
+### Price, task cost and latency
+
+Read the [definitions](https://artificialanalysis.ai/methodology#definitions) and [API performance methodology](https://artificialanalysis.ai/methodology/performance-benchmarking), retrieved 2026-09-05, before comparing efficiency charts:
+
+- Token prices are USD per million native tokens. AA's blended price assumes cache-hit, input and output tokens in a **7:2:1** ratio. That synthetic mix is not your workflow's bill.
+- Intelligence Index cost per task uses actual token consumption, provider prices and typical measured cache hit rates, weighted by the index's evaluation weights. It is neither the total cost of running the suite nor DeepSWE dollars per task. The leaderboard's `$` column above is this value.
+- Output speed uses standardized `o200k_base` tokens after the first chunk. The default workload is **10k input tokens**; the usual displayed result is the median over **72 hours**. The **100k** workload instead uses a **14-day** median. These are API measurements, not coding-agent completion times.
+- Time to first token can mean the first reasoning token. Time to first answer token includes thinking time. Compare these separately from output speed when interactive latency matters.
+- The homepage's Intelligence Index **Time per Task** estimates weighted decode time from output tokens and speed; it excludes TTFT and overhead. Do not call it measured end-to-end wall time. The Coding Agent Index execution-time metric does measure wall time.
+
+## Role to benchmark map
+
+| Role | Primary evidence | Cross-check | Measured leaders on 2026-09-05 |
+| --- | --- | --- | --- |
+| Debugger / coding worker | Datacurve DeepSWE pass@1, cost and steps | AA Terminal-Bench v2.1; Coding Agent Index with the actual agent identified | DeepSWE: astra [xhigh], gemini-3.8-flash [high], opus-5 [max] at 74%; Terminal-Bench: Fable 5.1 max 91%, Astra high 90% |
+| Reviewer / judgment gate | Task-specific Atomic evals and DeepSWE for code judgments | AA-Briefcase and knowledge reliability for broader judgments; SWE-Atlas-QnA for code-reading judgments | AA-Briefcase: Fable 5.1 max and Opus 5 max 58%; SWE-Atlas-QnA: Claude Code + Fable 5.1 56% |
+| Planner / orchestrator | AA-Briefcase, GDPval-AA v2 | 𝜏³-Banking for tool interaction | GDPval: Fable 5.1 max 63%, Opus 5 max 62%, Muse Spark 1.3 max 61%; 𝜏³: Muse Spark 1.3 max 52%, Grok 4.6 high 51% |
+| Research | AA-LCR v1.1, GDP.pdf | AA-Omniscience accuracy and non-hallucination | GDP.pdf: Astra max 33%; LCR: Kimi K3 max 89%; accuracy: Fable 5.1 max 67%; non-hallucination: GLM-5.3-Flash 72% |
+| Domain-specific work | Matching AA capability index | Its component evaluations | Legal: Muse Spark 1.3 xhigh and Kimi K3 max 95% on Harvey LAB-AA; SaaS automation: Gemini 3.7 Flash high 63%; incidents: Sol max 56% on ITBench-AA |
+
+See [Model Selection](/models/model-selection) for a small dated shortlist and production effort guidance. Benchmark settings are measurement configurations, not instructions to raise every role's effort.
+
+## Keeping the docs fresh
+
+1. Record each source's retrieval date separately from its publication or snapshot date. Follow the rendered charts and methodology, not just an old article's score.
+2. Preserve exact model, reasoning configuration, agent, benchmark version and units. A changed index or agent can change the ranking without a new model release.
+3. Say **unmeasured on the named benchmark and date**. Missing text extraction is not evidence of absence; inspect the rendered page. Never transfer a predecessor's score.
+4. Check the configured catalog and live provider access separately. These docs do not change runtime routing or model defaults.
+5. AA's per-evaluation numbers are only in client-rendered Recharts bar charts, so a plain HTTP fetch returns headings without values. To refresh them, open the model page in a headless browser, scroll the whole page so every chart animates in, then read each chart's `foreignObject` labels (model names, in bar order) alongside its `svg text` nodes (values, in the same order). The DeepSWE leaderboard and the AA evaluation leaderboards (AA-Briefcase, GDPval-AA v2) render as text and fetch cleanly.
+
+## Related
+
+- [Model Selection](/models/model-selection)
+- [Pareto Efficiency](/models/pareto-efficiency)