@event4u/agent-config 2.20.0 → 2.21.0

package/docs/contracts/compression-default-kill-criterion.md

@@ -5,14 +5,16 @@ keep-beta-until: 2026-08-14
 
 # Compression default — kill-criterion
 
-> **Status:** parked, criterion-deferred · **Owner:** `step-4-measurement-and-benchmark.md`
-> closeout phase · **Source:** [`council-synthesis.md` § 7](../../agents/audit-2026-05-14-north-star/council-synthesis.md)
+> **Status:** v1-measured · criterion not met · default stays `off` · **Owner:** `step-16-caveman-substance.md`
+> Phase 1 closeout · **Sources:** [`bench/reports/caveman-v1.md`](../../bench/reports/caveman-v1.md) ·
+> [`council-synthesis.md` § 7](../../agents/audit-2026-05-14-north-star/council-synthesis.md) ·
+> [`caveman-v1-kc-verdict.json`](../../agents/council-responses/caveman-v1-kc-verdict.json) <!-- council-ref-allowed: ADR decision trace for v1 kill-criterion verdict -->
 
 ## Rule
 
 ```
-DEFAULT STAYS OFF UNTIL `task bench` PRODUCES A NUMBER.
-DECISION OWNED BY step-4 CLOSEOUT, NOT BY THIS DOC OR BY step-99.
+DEFAULT STAYS OFF UNTIL `task bench -- --caveman` PRODUCES A POSITIVE vs_terse MEDIAN.
+DECISION OWNED BY THE NEXT BENCH CLOSEOUT, NOT BY THIS DOC.
 ```
 
 1. **Current state.** `caveman.speak_scope` defaults `off`. Carve-outs
@@ -21,49 +23,94 @@ DECISION OWNED BY step-4 CLOSEOUT, NOT BY THIS DOC OR BY step-99.
    [`caveman-speak`](../../.agent-src.uncompressed/rules/caveman-speak.md)
    but the feature is non-promoted: no skill recommends turning it on,
    no preset enables it, no profile depends on it.
-2. **Baseline window.** 60 days from the first green run of
-   `task bench` against the locked 25-prompt corpus
-   (`step-4-measurement-and-benchmark.md`
-   Phase 2). The corpus, the model, and the cost-tracker are frozen
-   for the window; mid-window changes restart the clock.
-3. **Decision points.** After the window closes, `step-4` closeout
-   reads `docs/parity/bench.json` and applies exactly one of:
-
-   | Measured tokens saved | Quality regression on corpus | Verdict |
+2. **Baselines.** Every published `bench/reports/caveman-v<N>.{json,md}`
+   measures three arms (`compressed` · `terse-control` ·
+   `uncompressed`) and reports two savings columns:
+   - `vs_raw` — median savings against the uncompressed arm.
+   - `vs_terse` — **load-bearing** median savings against the
+     `Answer concisely.` terse-control arm. `vs_raw` is inflated by the
+     carve-out-tax-free pure-prose case and is **not** the gate metric.
+3. **Decision table.** Read the latest `bench/reports/caveman-v<N>.md`
+   and apply exactly one of:
+
+   | Measured `vs_terse` median | Quality regression on corpus | Verdict |
    |---|---|---|
-   | < 30 % | any | **Deprecate** — remove `caveman-speak` rule, archive `caveman-compress` script, retire `caveman.*` settings keys with a one-release deprecation window |
-   | ≥ 30 % | < 5 % | **Flip default on** — `caveman.speak_scope` defaults to a non-`off` value, carve-outs stay, statusline surfaces lifetime tokens saved |
-   | ≥ 30 % | ≥ 5 % | **Hold** — repeat the window once with tuned intensity ladder; second hold → deprecate |
+   | < 0 % | any | **Criterion not met — defer.** Keep default `off`. No telemetry multiplier. Next move owned by the corpus-widening / methodology-revision step that produces `caveman-v<N+1>`. |
+   | 0 % – < 30 % | any | **Hold.** Keep default `off`. Authorised follow-up: widen corpus or tune carve-out share; no default flip. |
+   | ≥ 30 % | < 5 % | **Flip default on** — `caveman.speak_scope` defaults to a non-`off` value (separate roadmap), carve-outs stay, statusline surfaces lifetime tokens saved. |
+   | ≥ 30 % | ≥ 5 % | **Hold** — repeat the window once with tuned intensity ladder; second hold → deprecate. |
 
    "Quality regression" = host-side rubric on the corpus per
-   `step-4-measurement-and-benchmark.md` Phase 3. Numbers checked into
-   `docs/parity/bench.json` as the decision artefact.
+   `benchmark-report-schema.md`. Numbers checked into the published
+   `caveman-v<N>.json` as the decision artefact.
 4. **No interim flip.** The default does not move on anecdote,
-   gut feeling, or a single benchmark snapshot. The 60-day window and
-   the table above are the only path to a default change.
+   gut feeling, or a single positive prompt. Only a published
+   `caveman-v<N>` report with a `vs_terse` median in the "Flip" row
+   above authorises a default change, under a follow-up roadmap.
+
+## v1 verdict (2026-05-16)
+
+[`bench/reports/caveman-v1.md`](../../bench/reports/caveman-v1.md)
+landed 30 calls · $0.0805 · 0 errors · `claude-sonnet-4-5`:
+
+| Metric | Median | p10 | p90 |
+|---|---:|---:|---:|
+| `vs_raw` savings | +23.51 % | -18.29 % | +52.53 % |
+| **`vs_terse` savings** | **−9.27 %** | **−109.85 %** | +51.32 % |
+| Realised carve-out share (compressed arm) | 30.67 % | — | — |
+
+Per row 1 of the table, the v1 verdict is **criterion not met — defer**.
+Default stays `off`; no telemetry multiplier ships; no rule retirement
+in this roadmap. Wins exist only on pure-prose prompts (caveman-09
++50.5 %, caveman-10 +58.4 %); carve-out-heavy prompts drag the median
+negative (caveman-04 path-list −108 %, caveman-06 mode-marker −123 %).
+
+### Council split (recorded, not decisive)
+
+Council run [`caveman-v1-kc-verdict.json`](../../agents/council-responses/caveman-v1-kc-verdict.json) <!-- council-ref-allowed: ADR decision trace for v1 kill-criterion verdict -->
+(2 members · 1 round · $0.0514 actual) split:
+
+- **`claude-sonnet-4-5`** → Decision A.1 (deprecate now) + Decision B.3
+  (suspend telemetry). Reasoning: the roadmap pinned `vs_terse` as
+  load-bearing; the data falsified it; retreating to `vs_raw` is
+  post-hoc rationalisation.
+- **`gpt-4o`** → Decision A.3 (hold + re-bench with widened corpus +
+  revised terse-control prompt) + Decision B.2 (per-category
+  multipliers, suppress negatives). Reasoning: 10 prompts is a
+  razor-thin sample; the terse-control prompt may under-compress; the
+  carve-out validator (Phase 4) is not yet shipped, so we are
+  measuring a half-implemented feature.
+
+**Synthesis (criterion-not-met + defer).** Both members agreed `vs_terse`
+is the right gate. Neither's strongest path is taken in full inside
+step-16: deprecation is reserved for a follow-up roadmap once v2 confirms
+v1; re-bench is reserved for a follow-up roadmap with the methodology
+revision the council requested. Step-16 ships the infrastructure (corpus,
+bench arm, validator), records the v1 verdict, suspends the telemetry
+multiplier, and hands the deprecate-vs-rebench call to the v2 roadmap.
 
 ## Why this is parked, not decided
 
-The council split (Opus = remove now, o1 = measure-then-decide) is
-real. Either branch is wrong-shaped without numbers. The kill-criterion
-gives the audit a deterministic resolution path and stops every
-downstream roadmap from re-litigating compression on every PR.
+The 2026-05-14 council split (Opus = remove now, o1 = measure-then-decide)
+predated v1 numbers. The 2026-05-16 council split (Sonnet = deprecate now,
+GPT-4o = re-bench) is informed by v1 but disagrees on which methodological
+weakness is decisive. The kill table above gives every future bench run a
+deterministic resolution path and stops every downstream roadmap from
+re-litigating compression on every PR.
 
 ## Cross-references
 
-- ``step-99-north-star-restructure.md` § Phase 4`
-  — parks this criterion, does not decide.
-- `step-4-measurement-and-benchmark.md`
-  — owns `task bench`, the corpus, and the closeout that applies the
-  table above.
-- `step-10-caveman-parity.md`
-  — implements the carve-outs and the statusline integration the
-  "flip default on" branch depends on; blocks the default flip until
-  acceptance is green.
+- [`bench/reports/caveman-v1.md`](../../bench/reports/caveman-v1.md)
+  — v1 measurement; canonical baseline this doc cites.
+- [`docs/benchmarks.md`](../benchmarks.md)
+  — cadence + when the next bench run is mandatory.
+- [`caveman-telemetry`](caveman-telemetry.md)
+  — multiplier contract; records the suspended state v2 must lift.
 - [`caveman-speak`](../../.agent-src.uncompressed/rules/caveman-speak.md)
   — runtime rule; reads `caveman.speak_scope` from settings.
 
 ## Done
 
-This doc exists to keep the decision visible. It is **not** an action
-item. `step-4` closeout closes the loop.
+This doc reflects the v1 verdict. It is **not** an action item. The next
+bench closeout (against `caveman-v2` once a widened corpus or revised
+methodology is shipped) closes the loop.

package/docs/contracts/cost-summary-schema.md

@@ -0,0 +1,107 @@
+---
+stability: beta
+keep-beta-until: 2026-08-15
+---
+
+# cost-summary schema (`cost-summary/v1`)
+
+Stable JSON contract for inter-tool consumption of cost-tracking data
+emitted by [`scripts/cost_summary.py`](../../scripts/cost_summary.py).
+Schema-versioned so downstream consumers can pin and migrate explicitly.
+
+Design reference: Ruflo `scripts/summary.mjs` (upstream cite). Our shape
+diverges to align with the local `agents/cost-tracking/sessions.jsonl`
+fields and the caveman-suspended-multiplier contract.
+
+## Envelope
+
+```json
+{
+  "schema_version": "cost-summary/v1",
+  "generated_at": "2026-05-16T23:45:00Z",
+  "totals": { ... },
+  "by_session": [ ... ],
+  "by_conversation": [ ... ],
+  "by_model": [ ... ]
+}
+```
+
+| Field | Type | Notes |
+|---|---|---|
+| `schema_version` | string | Pinned to `cost-summary/v1`. Downstream consumers MUST refuse unknown versions. |
+| `generated_at` | string (ISO-8601 UTC, `Z` suffix) | Emit time. |
+| `totals` | object | Lifetime aggregates — see `totals` below. |
+| `by_session` | array | Per `sessionId` row; ordered by `sessionId` ascending. |
+| `by_conversation` | array | Per `conversation_id` row; ordered by `conversation_id` ascending. |
+| `by_model` | array | Per `model` row; ordered by `model` ascending. |
+
+## `totals` shape
+
+```json
+{
+  "sessions": 123,
+  "total_cost_usd": 1.2345,
+  "input_tokens": 100000,
+  "output_tokens": 50000,
+  "caveman_delta_tokens": 0,
+  "caveman_multiplier_version": "v1",
+  "caveman_multiplier_active": false
+}
+```
+
+`caveman_delta_tokens` is always `0` while
+`caveman_multiplier_active == false` — see
+[`caveman-telemetry.md`](caveman-telemetry.md) for the suspension contract.
+
+## `by_session` / `by_conversation` row shape
+
+```json
+{
+  "key": "<sessionId or conversation_id>",
+  "sessions": 12,
+  "total_cost_usd": 0.4567,
+  "input_tokens": 8000,
+  "output_tokens": 4500,
+  "caveman_delta_tokens": 0
+}
+```
+
+The `key` field is the grouping identifier; consumers identify the
+group by inspecting which array the row lives in.
+
+## `by_model` row shape
+
+```json
+{
+  "model": "claude-3-5-sonnet-20241022",
+  "sessions": 12,
+  "total_cost_usd": 0.4567,
+  "input_tokens": 8000,
+  "output_tokens": 4500
+}
+```
+
+`by_model` omits caveman fields — the multiplier is dialect-scoped, not
+model-scoped.
+
+## Stability guarantees
+
+- **Field additions** are **non-breaking**: consumers MUST ignore unknown fields.
+- **Field removals or renames** bump the `schema_version` minor (`v1` → `v2`).
+- **Type changes** bump the major (`v1.*` → `v2.0`).
+- Downstream consumers SHOULD pin to a specific `schema_version` and
+  refuse unknown ones; the pin is the migration boundary.
+
+## Downstream consumers
+
+- `agent-status` skill — surfaces lifetime / current-conversation slice.
+- Future `cost-export-to-monitoring` scripts (deferred; trigger:
+  consumer request) would wrap this JSON to push to Prometheus / OTLP.
+
+## See also
+
+- [`caveman-telemetry.md`](caveman-telemetry.md) — defines the
+  `caveman_*` fields and the suspended-multiplier contract.
+- [`scripts/cost_summary.py`](../../scripts/cost_summary.py) — implementation.
+- [`scripts/cost_by_conversation.py`](../../scripts/cost_by_conversation.py) — narrower per-conversation lens with the same JSONL source.
+- [`scripts/caveman_stats.py`](../../scripts/caveman_stats.py) — caveman-only delta lens with the same JSONL source.

package/docs/contracts/file-ownership-matrix.json

@@ -1800,6 +1800,12 @@
       "load_context": [],
       "load_context_eager": []
     },
+    ".agent-src.uncompressed/skills/compress-memory/SKILL.md": {
+      "kind": "skill",
+      "rule_type": null,
+      "load_context": [],
+      "load_context_eager": []
+    },
     ".agent-src.uncompressed/skills/content-funnel-design/SKILL.md": {
       "kind": "skill",
       "rule_type": null,
@@ -6396,6 +6402,13 @@
       "via": "self",
       "depth": 0
     },
+    {
+      "source": ".agent-src.uncompressed/rules/caveman-speak.md",
+      "target": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "type": "READ_ONLY",
+      "via": "body_link",
+      "depth": 1
+    },
     {
       "source": ".agent-src.uncompressed/rules/cli-output-handling.md",
       "target": ".agent-src.uncompressed/rules/cli-output-handling.md",
@@ -8048,6 +8061,34 @@
       "via": "self",
       "depth": 0
     },
+    {
+      "source": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "target": ".agent-src.uncompressed/rules/caveman-speak.md",
+      "type": "READ_ONLY",
+      "via": "body_link",
+      "depth": 1
+    },
+    {
+      "source": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "target": ".agent-src.uncompressed/rules/role-mode-adherence.md",
+      "type": "READ_ONLY",
+      "via": "body_link",
+      "depth": 1
+    },
+    {
+      "source": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "target": ".agent-src.uncompressed/skills/agents-md-thin-root/SKILL.md",
+      "type": "READ_ONLY",
+      "via": "body_link",
+      "depth": 1
+    },
+    {
+      "source": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "target": ".agent-src.uncompressed/skills/compress-memory/SKILL.md",
+      "type": "WRITE",
+      "via": "self",
+      "depth": 0
+    },
     {
       "source": ".agent-src.uncompressed/skills/content-funnel-design/SKILL.md",
       "target": ".agent-src.uncompressed/skills/activation-design/SKILL.md",

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@event4u/agent-config",
-    "version": "2.20.0",
+    "version": "2.21.0",
     "description": "Shared agent configuration \u2014 skills, rules, commands, guidelines, and templates for AI coding tools",
     "license": "MIT",
     "private": false,

package/scripts/_lib/bench_caveman.py

@@ -0,0 +1,273 @@
+# Caveman compression bench — step-16 Phase 1 Step 4.
+#
+# Three-arm live bench against bench/corpora/caveman/prompts.yaml:
+#   compressed     — system prompt embeds caveman-speak rule (aggressive).
+#   terse_control  — system prompt = "Answer concisely. …" (carve-out-free baseline).
+#   uncompressed   — generic helpful-assistant system prompt.
+#
+# Token counts come from Anthropic API `usage` (authoritative). Carve-out
+# share is measured via regex extraction on the reply text; chars/4 yields
+# an estimated carve-out-token figure for the carve-out-tax accounting.
+#
+# Cost-touch: 10 prompts × 3 arms × claude-sonnet-4-5 (~$3/M in, ~$15/M out).
+"""Caveman compression bench runner."""
+from __future__ import annotations
+
+import re
+import statistics
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+# ── system prompts per arm ──────────────────────────────────────────────
+
+SYSTEM_PROMPT_COMPRESSED = """You are speaking in CAVEMAN-SPEAK mode (speak_scope=aggressive).
+
+Compress all body prose to caveman grammar:
+- Drop articles (the, a, an).
+- Drop linking auxiliaries (is, are, was, be) where unambiguous.
+- Drop pronouns when context is clear.
+- Keep nouns, verbs, key adjectives, negation, numbers.
+- Example: "I will now check the file and see if it exists" -> "Check file. Exists?"
+
+Carve-outs — preserve BYTE-FOR-BYTE (do NOT compress these):
+1. Triple-backtick code/literal blocks (any language, including ALL-CAPS Iron-Law fences).
+2. Numbered-options lines matching ^\\d+\\.\\s + a **Recommendation:** label.
+3. Backtick spans (file paths, command names, identifiers).
+4. Status markers: lines starting with ❌, ⚠️, or ✅.
+5. Mode markers.
+6. Markdown tables.
+7. Deliverables (PR titles, commit messages, ticket summaries, articles, the prompt
+   line of any single question asked to the user).
+
+Apply caveman compression aggressively to every other prose surface."""
+
+SYSTEM_PROMPT_TERSE = (
+    "Answer concisely. Skip preamble. Do not restate the question. "
+    "Avoid filler phrases ('Let me', 'Here is', 'I will'). Get to the answer."
+)
+
+SYSTEM_PROMPT_UNCOMPRESSED = (
+    "You are a helpful AI assistant. Answer the user's question clearly and completely."
+)
+
+ARMS: tuple[str, ...] = ("compressed", "terse_control", "uncompressed")
+ARM_SYSTEM_PROMPT: dict[str, str] = {
+    "compressed": SYSTEM_PROMPT_COMPRESSED,
+    "terse_control": SYSTEM_PROMPT_TERSE,
+    "uncompressed": SYSTEM_PROMPT_UNCOMPRESSED,
+}
+
+# ── carve-out detection ────────────────────────────────────────────────
+
+_RE_TRIPLE_BACKTICK = re.compile(r"```[\s\S]*?```")
+_RE_BACKTICK_SPAN = re.compile(r"`[^`\n]+`")
+_RE_NUMBERED_LINE = re.compile(r"^>?\s*\d+\.\s.*$", re.MULTILINE)
+_RE_STATUS_LINE = re.compile(r"^(❌|⚠️|✅).*$", re.MULTILINE)
+_RE_TABLE_LINE = re.compile(r"^\s*\|.*\|\s*$", re.MULTILINE)
+_RE_RECOMMENDATION = re.compile(r"^\*\*(Recommendation|Empfehlung):\*\*.*$", re.MULTILINE)
+
+
+def carve_out_chars(text: str) -> int:
+    """Sum byte-length of every carve-out region (union, no double-count)."""
+    if not text:
+        return 0
+    mask = bytearray(len(text))
+    for pattern in (
+        _RE_TRIPLE_BACKTICK, _RE_BACKTICK_SPAN, _RE_NUMBERED_LINE,
+        _RE_STATUS_LINE, _RE_TABLE_LINE, _RE_RECOMMENDATION,
+    ):
+        for m in pattern.finditer(text):
+            for i in range(m.start(), m.end()):
+                mask[i] = 1
+    return sum(mask)
+
+
+# ── data shapes ────────────────────────────────────────────────────────
+
+@dataclass
+class ArmResult:
+    arm: str
+    text: str
+    input_tokens: int
+    output_tokens: int
+    latency_ms: int
+    output_chars: int
+    carve_out_chars: int
+    error: str | None = None
+
+    @property
+    def realised_carve_out_pct(self) -> float:
+        return self.carve_out_chars / self.output_chars if self.output_chars else 0.0
+
+
+@dataclass
+class PromptResult:
+    id: str
+    category: str
+    expected_carve_out_pct: float
+    arms: dict[str, ArmResult] = field(default_factory=dict)
+
+    @property
+    def savings_vs_raw(self) -> float | None:
+        c = self.arms.get("compressed")
+        u = self.arms.get("uncompressed")
+        if not c or not u or u.output_tokens == 0:
+            return None
+        return 1.0 - (c.output_tokens / u.output_tokens)
+
+    @property
+    def savings_vs_terse(self) -> float | None:
+        c = self.arms.get("compressed")
+        t = self.arms.get("terse_control")
+        if not c or not t or t.output_tokens == 0:
+            return None
+        return 1.0 - (c.output_tokens / t.output_tokens)
+
+
+# ── corpus + runner ────────────────────────────────────────────────────
+
+def load_corpus(corpus_path: Path) -> list[dict[str, Any]]:
+    """Read bench/corpora/caveman/prompts.yaml → list of prompt dicts."""
+    data = yaml.safe_load(corpus_path.read_text(encoding="utf-8")) or {}
+    prompts = data.get("prompts") or []
+    if not prompts:
+        raise ValueError(f"empty corpus: {corpus_path}")
+    return prompts
+
+
+def run_arm(
+    client: Any,
+    arm: str,
+    user_prompt: str,
+    *,
+    max_tokens: int = 1024,
+) -> ArmResult:
+    """Invoke one arm against the live API. Returns ArmResult including text."""
+    t0 = time.monotonic()
+    system = ARM_SYSTEM_PROMPT[arm]
+    try:
+        resp = client.ask(system, user_prompt, max_tokens=max_tokens)
+    except Exception as exc:  # noqa: BLE001
+        latency_ms = int((time.monotonic() - t0) * 1000)
+        return ArmResult(arm=arm, text="", input_tokens=0, output_tokens=0,
+                         latency_ms=latency_ms, output_chars=0, carve_out_chars=0,
+                         error=str(exc))
+    return ArmResult(
+        arm=arm, text=resp.text or "",
+        input_tokens=int(resp.input_tokens or 0),
+        output_tokens=int(resp.output_tokens or 0),
+        latency_ms=int(resp.latency_ms or (time.monotonic() - t0) * 1000),
+        output_chars=len(resp.text or ""),
+        carve_out_chars=carve_out_chars(resp.text or ""),
+        error=resp.error,
+    )
+
+
+# ── aggregation ────────────────────────────────────────────────────────────
+
+def _stats(values: list[float]) -> dict[str, float]:
+    """Median / p10 / p90 / stdev / n on a list of floats. Empty → zeros."""
+    if not values:
+        return {"n": 0, "median": 0.0, "p10": 0.0, "p90": 0.0, "stdev": 0.0}
+    s = sorted(values)
+    n = len(s)
+    def _pct(p: float) -> float:
+        if n == 1:
+            return s[0]
+        k = (n - 1) * p
+        lo, hi = int(k), min(int(k) + 1, n - 1)
+        return s[lo] + (s[hi] - s[lo]) * (k - lo)
+    return {
+        "n": n,
+        "median": statistics.median(s),
+        "p10": _pct(0.10),
+        "p90": _pct(0.90),
+        "stdev": statistics.pstdev(s) if n > 1 else 0.0,
+    }
+
+
+def aggregate_results(results: list[PromptResult]) -> dict[str, Any]:
+    """Compute median/p10/p90 for compression metrics across the corpus."""
+    vs_raw = [r.savings_vs_raw for r in results if r.savings_vs_raw is not None]
+    vs_terse = [r.savings_vs_terse for r in results if r.savings_vs_terse is not None]
+    realised_carve_pct = [
+        r.arms["compressed"].realised_carve_out_pct
+        for r in results if "compressed" in r.arms and r.arms["compressed"].output_chars
+    ]
+    expected_carve_pct = [r.expected_carve_out_pct for r in results]
+
+    per_arm_tokens: dict[str, list[int]] = {a: [] for a in ARMS}
+    for r in results:
+        for arm in ARMS:
+            ar = r.arms.get(arm)
+            if ar:
+                per_arm_tokens[arm].append(ar.output_tokens)
+
+    return {
+        "savings_vs_raw": _stats(vs_raw),
+        "savings_vs_terse": _stats(vs_terse),
+        "realised_carve_out_pct": _stats(realised_carve_pct),
+        "expected_carve_out_pct": _stats(expected_carve_pct),
+        "output_tokens": {
+            arm: _stats([float(v) for v in per_arm_tokens[arm]]) for arm in ARMS
+        },
+    }
+
+
+def compute_cost(results: list[PromptResult], pricing: dict[str, float]) -> dict[str, Any]:
+    """Sum input/output tokens across all arms; cost from per-1M pricing dict."""
+    totals = {"input_tokens": 0, "output_tokens": 0, "calls": 0, "errors": 0}
+    per_arm: dict[str, dict[str, int]] = {a: {"input_tokens": 0, "output_tokens": 0, "calls": 0} for a in ARMS}
+    for r in results:
+        for arm, ar in r.arms.items():
+            totals["input_tokens"] += ar.input_tokens
+            totals["output_tokens"] += ar.output_tokens
+            totals["calls"] += 1
+            if ar.error:
+                totals["errors"] += 1
+            per_arm[arm]["input_tokens"] += ar.input_tokens
+            per_arm[arm]["output_tokens"] += ar.output_tokens
+            per_arm[arm]["calls"] += 1
+    cost_usd = (
+        totals["input_tokens"] / 1e6 * pricing.get("input", 0.0)
+        + totals["output_tokens"] / 1e6 * pricing.get("output", 0.0)
+    )
+    totals["total_cost_usd"] = round(cost_usd, 6)
+    return {"totals": totals, "per_arm": per_arm}
+
+
+# ── orchestrator ───────────────────────────────────────────────────────────
+
+def run_caveman_bench(
+    client: Any,
+    corpus_path: Path,
+    *,
+    max_prompts: int | None = None,
+    max_tokens: int = 1024,
+    on_progress: Any = None,
+) -> list[PromptResult]:
+    """Run all three arms over the corpus. Returns per-prompt results."""
+    prompts = load_corpus(corpus_path)
+    if max_prompts:
+        prompts = prompts[:max_prompts]
+    results: list[PromptResult] = []
+    total = len(prompts) * len(ARMS)
+    done = 0
+    for p in prompts:
+        pr = PromptResult(
+            id=str(p["id"]),
+            category=str(p.get("category", "unknown")),
+            expected_carve_out_pct=float(p.get("expected_carve_out_pct", 0.0)),
+        )
+        for arm in ARMS:
+            ar = run_arm(client, arm, str(p["prompt"]), max_tokens=max_tokens)
+            pr.arms[arm] = ar
+            done += 1
+            if on_progress:
+                on_progress(done, total, pr.id, arm, ar)
+        results.append(pr)
+    return results