@ssheleg/agent-stack 0.17.1 → 0.18.0

package/CHANGELOG.md

@@ -1,3 +1,51 @@
+## v0.18.0 — how many runs before a difference is real, and the trajectory rule between its two measured edges
+
+`agent-evals` shipped 315 lines of doctrine about **what** to assert with **no
+`references/` directory at all**, and said nothing anywhere about how many times to run
+anything. A skill whose whole job is answering *did it get better* had no way to say
+whether a number was a result or noise.
+
+**`references/statistics.md`** is that layer, and every figure in it was recomputed rather
+than quoted:
+
+- `SE(p) = √(p(1−p)/n)` — at n=100, p=0.70 the 95% band is **±8.98 pp**, so a 73%-vs-70%
+  comparison on a hundred cases is a number inside its own noise. Error falls as `1/√n`,
+  which makes the remedy *more tasks*, not more argument.
+- **`pass@k` and `pass^k` differ by 91 points on the same agent** — at p=0.6, k=5 they are
+  99.0% and 7.8%. The first is a capability ceiling a human picks from; the second is what
+  a payment or a permission change needs. An operation with side effects may not "retry
+  until it works", so `pass@k` is not available to it as a metric at all.
+- **Trials are not independent, and the published data proves it.** τ-bench's airline
+  Pass^k for claude-3-5-sonnet runs 0.460 → 0.326 → 0.263 → 0.225, where independence from
+  Pass^1 would predict 0.460 → 0.212 → 0.097 → 0.045. Successes cluster by task, not by
+  trial. So `pass^k` cannot be computed from `pass^1`, and Anthropic's `0.75³ ≈ 42%` is the
+  right shape for an argument and the wrong number for a gate. **This one is ours** — it
+  came out of recomputing the table rather than restating it.
+- Pairing on the same tasks and the same 3–5 seeds with McNemar or a paired bootstrap,
+  because the task-difficulty variance the previous point measures is exactly what pairing
+  removes.
+- **The harness is a variable**: 6 pp between the most- and least-resourced setups on
+  Terminal-Bench 2.0 (p<0.01), moving within noise from 1× to 3× (p=0.40) and lifting ~4 pp
+  from 3× to uncapped — because generous headroom lets the agent attempt strategies a tight
+  cap forbids. Two caps measure two agents. The remedy is a floor *and* a ceiling.
+- A ladder for what a given piece of evidence authorises next, ending on the rule people
+  skip: **4/4 on a slice is not 100% system-wide.**
+
+**§5's trajectory rule moved between two measurements rather than being deleted.** It read
+*"Judge the trajectory, not just the answer. Right tools, right order, right arguments."*
+Anthropic calls exact tool-order assertions *"too rigid … agents regularly find valid
+approaches that eval designers didn't anticipate"*, with a worked case of an agent that
+solved a τ²-bench booking task through a policy loophole and failed the eval while serving
+the user better. But the opposite edge is measured too: a grader blind to the trajectory
+misses **44% of safety violations and 13% of robustness failures**. So the rule now reads
+*read the trajectory; do not match it* — assert what was produced and what changed, and use
+the trajectory as a **set and a forbidden list** for the claims an outcome cannot carry.
+§2's axis table lost its `→` sequence example for the same reason.
+
+The single-step example keeps its *"must call `find_meeting_times` first"*, with a sentence
+saying why: at that granularity the fixture **is** one decision, so ordering is the subject
+rather than a proxy for it. Across a trajectory it stops being one.
+
 # Changelog
 
 ## v0.17.1 — the evals run for the first time, and the tails go to zero

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.17.1",
+  "version": "0.18.0",
   "scripts": {
     "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
   },

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "agent-stack",
   "displayName": "Agent Stack",
   "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-  "version": "0.17.1",
+  "version": "0.18.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -63,6 +63,10 @@ point — tool name, argument shape.
 > "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`,
 > `schedule_meeting` and `send_email` available, must call `find_meeting_times` first.
 
+This is the one granularity where *first* is a legitimate assertion: the fixture is a
+single decision, so the ordering claim is the subject rather than a proxy for it. Across a
+whole trajectory it stops being one — see §5.
+
 Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These
 break on a graph refactor, and a suite that fails on every refactor gets deleted.
 
@@ -72,7 +76,7 @@ Assert on three axes at once, with three different mechanisms:
 
 | Axis | Assert | With |
 |---|---|---|
-| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers |
+| Trajectory | what the run **must not** do, and what it must have touched — never the order | set/subset matchers, forbidden-call lists |
 | Final response | quality, tone, policy compliance | rubric or judge |
 | **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect |
 
@@ -147,8 +151,20 @@ That is what makes a judge reproducible and a disagreement resolvable.
 assertions, tool-call correctness — all deterministic, all faster and cheaper than a model
 call. Send to a judge only what cannot be decided by code.
 
-**Judge the trajectory, not just the answer.** Right tools, right order, right arguments.
-An agent that reaches a correct answer through three wrong tool calls is a latent outage.
+**Read the trajectory; do not match it.** An agent that reaches a correct answer through
+three wrong tool calls is a latent outage — and asserting the *sequence* to catch that is
+measurably the wrong instrument. Anthropic names the instinct and rejects its strict form:
+exact tool-order assertions are *"too rigid and results in overly brittle tests, as agents
+regularly find valid approaches that eval designers didn't anticipate"*, and the worked
+case is an agent that solved a τ²-bench booking task through a policy loophole, failing
+the eval as written while serving the user better. Grade **what was produced and what
+changed**, and let the path vary.
+
+The opposite edge is measured too, so this is not "grade the final answer": a grader blind
+to the trajectory misses **44% of safety violations and 13% of robustness failures**,
+because a policy breach on the way to a correct result leaves no trace in the outcome. Use
+the trajectory for the claims the outcome cannot carry — a forbidden call, a missing
+confirmation, a secret read — as a **set and a forbidden list**, never as an order.
 
 **Calibrate the judge before trusting it.** Collect human labels on the same traces,
 measure agreement, iterate the judge prompt until agreement is high — *then* let it score
@@ -304,6 +320,16 @@ None of the above runs without these, and they are the part people skip:
 - [ ] Every production failure minimised into a permanent fixture
 - [ ] Annotation queue with filters, and the two reviewer roles kept separate
 - [ ] Simulated users trained on real transcripts, with adversarial personas
+- [ ] Every comparison that changes a decision states `n`, `k`, the reducer and the noise
+      band — `references/statistics.md`; a three-point gap on a hundred cases is noise
+
+---
+
+## References
+
+| Load | When |
+|---|---|
+| [`references/statistics.md`](references/statistics.md) | a number is about to change a decision — how many runs before a difference is real, `pass@k` vs `pass^k` and why trials are not independent, paired comparison, the harness as a variable, and what a given piece of evidence authorises next |
 
 ---
 

package/plugins/agent-stack/skills/agent-evals/references/statistics.md

@@ -0,0 +1,201 @@
+# Statistics for agent evals — how many runs before a difference is real
+
+**Load this when a number is about to change a decision:** picking between two models or
+two harness configurations, setting a release threshold, deciding whether a regression is
+real, or writing the sentence "X is better than Y" in a document somebody will act on.
+
+Everything here is arithmetic over binary outcomes. It needs no library and no
+statistician, and it is the layer most eval advice skips — including this skill's own
+first eight sections, which say what to assert and never say how many times.
+
+## Contents
+
+- [The band around a pass rate](#the-band-around-a-pass-rate)
+- [pass@k and pass^k are different questions](#passk-and-passk-are-different-questions)
+- [Trials are not independent, and the published data says so](#trials-are-not-independent-and-the-published-data-says-so)
+- [Pairing: same tasks, same seeds, per-task deltas](#pairing-same-tasks-same-seeds-per-task-deltas)
+- [The harness is a variable, so pin it](#the-harness-is-a-variable-so-pin-it)
+- [What a number authorises](#what-a-number-authorises)
+- [The report contract](#the-report-contract)
+
+---
+
+## The band around a pass rate
+
+A pass rate over `n` cases is an estimate, and its standard error is
+
+```
+SE(p) = sqrt( p * (1 - p) / n )
+```
+
+The 95% band is roughly `±1.96 · SE`. Computed, not quoted:
+
+| n | p | 95% band |
+|---|---|---|
+| 100 | 0.70 | **±8.98 pp** |
+| 400 | 0.70 | ±4.49 pp |
+| 1000 | 0.70 | ±2.84 pp |
+
+```python
+import math
+def band(p, n): return 1.96 * math.sqrt(p * (1 - p) / n) * 100   # percentage points
+```
+
+**So "the new one gets 73% where the old one got 70%, on a hundred cases" is not a
+result.** It is a number inside its own noise. The error shrinks as `1/√n`, which is the
+whole practical consequence: **the fix for a 2–3 pp expected gain is more tasks, not more
+argument.** Quadrupling the set halves the band.
+
+A corollary worth stating because leaderboards invite the opposite: **differences below
+about 3 pp deserve scepticism until both configurations are documented and matched.**
+
+> The formula assumes independent cases. A benchmark whose tasks share a fixture, an
+> environment or a generator violates that, and the true band is wider than this. Wider,
+> never narrower — so the table is a floor on your uncertainty, not a ceiling.
+
+## pass@k and pass^k are different questions
+
+Two metrics, one letter apart, measuring opposite things.
+
+| Metric | Formula | Asks | Who it is for |
+|---|---|---|---|
+| `pass@k` | `1 − (1 − p)^k` | did **at least one** of k attempts succeed | a **capability ceiling** — a human picks the best of k |
+| `pass^k` | `p^k` (if independent) | did **every** one of k succeed | a **reliability floor** — nobody is picking |
+
+At `p = 0.6`, `k = 5`:
+
+- `pass@5 = 1 − 0.4⁵ = 0.98976` → **99.0%**
+- `pass^5 = 0.6⁵ = 0.07776` → **7.8%**
+
+**A 91-point gap between two numbers describing the same agent.** The first makes a demo;
+the second is what a payment, a refund or a permission change actually needs.
+
+Anthropic states the same arithmetic for the everyday case: at a 75% per-trial rate, three
+trials all passing is `0.75³ ≈ 42%` — verified, `0.421875`.
+
+**The rule that follows: an operation with side effects may not "retry until it works."**
+If a failed attempt leaves a charge, a message or a mutated row behind, `pass@k` is not
+available to you as a metric — you cannot pick the best of five refunds. Sample in a
+sandbox or a rollback-capable environment, and count **every** failure.
+
+**A report that gives k without saying which k it means is unreadable.** *k independent
+samples of one task* and *k consecutive tasks on one live pipeline* are different claims.
+
+## Trials are not independent, and the published data says so
+
+`pass^k = p^k` assumes each trial is a fresh coin flip. Real benchmarks do not behave that
+way, and the τ-bench leaderboard is the cleanest demonstration — claude-3-5-sonnet on the
+airline domain, published Pass^k beside what independence would predict from Pass^1:
+
+| k | published Pass^k | `0.460^k` if independent |
+|---|---|---|
+| 1 | 0.460 | 0.460 |
+| 2 | **0.326** | 0.212 |
+| 3 | **0.263** | 0.097 |
+| 4 | **0.225** | 0.045 |
+
+The observed curve falls far slower than independence predicts. The reason is not
+mysterious: **some tasks are reliably easy and some reliably hard**, so successes cluster
+by task rather than scattering by trial. Positive correlation across trials of the same
+task.
+
+Two consequences, and they cut in opposite directions:
+
+- **You cannot compute `pass^k` from `pass^1`.** Exponentiating a headline rate gives a
+  number far below the truth. Measure `pass^k` directly, at the k you care about.
+- **Anthropic's `0.75³ ≈ 42%` is a worst case, not a forecast.** It is the right shape for
+  an argument — *consistency is a much harder bar* — and the wrong number to put in a
+  release gate.
+
+The other half of independence is the harness, not the task: Anthropic requires each trial
+start from a clean environment, because *"unnecessary shared state between runs (leftover
+files, cached data, resource exhaustion) can cause correlated failures."* Correlated
+failures break the arithmetic above — so isolation is not hygiene, it is what makes the
+metric mean anything.
+
+## Pairing: same tasks, same seeds, per-task deltas
+
+**Never subtract two independent averages.** Run both configurations over the *same* task
+list with the *same* fixed seeds, record a per-task win/loss/tie, and test the deltas.
+
+```
+for task in tasks:            # identical list
+    for seed in seeds:        # identical seeds, 3-5 of them
+        a = run(config_A, task, seed)
+        b = run(config_B, task, seed)
+        delta[task, seed] = a - b
+```
+
+- **3–5 seeds per configuration**, reporting mean and spread. A single run screens a
+  direction; it does not establish one.
+- Test the paired deltas with **McNemar** (binary outcomes) or a **paired bootstrap**.
+  Pairing removes the task-difficulty variance that dominates the unpaired comparison —
+  which is exactly the correlation the previous section measured.
+- Testing several hypotheses at once needs a **multiple-comparisons correction**, or an
+  independent re-run of whichever ones came out positive. Five hypotheses at p<0.05 gives
+  you roughly a one-in-four chance of a false positive somewhere.
+
+**Ship on three conditions, not one:** the difference exceeds the noise band, it survives
+the paired analysis, and it reproduces on a rerun.
+
+## The harness is a variable, so pin it
+
+The container spec is part of the measurement. On Terminal-Bench 2.0 the gap between the
+most- and least-resourced setups was **6 percentage points (p < 0.01)** — larger than most
+model differences anyone argues about.
+
+The shape of the effect matters more than the number:
+
+- Between **1× and 3×** the task's specified resources, scores move **within noise
+  (p = 0.40)**.
+- From 3× to uncapped, infrastructure errors drop a further **1.6 pp** and success jumps
+  nearly **4 pp** — because the extra headroom lets the agent attempt strategies that only
+  work with it: pulling large dependencies, spawning expensive subprocesses, running
+  memory-hungry suites.
+- Infrastructure errors alone: **5.8% of tasks at 1×**, cut to **2.1% at a 3× ceiling
+  (p < 0.001)**.
+
+So a tight cap and a generous cap **measure different agent strategies**, not the same
+agent more or less precisely. The remedy is a **floor and a ceiling**, calibrated so that
+scores at both fall within noise of each other — not a single pinned value.
+
+The effect is task-distribution dependent and does not transfer: SWE-bench moves only
+**1.54 pp from 1× to 5×**. Measure it for your own suite rather than importing a multiplier.
+
+## What a number authorises
+
+Evidence licenses the next action its scope supports, and nothing further. A worked
+three-round loop on a deliberately narrow slice — four tasks, one run each, model, seed,
+step limit and environment fixed, arm order alternated, **one variable changed per round**:
+
+| Round | The only change | Success | Tokens vs control | What it authorised |
+|---|---|---|---|---|
+| H1 | added navigation and final-check instructions | 25% → 25% | 0.47× | the prompt is not the bottleneck — stop tuning it |
+| H5 | accessibility feed → UIAutomator tree | 25% → **100%** | **2.498×** | right mechanism, too expensive — try to cheapen it |
+| H5C | prune invisible/textless/non-actionable nodes | 100% → 100% | **0.506×** | qualifies for a full rerun |
+
+Two rules fall out, and the second is the one people skip:
+
+- **Observation before prompt.** More detailed instructions cannot restore information the
+  agent never received. When a score will not move, ask what the agent could see before
+  asking how it was asked.
+- **4/4 on a slice is not 100% system-wide, and must not be reported as one.** With four
+  tasks per arm these numbers can decide whether a larger rerun is worth paying for. They
+  cannot estimate success across the benchmark. The ladder's output is *the next
+  experiment*, not a result.
+
+## The report contract
+
+A comparison that will change a decision states all of these, or it is a claim rather than
+a measurement:
+
+- [ ] `n` — how many tasks, and `k` — how many runs each
+- [ ] which reducer: `pass@k`, `pass^k`, `mean`, and **which k means what**
+- [ ] the noise band for that `n`, computed
+- [ ] paired or unpaired; if paired, the seed set
+- [ ] the harness configuration: resource floor and ceiling, isolation between trials
+- [ ] whether the difference reproduced on a rerun
+- [ ] what the scope of the evidence authorises next — not what it suggests
+
+**A green suite with none of these is a number, not a verdict.** That distinction is the
+reason this file exists.