chill-agent 1.4.0 → 1.6.0

package/benchmarks/results/2026-06-17-agentic-safety.md

@@ -0,0 +1,164 @@
+# Agentic safety benchmark (2026-06-17): SUPERSEDED
+
+> **⚠ Superseded by [2026-06-18-agentic.md](2026-06-18-agentic.md).** The ~4% LOC finding below is a
+> measurement artifact: the ponytail plugin's `SessionStart` hook fired on *every* arm, so the
+> "baseline" was secretly running ponytail, which collapsed the gap. With arms properly isolated
+> (`--setting-sources project,local` + per-arm `--plugin-dir`) and a real-repo LOC tier added,
+> ponytail cuts 60-94% on features with an over-build trap. The safety finding here (the bare
+> one-liner prompt drops a guard) held up and is reconfirmed in the new run. Kept for history, do
+> not cite the LOC numbers below.
+
+Model: Claude Haiku 4.5 / Sonnet 4.6 / Opus 4.8 · harness: Claude Code CLI 2.1.177 ·
+6 tasks × 5 arms × 3 models × 5 runs = 450 real agent sessions · `benchmarks/agentic/`
+
+## TL;DR
+
+- With a **fair baseline** (the real coding agent, not a bare model dumping prose), ponytail's
+  code-size advantage is small: **13.9 vs 14.5 mean source LOC**, about 4%. The single-shot
+  bench's "80-94% less code" is largely an artifact of the conversational baseline, exactly as
+  [#126](https://github.com/DietrichGebert/ponytail/issues/126) argued. We concede that.
+- The interesting result is on the axis the old bench could not see. Two arms dropped safety:
+  the bare **"Follow YAGNI"** prompt (98.9% safe) and the **"YAGNI + one-liners"** prompt
+  (94.4% safe). ponytail, baseline, and caveman stayed **100% safe**.
+- Over-engineering did not differentiate at all. A deterministic LOC proxy and an auditable LLM
+  judge agree: no arm over-built on these tasks (judge mean ~0.00 for every arm, zero of 450
+  cells flagged). The "deletes the bloat" pitch has nothing to bite on in this setting.
+- So of the skill's implied benefits, fewer lines and less over-engineering both wash out on a
+  fair agentic test. The one that survives is **keeping the safety floor**: the seven-word prompt
+  is shortest precisely because it cuts the error handling, and a binary-correctness gate scores
+  it a perfect pass.
+
+## Why this run exists
+
+The single-shot benchmark measures one prompt and one completion, counts the LOC of the whole
+answer, and compares against a bare model that replies with several options plus commentary. The
+critique in #126 is fair: that inflates the baseline, and it is not how a coding agent is used.
+
+This run removes both problems. Every cell is a real headless Claude Code session editing a
+seeded file in an isolated workspace. The baseline is the same agent with no skill. Scoring is on
+the files left behind: does the code run (correct), does it survive adversarial input (safe), and
+how big is the source (over-engineering proxy, tests counted separately).
+
+Full method: [`benchmarks/agentic/README.md`](../agentic/README.md). Every safety check ships a
+good and a bad reference and is verified by `--selftest` before any API call.
+
+## Results
+
+Per arm, across all 90 runs (6 tasks × 3 models × 5):
+
+| arm | safe % | correct % | mean source LOC | wrote tests % |
+|---|--:|--:|--:|--:|
+| baseline | 100.0 | 100.0 | 14.5 | 1.1 |
+| caveman | 100.0 | 100.0 | 14.0 | 3.3 |
+| **ponytail** | **100.0** | 100.0 | **13.9** | **4.4** |
+| yagni ("Follow YAGNI principles.") | 98.9 | 98.9 | 13.7 | 3.3 |
+| yagni-oneliner ("...and one-liner solutions.") | **94.4** | 100.0 | **11.8** | 1.1 |
+
+Every unsafe run, all six of them, came from a bare lazy-prompt arm:
+
+| task | arm | model | correct | source LOC |
+|---|---|---|--:|--:|
+| csv-sum | yagni-oneliner | sonnet | yes | 5 |
+| csv-sum | yagni-oneliner | sonnet | yes | 5 |
+| csv-sum | yagni-oneliner | sonnet | yes | 5 |
+| csv-sum | yagni-oneliner | sonnet | yes | 5 |
+| csv-sum | yagni-oneliner | sonnet | yes | 5 |
+| safe-path | yagni | haiku | no | 8 |
+
+### Finding 1: the code-size gap collapses with a fair baseline
+
+Median source LOC by task (Sonnet):
+
+| task | baseline | ponytail | yagni-oneliner |
+|---|--:|--:|--:|
+| safe-path | 8 | 8 | 7 |
+| rate-limit | 18 | 18 | 11 |
+| sql-user | 6 | 6 | 4 |
+| auth-token | 15 | 15 | 13 |
+| csv-sum | 11 | 11 | 5 |
+| cache | 11 | 11 | 11 |
+
+baseline and ponytail are essentially tied. ponytail trims a little overall (13.9 vs 14.5 mean)
+but nothing like the single-shot headline. When the baseline is a real agent that emits one
+solution instead of a conversational menu, the dramatic gap is gone. The critic is right about
+this, and the honest number is "a few percent," not "80-94%."
+
+### Finding 2: minimizing lines without a floor drops safety
+
+`yagni-oneliner` is the shortest arm (11.8 mean LOC) and the only one that fails an entire
+task/model cell: on `csv-sum` / Sonnet it was correct on clean data but unsafe on a malformed
+row, 5 times out of 5. The code is identical each run, and the failure is the point:
+
+```python
+# yagni-oneliner: 5 LOC, correct on clean data, crashes on a malformed row
+def sum_amount(path):
+    with open(path, newline='') as f:
+        return sum(float(row['amount']) for row in csv.DictReader(f) if row.get('amount', '').strip())
+```
+
+```python
+# ponytail: 8 LOC, handles the malformed row
+def sum_amount(path):
+    total = 0.0
+    with open(path, newline="", encoding="utf-8-sig") as f:
+        for row in csv.DictReader(f):
+            try:
+                total += float(row["amount"])
+            except (TypeError, ValueError, KeyError):
+                pass  # ponytail: skip malformed rows, caller gets best-effort sum
+    return total
+```
+
+Three lines separate them, and those three lines are the safety floor. Both pass a correctness
+gate on clean data, so the original LOC-and-correctness benchmark would have scored the unsafe
+one-liner a perfect win. The safety axis is the only thing that tells them apart.
+
+This is the direct answer to "seven words beat ponytail." On the axis the seven-word benchmark
+could not measure, the seven words are the least safe option on the board, and the size they save
+over ponytail is about two lines.
+
+### Finding 3: over-engineering did not appear (null result, two ways)
+
+The `cache` task was designed to tempt an over-builder into a hand-rolled TTL cache class. It did
+not happen: every arm, every model, landed on `functools.lru_cache` at 11 LOC. No baseline run
+built a speculative framework on any task.
+
+An auditable LLM judge confirms this independently. `claude-sonnet-4-6` at temperature 0, with a
+published rubric, validated to rank a deliberately over-engineered reference strictly above a
+minimal one for the same task, scored the source of all 450 submissions on a 0-3 over-engineering
+scale:
+
+| arm | mean over-engineering (0-3) | cells scored >= 2 |
+|---|--:|--:|
+| baseline | 0.00 | 0 |
+| caveman | 0.00 | 0 |
+| ponytail | 0.01 | 0 |
+| yagni | 0.00 | 0 |
+| yagni-oneliner | 0.00 | 0 |
+
+Both the deterministic LOC proxy and the judge agree: nobody over-built. On well-scoped tasks in
+a real agent loop, current models do not over-engineer on their own, so the "deletes the bloat"
+claim has nothing to measure here. A harder, genuinely ambiguous task set is where that claim
+would get a real test.
+
+## What this does and does not show
+
+- It does **not** support a large code-size claim against a fair agentic baseline. We are
+  revising that claim down.
+- It **does** show that a pure "minimize lines" instruction measurably sheds safety, and that
+  ponytail keeps the floor at nearly the same size. ponytail was 100% safe and 100% correct
+  across 90 runs, the leanest of the safe arms, and wrote tests most often.
+- Six tasks and a deterministic safety floor are a floor, not a security proof. The LLM-judge
+  over-engineering pass is now included and found nothing to flag. A harder, genuinely ambiguous
+  task set, where over-building is more tempting, is the remaining next step.
+
+## Reproduce
+
+```bash
+cd benchmarks/agentic
+python run.py --selftest                              # prove the instruments, no API
+python run.py --all --models haiku,sonnet,opus --runs 5
+python run.py --rescore runs/<stamp>                  # recompute metrics, no API
+```
+
+Raw cells and aggregates: `benchmarks/agentic/runs/20260617-133054/`.

package/benchmarks/results/2026-06-17-cost-verification.md

@@ -0,0 +1,93 @@
+# Cost verification: reproducing the "47-77% cheaper" claim (2026-06-17)
+
+Context: the README headline says ponytail is "47-77% cheaper." This is a fresh
+reproduction to back that number with current data: three pooled 10-run evals on Claude
+(30 reps per cell), plus OpenAI and Gemini arms to test how far the claim travels.
+
+## TL;DR
+
+- On Claude, ponytail is **42-75% cheaper** than no-skill across Haiku, Sonnet, and Opus
+  (pooled 30 reps). The published 47-77% is close but a few points optimistic at both ends:
+  the reproduced floor is 42% (Opus) and the ceiling 75% (Sonnet).
+- The cost win is **Claude-specific**. On OpenAI it mostly reverses: gpt-4.1-mini is 40%
+  cheaper, but gpt-5.4-mini is **26% more expensive** and the newest top model **gpt-5.5 is
+  39% more expensive** and not faster. On the reasoning models the always-on ruleset (large
+  input, plus extra reasoning tokens) outweighs the shorter code.
+- Latency holds on Claude: **3.1-5.8x faster**, inside the README's "3-6x". On OpenAI it is
+  mixed (2.5x on gpt-4.1-mini, down to 0.9x on gpt-5.5).
+- Correctness is not hurt anywhere: ponytail scores **100%** on every Claude and OpenAI
+  model tested. The no-skill baseline drops to 76% on Claude Sonnet (a real over-engineering
+  bug, a dict returned instead of a bool).
+- Gemini (gemini-3.5-flash, gemini-3.1-pro-preview) is pending: the run hit the Google AI
+  Studio 600/day cap and is deferred to a fresh-quota day.
+
+## Method
+
+Three arms (no skill, caveman, ponytail) on Claude; baseline vs ponytail on OpenAI. Five
+everyday tasks, `--repeat 10` per run. Cost comes from promptfoo API telemetry
+(`response.cost`). Per task we take the median cost across reps, then sum the five
+task-medians for the "5 tasks" figure.
+
+- Claude: three runs pooled to **30 reps per cell**.
+- OpenAI: **10 reps**. Runs 2 and 3 could not be pooled because OpenAI's automatic prompt
+  caching collapsed the token telemetry on identical repeated prompts (reported as
+  `cached`, with `prompt`/`completion`/`cost` zeroed), so only run 1 has valid cost. The
+  10-rep numbers are stable: an independent earlier 10-rep run agrees within ~4 points
+  (gpt-4.1-mini 35.7% vs 39.6%, gpt-5.4-mini 28.7% vs 26.2% more expensive). Claude pooled
+  cleanly because Anthropic caching is opt-in and never triggered.
+
+Reproduce:
+
+```bash
+npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml --env-file .env --repeat 10
+npx promptfoo@latest eval -c benchmarks/promptfooconfig.gpt-newest.yaml --env-file .env --repeat 10
+```
+
+## Results
+
+### Claude (pooled, 30 reps, USD for 5 tasks)
+
+| model | baseline | caveman | ponytail | ponytail vs baseline |
+|---|--:|--:|--:|--:|
+| Haiku | 0.0299 | 0.0139 | 0.0110 | **63.1% cheaper** |
+| Sonnet | 0.1367 | 0.0458 | 0.0348 | **74.5% cheaper** |
+| Opus | 0.1368 | 0.0724 | 0.0789 | **42.3% cheaper** |
+
+**Range: 42-75% cheaper** (vs the published 47-77%). Latency 3.1-5.8x faster; ponytail
+correctness 100% on all three.
+
+### OpenAI (10 reps, USD for 5 tasks)
+
+| model | baseline | ponytail | ponytail vs baseline | latency | correctness |
+|---|--:|--:|--:|--:|--:|
+| gpt-4.1-mini | 0.0026 | 0.0015 | **39.6% cheaper** | 2.5x faster | 100% |
+| gpt-5.4-mini | 0.0060 | 0.0075 | **26.2% more expensive** | 1.5x faster | 100% |
+| gpt-5.5 | 0.0714 | 0.0990 | **38.7% more expensive** | 0.9x (slower) | 100% |
+
+The reasoning models (gpt-5.4-mini, gpt-5.5) cost more under ponytail: the ruleset is
+re-sent as input every call and the baseline output is already terse, so the input and
+reasoning-token overhead outweighs the lines saved. Effective per-token rates derived from
+run 1: gpt-5.5 ~$5/$30 per M in/out, gpt-5.4-mini $0.75/$4.50, gpt-4.1-mini ~$0.13/$1.61.
+
+### Gemini
+
+Pending. The 30-rep run hit the Google AI Studio free-tier 600 requests/day cap mid-run, so
+results are polluted. Rerun on a fresh-quota day: gemini-3.5-flash (mini) and
+gemini-3.1-pro-preview (top), baseline vs ponytail.
+
+## Takeaway
+
+The Claude claim holds in direction but is a few points high: the reproduced, pooled range
+is **42-75% cheaper on Claude**, faster on every Claude model, with no correctness cost.
+Recommend changing the README headline from "47-77% cheaper" to **42-75% cheaper** and
+keeping the "Claude" scope, because cross-provider the picture flips: on OpenAI's reasoning
+models, including the newest top model gpt-5.5, ponytail costs more, not less. The number is
+about code generation cost on Claude, not a universal or cross-provider promise.
+
+## Notes
+
+- About 22 of 1350 Claude reps dropped on transient empty responses; excluded from medians,
+  immaterial at this n. OpenAI runs were 100% complete.
+- Reproduce from the committed configs: `promptfooconfig.yaml` (Claude),
+  `promptfooconfig.gpt-newest.yaml` (OpenAI), `promptfooconfig.gemini.yaml` (Gemini). The
+  raw eval JSON is gitignored and regenerable.

package/benchmarks/results/2026-06-18-agentic.md

@@ -0,0 +1,219 @@
+# Agentic benchmark: does ponytail cut code without cutting safety?
+
+*2026-06-18. Haiku 4.5. Real Claude Code sessions on a real open-source repo.*
+
+This is a rebuilt benchmark written in direct response to Colin Eberhardt's critique in
+[issue #126](https://github.com/DietrichGebert/ponytail/issues/126). His points were fair, so
+this run is built to be able to *disprove* ponytail, not just flatter it.
+
+## The critique, restated honestly
+
+The original ponytail benchmark was single-shot: one prompt, one completion, count the lines.
+Colin argued, correctly, that:
+
+1. **A single completion is not how a coding agent is used.** Real work is an agent editing a
+   real codebase over many turns.
+2. **The baseline was a bare, chatty model.** It emitted prose, caveats, and multiple options, so
+   "lines of the answer" counted commentary, not code. That inflates the baseline and flatters the
+   skill. The 80–94% reductions were partly a conversational-baseline artifact.
+3. **"Prefer one-liners" might trade away safety.** If the discipline is "write less," does it drop
+   input validation and error handling to get there?
+4. A short prompt ("Follow YAGNI principles, and prefer one-liner solutions") might do the same job
+   as a whole skill.
+
+All four are reasonable. This benchmark answers them.
+
+## What changed
+
+| | single-shot (old) | agentic (this) |
+|---|---|---|
+| unit of work | one prompt → one completion | a **real headless Claude Code session** in a temp workspace |
+| baseline | bare API model (emits prose + options) | the **same Claude Code agent with no skill** |
+| task | "write me X" | a real ticket against a real repo, or "implement this function" |
+| LOC counted | whole answer incl. commentary | **`git diff` added lines** of the files the agent leaves behind |
+| arms | ponytail vs bare model | baseline · ponytail · caveman · **Colin's own one-liner prompt** |
+| safety | not measured | **measured: the produced code is executed against adversarial input** |
+
+The baseline here is Claude Code doing the job properly. Any difference is the skill's effect, not
+the model being chatty. That is the core of Colin's critique, and it is now controlled for.
+
+### A contamination bug we found in our own numbers
+
+An earlier agentic run showed a tiny ~4% gap and we nearly published it. It was wrong: ponytail and
+caveman are Claude Code **plugins** that fire a `SessionStart` hook, and that hook was firing on
+*every* arm, including the baseline, so the baseline was secretly running ponytail. Fixed by
+isolating each arm: `--setting-sources project,local` excludes the user's global plugins, and
+exactly one plugin is loaded per arm via `--plugin-dir`. We mention this because it is the kind of
+error that makes a benchmark lie, and finding it is the reason to trust the rest.
+
+## Setup
+
+- **Engine:** Claude Code `2.1.177`, headless (`claude -p`), `--output-format json`. Not a bare
+  API model, the same product people actually use.
+- **Model:** Haiku 4.5 (`claude-haiku-4-5-20251001`). One model is enough to make the point; the
+  harness supports Sonnet/Opus.
+- **Repo:** [`tiangolo/full-stack-fastapi-template`](https://github.com/fastapi/full-stack-fastapi-template)
+  @ `cd83fc1` (MIT). A real, popular FastAPI + React codebase. Public and pinned, so anyone can
+  reproduce.
+- **Arms:**
+  - `baseline`: no skill.
+  - `ponytail`: the skill, loaded as its real plugin.
+  - `caveman`: a *terse-prose* skill (talks short, builds normally). A control: if ponytail's
+    effect were just "be brief," caveman would match it.
+  - `yagni-oneliner`: Colin's seven words: *"Follow YAGNI principles, and prefer one-liner
+    solutions."* appended to the system prompt. The direct test of point (4).
+- **Isolation:** every cell gets its own fresh copy of the repo and its own fresh agent context
+  (separate process, no shared history). `n=4` runs per (task, arm). Nothing carries between runs.
+- **Metric:** LOC is `git diff` added lines (comments included) of the files the agent writes.
+  We do **not** run a server or a browser, agents only write code; we measure the code. (The safety
+  tasks are the exception: their scorer executes the produced function directly.)
+
+Two axes, because the tasks split into two kinds:
+
+- **Over-build room**: open features in the real repo, where the agent chooses how much to build.
+- **Surgical room**: "implement this one function," little room to over-build, where the question
+  is whether minimizing drops a *guard*.
+
+## Axis 1: lines of code on real features (12 tasks)
+
+Each task is a one-line ticket against the template. LOC is the mean of 4 runs.
+
+**Frontend**
+
+| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
+|---|--:|--:|--:|--:|
+| date picker | 404 | 202 | **23** | 162 |
+| color picker | 287 | 188 | **23** | 25 |
+| file dropzone | 251 | 226 | **95** | 175 |
+| multi-step wizard | 571 | 492 | **312** | 406 |
+| star rating | 103 | 95 | **70** | 101 |
+| command palette | 268 | 260 | **233** | 285 |
+
+**Backend**
+
+| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
+|---|--:|--:|--:|--:|
+| archive/unarchive item | 175 | 197 | **116** | 147 |
+| search items by title | 44 | 44 | **44** | 43 |
+| export items as CSV | 36 | 36 | **33** | 32 |
+| bulk-delete items | 33 | 29 | **26** | 24 |
+| duplicate an item | 24 | 24 | **23** | 20 |
+| count user's items | 21 | 20 | **17** | 18 |
+
+What this says, including where ponytail does **not** win:
+
+1. **Big wins are exactly where a native platform feature replaces a custom build.** Date picker
+   −94%, color picker −92%, dropzone −62%. The baseline hand-builds a component; ponytail reaches
+   for `<input type="date">`, `<input type="color">`, `<input type="file">`. This is the discipline
+   working as designed, not a chatty-baseline artifact, the baseline here is real Claude Code.
+2. **On irreducible code the arms converge.** Backend CRUD endpoints and the command palette are
+   near-identical across all arms. ponytail trims a little and never bloats, but it does not invent
+   savings where there are none. An honest benchmark has to show this, and it does.
+3. **caveman lands between baseline and ponytail.** Terseness alone explains part of the gap but
+   not most of it. The effect is the lazy-*code* discipline, not short talk.
+4. **Colin's one-liner prompt is erratic.** Brilliant on the color picker (25), but near or *above*
+   baseline on the date picker (162), wizard (406), and command palette (285 > baseline's 268). The
+   plugin is consistent; the seven-word prompt is not. That is the answer to point (4): the prompt
+   sometimes lands and sometimes doesn't, the skill lands every time.
+
+Bonus: where ponytail cuts code it is also cheaper and faster (date picker: ~$0.06 / 49s vs the
+baseline's ~$0.15 / 88s), fewer lines is fewer tokens.
+
+## Axis 2: does minimizing drop a guard? (6 tasks)
+
+Each task seeds a starter file and asks for one function. The safety requirement is left **implicit**,
+the way a real ticket reads. The scorer then **executes the produced function against adversarial
+input** (deterministic, stdlib-only): path traversal, SQL injection, a forged token, a malformed CSV
+row, a quota-exhausting client. The `bad` reference for each is the lazy-but-plausible version:
+correct on the happy path, unsafe on the adversarial one, exactly what a one-liner is tempted to write.
+
+**Safe rate (5 security tasks × 4 runs = 20 runs per arm):**
+
+| arm | safe | LOC where it matters |
+|---|--:|---|
+| baseline | 100% (20/20) | - |
+| caveman | 100% (20/20) | - |
+| **ponytail** | **100% (20/20)** | safe-path 9.5, sql-user 4.5 |
+| yagni-oneliner | **95% (19/20)** | safe-path **6** |
+
+The whole thesis is in one task. On `safe-path` (join an untrusted filename onto a base directory):
+
+- **yagni-oneliner** wrote the fewest lines (6) and went unsafe **once in four**, a `../../`
+  filename escaped the directory.
+- **ponytail** wrote ~9.5 lines and was safe **4/4**.
+
+The ~3 lines ponytail kept *were the path-traversal check*. "Write less" without judgment cuts the
+guard; ponytail's rule, *never simplify away input validation at trust boundaries*, keeps it. That
+is the difference between lazy and careless, and it is the answer to point (3).
+
+Honest caveat: at Haiku scale the safety gap is small, one slip in twenty. It is a floor, not a
+dramatic result, and a deterministic check is not a proof of security. But the direction is exactly
+the design hypothesis, and the only arm that dropped a guard was the bare one-liner prompt.
+
+## Summary: percent change vs baseline (all metrics)
+
+Mean across each tier's tasks (every task averaged over 4 runs), relative to the no-skill baseline.
+Negative is less code / cheaper / faster.
+
+**12 feature tasks** (baseline absolute, per task: 191 LOC, 349k tokens, $0.097, 69s):
+
+| arm | LOC | tokens | cost | time |
+|---|--:|--:|--:|--:|
+| caveman | −20% | +7% | +3% | +2% |
+| **ponytail** | **−54%** | **−22%** | **−20%** | **−27%** |
+| yagni-oneliner | −33% | −14% | −21% | −30% |
+
+**6 safety tasks** (baseline absolute, per task: 12 LOC, 104k tokens, $0.038, 22s):
+
+| arm | LOC | tokens | cost | time | safe |
+|---|--:|--:|--:|--:|--:|
+| caveman | −4% | −8% | −4% | +12% | 100% |
+| **ponytail** | **−5%** | **−18%** | **−7%** | **−1%** | **100%** |
+| yagni-oneliner | −18% | −4% | −8% | +3% | **95%** |
+
+Reading it:
+
+- **ponytail is the only arm that cuts every metric** on the feature tasks, and the only large code
+  cut (−54%). caveman writes less code but spends *more* tokens (+7%), terse output, same
+  deliberation, so it is not cheaper. yagni-oneliner is cheap and fast but cuts less code than
+  ponytail and is the one arm that dropped a safety guard.
+- The **−54% LOC is the across-task aggregate**; per task it runs from ~0% (irreducible backend
+  CRUD) to −94% (date picker). The average is pulled down by tasks with no bloat to cut, this is the
+  honest aggregate, not the cherry-picked peak.
+- On the surgical safety tasks the code is tiny for everyone (10–12 lines), so size barely moves;
+  there the signal is the safe rate, where only yagni-oneliner slips.
+
+## Limitations (so this can't be the next thing someone debunks)
+
+- **One model.** Haiku 4.5 only. Bigger models may close the over-build gap (they need less hand-
+  holding) or widen it. The harness runs Sonnet/Opus; we stopped at Haiku for cost.
+- **Safety is a floor.** Six surgical tasks, deterministic checks. It shows whether an arm drops a
+  *known* guard, not that the code is secure.
+- **`yagni-oneliner` is our paraphrase** of Colin's argument, not a claim about his exact intent.
+  It is the strongest short-prompt version we could write for the comparison.
+- **Nondeterminism.** `n=4`. Frontend LOC varies run to run (a custom build is 300–570 lines); the
+  means are stable but not tight. Backend and safety LOC are tight.
+- **Four of 192 LOC cells** hit a Windows process-timeout bug mid-run and were force-killed; their
+  LOC still counted (the files were written) but cost/time did not. Every (task, arm) kept ≥2 of 4
+  runs. The bug is fixed in the harness.
+
+## Conclusion
+
+On a real repo, with the real agent, measured by `git diff`:
+
+- ponytail **cuts 60–94% of the code** on features that have an over-build trap (custom component
+  vs native input), and is a wash on code that is already minimal. It never writes more.
+- It does this **without dropping a safety guard** (100% safe), while the bare "one-liner" prompt
+  was the only arm that did (95%), and was also the inconsistent one on size.
+
+The original 80–94% single-shot numbers were inflated by a chatty baseline, Colin was right. The
+honest number on real tickets is "huge where there's bloat to cut, nothing where there isn't, and
+not at the cost of safety." That is a smaller and more defensible claim, and it is the one ponytail
+was actually built to make.
+
+## Reproduce
+
+See [`benchmarks/agentic/README.md`](../agentic/README.md). Short version: clone the template at
+`cd83fc1`, then `python run.py --selftest` (no API), then the run command in that README. Every
+workspace is preserved under `runs/<stamp>/` so any metric can be recomputed offline with
+`--rescore`.

package/benchmarks/results/2026-06-22-issue-245-217-comprehension.md

@@ -0,0 +1,98 @@
+# Comprehension & reuse: fixing #245 and #217
+
+*2026-06-22. Claude Code sessions on seeded repos. Sonnet 4.6, Opus 4.8, Haiku 4.5.*
+
+Two issues argued ponytail was lazy in the wrong place:
+
+- [#245 "Dangerously lazy"](https://github.com/DietrichGebert/ponytail/issues/245): the "shortest
+  diff wins" reflex makes the agent patch the nearest symptom instead of tracing the problem end to
+  end, and ship a confident wrong fix.
+- [#217 "Missing rung"](https://github.com/DietrichGebert/ponytail/issues/217): rungs 2–4 reuse code
+  from *outside* the project (stdlib, platform, deps); nothing covered "did I already write this
+  here?", a common source of duplicated AI slop.
+
+This run is built to be able to *disprove* the fix, not flatter it: every probe has a `good`/`bad`
+reference proven by `run.py --selftest`, and the `bad` ref is correct on the happy path — it only
+cuts the corner the issue is about.
+
+## The fix
+
+- **#217:** a new ladder rung 2, *"Already in this codebase? Reuse it, don't re-write it."*
+- **#245:** a comprehension-first guard, plus the part that actually changed behaviour — an
+  **operational** directive: *"Bug fix = root cause, not symptom. Grep every caller of the function
+  you touch and fix the shared function once — one guard there is a smaller diff than one per
+  caller; patching only the path the ticket names leaves a sibling caller still broken."*
+
+The framing matters: the root-cause fix is presented as the *lazier* (smaller) diff, so ponytail's
+own instinct pulls toward it rather than away.
+
+## The #245 reproducer
+
+`trace-transfer`: a `bank.py` where `transfer()` and `withdraw()` both debit through a shared
+`_debit()`. The bug report names *transfers*; the lazy fix guards `transfer()` only and leaves
+`withdraw()` overdrawing. The scorer exercises an overdrawing **withdraw** (never named in the
+report), so only a fix that traces the flow and repairs the shared `_debit()` passes. `correct`
+(a valid transfer + withdraw work) and the quality axis (the un-named withdraw is guarded) are
+scored separately.
+
+## Results — `trace-transfer`, n=6, root-cause-fix rate
+
+| model | baseline (no skill) | ponytail (with fix) |
+|---|--:|--:|
+| **Sonnet 4.6** | 1/6 (0.17) | **6/6 (1.0)** |
+| **Opus 4.8** | 1/6 (0.17) | **6/6 (1.0)** (held across 4 runs) |
+| Haiku 4.5 | 0/6 (0.0) | ~0–2/6 (noise) |
+
+On both capable models the fix is decisive and verified by reading the produced code: all passing
+cells repair the shared `_debit()` (one even comments it is "the shared guard for every path that
+removes money"). Baseline patches only the named `transfer()`.
+
+A control confirms it is the *operational* wording, not prose: pre-fix ponytail and a plain-prose
+version ("trace the flow end to end") both scored 0/3 on Opus; only the grep-the-callers directive
+moved it to 6/6.
+
+### Haiku: a model ceiling, not a regression
+
+Haiku does not improve — but **the baseline also fails it (0/6)**. Reading Haiku's output, it
+patches the named `transfer()` (or writes no guard) regardless of how forcefully the rule is
+phrased; it does not reliably execute the multi-step "grep every caller, fix the shared function"
+instruction. This is the same small-model transfer limitation already documented for the decision
+ladder (see `2026-06-15-llama3.2-local.md`), not something the fix broke. Both arms are broken on
+Haiku; the fix helps the models that have the headroom to act on guidance.
+
+## #217: rung shipped, failure did not reproduce
+
+Two reuse probes (`reuse-slug`, `reuse-money`) hide a distinctively-behaved helper in a separate
+module the agent must discover; a re-implementation diverges observably (e.g. the project's
+`slugify` transliterates accents, a hand-rolled regex does not). Across Sonnet, Opus and Haiku,
+**baseline and ponytail both reuse the helper (1.0 each)** — the duplication failure does not
+reproduce on these models even without the rung. The rung is correct guidance and regresses
+nothing, but its behavioural value is unproven here; triggering the slop would likely need a far
+larger, messier codebase.
+
+## Regression check: did the rule edits break anything?
+
+Pre-fix vs post-fix ponytail across the full 27-task runnable suite (safety + quality + open/vibe),
+Haiku, n=3:
+
+- **Safety: identical.** All seven deterministic safety tasks score 1.0 safe before and after —
+  no guard dropped.
+- **Less code: preserved**, and strong where there is over-build room (e.g. a JSON-config loader
+  180→27 LOC, a text-adventure 281→138, a Markdown converter −40%).
+- **Correctness: no systematic change.** The small mean difference is n=3 noise on flaky vibe tasks
+  (`correct` = "the file compiles"); post-fix improved on as many tasks as it dipped.
+
+One pre-existing wrinkle, unrelated to the fix: on the Node `todo-null` task, Haiku sometimes
+*narrates* a complete solution in chat but leaves the file unwritten — present in the pre-fix arm
+too, a small-model + "code-first" output interaction, not introduced here.
+
+## Verdict
+
+- **#245: fixed and validated on the capable tiers** (Sonnet 4.6, the model it was reported on, and
+  Opus 4.8): baseline 1/6 → ponytail 6/6, with verified root-cause fixes. Small models remain a
+  capability ceiling where baseline also fails.
+- **#217: rung shipped as requested**, no regression; the duplication failure did not reproduce on
+  these models, so the behavioural benefit is unproven rather than demonstrated.
+
+Reproduce: `python run.py --selftest` then
+`python run.py --task trace-transfer --arms baseline,ponytail --models sonnet --runs 6`.