chill-agent 1.5.0 → 1.6.0

package/benchmarks/results/2026-06-12-caveman-vs-ponytail.md

@@ -0,0 +1,71 @@
+# Caveman vs Ponytail — 2026-06-12
+
+5 coding tasks × fresh subagent per config, same model. Tokens = agent total
+(includes thinking). Code lines = fenced blocks in the deliverable, approx.
+n=1 per cell — durations carry noise.
+
+## Ponytail v1 (before this benchmark)
+
+| Task | Baseline | Caveman | Ponytail v1 |
+|---|---|---|---|
+| email | 31,971 tok · 104s · ~34 loc | 26,464 · 20s · ~25 | 27,582 · 32s · 6 |
+| debounce | 23,966 · 25s · ~38 | 26,496 · 19s · ~36 | 27,812 · 33s · 6 |
+| csv-sum | 22,607 · 11s · 6 | 26,062 · 13s · 6 | 26,867 · 20s · 7 |
+| react-countdown | 44,629 · 208s · ~190 | 26,656 · 21s · ~30 | 28,748 · 49s · 13 |
+| rate-limit | 38,782 · 131s · ~25 | 32,732 · 63s · ~20 | 32,579 · 93s · ~21 |
+| **Total** | **161,955 · 479s · ~293** | **138,410 · 136s · ~117** | **143,588 · 228s · ~53** |
+
+### v1 findings
+
+1. Code minimalism: ponytail dominated (2.2× fewer lines than caveman, 5.5× fewer than baseline).
+2. Total tokens: caveman won by ~4% — ponytail wrote minimal code, then long
+   "skipped on purpose" essays. Prose ate the code savings.
+3. Speed: caveman 136s vs ponytail 228s — ponytail deliberated about what not to build.
+4. Floor effect: on already-minimal tasks (csv-sum) both skills pay ~3k tokens
+   skill-read tax over baseline.
+
+## Ponytail v2 (after fixes)
+
+v2 changes: Output cap (code + ≤3 short lines, "if the explanation is longer
+than the code, delete the explanation"), ladder-is-a-reflex clause
+(anti-deliberation), ship-and-question rule (never stall on "do you need X?").
+
+| Task | Ponytail v2 | Δ vs v1 | Δ vs caveman |
+|---|---|---|---|
+| email | 26,705 · 21s · 5 loc | −877 tok · −11s | +241 tok · +1s |
+| debounce | 27,185 · 25s · 6 | −627 · −8s | +689 · +6s |
+| csv-sum | 26,278 · 15s · 6 | −589 · −5s | +216 · +2s |
+| react-countdown | 27,598 · 29s · 13 | −1,150 · −20s | +942 · +8s |
+| rate-limit | 28,858 · 48s · 17 | −3,721 · −45s | −3,874 · −15s |
+| **Total** | **136,624 · 158s · 47** | **−6,964 (−4.8%) · −70s (−31%)** | **−1,786 (−1.3%) · +22s** |
+
+## Ponytail v3 (skill file compressed)
+
+v3 change: SKILL.md 115 → 95 lines, same substance — the minimalism skill
+should not be 2× caveman's length. Cuts read cost per invocation and
+injection cost per session.
+
+| Task | Ponytail v3 | Δ vs v2 | Δ vs caveman |
+|---|---|---|---|
+| email | 26,573 · 19s · 5 loc | −132 · −2s | +109 · −1s |
+| debounce | 26,745 · 22s · 5 | −440 · −3s | +249 · +3s |
+| csv-sum | 26,251 · 15s · 6 | −27 · 0s | +189 · +2s |
+| react-countdown | 26,961 · 22s · 13 | −637 · −7s | +305 · +1s |
+| rate-limit | 29,179 · 49s · 18 | +321 · +1s | −3,553 · −14s |
+| **Total** | **135,709 · 127s · 47** | **−915 · −31s (−20%)** | **−2,701 (−2.0%) · −9s (−7%)** |
+
+## Verdict (v3)
+
+| Area | Winner |
+|---|---|
+| Code size | **Ponytail** — 47 vs 117 lines (2.5×) |
+| Deliverable prose | **Ponytail** — capped at 3 skip-lines, under caveman's gotcha lists |
+| Total tokens (cost) | **Ponytail** — 135.7k vs 138.4k (−2.0%) |
+| Wall time | **Ponytail** — 127s vs 136s (−7%; n=1, treat as parity-or-better) |
+| Follow-up prevention | **Ponytail** — every skip names its escalation trigger |
+
+Both skills demolish the no-skill baseline: −16% tokens, ~3× faster, and the
+baseline's degenerate cases (190-line countdown dashboard, 208s) simply don't
+happen. Remaining ~3.6k floor tax vs baseline on trivial tasks is mostly the
+benchmark's explicit SKILL.md read — production sessions get rules injected
+by the SessionStart hook and don't pay it.

package/benchmarks/results/2026-06-12-v4-hardening-vs-caveman.md

@@ -0,0 +1,125 @@
+# Ponytail v4 hardening — A–F benchmark vs Caveman (2026-06-12)
+
+Response to the hardening brief in `C:\dev\ponytail-bench\PONYTAIL-BENCHMARK-WRITEUP.md`.
+Harness reused as-is: same 6 tasks (specs reconstructed in `ponytail-bench\specs.md` —
+the originals were not preserved; both new arms got identical text), same scorer
+(`score.py`, arms now auto-discovered), same adversarial probes (`probe_e.py`,
+`probe_f.py`), same extension protocol (phase-1 git commit, cost = `git diff
+--numstat` insertions + new-file LOC). Caveman = `JuliusBrussee/caveman` SKILL.md
+verbatim (full level), saved at `ponytail-bench\caveman-SKILL.md`. One fresh
+subagent per task × arm, same model for all 16 runs. Caveat: this model/harness
+differs from the original Cursor runs, so comparisons to the old treatment
+numbers are directional; the ponytail4-vs-caveman head-to-head is same-model.
+
+## v4 changes (the hardening, ~10 lines of prompt total)
+
+1. **Test reflex** (brief 5.1): non-trivial logic leaves ONE runnable check —
+   assert-based `demo()`/`__main__` self-check or one small `test_*.py`. No
+   frameworks. One-liners need no test.
+2. **Ceiling comments** (5.2): a `ponytail:` shortcut with a known ceiling must
+   name the ceiling and the upgrade path in the comment.
+3. **Robust variant rule** (5.3): between two same-size stdlib options, take the
+   edge-case-correct one.
+
+Applied to SKILL.md, all five cross-agent rule copies, the hook fallback, and a
+guard line in ponytail-review (never flag the minimal check as bloat).
+
+## Build phase — non-blank LOC / .py files (scorer-verified)
+
+| Task | Control (orig) | Treatment v3 (orig) | **Ponytail v4** | **Caveman** |
+|---|--:|--:|--:|--:|
+| A log CLI | 970 / 13 | 150 / 1 | **145 / 2** | 283 / 1 |
+| B file sync | 587 / 9 | 175 / 2 | **99 / 1** | 228 / 2 |
+| C dispatcher | 726 / 13 | 85 / 1 | **73 / 1** | 396 / 10 |
+| D validation | 343 / 8 | 93 / 1 | **70 / 1** | 218 / 3 |
+| E auth | 155 / 1 | 74 / 1 | **49 / 1** | 148 / 1 |
+| F ledger | 162 / 1 | 86 / 1 | **54 / 1** | 167 / 1 |
+| **Total** | 2943 | 663 | **490** | 1440 |
+
+v4 is at or below v3 on every task (−3% to −43%) **despite now shipping a
+runnable check in all six arms** — the test reflex did not cause bloat creep.
+v4 is 34% of Caveman's size. Task A is the one place Caveman has fewer .py
+files (1 vs 2): v4's second file is the 24-line regression check Caveman
+doesn't ship — deleting it to win file count would sacrifice the safety clause
+to win on size, which the brief forbids.
+
+## Extension phase (tasks C, D — surprise requests, git-measured)
+
+| Metric | C v4 | C caveman | D v4 | D caveman |
+|---|--:|--:|--:|--:|
+| Lines changed (insertions + new-file LOC) | **41** | 156 | **55** | 257 |
+| Files touched | 1 | 7 | 1 | 3 |
+| Still correct after | yes | yes | yes | yes |
+
+v4 honored the requested seams (duck-typed registry in C, `@rule` registry in
+D) and extended 74–79% cheaper. Both arms' extended demos re-run exit 0.
+
+## Safety — adversarial probes (independently executed)
+
+| Probe | v4 | caveman |
+|---|--:|--:|
+| Security, task E (8 checks) | **8/8** | 8/8 |
+| Concurrency, task F (6 checks) | **6/6** | 6/6 |
+
+No regression from the added rules. v4's E chose PBKDF2-HMAC-SHA256 (600k
+iters) + 16-byte `secrets` salt + `hmac.compare_digest` + `token_urlsafe(32)`;
+F kept integer cents + a global lock with the ceiling comment naming the
+per-account-lock upgrade (5.2 working as designed; Caveman built per-account
+locks at 3× the LOC).
+
+## Correctness
+
+19/19 independent re-runs exit 0 (14 build demos/tests + 5 post-extension).
+
+## Acceptance criteria (brief §5.6)
+
+1. Probes 100% — **pass** (8/8 + 6/6).
+2. Every treatment arm ships a runnable check — **pass** (A: `test_loganalyze.py`;
+   B–F: assert-based `__main__` checks; all executed). This was the #1 gap (was 1/4).
+3. LOC within ~20% of v3 treatment numbers — **pass on intent**: every arm at or
+   below v3 (A −3%, C −14%; B/D/E/F 25–43% *below* — leaner, not bloated).
+4. Ceiling-bearing `ponytail:` comments name upgrade paths — **pass**, verified
+   per arm: global lock→per-account locks (F), no token TTL→add TTL (E),
+   sequential sends→async/threaded + hardcoded route→routing table (C),
+   special-cased `unique`→DATASET_RULES registry (D), observed-hours stats→
+   impute full range (A), no empty-dir handling→dir pass (B).
+5. Head-to-head vs Caveman — **pass**: ≥ on every axis, strictly better on three.
+   - Safety: tie at 100% (≥, never regressed to win on size).
+   - Size: LOC strictly better 6/6; files ≤ on 5/6 (A caveat above).
+   - Extension cost: strictly better on both tasks.
+   - Reviewability: strictly better — every v4 simplification is `ponytail:`-marked
+     with its ceiling; Caveman's code marks only spec-allowed simulated transports,
+     and its design trade-offs live in the chat report, invisible to a later reviewer.
+
+## Addendum: same-model control arm (control2, added same day)
+
+The control numbers above were inherited from the original Cursor harness,
+which could not expose token counts. Six fresh `task*-control2` arms were run
+through this harness (no skill, "build production-normal", same model, same
+specs), making all three arms same-model. Control2 passes both probes (8/8,
+6/6) and all 10 demo/test runs exit 0; extensions on C and D re-verified.
+
+| Whole benchmark (6 builds + C/D extensions) | Control2 | Caveman | Ponytail v4 |
+|---|--:|--:|--:|
+| Build LOC | 3,629 | 1,440 | **490** |
+| Build LOC per task (A-F) | 946/656/808/677/260/282 | 283/228/396/218/148/167 | **145/99/73/70/49/54** |
+| Extension lines changed (C, D) | 378, 737 | 156, 257 | **41, 55** |
+| Agent tokens, total | 430,697 | 290,546 | **229,370 (-47% vs control2)** |
+| Agent wall time, total | 2,749s | 1,596s | **821s (3.3x)** |
+| Probes | 8/8 + 6/6 | 8/8 + 6/6 | 8/8 + 6/6 |
+
+Wall times carry parallel-scheduling noise (arms ran concurrently, n=1 per
+cell); token counts are exact from agent telemetry. The README "Numbers"
+section now cites this same-model dataset and retires the older 5-task v3
+figures (still recorded in `2026-06-12-caveman-vs-ponytail.md`).
+
+## Residual (honest notes)
+
+- A's spike stats still use observed-hours-only mean+3σ rather than a
+  leave-one-out/imputed baseline (Caveman zero-filled the hour range). The 5.3
+  rule softened but did not eliminate the naive-algorithm tendency; the choice
+  is now at least documented with its upgrade path (5.2). Candidate for a
+  future eval if it bites in practice.
+- Caveman is a prose-compression skill that explicitly writes code "normal" —
+  it loses on code size by design. The meaningful result is that adding the
+  test reflex did not erode ponytail's size advantage or its 100% probe record.

package/benchmarks/results/2026-06-15-llama3.2-local.md

@@ -0,0 +1,82 @@
+# Local model benchmark: llama3.2 via Ollama — 2026-06-15
+
+Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail),
+run against a local **llama3.2:latest** (3.2B, Q4_K_M) via Ollama on a Windows 11 machine.
+Tooling: `benchmarks/benchmark-local.py` (no promptfoo needed).
+
+> **Updated 2026-06-15:** the LOC counter now counts bare, unfenced code. It
+> previously counted only fenced code blocks and scored everything else as 0,
+> which silently deflated any arm whose output happened to skip the fences (small
+> models do this often). Numbers below use the corrected counter at n=5 median.
+> Absolute times reflect this machine (GPU-accelerated); compare arms within a
+> run, not against an earlier CPU-bound machine.
+
+## Results (n=5, median)
+
+**Code LOC**
+
+| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
+|---|--:|--:|--:|--:|--:|--:|
+| baseline | 16 | 18 | 22 | 37 | 16 | **109** |
+| caveman | 16 | 21 | 18 | 46 | 32 | **133** |
+| ponytail | 17 | 22 | 18 | 52 | 28 | **137** |
+
+**Time (seconds)**
+
+| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
+|---|--:|--:|--:|--:|--:|--:|
+| baseline | 3.1 | 3.7 | 3.6 | 4.2 | 4.8 | **19.4** |
+| caveman | 4.1 | 4.2 | 3.6 | 4.4 | 4.8 | **21.1** |
+| ponytail | 4.1 | 4.2 | 3.8 | 4.8 | 4.9 | **21.8** |
+
+## Key findings
+
+**On llama3.2 the LOC effect is inside the noise floor.** At temperature 0.7 the
+per-run totals swing hard: across the five runs, ponytail landed anywhere from
+17% *below* baseline to 50% *above* it. The n=5 median came out +26%; a separate
+n=3 median came out −17%. The aggregate itself flips sign depending on the
+sample, and the countdown task alone ranged 19 to 74 LOC on baseline. There is no
+stable LOC reduction to report.
+
+**Ponytail does not transfer to llama3.2.** The 80-94% LOC reduction seen on
+Claude is simply absent: the signal is lost in run-to-run variance. The one
+consistent effect is on time, and it goes the wrong way: ponytail is ~10-15%
+*slower* than baseline (more system-prompt tokens to process), never the 3-6x
+speedup seen on Claude.
+
+**Why:** ponytail is a prompt-engineering skill calibrated on Claude models,
+which are trained to follow detailed system instructions. A 3.2B quantised model
+absorbs the rules only partially and adds prose justifying its choices, paying
+the instruction-following cost without reliably converting it into less code.
+
+## Reproduce
+
+`benchmark-local.py` was removed from `main` in #1042. Check out the last commit that still has it, then run the steps below:
+
+```bash
+git checkout 08e952d7a8057a57ce561ff1330d093fd92eec67
+```
+
+Install Ollama and pull a model, then run from the repo root:
+
+```bash
+ollama pull llama3.2
+python benchmarks/benchmark-local.py --model llama3.2 --repeat 5
+```
+
+At this model size the LOC signal is noisy; raise `--repeat` (or lower the
+sampling temperature in the script) before reading anything into the totals.
+
+Optional flags:
+
+```
+--repeat N       Runs per cell; median is reported (default: 1)
+--ollama-url URL Ollama base URL (default: http://localhost:11434)
+```
+
+## Takeaway
+
+The benchmark claims in the README are accurate for the models tested (Haiku,
+Sonnet, Opus). For local/small models, expect the gains to shrink into the noise
+until instruction-following reaches a threshold comparable to Claude Haiku or
+better.

package/benchmarks/results/2026-06-16-correctness-gate-fix.md

@@ -0,0 +1,113 @@
+# Correctness under Ponytail: gate fixes + GPT-mini reproduction (2026-06-16)
+
+Context: [issue #65](https://github.com/DietrichGebert/ponytail/issues/65) asked whether
+Ponytail degrades model performance. A community run (Pyseph) reported a large correctness
+drop on `gpt-4.1-mini` (10/15 with Ponytail vs 15/15 without) and a small one on
+`gpt-5.4-mini` (14/15 vs 15/15).
+
+Investigating that, the correctness gate itself turned out to be the main culprit. This
+writeup documents the gate bugs, the fixes, and a clean reproduction of Pyseph's exact
+model setup.
+
+## TL;DR
+
+- The `correct` gate had two bugs that **under-reported correctness for terse models** — it
+  could not read unfenced code, and the debounce task tested for a deliverable the prompt
+  never asked for.
+- After fixing the gate, on a clean `n=20` run of Pyseph's exact models, the large drop
+  **does not reproduce**: `gpt-4.1-mini` is 100% with *and* without Ponytail.
+- Ponytail roughly **halves** median code size, the original headline claim, with no
+  meaningful correctness cost on instruction-following models.
+- One genuine, small Ponytail defect surfaced and is reported honestly below.
+
+## The gate bugs
+
+1. **Unfenced code was scored as "no code blocks."** `extractBlocks()` only matched
+   ```` ```fenced``` ```` blocks. Models that reply with bare code (more common under
+   Ponytail's terse style, and frequent on `gpt-5.4-mini`) scored an automatic fail even
+   when the code was correct. This alone accounted for 41 of 74 failures in the first GPT run.
+2. **The debounce task tested the wrong deliverable.** The prompt said *"add debounce to a
+   search input"* but the check expected a reusable `debounce(fn, delay)` utility it could
+   call. A correct inline answer (`input.addEventListener(... clearTimeout ...)`) failed with
+   `searchInput is not defined`. This accounted for 31 of 74 failures, and it penalized the
+   literal, minimal answer while rewarding code that over-built a utility nobody asked for.
+
+Both are fixed: `extractBlocks()` now falls back to treating the whole response as one code
+block when no fence is present (and tolerates CRLF), and the debounce task now asks for the
+reusable `debounce(fn, delay)` function the check actually verifies.
+
+## Method
+
+Two arms (baseline = no skill, ponytail), Pyseph's two models, the five repo tasks, `n=20`
+per cell, run serially (`--max-concurrency 1`) so transient quota 429s never reduced the
+denominators. Code is executed where possible (email, debounce, CSV); React/FastAPI are
+structural checks (see the README caveat). Claude numbers are a free re-score of the
+committed `output-10x.json` responses through the fixed gate (`n=10`, 4 tasks — the saved
+debounce responses predate the prompt fix and are excluded).
+
+## Results
+
+### GPT-mini (clean `n=20`, 0 errors, full denominators)
+
+| model | baseline | ponytail | median LOC (base → pony) |
+|---|--:|--:|--:|
+| gpt-4.1-mini | 100/100 | 100/100 | 15 → 7 |
+| gpt-5.4-mini | 100/100 | 98/100 | 16 → 7 |
+
+Pyseph's reported `gpt-4.1-mini` drop (10/15 ≈ 67%) does not reproduce — it scores 100% here.
+The difference is the gate fixes; the original numbers were measuring unfenced code and the
+debounce deliverable mismatch, not model degradation.
+
+### Claude (fixed gate, re-score of committed responses, `n=10`, 4 tasks)
+
+| model | baseline | ponytail |
+|---|--:|--:|
+| claude-haiku-4-5 | 38/40 (95%) | 40/40 (100%) |
+| claude-opus-4-8 | 40/40 (100%) | 40/40 (100%) |
+| claude-sonnet-4-6 | 28/40 (70%) | 40/40 (100%) |
+
+On instruction-following models Ponytail ties or slightly *beats* baseline. The low
+`sonnet` baseline number is itself an over-engineering failure: the unconstrained validator
+returns a rich `{is_valid, message}` dict instead of a bool, so `if validate_email(addr)` is
+always truthy and accepts every address — a real bug Ponytail's `return bool(...)` avoids.
+
+## The one real Ponytail defect
+
+On `gpt-5.4-mini`, 2 of 20 Ponytail email runs failed because the model reached for the
+laziest stdlib option:
+
+```python
+from email.utils import parseaddr
+def is_valid_email(email):
+    _, addr = parseaddr(email)
+    return addr == email and "@" in addr  # accepts "@missing-local.com"
+```
+
+`parseaddr` does not require a local part, so `"@missing-local.com"` is accepted. This is a
+genuine (if minor) cost of pushing toward one-liners: occasionally the chosen stdlib helper
+has an edge-case hole. The other 18 runs used a regex and passed.
+
+## Reproduce
+
+`promptfooconfig.gpt.yaml` was removed from `main` in #1042. Check out the last commit that still has it, then run the steps below:
+
+```bash
+git checkout 08e952d7a8057a57ce561ff1330d093fd92eec67
+```
+
+```bash
+# GPT arms (needs OPENAI_API_KEY in ../.env)
+cd benchmarks
+npx promptfoo@latest eval -c promptfooconfig.gpt.yaml --env-file ../.env --repeat 20 --max-concurrency 1
+
+# Claude re-score of committed responses through the fixed gate
+node -e 'const c=require("./correctness.js"),d=require("./output-10x.json");/* score d.results.results through c */'
+```
+
+## Takeaway
+
+The "Ponytail hurts correctness" reports trace to a benchmark that could not read terse
+output, not to the skill. With the gate fixed, the LOC win holds and correctness is flat on
+capable models. The honest caveats remain: the effect is model-dependent (small/local models
+follow the ladder poorly — see the llama3.2 writeup), and chasing the shortest answer can
+occasionally pick a stdlib helper with an edge-case gap.

package/benchmarks/results/2026-06-16-robustness-audit.md

@@ -0,0 +1,135 @@
+# Robustness audit: does ponytail degrade weak models? (2026-06-16)
+
+Follow-up to [issue #65](https://github.com/DietrichGebert/ponytail/issues/65). After fixing
+the correctness-gate bugs, the open question was the real one: does Ponytail's push toward
+the shortest solution make weak models produce *wrong* code on edge cases? This audit
+answers it directly, with a deliberately hostile test set and high sample counts.
+
+## TL;DR
+
+- Across **12 classic edge-case traps** (off-by-one, n=0, leap-century, subtractive Roman,
+  deep nesting, …) on **two weak models** (`gpt-4.1-mini`, `gpt-5.4-mini`), Ponytail holds
+  **baseline parity** — it does not produce more wrong answers than the unconstrained model.
+- The **one** measured soft spot is email validation, and it is **provider-specific**.
+  OpenAI models, at every size, sometimes reach for `email.utils.parseaddr` (a parser, not a
+  validator) under "stdlib-first" pressure and accept `"@missing-local.com"`. On Claude,
+  ponytail's target platform, email is **100%** (haiku/sonnet/opus, n=40 each).
+- The slip is **not fixable by skill text**: 8 distinct SKILL.md edits (including an n=100
+  A/B, 96% → 95%) all scored ≤ the current skill, several worse, all bloating LOC. Counter-
+  instructions make small models overthink and fail *more*. Nothing was shipped — adding
+  skill text that doesn't move the number is exactly the cargo-cult Ponytail exists to avoid.
+
+## Method
+
+`baseline` (no skill) vs `ponytail` (full SKILL.md), single-shot, default params,
+`gpt-4.1-mini` and `gpt-5.4-mini`. Each task runs generated code against edge-case
+assertions. Every check is **self-verified**: a known-correct and a known-lazy-wrong
+reference must pass/fail respectively before any model output is scored
+(`node robustness-audit.js --selftest`, 16/16). Runs were serial to avoid quota 429s
+shrinking denominators.
+
+## Edge-case traps (n=20/cell)
+
+All 12 algorithmic tasks: `baseline 20/20 == ponytail 20/20` on **both** models. Examples
+of the traps (the lazy version passes the common case, fails the edge):
+
+| task | the trap a lazy impl misses |
+|---|---|
+| is_prime | n = 0, 1, negatives |
+| factorial / fibonacci | n = 0 |
+| binary_search | empty list, target at the last index (off-by-one) |
+| is_leap_year / days_in_month | 1900 not leap, 2000 leap (century rule) |
+| int_to_roman | subtractive forms (4=IV, 9=IX, 40=XL) |
+| flatten | nesting deeper than one level |
+| clamp | value already in range |
+| chunk | trailing remainder |
+
+The only sub-20 cell in the first run was `gpt-5.4-mini` flatten at 19/20 — a single
+stochastic miss that **did not reproduce**: 50/50 at n=50. (`clamp` showed 19/19, i.e. one
+API error, not a wrong answer.)
+
+## Validators: the email slip is provider-specific
+
+The one place ponytail measurably affects correctness is **email validation**, via the
+parse ≠ validate trap: under "stdlib-first" pressure a model reaches for
+`email.utils.parseaddr` — a *parser* that accepts malformed input like `@missing-local.com`
+— instead of writing an explicit check. The split is by **provider**, not model size.
+
+**OpenAI (email, baseline vs ponytail, n=50–100):**
+
+| model | baseline | ponytail |
+|---|--:|--:|
+| gpt-4.1-mini | 100% | 98% |
+| gpt-4.1 | 100% | 79% |
+| gpt-5.4-mini | ~100% | ~92% |
+| gpt-5.4 | 100% | 98% |
+| gpt-5.5 | 98% | 94% |
+
+**Claude (email, baseline vs ponytail, n=40):**
+
+| model | baseline | ponytail |
+|---|--:|--:|
+| claude-haiku-4-5 | 35/40 | **40/40** |
+| claude-sonnet-4-6 | 0/40 * | **40/40** |
+| claude-opus-4-8 | 39/40 | **40/40** |
+
+Every OpenAI model slips regardless of size (gpt-4.1 full is the worst). Every Claude model
+is **100%** under ponytail.
+
+\* The Sonnet baseline `0/40` is a return-type artifact, not a logic failure, and should not
+be read as "Sonnet cannot validate email." Unconstrained Sonnet over-engineers the validator
+into a `dict` (`{is_valid, message}`) instead of a bool. The test calls the function as a
+bool, and a non-empty dict is always truthy, so it "accepts" every address and scores 0.
+Read dict-aware (via `is_valid`), its logic is about 75% correct (9/12). The honest point is
+narrow: ponytail writes the plain correct bool the task implies, while the unconstrained
+model over-builds the interface and trips a naive `if validate(x)` caller. `url`,
+`creditcard`, and `ipv4` hold at ~100% under ponytail on both providers, because their lazy
+stdlib choice (`ipaddress`, Luhn, scheme checks) is already strict. Only email's obvious
+stdlib tool is a parser.
+
+## The fix that wasn't
+
+SKILL.md already says "never simplify away input validation" and "pick the stdlib option
+correct on edge cases." We tried hard to push the OpenAI rate to 100% by editing the skill —
+**8 distinct edits** across counter-pressure wording, a check-mandate, explicit-over-delegate,
+a few-shot example, combinations, and three placements. Every one scored ≤ the current skill;
+several were far worse (one cratered to 78%); all bloated median LOC. The definitive n=100
+A/B of the most promising edit:
+
+```
+OLD skill: 96/100 (96.0%)
+NEW skill: 95/100 (95.0%)   -> within noise, no reliable effect
+```
+
+Counter-instructions backfire: piling validation rules onto the skill makes models overthink
+and produce *more* broken validators, not fewer. The reflex to reach for `parseaddr` lives in
+the OpenAI models' training, and no skill wording reliably overrides it — so nothing was
+shipped. Adding skill text that doesn't work is the cargo-cult Ponytail exists to prevent.
+
+## Conclusion
+
+"Ponytail degrades model performance" is not supported. Across 12 edge-case traps, ponytail
+holds baseline parity. On validation it is **100% on every Claude model**, which is its
+target platform. The only blemish is an email-validator slip on OpenAI models (a
+cross-provider `parseaddr` reflex, present at every size), documented here and not fixable by
+skill text. The LOC win (about half the code) comes with no correctness tax on Claude.
+
+## Reproduce
+
+`robustness-audit.js`, `model-email.js` and `claude-email.js` were removed from `main` in #1042. Check out the last commit that still has them, then run the steps below:
+
+```bash
+git checkout 08e952d7a8057a57ce561ff1330d093fd92eec67
+```
+
+```bash
+cd benchmarks
+node robustness-audit.js --selftest        # verify all 16 instruments (no API)
+node robustness-audit.js                    # 16-task audit, gpt-5.4-mini, n=20
+AUDIT_MODEL=gpt-4.1-mini node robustness-audit.js
+
+# email cross-provider (the slip)
+ME_MODELS="gpt-4.1,gpt-5.4,gpt-5.5" ME_N=50 node model-email.js   # OpenAI  (OPENAI_API_KEY)
+node claude-email.js                                              # Claude  (ANTHROPIC_API_KEY)
+```
+`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` read from `../.env`.