tether-sdk 0.2.9__tar.gz → 0.2.11__tar.gz

tether_sdk-0.2.11/.claude/gru/active/north-star.md

@@ -0,0 +1,64 @@
+# North Star: Polyglot-Python 80% on qwen3.5:9b
+- Slug: north-star
+- Status: active
+- Started: 2026-10-27   Updated: 2026-10-27   Closed: —
+- Branches / PRs: branch fix/north-star-cheap-wins (worktree .claude/worktrees/north-star-c852b8), tracker #173
+- Budget: warn 300k / stop 400k paid tokens, max escalations 3, Tier 3 never without ask   Used: ~small (gru only)
+
+## Goal
+Polyglot-Python (34 ex, 2 trials, `--poly-verify`, default 8192 ctx) pass rate on ollama/qwen3.5:9b: 51.5% -> 80% by v0.3.0. Lanes: default (official) and `swe` role (#172, must not score below default).
+
+## Requests
+- 2026-10-27 PO: "Get to the North Star"; dev too slow, benches unreported. Asked to review the last cross-session and close it (done: v0-2-bugs archived).
+
+## Decisions
+- 2026-10-27 PO answers: lane 1 = deterministic fixes first (no GPU); merge proof = tests + offline replay, one batched full 34x2 bench after 2-3 PRs, reported immediately; budget default 300k/400k.
+- Ranked levers (Opus assessment): (1) runaway thinking hits num_ctx, +8-12pp, two attempts inconclusive (think=false 20 rows +3; length-retry rejected); (2) compact test output (relativize 195-char workspace path = 14% of obs chars, tracebacks = 51%), +3-6pp; (3) 4/68 runs died on 120s ProviderError timeout ~6pp (floor #171); (4) test-file access #170.
+- STOP list: nudge variants, whole-set 24-run A/Bs, wall cap as part of the north star.
+
+## Problems
+- 14 exercises never pass: book-store bowling connect forth hangman paasio poker pov react rest-api scale-generator sgf-parsing transpose zipper. Bench GPU: 8GB RTX 5050, one Ollama.
+
+## Plan (task cards)
+- A: ollama default timeout 120 -> 300 (tests/providers/test_ollama.py test written; docs/reference/providers.md line 95 says 120). Gru patches (1 line).
+- B: bash output: strip workspace prefix + ==== / ---- rule lines in sandbox.execute_bash_with_code (tests/test_sandbox.py test written). Gru patches (small).
+- C (next): collapse unittest tracebacks; build bench/replay.py offline measure first.
+- Then: output headroom / bounded thinking (#135/#147), #170, #171.
+
+## Ledger
+| Task | Tier | Model | Harness | Session | Attempt | Tokens | Time | Result |
+|---|---|---|---|---|---|---|---|---|
+| A,B | plan | gru | - | - | 1 | small | - | tests written failing, patch pending |
+
+## Progress log
+- 2026-10-27 branch fix/north-star-cheap-wins; failing tests added (sandbox prefix, ollama timeout). Closed v0-2-bugs log.
+
+## Next steps
+- [ ] Patch A and B, update docs (providers.md, release notes), unit+e2e, PR with `north star:` line
+- [ ] bench/replay.py offline proof for B
+- [ ] One batched full 34x2 after A+B+C, report at once
+- 2026-10-27 merged #180 (timeout 300s, compact bash, canary). PR #181 traceback collapse (canary +0.12, offline -12.6% obs chars, e2e 69). PR #182 outage retry (#171, e2e new scenario). PO: 80% is the impossible focus goal; wants v0.2.10 plan ranked by north star. Canary tool: bench/canary.py (12 min, 7 rows). Note: bench `pass` is graded by exercise tests, so #177 (done at max steps) does NOT move the bench number, only the user-facing floor.
+
+## Progress log: Coding Ladder (2026-10-28 .. 10-29, backfilled from the session)
+- 2026-10-28: v0.2.10 cut (#182,#184,#185,#186,#187). Full 34x2 Polyglot on qwen3.5:9b: 35/68 = 51.5% (baseline 35/68), mean 0.718 -> 0.763, wall caps 11 -> 14. Flat. Blanket think=false rejected (2/12); 16384 ctx rejected (-0.21, wall caps 11% -> 67%). Method fixed in #189 (bench/compare.py paired CI, --seeded, --tasks paired).
+- 2026-10-28: Coding Ladder built (#190, #191, #193, #194, #201): bench/ladder (24 tasks, graders proven by validate.py, run.py --harness tether|naive|opencode|claude --single --provider ollama|nvidia|opencode, report/snapshot/charts). Max in-VRAM ctx: 9b 40960, 4b 81920, 7b 131072.
+- Local results: 9b and 4b both write 3 / bugfix 3 / plan 6 / review 2; coder-7b write 2 / bugfix 0 / plan 1 / review 0. Harness table (24 tasks once): 9b tether 16 (0.82) vs bare 15 vs opencode 13 (6x slower); 4b tether 13 vs bare 16; 7b tether 1 vs bare 0. Tier 4 (unsaid cases) is a wall for every local model (#200).
+- Fixes from the ladder: #192 (#177 green-test implicit gate), #195 (plain-text JSON tool call), #196 (edit --file/--content flags), #197 (wrap-up turn before read-only stall ends), #198 (done --stdin, edit --method). Issues #199 (63% of steps are reads, 24% repeats), #200 (tier 4).
+- 2026-10-29: docs merged (#202): docs/ladder/{index,results}.md, bench/ladder/baseline, charts. Memory note: never `git add -A` in this worktree (bench output dirs untracked; 6156 files had to be untracked).
+
+### Round 2: free hosted models (IN PROGRESS, not finished)
+- Runner: /tmp/claude-1000/chain6.sh (nohup; finished marker /tmp/claude-1000/chain6.done). Data: hosted-out/<harness>-<model>/results.jsonl in the north-star worktree (untracked), logs /tmp/claude-1000/hosted-*.log. Models: nvidia/nemotron-3-ultra-550b-a55b, minimaxai/minimax-m3.1-flash (NVIDIA free API, key in NVIDIA_API_KEY), deepseek-v4.1-flash-free (opencode provider only; NVIDIA's deepseek-v4.1-flash never answers, OpenCode free tier is for its own client only so no tether/bare cell).
+- Done (24 tasks once, ctx 32768): nemotron tether 19/24 (mean 0.918, 11.5 steps, 29 min) vs bare 16/24 (0.849, 4.7 steps, 20 min): tether helps a strong free model. minimax tether 6/24 (0.32, 195 min, some wall-cap timeouts) vs bare 6/24 (0.32): weak and slow, no harness difference.
+- INVALID cells to redo: opencode on nemotron and minimax (0/24): HTTP 401 "authorization header missing", the `{env:NVIDIA_API_KEY}` apiKey in the workspace opencode.json is not resolved. Try provider `env: ["NVIDIA_API_KEY"]` in the config (do not write the key into the workspace file: the model can read it).
+- Still running: deepseek via opencode (12/24 done; 1 pass so far; bugfix-5 took 11,075 s with steps=0: the wall cap (1200 s) did not kill that run, check run.py's poll/kill loop with sandbox.sh/bwrap children), then the two tether ladders (hosted-out/ladder-*).
+### Next steps (resume)
+- [ ] Wait for/inspect chain6 (hosted ladders), kill stuck opencode runs if still hung; fix the opencode 401 and re-run opencode nemotron/minimax.
+- [ ] Re-run tether on qwen3.5:4b to check the -3 gap vs the bare loop (n=1 so far), paired seeds.
+- [ ] Add round 2 to bench/ladder/baseline (snapshot.py), charts (charts.py MODELS is hard-coded to the 3 local models: make it data-driven), and a "hosted models" section in docs/ladder/results.md; PR + merge.
+- [ ] Issues #199 (repeat reads), #200 (tier 4), #183 (empty thinking turns) remain open; v0.2.11 candidates.
+
+## Progress log: #199/#136 (2026-11-02)
+- Branch fix/136-199-repeat-reads (3 code commits + tests). opencode `--standalone` is the fix for the NVIDIA 401 (the `opencode serve` process lacked the env key); bench/ladder/run.py has `--provider openrouter` and passes `--standalone`.
+- Card A (#136, 10 lines) on tether+nemotron: FAILED, stalled after 13 steps / 82k prompt tokens, reads only, 4 cycle_stall (this is #199 itself). Retried on opencode+nemotron: correct diff first try. Card B (#199) on tether+nemotron: COMPLETED, 29 steps, 205k prompt tokens, diff to spec, I fixed lint and golden. Tests 419 pass, cov 100%.
+- Tether friction: card A stall with no edit at 13 steps (model reads ranges of one file instead of editing). Not yet filed.
+- NOT measured yet: repeat-read share / steps with the change. Proposed: paired seeds, tether before vs after, qwen3.5:9b, ladder tiers 3-6 (12 tasks) x3 seeds, bench/compare.py.

{tether_sdk-0.2.9/.claude/gru/active → tether_sdk-0.2.11/.claude/gru/archive}/v0-2-bugs.md

@@ -1,7 +1,7 @@
 # v0.2 bugs
 - Slug: v0-2-bugs
-- Status: active
-- Started: 2026-10-20   Updated: 2026-10-24   Closed: —
+- Status: done
+- Started: 2026-10-20   Updated: 2026-10-24   Closed: 2026-10-27
 - Branches / PRs: branch `claude/keep-feature-context-resume-0ec918` (worktree); base `main` at 125b386 (release v0.2.2)
 
 ## Goal
@@ -108,3 +108,29 @@ Track and fix bugs found in Tether v0.2.x. Exact bug list not yet given by PO.
 - 2026-10-25 PO: milestones reshuffled: v0.2.8 = release now (#144 #145 + sandbox venv + CLI/providers docs), v0.2.9 = docs overhaul (#150-#154, #63-#66), v0.2.10 = #27 #79 #132-#137. Merged: #146 #148 #149 #155 #156 #157 #158. Release PR v0.2.8 (version bump, notes) prepared; GitHub Release only on PO yes.
 - 2026-10-25 RELEASED v0.2.8 'Room to Answer' (release PR #159 -> f4370b1, GitHub Release, delivery.yml success, PyPI tether-sdk 0.2.8 live, milestone closed). Docs v0.2.9 started: #151 site structure PR (nav = Start here/Guides/Concepts/Reference/Help, 'For:' line per page, CONTRIBUTING.md replaces docs/testing.md + releasing.md, tests/test_docs_structure.py). Lesson: ruff format --check also formats Python blocks in docs; renamed anchors break the strict mkdocs build via old release-notes links -> run both before pushing docs.
 - 2026-10-25 PO: 'merge all docs PRs and cut the next version'. Docs plan (#150) done: #155 #156 #157 #160 #161 #162 #165 #166 #167 merged, epic closed. Found #163 while writing guide 1 (qwen3.5:9b fixes the test, passes, never calls done -> FAILED Stalled); #164 merged on PO yes (Rule 3 reading: passing verify gates at stall time -> COMPLETED). Release PR v0.2.9 prepared. Next: v0.2.10 = #27 #79 #132-#137 (8 issues; pick by evidence, e.g. bench failure breakdown after v0.2.9). Tether friction this stretch: docs rewrites by Nemotron invent API facts and end without done; harness.py cards need 40+ steps.
+- 2026-10-26 v0.2.9 released (#168). Polyglot fast set + poly-verify on v0.2.9: 12/24, mean 0.765; 8192 vs 16384 context on 12 exercises: 16384 not a regression (easy 9/12 both, mean 0.872->0.937; hard 3->5/12) but ~+50% s/step (decode-bound), more 600s bench-cap timeouts; default stays 8192 (token saver), --context-tokens bench option on branch bench/137-context-tokens (no PR). Root-cause read of the breakdown found #169: bare `done` / typed `act("done", ...)` rejected, 8 of 36 runs wasted 1-13 steps, 2 false FAILED at score 1.0.
+- 2026-10-26 #169 fix (PR on branch fix/169-bare-done): tests first (e4f0feb), fix 143d79e. Ledger: #169 card A (bare done) | Tier 0 | z-ai/glm-5.3 | tether | ses_152e5df2 | attempt 2 | 7 steps, 6.6k tokens | ok (3 lines). Card B (act() text unwrap) | nemotron-3-ultra ses_d9caa19b (combined card A+B): 25 steps, 230k prompt tokens, zero edits; glm-5.3 ses (800s timeout, rate-limited, zero edits) -> escalated to plan level, 5-line patch written by gru.
+- Tether friction (#169): (1) nemotron-3-ultra on a 2-file card read 3 large files and ran pytest 12 times in 25 steps without ever calling `edit write` (read-only loop, no pre-change nudge fired before step 12, prompt 230k tokens); (2) nvidia free tier + glm-5.3 ~70s per step (rate limit) so an 800s timeout ended with no edit. Cards under ~15 lines and one file each worked (glm-5.3 7 steps).
+- 2026-10-26 #169 bench (done-fix vs v0.2.9, qwen3.5:9b, poly-verify, 2 trials, 8192): done-rejected steps 18 (4 runs) -> 0; pass 12/24 -> 10/24, mean 0.765 -> 0.694 (one infra outage cell rerun). Within noise: every failing run in the new set made 0 done calls (30 steps, never called done), so none touched the changed code. Affine-cipher t0 28 -> 7 steps, bottle-song t0 21 -> 14. Remaining dominant failure: max_steps without calling done (rewrite loops, e.g. food-chain t1 scores 1.0 yet FAILED). e2e 69 passed, unit 391 passed, 100% cov.
+- 2026-10-27 North star set (#173, AGENTS.md "North Star"): Polyglot-Python on qwen3.5:9b default 8192, full 34: 80% by v0.3.0. Baseline pinned (#178, bench/history.jsonl): main@b25cbbe 35/68 = 51.5%, mean 0.718; 17 fails at 30-step cap, 11 at the 600s bench wall cap, 14 exercises never pass (book-store bowling connect forth hangman paasio poker pov react rest-api scale-generator sgf-parsing transpose zipper). Naive no-harness loop on fast-12: 3/24 (12.5%) vs tether 12/24.
+- 2026-10-27 Experiments, all unmerged (data on issues): ctx 16384 fast-12 11/24 vs 12/24 (mean 0.807 vs 0.711, 9/13 fails hit 600s cap) -> rejected (#170); paging nudge flat; bounded cmd replay worse (10->8/24); think=false on 20 targeted rows 9/20 vs 6/20 (net +3, pre-reg +4) ; adaptive length-retry (branch fix/135-length-retry) 1/12 on failing rows -> rejected (#135); blanket think=false on fast-12 stopped at 11/24 runs (7 pass), inconclusive.
+- 2026-10-27 OPUS ASSESSMENT (architect, read-only, 13 tool calls over base34 t.db): 
+  * First-order failure nobody had measured = runaway thinking hitting num_ctx: 53 assistant turns with no action, 45 have prompt+completion in [8192,9216) (re-verified by gru: 45/53, 15 runs, 3672 s). Model thinks until the window is full, Ollama stops, tether records "(no output)" and nudges; ~85-110 s per turn. In the 11 wall-cap runs 41% of model time was empty turns: wall-cap failures are this, not slow progress. context.py:333 budgets prompt to 3/4 of ctx and leaves ~2048 for output; thinking does not fit.
+  * 4 of 68 runs died of ProviderError timeout 120s x3 (book-store t1, transpose t0, hangman t1, pig-latin t1): unretried infra loss ~6pp, breaks the floor (#171).
+  * Second mechanism: read-loop to the step cap (paasio poker react zipper sgf pov-t1 grep-t0 phone-t0 bowling-t1; fast 6-11 s/step). Test-run output (unittest tracebacks) = 51% of all observation chars; the 195-char bench workspace path = 14% of observation chars (~60 tokens per traceback line); react prints 6.5KB per test run, paasio 12KB.
+  * Why no progress: picked by proxy (re-read %) not mechanism; thinking not recorded (#32); 24-run bench sd ~2.4 passes = a +10pp change is ~1 sigma; whole-set A/B dilutes effects; the 600s wall cap measures GPU speed and thinking length (confound; ctx16 raised mean score but lost on wall); infra deaths baked into baseline.
+  * Fast assessment tiers: T0 offline replay `bench/replay.py` (~150 lines; rebuild each step's prompt from a recorded t.db through the candidate assembler; prompt tokens, output headroom, turns exceeding num_ctx, test-output chars; seconds, no GPU; proves prompt shape only). T1 determinism: same exercise twice with --param seed=1 (temp already 0.2), compare command sequences; first divergence = did the change fire; identical = zero effect (10 min). T2 counterfactual fork: copy a failing t.db, cut where the mechanism first fires, resume with the change (5-15 min/run). T3 targeted subset: only rows where the mechanism fired + 4 always-pass heavy-thinking canaries (zebra t1, vlq t0, two-bucket t0, go-counting t1), paired seeds, sequential stop (ship +4 net flips, kill at -2), <=1h. T4 full 34x2 overnight. Rule: a change gets GPU time only if T0/T1 shows its target metric moves.
+  * Parallelism: GPU is an 8GB RTX 5050; two Ollama instances will not fit; try OLLAMA_NUM_PARALLEL=2 (~1.5x?) for 10 min, only after dropping the wall cap (contention distorts wall).
+  * Ranked levers: (1) output headroom + bounded thinking, expected +8-12pp (15 affected runs, partial 0.6-0.75): test think=false T3, real fix = on done_reason=="length" with no tool call retry once with think=false, reserve output headroom in context.py, scale provider timeout to the generation budget, overlaps #135/#147; (2) compact test-run output: relativize workspace_root in bash observations, collapse unittest tracebacks to `test_x:105 AssertionError: 2 != 4`, drop ====/---- lines, +3-6pp, verify via T0 then T3; (3) test-file access #170, +2-4pp low confidence (def test_ index when file exceeds cap).
+  * 2-week workflow: D1 build replay.py + seed determinism check + queue think=false T3; D2 headroom/think-retry PR; D3-4 output-compaction PR proven with T0 then T3; bench-rules PR (official cap = 30 steps only with 1800s safety wall, p50/p90 wall as indicator, short workspace path /tmp/tb/<ex>, provider timeout no longer kills a run silently, re-pin baseline overnight once); end wk1 full T4; wk2 #170 then #171. Opus does diagnosis/tooling/acceptance tests/verdicts, minions mechanical PRs only; one GPU queue. STOP: nudge variants, whole-set 24-run A/Bs, wall cap as part of the north star.
+- 2026-10-27 Result of following it (gru): lever 1 tested twice, neither clean: think=false 20 rows +3 net, adaptive length-retry rejected (empty turns 3672s -> 332s but 20 turns still >2500 completion tokens: thinking is slow even when it fits; blanket think=false ran 405 turns in the time retry ran 243). T0/T1/T2 never built.
+- 2026-10-27 PO feedback (verbatim gist): dev is far too slow ("1 line of code and an entire day benching it"), benches finish unreported, results get thrown away because of the impossible goal; gru is losing itself on the work; wants the Opus findings logged. LESSON: stop bench-per-line-of-code loops; decide with a fast tier first, ship deterministic/obviously-correct improvements with unit+e2e evidence, report every bench end immediately (watcher fires on the results file, not on the launcher shell).
+## Next steps (as of 2026-10-27, awaiting PO decision)
+- [ ] PO to choose: (a) finish blanket think=false fast-12 (~13 runs, 45 min), (b) offline output compaction (no GPU), (c) plan only. Nothing is running; no open PRs.
+
+
+## Wrap-up (2026-10-27, closed by PO request)
+- Outcome: v0.2.3 -> v0.2.9 released (PyPI tether-sdk 0.2.9). Bugs and docs overhaul done; goal shifted to the North Star (#173): Polyglot-Python 80% on qwen3.5:9b by v0.3.0. Baseline 35/68 = 51.5% (main@b25cbbe).
+- Durable lessons: (1) never bench per line of code: decide with a fast tier first (offline replay of recorded t.db, determinism check), GPU only for targeted rows; (2) report every bench end immediately; (3) cards under ~15 lines, one file, worked on tether minions (glm-5.3 7 steps); 2-file cards and docs rewrites on Nemotron Ultra loop or invent API facts; (4) run `ruff format --check` and strict mkdocs before docs pushes; (5) nudge/wording variants and whole-set 24-run A/Bs do not move the number (sd ~2.4 passes).
+- Open debt: #27 #79 #132-#137 #170 #171; branches unmerged: fix/170-bounded-cmd-replay (rejected), fix/135-length-retry (rejected), bench/137-context-tokens; Harness-Bench #123 blocked (docker firewall + key).
+- Tier cost: Tier 0 (Nemotron Ultra, GLM-5.3 on tether) solved only tiny single-file cards; everything else was finished at plan level by gru.

{tether_sdk-0.2.9 → tether_sdk-0.2.11}/AGENTS.md

@@ -28,6 +28,15 @@ Tether stays **lightweight, reliable, extensible, model-agnostic, easy to use, r
 
 Reviewers and the Team Leader cite the failing line when they reject a change.
 
+### North Star (the one number; see tracker #173)
+**Polyglot-Python pass rate on `ollama/qwen3.5:9b` (our weakest model), default config, full 34 exercises: 80% by v0.3.0.** Nothing important is set aside for it, but everything is ranked by it:
+1. Every issue, PR and benchmark must say how it moves this number (or tokens per solved exercise at the same pass rate). A benchmark that cannot show a change in it is not run; an issue that cannot name the link waits behind one that can.
+1b. Two lanes, both tracked: the **general-purpose default** (no role) is the official number and must reach 80%; the shipped **`swe` role** (#172) is reported next to it, must never score below the default, and gets its own target once its baseline is pinned.
+2. Weakest-model rule: every stronger model must score at least what `qwen3.5:9b` scores (and at least 80% once reached). If we plateau, stronger models may not fall below the plateau.
+3. Other indicators (tokens per pass, steps, wall-clock, prompt size, coverage) may slow or pause but never degrade between releases.
+4. Floor, a promise rather than a goal: every task delivers a real, honest, usable result in a fair time; no unretried outage, no empty result, no unrelated hallucinated deliverable (#171).
+5. Every PR body and issue-close comment ends with `north star: <before> -> <after> (runs, noise, run dir)`; the tracker's History table and `bench/history.jsonl` are updated at every release.
+
 ---
 
 ## 2. The Golden Rules (Non-Negotiable Invariants)
@@ -43,10 +52,10 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
 
 ### Rule 3: Explicit Termination (`act("done", stdin="...")`), Interactive Inquiry (`done --ask`), & Pre-`done` Verification Gates
 - Sessions do **not** terminate simply because the model emitted a text response without a tool call (small 9B models frequently "think out loud" before acting).
-- A task only completes when the agent explicitly calls `act("done", stdin="<final deliverable>")` (or `act("done '<short answer>'")`), pauses for human/parent input via `act("done --ask", stdin="<question>")` (transitioning the session to `WAITING_INPUT` and releasing the worker process until resumed via `send`), or hits `max_steps`.
+- A task only completes when the agent explicitly calls `act("done", stdin="<final deliverable>")` (or `act("done '<short answer>'")`), pauses for human/parent input via `act("done --ask", stdin="<question>")` (transitioning the session to `WAITING_INPUT` and releasing the worker process until resumed via `send`), or hits `max_steps`. A bare `done` (no answer, no stdin) completes with the result `Done.` and still runs the `verify` gates; `done --ask` with no question is an error (#169).
 - If the model emits plain text without calling `act`, the harness appends a brief deterministic nudge reminding it to either execute a command or call `done`.
-- After 6 consecutive read-only/verification steps following a change, the harness appends a one-shot `[NUDGE]` to that step's observation reminding the model to call `done` if the task is complete (0 static prompt tokens, #74). Before any change, 12 read-only steps in a row (then every 8 more) append a one-line `[NUDGE]` to make the edit or call `done` (#127). Seven reads of one file in a row with no change are a `cycle_stall`, and 4 stall warnings without a change end the session `FAILED` as `Stalled` (#92); `tether stop <id>` terminates a detached worker (#89).
-- When `verify` gate commands are configured (via `Harness(verify=[...])`, CLI `--verify`, or Skill YAML `verify: [...]`), `Harness.execute_session` deterministically executes each command inside the session's `SandboxPolicy` before allowing `done` to transition to `COMPLETED` (consuming 0 static prompt tokens until a gate fails). A gate that errors instead of failing (sandbox violation, exit 124/126/127) ends the session `FAILED` at once as `gate_error` ("gate misconfigured"), without counting toward the consecutive-failure abort. When the stall guard is about to end a session as `Stalled` and gates are configured, it runs them first: a change was made and every gate passes, so the session ends `COMPLETED` ("Finished by the harness"), since the deterministic gate is the completion criterion the caller asked for (#163).
+- After 6 consecutive read-only/verification steps following a change, the harness appends a one-shot `[NUDGE]` to that step's observation reminding the model to call `done` if the task is complete (0 static prompt tokens, #74). Before any change, 12 read-only steps in a row (then every 8 more) append a one-line `[NUDGE]` to make the edit or call `done` (#127). Seven reads of one file in a row with no change are a `cycle_stall`, and 4 stall warnings without a change end the session `FAILED` as `Stalled` (#92), after one `[WRAP UP]` turn that asks a read-only session for its final answer (the 4th warning triggers it, one more stall ends the session); `tether stop <id>` terminates a detached worker (#89).
+- When `verify` gate commands are configured (via `Harness(verify=[...])`, CLI `--verify`, or Skill YAML `verify: [...]`), `Harness.execute_session` deterministically executes each command inside the session's `SandboxPolicy` before allowing `done` to transition to `COMPLETED` (consuming 0 static prompt tokens until a gate fails). A gate that errors instead of failing (sandbox violation, exit 124/126/127) ends the session `FAILED` at once as `gate_error` ("gate misconfigured"), without counting toward the consecutive-failure abort. When the stall guard is about to end a session as `Stalled` and gates are configured, it runs them first: a change was made and every gate passes, so the session ends `COMPLETED` ("Finished by the harness"), since the deterministic gate is the completion criterion the caller asked for (#163). With no gates, the model's own last passing test run (runner-recognised, nothing changed since, still green when re-run) is the implicit gate, at a stall and at the step cap; a passing run after a change also appends a one-line `[NUDGE]` to call `done` (#177).
 
 ### Rule 4: Progressive `--help` (JIT Schema Disclosure)
 - The system prompt lists at most **1 line per enabled tool namespace** (`- <name>: <short summary>`, target **≤ 12 tokens per entry**).
@@ -205,7 +214,7 @@ uv run mypy tether/
 2. **Async-First Design:**
    - All model calls and I/O-bound tool executions must be `async`/`await` native, while seamlessly wrapping synchronous user `@tool` functions.
 3. **Graceful Degradation on 9B Hallucinations:**
-   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`), unwrapping `bash -c/-lc '<cmd>'` and `sh -c` wrappers, running an unknown first word through the enabled `bash` tool (never for disabled built-ins), registering extra quirk fixers with `register_normalizer`, mapping a native tool call named other than `act` to `<name> --k 'v'` so the JIT usage answers it, inferring the `edit` verb (`edit <path>` -> view/replace/write) and accepting `apply_patch` (`*** Begin Patch` or unified-diff patches, also when a diff arrives as `edit <path>` stdin) as an `edit` verb.
+   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`), rewriting `edit --file p --content c` flags into `edit write p` + stdin and `done --stdin 'x'` into `done` + stdin, unwrapping `bash -c/-lc '<cmd>'` and `sh -c` wrappers, unwrapping an `act("<cmd>", stdin="...")` call typed as text in `cmd` (also after `bash`) into the command and its `stdin` (#169), running an unknown first word through the enabled `bash` tool (never for disabled built-ins), registering extra quirk fixers with `register_normalizer`, mapping a native tool call named other than `act` (also written as plain-text JSON `{"name": ..., "arguments": {...}}`) to `<name> --k 'v'` so the JIT usage answers it, inferring the `edit` verb (`edit <path>` -> view/replace/write) and accepting `apply_patch` (`*** Begin Patch` or unified-diff patches, also when a diff arrives as `edit <path>` stdin) as an `edit` verb.
 4. **Golden traces:** behaviour the model sees (framing, folding, nudges, tool output) is locked by `tests/golden/*.snap.json`. A PR that changes it runs `UPDATE_GOLDEN=1 pytest tests/test_golden.py` and the snapshot diff is part of the review; unrelated traces must not change. See `CONTRIBUTING.md`.
 5. **E2E scenarios:** every feature or bug fix ships with a scenario in `e2e/` (real CLI, real ollama behind a fault-injecting proxy, local only, `TETHER_E2E=1 uv run pytest e2e`). Assert on what tether sends and records, never on the model's skill; script the reply or inject the fault so the test does not depend on model potency. **Before merging any PR, run `TETHER_E2E=1 uv run pytest e2e` locally and put the result in the PR description (CI cannot run it). Every failure must be mitigated before merge: fix the code, or fix a wrong scenario, or record in the PR why it is acceptable and what follows up. A PR is not merged with an unexplained red scenario.** See `CONTRIBUTING.md`.
 6. **Documentation Integrity:**
@@ -225,5 +234,5 @@ uv run mypy tether/
 - Hard stops (ask first): force push / history rewrite; merging a PR and cutting a release need the PO's explicit yes each time
 - Design docs: `AGENTS.md` sections 1-3, `docs/architecture.md`, `docs/reference/*`
 - Orchestration: **`/gru` runs this repo** (Team Leader + router). Strong model plans, writes the failing acceptance tests first, reviews and decides; tether does the typing as the minion at Tier 0 on the strongest free model (`nvidia/nvidia/nemotron-3-ultra-550b-a55b`, fallback `nvidia/z-ai/glm-5.3`), escalating one tier only on evidence. Tiers, context sizes and sensitive paths: `.claude/gru.json`. tether's own failures while doing a card are logged as friction and filed as issues (with the PO's OK).
-- Improvement loop: **every PR leaves tether a little better**, measured, not asserted: a behaviour change ships with a bench or e2e delta (Polyglot-Python fast set on `ollama/qwen3.5:9b`, `bench/`) in the PR body, reported honestly even when it is flat. After a merge, gru re-reads the next planned milestone and the other open bug issues and picks the one that clearly improves tether most (evidence from bench failure breakdowns and `tether stats` friction beats guesses); features and docs wait behind bugs that cost steps or pass rate.
+- Improvement loop: **every PR leaves tether a little better**, measured, not asserted: a behaviour change ships with a bench or e2e delta (Polyglot-Python on `ollama/qwen3.5:9b`, `bench/`) in the PR body as a `north star:` line (see North Star), reported honestly even when it is flat. After a merge, gru re-reads the next planned milestone and the other open bug issues and picks the one that clearly improves tether most (evidence from bench failure breakdowns and `tether stats` friction beats guesses); features and docs wait behind bugs that cost steps or pass rate.
 - gru store: .claude/gru/   (committed; legacy `feature-context` logs were moved here)

{tether_sdk-0.2.9 → tether_sdk-0.2.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tether-sdk
-Version: 0.2.9
+Version: 0.2.11
 Summary: General-purpose, headless-first LLM-Native micro-harness for small language models
 Project-URL: Homepage, https://github.com/igormcsouza/tether
 Project-URL: Repository, https://github.com/igormcsouza/tether

tether_sdk-0.2.11/bench/README.md

@@ -0,0 +1,81 @@
+# bench: model-in-the-loop checks (developers only)
+
+Not part of the `tether` package, not run in CI. Use it to onboard a new model or to check a
+release for regressions that deterministic tests cannot see (unit, golden and e2e tests stay the
+gate; see `CONTRIBUTING.md`).
+
+Nine tasks in `tasks/<name>/` (workspace in `ws/`, goal in `goal.txt`, hidden tests in `hidden/`):
+`focus` (one exact edit in a distractor-heavy repo, scope-checked), `plan` (an implementation plan, checked for named files, failure handling and tests), `codereview` (eight seeded bugs to find, no edits), `code-simple`, `code-medium`, `code-hard` (graded by hidden unit tests; editing the visible tests
+scores 0), `qa`, `content`, `review` (graded by constraint counts and keyword checks, so they are
+a smoke test for quality, not a judge).
+
+**Polyglot-Python** is the score we answer to. `--tasks polyglot` runs the 34 Python exercises of Aider's
+[polyglot-benchmark](https://github.com/Aider-AI/polyglot-benchmark) (cloned once into
+`~/.cache/tether-bench/polyglot-benchmark`, or set `POLYGLOT_DIR`); `--tasks polyglot-fast` runs a 12-exercise
+subset that mixes passes, partials and failures. Each task is graded by the exercise's own tests (the test file is
+restored before grading, so editing it cannot pass), `--poly-verify` adds a `--verify` gate on those tests. Then
+`python3 bench/report.py <out>` prints pass rate and mean score with a failure breakdown that keeps provider outages
+(`outage`, `timeout`) apart from harness outcomes (`max_steps`, `stalled`, `wrong_answer`, `worker_error`).
+
+**Did my change help? (15 min)** `python3 bench/canary.py /tmp/canary-<change> [--tether-bin <venv>/bin/tether]` runs 7 exercises
+once (4 flaky ones that can flip, 3 fast guards) and prints each score next to the pinned baseline with a verdict
+(`REJECT` / `PROMISING` / `FLAT`). A `PROMISING` canary only says "worth a paired A/B" (see *Judging a change* below), not "improved". `--tasks canary` is the same set for `run.py`.
+
+Three runners, so a gap can be attributed:
+
+| runner | what it is |
+|---|---|
+| `tether` | the CLI under test (`--tether-bin` picks a version or a venv) |
+| `naive` | `naive_agent.py`, a plain OpenAI-compatible tool loop (bash/read/write) with the same model |
+| `claude` | Claude Code in print mode, as a frontier baseline (`--models cc/sonnet`) |
+
+```bash
+export NVIDIA_API_KEY=...            # only for nvidia/* models; OLLAMA_HOST defaults to :11435
+uv run python bench/run.py --runners tether,naive --models ollama/qwen3.5:9b \
+    --tasks code-simple,qa --trials 2 --out /tmp/bench
+uv run python bench/run.py --runners tether --tether-bin /path/to/venv/bin/tether \
+    --models ollama/qwen3.5:9b --param think=false --out /tmp/bench-v0.2.3
+```
+
+Each run lands in `<out>/<runner>__<model>__<task>__t<k>/` with the workspace, stdout, the
+session DB (`t.db`) and `done.json`; `results.jsonl` collects them. Re-running skips finished
+cells; `--redo-infra` reruns cells that failed on provider errors (429, 503, timeouts).
+
+Read results with the friction events (`syntax_error`, `loop_stall`, `nudge`, ...) next to the
+score: a low score with many events is a harness problem, a low score with none is the model.
+Rate limits (NVIDIA: 40 requests/min) invalidate results silently; set `rpm:` in `providers.yaml`
+and keep parallelism low (`--par nvidia=2`).
+
+Caveats: one run per cell is noisy (about 0.1); the scores are regex and hidden-test based.
+
+## Coding ladder (`bench/ladder/`)
+
+24 coding tasks on a small `shop` project: 6 tiers x {write, bugfix, plan, review}. Tier 1 gives file, place and code; each tier gives less (named behaviour, find the file, edge cases left unsaid, several modules, an open goal). Code tasks are graded by hidden unittest files, plan/review tasks by regex rubrics (reviews lose points for false alarms). `python3 bench/ladder/validate.py` proves every grader (the start state fails, the reference solution passes).
+
+```bash
+python3 bench/ladder/run.py --out /tmp/ladder --models qwen3.5:9b,qwen3.5:4b,qwen2.5-coder:7b \
+    --ctx qwen3.5:9b=40960,qwen3.5:4b=65536,qwen2.5-coder:7b=32768 --tether-bin .venv/bin/tether
+python3 bench/ladder/report.py /tmp/ladder
+```
+
+Per (model, kind) the tiers are climbed in order. A tier gets up to 4 attempts, stopping at the first pass: thinking off; thinking on; thinking on with more steps and later folding; thinking off at temperature 0 with more steps. Four failures end that kind's ladder, so the next tier is never tried. A run is stopped as soon as the work passes its grader ("delivered"), so models that never call `done` do not burn GPU time; the report shows steps at delivery. Pick `--ctx` as the largest window that stays fully in VRAM (`/api/ps`: `size == size_vram`).
+
+**Harness comparison.** `run.py --harness tether|naive|opencode|claude --single` runs all 24 tasks once, no stopping, on the same model, so the harnesses can be set side by side (`report.py` prints one block per harness). `naive` is `bench/naive_agent.py` (a bare tool loop), `opencode` runs through the gru `sandbox.sh` and needs an `ollama` provider in `opencode.json` pointing at `http://localhost:11435/v1` plus a model alias created with the same `num_ctx` (`/v1` cannot set it), `claude` is Claude Code in print mode with a $1 budget cap per task.
+
+## Judging a change (what a number can and cannot tell you)
+
+Measured on the baseline and v0.2.10 full runs (#173): 13 of 34 exercises never pass, 11 always pass, and only 10 ever flip. 12 of 68 run slots flipped between the two runs, 6 up and 6 down, and the paired score delta was +0.045 with a 95% CI of [-0.029, +0.121]. A pass rate over 68 runs (standard error ~6 points) and a canary of rows picked *because they failed once* (regression to the mean) both mislead. Use this instead:
+
+1. **Mechanism first (minutes, no GPU).** A change must move its own metric before it earns a run: empty turns, test-file reads, tokens per observation, measured on recorded `t.db` files.
+2. **Greedy A/B.** `--param temperature=0` makes a run repeatable: the same code and prompt gave identical command sequences three times, including a run that hit the wall cap. At temperature 0.2 with `--seeded` short runs repeat but runs of 20+ steps drift apart (GPU float order), so seeded sampling only reduces noise. Run the baseline arm once per commit and reuse it; run only the new arm.
+3. **Compare, do not eyeball.** `python3 bench/compare.py <before-dir> <after-dir>` pairs runs by exercise and prints the score delta with a bootstrap 95% CI over exercises, the pass flips with an exact sign test, and the mechanism numbers (wall-cap rate, steps, nudges per run). `BETTER`/`WORSE` only when the CI excludes 0; otherwise `NO EVIDENCE`, with the number of runs it would take.
+4. **Pick rows before you look**, never by one failure. `--tasks paired` is a fixed 10-exercise set that flips or scores partially (about 40 min per arm).
+5. The official number stays the full 34 x 2 run, once per release.
+
+## History (the north-star ledger)
+
+`bench/history.jsonl` is the single source for the tracker's History table (#173). One JSON object per line: `date`, `version`, `model`, `runner`, `tasks`, `trials`, `context`, `runs`, `passed`, `pass_rate`, `mean_score`, failure counts, mean steps/prompt tokens/wall, `note`.
+
+A PR that changes behaviour the model sees appends one line for its own run (same model, default 8192 context, `--poly-verify`) and ends its body with `north star: <before> -> <after> (runs, noise, run dir)`. Noise is about 0.1 per run, so a claim needs enough runs to beat it; a flat result is appended and reported as flat.
+
+The official number is the full 34 exercises × 2 trials (68 runs) with the 600 s per-run wall cap. Use `--tasks polyglot-fast` for quick checks and label it `polyglot-fast-12`.

tether_sdk-0.2.11/bench/canary.py

@@ -0,0 +1,39 @@
+"""15-minute check: did this change help? `python3 bench/canary.py <out-dir> [--tether-bin ...]`.
+
+Runs 7 Polyglot exercises once (4 flaky ones that can flip, 3 fast guards), then compares each
+score with the pinned baseline (`canary_baseline.json`, mean of 2 runs on main@b25cbbe).
+Verdict: REJECT if a guard drops >0.2, PROMISING if the flaky mean gains >=0.1, else FLAT.
+Only PROMISING changes earn a full 34x2 run. One run per exercise is noisy: a FLAT verdict
+means "no evidence", not "proven useless".
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+import grade  # noqa: E402
+
+
+def main(out, extra):
+    cmd = [sys.executable, f"{HERE}/run.py", "--models", "ollama/qwen3.5:9b", "--tasks", "canary",
+           "--poly-verify", "--par", "ollama=1", "--out", out, *extra]
+    subprocess.run(cmd, check=True)
+    base = json.load(open(f"{HERE}/canary_baseline.json"))
+    now = {}
+    for line in open(f"{out}/results.jsonl"):
+        r = json.loads(line)
+        now[r["task"][5:]] = r["score"]
+    flaky = [e for e in base if e not in grade.GUARDS]
+    for e in base:
+        print(f"{e:20} {base[e]:.2f} -> {now.get(e, float('nan')):.2f}{'  (guard)' if e in grade.GUARDS else ''}")
+    gain = sum(now.get(e, 0) - base[e] for e in flaky) / len(flaky)
+    drops = [e for e in grade.GUARDS if now.get(e, 0) < base[e] - 0.2]
+    verdict = "REJECT" if drops else "PROMISING" if gain >= 0.1 else "FLAT"
+    print(f"flaky mean delta {gain:+.2f}; guard drops {drops or 'none'} -> {verdict}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2:])

tether_sdk-0.2.11/bench/canary_baseline.json

@@ -0,0 +1 @@
+{"phone-number": 0.69, "tree-building": 0.92, "grep": 0.66, "pig-latin": 0.91, "proverb": 1.0, "robot-name": 1.0, "list-ops": 1.0}

tether_sdk-0.2.11/bench/compare.py

@@ -0,0 +1,102 @@
+"""Paired A/B of two bench runs: `python3 bench/compare.py <before-dir> <after-dir>`.
+
+A pass rate over 68 runs has a standard error of ~6 points and 10 of 34 exercises carry all the
+flips, so "35/68 vs 35/68" or "9/20 vs 6/20" cannot tell a change from luck. This reports what can:
+the paired per-run score delta with a bootstrap 95% CI resampled over exercises (runs of one
+exercise are not independent), the up/down flips with an exact sign test, and the mechanism numbers
+(wall-cap rate, steps, empty turns) that move before the pass rate does.
+
+Verdict: BETTER / WORSE when the score CI excludes 0, else NO EVIDENCE with the number of paired
+runs it would take to see the observed delta (a delta of ~0 reports "no effect detectable").
+"""
+
+import json
+import math
+import random
+import statistics as st
+import sys
+from collections import defaultdict
+
+
+def load(path):
+    rows = defaultdict(dict)
+    for line in open(f"{path}/results.jsonl"):
+        r = json.loads(line)
+        rows[r["task"]][r["trial"]] = r
+    return rows
+
+
+def sign_p(up, down):
+    n = up + down
+    if n == 0:
+        return 1.0
+    k = min(up, down)
+    return min(1.0, 2 * sum(math.comb(n, i) for i in range(k + 1)) / 2**n)
+
+
+def mean_delta(by_task):
+    ds = [d for v in by_task.values() for d in v]
+    return st.mean(ds) if ds else 0.0
+
+
+def compare(a, b, boot=4000, seed=1):
+    tasks = sorted(set(a) & set(b))
+    by_task, up, down, pairs = {}, 0, 0, 0
+    for t in tasks:
+        ds = []
+        for i in set(a[t]) & set(b[t]):
+            ra, rb = a[t][i], b[t][i]
+            ds.append(rb["score"] - ra["score"])
+            up += (not ra["pass"]) and rb["pass"]
+            down += ra["pass"] and not rb["pass"]
+            pairs += 1
+        if ds:
+            by_task[t] = ds
+    rng, keys, means = random.Random(seed), list(by_task), []
+    for _ in range(boot):
+        pick = [by_task[rng.choice(keys)] for _ in keys]
+        means.append(mean_delta(dict(enumerate(pick))))
+    means.sort()
+    lo, hi = means[int(0.025 * boot)], means[int(0.975 * boot) - 1]
+    return by_task, pairs, up, down, lo, hi
+
+
+def mech(rows, tasks):
+    rs = [r for t in tasks for r in rows[t].values()]
+    n = len(rs)
+    empty = sum(r["events"].count("nudge") if isinstance(r["events"], list) else r["events"].get("nudge", 0) for r in rs)
+    return {
+        "runs": n,
+        "pass": sum(r["pass"] for r in rs) / n,
+        "score": st.mean(r["score"] for r in rs),
+        "wall_cap": sum(r.get("run_status") == "timeout" for r in rs) / n,
+        "steps": st.mean(r["steps"] for r in rs),
+        "wall_s": st.mean(r["wall_s"] for r in rs),
+        "nudges/run": empty / n,
+    }
+
+
+def main(pa, pb):
+    a, b = load(pa), load(pb)
+    by_task, pairs, up, down, lo, hi = compare(a, b)
+    tasks = list(by_task)
+    delta = mean_delta(by_task)
+    print(f"{len(tasks)} exercises, {pairs} paired runs")
+    ma, mb = mech(a, tasks), mech(b, tasks)
+    for k in ma:
+        fa, fb = ma[k], mb[k]
+        print(f"  {k:11} {fa:8.3f} -> {fb:8.3f}")
+    print(f"score delta {delta:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  (bootstrap over exercises)")
+    print(f"pass flips +{up} / -{down}  sign test p={sign_p(up, down):.2f}")
+    if lo > 0:
+        print("verdict: BETTER")
+    elif hi < 0:
+        print("verdict: WORSE")
+    else:
+        sd = st.pstdev([d for v in by_task.values() for d in v]) or 1e-9
+        need = int(math.ceil((1.96 * sd / max(abs(delta), 1e-3)) ** 2)) if delta else 0
+        print(f"verdict: NO EVIDENCE (CI spans 0)" + (f"; ~{need} paired runs to resolve {delta:+.3f}" if need else ""))
+
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2])

{tether_sdk-0.2.9 → tether_sdk-0.2.11}/bench/grade.py

@@ -180,6 +180,11 @@ def _codereview(_task, _ws, result):
 POLY_FAST = ("affine-cipher bottle-song list-ops paasio dot-dsl food-chain forth go-counting poker pov react "
              "rest-api").split()
 
+# ~15 min: 4 flaky exercises that can flip + 3 fast guards that must not regress (baseline in canary_baseline.json)
+POLY_CANARY = "phone-number tree-building grep pig-latin proverb robot-name list-ops".split()
+POLY_PAIRED = "grep go-counting pov dominoes beer-song hangman wordy forth paasio poker".split()  # A/B set, see README
+GUARDS = ("proverb", "robot-name", "list-ops")
+
 
 def poly_dir(ex=""):
     """Aider polyglot-benchmark Python exercises (cloned once, never vendored)."""

tether_sdk-0.2.11/bench/history.jsonl

@@ -0,0 +1,4 @@
+{"date": "2026-10-27", "version": "main@b25cbbe (post v0.2.9, #176)", "model": "ollama/qwen3.5:9b", "runner": "tether", "tasks": "polyglot-34", "trials": 2, "context": 8192, "note": "baseline; wall cap 600s; 14 exercises never pass", "runs": 68, "passed": 35, "pass_rate": 0.515, "mean_score": 0.718, "fail_max_steps": 17, "fail_wall_cap": 11, "mean_steps": 18.2, "mean_prompt_tokens": 82120, "mean_wall_s": 274}
+{"date": "2026-10-27", "version": "main@b25cbbe", "model": "ollama/qwen3.5:9b", "runner": "tether", "tasks": "polyglot-fast-12", "trials": 2, "context": 8192, "note": "same runs as baseline, fast subset", "runs": 24, "passed": 12, "pass_rate": 0.5, "mean_score": 0.711, "fail_max_steps": 8, "fail_wall_cap": 3, "mean_steps": 20.7, "mean_prompt_tokens": 94828, "mean_wall_s": 265}
+{"date": "2026-10-27", "version": "naive_agent", "model": "ollama/qwen3.5:9b", "runner": "naive", "tasks": "polyglot-fast-12", "trials": 2, "context": 8192, "note": "no-harness ceiling reference", "runs": 24, "passed": 3, "pass_rate": 0.125, "mean_score": 0.243, "fail_max_steps": 0, "fail_wall_cap": 0, "mean_steps": 4.7, "mean_prompt_tokens": 8054, "mean_wall_s": 55}
+{"date": "2026-10-28", "version": "v0.2.10", "model": "ollama/qwen3.5:9b", "runner": "tether", "tasks": "polyglot-34", "trials": 2, "context": 8192, "note": "v0.2.10 full run, --poly-verify; flat vs baseline: 14 wall-cap runs still lose to 4-6k-token empty turns", "runs": 68, "passed": 35, "pass_rate": 0.515, "mean_score": 0.763, "fail_max_steps": 18, "fail_wall_cap": 14, "mean_steps": 17.9, "mean_prompt_tokens": 77722, "mean_wall_s": 281}