tether-sdk 0.2.8__tar.gz → 0.2.10__tar.gz

tether_sdk-0.2.10/.claude/gru/active/north-star.md

@@ -0,0 +1,40 @@
+# North Star: Polyglot-Python 80% on qwen3.5:9b
+- Slug: north-star
+- Status: active
+- Started: 2026-10-27   Updated: 2026-10-27   Closed: —
+- Branches / PRs: branch fix/north-star-cheap-wins (worktree .claude/worktrees/north-star-c852b8), tracker #173
+- Budget: warn 300k / stop 400k paid tokens, max escalations 3, Tier 3 never without ask   Used: ~small (gru only)
+
+## Goal
+Polyglot-Python (34 ex, 2 trials, `--poly-verify`, default 8192 ctx) pass rate on ollama/qwen3.5:9b: 51.5% -> 80% by v0.3.0. Lanes: default (official) and `swe` role (#172, must not score below default).
+
+## Requests
+- 2026-10-27 PO: "Get to the North Star"; dev too slow, benches unreported. Asked to review the last cross-session and close it (done: v0-2-bugs archived).
+
+## Decisions
+- 2026-10-27 PO answers: lane 1 = deterministic fixes first (no GPU); merge proof = tests + offline replay, one batched full 34x2 bench after 2-3 PRs, reported immediately; budget default 300k/400k.
+- Ranked levers (Opus assessment): (1) runaway thinking hits num_ctx, +8-12pp, two attempts inconclusive (think=false 20 rows +3; length-retry rejected); (2) compact test output (relativize 195-char workspace path = 14% of obs chars, tracebacks = 51%), +3-6pp; (3) 4/68 runs died on 120s ProviderError timeout ~6pp (floor #171); (4) test-file access #170.
+- STOP list: nudge variants, whole-set 24-run A/Bs, wall cap as part of the north star.
+
+## Problems
+- 14 exercises never pass: book-store bowling connect forth hangman paasio poker pov react rest-api scale-generator sgf-parsing transpose zipper. Bench GPU: 8GB RTX 5050, one Ollama.
+
+## Plan (task cards)
+- A: ollama default timeout 120 -> 300 (tests/providers/test_ollama.py test written; docs/reference/providers.md line 95 says 120). Gru patches (1 line).
+- B: bash output: strip workspace prefix + ==== / ---- rule lines in sandbox.execute_bash_with_code (tests/test_sandbox.py test written). Gru patches (small).
+- C (next): collapse unittest tracebacks; build bench/replay.py offline measure first.
+- Then: output headroom / bounded thinking (#135/#147), #170, #171.
+
+## Ledger
+| Task | Tier | Model | Harness | Session | Attempt | Tokens | Time | Result |
+|---|---|---|---|---|---|---|---|---|
+| A,B | plan | gru | - | - | 1 | small | - | tests written failing, patch pending |
+
+## Progress log
+- 2026-10-27 branch fix/north-star-cheap-wins; failing tests added (sandbox prefix, ollama timeout). Closed v0-2-bugs log.
+
+## Next steps
+- [ ] Patch A and B, update docs (providers.md, release notes), unit+e2e, PR with `north star:` line
+- [ ] bench/replay.py offline proof for B
+- [ ] One batched full 34x2 after A+B+C, report at once
+- 2026-10-27 merged #180 (timeout 300s, compact bash, canary). PR #181 traceback collapse (canary +0.12, offline -12.6% obs chars, e2e 69). PR #182 outage retry (#171, e2e new scenario). PO: 80% is the impossible focus goal; wants v0.2.10 plan ranked by north star. Canary tool: bench/canary.py (12 min, 7 rows). Note: bench `pass` is graded by exercise tests, so #177 (done at max steps) does NOT move the bench number, only the user-facing floor.

{tether_sdk-0.2.8/.claude/gru/active → tether_sdk-0.2.10/.claude/gru/archive}/v0-2-bugs.md

@@ -1,7 +1,7 @@
 # v0.2 bugs
 - Slug: v0-2-bugs
-- Status: active
-- Started: 2026-10-20   Updated: 2026-10-24   Closed: —
+- Status: done
+- Started: 2026-10-20   Updated: 2026-10-24   Closed: 2026-10-27
 - Branches / PRs: branch `claude/keep-feature-context-resume-0ec918` (worktree); base `main` at 125b386 (release v0.2.2)
 
 ## Goal
@@ -106,3 +106,31 @@ Track and fix bugs found in Tether v0.2.x. Exact bug list not yet given by PO.
 - 2026-10-25 PO: docs plan. Audit -> epic #150 (steps: CLI ref #61, providers #60+#62, site structure #151, story guides #152, troubleshooting #153, benchmarks #154, README #63, llms.txt #64, polish #65/#66), all in milestone v0.2.10 (v0.2.11 closed, empty). AGENTS.md 5.6 now states the two audiences (docs site = users/integrators, no bug history; README = evaluate/build/contribute). #145 fix on branch fix/79-stats-offenders awaiting fast-set bench then PR.
 - 2026-10-25 Tether friction (docs cards on Nemotron Ultra): (1) ses_f6f847e5 CLI page and ses_7fb5863b providers page both ended at max steps WITHOUT calling done after the work was finished (keeps verifying); (2) rewriting a docs page it invented API facts (wrong ModelResponse fields, register_provider signature, nonexistent settings) -> docs rewrites by tether need a gru fact-check or assemble-from-source approach; (3) 5 steps in 10 min on a 156-line page (slow generation of long outputs on the free endpoint). Merged: #155 policy, #156 CLI ref (#61). PR for #60/#62 open. #145 PR waits for fast-set bench.
 - 2026-10-25 PO: milestones reshuffled: v0.2.8 = release now (#144 #145 + sandbox venv + CLI/providers docs), v0.2.9 = docs overhaul (#150-#154, #63-#66), v0.2.10 = #27 #79 #132-#137. Merged: #146 #148 #149 #155 #156 #157 #158. Release PR v0.2.8 (version bump, notes) prepared; GitHub Release only on PO yes.
+- 2026-10-25 RELEASED v0.2.8 'Room to Answer' (release PR #159 -> f4370b1, GitHub Release, delivery.yml success, PyPI tether-sdk 0.2.8 live, milestone closed). Docs v0.2.9 started: #151 site structure PR (nav = Start here/Guides/Concepts/Reference/Help, 'For:' line per page, CONTRIBUTING.md replaces docs/testing.md + releasing.md, tests/test_docs_structure.py). Lesson: ruff format --check also formats Python blocks in docs; renamed anchors break the strict mkdocs build via old release-notes links -> run both before pushing docs.
+- 2026-10-25 PO: 'merge all docs PRs and cut the next version'. Docs plan (#150) done: #155 #156 #157 #160 #161 #162 #165 #166 #167 merged, epic closed. Found #163 while writing guide 1 (qwen3.5:9b fixes the test, passes, never calls done -> FAILED Stalled); #164 merged on PO yes (Rule 3 reading: passing verify gates at stall time -> COMPLETED). Release PR v0.2.9 prepared. Next: v0.2.10 = #27 #79 #132-#137 (8 issues; pick by evidence, e.g. bench failure breakdown after v0.2.9). Tether friction this stretch: docs rewrites by Nemotron invent API facts and end without done; harness.py cards need 40+ steps.
+- 2026-10-26 v0.2.9 released (#168). Polyglot fast set + poly-verify on v0.2.9: 12/24, mean 0.765; 8192 vs 16384 context on 12 exercises: 16384 not a regression (easy 9/12 both, mean 0.872->0.937; hard 3->5/12) but ~+50% s/step (decode-bound), more 600s bench-cap timeouts; default stays 8192 (token saver), --context-tokens bench option on branch bench/137-context-tokens (no PR). Root-cause read of the breakdown found #169: bare `done` / typed `act("done", ...)` rejected, 8 of 36 runs wasted 1-13 steps, 2 false FAILED at score 1.0.
+- 2026-10-26 #169 fix (PR on branch fix/169-bare-done): tests first (e4f0feb), fix 143d79e. Ledger: #169 card A (bare done) | Tier 0 | z-ai/glm-5.3 | tether | ses_152e5df2 | attempt 2 | 7 steps, 6.6k tokens | ok (3 lines). Card B (act() text unwrap) | nemotron-3-ultra ses_d9caa19b (combined card A+B): 25 steps, 230k prompt tokens, zero edits; glm-5.3 ses (800s timeout, rate-limited, zero edits) -> escalated to plan level, 5-line patch written by gru.
+- Tether friction (#169): (1) nemotron-3-ultra on a 2-file card read 3 large files and ran pytest 12 times in 25 steps without ever calling `edit write` (read-only loop, no pre-change nudge fired before step 12, prompt 230k tokens); (2) nvidia free tier + glm-5.3 ~70s per step (rate limit) so an 800s timeout ended with no edit. Cards under ~15 lines and one file each worked (glm-5.3 7 steps).
+- 2026-10-26 #169 bench (done-fix vs v0.2.9, qwen3.5:9b, poly-verify, 2 trials, 8192): done-rejected steps 18 (4 runs) -> 0; pass 12/24 -> 10/24, mean 0.765 -> 0.694 (one infra outage cell rerun). Within noise: every failing run in the new set made 0 done calls (30 steps, never called done), so none touched the changed code. Affine-cipher t0 28 -> 7 steps, bottle-song t0 21 -> 14. Remaining dominant failure: max_steps without calling done (rewrite loops, e.g. food-chain t1 scores 1.0 yet FAILED). e2e 69 passed, unit 391 passed, 100% cov.
+- 2026-10-27 North star set (#173, AGENTS.md "North Star"): Polyglot-Python on qwen3.5:9b default 8192, full 34: 80% by v0.3.0. Baseline pinned (#178, bench/history.jsonl): main@b25cbbe 35/68 = 51.5%, mean 0.718; 17 fails at 30-step cap, 11 at the 600s bench wall cap, 14 exercises never pass (book-store bowling connect forth hangman paasio poker pov react rest-api scale-generator sgf-parsing transpose zipper). Naive no-harness loop on fast-12: 3/24 (12.5%) vs tether 12/24.
+- 2026-10-27 Experiments, all unmerged (data on issues): ctx 16384 fast-12 11/24 vs 12/24 (mean 0.807 vs 0.711, 9/13 fails hit 600s cap) -> rejected (#170); paging nudge flat; bounded cmd replay worse (10->8/24); think=false on 20 targeted rows 9/20 vs 6/20 (net +3, pre-reg +4) ; adaptive length-retry (branch fix/135-length-retry) 1/12 on failing rows -> rejected (#135); blanket think=false on fast-12 stopped at 11/24 runs (7 pass), inconclusive.
+- 2026-10-27 OPUS ASSESSMENT (architect, read-only, 13 tool calls over base34 t.db): 
+  * First-order failure nobody had measured = runaway thinking hitting num_ctx: 53 assistant turns with no action, 45 have prompt+completion in [8192,9216) (re-verified by gru: 45/53, 15 runs, 3672 s). Model thinks until the window is full, Ollama stops, tether records "(no output)" and nudges; ~85-110 s per turn. In the 11 wall-cap runs 41% of model time was empty turns: wall-cap failures are this, not slow progress. context.py:333 budgets prompt to 3/4 of ctx and leaves ~2048 for output; thinking does not fit.
+  * 4 of 68 runs died of ProviderError timeout 120s x3 (book-store t1, transpose t0, hangman t1, pig-latin t1): unretried infra loss ~6pp, breaks the floor (#171).
+  * Second mechanism: read-loop to the step cap (paasio poker react zipper sgf pov-t1 grep-t0 phone-t0 bowling-t1; fast 6-11 s/step). Test-run output (unittest tracebacks) = 51% of all observation chars; the 195-char bench workspace path = 14% of observation chars (~60 tokens per traceback line); react prints 6.5KB per test run, paasio 12KB.
+  * Why no progress: picked by proxy (re-read %) not mechanism; thinking not recorded (#32); 24-run bench sd ~2.4 passes = a +10pp change is ~1 sigma; whole-set A/B dilutes effects; the 600s wall cap measures GPU speed and thinking length (confound; ctx16 raised mean score but lost on wall); infra deaths baked into baseline.
+  * Fast assessment tiers: T0 offline replay `bench/replay.py` (~150 lines; rebuild each step's prompt from a recorded t.db through the candidate assembler; prompt tokens, output headroom, turns exceeding num_ctx, test-output chars; seconds, no GPU; proves prompt shape only). T1 determinism: same exercise twice with --param seed=1 (temp already 0.2), compare command sequences; first divergence = did the change fire; identical = zero effect (10 min). T2 counterfactual fork: copy a failing t.db, cut where the mechanism first fires, resume with the change (5-15 min/run). T3 targeted subset: only rows where the mechanism fired + 4 always-pass heavy-thinking canaries (zebra t1, vlq t0, two-bucket t0, go-counting t1), paired seeds, sequential stop (ship +4 net flips, kill at -2), <=1h. T4 full 34x2 overnight. Rule: a change gets GPU time only if T0/T1 shows its target metric moves.
+  * Parallelism: GPU is an 8GB RTX 5050; two Ollama instances will not fit; try OLLAMA_NUM_PARALLEL=2 (~1.5x?) for 10 min, only after dropping the wall cap (contention distorts wall).
+  * Ranked levers: (1) output headroom + bounded thinking, expected +8-12pp (15 affected runs, partial 0.6-0.75): test think=false T3, real fix = on done_reason=="length" with no tool call retry once with think=false, reserve output headroom in context.py, scale provider timeout to the generation budget, overlaps #135/#147; (2) compact test-run output: relativize workspace_root in bash observations, collapse unittest tracebacks to `test_x:105 AssertionError: 2 != 4`, drop ====/---- lines, +3-6pp, verify via T0 then T3; (3) test-file access #170, +2-4pp low confidence (def test_ index when file exceeds cap).
+  * 2-week workflow: D1 build replay.py + seed determinism check + queue think=false T3; D2 headroom/think-retry PR; D3-4 output-compaction PR proven with T0 then T3; bench-rules PR (official cap = 30 steps only with 1800s safety wall, p50/p90 wall as indicator, short workspace path /tmp/tb/<ex>, provider timeout no longer kills a run silently, re-pin baseline overnight once); end wk1 full T4; wk2 #170 then #171. Opus does diagnosis/tooling/acceptance tests/verdicts, minions mechanical PRs only; one GPU queue. STOP: nudge variants, whole-set 24-run A/Bs, wall cap as part of the north star.
+- 2026-10-27 Result of following it (gru): lever 1 tested twice, neither clean: think=false 20 rows +3 net, adaptive length-retry rejected (empty turns 3672s -> 332s but 20 turns still >2500 completion tokens: thinking is slow even when it fits; blanket think=false ran 405 turns in the time retry ran 243). T0/T1/T2 never built.
+- 2026-10-27 PO feedback (verbatim gist): dev is far too slow ("1 line of code and an entire day benching it"), benches finish unreported, results get thrown away because of the impossible goal; gru is losing itself on the work; wants the Opus findings logged. LESSON: stop bench-per-line-of-code loops; decide with a fast tier first, ship deterministic/obviously-correct improvements with unit+e2e evidence, report every bench end immediately (watcher fires on the results file, not on the launcher shell).
+## Next steps (as of 2026-10-27, awaiting PO decision)
+- [ ] PO to choose: (a) finish blanket think=false fast-12 (~13 runs, 45 min), (b) offline output compaction (no GPU), (c) plan only. Nothing is running; no open PRs.
+
+
+## Wrap-up (2026-10-27, closed by PO request)
+- Outcome: v0.2.3 -> v0.2.9 released (PyPI tether-sdk 0.2.9). Bugs and docs overhaul done; goal shifted to the North Star (#173): Polyglot-Python 80% on qwen3.5:9b by v0.3.0. Baseline 35/68 = 51.5% (main@b25cbbe).
+- Durable lessons: (1) never bench per line of code: decide with a fast tier first (offline replay of recorded t.db, determinism check), GPU only for targeted rows; (2) report every bench end immediately; (3) cards under ~15 lines, one file, worked on tether minions (glm-5.3 7 steps); 2-file cards and docs rewrites on Nemotron Ultra loop or invent API facts; (4) run `ruff format --check` and strict mkdocs before docs pushes; (5) nudge/wording variants and whole-set 24-run A/Bs do not move the number (sd ~2.4 passes).
+- Open debt: #27 #79 #132-#137 #170 #171; branches unmerged: fix/170-bounded-cmd-replay (rejected), fix/135-length-retry (rejected), bench/137-context-tokens; Harness-Bench #123 blocked (docker firewall + key).
+- Tier cost: Tier 0 (Nemotron Ultra, GLM-5.3 on tether) solved only tiny single-file cards; everything else was finished at plan level by gru.

{tether_sdk-0.2.8 → tether_sdk-0.2.10}/AGENTS.md

@@ -28,6 +28,15 @@ Tether stays **lightweight, reliable, extensible, model-agnostic, easy to use, r
 
 Reviewers and the Team Leader cite the failing line when they reject a change.
 
+### North Star (the one number; see tracker #173)
+**Polyglot-Python pass rate on `ollama/qwen3.5:9b` (our weakest model), default config, full 34 exercises: 80% by v0.3.0.** Nothing important is set aside for it, but everything is ranked by it:
+1. Every issue, PR and benchmark must say how it moves this number (or tokens per solved exercise at the same pass rate). A benchmark that cannot show a change in it is not run; an issue that cannot name the link waits behind one that can.
+1b. Two lanes, both tracked: the **general-purpose default** (no role) is the official number and must reach 80%; the shipped **`swe` role** (#172) is reported next to it, must never score below the default, and gets its own target once its baseline is pinned.
+2. Weakest-model rule: every stronger model must score at least what `qwen3.5:9b` scores (and at least 80% once reached). If we plateau, stronger models may not fall below the plateau.
+3. Other indicators (tokens per pass, steps, wall-clock, prompt size, coverage) may slow or pause but never degrade between releases.
+4. Floor, a promise rather than a goal: every task delivers a real, honest, usable result in a fair time; no unretried outage, no empty result, no unrelated hallucinated deliverable (#171).
+5. Every PR body and issue-close comment ends with `north star: <before> -> <after> (runs, noise, run dir)`; the tracker's History table and `bench/history.jsonl` are updated at every release.
+
 ---
 
 ## 2. The Golden Rules (Non-Negotiable Invariants)
@@ -43,10 +52,10 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
 
 ### Rule 3: Explicit Termination (`act("done", stdin="...")`), Interactive Inquiry (`done --ask`), & Pre-`done` Verification Gates
 - Sessions do **not** terminate simply because the model emitted a text response without a tool call (small 9B models frequently "think out loud" before acting).
-- A task only completes when the agent explicitly calls `act("done", stdin="<final deliverable>")` (or `act("done '<short answer>'")`), pauses for human/parent input via `act("done --ask", stdin="<question>")` (transitioning the session to `WAITING_INPUT` and releasing the worker process until resumed via `send`), or hits `max_steps`.
+- A task only completes when the agent explicitly calls `act("done", stdin="<final deliverable>")` (or `act("done '<short answer>'")`), pauses for human/parent input via `act("done --ask", stdin="<question>")` (transitioning the session to `WAITING_INPUT` and releasing the worker process until resumed via `send`), or hits `max_steps`. A bare `done` (no answer, no stdin) completes with the result `Done.` and still runs the `verify` gates; `done --ask` with no question is an error (#169).
 - If the model emits plain text without calling `act`, the harness appends a brief deterministic nudge reminding it to either execute a command or call `done`.
 - After 6 consecutive read-only/verification steps following a change, the harness appends a one-shot `[NUDGE]` to that step's observation reminding the model to call `done` if the task is complete (0 static prompt tokens, #74). Before any change, 12 read-only steps in a row (then every 8 more) append a one-line `[NUDGE]` to make the edit or call `done` (#127). Seven reads of one file in a row with no change are a `cycle_stall`, and 4 stall warnings without a change end the session `FAILED` as `Stalled` (#92); `tether stop <id>` terminates a detached worker (#89).
-- When `verify` gate commands are configured (via `Harness(verify=[...])`, CLI `--verify`, or Skill YAML `verify: [...]`), `Harness.execute_session` deterministically executes each command inside the session's `SandboxPolicy` before allowing `done` to transition to `COMPLETED` (consuming 0 static prompt tokens until a gate fails). A gate that errors instead of failing (sandbox violation, exit 124/126/127) ends the session `FAILED` at once as `gate_error` ("gate misconfigured"), without counting toward the consecutive-failure abort.
+- When `verify` gate commands are configured (via `Harness(verify=[...])`, CLI `--verify`, or Skill YAML `verify: [...]`), `Harness.execute_session` deterministically executes each command inside the session's `SandboxPolicy` before allowing `done` to transition to `COMPLETED` (consuming 0 static prompt tokens until a gate fails). A gate that errors instead of failing (sandbox violation, exit 124/126/127) ends the session `FAILED` at once as `gate_error` ("gate misconfigured"), without counting toward the consecutive-failure abort. When the stall guard is about to end a session as `Stalled` and gates are configured, it runs them first: a change was made and every gate passes, so the session ends `COMPLETED` ("Finished by the harness"), since the deterministic gate is the completion criterion the caller asked for (#163).
 
 ### Rule 4: Progressive `--help` (JIT Schema Disclosure)
 - The system prompt lists at most **1 line per enabled tool namespace** (`- <name>: <short summary>`, target **≤ 12 tokens per entry**).
@@ -205,9 +214,9 @@ uv run mypy tether/
 2. **Async-First Design:**
    - All model calls and I/O-bound tool executions must be `async`/`await` native, while seamlessly wrapping synchronous user `@tool` functions.
 3. **Graceful Degradation on 9B Hallucinations:**
-   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`), unwrapping `bash -c/-lc '<cmd>'` and `sh -c` wrappers, running an unknown first word through the enabled `bash` tool (never for disabled built-ins), registering extra quirk fixers with `register_normalizer`, mapping a native tool call named other than `act` to `<name> --k 'v'` so the JIT usage answers it, inferring the `edit` verb (`edit <path>` -> view/replace/write) and accepting `apply_patch` (`*** Begin Patch` or unified-diff patches, also when a diff arrives as `edit <path>` stdin) as an `edit` verb.
-4. **Golden traces:** behaviour the model sees (framing, folding, nudges, tool output) is locked by `tests/golden/*.snap.json`. A PR that changes it runs `UPDATE_GOLDEN=1 pytest tests/test_golden.py` and the snapshot diff is part of the review; unrelated traces must not change. See `docs/testing.md`.
-5. **E2E scenarios:** every feature or bug fix ships with a scenario in `e2e/` (real CLI, real ollama behind a fault-injecting proxy, local only, `TETHER_E2E=1 uv run pytest e2e`). Assert on what tether sends and records, never on the model's skill; script the reply or inject the fault so the test does not depend on model potency. **Before merging any PR, run `TETHER_E2E=1 uv run pytest e2e` locally and put the result in the PR description (CI cannot run it). Every failure must be mitigated before merge: fix the code, or fix a wrong scenario, or record in the PR why it is acceptable and what follows up. A PR is not merged with an unexplained red scenario.** See `docs/testing.md`.
+   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`), unwrapping `bash -c/-lc '<cmd>'` and `sh -c` wrappers, unwrapping an `act("<cmd>", stdin="...")` call typed as text in `cmd` (also after `bash`) into the command and its `stdin` (#169), running an unknown first word through the enabled `bash` tool (never for disabled built-ins), registering extra quirk fixers with `register_normalizer`, mapping a native tool call named other than `act` to `<name> --k 'v'` so the JIT usage answers it, inferring the `edit` verb (`edit <path>` -> view/replace/write) and accepting `apply_patch` (`*** Begin Patch` or unified-diff patches, also when a diff arrives as `edit <path>` stdin) as an `edit` verb.
+4. **Golden traces:** behaviour the model sees (framing, folding, nudges, tool output) is locked by `tests/golden/*.snap.json`. A PR that changes it runs `UPDATE_GOLDEN=1 pytest tests/test_golden.py` and the snapshot diff is part of the review; unrelated traces must not change. See `CONTRIBUTING.md`.
+5. **E2E scenarios:** every feature or bug fix ships with a scenario in `e2e/` (real CLI, real ollama behind a fault-injecting proxy, local only, `TETHER_E2E=1 uv run pytest e2e`). Assert on what tether sends and records, never on the model's skill; script the reply or inject the fault so the test does not depend on model potency. **Before merging any PR, run `TETHER_E2E=1 uv run pytest e2e` locally and put the result in the PR description (CI cannot run it). Every failure must be mitigated before merge: fix the code, or fix a wrong scenario, or record in the PR why it is acceptable and what follows up. A PR is not merged with an unexplained red scenario.** See `CONTRIBUTING.md`.
 6. **Documentation Integrity:**
    - Keep `README.md` and `AGENTS.md` synchronized whenever architectural interfaces or built-in tool signatures change.
    - **Always update documentation in the same PR** when introducing a new feature, changing an existing behavior, or altering a CLI/SDK interface. This covers `README.md`, `AGENTS.md`, the MkDocs pages under `docs/` (including `docs/reference/*` for new modules), and a `docs/release-notes.md` entry. A change without matching docs is incomplete.
@@ -225,5 +234,5 @@ uv run mypy tether/
 - Hard stops (ask first): force push / history rewrite; merging a PR and cutting a release need the PO's explicit yes each time
 - Design docs: `AGENTS.md` sections 1-3, `docs/architecture.md`, `docs/reference/*`
 - Orchestration: **`/gru` runs this repo** (Team Leader + router). Strong model plans, writes the failing acceptance tests first, reviews and decides; tether does the typing as the minion at Tier 0 on the strongest free model (`nvidia/nvidia/nemotron-3-ultra-550b-a55b`, fallback `nvidia/z-ai/glm-5.3`), escalating one tier only on evidence. Tiers, context sizes and sensitive paths: `.claude/gru.json`. tether's own failures while doing a card are logged as friction and filed as issues (with the PO's OK).
-- Improvement loop: **every PR leaves tether a little better**, measured, not asserted: a behaviour change ships with a bench or e2e delta (Polyglot-Python fast set on `ollama/qwen3.5:9b`, `bench/`) in the PR body, reported honestly even when it is flat. After a merge, gru re-reads the next planned milestone and the other open bug issues and picks the one that clearly improves tether most (evidence from bench failure breakdowns and `tether stats` friction beats guesses); features and docs wait behind bugs that cost steps or pass rate.
+- Improvement loop: **every PR leaves tether a little better**, measured, not asserted: a behaviour change ships with a bench or e2e delta (Polyglot-Python on `ollama/qwen3.5:9b`, `bench/`) in the PR body as a `north star:` line (see North Star), reported honestly even when it is flat. After a merge, gru re-reads the next planned milestone and the other open bug issues and picks the one that clearly improves tether most (evidence from bench failure breakdowns and `tether stats` friction beats guesses); features and docs wait behind bugs that cost steps or pass rate.
 - gru store: .claude/gru/   (committed; legacy `feature-context` logs were moved here)

tether_sdk-0.2.8/docs/testing.md → tether_sdk-0.2.10/CONTRIBUTING.md

@@ -1,4 +1,24 @@
-# Testing: golden traces
+# Contributing to tether
+
+Thanks for helping. Tether stays **lightweight, reliable, extensible, model-agnostic, easy to use, resourceful and a token saver**: that list (the *Philosophy Gate*) and the architectural rules live in [AGENTS.md](AGENTS.md), which is the authoritative guide for contributors and coding agents. Read its sections 1-3 before changing behaviour.
+
+## Setup and checks
+
+```bash
+uv sync                                                        # environment
+uv run ruff check . && uv run ruff format --check .            # lint and format (also formats Python in docs)
+uv run mypy tether/                                            # types
+uv run pytest --cov=tether --cov-report=term-missing --cov-fail-under=100   # unit tests, 100% coverage (CI)
+TETHER_E2E=1 uv run pytest e2e                                 # e2e, local only (needs the tether-ollama container)
+```
+
+## Pull requests
+
+- One PR per task, Conventional Commits (`feat(scope): ... (#N)`, `fix(...)`, `docs(...)`, `release: vX.Y.Z ...`), squash-merged to `main`.
+- Update the docs in the same PR (README, the MkDocs pages, `docs/release-notes.md`); see *Two audiences, two jobs* in AGENTS.md section 5.6 for what belongs where.
+- A behaviour change carries a measured result in the PR body (bench or e2e delta), reported honestly even when it is flat. Put the `TETHER_E2E=1 uv run pytest e2e` result in the PR; CI cannot run it.
+
+## Testing: golden traces, e2e scenarios and the model bench
 
 Tether is a deterministic envelope around a random model, so a change to framing, folding,
 nudges, tools or retries can silently change what the model is shown. Golden traces make that
@@ -20,7 +40,7 @@ Shipped traces: a real researcher run (web stubbed, plain-text `done` nudges), a
 coding run covering the edit failure modes seen with small models, and a `done --ask` /
 `send` resume, and a range ping-pong (alternating views of one file; the latest read of each range stays unfolded, older ones fold). Add one for every behaviour that must not regress.
 
-## End-to-end scenarios (local, real processes, real ollama)
+### End-to-end scenarios (local, real processes, real ollama)
 
 `e2e/` drives the real `tether` CLI in a throwaway workspace. A small fault-injecting proxy
 (`e2e/proxy.py`) sits between tether and ollama, so a scenario can force the situation instead of
@@ -83,7 +103,7 @@ judge (answer quality).
 **Rule:** a new feature or bug fix ships with a scenario here. If a model must misbehave to
 trigger it, script the reply; if infrastructure must fail, add a fault to the proxy.
 
-## Model conformance bench (developers only)
+### Model conformance bench (developers only)
 
 `bench/` holds nine tasks (the `focus` task checks that an exact single-file goal is not lost in a distractor-heavy repo) with hidden-test graders and three runners (Tether, a plain tool loop,
 Claude Code) so the same model can be compared across harnesses and releases. It needs real
@@ -91,3 +111,14 @@ models, so it is not in CI and not in the package: use it when onboarding a mode
 release to see friction that scripted tests cannot (the shell-wrapper, bare-command and
 tool-name failures behind #109 were found this way). Anything it finds that can be scripted
 becomes a unit, golden or e2e test. See `bench/README.md`.
+
+## Releasing
+
+PyPI is published only from a GitHub Release, never from a push or tag.
+
+0. Re-run the commands in `docs/guides/` against the release candidate and fix any output that changed.
+1. Bump `__version__` in `tether/__init__.py` (and the version assertion in `tests/test_tether.py`) and update `docs/release-notes.md`.
+2. Merge to `main`.
+3. Create a GitHub Release with tag `v<version>` (for example `v0.1.0`). The tag must match `__version__` (a leading `v` is ignored) or the workflow fails before building.
+
+The `delivery.yml` workflow then verifies the tag, builds the sdist and wheel, publishes to PyPI with trusted publishing (no secrets) and attaches both files to the release. Pre-releases are skipped: nothing is published or attached.

tether_sdk-0.2.10/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.5
+Name: tether-sdk
+Version: 0.2.10
+Summary: General-purpose, headless-first LLM-Native micro-harness for small language models
+Project-URL: Homepage, https://github.com/igormcsouza/tether
+Project-URL: Repository, https://github.com/igormcsouza/tether
+Project-URL: Documentation, https://igormcsouza.github.io/tether/
+Author-email: Igor Souza <igormcsouza@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,cli,harness,llm,slm,sqlite
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pyyaml>=6.0.1
+Provides-Extra: mcp
+Requires-Dist: mcp<3,>=2.0; extra == 'mcp'
+Description-Content-Type: text/markdown
+
+# 🪢 Tether
+
+**A small, headless-first harness that lets small local models (and frontier ones) do real work with a tiny prompt.**
+
+[![PyPI](https://img.shields.io/pypi/v/tether-sdk?color=0d9488&label=PyPI%20%7C%20tether-sdk)](https://pypi.org/project/tether-sdk/)
+[![Docs](https://img.shields.io/badge/Docs-igormcsouza.github.io%2Ftether-0d9488)](https://igormcsouza.github.io/tether/)
+[![Python](https://img.shields.io/badge/Python-3.12%2B-3b82f6)](#install-and-first-run)
+[![Models](https://img.shields.io/badge/Models-Ollama%20%7C%20OpenAI--compatible%20%7C%20Claude-10b981)](https://igormcsouza.github.io/tether/reference/providers/)
+
+```bash
+uv tool install tether-sdk
+tether run --model ollama/qwen3.5:9b --tools bash,edit --verify "pytest -q" "fix the failing tests"
+```
+
+## Why Tether
+
+Most agent frameworks treat the context window as a dumping ground. Before your task is read, they inject **3,000 to 8,000 tokens** of JSON tool schemas and system prompt, then append every tool output forever. On a frontier model that is money. On a 9B model running on your laptop it is fatal: attention is diluted, small models fumble nested JSON, and changing the tool list between turns throws away the inference server's KV cache.
+
+Tether is built the other way round:
+
+- **One tool, not thirty.** The model sees a single `act(cmd, stdin)` schema. Everything else (`bash`, `edit`, `web`, `mem`, your own functions) is a command behind it, off by default. Fixed framing is under 150 tokens.
+- **Old output folds away.** Tool results stay in full for two turns, then become a one-line tombstone. Everything lives in SQLite and comes back by search.
+- **The envelope is deterministic.** Sandboxing, state, step limits, verification gates and termination are plain code. The model only reasons; it is never trusted with safety.
+
+## How well does it work
+
+Tether answers to one number: how many real coding exercises a small local model finishes. On the Polyglot-Python set (Aider's 34 Python exercises, graded by each exercise's own tests), `qwen3.5:9b` through Ollama, 12-exercise fast set, 2 trials each:
+
+| | v0.2.7 | v0.2.8 |
+|---|---|---|
+| Passes | 10 of 24 | 11 of 24 |
+| Mean score | 0.741 | 0.756 |
+| "Tool call cut off" hints | 29 | 2 |
+| Steps that did nothing | 19% | 2.2% |
+
+One run is noisy (about 0.1), so read the pass count and the failure breakdown, not one decimal. Method, how to reproduce it and fair-comparison rules: [Benchmarks](https://igormcsouza.github.io/tether/benchmarks/). Every change that touches model-visible behaviour has to move this number or leave it alone.
+
+## How it works
+
+```mermaid
+flowchart LR
+    You["CLI · SDK · MCP"] --> Loop
+    subgraph Envelope["Deterministic envelope"]
+        Loop["bounded loop<br/>steps · stall guard · gates"] --> Mux["act(cmd, stdin)<br/>command bus"]
+        Mux --> Sand["sandbox<br/>bwrap · workspace jail"]
+        Sand --> Tools["bash · edit · web · mem · agent · your @tool"]
+        Tools --> Fold["fold old output"]
+        Fold --> DB[("SQLite WAL + FTS5<br/>.tether/tether.db")]
+        DB --> Loop
+    end
+    Loop <-->|"messages"| Model["LLM<br/>ollama · openai · anthropic"]
+```
+
+A session is a row in SQLite, so it survives crashes, runs detached (`--headless`), takes new instructions while running (`tether send`), and can be read by another agent as compact JSON. A task ends only when the model calls `done`, after your `--verify` commands pass.
+
+## Why this and not that
+
+| Decision | Instead of | Why |
+|---|---|---|
+| One `act` tool, commands behind it | A JSON schema per tool | CodeAct and SWE-agent results: one executable channel beats many schemas, most for 7-9B models; and the prompt prefix never changes, so the KV cache survives. |
+| Progressive `--help` | Full usage up front | A tool costs about a dozen tokens until the model asks or gets it wrong. |
+| SQLite (WAL, FTS5) as memory | An ever-growing transcript | Context stays a bounded working set; the rest is searchable. State is ACID, so sessions resume and agents share results. |
+| Kernel sandbox (`bwrap`), env scrubbing, SSRF guard | "Please be careful" in the prompt | Safety that depends on the model obeying is not safety. |
+| Explicit `done` plus verification gates | Letting the model declare success | Small models stop early or never stop. The gate is a command you wrote. |
+| Thin `httpx` adapters, 2 runtime dependencies | Vendor SDKs, LangChain-style frameworks | Easy to read, easy to add a backend (`providers.yaml`, no code), nothing to upgrade around. |
+
+The research behind it: Maia et al. (2026) on probabilistic core and deterministic envelope, Wang et al. (2024, CodeAct), Yang et al. (2024, SWE-agent) and Packer et al. (2023, MemGPT); see [References](#references) and [Architecture](https://igormcsouza.github.io/tether/architecture/).
+
+## Install and first run
+
+```bash
+uv tool install tether-sdk            # or: pip install tether-sdk
+ollama pull qwen3.5:9b                # any model works; see the providers page
+tether run --model ollama/qwen3.5:9b --tools bash,edit "create hello.txt containing hi"
+```
+
+Everything else (headless sessions, skills, pipelines, MCP, the CLI reference, writing your own tool or provider) is in the docs: **[igormcsouza.github.io/tether](https://igormcsouza.github.io/tether/)**. Start with the guides: [fix a failing test](https://igormcsouza.github.io/tether/guides/fix-a-failing-test/), [use it from Claude Code or Codex](https://igormcsouza.github.io/tether/guides/use-tether-from-your-agent/).
+
+## Build on it
+
+```python
+import asyncio
+
+from tether import Harness, tool
+
+
+@tool(name="price", description="Look up the unit price of a fruit")
+def price(fruit: str) -> str:
+    return f"{fruit}: 0.50 EUR"
+
+
+async def main() -> None:
+    harness = Harness(model="ollama/qwen3.5:9b", tools=[price])
+    session = await harness.run("What does an apple cost?")
+    print(session.result)
+
+
+asyncio.run(main())
+```
+
+`Harness`, `Pipeline`, `Skill`, `SandboxPolicy`, `@tool` and `register_provider` are the public surface. The agent-facing JSON and the optional `tether mcp` server (`tether-sdk[mcp]`) are specified in the [reference](https://igormcsouza.github.io/tether/reference/agent-interface/).
+
+## Contributing
+
+Tether stays lightweight, reliable, extensible, model-agnostic, easy to use, resourceful and a token saver; a change that fails one of those lines is reworked. Read [AGENTS.md](AGENTS.md) (the rules and the module map) and [CONTRIBUTING.md](CONTRIBUTING.md) (setup, checks, golden traces, e2e, the bench, releasing). Open an issue before a large change; small fixes with a test and a docs line are welcome as they are.
+
+```text
+tether/        harness · multiplexer · context · store · sandbox · providers/ · tools/
+tests/         unit, golden traces (what the model sees), 100% coverage
+e2e/           real CLI and real Ollama behind a fault-injecting proxy (local)
+bench/         Polyglot-Python and model checks (developers)
+docs/          the user site (MkDocs)
+```
+
+---
+
+## References
+
+1. **Maia, P. H. M., et al. (2026).** *LLM-Native Applications as a Paradigm for AI-Driven Software Architecture*.
+2. **Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., & Ji, H. (2024).** *Executable Code Actions Elicit Better LLM Agents (CodeAct)*. In **Proceedings of the 41st International Conference on Machine Learning (ICML 2024)**. [arXiv:2402.01030](https://arxiv.org/abs/2402.01030)
+3. **Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024).** *SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering*. In **Advances in Neural Information Processing Systems (NeurIPS 2024)**. [arXiv:2405.15793](https://arxiv.org/abs/2405.15793)
+4. **Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023).** *MemGPT: Towards LLMs as Operating Systems*. [arXiv:2310.08560](https://arxiv.org/abs/2310.08560)

tether_sdk-0.2.10/README.md

@@ -0,0 +1,122 @@
+# 🪢 Tether
+
+**A small, headless-first harness that lets small local models (and frontier ones) do real work with a tiny prompt.**
+
+[![PyPI](https://img.shields.io/pypi/v/tether-sdk?color=0d9488&label=PyPI%20%7C%20tether-sdk)](https://pypi.org/project/tether-sdk/)
+[![Docs](https://img.shields.io/badge/Docs-igormcsouza.github.io%2Ftether-0d9488)](https://igormcsouza.github.io/tether/)
+[![Python](https://img.shields.io/badge/Python-3.12%2B-3b82f6)](#install-and-first-run)
+[![Models](https://img.shields.io/badge/Models-Ollama%20%7C%20OpenAI--compatible%20%7C%20Claude-10b981)](https://igormcsouza.github.io/tether/reference/providers/)
+
+```bash
+uv tool install tether-sdk
+tether run --model ollama/qwen3.5:9b --tools bash,edit --verify "pytest -q" "fix the failing tests"
+```
+
+## Why Tether
+
+Most agent frameworks treat the context window as a dumping ground. Before your task is read, they inject **3,000 to 8,000 tokens** of JSON tool schemas and system prompt, then append every tool output forever. On a frontier model that is money. On a 9B model running on your laptop it is fatal: attention is diluted, small models fumble nested JSON, and changing the tool list between turns throws away the inference server's KV cache.
+
+Tether is built the other way round:
+
+- **One tool, not thirty.** The model sees a single `act(cmd, stdin)` schema. Everything else (`bash`, `edit`, `web`, `mem`, your own functions) is a command behind it, off by default. Fixed framing is under 150 tokens.
+- **Old output folds away.** Tool results stay in full for two turns, then become a one-line tombstone. Everything lives in SQLite and comes back by search.
+- **The envelope is deterministic.** Sandboxing, state, step limits, verification gates and termination are plain code. The model only reasons; it is never trusted with safety.
+
+## How well does it work
+
+Tether answers to one number: how many real coding exercises a small local model finishes. On the Polyglot-Python set (Aider's 34 Python exercises, graded by each exercise's own tests), `qwen3.5:9b` through Ollama, 12-exercise fast set, 2 trials each:
+
+| | v0.2.7 | v0.2.8 |
+|---|---|---|
+| Passes | 10 of 24 | 11 of 24 |
+| Mean score | 0.741 | 0.756 |
+| "Tool call cut off" hints | 29 | 2 |
+| Steps that did nothing | 19% | 2.2% |
+
+One run is noisy (about 0.1), so read the pass count and the failure breakdown, not one decimal. Method, how to reproduce it and fair-comparison rules: [Benchmarks](https://igormcsouza.github.io/tether/benchmarks/). Every change that touches model-visible behaviour has to move this number or leave it alone.
+
+## How it works
+
+```mermaid
+flowchart LR
+    You["CLI · SDK · MCP"] --> Loop
+    subgraph Envelope["Deterministic envelope"]
+        Loop["bounded loop<br/>steps · stall guard · gates"] --> Mux["act(cmd, stdin)<br/>command bus"]
+        Mux --> Sand["sandbox<br/>bwrap · workspace jail"]
+        Sand --> Tools["bash · edit · web · mem · agent · your @tool"]
+        Tools --> Fold["fold old output"]
+        Fold --> DB[("SQLite WAL + FTS5<br/>.tether/tether.db")]
+        DB --> Loop
+    end
+    Loop <-->|"messages"| Model["LLM<br/>ollama · openai · anthropic"]
+```
+
+A session is a row in SQLite, so it survives crashes, runs detached (`--headless`), takes new instructions while running (`tether send`), and can be read by another agent as compact JSON. A task ends only when the model calls `done`, after your `--verify` commands pass.
+
+## Why this and not that
+
+| Decision | Instead of | Why |
+|---|---|---|
+| One `act` tool, commands behind it | A JSON schema per tool | CodeAct and SWE-agent results: one executable channel beats many schemas, most for 7-9B models; and the prompt prefix never changes, so the KV cache survives. |
+| Progressive `--help` | Full usage up front | A tool costs about a dozen tokens until the model asks or gets it wrong. |
+| SQLite (WAL, FTS5) as memory | An ever-growing transcript | Context stays a bounded working set; the rest is searchable. State is ACID, so sessions resume and agents share results. |
+| Kernel sandbox (`bwrap`), env scrubbing, SSRF guard | "Please be careful" in the prompt | Safety that depends on the model obeying is not safety. |
+| Explicit `done` plus verification gates | Letting the model declare success | Small models stop early or never stop. The gate is a command you wrote. |
+| Thin `httpx` adapters, 2 runtime dependencies | Vendor SDKs, LangChain-style frameworks | Easy to read, easy to add a backend (`providers.yaml`, no code), nothing to upgrade around. |
+
+The research behind it: Maia et al. (2026) on probabilistic core and deterministic envelope, Wang et al. (2024, CodeAct), Yang et al. (2024, SWE-agent) and Packer et al. (2023, MemGPT); see [References](#references) and [Architecture](https://igormcsouza.github.io/tether/architecture/).
+
+## Install and first run
+
+```bash
+uv tool install tether-sdk            # or: pip install tether-sdk
+ollama pull qwen3.5:9b                # any model works; see the providers page
+tether run --model ollama/qwen3.5:9b --tools bash,edit "create hello.txt containing hi"
+```
+
+Everything else (headless sessions, skills, pipelines, MCP, the CLI reference, writing your own tool or provider) is in the docs: **[igormcsouza.github.io/tether](https://igormcsouza.github.io/tether/)**. Start with the guides: [fix a failing test](https://igormcsouza.github.io/tether/guides/fix-a-failing-test/), [use it from Claude Code or Codex](https://igormcsouza.github.io/tether/guides/use-tether-from-your-agent/).
+
+## Build on it
+
+```python
+import asyncio
+
+from tether import Harness, tool
+
+
+@tool(name="price", description="Look up the unit price of a fruit")
+def price(fruit: str) -> str:
+    return f"{fruit}: 0.50 EUR"
+
+
+async def main() -> None:
+    harness = Harness(model="ollama/qwen3.5:9b", tools=[price])
+    session = await harness.run("What does an apple cost?")
+    print(session.result)
+
+
+asyncio.run(main())
+```
+
+`Harness`, `Pipeline`, `Skill`, `SandboxPolicy`, `@tool` and `register_provider` are the public surface. The agent-facing JSON and the optional `tether mcp` server (`tether-sdk[mcp]`) are specified in the [reference](https://igormcsouza.github.io/tether/reference/agent-interface/).
+
+## Contributing
+
+Tether stays lightweight, reliable, extensible, model-agnostic, easy to use, resourceful and a token saver; a change that fails one of those lines is reworked. Read [AGENTS.md](AGENTS.md) (the rules and the module map) and [CONTRIBUTING.md](CONTRIBUTING.md) (setup, checks, golden traces, e2e, the bench, releasing). Open an issue before a large change; small fixes with a test and a docs line are welcome as they are.
+
+```text
+tether/        harness · multiplexer · context · store · sandbox · providers/ · tools/
+tests/         unit, golden traces (what the model sees), 100% coverage
+e2e/           real CLI and real Ollama behind a fault-injecting proxy (local)
+bench/         Polyglot-Python and model checks (developers)
+docs/          the user site (MkDocs)
+```
+
+---
+
+## References
+
+1. **Maia, P. H. M., et al. (2026).** *LLM-Native Applications as a Paradigm for AI-Driven Software Architecture*.
+2. **Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., & Ji, H. (2024).** *Executable Code Actions Elicit Better LLM Agents (CodeAct)*. In **Proceedings of the 41st International Conference on Machine Learning (ICML 2024)**. [arXiv:2402.01030](https://arxiv.org/abs/2402.01030)
+3. **Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024).** *SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering*. In **Advances in Neural Information Processing Systems (NeurIPS 2024)**. [arXiv:2405.15793](https://arxiv.org/abs/2405.15793)
+4. **Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023).** *MemGPT: Towards LLMs as Operating Systems*. [arXiv:2310.08560](https://arxiv.org/abs/2310.08560)

{tether_sdk-0.2.8 → tether_sdk-0.2.10}/bench/README.md

@@ -2,7 +2,7 @@
 
 Not part of the `tether` package, not run in CI. Use it to onboard a new model or to check a
 release for regressions that deterministic tests cannot see (unit, golden and e2e tests stay the
-gate; see `docs/testing.md`).
+gate; see `CONTRIBUTING.md`).
 
 Nine tasks in `tasks/<name>/` (workspace in `ws/`, goal in `goal.txt`, hidden tests in `hidden/`):
 `focus` (one exact edit in a distractor-heavy repo, scope-checked), `plan` (an implementation plan, checked for named files, failure handling and tests), `codereview` (eight seeded bugs to find, no edits), `code-simple`, `code-medium`, `code-hard` (graded by hidden unit tests; editing the visible tests
@@ -17,6 +17,10 @@ restored before grading, so editing it cannot pass), `--poly-verify` adds a `--v
 `python3 bench/report.py <out>` prints pass rate and mean score with a failure breakdown that keeps provider outages
 (`outage`, `timeout`) apart from harness outcomes (`max_steps`, `stalled`, `wrong_answer`, `worker_error`).
 
+**Did my change help? (15 min)** `python3 bench/canary.py /tmp/canary-<change> [--tether-bin <venv>/bin/tether]` runs 7 exercises
+once (4 flaky ones that can flip, 3 fast guards) and prints each score next to the pinned baseline with a verdict
+(`REJECT` / `PROMISING` / `FLAT`). Only `PROMISING` earns the full 34x2 run. `--tasks canary` is the same set for `run.py`.
+
 Three runners, so a gap can be attributed:
 
 | runner | what it is |
@@ -43,3 +47,11 @@ Rate limits (NVIDIA: 40 requests/min) invalidate results silently; set `rpm:` in
 and keep parallelism low (`--par nvidia=2`).
 
 Caveats: one run per cell is noisy (about 0.1); the scores are regex and hidden-test based.
+
+## History (the north-star ledger)
+
+`bench/history.jsonl` is the single source for the tracker's History table (#173). One JSON object per line: `date`, `version`, `model`, `runner`, `tasks`, `trials`, `context`, `runs`, `passed`, `pass_rate`, `mean_score`, failure counts, mean steps/prompt tokens/wall, `note`.
+
+A PR that changes behaviour the model sees appends one line for its own run (same model, default 8192 context, `--poly-verify`) and ends its body with `north star: <before> -> <after> (runs, noise, run dir)`. Noise is about 0.1 per run, so a claim needs enough runs to beat it; a flat result is appended and reported as flat.
+
+The official number is the full 34 exercises × 2 trials (68 runs) with the 600 s per-run wall cap. Use `--tasks polyglot-fast` for quick checks and label it `polyglot-fast-12`.

tether_sdk-0.2.10/bench/canary.py

@@ -0,0 +1,39 @@
+"""15-minute check: did this change help? `python3 bench/canary.py <out-dir> [--tether-bin ...]`.
+
+Runs 7 Polyglot exercises once (4 flaky ones that can flip, 3 fast guards), then compares each
+score with the pinned baseline (`canary_baseline.json`, mean of 2 runs on main@b25cbbe).
+Verdict: REJECT if a guard drops >0.2, PROMISING if the flaky mean gains >=0.1, else FLAT.
+Only PROMISING changes earn a full 34x2 run. One run per exercise is noisy: a FLAT verdict
+means "no evidence", not "proven useless".
+"""
+
+import json
+import os
+import subprocess
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+import grade  # noqa: E402
+
+
+def main(out, extra):
+    cmd = [sys.executable, f"{HERE}/run.py", "--models", "ollama/qwen3.5:9b", "--tasks", "canary",
+           "--poly-verify", "--par", "ollama=1", "--out", out, *extra]
+    subprocess.run(cmd, check=True)
+    base = json.load(open(f"{HERE}/canary_baseline.json"))
+    now = {}
+    for line in open(f"{out}/results.jsonl"):
+        r = json.loads(line)
+        now[r["task"][5:]] = r["score"]
+    flaky = [e for e in base if e not in grade.GUARDS]
+    for e in base:
+        print(f"{e:20} {base[e]:.2f} -> {now.get(e, float('nan')):.2f}{'  (guard)' if e in grade.GUARDS else ''}")
+    gain = sum(now.get(e, 0) - base[e] for e in flaky) / len(flaky)
+    drops = [e for e in grade.GUARDS if now.get(e, 0) < base[e] - 0.2]
+    verdict = "REJECT" if drops else "PROMISING" if gain >= 0.1 else "FLAT"
+    print(f"flaky mean delta {gain:+.2f}; guard drops {drops or 'none'} -> {verdict}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1], sys.argv[2:])

tether_sdk-0.2.10/bench/canary_baseline.json

@@ -0,0 +1 @@
+{"phone-number": 0.69, "tree-building": 0.92, "grep": 0.66, "pig-latin": 0.91, "proverb": 1.0, "robot-name": 1.0, "list-ops": 1.0}

{tether_sdk-0.2.8 → tether_sdk-0.2.10}/bench/grade.py

@@ -180,6 +180,10 @@ def _codereview(_task, _ws, result):
 POLY_FAST = ("affine-cipher bottle-song list-ops paasio dot-dsl food-chain forth go-counting poker pov react "
              "rest-api").split()
 
+# ~15 min: 4 flaky exercises that can flip + 3 fast guards that must not regress (baseline in canary_baseline.json)
+POLY_CANARY = "phone-number tree-building grep pig-latin proverb robot-name list-ops".split()
+GUARDS = ("proverb", "robot-name", "list-ops")
+
 
 def poly_dir(ex=""):
     """Aider polyglot-benchmark Python exercises (cloned once, never vendored)."""

tether_sdk-0.2.10/bench/history.jsonl

@@ -0,0 +1,3 @@
+{"date": "2026-10-27", "version": "main@b25cbbe (post v0.2.9, #176)", "model": "ollama/qwen3.5:9b", "runner": "tether", "tasks": "polyglot-34", "trials": 2, "context": 8192, "note": "baseline; wall cap 600s; 14 exercises never pass", "runs": 68, "passed": 35, "pass_rate": 0.515, "mean_score": 0.718, "fail_max_steps": 17, "fail_wall_cap": 11, "mean_steps": 18.2, "mean_prompt_tokens": 82120, "mean_wall_s": 274}
+{"date": "2026-10-27", "version": "main@b25cbbe", "model": "ollama/qwen3.5:9b", "runner": "tether", "tasks": "polyglot-fast-12", "trials": 2, "context": 8192, "note": "same runs as baseline, fast subset", "runs": 24, "passed": 12, "pass_rate": 0.5, "mean_score": 0.711, "fail_max_steps": 8, "fail_wall_cap": 3, "mean_steps": 20.7, "mean_prompt_tokens": 94828, "mean_wall_s": 265}
+{"date": "2026-10-27", "version": "naive_agent", "model": "ollama/qwen3.5:9b", "runner": "naive", "tasks": "polyglot-fast-12", "trials": 2, "context": 8192, "note": "no-harness ceiling reference", "runs": 24, "passed": 3, "pass_rate": 0.125, "mean_score": 0.243, "fail_max_steps": 0, "fail_wall_cap": 0, "mean_steps": 4.7, "mean_prompt_tokens": 8054, "mean_wall_s": 55}