tether-sdk 0.2.4__tar.gz → 0.2.5__tar.gz

{tether_sdk-0.2.4 → tether_sdk-0.2.5}/AGENTS.md

@@ -45,7 +45,7 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
 - Sessions do **not** terminate simply because the model emitted a text response without a tool call (small 9B models frequently "think out loud" before acting).
 - A task only completes when the agent explicitly calls `act("done", stdin="<final deliverable>")` (or `act("done '<short answer>'")`), pauses for human/parent input via `act("done --ask", stdin="<question>")` (transitioning the session to `WAITING_INPUT` and releasing the worker process until resumed via `send`), or hits `max_steps`.
 - If the model emits plain text without calling `act`, the harness appends a brief deterministic nudge reminding it to either execute a command or call `done`.
-- After 6 consecutive read-only/verification steps following a change, the harness appends a one-shot `[NUDGE]` to that step's observation reminding the model to call `done` if the task is complete (0 static prompt tokens, #74).
+- After 6 consecutive read-only/verification steps following a change, the harness appends a one-shot `[NUDGE]` to that step's observation reminding the model to call `done` if the task is complete (0 static prompt tokens, #74). Seven reads of one file in a row with no change are a `cycle_stall`, and 4 stall warnings without a change end the session `FAILED` as `Stalled` (#92); `tether stop <id>` terminates a detached worker (#89).
 - When `verify` gate commands are configured (via `Harness(verify=[...])`, CLI `--verify`, or Skill YAML `verify: [...]`), `Harness.execute_session` deterministically executes each command inside the session's `SandboxPolicy` before allowing `done` to transition to `COMPLETED` (consuming 0 static prompt tokens until a gate fails). A gate that errors instead of failing (sandbox violation, exit 124/126/127) ends the session `FAILED` at once as `gate_error` ("gate misconfigured"), without counting toward the consecutive-failure abort.
 
 ### Rule 4: Progressive `--help` (JIT Schema Disclosure)
@@ -62,7 +62,7 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
   ```text
   [obs_3 folded: `web fetch https://...` (3,420 chars) | use `mem recall obs_3` to expand]
   ```
-- Exception: the latest read of each of the last 8 files (`edit view <p>`, `bash cat/head/tail/sed -n <p>`; paths normalized against the workspace) is not folded until `<p>` is edited (`edit write|replace`, or any other bash command naming it: `sed -i`, `tee`, redirects, `python -c`) or re-read. The 2,400-char head+tail elision still applies, and the hard context budget remains #81. The `mem recall` hint is shown only when `mem` is enabled (else "re-run the command").
+- Exception: the latest read of each of the last 8 files (`edit view <p>`, `bash cat/head/tail/sed -n <p>`; paths normalized against the workspace) is not folded until `<p>` is edited (`edit write|replace`, or any other bash command naming it: `sed -i`, `tee`, redirects, `python -c`) or re-read. The 2,400-char head+tail elision still applies, and under `max_context_tokens` pressure it is folded like any other (oldest first). The `mem recall` hint is shown only when `mem` is enabled (else "re-run the command").
 - History is replayed as structured `tool_calls` + `tool` results (providers with `structured_history = True`; textual `act('...')` form only as fallback), with `stdin` replayed as a real 400-char prefix, never a `<N chars>` placeholder. A call and its result are trimmed together.
 
 ### Rule 6: Deterministic Security Envelope (Never Trust the LLM for Safety)
@@ -71,7 +71,7 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
   - All `edit` and `bash` paths are canonicalized (`Path.resolve()`) and confined to `workspace_root`.
   - Sensitive files/directories (`.env*`, `.git/config`, `.git/hooks`, `~/.ssh`, `.tether/`) are deterministically protected from mutation or read exfiltration.
   - `bash` subprocesses get an allowlisted environment only (secrets with any name are dropped; opt-in via `TETHER_ENV_PASSTHROUGH`). `web fetch` blocks loopback/private/metadata addresses (SSRF guard). `SandboxPolicy(network=False)` / `--no-network` adds `--unshare-net` (fails closed without `bwrap`; children can only narrow it).
-  - On Linux, `bash` automatically uses **`bwrap` (Bubblewrap) / `unshare`** when available (read-only `/`, hidden `/home` and `/var/tmp` tmpfs, `/run` unix sockets such as `docker.sock` masked, read-write bind on `workspace_root` with secret-looking files at any depth (`.env*`, `*.pem`, `*.key`, ...), `.tether/`, `.git/config` and `.git/hooks` masked at the mount level, `PR_SET_NO_NEW_PRIVS`), falling back to a strict deterministic command allowlist/denylist (`rm -rf /`, `sudo`, `dd`, `mkfs`, `chmod 777`, etc. blocked).
+  - On Linux, `bash` automatically uses **`bwrap` (Bubblewrap) / `unshare`** when available (read-only `/`, hidden `/home` and `/var/tmp` tmpfs (existing `PATH` dirs under them, e.g. `~/.local/bin`, are re-bound read-only so `uv`/`node` still run, #76), `/run` unix sockets such as `docker.sock` masked, read-write bind on `workspace_root` with secret-looking files at any depth (`.env*`, `*.pem`, `*.key`, ...), `.tether/`, `.git/config` and `.git/hooks` masked at the mount level, `PR_SET_NO_NEW_PRIVS`), falling back to a strict deterministic command allowlist/denylist (`rm -rf /`, `sudo`, `dd`, `mkfs`, `chmod 777`, etc. blocked).
   - Sub-agents spawned via `agent spawn` strictly inherit or narrow their parent session's `SandboxPolicy` (no privilege escalation).
 
 ### Rule 7: Zero Framework Bloat
@@ -85,7 +85,7 @@ Reviewers and the Team Leader cite the failing line when they reject a change.
 ```text
 tether/
 ├── __init__.py          # Public SDK exports: Harness, Pipeline, Session, Skill, SandboxPolicy, tool
-├── cli.py               # Headless-capable CLI: `tether run|pipeline|status|read|query|send|list|wait|stats|retro|events|misses|mcp` (`--json` + documented exit codes)
+├── cli.py               # Headless-capable CLI: `tether run|pipeline|status|read|query|send|list|wait|stop|stats|retro|events|misses|mcp` (`--json` + documented exit codes)
 ├── harness.py           # Control Plane: bounded agent loop, pipelines, SQLite inbox, worker spawn
 ├── pipeline.py          # Workflow Plane: declarative multi-stage YAML pipelines & model routing
 ├── api.py               # Agent-facing JSON session view, exit codes, events (`--json`)
@@ -166,6 +166,7 @@ tether/
 - If no provider prefix is given, defaults to `ollama/<model>`.
 - Custom SDK transports (e.g., internal corporate gateways, custom local runners, or test harnesses) can be injected via `Harness(provider=...)` or registered under a custom `<prefix>/<model>` via `register_provider(prefix, factory)`.
 - Declarative providers: `~/.tether/providers.yaml` (override with `TETHER_PROVIDERS`, never read from the workspace) maps a model-id prefix to a `kind` (`openai`, `ollama`, `anthropic` or `command`); `claude-cli/sonnet` uses the entry `claude-cli`. The `command` kind runs `argv` directly (never through a shell) with `{model}`, `{system}` and `{conversation}` placeholders, and reads the reply and usage from JSON paths. Built-in prefixes (`ollama`, `openai`, `anthropic`, `claude`) cannot be shadowed, keys are env var names (never values), and an invalid file raises `ValueError: providers.yaml: ...`. Implementation in `tether/providers/declarative.py`; docs in `docs/reference/providers.md`.
+- `providers.yaml` entries may set `params` (request-body parameters for HTTP kinds, overridable per call via `--param`, `Harness(params=)`, MCP `params`, stored in `sandbox_json`, #112) and `rpm` (requests per minute cap for HTTP kinds, sliding window shared across processes via a lock file, #110).
 - Environment variables:
   - `OLLAMA_HOST` (default `http://localhost:11434`); `TETHER_OLLAMA_FALLBACK=1` opts in to substituting an installed model when the requested one 404s (default: fail loudly with the installed list)
   - `OPENAI_API_KEY`, `OPENAI_BASE_URL` (default `https://api.openai.com/v1` — works with OpenRouter, vLLM, Groq, etc.)
@@ -204,7 +205,7 @@ uv run mypy tether/
 2. **Async-First Design:**
    - All model calls and I/O-bound tool executions must be `async`/`await` native, while seamlessly wrapping synchronous user `@tool` functions.
 3. **Graceful Degradation on 9B Hallucinations:**
-   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), and turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`).
+   - Small models occasionally emit `cmd` with minor quirks (e.g., leading `$`, wrapping backticks, or calling a tool name directly instead of `act`). The multiplexer must normalize trivial syntax quirks deterministically before failing. This includes unescaping literal `\n`/`\t` in one-line `stdin` for `edit write`/`edit replace` and `done` (never `bash`), turning a trailing shell heredoc in `cmd` (`edit write f <<'EOF' ... EOF`) into `stdin` (never for `bash`), unwrapping `bash -c/-lc '<cmd>'` and `sh -c` wrappers, running an unknown first word through the enabled `bash` tool (never for disabled built-ins), registering extra quirk fixers with `register_normalizer`, mapping a native tool call named other than `act` to `<name> --k 'v'` so the JIT usage answers it, inferring the `edit` verb (`edit <path>` -> view/replace/write) and accepting `apply_patch` (`*** Begin Patch` or unified-diff patches, also when a diff arrives as `edit <path>` stdin) as an `edit` verb.
 4. **Golden traces:** behaviour the model sees (framing, folding, nudges, tool output) is locked by `tests/golden/*.snap.json`. A PR that changes it runs `UPDATE_GOLDEN=1 pytest tests/test_golden.py` and the snapshot diff is part of the review; unrelated traces must not change. See `docs/testing.md`.
 5. **E2E scenarios:** every feature or bug fix ships with a scenario in `e2e/` (real CLI, real ollama behind a fault-injecting proxy, local only, `TETHER_E2E=1 uv run pytest e2e`). Assert on what tether sends and records, never on the model's skill; script the reply or inject the fault so the test does not depend on model potency. **Before merging any PR, run `TETHER_E2E=1 uv run pytest e2e` locally and put the result in the PR description (CI cannot run it). Every failure must be mitigated before merge: fix the code, or fix a wrong scenario, or record in the PR why it is acceptable and what follows up. A PR is not merged with an unexplained red scenario.** See `docs/testing.md`.
 6. **Documentation Integrity:**

{tether_sdk-0.2.4 → tether_sdk-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tether-sdk
-Version: 0.2.4
+Version: 0.2.5
 Summary: General-purpose, headless-first LLM-Native micro-harness for small language models
 Project-URL: Homepage, https://github.com/igormcsouza/tether
 Project-URL: Repository, https://github.com/igormcsouza/tether
@@ -265,8 +265,10 @@ $ tether stats
 # 7. Post-mortem: distill friction into <=2 short rules (diff; --apply writes ## Learned Rules)
 $ tether retro ses_8f92a1 [--apply]
 
-# 8. Call Tether from another agent: JSON output, exit codes (0 ok, 2 needs input, 3 gate, 4 max steps, 5 error)
+# 8. Call Tether from another agent: JSON output, exit codes (0 ok, 2 needs input, 3 gate, 4 max steps or stalled, 5 error)
 $ tether run --json --output-schema out.json "goal"
+$ tether stop ses_8f92a1                                 # terminate a running detached session
+$ tether run --param temperature=0 --param think=false "goal"   # per-call model parameters
 $ claude mcp add tether -- tether mcp --workspace "$PWD"   # MCP server (needs: pip install "tether-sdk[mcp]")
 ```
 

{tether_sdk-0.2.4 → tether_sdk-0.2.5}/README.md

@@ -244,8 +244,10 @@ $ tether stats
 # 7. Post-mortem: distill friction into <=2 short rules (diff; --apply writes ## Learned Rules)
 $ tether retro ses_8f92a1 [--apply]
 
-# 8. Call Tether from another agent: JSON output, exit codes (0 ok, 2 needs input, 3 gate, 4 max steps, 5 error)
+# 8. Call Tether from another agent: JSON output, exit codes (0 ok, 2 needs input, 3 gate, 4 max steps or stalled, 5 error)
 $ tether run --json --output-schema out.json "goal"
+$ tether stop ses_8f92a1                                 # terminate a running detached session
+$ tether run --param temperature=0 --param think=false "goal"   # per-call model parameters
 $ claude mcp add tether -- tether mcp --workspace "$PWD"   # MCP server (needs: pip install "tether-sdk[mcp]")
 ```
 

tether_sdk-0.2.5/bench/README.md

@@ -0,0 +1,37 @@
+# bench: model-in-the-loop checks (developers only)
+
+Not part of the `tether` package, not run in CI. Use it to onboard a new model or to check a
+release for regressions that deterministic tests cannot see (unit, golden and e2e tests stay the
+gate; see `docs/testing.md`).
+
+Seven tasks in `tasks/<name>/` (workspace in `ws/`, goal in `goal.txt`, hidden tests in `hidden/`):
+`focus` (one exact edit in a distractor-heavy repo, scope-checked), `code-simple`, `code-medium`, `code-hard` (graded by hidden unit tests; editing the visible tests
+scores 0), `qa`, `content`, `review` (graded by constraint counts and keyword checks, so they are
+a smoke test for quality, not a judge).
+
+Three runners, so a gap can be attributed:
+
+| runner | what it is |
+|---|---|
+| `tether` | the CLI under test (`--tether-bin` picks a version or a venv) |
+| `naive` | `naive_agent.py`, a plain OpenAI-compatible tool loop (bash/read/write) with the same model |
+| `claude` | Claude Code in print mode, as a frontier baseline (`--models cc/sonnet`) |
+
+```bash
+export NVIDIA_API_KEY=...            # only for nvidia/* models; OLLAMA_HOST defaults to :11435
+uv run python bench/run.py --runners tether,naive --models ollama/qwen3.5:9b \
+    --tasks code-simple,qa --trials 2 --out /tmp/bench
+uv run python bench/run.py --runners tether --tether-bin /path/to/venv/bin/tether \
+    --models ollama/qwen3.5:9b --param think=false --out /tmp/bench-v0.2.3
+```
+
+Each run lands in `<out>/<runner>__<model>__<task>__t<k>/` with the workspace, stdout, the
+session DB (`t.db`) and `done.json`; `results.jsonl` collects them. Re-running skips finished
+cells; `--redo-infra` reruns cells that failed on provider errors (429, 503, timeouts).
+
+Read results with the friction events (`syntax_error`, `loop_stall`, `nudge`, ...) next to the
+score: a low score with many events is a harness problem, a low score with none is the model.
+Rate limits (NVIDIA: 40 requests/min) invalidate results silently; set `rpm:` in `providers.yaml`
+and keep parallelism low (`--par nvidia=2`).
+
+Caveats: one run per cell is noisy (about 0.1); the scores are regex and hidden-test based.

tether_sdk-0.2.5/bench/grade.py

@@ -0,0 +1,148 @@
+"""Deterministic graders. grade(task, ws, result_text) -> {"score": 0..1, "pass": bool, "detail": {...}}"""
+import ast
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+TASKS = os.path.join(HERE, "tasks")
+
+
+def _tree_hash(path):
+    h = hashlib.sha256()
+    for root, _, files in sorted(os.walk(path)):
+        for f in sorted(files):
+            if f.endswith(".pyc"):
+                continue
+            p = os.path.join(root, f)
+            h.update(os.path.relpath(p, path).encode())
+            h.update(open(p, "rb").read())
+    return h.hexdigest()
+
+
+def _code(task, ws, _result):
+    tmp = tempfile.mkdtemp(prefix="g_")
+    try:
+        shutil.copytree(ws, tmp, dirs_exist_ok=True)
+        orig_tests = os.path.join(TASKS, task, "ws", "tests")
+        tampered = os.path.isdir(orig_tests) and _tree_hash(orig_tests) != _tree_hash(os.path.join(ws, "tests"))
+        shutil.copy(os.path.join(TASKS, task, "hidden", "test_hidden.py"), os.path.join(tmp, "_hidden_test.py"))
+        try:
+            p = subprocess.run([sys.executable, "-m", "unittest", "-v", "_hidden_test"], cwd=tmp, capture_output=True,
+                               text=True, timeout=90)
+            out = p.stderr + p.stdout
+        except subprocess.TimeoutExpired:
+            return {"score": 0.0, "pass": False, "detail": {"error": "hidden tests timed out"}}
+        ok = len(re.findall(r"\.\.\. ok$", out, re.M))
+        bad = len(re.findall(r"\.\.\. (FAIL|ERROR)$", out, re.M))
+        total = ok + bad
+        score = ok / total if total else 0.0
+        if tampered:
+            return {"score": 0.0, "pass": False, "detail": {"tampered_tests": True, "hidden_ok": ok, "hidden_total": total}}
+        return {"score": round(score, 3), "pass": total > 0 and bad == 0, "detail": {"hidden_ok": ok, "hidden_total": total}}
+    finally:
+        shutil.rmtree(tmp, ignore_errors=True)
+
+
+def _json_array(text):
+    m = re.search(r"\[.*\]", text or "", re.S)
+    if not m:
+        return None
+    for parse in (json.loads, ast.literal_eval):
+        try:
+            v = parse(m.group(0))
+            return v if isinstance(v, list) else None
+        except (ValueError, SyntaxError):
+            pass
+    return None
+
+
+def _qa(_task, _ws, result):
+    arr = _json_array(result)
+    checks = [
+        lambda a: "17:25" in a,
+        lambda a: re.search(r"\b391(\.0+)?\b", a) is not None,
+        lambda a: "canberra" in a.lower(),
+        lambda a: re.sub(r"\s", "", a) == "[1,3]",
+        lambda a: re.search(r"\b(10|ten)\b", a.lower()) is not None,
+        lambda a: ("1024" in a or "2**10" in a.replace(" ", "") or "2^10" in a.replace(" ", "")) and re.search(r"\b24\b", a) is not None,
+    ]
+    if arr is None or len(arr) != 6:
+        return {"score": 0.0, "pass": False, "detail": {"error": "not a JSON array of 6"}}
+    got = [bool(c(str(a))) for c, a in zip(checks, arr)]
+    return {"score": round(sum(got) / 6, 3), "pass": all(got), "detail": {"correct": got}}
+
+
+def _content(_task, _ws, result):
+    t = (result or "").strip()
+    words = len(re.findall(r"\S+", t))
+    lines = [ln for ln in t.splitlines() if ln.strip()]
+    sentences = re.split(r"(?<=[.?])\s+", t.replace("\n", " "))
+    last = sentences[-1] if sentences else ""
+    c = {
+        "<=120 words": words <= 120 and words > 0,
+        "3 bullets": sum(1 for ln in lines if ln.startswith("- ")) == 3,
+        "snapshot": "snapshot" in t.lower(),
+        "rollback": "rollback" in t.lower(),
+        "CTA last": "tidepool.dev" in last.lower(),
+        "no !": "!" not in t,
+        "no banned": not re.search(r"revolutionary|game-changing", t, re.I),
+    }
+    return {"score": round(sum(c.values()) / len(c), 3), "pass": all(c.values()), "detail": {"checks": c, "words": words}}
+
+
+FLAWS = {
+    "selection/no-randomisation": r"random|self-select|selection bias|volunteer|chose to install|non-?random|assign",
+    "causal-overreach": r"causal|causation|\bcause|correlat|overreach|overstat",
+    "tiny-sample/generalise": r"small sample|sample size|n\s*=\s*1[12]|23 (student|participant)|under-?power|low power|generali[sz]|single (university|course)|one (course|university)",
+    "multiple-comparisons": r"multiple (comparison|testing|outcome|test)|bonferroni|family-?wise|false[- ]positive|correction|cherry|selective",
+    "number-inconsistency": r"inconsisten|discrepan|mismatch|contradict|12\.4.{0,80}(8\.1|7\.2)|(8\.1|7\.2).{0,80}12\.4",
+    "baseline-gpa-imbalance": r"gpa|baseline|prior (performance|achievement|ability)|confound|not matched|unmatched|equivalen",
+    "unblinded-grader": r"blind|grader|grading|instructor|assessor|rater",
+    "no-effect-size/CI": r"effect size|confidence interval|\bCIs?\b|cohen",
+}
+
+
+def _review(_task, _ws, result):
+    t = result or ""
+    hit = {k: bool(re.search(v, t, re.I)) for k, v in FLAWS.items()}
+    n = len(re.findall(r"^\s*\d+[.)]", t, re.M))
+    return {"score": round(sum(hit.values()) / len(hit), 3), "pass": sum(hit.values()) >= 6,
+            "detail": {"found": [k for k, v in hit.items() if v], "missed": [k for k, v in hit.items() if not v], "items": n}}
+
+
+def _focus(task, ws, result):
+    """_code + scope: the goal allows editing mini/cli.py only; other changes halve the score."""
+    r = _code(task, ws, result)
+    orig = os.path.join(TASKS, task, "ws")
+    touched = []
+    for root, dirs, files in os.walk(orig):
+        for f in files:
+            rel = os.path.relpath(os.path.join(root, f), orig)
+            p = os.path.join(ws, rel)
+            if rel != os.path.join("mini", "cli.py") and (
+                not os.path.exists(p) or open(p, "rb").read() != open(os.path.join(root, f), "rb").read()
+            ):
+                touched.append(rel)
+    r["detail"]["touched_other"] = touched
+    if touched:
+        r["score"], r["pass"] = round(r["score"] / 2, 3), False
+    return r
+
+
+def grade(task, ws, result):
+    fn = {"qa": _qa, "content": _content, "review": _review, "focus": _focus}.get(task, _code)
+    try:
+        return fn(task, ws, result)
+    except Exception as e:  # grader bug must never look like a model result
+        return {"score": 0.0, "pass": False, "detail": {"grader_error": repr(e)}}
+
+
+if __name__ == "__main__":
+    task, ws, rf = sys.argv[1:4]
+    print(json.dumps(grade(task, ws, open(rf).read() if os.path.exists(rf) else "")))

tether_sdk-0.2.5/bench/naive_agent.py

@@ -0,0 +1,92 @@
+"""Control harness: plain OpenAI-compatible tool loop (bash/read_file/write_file), append-only history,
+stop when the model answers without a tool call. Same model, different harness => isolates tether's effect."""
+import argparse
+import json
+import os
+import subprocess
+import time
+
+import httpx
+
+TOOLS = [
+    {"type": "function", "function": {"name": "bash", "description": "Run a shell command in the workspace.",
+     "parameters": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}}},
+    {"type": "function", "function": {"name": "read_file", "description": "Read a text file.",
+     "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
+    {"type": "function", "function": {"name": "write_file", "description": "Create or overwrite a file.",
+     "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
+                    "required": ["path", "content"]}}},
+]
+SYSTEM = ("You are a capable assistant working in a project directory with tools. Use the tools to do the task. "
+          "When finished, reply with a plain-text final answer and no tool call.")
+
+
+def run_tool(name, args, ws):
+    try:
+        if name == "bash":
+            p = subprocess.run(args["command"], shell=True, cwd=ws, capture_output=True, text=True, timeout=60)
+            return (p.stdout + p.stderr)[-10000:] or f"(exit {p.returncode})"
+        if name == "read_file":
+            return open(os.path.join(ws, args["path"])).read()[:20000]
+        if name == "write_file":
+            p = os.path.join(ws, args["path"])
+            os.makedirs(os.path.dirname(p) or ws, exist_ok=True)
+            open(p, "w").write(args["content"])
+            return "ok"
+        return f"unknown tool {name}"
+    except Exception as e:
+        return f"error: {e}"
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base-url", required=True)
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--key-env", default="")
+    ap.add_argument("--ws", required=True)
+    ap.add_argument("--goal", required=True)
+    ap.add_argument("--max-turns", type=int, default=40)
+    ap.add_argument("--result-file", required=True)
+    ap.add_argument("--stats-file", required=True)
+    a = ap.parse_args()
+    hdr = {"Authorization": f"Bearer {os.environ.get(a.key_env, 'x')}"}
+    msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": a.goal}]
+    pt = ct = turns = 0
+    peak = 0
+    final, status = "", "max_turns"
+    t0 = time.time()
+    with httpx.Client(timeout=300) as c:
+        for turns in range(1, a.max_turns + 1):
+            for attempt in range(4):
+                r = c.post(f"{a.base_url}/chat/completions", headers=hdr,
+                           json={"model": a.model, "messages": msgs, "tools": TOOLS, "temperature": 0.2})
+                if r.status_code in (429, 500, 502, 503, 504) and attempt < 3:
+                    time.sleep(2 * (attempt + 1))
+                    continue
+                break
+            if r.status_code != 200:
+                status, final = "http_error", f"HTTP {r.status_code}: {r.text[:300]}"
+                break
+            d = r.json()
+            u = d.get("usage") or {}
+            pt += u.get("prompt_tokens", 0)
+            ct += u.get("completion_tokens", 0)
+            peak = max(peak, u.get("prompt_tokens", 0))
+            m = d["choices"][0]["message"]
+            msgs.append({k: v for k, v in m.items() if k in ("role", "content", "tool_calls") and v is not None})
+            if not m.get("tool_calls"):
+                final, status = m.get("content") or "", "completed"
+                break
+            for tc in m["tool_calls"]:
+                try:
+                    args = json.loads(tc["function"]["arguments"] or "{}")
+                except ValueError:
+                    args = {}
+                msgs.append({"role": "tool", "tool_call_id": tc["id"], "content": run_tool(tc["function"]["name"], args, a.ws)})
+    open(a.result_file, "w").write(final)
+    json.dump({"status": status, "steps": turns, "prompt_tokens": pt, "completion_tokens": ct, "peak_tokens": peak,
+               "wall_s": round(time.time() - t0, 1)}, open(a.stats_file, "w"))
+
+
+if __name__ == "__main__":
+    main()

tether_sdk-0.2.5/bench/run.py

@@ -0,0 +1,173 @@
+"""Benchmark runner. Matrix = runners x models x tasks x trials. Results appended to <out>/results.jsonl.
+
+runners: tether (CLI under test; --tether-bin selects a version), naive (control loop), claude (Claude Code baseline).
+"""
+import argparse
+import asyncio
+import json
+import os
+import re
+import shutil
+import signal
+import sqlite3
+import sys
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import grade  # noqa: E402
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+TASK_CFG = {  # tether tools / step cap / wall timeout
+    "code-simple": ("edit,bash", 30, 600), "code-medium": ("edit,bash", 40, 900), "code-hard": ("edit,bash", 50, 1200),
+    "focus": ("edit,bash,mem", 30, 600),
+    "qa": ("", 10, 300), "content": ("", 10, 300), "review": ("edit", 15, 420),
+}
+
+
+def group(model):
+    return "nvidia" if model.startswith("openai/") else "ollama" if model.startswith("ollama/") else "claude" if model.startswith(("claude", "cc/")) else "nvidia"
+
+
+_HELP = {}
+
+
+def flags(tether_bin):
+    if tether_bin not in _HELP:
+        import subprocess
+        _HELP[tether_bin] = subprocess.run([tether_bin, "run", "--help"], capture_output=True, text=True).stdout
+    return _HELP[tether_bin]
+
+
+def build_cmd(runner, model, task, goal, ws, rd, tether_bin, params=()):
+    tools, steps, _ = TASK_CFG[task]
+    if runner == "tether":
+        h = flags(tether_bin)
+        cmd = [tether_bin, "run", "--model", model, "--max-steps", str(steps), "--db", f"{rd}/t.db"]
+        if "--result-file" in h:
+            cmd += ["--result-file", f"{rd}/result.txt", "--json"]
+        if tools:
+            cmd += ["--tools", tools]
+        for item in params:
+            cmd += ["--param", item]
+        return cmd + [goal]
+    if runner == "naive":
+        base, key = ("http://localhost:11435/v1", "") if model.startswith("ollama/") else ("https://integrate.api.nvidia.com/v1", "NVIDIA_API_KEY")
+        return [sys.executable, f"{HERE}/naive_agent.py", "--base-url", base, "--model", model.split("/", 1)[1] if model.startswith("ollama/") else model.removeprefix("nvidia/"),
+                "--key-env", key, "--ws", ws, "--goal", goal, "--max-turns", str(steps), "--result-file", f"{rd}/result.txt", "--stats-file", f"{rd}/stats.json"]
+    if runner == "claude":
+        return ["env", "-u", "ANTHROPIC_API_KEY", "claude", "-p", goal, "--model", model.removeprefix("cc/"), "--permission-mode", "bypassPermissions",
+                "--output-format", "json", "--setting-sources", "", "--disable-slash-commands", "--no-session-persistence", "--max-budget-usd", "3"]
+    raise SystemExit(f"unknown runner {runner}")
+
+
+def is_infra(r):
+    """Provider/network failure or a run that never got going: says nothing about model or harness."""
+    if r["runner"] == "tether" and r.get("exit") == 5:
+        try:
+            t = open(r["dir"] + "/stdout.txt", errors="replace").read() + open(r["dir"] + "/stderr.txt", errors="replace").read()
+        except OSError:
+            t = ""
+        return bool(re.search(r"unavailable after|ProviderError|429|503|Not Found", t))
+    return r["runner"] != "claude" and r["run_status"] == "timeout" and (r.get("steps") or 0) <= 3
+
+
+def tether_stats(db):
+    out = {}
+    try:
+        c = sqlite3.connect(db)
+        s = c.execute("select status, step_count, peak_tokens, total_prompt_tokens, total_completion_tokens from sessions where parent_id is null order by created_at limit 1").fetchone()
+        if s:
+            out.update(status=s[0], steps=s[1], peak_tokens=s[2], prompt_tokens=s[3], completion_tokens=s[4])
+        ev = dict(c.execute("select event_type, count(*) from turns where event_type is not null group by 1").fetchall())
+        out["events"] = {k: v for k, v in ev.items() if k not in ("ok", "completed")}
+    except Exception as e:
+        out["db_error"] = repr(e)
+    return out
+
+
+async def one(sem, rec, args, results):
+    runner, model, task, trial = rec
+    name = f"{runner}__{model.replace('/', '_').replace(':', '-')}__{task}__t{trial}"
+    rd = os.path.join(args.out, name)
+    ws = os.path.join(rd, "ws")
+    if os.path.exists(f"{rd}/done.json") and not args.force:
+        old = json.load(open(f"{rd}/done.json"))
+        if not (args.redo_infra and is_infra(old)):
+            results.append(old)
+            return
+    async with sem[group(model)]:
+        shutil.rmtree(rd, ignore_errors=True)
+        os.makedirs(rd)
+        shutil.copytree(f"{HERE}/tasks/{task}/ws", ws)
+        goal = open(f"{HERE}/tasks/{task}/goal.txt").read().strip()
+        cmd = build_cmd(runner, model, task, goal, ws, rd, args.tether_bin, args.param)
+        env = dict(os.environ, OLLAMA_HOST="http://localhost:11435", PYTHONDONTWRITEBYTECODE="1")
+        if model.startswith("openai/"):  # old tether has no providers.yaml: route openai/ to NVIDIA's OpenAI-compatible API
+            env.update(OPENAI_BASE_URL="https://integrate.api.nvidia.com/v1", OPENAI_API_KEY=os.environ.get("NVIDIA_API_KEY", ""))
+        t0 = time.time()
+        p = await asyncio.create_subprocess_exec(*cmd, cwd=ws, env=env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, start_new_session=True)
+        status = "ok"
+        try:
+            out, err = await asyncio.wait_for(p.communicate(), TASK_CFG[task][2])
+        except asyncio.TimeoutError:
+            os.killpg(p.pid, signal.SIGKILL)
+            out, err = await p.communicate()
+            status = "timeout"
+        wall = round(time.time() - t0, 1)
+        open(f"{rd}/stdout.txt", "wb").write(out)
+        open(f"{rd}/stderr.txt", "wb").write(err)
+        info = {"runner": runner, "model": model, "task": task, "trial": trial, "wall_s": wall, "exit": p.returncode, "run_status": status}
+        result = ""
+        if runner == "claude":
+            try:
+                j = json.loads(out.decode())
+                result = j.get("result") or ""
+                u = j.get("usage") or {}
+                info.update(steps=j.get("num_turns"), cost_usd=j.get("total_cost_usd"), is_error=j.get("is_error"),
+                            prompt_tokens=(u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0) + u.get("cache_read_input_tokens", 0)),
+                            completion_tokens=u.get("output_tokens"))
+            except Exception:
+                info["parse_error"] = True
+        elif runner == "naive":
+            try:
+                info.update(json.load(open(f"{rd}/stats.json")))
+                result = open(f"{rd}/result.txt").read()
+            except Exception:
+                info["parse_error"] = True
+        else:
+            result = open(f"{rd}/result.txt").read() if os.path.exists(f"{rd}/result.txt") else out.decode(errors="replace")
+            info.update(tether_stats(f"{rd}/t.db"))
+        if not os.path.exists(f"{rd}/result.txt"):
+            open(f"{rd}/result.txt", "w").write(result)
+        info.update(grade.grade(task, ws, result))
+        info["dir"] = rd
+        json.dump(info, open(f"{rd}/done.json", "w"))
+        with open(f"{args.out}/results.jsonl", "a") as f:
+            f.write(json.dumps(info) + "\n")
+        results.append(info)
+        print(f"[{len(results)}] {name}: score={info['score']} pass={info['pass']} {wall}s steps={info.get('steps')}", flush=True)
+
+
+async def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--runners", default="tether")
+    ap.add_argument("--models", required=True)
+    ap.add_argument("--tasks", default=",".join(TASK_CFG))
+    ap.add_argument("--trials", type=int, default=1)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--tether-bin", default="tether")
+    ap.add_argument("--param", action="append", default=[], help="forwarded to `tether run --param`")
+    ap.add_argument("--par", default="nvidia=4,ollama=1,claude=3")
+    ap.add_argument("--force", action="store_true")
+    ap.add_argument("--redo-infra", action="store_true", help="rerun cells that failed on provider errors")
+    args = ap.parse_args()
+    os.makedirs(args.out, exist_ok=True)
+    sem = {k: asyncio.Semaphore(int(v)) for k, v in (x.split("=") for x in args.par.split(","))}
+    recs = [(r, m, t, k) for r in args.runners.split(",") for m in args.models.split(",") for t in args.tasks.split(",") for k in range(args.trials)
+            if (r == "claude") == m.startswith("cc/")]
+    print(f"{len(recs)} runs", flush=True)
+    results = []
+    await asyncio.gather(*(one(sem, r, args, results) for r in recs))
+
+
+asyncio.run(main())

tether_sdk-0.2.5/bench/tasks/code-hard/goal.txt

@@ -0,0 +1 @@
+The calc library (expression evaluator in calc/) has bugs: `python3 -m unittest discover -s tests -t .` fails. Find and fix the root causes in the library code so it behaves as documented in calc/__init__.py. Do not edit the tests. Other behaviours of the library are covered by hidden tests, so fix the underlying causes, not just the failing inputs.

tether_sdk-0.2.5/bench/tasks/code-hard/hidden/test_hidden.py

@@ -0,0 +1,31 @@
+import unittest
+from calc import calc, CalcError
+
+
+class H(unittest.TestCase):
+    def test_basic(self):
+        for s, e in [("1+2*3", 7), ("(1+2)*3", 9), ("10/4", 2.5), ("2*3-4", 2), ("1 - 2 - 3", -4), ("8/2/2", 2),
+                     ("1.5e3+1", 1501), (".5+.5", 1), ("  7  ", 7)]:
+            self.assertAlmostEqual(calc(s), e, msg=s)
+
+    def test_pow(self):
+        for s, e in [("2^3^2", 512), ("2^10", 1024), ("(2^3)^2", 64), ("2^-1", 0.5), ("2*3^2", 18), ("4^0.5", 2)]:
+            self.assertAlmostEqual(calc(s), e, msg=s)
+
+    def test_unary(self):
+        for s, e in [("-2^2", -4), ("-2*3", -6), ("--3", 3), ("-(1+2)", -3), ("2*-3", -6), ("-2^2^2", -16), ("1--1", 2), ("+5", 5)]:
+            self.assertAlmostEqual(calc(s), e, msg=s)
+
+    def test_vars(self):
+        env = {"x1": 3, "my_var": 4, "y": 2, "a2b": 1}
+        for s, e in [("x1+1", 4), ("my_var*y", 8), ("a2b+x1", 4), ("x1^y", 9), ("-x1", -3)]:
+            self.assertAlmostEqual(calc(s, env), e, msg=s)
+
+    def test_funcs(self):
+        for s, e in [("min(3,1,2)", 1), ("max(1, 2^3)", 8), ("abs(-4)", 4), ("sqrt(16)", 4), ("max(abs(-2), min(5,6))", 5)]:
+            self.assertAlmostEqual(calc(s), e, msg=s)
+
+    def test_errors(self):
+        for s in ["1/0", "2/(1-1)", "x", "foo(1)", "abs(1,2)", "sqrt(-1)", "1 +", "(1", "1 $ 2", "min()", "1 2"]:
+            with self.assertRaises(CalcError, msg=s):
+                calc(s)

tether_sdk-0.2.5/bench/tasks/code-hard/ref/calc/__init__.py

@@ -0,0 +1,10 @@
+from .errors import CalcError
+from .evaluator import evaluate
+from .parser import Parser
+from .tokenizer import tokenize
+
+
+def calc(src, env=None):
+    """Evaluate an arithmetic expression: + - * / ^ (right assoc), unary minus, parens,
+    variables from env, functions min/max/abs/sqrt. Raises CalcError on any user error."""
+    return evaluate(Parser(tokenize(src)).parse(), env or {})

tether_sdk-0.2.5/bench/tasks/code-hard/ref/calc/errors.py

@@ -0,0 +1,2 @@
+class CalcError(Exception):
+    pass

tether_sdk-0.2.5/bench/tasks/code-hard/ref/calc/evaluator.py

@@ -0,0 +1,44 @@
+import math
+
+from .errors import CalcError
+
+FUNCS = {"min": (1, None, min), "max": (1, None, max), "abs": (1, 1, abs), "sqrt": (1, 1, math.sqrt)}
+
+
+def evaluate(node, env):
+    k = node[0]
+    if k == "num":
+        return node[1]
+    if k == "var":
+        if node[1] not in env:
+            raise CalcError(f"undefined variable {node[1]}")
+        return env[node[1]]
+    if k == "neg":
+        return -evaluate(node[1], env)
+    if k == "bin":
+        op = node[1]
+        a, b = evaluate(node[2], env), evaluate(node[3], env)
+        if op == "+":
+            return a + b
+        if op == "-":
+            return a - b
+        if op == "*":
+            return a * b
+        if op == "/":
+            if b == 0:
+                raise CalcError("division by zero")
+            return a / b
+        if op == "^":
+            return a**b
+    if k == "call":
+        name, args = node[1], node[2]
+        if name not in FUNCS:
+            raise CalcError(f"unknown function {name}")
+        lo, hi, fn = FUNCS[name]
+        if len(args) < lo or (hi is not None and len(args) > hi):
+            raise CalcError(f"{name}: wrong number of arguments")
+        vals = [evaluate(a, env) for a in args]
+        if name == "sqrt" and vals[0] < 0:
+            raise CalcError("sqrt of negative number")
+        return fn(*vals)
+    raise CalcError("bad node")