agent-testbench 0.1.0__tar.gz

agent_testbench-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.testbench/
+examples/*/outputs/
+.DS_Store

agent_testbench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sarp Tandoven
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_testbench-0.1.0/PKG-INFO

@@ -0,0 +1,301 @@
+Metadata-Version: 2.5
+Name: agent-testbench
+Version: 0.1.0
+Summary: Let a coding agent iterate on notebooks, models and UIs like an testbench: run, read the evidence, fix, rerun - locally, on Modal or on Colab, inside a budget.
+Project-URL: Homepage, https://github.com/sarptandoven/agent-testbench
+Project-URL: Issues, https://github.com/sarptandoven/agent-testbench/issues
+Author: Sarp Tandoven
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,claude,claude-code,colab,gpu,mcp,modal,notebooks,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: ipykernel>=6.29
+Requires-Dist: jupyter-client>=8
+Requires-Dist: nbclient>=0.10
+Requires-Dist: nbformat>=5.9
+Requires-Dist: pyyaml>=6
+Provides-Extra: all
+Requires-Dist: mcp>=1.2; extra == 'all'
+Requires-Dist: modal>=1.5; extra == 'all'
+Requires-Dist: playwright>=1.45; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: mcp>=1.2; extra == 'dev'
+Requires-Dist: modal>=1.5; extra == 'dev'
+Requires-Dist: playwright>=1.45; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2; extra == 'mcp'
+Provides-Extra: modal
+Requires-Dist: modal>=1.5; extra == 'modal'
+Provides-Extra: web
+Requires-Dist: playwright>=1.45; extra == 'web'
+Description-Content-Type: text/markdown
+
+# agent-testbench
+
+[![tests](https://github.com/sarptandoven/agent-testbench/actions/workflows/ci.yml/badge.svg)](https://github.com/sarptandoven/agent-testbench/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/agent-testbench.svg)](https://pypi.org/project/agent-testbench/)
+[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sarptandoven/agent-testbench/blob/main/LICENSE)
+
+**A test bench your coding agent operates.** Claude, GPT/Codex or any agent can start a live kernel on a Modal GPU,
+a Colab VM or your machine, run code and notebook cells in it, inspect variables and plots, install packages,
+move files, fix and rerun - and then prove the result with a clean, checked run. Everything stays inside a
+budget you set.
+
+```
+$ testbench session start gpu --backend modal --gpu T4 --idle-minutes 3 --max-minutes 8
+gpu  ready on modal T4  up 0.0 min, idle 0.0 min (stops after 3 idle / 8 total)  ~$0.000 so far
+Within budget: up to $0.16 (today so far $0.01 of $2.00).
+
+$ testbench exec -f check_gpu.py
+[check_gpu.py | ok | 3.3 s]
+20 matmuls of 4096x4096 on Tesla T4: 0.67 s
+=> torch.Size([4096, 4096])
+
+$ testbench exec "float(b.mean()), torch.cuda.max_memory_allocated() // 2**20"      # variables persist
+[code | ok | 0.2 s]
+=> (1023.1211547851562, 200)
+
+$ testbench session stop
+gpu stopped after 1.4 min, about $0.017
+```
+
+## What your agent can do with it
+
+- **Work live** in a kernel that stays up - on this machine, in a [Modal](https://modal.com) Sandbox with any
+  GPU (the machinery behind Modal Notebooks), or on a [Colab](https://colab.research.google.com) VM through
+  Google's Colab CLI. Run code, `.py` files, or a notebook's cells with its Colab form fields set; get back what
+  it printed, the last value, plots as image files, and errors with the line that raised them. Install
+  packages, run shell commands (`nvidia-smi`), list, upload and download files, and push local edits with
+  `sync` without losing variables.
+- **Prove with plans**: run notebooks (Colab ones included), scripts or web UIs end to end on any of the same
+  backends, checked against expectations - metric thresholds, files, video frame counts, text on a page - with
+  a report that names the failing cell and line, shows the evidence, and says what changed since the last run.
+- **Stay inside a budget**: every session and run is priced before it starts (worst case = its time limit on
+  its hardware), refused above your cap, sent to you for approval above your threshold, and stopped by the
+  provider at its limit even if your laptop goes to sleep. Sessions stop themselves when idle.
+- **Stay safe**: secrets are referred to by name only, paid APIs get strict mocks in tests, and the Claude Code
+  hook blocks raw cloud launches and commands that would print secrets.
+
+## Install
+
+```bash
+pip install "agent-testbench[all]"     # or: uv tool install "agent-testbench[all]"
+```
+
+The latest from GitHub: `pip install "agent-testbench[all] @ git+https://github.com/sarptandoven/agent-testbench"`.
+
+- **Modal**: `modal setup` once (your account, your card; [pricing](https://modal.com/pricing)).
+- **Colab**: `uv tool install google-colab-cli`, then log in once (`colab new` opens the sign-in; or set
+  `colab: {auth: adc}` if you use gcloud Application Default Credentials). Uses your Colab compute units.
+- **Web UI checks**: `playwright install chromium`.
+
+`testbench doctor` shows what this machine can do. Then, in your project:
+
+```bash
+testbench init          # writes testbench.yaml (budget, session defaults, a plan per notebook) and AGENTS.md
+```
+
+## Connect your agent
+
+**Claude Code** - the plugin adds the skills and the guard hook:
+
+```bash
+claude plugin marketplace add sarptandoven/agent-testbench
+claude plugin install agent-testbench@agent-testbench
+```
+
+Then ask in plain words ("get training working on a T4 and prove it with the smoke plan"), or call the skills:
+`/agent-testbench:iterate` and `/agent-testbench:setup`.
+
+**Claude Desktop, Cursor, Codex and other MCP clients**:
+
+```json
+{ "mcpServers": { "agent-testbench": { "command": "testbench", "args": ["mcp"] } } }
+```
+
+Tools: `session_start`, `session_exec`, `session_files`, `session_sync`, `session_install`, `session_shell`,
+`session_list`, `session_stop`, `plans`, `run_plan`, `wait_run`, `get_report`, `list_runs`, `cancel_run`,
+`spend`, `lint_notebook`, `add_note`. In Claude Code: `claude mcp add agent-testbench -- testbench mcp`.
+
+**Any other agent** reads the rules `testbench init` writes into `AGENTS.md`.
+
+## Live sessions
+
+```bash
+testbench session start                                     # default from testbench.yaml (local unless set)
+testbench session start big --backend modal --gpu A100-80GB --idle-minutes 10 --max-minutes 90
+testbench session start nb --backend colab --gpu T4
+
+testbench exec "x = load_data(); x.shape"                   # code
+testbench exec -f debug/check.py                            # a local file
+testbench exec --cells train.ipynb:1-4 --params '{"EPOCHS": 1}'   # notebook cells, form fields set
+testbench files get outputs/sample.png                      # ls / get / put on the session's machine
+testbench session sync                                      # copy local edits again; variables stay
+testbench session install "transformers==4.46.3"
+testbench session shell "nvidia-smi"
+testbench session list                                      # uptime, idle time, cost so far
+testbench session stop                                      # or let it stop itself
+```
+
+| backend | where the kernel runs | stops |
+|---|---|---|
+| `local` | this machine, in the project folder | after `idle_minutes` unused or `max_minutes` |
+| `modal` | a Modal Sandbox with your GPU (T4 ... H100, B200), project in `/work/project` and `/content`, a persistent volume at `/testbench` | Modal enforces both limits itself (a running command counts as activity), so it stops even if your laptop is off |
+| `colab` | a Colab VM (CPU, T4, L4, A100, H100 by subscription), project in `/content` | after `idle_minutes` unused or `max_minutes` (Colab may also reclaim VMs after an hour or two) |
+
+A call that runs past `--timeout` is interrupted; the kernel and its variables survive. One command runs at a
+time per session.
+
+**GPU libraries on Modal**: the default image is Debian slim with the NVIDIA driver but no CUDA toolkit.
+PyTorch works as it is (its wheels bring their own CUDA libraries - checked on a T4). Libraries that expect the
+CUDA toolkit on the machine, such as CuPy, fail there with errors like
+`libcurand.so.10: cannot open shared object file` (also checked). Give them the CUDA libraries they need as
+described in their install guide (for CuPy: docs.cupy.dev, "Installation"), and confirm it in a short GPU
+session (`testbench session start --backend modal --gpu T4 --max-minutes 5`) before a long run.
+
+Notebooks see a headless `google.colab` (uploads from files you name, downloads kept,
+secrets from the environment, Drive as a local folder), so Colab notebooks run as they are.
+
+## Plans
+
+```yaml
+project: my-model                 # names the Modal app and volume
+
+budget:
+  max_run_usd: 2.00               # refuse any run or session that could cost more
+  ask_above_usd: 0.50             # above this a person approves (--approve)
+  max_day_usd: 10.00              # everything today together
+
+sessions:
+  backend: modal
+  gpu: A10
+  idle_minutes: 10
+  max_minutes: 60
+
+modal:                            # image for Modal plans and sessions
+  python: "3.11"
+  pip: [torch==2.5.1, scikit-learn==1.5.2]
+  cpu: 4
+  memory_gb: 16
+
+plans:
+  smoke:                          # cheapest proof first
+    backend: local
+    steps:
+      - name: lint
+        run: testbench lint train.ipynb --strict
+        max_minutes: 1
+      - name: train tiny
+        notebook: train.ipynb
+        params: {EPOCHS: 1, SUBSET: 200}     # Colab form fields (NAME = value  #@param)
+        max_minutes: 5
+        expect:
+          json: {outputs/metrics.json: {loss: "< 2.0"}}
+
+  gpu-check:                      # a few minutes on the target GPU before a long run
+    backend: modal
+    gpu: A10
+    steps:
+      - name: train short
+        notebook: train.ipynb
+        params: {EPOCHS: 2}
+        max_minutes: 10           # also the hard stop on Modal's side
+        artifacts: [outputs/*.png]
+        expect:
+          json: {outputs/metrics.json: {accuracy: ">= 0.8"}}
+```
+
+```bash
+testbench plans                   # every plan, its hardware and worst-case cost
+testbench run smoke               # runs it and prints the report (follows up to 9 min; then `testbench wait`)
+testbench report                  # the latest report
+```
+
+A report leads with what failed: the cell, the error, the line that raised it, each expectation that missed and
+what was found instead, then what changed since the previous run of the plan ("Fixed: train", "Broke: eval",
+"accuracy 0.71 -> 0.93"), the evidence files to look at, and GPU use per step (an idle GPU shows as 0%).
+Steps are `notebook:`, `run:` (a shell command) or `web:` (a page driven in headless Chromium: click, fill,
+press, wait for, screenshot). The full reference is in [docs/reference.md](https://github.com/sarptandoven/agent-testbench/blob/main/docs/reference.md).
+
+## Guardrails
+
+- **Budget**: worst case = hardware price x time limit (+ start-up). Over `max_run_usd` or the day's cap:
+  refused. Over `ask_above_usd`: needs `--approve`, which the agent is told never to pass itself - and in Claude
+  Code the hook turns it into a question to you. Every finished run and session is written to a ledger;
+  `testbench spend --modal` also reads Modal's own bill.
+- **Hard stops**: plans and sessions on Modal carry their limits in Modal itself (function and Sandbox
+  timeouts, no retries, 2-second scale-down). Colab sessions and runs are stopped by their supervisor and always
+  released, even when a run fails.
+- **Secrets**: by name only; values come from Modal secrets or your environment. The hook refuses commands that
+  would print them.
+- **Paid APIs**: test with strict mocks (`agent_testbench.mocks.StrictAPI`): a module with the real client's
+  name that checks arguments against the published schema and refuses anything unexpected.
+- **Outside world**: the hook asks before `git push`, `gh repo create` and anything in `guard.ask`.
+
+The hook is a safety net for agents, not a security boundary; the budget checks and remote time limits are
+enforced in code either way.
+
+## Notebooks
+
+- `testbench nb split nb.ipynb cells/` and `testbench nb build cells/ nb.ipynb` - edit cells as plain files.
+- `testbench nb fields nb.ipynb` - the form fields a plan or `--params` can set.
+- `testbench lint nb.ipynb` - big saved outputs, embedded video, secrets without a fallback, `files.upload()`,
+  unpinned installs, the inline matplotlib backend leaking into child processes, broken form fields, hard-coded
+  `/content` paths.
+
+## Commands
+
+| | |
+|---|---|
+| `testbench session start/list/stop/sync/install/shell` | live kernels |
+| `testbench exec` / `testbench files` | work in a session |
+| `testbench plans` / `run` / `wait` / `report` / `runs` / `cancel` | plans |
+| `testbench estimate <plan>` / `spend [--modal]` | money |
+| `testbench lint` / `nb split/build/fields` | notebooks |
+| `testbench note "..."` | the project's lab notebook (`.testbench/NOTES.md`) |
+| `testbench init` / `doctor` / `mcp` | set up, check, serve MCP |
+
+## Examples
+
+- [`examples/tiny-model`](https://github.com/sarptandoven/agent-testbench/tree/main/examples/tiny-model): a Colab-style notebook that trains a small network, with plans
+  for local, Modal CPU, Modal T4 and Colab.
+  [Open in Colab](https://colab.research.google.com/github/sarptandoven/agent-testbench/blob/main/examples/tiny-model/tiny_model.ipynb)
+- [`examples/web-ui`](https://github.com/sarptandoven/agent-testbench/tree/main/examples/web-ui): a page checked in a headless browser - clicks, text, console errors, screenshots.
+
+## How it is tested
+
+`python -m pytest` runs everything that needs no account: config, notebooks executed in real kernels (errors,
+timeouts, images, magics, the Colab stand-in), expectations, budgets, lint, mocks, the generated Modal app, full
+CLI runs (pass, fail, fix, compare, cancel, timeouts, refusals), live sessions on this machine (persistent
+state, interrupts, notebook cells, files, auto-stop, budgets), the guard hook, the MCP server over stdio, and web
+steps in headless Chromium. It passes on Python 3.11 and 3.12.
+
+`tests/test_live.py` runs the same things on the real services and is opt-in because it costs money:
+`TESTBENCH_TEST_MODAL=1` (plans on CPU and a T4, cancelling a GPU run mid-flight, a live session, a T4 session)
+and `TESTBENCH_TEST_COLAB=1` (a plan and a live session on a Colab CPU VM). Colab T4 sessions and plans were also
+run by hand: PyTorch and CuPy on the GPU, a 70-second cell, an interrupted cell whose kernel kept its state, plots
+coming back as images.
+
+[docs/demo.md](https://github.com/sarptandoven/agent-testbench/blob/main/docs/demo.md) shows Claude Code, given only this plugin and a project with a planted bug, finding
+and fixing it and proving the fix with a run.
+
+## Limits
+
+- Costs are estimates (time x published price); `testbench spend --modal` reads Modal's actual bill. Colab
+  unit rates vary by account - put yours (`colab usage`) under `pricing: colab_units_per_hour:`.
+- It runs `.ipynb` files and live kernels; it does not drive the Colab or Modal Notebooks web pages themselves
+  (Modal Notebooks have no API; for steering a Colab tab in your browser see Google's
+  [colab-mcp](https://github.com/googlecolab/colab-mcp)).
+- Colab VMs can be reclaimed after an hour or two even while busy; use Modal for long jobs.
+- macOS and Linux. Windows is untested (the Colab CLI does not support it either).
+
+## License
+
+MIT

agent_testbench-0.1.0/README.md

@@ -0,0 +1,262 @@
+# agent-testbench
+
+[![tests](https://github.com/sarptandoven/agent-testbench/actions/workflows/ci.yml/badge.svg)](https://github.com/sarptandoven/agent-testbench/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/agent-testbench.svg)](https://pypi.org/project/agent-testbench/)
+[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sarptandoven/agent-testbench/blob/main/LICENSE)
+
+**A test bench your coding agent operates.** Claude, GPT/Codex or any agent can start a live kernel on a Modal GPU,
+a Colab VM or your machine, run code and notebook cells in it, inspect variables and plots, install packages,
+move files, fix and rerun - and then prove the result with a clean, checked run. Everything stays inside a
+budget you set.
+
+```
+$ testbench session start gpu --backend modal --gpu T4 --idle-minutes 3 --max-minutes 8
+gpu  ready on modal T4  up 0.0 min, idle 0.0 min (stops after 3 idle / 8 total)  ~$0.000 so far
+Within budget: up to $0.16 (today so far $0.01 of $2.00).
+
+$ testbench exec -f check_gpu.py
+[check_gpu.py | ok | 3.3 s]
+20 matmuls of 4096x4096 on Tesla T4: 0.67 s
+=> torch.Size([4096, 4096])
+
+$ testbench exec "float(b.mean()), torch.cuda.max_memory_allocated() // 2**20"      # variables persist
+[code | ok | 0.2 s]
+=> (1023.1211547851562, 200)
+
+$ testbench session stop
+gpu stopped after 1.4 min, about $0.017
+```
+
+## What your agent can do with it
+
+- **Work live** in a kernel that stays up - on this machine, in a [Modal](https://modal.com) Sandbox with any
+  GPU (the machinery behind Modal Notebooks), or on a [Colab](https://colab.research.google.com) VM through
+  Google's Colab CLI. Run code, `.py` files, or a notebook's cells with its Colab form fields set; get back what
+  it printed, the last value, plots as image files, and errors with the line that raised them. Install
+  packages, run shell commands (`nvidia-smi`), list, upload and download files, and push local edits with
+  `sync` without losing variables.
+- **Prove with plans**: run notebooks (Colab ones included), scripts or web UIs end to end on any of the same
+  backends, checked against expectations - metric thresholds, files, video frame counts, text on a page - with
+  a report that names the failing cell and line, shows the evidence, and says what changed since the last run.
+- **Stay inside a budget**: every session and run is priced before it starts (worst case = its time limit on
+  its hardware), refused above your cap, sent to you for approval above your threshold, and stopped by the
+  provider at its limit even if your laptop goes to sleep. Sessions stop themselves when idle.
+- **Stay safe**: secrets are referred to by name only, paid APIs get strict mocks in tests, and the Claude Code
+  hook blocks raw cloud launches and commands that would print secrets.
+
+## Install
+
+```bash
+pip install "agent-testbench[all]"     # or: uv tool install "agent-testbench[all]"
+```
+
+The latest from GitHub: `pip install "agent-testbench[all] @ git+https://github.com/sarptandoven/agent-testbench"`.
+
+- **Modal**: `modal setup` once (your account, your card; [pricing](https://modal.com/pricing)).
+- **Colab**: `uv tool install google-colab-cli`, then log in once (`colab new` opens the sign-in; or set
+  `colab: {auth: adc}` if you use gcloud Application Default Credentials). Uses your Colab compute units.
+- **Web UI checks**: `playwright install chromium`.
+
+`testbench doctor` shows what this machine can do. Then, in your project:
+
+```bash
+testbench init          # writes testbench.yaml (budget, session defaults, a plan per notebook) and AGENTS.md
+```
+
+## Connect your agent
+
+**Claude Code** - the plugin adds the skills and the guard hook:
+
+```bash
+claude plugin marketplace add sarptandoven/agent-testbench
+claude plugin install agent-testbench@agent-testbench
+```
+
+Then ask in plain words ("get training working on a T4 and prove it with the smoke plan"), or call the skills:
+`/agent-testbench:iterate` and `/agent-testbench:setup`.
+
+**Claude Desktop, Cursor, Codex and other MCP clients**:
+
+```json
+{ "mcpServers": { "agent-testbench": { "command": "testbench", "args": ["mcp"] } } }
+```
+
+Tools: `session_start`, `session_exec`, `session_files`, `session_sync`, `session_install`, `session_shell`,
+`session_list`, `session_stop`, `plans`, `run_plan`, `wait_run`, `get_report`, `list_runs`, `cancel_run`,
+`spend`, `lint_notebook`, `add_note`. In Claude Code: `claude mcp add agent-testbench -- testbench mcp`.
+
+**Any other agent** reads the rules `testbench init` writes into `AGENTS.md`.
+
+## Live sessions
+
+```bash
+testbench session start                                     # default from testbench.yaml (local unless set)
+testbench session start big --backend modal --gpu A100-80GB --idle-minutes 10 --max-minutes 90
+testbench session start nb --backend colab --gpu T4
+
+testbench exec "x = load_data(); x.shape"                   # code
+testbench exec -f debug/check.py                            # a local file
+testbench exec --cells train.ipynb:1-4 --params '{"EPOCHS": 1}'   # notebook cells, form fields set
+testbench files get outputs/sample.png                      # ls / get / put on the session's machine
+testbench session sync                                      # copy local edits again; variables stay
+testbench session install "transformers==4.46.3"
+testbench session shell "nvidia-smi"
+testbench session list                                      # uptime, idle time, cost so far
+testbench session stop                                      # or let it stop itself
+```
+
+| backend | where the kernel runs | stops |
+|---|---|---|
+| `local` | this machine, in the project folder | after `idle_minutes` unused or `max_minutes` |
+| `modal` | a Modal Sandbox with your GPU (T4 ... H100, B200), project in `/work/project` and `/content`, a persistent volume at `/testbench` | Modal enforces both limits itself (a running command counts as activity), so it stops even if your laptop is off |
+| `colab` | a Colab VM (CPU, T4, L4, A100, H100 by subscription), project in `/content` | after `idle_minutes` unused or `max_minutes` (Colab may also reclaim VMs after an hour or two) |
+
+A call that runs past `--timeout` is interrupted; the kernel and its variables survive. One command runs at a
+time per session.
+
+**GPU libraries on Modal**: the default image is Debian slim with the NVIDIA driver but no CUDA toolkit.
+PyTorch works as it is (its wheels bring their own CUDA libraries - checked on a T4). Libraries that expect the
+CUDA toolkit on the machine, such as CuPy, fail there with errors like
+`libcurand.so.10: cannot open shared object file` (also checked). Give them the CUDA libraries they need as
+described in their install guide (for CuPy: docs.cupy.dev, "Installation"), and confirm it in a short GPU
+session (`testbench session start --backend modal --gpu T4 --max-minutes 5`) before a long run.
+
+Notebooks see a headless `google.colab` (uploads from files you name, downloads kept,
+secrets from the environment, Drive as a local folder), so Colab notebooks run as they are.
+
+## Plans
+
+```yaml
+project: my-model                 # names the Modal app and volume
+
+budget:
+  max_run_usd: 2.00               # refuse any run or session that could cost more
+  ask_above_usd: 0.50             # above this a person approves (--approve)
+  max_day_usd: 10.00              # everything today together
+
+sessions:
+  backend: modal
+  gpu: A10
+  idle_minutes: 10
+  max_minutes: 60
+
+modal:                            # image for Modal plans and sessions
+  python: "3.11"
+  pip: [torch==2.5.1, scikit-learn==1.5.2]
+  cpu: 4
+  memory_gb: 16
+
+plans:
+  smoke:                          # cheapest proof first
+    backend: local
+    steps:
+      - name: lint
+        run: testbench lint train.ipynb --strict
+        max_minutes: 1
+      - name: train tiny
+        notebook: train.ipynb
+        params: {EPOCHS: 1, SUBSET: 200}     # Colab form fields (NAME = value  #@param)
+        max_minutes: 5
+        expect:
+          json: {outputs/metrics.json: {loss: "< 2.0"}}
+
+  gpu-check:                      # a few minutes on the target GPU before a long run
+    backend: modal
+    gpu: A10
+    steps:
+      - name: train short
+        notebook: train.ipynb
+        params: {EPOCHS: 2}
+        max_minutes: 10           # also the hard stop on Modal's side
+        artifacts: [outputs/*.png]
+        expect:
+          json: {outputs/metrics.json: {accuracy: ">= 0.8"}}
+```
+
+```bash
+testbench plans                   # every plan, its hardware and worst-case cost
+testbench run smoke               # runs it and prints the report (follows up to 9 min; then `testbench wait`)
+testbench report                  # the latest report
+```
+
+A report leads with what failed: the cell, the error, the line that raised it, each expectation that missed and
+what was found instead, then what changed since the previous run of the plan ("Fixed: train", "Broke: eval",
+"accuracy 0.71 -> 0.93"), the evidence files to look at, and GPU use per step (an idle GPU shows as 0%).
+Steps are `notebook:`, `run:` (a shell command) or `web:` (a page driven in headless Chromium: click, fill,
+press, wait for, screenshot). The full reference is in [docs/reference.md](https://github.com/sarptandoven/agent-testbench/blob/main/docs/reference.md).
+
+## Guardrails
+
+- **Budget**: worst case = hardware price x time limit (+ start-up). Over `max_run_usd` or the day's cap:
+  refused. Over `ask_above_usd`: needs `--approve`, which the agent is told never to pass itself - and in Claude
+  Code the hook turns it into a question to you. Every finished run and session is written to a ledger;
+  `testbench spend --modal` also reads Modal's own bill.
+- **Hard stops**: plans and sessions on Modal carry their limits in Modal itself (function and Sandbox
+  timeouts, no retries, 2-second scale-down). Colab sessions and runs are stopped by their supervisor and always
+  released, even when a run fails.
+- **Secrets**: by name only; values come from Modal secrets or your environment. The hook refuses commands that
+  would print them.
+- **Paid APIs**: test with strict mocks (`agent_testbench.mocks.StrictAPI`): a module with the real client's
+  name that checks arguments against the published schema and refuses anything unexpected.
+- **Outside world**: the hook asks before `git push`, `gh repo create` and anything in `guard.ask`.
+
+The hook is a safety net for agents, not a security boundary; the budget checks and remote time limits are
+enforced in code either way.
+
+## Notebooks
+
+- `testbench nb split nb.ipynb cells/` and `testbench nb build cells/ nb.ipynb` - edit cells as plain files.
+- `testbench nb fields nb.ipynb` - the form fields a plan or `--params` can set.
+- `testbench lint nb.ipynb` - big saved outputs, embedded video, secrets without a fallback, `files.upload()`,
+  unpinned installs, the inline matplotlib backend leaking into child processes, broken form fields, hard-coded
+  `/content` paths.
+
+## Commands
+
+| | |
+|---|---|
+| `testbench session start/list/stop/sync/install/shell` | live kernels |
+| `testbench exec` / `testbench files` | work in a session |
+| `testbench plans` / `run` / `wait` / `report` / `runs` / `cancel` | plans |
+| `testbench estimate <plan>` / `spend [--modal]` | money |
+| `testbench lint` / `nb split/build/fields` | notebooks |
+| `testbench note "..."` | the project's lab notebook (`.testbench/NOTES.md`) |
+| `testbench init` / `doctor` / `mcp` | set up, check, serve MCP |
+
+## Examples
+
+- [`examples/tiny-model`](https://github.com/sarptandoven/agent-testbench/tree/main/examples/tiny-model): a Colab-style notebook that trains a small network, with plans
+  for local, Modal CPU, Modal T4 and Colab.
+  [Open in Colab](https://colab.research.google.com/github/sarptandoven/agent-testbench/blob/main/examples/tiny-model/tiny_model.ipynb)
+- [`examples/web-ui`](https://github.com/sarptandoven/agent-testbench/tree/main/examples/web-ui): a page checked in a headless browser - clicks, text, console errors, screenshots.
+
+## How it is tested
+
+`python -m pytest` runs everything that needs no account: config, notebooks executed in real kernels (errors,
+timeouts, images, magics, the Colab stand-in), expectations, budgets, lint, mocks, the generated Modal app, full
+CLI runs (pass, fail, fix, compare, cancel, timeouts, refusals), live sessions on this machine (persistent
+state, interrupts, notebook cells, files, auto-stop, budgets), the guard hook, the MCP server over stdio, and web
+steps in headless Chromium. It passes on Python 3.11 and 3.12.
+
+`tests/test_live.py` runs the same things on the real services and is opt-in because it costs money:
+`TESTBENCH_TEST_MODAL=1` (plans on CPU and a T4, cancelling a GPU run mid-flight, a live session, a T4 session)
+and `TESTBENCH_TEST_COLAB=1` (a plan and a live session on a Colab CPU VM). Colab T4 sessions and plans were also
+run by hand: PyTorch and CuPy on the GPU, a 70-second cell, an interrupted cell whose kernel kept its state, plots
+coming back as images.
+
+[docs/demo.md](https://github.com/sarptandoven/agent-testbench/blob/main/docs/demo.md) shows Claude Code, given only this plugin and a project with a planted bug, finding
+and fixing it and proving the fix with a run.
+
+## Limits
+
+- Costs are estimates (time x published price); `testbench spend --modal` reads Modal's actual bill. Colab
+  unit rates vary by account - put yours (`colab usage`) under `pricing: colab_units_per_hour:`.
+- It runs `.ipynb` files and live kernels; it does not drive the Colab or Modal Notebooks web pages themselves
+  (Modal Notebooks have no API; for steering a Colab tab in your browser see Google's
+  [colab-mcp](https://github.com/googlecolab/colab-mcp)).
+- Colab VMs can be reclaimed after an hour or two even while busy; use Modal for long jobs.
+- macOS and Linux. Windows is untested (the Colab CLI does not support it either).
+
+## License
+
+MIT

agent_testbench-0.1.0/examples/tiny-model/README.md

@@ -0,0 +1,26 @@
+# tiny-model
+
+A Colab-style notebook (`tiny_model.ipynb`, built from `cells/`) that trains a small neural network on
+scikit-learn's bundled digits, saves `outputs/metrics.json` and a confusion-matrix plot, and offers the
+metrics for download. Its form fields (`HIDDEN`, `EPOCHS`, `LEARNING_RATE`, `SEED`) are set per plan.
+
+```bash
+pip install "agent-testbench[all]" scikit-learn==1.5.2 matplotlib==3.9.2
+testbench plans
+testbench run smoke          # lint + quick train here, free, seconds
+testbench run full-modal     # full train in a Modal CPU container (~$0.002)
+testbench run gpu-check      # nvidia-smi + train on a Modal T4 (~$0.013) - the report shows the GPU sat idle
+testbench run colab-cpu      # same notebook on a Colab CPU runtime (needs the Colab CLI)
+```
+
+Or work in it live, the way you would in Colab:
+
+```bash
+testbench session start                                   # or: --backend modal --gpu T4, or --backend colab
+testbench exec --cells tiny_model.ipynb:1-3 --params '{"EPOCHS": 10}'
+testbench exec "model.n_iter_, model.score(X_test, y_test)"
+testbench session stop
+```
+
+Try breaking it (`params: {HIDDEN: 0}` in a plan, or `LEARNING_RATE = 1.0` in `cells/02_settings.py` and
+`testbench nb build cells tiny_model.ipynb`) and read what the report says.

agent_testbench-0.1.0/examples/web-ui/README.md

@@ -0,0 +1,7 @@
+# web-ui
+
+A one-page counter checked the way a person would: `testbench run ui-check` serves `site/`, opens it in headless
+Chromium, clicks "Add one" twice, takes screenshots, and checks the page says `Count: 2` with no console errors.
+Needs `pip install "agent-testbench[web]"` and `playwright install chromium`.
+
+Break `site/app.js` (call a function that does not exist) and the report names the page error.