ctrlrelay 0.23.0__tar.gz → 0.25.0__tar.gz

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/CHANGELOG.md

@@ -7,6 +7,102 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.25.0] - 2026-11-06
+
+### Fixed
+
+- A sandboxed run (`agent.sandbox: bwrap`) failed at once with "Argument
+  list too long" on a machine with thousands of past agent project folders
+  (#362). The sandbox mounted each `~/.claude/projects/*/memory` on its own,
+  which ran past both the kernel's argument limit and bwrap's 9000 arguments.
+  It now mounts `~/.claude/projects` read-only once, opens only the run's
+  own transcript folder for writing, and passes its options to bwrap
+  through a file descriptor. A config that lists the old glob in
+  `sandbox_readonly_paths` should drop it.
+
+### Added
+
+- `agent.max_parallel` lets one machine run several agents at once (#210).
+  It is one cap over new issues, resumed answers and the security sweep.
+  Each poll's issues start side by side up to the free slots, in priority
+  order, never two of one repo at once, and the loop polls again while they
+  run. Unset keeps the old one-at-a-time behaviour. The status page's slot
+  count now shows the real number of runs and the cap, or "no cap".
+- A dev run saves each step it finishes (plan, code, publish, verify) in a
+  new `run_steps` table, and a run whose process died is continued after
+  its last saved step instead of starting over (#231). The branch must
+  still be at the saved commit. The agent is not run again for a saved
+  code step, and a saved pull request is reused rather than opened twice.
+  Proof and review always run again. `ctrlrelay run dev --from-step
+  plan|code|publish` starts a new run from a past step, keeping the old
+  branch head under `refs/ctrlrelay/before-retry/`.
+
+## [0.24.0] - 2026-11-05
+
+### Added
+
+- Each poll's new issues run in priority order, oldest first (#356). The
+  batch is sorted by the position of the issue's priority label in
+  `intake.priority_labels` (default `["P0", "P1", "P2", "P3"]`, compared
+  without case; an issue with several runs at its earliest; unlabeled
+  last), then by when the issue was opened, then by number. An issue whose
+  creation time cannot be read runs after the others with its label. `[]`
+  keeps the oldest-first order only. Each poll that found issues logs the
+  order once as `poll.batch.order`. The re-read before each dispatch and
+  the blocker hold are unchanged; the order is per machine and fixed for
+  the batch.
+
+- Cost per run in the numbers (#230). Each 7- and 30-day window's
+  `cost_per_run` (in `/api/status` and the status page's **Numbers** box)
+  now gives the median and total tokens per run that ended (`done` or
+  `failed`), counted as the run budget counts them (#223), and the median
+  and total `total_cost_usd` where the agent reported one. Runs with no
+  usage recorded (from before #223, or on a backend that is not measured)
+  are left out and counted in `runs_without_usage`, never read as 0; a
+  window with no measured run is `null` with the reason in `unavailable`.
+  The weekly chat summary gets a `tokens per run` line, `unknown` when not
+  measured.
+
+- A dev run's PR goes ready only with proof the change works (#221). The
+  agent names a proving test in its DONE checkpoint. The relay runs it on
+  the pushed commit (must pass) and on the base with only the branch's test
+  changes (must fail), in fresh copies, with a time limit, without the
+  GitHub token, and under bwrap with no network when the agent was
+  sandboxed. It also flags deleted, skipped or loosened tests and empty or
+  whitespace-only diffs. A result counts only with evidence the named test
+  ran (pytest's junit report, or the exit code and the test in the output
+  for other runners), and the PR's head must still be the proven commit when
+  it is marked ready. The outcome is `proven`, `failed` or `not proven`;
+  only `proven` (or a docs-only diff) lets the PR go ready. Otherwise the PR
+  stays a draft and the run fails with `PR #N held as a draft`. The PR gets
+  one comment, edited in place, with the test, the command and both results.
+  Off by default: set `repos[].automation.require_proof: true` for a repo
+  whose tests can run offline in a fresh copy. With it off, the agent is not
+  asked for a proof and nothing is posted.
+
+- Each machine's agent sandbox mode is shown (#305). `/api/status` carries
+  `node.sandbox` (`"none"` or `"bwrap"`; `null` for a peer on an older
+  version), each machine's card on the status page says `sandbox off` in the
+  warning colour when the agent runs unconfined, and the poller logs
+  `poller.sandbox_off` once at start with what to do about it.
+- A token budget for each run (#223). `agent.budget.soft_tokens` warns once
+  in chat and the log; `agent.budget.hard_tokens` stops the agent, records the
+  run as failed with the reason `budget: used N tokens, hard limit M`, and
+  says `💸 Over budget` in chat. `agent.budget.rules` sets other limits by
+  issue label, which covers `size:*`; the first match wins. Nothing is limited
+  until a limit is set. The count is input + cache writes + output tokens,
+  read live from the agent's transcript, and covers the whole issue session
+  across resumes and fix rounds. Cache reads are recorded but not counted.
+  A run stopped over budget or as stuck after it opened a PR puts the PR back
+  to draft (or posts a "Do not merge" comment when that fails), starts no
+  merge watcher for it, and names it in chat and the issue comment.
+- Each session records the tokens it used (`sessions.usage_tokens` and
+  `sessions.usage_detail`, with the cost the agent CLI reported). Running
+  cards on the status page and `/api/status` show `tokens_used`,
+  `cache_read_tokens`, `cost_usd`, `token_soft_limit` and `token_hard_limit`.
+  A reading that could not be parsed shows as unknown (`null`), never as 0,
+  and never stops a run on its own (#223).
+
 ## [0.23.0] - 2026-11-05
 
 ### Added
@@ -2525,7 +2621,9 @@ pipeline).
   per-phase implementation plans (Phase 0 through Phase 4).
 - `docs/Claude_Code_Project_Guide.md` — project development guide.
 
-[Unreleased]: https://github.com/AInvirion/ctrlrelay/compare/v0.23.0...HEAD
+[Unreleased]: https://github.com/AInvirion/ctrlrelay/compare/v0.25.0...HEAD
+[0.25.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.24.0...v0.25.0
+[0.24.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.23.0...v0.24.0
 [0.23.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.22.0...v0.23.0
 [0.22.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.21.0...v0.22.0
 [0.21.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.20.2...v0.21.0

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ctrlrelay
-Version: 0.23.0
+Version: 0.25.0
 Summary: Local-first orchestrator for headless coding agents across multiple GitHub repos
 Project-URL: Homepage, https://github.com/AInvirion/ctrlrelay
 Project-URL: Documentation, https://ainvirion.github.io/ctrlrelay/

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/docs/architecture.md

@@ -140,6 +140,14 @@ to confirm the PR is mergeable and CI is green. If not, it builds a fix prompt
 fix"), resumes the same agent session with that prompt, and re-verifies. Up
 to `DEFAULT_MAX_FIX_ATTEMPTS` (3) attempts.
 
+Then the proof step
+([`src/ctrlrelay/core/proof.py`](https://github.com/AInvirion/ctrlrelay/blob/main/src/ctrlrelay/core/proof.py),
+#221) runs the test the agent named, in fresh copies of the pushed commit
+(must pass) and of the base with only the branch's test changes (must fail),
+and scans the diff for weakened tests, when the repo turns it on. Only a proven run's PR is marked
+ready; any other outcome leaves it a draft and fails the run. See
+[`automation.require_proof`](configuration.md#reposautomationrequire_proof).
+
 `PRWatcher` (in
 [`src/ctrlrelay/core/pr_watcher.py`](https://github.com/AInvirion/ctrlrelay/blob/main/src/ctrlrelay/core/pr_watcher.py))
 polls a PR for merge status; the post-merge handler in

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/docs/cli.md

@@ -192,7 +192,7 @@ without waiting for a real pipeline run.
 ### `run dev`
 
 ```bash
-ctrlrelay run dev -i ISSUE [-r REPO] [-c PATH]
+ctrlrelay run dev -i ISSUE [-r REPO] [-c PATH] [--from-step STEP]
 ```
 
 | Flag | Required | Default | Description |
@@ -200,6 +200,7 @@ ctrlrelay run dev -i ISSUE [-r REPO] [-c PATH]
 | `--issue` / `-i` | yes | — | GitHub issue number. |
 | `--repo` / `-r` | when more than one repo is configured | — | Limit to a single `owner/repo`. |
 | `--config` / `-c` | no | `config/orchestrator.yaml` | Config path. |
+| `--from-step` | no | — | Start again from `plan`, `code` or `publish`, taking the earlier steps from the issue's newest run that saved them and moving the branch back. See [A run whose process died](operations.md#a-run-whose-process-died). |
 
 Acquires the per-repo lock, creates the worktree, spawns the repo's agent
 (Claude Code or OpenCode, per `agent.type` / `repos[].agent_type`) with the

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/docs/configuration.md

@@ -40,6 +40,7 @@ repos:        [ ... ]
 | `transport` | object | **yes** | — | See [transport](#transport). |
 | `dashboard` | object | no | (defaults) | See [dashboard](#dashboard). |
 | `web` | object | no | (defaults) | The read-only status page. See [web](#web). |
+| `intake` | object | no | (defaults) | The order each poll's new issues run in. See [intake](#intake). |
 | `claims` | object | no | (defaults) | Claim timings for several machines on one account. See [claims](#claims). |
 | `repos` | list | no | `[]` | See [repos](#repos). |
 | `schedules` | object | no | (defaults) | See [schedules](#schedules). |
@@ -208,6 +209,7 @@ agent:
 | `type` | string | `"claude"` | Default backend for every repo: `"claude"` or `"opencode"`. See [agent backends](#agent-backends). |
 | `binary` | string | `"claude"` | Path to the `claude` executable. The bare name `"claude"` is auto-resolved at startup using `shutil.which("claude")`, then `~/.local/bin/claude`, `/usr/local/bin/claude`, `/opt/homebrew/bin/claude`. Set an absolute path to skip lookup (useful under launchd/systemd where PATH is minimal). |
 | `default_timeout_seconds` | int | `1800` | Per-session timeout passed to `asyncio.wait_for`. Sessions that exceed this are killed and reported as failed. A session going in circles is stopped before this; see [stuck runs](operations.md#stuck-runs). |
+| `max_parallel` | int (at least 1) | unset | How many agent runs this machine may have going at once, counted over new issues, resumed answers and the security sweep together. With it set, each poll's issues start side by side up to the free slots, in priority order, and never two of one repo at once; the poll loop polls again while they run. Unset keeps the old behaviour: new issues one at a time, and no shared limit. The status page shows `slots busy/max_parallel`, or `slots N (no cap)` when unset. See [running several agents at once](operations.md#running-several-agents-at-once). |
 | `output_format` | string | `"json"` | Not used. The relay always runs `claude -p --output-format stream-json --verbose`, because only that output carries the subscription usage it reads (see [usage](#usage)). Kept so existing configs still load. |
 
 ctrlrelay always invokes claude with `--dangerously-skip-permissions` so the
@@ -217,6 +219,41 @@ modes either need every shell command allow-listed or filter by command text,
 which a wrapper script walks past, so they are not used as a guard. The guard
 is the next section: the agent holds no GitHub credentials.
 
+### agent.budget
+
+A token budget for each run (#223). Both limits unset (the default): no limit,
+and nothing changes. Usage is measured and shown on the status page either
+way. See [run budgets](operations.md#run-budgets) for what is counted.
+
+```yaml
+agent:
+  budget:
+    soft_tokens: 2000000     # warn once in chat and the log
+    hard_tokens: 6000000     # stop the agent
+    rules:                   # first rule naming any of the issue's labels wins
+      - labels: ["size:S", "size:XS"]
+        soft_tokens: 500000
+        hard_tokens: 1500000
+      - labels: ["size:L", "size:XL"]
+        hard_tokens: 15000000   # soft_tokens left out: 2000000 from above
+      - labels: ["no-budget"]
+        soft_tokens: null       # null: no limit for these runs
+        hard_tokens: null
+```
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `soft_tokens` | int or null | `null` | When a run's total first reaches it, chat gets one line and the log gets `budget.soft_limit`. Once per session. |
+| `hard_tokens` | int or null | `null` | When a run's total reaches it, the agent is stopped and the run is recorded as failed with the reason `budget: used N tokens, hard limit M`. Must be at least `soft_tokens`. |
+| `rules` | list | `[]` | Other limits by issue label. Each rule has `labels` (a non-empty list) and optionally `soft_tokens` and `hard_tokens`. Labels match without regard to case. A size is a label (`size:S`), so this covers sizes too. A limit a rule leaves out comes from the top level; one set to `null` is no limit. |
+
+The count is input tokens + cache writes + output tokens; cache reads are
+recorded but not counted. The budget is for the whole issue session, across
+resumes and CI fix rounds. Unknown keys and a hard limit below the soft one
+(after a rule inherits from the top level) are refused when the config loads.
+Only the `claude` backend is measured. Runs with no issue (secops) get the
+top-level limits for each spawn.
+
 ### What the agent can reach (#219)
 
 The dev and task agents run **without GitHub access**. ctrlrelay starts them
@@ -290,6 +327,14 @@ your user. With `sandbox: "none"` a deliberately hostile agent can read
 and use them by hand, and it can write anywhere your user can, including your
 dotfiles and ctrlrelay's install. On macOS the login keychain is reachable
 whatever the environment says. `sandbox: "bwrap"` closes this on Linux.
+macOS has no sandbox yet, so a macOS node should only take issues from
+repositories you trust.
+
+So that an unconfined node is never a surprise (#305), each machine's mode is
+in `/api/status` as `node.sandbox` (`"none"` or `"bwrap"`; `null` for a peer on
+an older version), its card on the status page says `sandbox off` in the
+warning colour when it is `"none"`, and the poller logs `poller.sandbox_off`
+once at start.
 
 **`sandbox: "bwrap"` (#236).** The dev and task agents run under bubblewrap
 in a read-only copy of the machine (the secops agent, which needs your GitHub
@@ -317,9 +362,10 @@ login, does not):
   snapshots, which your own sessions run or write back (the sandboxed run
   gets an empty, writable one of each);
 - read-only again, over those: `sandbox_readonly_paths` (the agent CLI's
-  settings, rules, hooks, plugins, skills, browser host and install, and each
-  project's memory under `~/.claude/projects/*/memory`, which your own
-  sessions load) and, in the repository, `config`, `config.worktree`,
+  settings, rules, hooks, plugins, skills, browser host and install, and the
+  whole `~/.claude/projects` folder with every project's memory, which your
+  own sessions load; only the run's own transcript folder in it is writable)
+  and, in the repository, `config`, `config.worktree`,
   `hooks`, `modules`, the other runs' worktree directories, a linked
   worktree's `commondir` and `gitdir`, and the worktree's `.git` pointer,
   which every later git command there reads. So the agent cannot change git
@@ -394,7 +440,7 @@ then run the check again. On a single-user laptop this is the whole setup.
 | `sandbox` | `"none"` or `"bwrap"` | `"none"` | `"bwrap"` runs the dev and task agents in the read-only sandbox above. |
 | `sandbox_hide_paths` | list of strings | `~/.ssh`, `~/.config/gh`, `~/.git-credentials`, `~/.config/git/credentials`, `~/.netrc`, `~/.local/share/keyrings`, `$XDG_RUNTIME_DIR/bus`, `~/.gnupg`, `~/.aws`, `~/.azure`, `~/.config/gcloud`, `~/.kube`, `~/.docker/config.json`, `~/.npmrc`, `~/.pypirc`, `~/.cargo/credentials`, `~/.cargo/credentials.toml`, `~/.config/doctl`, `~/.password-store`, `~/.1password`, `~/.docker/run`, `~/.docker/desktop`, `~/.config/ctrlrelay`, `$XDG_RUNTIME_DIR`, `/run/user/<uid>`, `/run/docker.sock`, `/var/run/docker.sock`, `~/.claude/shell-snapshots`, `~/.claude/session-env`, `~/.claude/ide`, `~/.claude/file-history` | Made empty under `sandbox: bwrap`. A directory becomes an empty tmpfs, a file or socket becomes `/dev/null`, a missing path is skipped. |
 | `sandbox_writable_paths` | list of strings | `~/.claude/*` | Writable under `sandbox: bwrap`, besides the run's own paths. A last part of `*` opens each entry in the directory now (not links), not the directory itself. |
-| `sandbox_readonly_paths` | list of strings | `~/.claude/settings.json`, `settings.local.json`, `CLAUDE.md`, `agents`, `commands`, `skills`, `hooks`, `plugins`, `output-styles`, `scripts`, `rules`, `chrome`, `local`, `projects/*/memory` (all under `~/.claude`) | Read-only again inside a writable path. One `*` part matches the directories there at the start of each run; a missing path under a writable one is created first (an empty `memory` directory in each project, for one). |
+| `sandbox_readonly_paths` | list of strings | `~/.claude/settings.json`, `settings.local.json`, `CLAUDE.md`, `agents`, `commands`, `skills`, `hooks`, `plugins`, `output-styles`, `scripts`, `rules`, `chrome`, `local`, `projects` (all under `~/.claude`) | Read-only again inside a writable path. One `*` part matches the directories there at the start of each run; a missing path under a writable one is created first. The run's own transcript folder under `projects` is made writable again by the relay. Before 0.25.0 the default had `projects/*/memory`, one mount per project folder; on a home with thousands of them bwrap could not start (#362). A config that still lists that glob has the same problem, so drop it. |
 | `sandbox_cache_dir` | string | `~/.ctrlrelay/agent-cache` | The agent's own package cache under `sandbox: bwrap`. |
 
 ## transport
@@ -545,6 +591,29 @@ web:
 The page has no login and shows repository names, issue titles and the
 questions agents asked. **Never expose it to the internet.**
 
+## intake
+
+The order each poll's new issues run in (#356). A poll gathers one batch of
+new issues and works through it before the next poll; this sets the order of
+that batch.
+
+```yaml
+intake:
+  priority_labels: ["P0", "P1", "P2", "P3"]
+```
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `priority_labels` | list of strings | `["P0", "P1", "P2", "P3"]` | Issues carrying an earlier label in this list run first; labels are compared without case, and an issue carrying several runs at its earliest. Issues carrying none of them run last. Within the same label, the oldest issue runs first (by when it was opened, then by number). An issue whose creation time cannot be read runs after the others with its label. `[]` keeps the oldest-first order only. Each label must be a non-empty string and listed once. |
+
+Each poll that found new issues logs the order it will run them in, as one
+`poll.batch.order` event (`order`: `owner/repo#N` in run order). The order is
+fixed when the poll ends: an urgent issue opened while a batch runs waits for
+the next poll. Each issue is still read again just before it starts, and an
+issue blocked by an open issue is still held until the blocker closes; those
+can skip an issue, never reorder the batch. The order applies to this machine
+only, not across machines.
+
 ## claims
 
 How machines that share one GitHub account keep off each other's issues.
@@ -719,9 +788,11 @@ and ask the operator), or `never` (skip).
 | `include_labels` | `[]` | Issue labels that opt an issue **into** the dev pipeline regardless of who is (or isn't) assigned. See [include_labels](#reposautomationinclude_labels) below. |
 | `require_labels` | `[]` | Issue labels that gate the plain-assignment trigger: when set, assignment alone is no longer enough — the issue must also carry one of these labels. See [require_labels](#reposautomationrequire_labels) below. |
 | `ci_wait_timeout_seconds` | `600` | Unused since #219 (the agent no longer waits on CI; ctrlrelay does). Still accepted so existing configs load. See [ci_wait_timeout_seconds](#reposautomationci_wait_timeout_seconds) below. |
+| `require_proof` | `false` | A dev run's PR goes ready only when the relay itself proved the change: a test that fails on the base and passes on the branch, and no weakened tests. See [require_proof](#reposautomationrequire_proof) below. |
 
-The current secops and dev pipelines read these settings to bias their prompts
-to the agent — they're not enforced by hard-coded checks.
+The current secops and dev pipelines read the policy settings to bias their
+prompts to the agent — they're not enforced by hard-coded checks.
+`require_proof` is the exception: the relay enforces it.
 
 ### repos[].automation.operator_prs
 
@@ -922,13 +993,120 @@ repos:
 - This does not change the overall session timeout (`agent.default_timeout_seconds`,
   default 1800s) — it only bounds the one `ci wait` call.
 
+### repos[].automation.require_proof
+
+Off by default (#221). Green CI says a change did not break what was already
+tested; it does not say the issue was fixed. With `require_proof` the relay
+checks real evidence before it hands a dev run's PR over as ready.
+
+The agent names its proof in its DONE checkpoint:
+
+```json
+"proof": {"command": ["uv", "run", "--offline", "pytest", "tests/test_x.py::test_y"],
+          "tests": ["tests/test_x.py::test_y"]}
+```
+
+After CI is green, the relay runs `command` twice, as an argument list with
+no shell, each time in a fresh copy of committed content (never the agent's
+worktree):
+
+- on the commit it pushed, where it must pass;
+- on the base commit (the merge base with the default branch) with only the
+  branch's changes to test files applied, where it must fail.
+
+Each run has a 10-minute limit and never gets the operator's GitHub token.
+When the agent ran under `agent.sandbox: bwrap`, the proof runs under the
+same kind of bwrap confinement, with no network and only the copy and the
+agent's cache dir writable. When the agent ran with `sandbox: none`, it
+already ran arbitrary code as this user, so the proof runs unconfined too.
+The copies have no untracked files, so the command must set up what it needs
+offline.
+
+A run counts only with evidence that the named tests ran. For a pytest
+command (`pytest`, `uv run pytest`, `python -m pytest`) the relay adds
+`--junitxml` and reads the report: on the branch each named test must have
+passed, on the base each must have failed or errored. pytest exiting 2 to 5
+(a collection error, a usage error, no tests collected, or `uv run` failing
+to start it), no report, or a named test missing from it is no evidence.
+One exception, on the base only: a collection error in the file of a named
+test (a new test file that imports code only the branch has) counts as
+failing before, recorded as "could not import on base". A collection error
+only in other files does not. For any other runner the command must exit 0 on the
+branch and 1 on the base, and each named test (the part after the last
+`::`) must appear in its output.
+
+There are three outcomes:
+
+| Outcome | When |
+|---|---|
+| `proven` | The named tests failed on the base and passed on the branch. |
+| `failed` | The named tests ran on the branch and did not pass; they ran on the base and passed, so they prove nothing; the diff has no real change (empty or whitespace only); or the diff weakens tests. |
+| `not proven` | No proof was named, the proof is malformed or names a test outside the test files, there is no evidence the named tests ran (a setup, dependency or network failure, a usage error, a collection error), the command could not start, it timed out, or the branch moved after the proof ran. |
+
+Only `proven` lets the PR go ready. Two exemptions, recorded as `not
+required`: a diff that only touches docs (`*.md`, `*.rst`, `*.adoc`,
+`docs/**`), and a repo with `require_proof: false`.
+
+**The test weakening scan** reads the diff's test files (`tests/`, `test_*.py`,
+`*_test.py`, `conftest.py`, `*.test.ts`, `*.spec.js`, `__tests__/` and the
+like) and the test runner's config. These are blockers:
+
+- a deleted test file, in any language;
+- a test file renamed to a name the runner does not collect, or moved out
+  of the tests;
+- a removed test function;
+- an added skip, xfail or `.only` marker: decorators, a module-level
+  `pytestmark`, `pytest.importorskip`, unittest skips, and JS `.skip`,
+  `xit`, `it.only`;
+- an added line in runner config that can deselect or skip: `addopts`,
+  `--deselect`, `-k`, `testpaths`, `norecursedirs`, `collect_ignore` (in
+  `pytest.ini`, `pyproject.toml`, `setup.cfg`, `tox.ini`, `conftest.py`),
+  and jest's `testPathIgnorePatterns`, `testMatch`, `testRegex`;
+- a bare `return` added as the first statement of an existing Python test;
+- an assert line removed from a diff hunk that adds no assert.
+
+An assert line removed from a hunk that also adds one is an edit. The scan
+cannot tell a corrected expectation from a loosened one, so it is shown as a
+warning on the PR and does not block. Lines are matched by text, so a test or
+an assert moved to another file is not a hit. Any blocker makes the run
+`failed`, unless the plan's acceptance criteria say the change is intended
+with an item `Test removal intended: <why>`. Python and JS/TS are read; a
+test file in another language makes the scan report `not checked`, never
+`clean`.
+
+If no test can show the change, the plan's acceptance criteria can say `No
+test possible: <why>`. The run is still `not proven` and the PR stays a
+draft for a person to decide; the PR comment says why.
+
+**What the PR shows.** One comment, edited in place on a later run: the
+outcome and why, the proving test, the command, the fail-before and
+pass-after results with a short redacted tail of each, and any scan hits.
+A run that is not proven fails like any required step after the PR exists
+(see [below](#reposteps)): the PR stays a draft, and chat gets `❌ Failed on
+#N: PR #M held as a draft: not proven …`. A PR GitHub would not open as a
+draft gets a comment telling readers not to merge it.
+
+With `require_proof` on, the PR is opened as a draft, as with
+`steps.review: required`. When both are on, the review step marks it ready
+after the proof passed and the review posted. Right before it is marked
+ready, the relay reads the PR's head from GitHub: if it is not the commit the
+proof ran on, or cannot be read, the run is `not proven` and the PR stays a
+draft.
+
+Turn it on only for a repo whose tests can run offline in a fresh copy of
+the commit. With it off (the default) the agent is not asked for a proof,
+the PR is opened as before, and no proof comment is posted.
+
 ### repos[].steps
 
 The dev pipeline does its work in a fixed order: **plan** (an optional
 planning session, off unless asked for), **code** (the agent run and
 any questions it asks), **publish** (the relay pushes the branch and opens the
 pull request), **verify** (wait for CI and mergeability, asking the agent to
-fix what fails, up to `max_fix_attempts` rounds) and **review** (the
+fix what fails, up to `max_fix_attempts` rounds), **proof** (the relay runs
+the agent's proving test before and after; switched by
+[`automation.require_proof`](#reposautomationrequire_proof), not here) and
+**review** (the
 configured reviewer reads the pushed commit and posts its findings). The order
 is not configurable. `steps` says how strict each step is for this repo.
 Only the dev pipeline reads it. The task and secops pipelines have no
@@ -976,7 +1154,10 @@ that can each be checked by a command or by reading the diff. The relay then:
   outside the worktree, so it is never committed;
 - posts the criteria list as one comment on the issue;
 - adds the plan to the code session's prompt under `## Plan`, telling the
-  agent the criteria are what "done" means.
+  agent the criteria are what "done" means;
+- reads two items from the criteria for the proof step: `Test removal
+  intended: <why>` and `No test possible: <why>` (see
+  [require_proof](#reposautomationrequire_proof)).
 
 If the issue is too vague to plan, the planning session asks one question,
 through the same chat loop the code session uses. No code session runs and no
@@ -1047,7 +1228,7 @@ exists.** The ordinary failure path, unchanged:
 - chat gets `❌ Failed on #N: …`, never `✅ PR ready`.
 
 A resumed session (an answer that arrived after the run stopped to ask) goes
-through the same publish and verify steps. It did not run the review before
+through the same publish, verify and proof steps. It did not run the review before
 these switches, and with `review` unset or `on` it still does not; with
 `review: required` it does.
 
@@ -1101,8 +1282,10 @@ schedules:
 Once a week, each node sends one chat message with its own numbers for
 the last 7 days, the same ones its status page shows: runs started, done,
 failed, blocked and stuck, the success rate, the median time to PR and to
-merge, and how many issues the relay finished were reopened. A number the
-node cannot read is written `unknown`, with the reason, never 0.
+merge, how many issues the relay finished were reopened, and the median
+tokens per run (with the cost, when the agent reported one) over the runs
+that ended with their usage recorded. A number the node cannot read is
+written `unknown`, with the reason, never 0.
 
 The daemon checks every five minutes. It sends at the first check after
 `day` at `hour`:00 that finds nothing delivered since then, so a node that

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/docs/operations.md

@@ -304,6 +304,18 @@ Added in #247:
 |---|---|
 | `node` | This machine's own block: `name` (`node_id`); `usage`, the agent subscription's usage as the last run or probe read it: `status`, `read_at`, and for `five_hour` and `seven_day` each `{used (0 to 1), resets_at}`, or `"unknown"` for a window it could not read; the whole `usage` is `"unknown"` before any reading. `pause_at` is `usage.pause_at`, and `paused` is `{reason, until}` while the machine takes no new issues, else `null`. The page shows it in the **Usage** box. `node` is `null`, with a line in `errors`, only if reading it failed outright. |
 
+Added in #223, on every `running` card (see [Run budgets](#run-budgets)):
+
+| Field | What |
+|---|---|
+| `tokens_used` | Tokens the session has used so far, over all its spawns: input + cache writes + output, what the budget counts. `null` before anything was measured, on a backend that is not measured, or when a transcript row could not be read (then `reason` gives the number of rows and the floor). Never `0` for unknown. |
+| `cache_read_tokens` | Cache reads, recorded but not counted. `null` as for `tokens_used`. |
+| `cost_usd` | The sum of `total_cost_usd` the agent CLI reported at the end of each spawn. Information only. `null` until a spawn reported one. |
+| `token_soft_limit`, `token_hard_limit` | The limits applied to this run. `null` when no limit applies or nothing was measured yet. |
+
+The page shows them in a `tokens:` row on each running card; the count is in
+the warning colour once it passes the soft limit.
+
 Added in #237: `node.github_api` is the account's GitHub API budget as this
 machine last read it: `read_at`, and for `core`, `search` and `graphql` each
 `{remaining, limit, resets_at, state}` with `state` one of `ok`, `low`, `out`,
@@ -340,6 +352,12 @@ Added in #307:
 | `node.config_fingerprint` | `{v, repos_hash, profiles, repos, schedules}`, `v` is `2`. `profiles` maps a profile hash to `{automation, steps}`, each a map of field name to a hash of its value; `repos` maps each repo name to its profile hash, so repos with the same configuration share one profile; `repos_hash` is the hash of the sorted, lowercased repo names; `schedules` maps each `schedules` field to a hash. Hashes and repo names only, never a value. `null` if it could not be built. A peer's fingerprint is unknown unless every hash is 16 lowercase hex characters, `repos_hash` is the hash of its own repo names, each profile key is the hash of that profile, and each repo names a profile it carries. A field's hash cannot be checked against its value, which is never sent. |
 | `fleet.peers[].drift` | `{status, reason, only_here, only_there, fields, lines}`. `status` is `same`, `differs` or `unknown` (with `reason`: not read yet, not reachable, the peer sends no fingerprint in this format, or the comparison failed); `fields` maps a field path such as `automation.exclude_labels` to the repos it differs on; `lines` is what the page shows. Leaves out `fleet.expected_drift`. The peer's own fingerprint is not repeated in `fleet.peers[].node`. |
 
+Added in #305:
+
+| Field | What |
+|---|---|
+| `node.sandbox` | This machine's `agent.sandbox`: `"none"` (the agent runs with the user's full access) or `"bwrap"`. In `fleet.peers[].node` it is `null` when the peer is on an older version or sent anything else, which means unknown, never off. The page shows `sandbox off` in the warning colour for `"none"`. |
+
 Added in #230: `numbers` is how this machine's runs went, for the last 7 and
 the last 30 days (`numbers.windows`, each with `days`, `since`/`since_ts`,
 `until`/`until_ts`). The page shows it in the **Numbers** box, with a table
@@ -354,7 +372,7 @@ control plane. Every field is defined in `src/ctrlrelay/web/numbers.py`.
 | `time_to_pr_s`, `time_to_merge_s` | `{median, n}`: seconds from the run's start to its first `pr_opened` (or `merged`) event, over the `n` runs that have one. Events stamped after now are not read. `null` when events cannot be read, or when a run in the window started before the oldest event still kept: its PR event may have been deleted. The daemon deletes events older than `web.history_days` (7 by default), so with the default the 30-day times are `null`. **To get them, set `web.history_days` to 30**; they appear once the events kept reach back past every run in the window (up to 30 days after the change, since deleted events do not come back). |
 | `came_back` | Distinct issues dispatched again in the window after an earlier run on the same issue had ended `done`. The window's `came_back_issues` lists them (`repo`, `issue_number`, `url`; at most 50). |
 | `reopened` | Issues the relay finished (it saw their PR merge and closed them) that were reopened on GitHub in the window, dated by GitHub's `reopened` event. Counted whether or not the relay picks the issue up again; one per finish. The window's `reopened_issues` lists them. The poller notices a reopen only for an issue its search returns (open and assigned to the operator, or wearing a trigger label), with no extra GitHub call unless a finished issue shows up open. Finishes are recorded in the state database from this version on, plus those still in the kept `merged` events when the table is created. `null` on a database the daemon has not migrated yet. A `reopened` event goes to the feed when one is recorded. |
-| `cost_per_run` | Always `null` for now: cost per run is not recorded yet (#223). The window's `unavailable` says this, and why any other field is `null`. |
+| `cost_per_run` | What the runs that ended (`done` or `failed`) in the window used, from the token usage the run budget records on each run (#223, see [Run budgets](#run-budgets)): `{tokens, cost_usd, runs_without_usage}`. `tokens` is `{median, total, n}` over the `n` ended runs whose usage was recorded and read in full, counted as the budget counts (input + cache writes + output; cache reads left out). `cost_usd` is `{median, total, n}` in US dollars over those of them whose agent reported a cost (`total_cost_usd`, information only), or `null` when none did. `runs_without_usage` is how many ended runs are left out: no usage recorded (runs from before #223, or a backend that is not measured, such as `opencode`) or usage that could not be read in full. They are never counted as 0. Blocked and running runs are left out, since their count still grows. `null` when no ended run in the window has its usage measured, or on a database the daemon has not migrated yet; the window's `unavailable` says which, and why any other field is `null`. |
 | `stuck` | Runs the relay stopped as stuck (#224), also counted in `failed`. `null` only on a database the daemon has not migrated yet. |
 
 `numbers` is `null`, with a line in `errors`, when the sessions cannot be read.
@@ -712,6 +730,138 @@ A stopped run is recorded as `failed` with a `stuck_reason` in the
 the reason, and chat gets `🔁 Stuck on #<n>, stopped: <reason>`. The status
 page's numbers count these runs in `stuck`.
 
+## Run budgets
+
+Each run can have a token budget (#223), set in
+[`agent.budget`]({{ '/configuration/' | relative_url }}#agentbudget). There is no
+limit until one is set.
+
+- **What is counted.** Every 30 seconds ctrlrelay adds up the usage in the
+  agent's live transcript, sub-agents included: input tokens, cache writes
+  and output tokens. Cache reads are recorded but not counted. One API
+  message written as several rows is counted once.
+- **Per session, not per spawn.** A dev or task run spawns the agent more
+  than once (the first run, resumes after a question, CI fix rounds). They
+  all count against the same budget, because each spawn starts from what the
+  session row already holds.
+- **Soft limit.** The first time the total passes it, chat gets one line
+  (`💰 #<n> (<repo>) has used <N> tokens, past its soft limit ...`) and the
+  log gets `budget.soft_limit`. Once per session.
+- **Hard limit.** The agent is stopped through the same path as a stuck run.
+  The run is recorded as `failed` with `stuck_reason` set to
+  `budget: used <N> tokens, hard limit <M>`, its summary starts with
+  `Stopped:`, the issue comment gives the reason, and chat gets
+  `💸 Over budget on #<n>, stopped: <reason>`. A checkpoint the agent wrote
+  just before the stop is not used: the work is not handed over as ready.
+  When no pull request exists yet, the unfinished work is not saved to a
+  branch (only a timeout does that). The status page's numbers count the
+  run in `failed`, not in `stuck`.
+- **A stop after the PR was opened** (over budget in a CI fix round, or
+  stuck there): the PR is put back to draft and kept with the work so far,
+  no merge watcher is started for it, and chat and the issue comment say
+  `PR #<n> was kept as a draft with the work so far.` If GitHub refuses the
+  draft, a "Do not merge" comment goes on the PR instead, and the messages
+  say so.
+- **A floor, not an exact bill.** Only API calls that write an assistant
+  row to the transcript are counted; calls the CLI makes on its own without
+  one (for example, short helper calls) are not. The count can be a little
+  under what the subscription was charged.
+- **Unknown is not zero.** A transcript row whose usage cannot be read adds
+  nothing and is counted as unreadable; the log gets `budget.usage_unreadable`
+  and the page shows the total as unknown. What could be read is still a
+  true floor, so a run is stopped only when that floor reaches the hard limit.
+  An unreadable row on its own never stops a run.
+- **What is stored.** `sessions.usage_tokens` is the counted total (a floor
+  when rows were unreadable) and `sessions.usage_detail` holds the breakdown,
+  the unreadable row count, the cost and the limits applied, as JSON.
+  `cost_usd` is what the agent CLI's final `result` event reported
+  (`total_cost_usd`), summed over the session's spawns; it is information
+  only.
+- **Backends.** Only the `claude` backend writes a transcript ctrlrelay can
+  read. A run on `opencode` is not measured; with a limit set, the log gets
+  `dispatcher.budget.unmeasured` and nothing is enforced.
+
+## A run whose process died
+
+A dev run saves each step it finishes: `plan`, `code`, `publish` and
+`verify` (#231). Each is a row in the `run_steps` table of the state DB,
+with the run branch's commit at that moment and what the later steps need
+(the plan, the agent's result, the pull request). No commit is added to the
+branch for it.
+
+When the relay process dies in the middle of a run (out of memory, a
+restart, a power cut), the poller settles that run when it starts again and
+hands the issue back. The next run for the issue then continues after the
+last saved step:
+
+- it checks that the branch is still at the saved commit, and starts from
+  the beginning if not (or stops with the reason, when the dead run had
+  already opened its pull request);
+- it does not run those steps again: no new agent session for a saved code
+  step, no second push or pull request for a saved publish step;
+- it comments on the issue which run it continues and from where;
+- `proof` and `review` always run again, because they read the pull
+  request as it is now.
+
+A dead run that had opened its pull request is handed back too, and only
+that pull request is reused. A run that failed for its own reasons is not
+continued; the next run starts from the beginning, as before.
+
+To start again from a past step by hand, for example to redo the code with
+the same plan:
+
+```bash
+ctrlrelay run dev --issue 42 --repo your-org/your-repo --from-step code
+```
+
+`--from-step` takes `plan`, `code` or `publish`, uses the newest run of the
+issue that saved steps, and moves the branch back to where it was before
+that step. The old head is kept at `refs/ctrlrelay/before-retry/<run id>`
+in the relay's clone. It refuses a branch behind an open pull request, and a
+branch that is on GitHub is synced back to GitHub's head; delete the remote
+branch first in that case.
+
+The log events are `dev.step.saved`, `dev.resume.restored` and
+`dev.resume.head_moved`.
+
+## Running several agents at once
+
+Set `agent.max_parallel` to let one machine run more than one agent (#210):
+
+```yaml
+agent:
+  max_parallel: 3
+```
+
+- **One cap over everything.** New issues from the poll, runs resumed after
+  you answer a question, and the scheduled security sweep all take a slot
+  from the same pool. A run takes its slot before the repo lock, and gives
+  it back when it ends.
+- **New issues side by side.** Polled issues wait in a backlog, in priority
+  order, and start as slots free up. Two issues of the same repo never run
+  together: the second waits, and an issue of another repo behind it goes
+  first. The loop keeps polling on its interval whatever is running, so one
+  slow issue never holds up new ones.
+- **A run holds its slot for its whole life**, including the wait for CI
+  and the fix rounds, and while it waits in the run for your answer. Once
+  it gives up waiting and saves the question, the slot is free again, and
+  the resume takes one when your answer comes. Two pull requests waiting on
+  slow CI can fill `max_parallel: 2`.
+- **A hold placed while a run waited for its slot is respected.** The resume
+  sweeper and the security sweep check again once they have a slot, so a
+  `self-update` that started meanwhile is not raced.
+- **On shutdown** no new run starts in a slot a stopping run frees, and
+  issues polled but not started are offered again after the restart.
+- **Unset** keeps the old behaviour: new issues one at a time, and nothing
+  limits the total.
+
+Each run uses its own agent session, worktree and memory. Size the number to
+the machine and to your plan's usage limits. Start with 2.
+
+The status page shows `slots busy/max`, and `slots N (no cap)` when it is
+unset. The log event `poll.batch.parallel` names each batch, the limit and
+how many slots were busy.
+
 ## When to restart
 
 | You changed... | Restart... |

{ctrlrelay-0.23.0 → ctrlrelay-0.25.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ctrlrelay"
-version = "0.23.0"
+version = "0.25.0"
 description = "Local-first orchestrator for headless coding agents across multiple GitHub repos"
 readme = "README.md"
 requires-python = ">=3.12"