@giamme/forge 0.11.1 → 0.12.0

package/CHANGELOG.md

@@ -5,6 +5,40 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and versions follow the format `{major}.{minor}[.{patch}]` ([...] parts appear only when non-zero).
 
+## [0.12] - 2026-10-27
+
+Decomposed runs can now be steered while they run and stop cleanly when interrupted: drain, stop, retry or resize a live run from forge control or forge-ui, an infrastructure stop pauses only the model or CLI it hit, and verification can run as parallel lanes that rerun only what failed and name the failing tests.
+
+### Highlights
+
+- A running decomposed plan can be steered: `forge control drain` lets running tasks finish and starts nothing new until `forge control resume`; `forge control stop` ends the run once running tasks are done (exit 130 without verifying, unless every task merged; not-started tasks stay pending for the next `run`), and `stop --now` interrupts them; `forge control set <plan> max-parallel N` changes how many tasks run at once.
+- Verification can run as lanes: declare them in `.forge/verify.d/` (one file per lane) or with `run --verify-lane 'name[options]=command'`, and forge runs them side by side (`--verify-parallel`, 3 by default), reruns only a lane that failed, and calls a lane that passed on rerun flaky. Lanes can read JUnit or TAP reports to name the failing tests, and a baseline of known failures keeps those from failing the run. The run summary and the whole-run review name each lane and its state.
+- An infrastructure stop no longer pauses the whole decomposed run: a rate limit pauses only that model and a quota or sign-in stop only that CLI, each with its own wait and --infra-retries budget, while tasks routed elsewhere keep running. `forge control show` lists the active pauses.
+
+### Added
+
+- `run --infra-backoff` sets how long forge waits between infrastructure retries: `exp:<base>[,<cap>]` (the default, `exp:60`), `linear:<start>,<step>[,<cap>]` or a list of waits such as `10,20,30`. `forge control set <plan> infra-backoff` changes it while the run goes on, and each wait message names the shape it used.
+- A running decomposed plan can retry a stopped or failed task, optionally on another model (`forge control retry <plan> <task> [--dwarf <spec>]`), and pick up a row you added to tasks.tsv (`forge control add <plan> <task>`) without ending the run. The scheduler applies or refuses each request at its next pass and records the outcome in control.log.
+- `run --infra-fallback`: when an implementer's model is at capacity, the task moves at once to the next model of its own tier pool instead of waiting, and the move is recorded like an escalation. The busy model is still paused and counted, so a pool that is entirely at capacity still ends the run.
+- forge-ui now shows a verification while it runs: its current step, last output line and idle time, and, for a lanes verification, each lane's state, runs, test counts and failing tests, with known failures marked.
+- forge-ui serves every verification log, each lane's and each rerun's, as a numbered tail or in 400-line pages, so the cause of a failed verification can be read without opening the run dir.
+- The verification panel in forge-ui now follows a running verification live (current step, last line, elapsed time, a warning after five silent minutes, the log's tail) and, when it fails, opens on the failing tests, marked new or known, each linking to its line in the log.
+- forge-ui's Execution view shows verification lanes as their own rows, with every run and rerun at its real time, and the 3D finish node shows one light per lane; a verification that is still running reads RUNNING instead of its previous result.
+- `--escalate-effort` (on `run` and `retry`, or `forge control set <plan> escalate-effort on`): when a task has failed its escalation limit on the top tier's model, forge retries it once more at the next effort that model accepts before stopping it with a split suggestion. Turning it on also releases tasks the policy already stopped.
+- forge-ui's Controls panel can now drain, resume and stop a live run (gracefully or at once), change max parallel, the infrastructure backoff and the effort step, and retry a stopped task now with a chosen model. It also shows each infrastructure pause with a countdown, and whether a queued retry was applied or refused.
+
+### Changed
+
+- The forecast `run` prints before dispatching lists the tasks that don't have enough history yet on one line instead of a row each, and is a single line when no task has enough history, so a new setup no longer opens every run with a table of "not enough history" rows. `forge dataset forecast <plan>` still shows every task.
+
+### Fixed
+
+- When the plan forecast that `run` prints runs out of its time budget while saving, it no longer leaves a stray `.forecast.json.<pid>` file in the plan.
+- Interrupting a decomposed run no longer leaves an implementer or reviewer CLI running: forge waits up to 10 s (FORGE_INTERRUPT_GRACE) for each task to stop its harness, and a harness left behind by a killed run is stopped when the next run starts.
+- Interrupting `forge-parallel.sh run` (Ctrl-C or SIGTERM) no longer goes on to verify a half-merged integration branch: the run stops, frees the plan lock within seconds, exits 130 and prints the command that continues each interrupted task. A signal during verification stops the whole verify command and skips its rerun.
+- A task that finished just as a decomposed run was interrupted is now reaped first: its passing work still merges and its result is kept, and only the tasks still running are marked INTERRUPTED.
+- On macOS, a Ripwire context call whose output was too long or that ran too long no longer fails with a PermissionError; it is skipped with its reason, as intended.
+
 ## [0.11.1] - 2026-10-26
 
 A bug-fix release for forge-ui and forge issues: the Settings and Open findings views keep your edits and selections honest, forge ui open handles a read-only server with another configuration, and refused labels and warnings say what's wrong.

package/SKILL.md

@@ -57,8 +57,10 @@ For explanation requests, explain the workflow without spending model quota.
   overrides a reviewer, so run it (with the user's `--reason` and
   `--approved`) only on the user's explicit instruction; never self-accept.
 - When you report a run, read `<plan>/control.log` and mention any human change made mid-run
-  (a threshold, a budget, a model pin; `forge control show <plan>` lists them). Never set a
-  control, pin or delete a run with `forge control` / `forge runs rm` unless the user asks.
+  (a threshold, a budget, a model pin, a drain, stop or queued retry; `forge control show <plan>`
+  lists them). Never set a control, pin, `drain|stop|retry` or delete a run with `forge control` /
+  `forge runs rm` unless the user asks. Exit `130` / `INTERRUPTED`: the run was signalled or
+  stopped; relay the `retry` lines it printed.
 - Exit `8` / status `INFRA` is a quota, auth or network stop, not a task failure and no
   attempt was spent: report it, say what to fix, and re-run only when told. For a multi-task
   plan, offer the opt-in whole-run review (`review`, or `integrate --final-review <qa-spec>`);

package/VERSION

@@ -1 +1 @@
-0.11.1
+0.12

package/docs/artifacts.md

@@ -15,7 +15,7 @@ as if the dwarf had written it.
   <role>.cmd          the exact command line executed
   <role>.log          full transcript
   <role>.last         the final message — read this first
-  <role>.infra        only after an infrastructure failure: class, retry_after, detail
+  <role>.infra        only after an infrastructure failure: class, retry_after, detail, harness, model
   <role>.orphans      only when processes were left running after the harness exited
   <role>.session      session id reported by claude, openclaude or codex; absent when the dispatch published none
   <role>.continue     only when the dispatch was asked to continue a reviewer session: requested=, mode=continued|failed, reason=
@@ -49,9 +49,10 @@ A decomposed run adds:
   tasks/<id>/attempt, attempts.tsv     attempts spent; one audit row per entry (incl. refunded stops)
   tasks/<id>/known_issues.md, qa.gate  findings tolerated by --qa-threshold, and the gate's decision
   tasks/<id>/guard.txt                 what forge observed about the implementer (self-commit, orphans, promises)
-  tasks/<id>/infra.txt, resume, noretry  infrastructure stop record; QA-only resume marker; no-progress marker
+  tasks/<id>/infra.txt, resume, noretry  infrastructure stop record (<role>.infra plus stage=); QA-only resume marker; no-progress marker
+  tasks/<id>/<role>.harness            only while a guarded dispatch runs: its harness's pid and start time, for forge-guard.py stop
   tasks/<id>/qa_threshold              optional per-task blocking threshold: P0-P3, or none (exempt from the plan's)
-  tasks/<id>/escalations.tsv           epoch, attempt, from-spec, to-spec, source (policy|retry|pin); older rows have 4 columns
+  tasks/<id>/escalations.tsv           epoch, attempt, from-spec, to-spec, source (policy|retry|pin|fallback); older rows have 4 columns
   tasks/<id>/escalation.pin            a human's one-shot model for the next full retry: one dwarf spec on one line
   tasks/<id>/escalation.pin.applying   the pin while a retry has claimed it; removed once the retry is recorded
   tasks/<id>/retry_findings.commit     commit the carried review judged
@@ -64,18 +65,40 @@ A decomposed run adds:
   tasks/<id>/retry_budget, retries.used   live per-task retry budget (a number); retries spent (kept after the run)
   results.tsv                          id  status  branch
   timeout, qa_threshold, verify_retries  options remembered from run/retry flags (also escalate_after)
+  escalate_effort                      on: the retry policy's effort step (--escalate-effort; absent = off)
+  infra_backoff, infra_fallback        --infra-backoff's shape; on while --infra-fallback is set
+  max_parallel                         the concurrency limit: written by each run from --max-parallel, changed live by forge control
   reviewer_session                     holds on; present while plan --reviewer-session is on
   <final-review-dir>/review.meta       final-review-<epoch>-<pid>/ or integrate-*/final-review/; base=, tip=, source=, spec=, threshold=, integration=, previous=, since=; session= and session_reason= while reviewer_session is set
-  control.log                          one JSON line per human change made with forge control
+  control.log                          one JSON line per human change made with forge control, and per request outcome
+  control.requests/                    retry and add requests queued by forge control: one JSON file each, removed once settled
   scheduler.live                       only while a scheduler runs: {"pid", "started"}
+  drain, stop                          only while a scheduler runs: forge control drain (empty) and stop (graceful|now)
+  infra.paused                         only while something is paused on infrastructure: {"scopes": [...]}
   control.lock                         the lock that serialises changes to the plan's control files, created by whichever takes it first (forge control, the scheduler or an escalation record) and left in place
   retry_budget, infra_budget           only while a scheduler runs: its live budgets (a number each)
   integration.base                     commit the integration branch started from
-  verification.{status,exit,log,command,fingerprint}   the combined verification
-  verification.{flaky,retry.log}       only when a failing command passed on rerun
+  verification.{status,exit,log,command,fingerprint}   the combined verification; status PASS, FAIL or INTERRUPTED (a signal stopped it)
+  verification.{flaky,retry.log}       only when a failing command passed on rerun (lanes: flaky alone, per lane)
+  verification.commit                  the integration commit that was verified
+  verification.runs.tsv                one row per run of the verify command or of a lane (below)
+  verification.run-<n>.log             single command: each run's own copy of its log
+  verification.lanes.json              lanes only: every lane's state, runs, step and tests, rewritten as they change
+  verification.lane.<name>.log         lanes only: the lane's run in progress
+  verification.lane.<name>.run-<k>.log lanes only: each finished run of the lane
   verification.coverage.txt            suites the verify command does not run (advisory)
+  verify_cmd, verify.d/                the plan's --verify command, or its --verify-lane lanes (one file per lane)
+  verify_parallel                      lanes run at once (--verify-parallel; default 3)
 ```
 
+`verification.runs.tsv` is tab-separated, one row per run: `n`, `kind` (`initial`, `rerun`, or
+`jev_retry` for a single command), `exit`, `tree_changed` (`0`/`1`), `duration_s` and `log`. With
+lanes each row has two more columns, `lane` and `started` (epoch seconds); lanes run side by
+side, so `tree_changed` is `1` only when the whole verification was a single lane run that
+changed the tree (`verification.lanes.json`'s `mutated` says whether it changed at all). In lanes
+mode `verification.flaky` holds `lanes=<names>` and, per flaky lane, `<lane>.first_exit=` and
+`<lane>.retries_used=`.
+
 with worktrees and branches at:
 
 ```
@@ -86,11 +109,21 @@ forge/<run-id>-integration                       integration branch
 
 `control.log` lines look like
 `{"at": 1761200000.5, "actor": "ui", "action": "set", "task": null, "key": "qa-threshold", "value": "P2", "previous": "P1"}`
-(`actor` is `cli` or `ui`; `action` is `set`, `pin` or `unpin`; `key` is the control's name, or `pin`; `task` is null for a plan-wide
-control). `scheduler.live`, `retry_budget`, `infra_budget` and the per-task `retry_budget` exist
-only while the scheduler is live: it publishes the flags it was given when it starts, re-reads them
-at every decision and removes them when it stops, which is why `forge control` refuses a budget
-change outside a live run. See [live controls](decomposed-runs.md#changing-a-run-while-it-works).
+(`actor` is `cli` or `ui`; `action` is `set`, `pin`, `unpin`, `drain`, `resume`, `stop`, `retry` or
+`add`; `key` is the control's name, or `pin`; `task` is null for a plan-wide control). A queued
+`retry` or `add` also carries its `request` id, and the scheduler later appends the outcome under
+the same id:
+`{"at": 1761200030.1, "actor": "scheduler", "action": "retry", "task": "docs", "key": null, "value": "sol:high", "previous": null, "request": "…", "outcome": "refused", "reason": "the task is running"}`
+(`outcome` is `applied` or `refused`). `scheduler.live`, `retry_budget`, `infra_budget`, the
+per-task `retry_budget`, `drain`, `stop` and `infra.paused` exist only while the scheduler is live:
+it publishes the flags it was given when it starts, re-reads them at every decision and removes
+them when it stops, which is why `forge control` refuses a budget change, a drain or a stop outside
+a live run. A request still queued then is refused (`run ended`, or `left by an earlier run` at the
+next start). See [live controls](decomposed-runs.md#changing-a-run-while-it-works).
+
+`infra.paused` holds one entry per paused scope: `scope` (`all`, `harness:<h>` or
+`model:<h>/<m>`), `class`, `harness`, `model`, the `tasks` it holds, `cycle` (which wait this is),
+`until` (epoch seconds, null while not waiting) and `exhausted` (its waits are used up).
 
 The worktree root is a **sibling of the repo, never `TMPDIR`**. Isolated worktrees are never
 pushed, so a temp root that gets cleaned takes the only copy of that work with it.

package/docs/commands.md

@@ -16,7 +16,7 @@ harness (`/forge`); this command installs forge and drives its optional tools.
 | `forge version [--check]` | the installed version, and whether a newer release is out |
 | `forge fractal …` | opt-in Fractal execution; see [the command group](fractal.md#forge-fractal-command-group) |
 | `forge runs [--repo DIR \| --all] [--limit N] [--json]` | this repo's recent runs, newest first: run id, kind, age, task states, verification and final review, goal, and the run dir. Read-only; the same summaries forge-ui shows |
-| `forge control show\|set\|pin\|unpin PLAN_DIR …` | change a live plan: `set` takes `qa-threshold`, `timeout`, `escalate-after`, `verify-retries`, `retry-budget` or `infra-retries` (the last two only while a scheduler runs; `--task ID` for `qa-threshold` and `retry-budget`, value `inherit` drops the override); `pin PLAN_DIR TASK SPEC` sets the model the task's next full retry uses once, `unpin` drops it; `--json`. Every change is logged in `control.log`. See [live controls](decomposed-runs.md#changing-a-run-while-it-works) |
+| `forge control show\|set\|pin\|unpin\|drain\|resume\|stop\|retry\|add PLAN_DIR …` | change a live plan: `set` takes `qa-threshold`, `timeout`, `escalate-after`, `escalate-effort` (`on`/`off`), `verify-retries`, `infra-backoff` (a backoff shape; `none` drops it), `retry-budget`, `infra-retries` or `max-parallel` (1-64) (the last three only while a scheduler runs; `--task ID` for `qa-threshold` and `retry-budget`, value `inherit` drops the override); `pin PLAN_DIR TASK SPEC` sets the model the task's next full retry uses once, `unpin` drops it. While a scheduler runs: `drain` starts nothing new until `resume` (a run with nothing left to start ends as usual); `stop [--now]` ends the run once the running tasks finish (`--now` interrupts them; `run` exits 130 without verifying, unless every task had merged by then: it then verifies and exits 0 as usual); `retry PLAN_DIR TASK [--dwarf SPEC]` and `add PLAN_DIR TASK` (a row appended to `tasks.tsv`) queue a request the scheduler applies or refuses. `show` (`--json` for JSON) also says whether the run is drained or stopping and lists its infrastructure pauses. Every change and every request's outcome is logged in `control.log`. Exit 2 invalid input, 3 refused. See [live controls](decomposed-runs.md#changing-a-run-while-it-works) |
 | `forge config show [--repo DIR] [--json]` / `forge config set <key> <value> --repo DIR [--json]` / `forge config unset <key> --repo DIR [--json]` | one view of every config source (registry, Jev, the repo's `.forge/verify`, `setup` and `verify-subset`, dataset, `FORGE_*` variables with secrets hidden); `set KEY VALUE` / `unset KEY` edit `repo.verify`, `repo.setup`, `repo.verify-subset` (must contain `{files}`) and, for `set`, `jev.enabled true\|false` (that one does not need `--repo`). `-` as a value reads stdin. Exit 2 on invalid input |
 | `forge runs rm RUN_DIR [--dry-run] [--yes] [--no-archive] [--worktrees] [--branches] [--json]` | delete a run directory but keep what it taught the dataset; see [below](#forge-runs-rm) |
 | `forge issues RUN [--apply] …` | file the review findings a run left open as GitHub issues; see [below](#forge-issues) |
@@ -191,8 +191,13 @@ forge-dispatch.sh dwarf   <spec> --prompt-file <f> [--repo <dir>] [--run-dir <d>
 forge-dispatch.sh qa      <spec> --prompt-file <f> [--repo <dir>] [--run-dir <d>] [--yolo] [--dry-run]
                                  [--native-review [--review-base <ref>]]
 forge-dispatch.sh planner <spec> --prompt-file <f> [--repo <dir>] [--run-dir <d>] [--dry-run]
+forge-dispatch.sh efforts --spec <spec>
 ```
 
+`efforts` prints the efforts the spec's pairing accepts, weakest first, one per line (nothing
+when the pairing has no effort dial); a literal model id with a harness takes that harness's
+whole ladder. It never dispatches and spends no quota. The retry policy's effort step reads it.
+
 | Option | Meaning |
 |---|---|
 | `--prompt-file <f>` | file holding the prompt (required for `dwarf`/`qa`) |
@@ -210,7 +215,7 @@ forge-dispatch.sh planner <spec> --prompt-file <f> [--repo <dir>] [--run-dir <d>
 `8` an **infrastructure failure** (quota or usage limit, auth, rate limit, network, or an
 empty answer) — nothing about the work was judged, so callers pause instead of counting a
 failure. On `8`, `<run-dir>/<role>.infra` holds `class=`, `retry_after=`, `detail=`, `rc=`,
-`role=`.
+`role=`, `harness=`, `model=`.
 
 **Timeout precedence.** An explicit `--timeout`, then `FORGE_TIMEOUT` (an explicit `0` wins
 and disables the watchdog), then the optional fifth `timeout` column of the model's
@@ -328,15 +333,21 @@ forge-parallel.sh plan      <plan-dir> --repo <dir> [routing flags] [--planner <
                                      [--reviewer-session | --no-reviewer-session]
                                      [--explore E] [--ui | --no-ui] [--fail-first | --no-fail-first]
 forge-parallel.sh run       <plan-dir> [--max-parallel N] [--yolo-dwarf] [--yolo-qa] [--dry-run]
-                                     [--verify <cmd>] [--setup <cmd>] [--timeout S]
-                                     [--retry-failed N] [--infra-retries M]
+                                     [--verify <cmd> | --verify-lane 'name[opts]=cmd' ...]
+                                     [--verify-parallel N] [--setup <cmd>] [--timeout S]
+                                     [--retry-failed N] [--infra-retries M] [--infra-backoff SHAPE]
+                                     [--infra-fallback | --no-infra-fallback]
                                      [--qa-threshold P0|P1|P2|P3|none] [--verify-retries N]
+                                     [--escalate-after N] [--escalate-effort | --no-escalate-effort]
                                      [--fractal | --no-fractal] [--fractal-<limit> <n>]
 forge-parallel.sh retry     <plan-dir> <task-id> [--dwarf <spec>] [--timeout S]
                                      [--qa-threshold Pn|none] [--verify-retries N]
+                                     [--verify <cmd> | --verify-lane 'name[opts]=cmd' ...] [--verify-parallel N]
+                                     [--escalate-after N] [--escalate-effort | --no-escalate-effort]
 forge-parallel.sh accept    <plan-dir> <task-id> --reason "<why>" --approved
 forge-parallel.sh review    <plan-dir> [--qa <spec>] [--yolo-qa] [--qa-threshold Pn]
 forge-parallel.sh integrate <plan-dir> --approved [--final-review <qa-spec>] [--verify-retries N]
+                                     [--verify-parallel N]
 forge-parallel.sh split     <plan-dir> <new-plan-dir> --tasks <id,id> [--dry-run]
 forge-parallel.sh combine   <new-plan-dir> <plan-dir> <plan-dir>... [--verify <cmd>] [--dry-run]
 ```
@@ -355,17 +366,44 @@ forge-parallel.sh combine   <new-plan-dir> <plan-dir> <plan-dir>... [--verify <c
 |---|---|---|---|
 | `--timeout S` | run, retry | `timeout` | per-dispatch timeout in seconds (`0` disables); wins over `FORGE_TIMEOUT` |
 | `--retry-failed N` | run | no — this run only | re-queue a task the reviewer rejected up to `N` more times inside the run |
-| `--infra-retries M` | run | no — this run only | wait out up to `M` infrastructure stops instead of pausing |
+| `--infra-retries M` | run | no — this run only | wait out up to `M` infrastructure stops per paused scope (a model, a harness, or everything) instead of ending the run |
+| `--infra-backoff SHAPE` | run | `infra_backoff` | the waits between those stops when the provider gave no retry-after: `exp:B[,C]`, `linear:S,T[,C]` or `list:a,b,...` in seconds (default `exp:60`) |
+| `--infra-fallback` | run | `infra_fallback` | a dwarf stopped by a rate limit moves to the next healthy model of its tier pool instead of waiting (off by default; `--no-infra-fallback` turns it off) |
 | `--qa-threshold Pn` | run, retry | `qa_threshold` | least severe finding that still blocks; milder ones become known issues; `none` clears |
-| `--verify-retries N` | run, retry, integrate | `verify_retries` | reruns of a failed verify command (default `1`, `0` = strict single run) |
+| `--verify-retries N` | run, retry, integrate | `verify_retries` | reruns of a failed verify command, or of each failed lane (default `1`, `0` = strict single run) |
+| `--verify-lane 'name[opts]=cmd'` | run, retry | `verify.d/<name>` | one verification lane; repeatable. Options as in a lane file, comma-separated (`group=g`, `first`, `retries=n`, `reports=dir`, `baseline=file`). The first one of an invocation replaces the plan's earlier lanes and `--verify`; see [verification lanes](../references/decompose.md#execution-checks-and-integration-verification) |
+| `--verify-parallel N` | run, retry, integrate | `verify_parallel` | lanes run at once, 1 to 32 (default `3`) |
+| `--escalate-after N` | run, retry | `escalate_after` | failed attempts on one model before the next goes a tier up (default `3`, `0` = off) |
+| `--escalate-effort` | run, retry | `escalate_effort` | at the top tier, retry the same model once more at its next accepted effort before stopping (off by default; `--no-escalate-effort` turns it off) |
 
 **Exit codes:** `0` ok · `2` usage or validation · `3` precondition (not a git repo, …) ·
 `5` some task failed QA, or the combined verification failed · `6` integration conflict ·
 `8` **paused on an infrastructure failure** (quota, auth, rate limit, network): no attempt was
-spent and nothing was judged — fix the cause and run the same command again.
+spent and nothing was judged — fix the cause and run the same command again ·
+`130` **interrupted or stopped** (a signal, or `forge control stop`): the running tasks are
+`INTERRUPTED` and verification was skipped or stopped; the summary prints the `retry` for each.
 
 The full plan → run → integrate walkthrough is in [decomposed runs](decomposed-runs.md#plan-run-integrate). `retry`, `accept` and `review` are covered in [reviews, retries and recovery](reviews.md); resuming, `split`/`combine`, scope drift and `tasks.tsv` in [decomposed runs](decomposed-runs.md).
 
+## `scripts/forge-verify.py`
+
+The verification lane runner `run`, `retry` and `integrate` use when lanes are declared. You can
+run it by hand to check or try a lanes directory:
+
+```
+forge-verify.py lanes --lanes-dir DIR [--checkout DIR] [--json]
+forge-verify.py run   --checkout DIR --out DIR --lanes-dir DIR [--parallel N] [--retries N]
+                      [--fingerprint-cmd CMD]
+```
+
+`lanes` lists the lanes with their options and checks them (with `--checkout`, also that
+`reports=` and `baseline=` stay inside it and the baseline exists). `run` runs them in the
+checkout as a forge run does and writes `verification.lanes.json`, `verification.runs.tsv` and
+the per-lane logs into `--out` (`--parallel` default 3, `--retries` default 1).
+
+**Exit codes:** `0` every lane passed and the tree is unchanged · `1` a lane failed, or the tree
+changed · `2` configuration error (nothing ran) · `130` interrupted.
+
 ## `scripts/forge-memory.sh` and `scripts/forge-install.sh`
 
 See [project memory](memory.md#scriptsforge-memorysh) and [install](install.md#scriptsforge-installsh).

package/docs/decomposed-runs.md

@@ -193,6 +193,14 @@ all their child invocations and QA. See [decomposed Fractal execution](../refere
 
 ## Resuming an interrupted run
 
+Ctrl-C (or a SIGTERM) on `run` stops the running tasks' process groups, waiting up to
+`FORGE_INTERRUPT_GRACE` seconds (10 by default) before killing what is left, and stops any
+harness that outlived its dispatch. A task that had already finished is reaped first (a PASS
+still merges); only the tasks still running become `INTERRUPTED`, with their partial work kept
+in the worktree. Verification is skipped, or stopped if it had started (`verification.status`
+`INTERRUPTED`, not rerun). The summary lists each interrupted task with its `retry` command,
+the plan lock is released, and `run` exits 130.
+
 Run `run` again to skip MERGED tasks, merge saved PASS results after fingerprint checks,
 and dispatch eligible pending tasks. Tasks paused on an infrastructure failure (`INFRA`) are
 dispatched again without any flag, since nothing was wrong with the work. Failed and
@@ -200,6 +208,9 @@ interrupted tasks need an explicit `retry` (or `run --retry-failed N` for reject
 Their worktrees and pinned baselines remain available. Concurrent run/retry/integrate
 operations for the same plan are excluded by an advisory lock.
 
+To pause a run on purpose, drain it instead of signalling it (see below): nothing is
+interrupted, and `forge control resume` continues the same run.
+
 For a Fractal-backed run, use `forge fractal resume RUN_ID` to recover recorded execution
 and pipeline checkpoints. A task/subtree selector resumes only that execution scope.
 Explicit `retry` retains the backend and frozen pools; a replacement dwarf must already
@@ -213,12 +224,35 @@ belong to that pool. See [recovery](../references/fractal.md#limits-and-lifecycl
 forge control show <plan-dir> [--json]                    # every control, per-task overrides, pins, recent changes
 forge control set <plan-dir> qa-threshold P2              # also --task <id>; P0-P3, none, or inherit (task only)
 forge control set <plan-dir> retry-budget 3 [--task <id>] # automatic retries; live scheduler only
-forge control set <plan-dir> infra-retries 2              # infrastructure waits; live scheduler only
+forge control set <plan-dir> infra-retries 2              # infrastructure waits per scope; live scheduler only
+forge control set <plan-dir> max-parallel 2               # tasks at once, 1-64; live scheduler only
+forge control set <plan-dir> infra-backoff linear:60,60   # the next infrastructure wait; none = exp:60
+forge control set <plan-dir> escalate-effort on           # the effort step before a policy stop
 forge control set <plan-dir> timeout|escalate-after|verify-retries <n>
 forge control pin <plan-dir> <task> sol:high:codex        # the model the next full retry uses, once
 forge control unpin <plan-dir> <task>
 ```
 
+And to steer the scheduler itself (each needs a live run, exit 3 otherwise):
+
+```bash
+forge control drain <plan-dir>                    # start nothing new; running tasks finish, the run waits for resume
+forge control resume <plan-dir>                   # end the drain
+forge control stop <plan-dir>                     # finish what runs, start nothing, end the run (exit 130)
+forge control stop <plan-dir> --now               # interrupt the running tasks too, like Ctrl-C
+forge control retry <plan-dir> docs --dwarf sol:high   # queue a retry of a stopped or failed task
+forge control add <plan-dir> docs-fix             # queue a row you appended to tasks.tsv (with its prompt.md)
+```
+
+A stop skips verification and exits 130, unless every task had merged by then, and a drain
+with nothing left to start lets the run end as usual: both act on work still to do.
+`retry` and `add` only queue a request in `control.requests/`; the scheduler applies or refuses
+it at its next pass and logs the outcome in `control.log` under the same request id (`forge
+control show` lists the log). A retry is prepared exactly as `forge-parallel.sh retry` prepares one,
+so a task the policy stopped needs `--dwarf`; it spends no automatic retry budget. An added row
+may depend only on tasks the run already knows. Neither outlives the run that was live when it
+was queued, and both are refused once the run is stopping.
+
 Each change is validated, written under `control.lock`, and says when it applies:
 
 - **The QA threshold is read once per attempt**, at the start of the task's next attempt; the
@@ -244,8 +278,15 @@ Each change is validated, written under `control.lock`, and says when it applies
   worktree. A marker the next retry will discard is reported as `pinned`.
   Fractal plans never apply pins, and a MERGED task refuses one.
 - **Who decided an escalation.** The last column of `tasks/<id>/escalations.tsv` is `policy`
-  (`--escalate-after`), `retry` (a typed `retry --dwarf`) or `pin`; rows from before it was added
+  (`--escalate-after`), `retry` (a typed `retry --dwarf`), `pin` or `fallback`
+  (`--infra-fallback` moved a task off a model at capacity); rows from before it was added
   have four columns and read as `policy`.
+- **Max parallel and the backoff apply at the next decision.** A lower `max-parallel` lets the
+  running tasks finish and never stops one; `infra-backoff` shapes the next infrastructure wait,
+  not the one already counting down.
+- **The effort step re-decides stopped tasks.** Turning `escalate-effort` on clears the policy's
+  stop markers, so a task stopped at the top tier is tried at its model's next effort on its
+  next retry.
 
 Every change is appended to `control.log` (one JSON line: `at`, `actor` `cli` or `ui`, `action`,
 `task`, `key`, `value`, `previous`) and recorded in the dataset as a `control` event, so a run's

package/docs/forge-ui.md

@@ -55,6 +55,20 @@ following its log as it grows. It is read from the agent CLI's own log, whicheve
 wrote it (as plain text for those that log no events yet), or for an older claude run from
 Claude's session file on this machine.
 
+The *Verification* node opens the run's verification in the inspector. While it runs you see
+for how long, the step a lane announced (`::forge step`, see
+[verification lanes](../references/decompose.md#execution-checks-and-integration-verification))
+and its last printed line, a warning after five minutes with no output, and the live output of
+the lane that printed last (or of the one you choose). Once it fails, the failing tests come
+first, by lane and marked new or known (from the lanes' JUnit or TAP reports), each with its
+message, an excerpt and a link that opens the lane's log at that line; a lane without reports
+shows the end of its log and the step it ended in. Below, every lane with its state (flaky and
+known called out), time, runs and test counts; a lane's name opens its output. A single verify
+command shows the newest of its logs. On the *Execution*
+view the verification has rows of its own after the tasks, one per lane with a bar per run and
+reruns marked (a single command's runs sit back to back, dashed, because only their durations
+are known), and its finish node counts the lanes and reruns.
+
 ### Controls
 
 forge-ui reads everything, and it changes things only by calling forge's own commands: it
@@ -64,12 +78,19 @@ indistinguishable from the same command typed in a terminal, except that `contro
 `ui` instead of `cli`. Every change asks you to confirm and says when it takes effect.
 
 - **Run controls** (a plan's inspector): the QA threshold, the per-attempt timeout, how many
-  failures before escalating, verification retries, and, while a scheduler is live, the retry
-  and infrastructure-retry budgets. A task has its own panel for a threshold override
-  (including *None*, which exempts it from the run's threshold), its own retry budget, and a
-  **model pin**: the model its next full retry uses, once. Each is `forge control`; see
-  [decomposed runs](decomposed-runs.md#changing-a-run-while-it-works). A control that cannot
-  apply (a budget with no live scheduler, a merged task) is disabled and says why.
+  failures before escalating, the effort step, verification retries, the infrastructure backoff,
+  and, while a scheduler is live, the retry and infrastructure-retry budgets and max parallel.
+  A live run also gets **Drain** (start nothing new; *Resume* ends it), **Stop** (finish what
+  runs, then end the run) and **Stop now** (interrupt the running tasks), and one line per
+  infrastructure pause: the model or harness it holds, the class, its tasks, and a countdown to
+  its next try, or *budget used up*. A task has its own panel for a threshold override
+  (including *None*, which exempts it from the run's threshold), its own retry budget, a
+  **model pin**: the model its next full retry uses, once, and, on a live run, **Retry now**
+  for a task that stopped or failed, on the policy's choice or a model you name. A retry is
+  queued; the panel says *queued*, then *applied* or *refused* with the scheduler's reason.
+  Each is `forge control`; see [decomposed runs](decomposed-runs.md#changing-a-run-while-it-works).
+  A control that cannot apply (a budget with no live scheduler, a merged task) is disabled and
+  says why.
 - **Open findings:** the review findings a run left open (tolerated by `--qa-threshold`,
   PLAUSIBLE, or left by the whole-run review), for this run or every run of its repository.
   Findings of an outdated review carry an "outdated review" marker, and findings a person

package/docs/reviews.md

@@ -103,15 +103,30 @@ decomposed task then ends as **`INFRA`**, not `ERROR`/`FAIL`, and:
 - **no attempt is spent** — a dwarf-stage stop refunds the counter; a QA-stage stop keeps the
   dwarf's committed work and re-runs **QA only** next time (`tasks/<id>/resume`, revalidated
   against the branch tip and fingerprint before it is trusted);
-- the scheduler stops dispatching, lets running tasks drain, leaves unstarted ones `PENDING`
-  (never `BLOCKED`) and `run` exits **8**: fix the cause, then run the same command again;
-- `run --infra-retries M` waits instead — up to `M` times, using the provider's retry-after
-  hint when it gave one (capped at 6 hours) and otherwise 60 s doubling to 30 min
-  (`FORGE_INFRA_BACKOFF` sets the base) — and resumes in the same run;
+- the scheduler pauses only what the stop is about: a **rate limit** pauses that model on that
+  harness, a **quota** or **auth** stop pauses that harness, and anything else (network, an
+  empty answer) pauses everything, letting running tasks drain. A task whose next dispatch
+  needs a paused model or harness waits; the others keep running. Unstarted tasks stay
+  `PENDING` (never `BLOCKED`), and once nothing else can run `run` exits **8**: fix the cause,
+  then run the same command again;
+- `run --infra-retries M` waits instead — up to `M` waits per paused scope, using the provider's
+  retry-after hint when it gave one (capped at 6 hours) and otherwise the backoff shape — and
+  resumes that scope's tasks in the same run. `--infra-backoff` picks the shape: `exp:60` by
+  default (60 s doubling to 30 min; `FORGE_INFRA_BACKOFF` sets that base), `exp:<base>,<cap>`,
+  `linear:<start>,<step>[,<cap>]` or `list:<a>,<b>,...`. `forge control set infra-backoff`
+  changes it for the next wait, and `forge control show` (and forge-ui) list each pause with
+  the time left;
+- `run --infra-fallback` moves an implementer stopped by a rate limit to the next model of its
+  own tier pool that nobody paused, at once and with no attempt spent (recorded in
+  `escalations.tsv` with source `fallback`); a pinned task, or a pool with no healthy member,
+  waits as above;
 - verification is skipped for a paused run (it is unfinished); the next `run` verifies at its end.
 
+For a planned outage, `forge control drain <plan>` lets the running tasks finish and starts
+nothing new until `forge control resume <plan>`, inside the same run.
+
 `.infra` (`dwarf.infra`/`qa.infra`) holds `class=`, `retry_after=`, `detail=`, `rc=`,
-`role=`. Detection is deliberately strict: strong phrases only, and ordinary text that merely
+`role=`, `harness=`, `model=`. Detection is deliberately strict: strong phrases only, and ordinary text that merely
 mentions "rate limit" in a reply that otherwise succeeded is never classified. A false
 positive costs a pause, not a spent attempt. Under Fractal a dwarf-side infrastructure stop
 still collapses to Fractal's own "failed" handling; QA-side stops propagate as above.
@@ -297,6 +312,16 @@ intermittent failure can be a real bug, so it is never passed off as a clean pas
 that changes the tree is never retried. This deterministic rerun comes before Jev's optional
 model-assisted triage, which then does not run.
 
+With [verification lanes](../references/decompose.md#execution-checks-and-integration-verification)
+the rerun is per lane: only a lane that failed runs again, alone, and a lane that passes on its
+rerun is `flaky`. The run is then a pass, but `run`/`retry`/`integrate` print
+`FLAKY: lane <name> failed once and passed on rerun`, and `verification.flaky` records each
+such lane (`lanes=`, `<lane>.first_exit=`, `<lane>.retries_used=`); a lane whose every failing
+test is in its baseline prints `known failures only: lane <name>`. A whole-run review's
+verification summary lists every lane with its state (`unit: pass`, `db: flaky`, ...) and says
+again which lane failed, which passed only on a rerun ("so its pass is not clean") and which
+failed only on known tests. In lanes mode there is no whole-command rerun and no Jev triage.
+
 ## Native review
 
 `--native-review` switches a codex reviewer to its purpose-built reviewer, for a diff too large

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@giamme/forge",
-  "version": "0.11.1",
+  "version": "0.12.0",
   "description": "Multi-model coding with a reviewer that reads the code: plan, build in parallel worktrees, review every real diff, merge only what passes.",
   "keywords": ["claude-code", "codex", "opencode", "agent-skill", "code-review", "multi-model"],
   "license": "MIT",

package/references/dataset.md

@@ -350,7 +350,9 @@ run.
   wave), and the expected number of first-try passes.
 - It never creates the store, and prints nothing when the store is missing or has no history.
 
-`forge-parallel.sh run` prints the same block before it dispatches. It runs with a time budget
+`forge-parallel.sh run` prints the same block before it dispatches, shortened: the tasks without
+enough history share one line instead of a row each, and when no task has enough, the whole block
+is one line pointing at `forge dataset forecast <plan>`. It runs with a time budget
 (`FORGE_FORECAST_BUDGET_S`, default 3 seconds), prints one line and moves on if reading the
 history takes longer, never prints errors and never changes the exit code. `--no-forecast` or
 `FORGE_FORECAST=off` turn it off. The block is written to the plan as `forecast.json` and