faberun 0.9.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,9 @@
2
2
 
3
3
  <h1 align="center">faberun</h1>
4
4
 
5
- <p align="center">From intent to running software.</p>
5
+ <p align="center"><strong>From intent to running software.</strong></p>
6
+
7
+ <p align="center">Graph engineering for coding agents.</p>
6
8
 
7
9
  <p align="center">
8
10
  <a href="https://github.com/feliperun/faberun/actions/workflows/ci.yml"><img src="https://github.com/feliperun/faberun/actions/workflows/ci.yml/badge.svg" alt="CI status"></a>
@@ -10,21 +12,162 @@
10
12
  <a href="https://www.npmjs.com/package/faberun"><img src="https://img.shields.io/npm/v/faberun" alt="npm version"></a>
11
13
  </p>
12
14
 
13
- Faberun is a development orchestration system that turns intent into verified
14
- software. It manages the process around software creation: plans, tasks,
15
- dependencies, execution, validation, evidence, retries and progress toward a
16
- defined outcome. It is not another coding agent, and it does not generate code
17
- without a definition of done.
15
+ Faberun is a graph-engineering runtime for building software with coding agents.
16
+ It sits above the models and the harnesses and turns an implementation plan into
17
+ a durable graph of tasks, dependencies, workers, proofs, reviews, retries and
18
+ integration gates.
19
+
20
+ Harnesses are execution environments: Claude Code, Codex, the Gemini, DeepSeek
21
+ and GLM shells, and a generic `exec-jsonl` door for whatever comes next. Models
22
+ are the engines inside them. Faberun owns the graph, the state and the
23
+ definition of done around both.
24
+
25
+ The goal is not to make a group of agents talk to each other. It is to use the
26
+ right model for each kind of work, keep the process independent of any single
27
+ model or harness, and require evidence before software counts as done.
28
+
29
+ ## Why Faberun
30
+
31
+ Coding agents are already very good at changing repositories. The remaining
32
+ problem is everything around that execution.
33
+
34
+ A frontier model earns its cost when it reasons about architecture, challenges a
35
+ design, writes a specification or reviews a difficult decision. The same model
36
+ is usually unnecessary for a small, well-defined implementation task a much
37
+ cheaper one performs well. Without an orchestration layer, though, the model
38
+ driving the current session becomes the model doing everything.
39
+
40
+ Faberun separates those concerns. One campaign can use a frontier model to
41
+ reason about the problem and author a plan, a second strong model to attack that
42
+ plan, lower-cost models to execute well-defined implementation nodes, a model
43
+ from another vendor to judge the work that genuinely needs judgment, and plain
44
+ deterministic commands wherever no model is needed at all. The expensive
45
+ intelligence goes where it matters, and the rest becomes an execution problem.
46
+
47
+ It also takes the campaign out of the lifetime of one chat. The work has durable
48
+ state on disk, so Claude Code can start a campaign and Codex, DeepSeek, GLM or
49
+ any other adapted harness can continue it later without reconstructing the
50
+ process from a conversation history.
51
+
52
+ ## Graph engineering
18
53
 
19
- Faberun is model- and harness-agnostic. Claude Code, Codex, OpenCode and
20
- whatever comes next are workers; Claude, GPT, Gemini, DeepSeek, GLM and other
21
- models are engines; Faberun sits above them. It keeps the intent, coordinates
22
- the work, tracks what was actually completed, validates the result and decides
23
- what should happen next.
54
+ Faberun treats development as an executable graph rather than a long
55
+ conversation.
24
56
 
25
- Software should be built, not merely generated. A craftsman does not depend on
26
- one hammer, so Faberun does not depend on one model or one agent: tools,
27
- models and harnesses can change, and the work remains.
57
+ ```text
58
+ intent
59
+ └─ campaign
60
+ └─ contract
61
+ ├─ node A ── proof ── review ──┐
62
+ ├─ node B ── proof ── review ──┼── integration ── promotion
63
+ └─ node C ── proof ── review ──┘
64
+ ```
65
+
66
+ A campaign carries the durable intent. A contract turns part of that intent into
67
+ a schema-versioned DAG. Each node carries a closed packet: what the worker may
68
+ read, what it may change, what done means, and how that claim has to be proven.
69
+ The graph decides what can run, what must wait, what may retry, what needs a
70
+ human and what is allowed to advance. The models are replaceable participants
71
+ inside it. The graph remains.
72
+
73
+ ## What is different
74
+
75
+ ### Use the right model for the job
76
+
77
+ A runtime is one harness running one model. Workers, judges and planning
78
+ sessions need not share a provider or a capability tier, so high-cost reasoning
79
+ and low-cost execution live in the same campaign instead of one model owning the
80
+ whole process. Changing providers is a routing decision, not a rewrite of the
81
+ workflow.
82
+
83
+ ### Proof before opinion
84
+
85
+ A worker saying "done" is not evidence that it is. Every definition-of-done item
86
+ declares how it can be proven: a command that must succeed, a path that must
87
+ exist, or — only when the criterion cannot be mechanised — the judgment of a
88
+ judge. Mechanical verification always runs first, and a node whose proofs are
89
+ all mechanical settles without spending a token on review. A gate may also
90
+ declare `skipWhen`, so a change that is green and small enough skips the judge
91
+ by policy rather than by accident.
92
+
93
+ ### Independent review
94
+
95
+ When judgment is required, the worker does not grade itself. Validation refuses
96
+ a gated node whose worker and judge resolve to the same vendor. The judge reads
97
+ the recorded result and its evidence, never the reasoning that produced it, and
98
+ never re-runs the work. The point is not to make two agents agree; it is to keep
99
+ implementation and acceptance as separate responsibilities.
100
+
101
+ ### Isolated execution
102
+
103
+ Each attempt runs in its own git worktree, so independent nodes run at the same
104
+ time without sharing a mutable working tree, and accepted work reaches the
105
+ repository through controlled refs instead of being copied into the operator's
106
+ checkout. The operator keeps working while the campaign runs.
107
+
108
+ ### Durable campaigns
109
+
110
+ A run is not tied to the terminal that launched it. The controller runs
111
+ detached, state is persisted under `.runs/`, `supervise` resumes a controller
112
+ that died, and a campaign carries its intent across many runs and sessions. If a
113
+ provider goes down or a harness session ends, the work does not have to be
114
+ re-authored.
115
+
116
+ ### Safe integration
117
+
118
+ Passing a worker's own checks is not the end of the process. An accepted attempt
119
+ is sealed, integrated onto the run ref and verified again in the candidate
120
+ state, and the phase-terminal node runs the contract's final verification. If
121
+ combining individually valid changes breaks the system, the campaign does not
122
+ promote them.
123
+
124
+ ### Measurable model economics
125
+
126
+ Every invocation records its usage, routing and outcome, and a closed campaign
127
+ keeps that ledger in its own record under `docs/campaigns/<id>/ledger/`. It makes
128
+ questions like these measurable instead of anecdotal: which model handles this
129
+ class of task reliably, where a frontier model is actually worth its price,
130
+ which nodes close mechanically with no judge at all, how many revisions a worker
131
+ needs before acceptance, what one closed checkpoint costs, and how often a
132
+ fallback saves a campaign. The objective is not to spend less. It is to know
133
+ where expensive intelligence creates value.
134
+
135
+ ## Campaigns, not synthetic teams
136
+
137
+ Faberun does not model an agent process as a human organisation. It needs no "AI
138
+ architect", "AI developer", "AI QA engineer" and "AI product manager" merely
139
+ because human teams carry those titles: models are general enough that those
140
+ boundaries are mostly artificial. The roles it does define are the ones that
141
+ stay useful for agents.
142
+
143
+ | role | responsibility |
144
+ | --- | --- |
145
+ | worker | produces a result inside its packet. |
146
+ | judge | decides independently whether a result that needs judgment satisfies its contract. |
147
+ | controller | advances the graph deterministically. |
148
+ | operator | supplies the intent and intervenes when a decision genuinely needs a human. |
149
+
150
+ Everything else belongs in the contract.
151
+
152
+ ## From spec to contract
153
+
154
+ Planning is part of the product, and it is adversarial on purpose. `faberun spec
155
+ validate` checks a specification's front matter, its `R<n>` requirements and the
156
+ proof each one declares, and invokes no model to do it. `faberun plan` then runs
157
+ the debate in budgeted, isolated runs: an author drafts, a reviewer from another
158
+ vendor attacks the draft holding only the spec, the repository facts and the
159
+ plan under review, and a revision round follows while a critical finding
160
+ remains. A converged draft is sized, routed and frozen into a contract. Freezing
161
+ never launches, and a plan that never converges ends `contested`, with its open
162
+ findings recorded as a campaign question instead of a contract.
163
+
164
+ None of it is mandatory. The traceability rules are advisory unless you ask for
165
+ `--strict-traceability`, a document with no front matter validates as `legacy`
166
+ rather than being rejected, and you can skip planning altogether and author the
167
+ contract by hand. What Faberun needs is the executable part of the plan:
168
+ objective, nodes, dependencies, read and write scope, definitions of done,
169
+ verification, runtime policy and integration behaviour. Your specification
170
+ process stays yours.
28
171
 
29
172
  ## Install
30
173
 
@@ -92,29 +235,35 @@ The operator writes the intent into a campaign and an authored contract, a
92
235
  schema-versioned DAG whose nodes each carry a closed task packet. The controller
93
236
  schedules every dependency-ready node and dispatches its packet to a worker
94
237
  inside an attempt worktree, where the worker sees only the files the packet
95
- names. The controller then runs the deterministic verification once, and a
96
- judge from a different vendor reviews the recorded result without re-running it.
97
- A passing attempt is sealed and integrated onto the run ref, a campaign promotes
98
- each run onto its landing branch, and the orchestrator lands that branch.
99
- Campaigns, handoffs and the `supervise` watchdog carry the work across sessions,
100
- so an interrupted run is continued in place instead of being re-authored.
238
+ names. Faberun then judges the result in stages, and a free slot dispatches the
239
+ next node while another one is still being verified.
101
240
 
102
241
  ```text
103
- intent
104
- └─ contract: validate · preflight
105
- └─ controller
106
- ├─ worker attempt in an attempt worktree
107
- └─ deterministic verification · cross-vendor judge
108
- └─ integration ref · promotion · landing branch
242
+ packet
243
+ └─ worker attempt in its own worktree
244
+ └─ mechanical proof ── failure ─→ retry · route · attention
245
+ └─ judge, when required ── rejection ─→ revision
246
+ └─ seal ─→ integration ref ─→ candidate verification
247
+ └─ promotion onto the campaign's landing branch
109
248
  ```
110
249
 
111
- The vocabulary is in [Concepts](docs/CONCEPTS.md), and the layers, process
112
- model and gates are in [Architecture](docs/ARCHITECTURE.md).
250
+ A judge reviews recorded evidence rather than re-running arbitrary work. A
251
+ passing attempt is sealed and integrated onto the run ref, a campaign promotes
252
+ each finished run onto its landing branch, and the orchestrator lands that
253
+ branch. Campaigns, handoffs and the `supervise` watchdog carry the work across
254
+ sessions, so an interrupted run is continued in place instead of being
255
+ re-authored.
113
256
 
114
- ## Harnesses
257
+ The vocabulary is in [Concepts](docs/CONCEPTS.md), and the layers, process model
258
+ and gates are in [Architecture](docs/ARCHITECTURE.md).
115
259
 
116
- A runtime is one harness running one model. Any runtime can be a worker, and a
117
- judge of another vendor reviews what it produced.
260
+ ## Harnesses and models
261
+
262
+ Faberun keeps three things apart. A **model** is the reasoning engine, a
263
+ **harness** is the environment that gives a model access to code and tools, and
264
+ a **runtime** is one configured harness-and-model pair Faberun can dispatch. Any
265
+ runtime can act as a worker or as a judge, according to the contract and the
266
+ routing policy.
118
267
 
119
268
  | Harness | Default vendor | Example model |
120
269
  | --- | --- | --- |
@@ -126,6 +275,19 @@ judge of another vendor reviews what it produced.
126
275
  | `exec-jsonl` | declared per runtime | the model its command names |
127
276
  | `replay` | declared per runtime | the recorded model |
128
277
 
278
+ Harness and model support are adapter concerns. The orchestration above them
279
+ depends on no single vendor.
280
+
281
+ ## Philosophy
282
+
283
+ Software should be built, not merely generated. Generating plausible code is
284
+ becoming cheap; the valuable part is the system around the generation —
285
+ preserving intent, decomposing work, choosing the appropriate intelligence,
286
+ isolating effects, proving outcomes, recovering from failure, and recording why a
287
+ change was accepted. A craftsman does not depend on one hammer, and Faberun does
288
+ not depend on one model. Models change, harnesses change, providers change. The
289
+ intent, the graph, the evidence and the finished work remain.
290
+
129
291
  ## Documentation
130
292
 
131
293
  - [Documentation map](docs/README.md): every document and the question it answers.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "faberun",
3
- "version": "0.9.0",
3
+ "version": "0.10.0",
4
4
  "description": "Faberun is a development orchestration system that turns intent into verified software: harness- and model-agnostic, it keeps the intent, coordinates the work, verifies the result and decides what happens next.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -290,6 +290,25 @@ export function validateContractAgainstRef(raw, contractPath, context = {}) {
290
290
  }
291
291
  }
292
292
 
293
+ /**
294
+ * Validate a contract for a launch, the one path the CLI's `run` command and
295
+ * the scheduler's own re-validation both go through: against `baseRef` when
296
+ * one is given, the checkout otherwise. `repo` defaults to the contract's own
297
+ * `cwd` when the caller does not know a better one (the campaign chain always
298
+ * passes its own), since a contract's `cwd` is always inside the repo it
299
+ * names.
300
+ *
301
+ * @param {Record<string, unknown>} raw
302
+ * @param {string} contractPath
303
+ * @param {{repo?: string, baseRef?: string}} [context]
304
+ * @returns {ValidatedContract}
305
+ */
306
+ export function validateContractForLaunch(raw, contractPath, context = {}) {
307
+ if (!context.baseRef) return validateContract(raw, contractPath);
308
+ const repo = context.repo ?? resolve(dirname(contractPath), typeof raw.cwd === "string" ? raw.cwd : ".");
309
+ return validateContractAgainstRef(raw, contractPath, { repo, baseRef: context.baseRef });
310
+ }
311
+
293
312
  /**
294
313
  * Reduce a run's progress to the one decision the chain makes. An in-flight run
295
314
  * is `unfinished` even though `reduceRunOutcome` already names its running nodes
package/src/cli.mjs CHANGED
@@ -24,6 +24,7 @@ import {
24
24
  validBootstrapNonce,
25
25
  } from "./run/lock.mjs";
26
26
  import { renderRunHandoff } from "./campaign/index.mjs";
27
+ import { validateContractForLaunch } from "./campaign/chain.mjs";
27
28
  import { campaignCli } from "./cli/campaign.mjs";
28
29
  import { seatCli } from "./cli/seat.mjs";
29
30
  import { initCommand } from "./cli/init.mjs";
@@ -291,9 +292,9 @@ async function main(argv) {
291
292
  if (command === "run") {
292
293
  warnIfNoTransport();
293
294
  const absolute = resolve(target);
294
- const contract = validateContract(JSON.parse(readFileSync(absolute, "utf8")), absolute);
295
- const runDir = join(contract.cwd, ".runs", contract.id);
296
295
  const baseRef = typeof values["base-ref"] === "string" && values["base-ref"] ? values["base-ref"] : undefined;
296
+ const contract = validateContractForLaunch(JSON.parse(readFileSync(absolute, "utf8")), absolute, { baseRef });
297
+ const runDir = join(contract.cwd, ".runs", contract.id);
297
298
  setLaunchBaseRef(baseRef);
298
299
  // The base is what every worktree is cut from; a dirty tree only blocks
299
300
  // when the cwd HEAD *is* that base. A `--base-ref` elsewhere leaves the
@@ -314,7 +315,7 @@ async function main(argv) {
314
315
  return;
315
316
  }
316
317
  for (const warning of [...contract.warnings, ...reusedDoneWarnings(contract)]) process.stdout.write(`${advisoryToken()} ${warning}\n`);
317
- const result = await runContract(target, { detachedBootstrap: hasDetachedBootstrapNonce() });
318
+ const result = await runContract(target, { detachedBootstrap: hasDetachedBootstrapNonce(), baseRef });
318
319
  if (!result.ok) process.exitCode = 1;
319
320
  return;
320
321
  }
@@ -3,11 +3,13 @@
3
3
  * rule 12).
4
4
  *
5
5
  * `contract.finalVerification` is the contract-wide proof that the phase as a
6
- * whole closes: the controller runs it before the judge on the phase-terminal
7
- * node (the node no other node depends on), so no final checkpoint is ever
8
- * approved on partial verification. `contract.sharedVerification` carries the
9
- * same command schema but is appended to every node's attempt and integration
10
- * candidate, for the fast repository ratchets a node's write set can break.
6
+ * whole closes: the controller runs it before the judge, once, on whichever
7
+ * phase-terminal node (a node no other node depends on) is the last of them
8
+ * to settle, so no final checkpoint is ever approved on partial verification
9
+ * and a flake on one phase-terminal node cannot also cost its siblings a
10
+ * revision. `contract.sharedVerification` carries the same command schema but
11
+ * is appended to every node's attempt and integration candidate, for the fast
12
+ * repository ratchets a node's write set can break.
11
13
  * The persisted node-snapshot shape for verification evidence lives here too,
12
14
  * next to the schema it records.
13
15
  */
@@ -58,20 +60,49 @@ export function sharedVerificationCommands(contract) {
58
60
  return contract.sharedVerification ?? [];
59
61
  }
60
62
 
63
+ /**
64
+ * Whether no other node in the contract depends on this one. `finalVerification`
65
+ * is a candidate to run on a phase-terminal node only; a node with a dependant
66
+ * never carries it, no matter which of its siblings settles last.
67
+ *
68
+ * @param {{nodes: {id: string, dependsOn: string[]}[]}} contract
69
+ * @param {{id: string}} node
70
+ * @returns {boolean}
71
+ */
72
+ export function phaseTerminalNode(contract, node) {
73
+ return !contract.nodes.some((candidate) => candidate.dependsOn.includes(node.id));
74
+ }
75
+
61
76
  /**
62
77
  * The contract's `finalVerification` commands when this node is the one that
63
- * closes the phase, otherwise none. A node is phase-terminal when no other
64
- * node in the contract depends on it.
78
+ * closes the phase, otherwise none. A phase can have several phase-terminal
79
+ * nodes (several nodes no other node depends on); the suite runs once for the
80
+ * phase, on whichever of them is the last to settle, not on every one of them.
81
+ *
82
+ * `settledIds` is the set of sibling node ids (any status a node does not
83
+ * leave on its own -- see `SETTLED` in `engine/prompts.mjs`) already reached
84
+ * at the moment this call is made. A phase-terminal node carries the suite
85
+ * exactly when every *other* phase-terminal node is already in that set --
86
+ * whichever node that is true for, in whatever order nodes actually settle,
87
+ * including a node retried after every sibling has already closed. Omitting
88
+ * `settledIds` (the scheduler's budget estimate, which has no run to inspect)
89
+ * falls back to the old per-node-shape answer: any phase-terminal node may
90
+ * still turn out to be the one that carries it, so the estimate stays an
91
+ * upper bound.
65
92
  *
66
93
  * @param {{finalVerification?: VerificationCommand[], nodes: {id: string, dependsOn: string[]}[]}} contract
67
94
  * @param {{id: string}} node
95
+ * @param {Set<string>} [settledIds]
68
96
  * @returns {VerificationCommand[]}
69
97
  */
70
- export function finalVerificationCommands(contract, node) {
98
+ export function finalVerificationCommands(contract, node, settledIds) {
71
99
  const commands = contract.finalVerification ?? [];
72
100
  if (commands.length === 0) return [];
73
- const hasDependant = contract.nodes.some((candidate) => candidate.dependsOn.includes(node.id));
74
- return hasDependant ? [] : commands;
101
+ if (!phaseTerminalNode(contract, node)) return [];
102
+ if (!settledIds) return commands;
103
+ const siblings = contract.nodes.filter((candidate) => candidate.id !== node.id && phaseTerminalNode(contract, candidate));
104
+ const isLastToClose = siblings.every((sibling) => settledIds.has(sibling.id));
105
+ return isLastToClose ? commands : [];
75
106
  }
76
107
 
77
108
  /**
@@ -31,15 +31,15 @@ import {
31
31
  terminalErrorCode,
32
32
  } from "./lifecycle.mjs";
33
33
  import { delay, errorCode } from "../util.mjs";
34
- import { alreadyNotified, notifyQueueFor, notifyQueuesByRun, renderCampaignHandoffSafely } from "./notify-queue.mjs";
35
- import { detectStalls, terminateProcess } from "./process.mjs";
34
+ import { alreadyNotified, emitNodeAdvisories, notifyQueueFor, notifyQueuesByRun, renderCampaignHandoffSafely } from "./notify-queue.mjs";
35
+ import { detectStalls, invocationAlive, terminateProcess } from "./process.mjs";
36
36
  import { transition, writeNode } from "./state.mjs";
37
37
  import { listNodeSnapshots, readNodeSnapshot } from "../run/node-store.mjs";
38
38
  import { render, renderFinalReport, writeFindingsArtifact } from "../report/final.mjs";
39
39
  import { operationNextState, providerReceipts, settleInvocation } from "../run/operations.mjs";
40
40
  import { appendUsageRecord, invocationCost, invocationUsage, recordInvocationUsage } from "../run/usage.mjs";
41
41
  import { captureNodeScopeBoundaries, checkWorkerScope, emptyScope } from "./scope.mjs";
42
- import { validateContract } from "../contract/index.mjs";
42
+ import { validateContractForLaunch } from "../campaign/chain.mjs";
43
43
  import { validateNodeSnapshot } from "../contract/snapshot.mjs";
44
44
  import { finalVerificationCommands, sharedVerificationCommands } from "../contract/final-verification.mjs";
45
45
  import { startJudge, startWorker } from "./dispatch.mjs";
@@ -131,16 +131,22 @@ export function nodeBudgetBasisMs(contract, node) {
131
131
  * never touches phase 2's parked `blocked`/`failed`/`exhausted`/`stalled`
132
132
  * states, and never overwrites the `runtime_tier_exhausted` waiting shape.
133
133
  *
134
+ * A node whose closed job is being settled in the background (its id is a key
135
+ * of `pendingSettlements`) is not this dead end either: its invocation has
136
+ * already exited and left `running`, but the settlement promise still owns
137
+ * deciding what happens to it, so it is left alone until that promise resolves.
138
+ *
134
139
  * @param {string} runDir
135
140
  * @param {Map<string, NodeSnapshot>} states
136
141
  * @param {Map<string, Job>} running
137
142
  * @param {LockHandle|null} lock
143
+ * @param {Map<string, Promise<void>>} [pendingSettlements]
138
144
  * @returns {string[]} the node ids this pass parked
139
145
  */
140
- export function enforceRunningInvariant(runDir, states, running, lock) {
146
+ export function enforceRunningInvariant(runDir, states, running, lock, pendingSettlements = new Map()) {
141
147
  const parked = [];
142
148
  for (const [nodeId, state] of states) {
143
- if (state.status !== "running" || running.has(nodeId)) continue;
149
+ if (state.status !== "running" || running.has(nodeId) || pendingSettlements.has(nodeId)) continue;
144
150
  transition(runDir, state, "blocked", {
145
151
  phase: state.phase,
146
152
  error: {
@@ -175,14 +181,17 @@ function renderFingerprint(states) {
175
181
 
176
182
  /**
177
183
  * @param {string} contractPath
178
- * @param {{detachedBootstrap?: boolean}} [options] `detachedBootstrap` is set
179
- * only by the CLI entry when this process is its own detached child, and
180
- * makes the controller wait for the launcher's acknowledgement
184
+ * @param {{detachedBootstrap?: boolean, baseRef?: string}} [options]
185
+ * `detachedBootstrap` is set only by the CLI entry when this process is its
186
+ * own detached child, and makes the controller wait for the launcher's
187
+ * acknowledgement; `baseRef` is the CLI's own `--base-ref`, re-validated
188
+ * here so a launch and the run it starts agree about what the contract was
189
+ * checked against
181
190
  * @returns {Promise<RunOutcome>}
182
191
  */
183
192
  export async function runContract(contractPath, options = {}) {
184
193
  const absoluteContractPath = resolve(contractPath);
185
- const contract = validateContract(JSON.parse(readFileSync(absoluteContractPath, "utf8")), absoluteContractPath);
194
+ const contract = validateContractForLaunch(JSON.parse(readFileSync(absoluteContractPath, "utf8")), absoluteContractPath, { baseRef: options.baseRef });
186
195
  const runDir = join(contract.cwd, ".runs", contract.id);
187
196
  if (existsSync(runDir)) throw new Error(`run already exists: ${runDir}`);
188
197
  mkdirSync(join(contract.cwd, ".runs"), { recursive: true });
@@ -319,12 +328,13 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
319
328
  statusFingerprint = fingerprint;
320
329
  render(runDir, runsDir, contract, states, renderLock);
321
330
  };
322
- // The tick that owns a running node's verification can be minutes long
323
- // (`executeControllerVerification` is awaited on the critical path below),
324
- // and that whole time status.json would otherwise report whatever the last
325
- // tick left it at. A timer renders between ticks too; it is cheap even when
326
- // idle because `renderFingerprint` still change-detects, so a quiet run
327
- // writes nothing extra.
331
+ // A node's own settlement (its controller verification, its judge round) can
332
+ // be minutes long, and it now runs off the tick's critical path in
333
+ // `pendingSettlements` so dispatch never waits behind it -- but that whole
334
+ // time status.json would otherwise report whatever the last tick left it at.
335
+ // A timer renders between ticks too; it is cheap even when idle because
336
+ // `renderFingerprint` still change-detects, so a quiet run writes nothing
337
+ // extra.
328
338
  const statusTimer = setInterval(() => renderStatusIfChanged(), contract.pollIntervalMs);
329
339
  statusTimer.unref();
330
340
  let handoffFingerprint = statesFingerprint(states);
@@ -364,14 +374,31 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
364
374
 
365
375
  /** @type {Map<string, Job>} */
366
376
  const running = new Map();
377
+ // One promise per node currently settling a closed job -- its controller
378
+ // verification, its candidate verification, its judge round -- kept off the
379
+ // tick's critical path so an eligible sibling still dispatches into a free
380
+ // slot while this node's own invocation has already exited. A node's own
381
+ // steps stay ordered because only one settlement per node is ever in flight
382
+ // (see the dispatch loop below); a *different* node's settlement is queued
383
+ // behind whichever one is already running (`settlementQueue`), not run
384
+ // alongside it -- `finalizeClosedJobs` shares state a concurrent second call
385
+ // would corrupt: the phase-continuation selection that picks at most one
386
+ // live node to carry a session forward, and `repo/integrate.mjs`'s one
387
+ // candidate ref and worktree per run. Only *dispatching a sibling* skips
388
+ // ahead of a node's settlement; two nodes' settlements never interleave.
389
+ /** @type {Map<string, Promise<void>>} */
390
+ const pendingSettlements = new Map();
391
+ /** @type {Promise<void>} */
392
+ let settlementQueue = Promise.resolve();
367
393
  let canceled = false;
368
394
  const cancel = () => { canceled = true; };
369
395
  process.once("SIGINT", cancel);
370
396
  process.once("SIGTERM", cancel);
371
397
  process.once("SIGHUP", cancel);
372
398
  // The heartbeat's `at` is owned by an unref'd timer inside this writer, never
373
- // by the loop body below: the loop awaits controller verification on its own
374
- // critical path and must keep answering "the process is alive" while it does.
399
+ // by the loop body below: a node's settlement can still run for minutes off
400
+ // the tick's critical path (`pendingSettlements`), and the heartbeat must
401
+ // keep answering "the process is alive" while it does.
375
402
  const heartbeat = createHeartbeat({ runDir, intervalMs: HEARTBEAT_INTERVAL_MS });
376
403
  const heartbeatNodes = new Map(contract.nodes.map((node) => [node.id, node]));
377
404
  let heartbeatFingerprint = statesFingerprint(states);
@@ -380,11 +407,89 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
380
407
  const activeHeartbeatNodes = () => [...states.values()]
381
408
  .filter((state) => state.status === "running" && heartbeatNodes.has(state.id))
382
409
  .map((state) => ({ nodeId: state.id, budgetBasis: nodeBudgetBasisMs(contract, /** @type {ValidatedNode} */ (heartbeatNodes.get(state.id))) }));
410
+ // A programmer error surfacing inside a background settlement must still
411
+ // crash the whole run, exactly as an unguarded `await finalizeClosedJobs`
412
+ // used to: it is recorded here and thrown from the top of the loop on the
413
+ // very next tick, rather than immediately, so it cannot itself become the
414
+ // block a sibling's dispatch is waiting behind. A lost lock is not this --
415
+ // the loop's own `lock.assert()` calls surface that same condition on their
416
+ // own schedule, so it is left for them.
417
+ /** @type {unknown} */
418
+ let backgroundSettlementFailure = null;
419
+ // Mark every job that closed this tick as settling, and free its slot,
420
+ // without waiting for any of them: that alone is what lets an eligible
421
+ // sibling dispatch into the freed slot while this node's minutes-long
422
+ // controller verification or judge round is still running. The actual
423
+ // settlement work is chained onto `settlementQueue`, one node at a time in
424
+ // the order its job closed, so it still runs exactly as serialized against
425
+ // every *other* node's settlement as it did when this loop awaited
426
+ // `finalizeClosedJobs` directly -- `finalizeClosedJobs` runs against a
427
+ // one-entry map per node, so this is one call per node rather than the one
428
+ // batched call it used to be, but the chain still runs them one at a time.
429
+ // A settlement may itself dispatch the node's next phase (a judge, a
430
+ // revision) through the same `startJudge`/`startWorker` calls dispatch below
431
+ // uses; those land in `slot`, not the real `running`, so they are copied
432
+ // back into `running` in a `finally` -- unconditionally, win or lose, so a
433
+ // job a settlement started before failing (a lost lock, a programmer error)
434
+ // is still visible to the cleanup sweeps below rather than leaked. The
435
+ // node's own steps stay ordered by never starting a second settlement for a
436
+ // node whose first has not yet cleared `pendingSettlements`.
437
+ const settleClosedJobsInBackground = () => {
438
+ for (const [nodeId, job] of [...running]) {
439
+ if (pendingSettlements.has(nodeId) || !job.closed || invocationAlive(job.invocation)) continue;
440
+ running.delete(nodeId);
441
+ const slot = new Map([[nodeId, job]]);
442
+ const settlement = settlementQueue
443
+ .then(() => finalizeClosedJobs(contract, runDir, states, slot, lock, campaign.path))
444
+ .finally(() => {
445
+ for (const [settledId, settledJob] of slot) running.set(settledId, settledJob);
446
+ pendingSettlements.delete(nodeId);
447
+ });
448
+ // The queue itself must never reject -- a rejected settlement (a lost
449
+ // lock, a programmer error) would otherwise wedge every node queued
450
+ // behind it. The rejection still reaches whoever awaits the real
451
+ // `settlement` promise (`pendingSettlements`, below).
452
+ settlementQueue = settlement.catch(() => {});
453
+ // Handled here so an in-flight settlement never becomes an unhandled
454
+ // rejection when nobody happens to await `pendingSettlements` before the
455
+ // process exits; the original promise, still held below, carries the
456
+ // rejection to whichever checkpoint (cancel, shutdown) awaits it.
457
+ settlement.catch((error) => {
458
+ if (!(error instanceof LockLostError) && backgroundSettlementFailure === null) backgroundSettlementFailure = error;
459
+ });
460
+ pendingSettlements.set(nodeId, settlement);
461
+ }
462
+ };
463
+ // Captured once, before the loop, rather than re-derived every tick: it is
464
+ // "parked when this controller invocation started" (autoRetryParkedNodes's
465
+ // own contract), and a node a background settlement parks between two ticks
466
+ // -- rather than synchronously within one, as it always did before
467
+ // settlement moved off the tick's critical path -- must still read as newly
468
+ // parked whichever later tick first observes it, not as already-parked
469
+ // because a per-tick snapshot happened to be taken after it landed.
470
+ const parkedBefore = new Set([...states.values()].filter((state) => PARKED.has(state.status)).map((state) => state.id));
471
+ const anyUnsettled = () => [...states.values()].some((state) => !SETTLED.has(state.status));
383
472
  try {
384
- while ([...states.values()].some((state) => !SETTLED.has(state.status))) {
473
+ // A `while` that re-checked this at the very top of every tick would exit
474
+ // the instant a background settlement flips the run's last unsettled node
475
+ // straight to a terminal status between two ticks -- before the tick body
476
+ // that would have run `autoRetryParkedNodes` against that new status ever
477
+ // gets to. The entry guard skips the loop entirely when there is nothing
478
+ // to do at all (a resume of an already-settled run still does zero
479
+ // iterations); once inside, the exit check moves to the bottom, after the
480
+ // body, so that body always sees a freshly-parked node at least once
481
+ // before the loop is allowed to end.
482
+ if (anyUnsettled()) for (;;) {
385
483
  lock.assert();
484
+ if (backgroundSettlementFailure !== null) throw backgroundSettlementFailure;
386
485
  if (existsSync(join(runDir, "cancel.request.json"))) canceled = true;
387
486
  if (canceled) {
487
+ // A settlement already in flight owns the one decision a cancellation
488
+ // must not race: what this node's own invocation resolved to. Let it
489
+ // land on its real terminal status (and, when it re-dispatched a judge
490
+ // or a revision, on the new job that landed in `running`) before this
491
+ // branch decides which nodes are merely canceled.
492
+ await Promise.all([...pendingSettlements.values()]);
388
493
  const jobs = [...running.values()];
389
494
  await Promise.all(jobs.map((job) => terminateProcess(job)));
390
495
  const envelopes = new Map();
@@ -408,8 +513,14 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
408
513
  break;
409
514
  }
410
515
 
411
- const parkedBefore = new Set([...states.values()].filter((state) => PARKED.has(state.status)).map((state) => state.id));
412
- await finalizeClosedJobs(contract, runDir, states, running, lock, campaign.path);
516
+ // Advisory spend lines are checked every tick, closed job or not: a
517
+ // crossing must be visible while the spend is happening on a node still
518
+ // running, not only once it closes. `finalizeClosedJobs` used to open
519
+ // with this same check; calling it once here, rather than once per
520
+ // node settled this tick, is what keeps it at one pass per tick now
521
+ // that settlement runs per node in `settleClosedJobsInBackground`.
522
+ await emitNodeAdvisories(contract, runDir, states);
523
+ settleClosedJobsInBackground();
413
524
  await detectStalls(contract, running, async (job, status, error) => {
414
525
  const envelope = recordInvocationUsage(job);
415
526
  job.state.usage = invocationUsage(job.state);
@@ -450,8 +561,11 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
450
561
  writeNode(runDir, job.state, lock);
451
562
  });
452
563
  // A node left `running` with no job is a dead end; park it before the
453
- // dispatch pass so it cannot hide behind a healthy sibling.
454
- enforceRunningInvariant(runDir, states, running, lock);
564
+ // dispatch pass so it cannot hide behind a healthy sibling. A node
565
+ // whose closed job is settling in the background is not that dead end
566
+ // -- `pendingSettlements` is what tells this apart from one truly
567
+ // abandoned.
568
+ enforceRunningInvariant(runDir, states, running, lock, pendingSettlements);
455
569
  // Before dependants are blocked, a node that parked on this tick gets
456
570
  // its one automatic retry: it becomes pending, so `blockDependents` sees
457
571
  // nothing to block and the dependants stay `pending`/`phase: "waiting"`
@@ -463,14 +577,37 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
463
577
  if (slots > 0) {
464
578
  const ready = contract.nodes.filter((node) => {
465
579
  const state = states.get(node.id);
466
- return state?.status === "pending" && node.dependsOn.every((id) => states.get(id)?.status === "done");
580
+ return state?.status === "pending" && !pendingSettlements.has(node.id)
581
+ && node.dependsOn.every((id) => states.get(id)?.status === "done");
467
582
  });
468
583
  for (const node of ready.slice(0, slots)) {
469
584
  const state = states.get(node.id);
470
585
  if (!state || routingBackoffActive(state, state.phase)) continue;
586
+ // A node recovered pending a re-ask judge (its own worker attempt
587
+ // already accepted, `state.result` durable) reaches `settleDone` /
588
+ // `integrateAttempt` exactly like a closed job's own settlement
589
+ // does, on the same per-run candidate ref and worktree
590
+ // `settlementQueue` exists to serialize -- so it is dispatched the
591
+ // same way: chained onto the queue rather than awaited here, using
592
+ // its own one-entry `slot` merged back into `running` in a
593
+ // `finally`. The node's own order is untouched (still one entry at
594
+ // a time, gated by `pendingSettlements`); only the tick stops
595
+ // waiting behind it.
471
596
  if (state.phase === "judge" && state.result) {
472
- await applyJudgeRound(await startJudge(contract, node, state, runDir, running, state.result, lock, states, campaign.path),
473
- contract, node, state, runDir, running, lock, states, campaign.path, state.result);
597
+ const workerResult = state.result;
598
+ const slot = new Map();
599
+ const settlement = settlementQueue
600
+ .then(() => startJudge(contract, node, state, runDir, slot, workerResult, lock, states, campaign.path))
601
+ .then((round) => applyJudgeRound(round, contract, node, state, runDir, slot, lock, states, campaign.path, workerResult))
602
+ .finally(() => {
603
+ for (const [settledId, settledJob] of slot) running.set(settledId, settledJob);
604
+ pendingSettlements.delete(node.id);
605
+ });
606
+ settlementQueue = settlement.catch(() => {});
607
+ settlement.catch((error) => {
608
+ if (!(error instanceof LockLostError) && backgroundSettlementFailure === null) backgroundSettlementFailure = error;
609
+ });
610
+ pendingSettlements.set(node.id, settlement);
474
611
  continue;
475
612
  }
476
613
  state.attempt += 1;
@@ -499,10 +636,46 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
499
636
  }
500
637
  }
501
638
  heartbeat.setActive(activeHeartbeatNodes());
502
- if ([...states.values()].some((state) => !SETTLED.has(state.status))) await delay(contract.pollIntervalMs);
639
+ // A background settlement can flip a node straight to a parked
640
+ // terminal status during this same tick's own later awaits
641
+ // (`detectStalls`, `notifyStateChanges`), after the auto-retry pass
642
+ // above already ran and saw it as not-yet-parked. Running it once more
643
+ // here, immediately before the exit check, is what stops the loop from
644
+ // mistaking that freshly-parked node for settled and exiting before it
645
+ // ever got its automatic retry; a node it reopens dispatches on the
646
+ // next tick rather than this one, which `anyUnsettled()` below still
647
+ // correctly keeps the loop alive for.
648
+ autoRetryParkedNodes(contract, runDir, states, lock, parkedBefore);
649
+ if (!anyUnsettled()) break;
650
+ await delay(contract.pollIntervalMs);
503
651
  }
652
+ // The loop above exits (by the bottom check above or the cancel branch's
653
+ // `break`) the instant every node's state looks settled, but a
654
+ // settlement's own trailing work -- sealing a candidate's acceptance,
655
+ // removing its worktree, enqueueing its terminal notification -- can
656
+ // still be running after the transition that made the state look
657
+ // terminal. Nothing past this point (the final render, the findings
658
+ // artifact, the run-terminal notification, releasing the lock) may run
659
+ // ahead of that trailing work.
660
+ await Promise.all([...pendingSettlements.values()]);
661
+ // A settlement's own `finally` deletes it from `pendingSettlements` the
662
+ // instant it settles, win or lose -- so a rejection recorded here can
663
+ // already be gone from the map above by the time this line runs, with
664
+ // nothing left to await it. The loop's own top-of-tick check
665
+ // (`if (backgroundSettlementFailure !== null) throw ...`) cannot save
666
+ // that case either: the loop has already exited. Checked again here,
667
+ // once, so a background settlement failure can never read as a clean
668
+ // finish just because it settled on the same tick the run's last node
669
+ // did.
670
+ if (backgroundSettlementFailure !== null) throw backgroundSettlementFailure;
504
671
  } catch (error) {
505
672
  if (!(error instanceof LockLostError)) throw error;
673
+ // A settlement still merges whatever it started into `running` in its own
674
+ // `finally`, win or lose, regardless of whether anything awaits it; wait
675
+ // for that to land (never rejecting itself, so a second lock-loss here
676
+ // cannot mask the one already being handled) before the termination sweep
677
+ // reads `running`.
678
+ await Promise.allSettled([...pendingSettlements.values()]);
506
679
  await Promise.all([...running.values()].map((job) => terminateProcess(job)));
507
680
  notifyQueuesByRun.delete(runDir);
508
681
  return { runDir, states, ok: false, error };
@@ -165,7 +165,7 @@ export async function settleDone(contract, node, state, runDir, lock, states, ca
165
165
  attemptSha: sealed.sha,
166
166
  branch: state.worktree.branch,
167
167
  verificationEvidence: state.verification,
168
- verifyCandidate: (candidateWorkspace) => verifyCandidateWorkspace(contract, node, state, runDir, candidateWorkspace),
168
+ verifyCandidate: (candidateWorkspace) => verifyCandidateWorkspace(contract, node, state, runDir, candidateWorkspace, lock),
169
169
  onAccepted: async (transaction) => {
170
170
  const acceptedPath = state.worktree?.path ?? attemptWorktreePath(runDir, contract.id, node.id, transaction.attempt);
171
171
  if (state.attempt === transaction.attempt && state.status !== "done") {
@@ -11,8 +11,10 @@
11
11
  import { attemptWorkspace } from "../repo/worktree.mjs";
12
12
  import { boundedUtf8, errorMessage } from "../util.mjs";
13
13
  import { compactVerification } from "../contract/verification.mjs";
14
- import { finalVerificationCommands, sharedVerificationCommands } from "../contract/final-verification.mjs";
14
+ import { finalVerificationCommands, phaseTerminalNode, sharedVerificationCommands } from "../contract/final-verification.mjs";
15
15
  import { join } from "node:path";
16
+ import { listNodeSnapshots, readNodeSnapshot } from "../run/node-store.mjs";
17
+ import { SETTLED } from "./prompts.mjs";
16
18
  import { terminateInvocation } from "./process.mjs";
17
19
  import { writeNode } from "./state.mjs";
18
20
  import { runVerification } from "./run-command.mjs";
@@ -28,6 +30,34 @@ import { runVerification } from "./run-command.mjs";
28
30
  /** @typedef {import("../contract/index.mjs").VerificationState} VerificationState */
29
31
  /** @typedef {{index: number, total: number, argv: string}} VerificationProgress */
30
32
 
33
+ /**
34
+ * The sibling phase-terminal node ids already settled, read straight off
35
+ * disk: every node snapshot is persisted from the run's first tick (see
36
+ * `runContract`), so a node still `pending` reads back honestly, not as
37
+ * missing. `finalVerificationCommands` uses this to decide, at the moment a
38
+ * phase-terminal node's own verification runs, whether it is the one that
39
+ * closes the phase -- recomputed fresh on every call, so a retried node sees
40
+ * its siblings' current status each time, not a decision frozen from an
41
+ * earlier attempt.
42
+ *
43
+ * @param {string} runDir
44
+ * @param {ValidatedContract} contract
45
+ * @param {ValidatedNode} node
46
+ * @returns {Set<string>}
47
+ */
48
+ function settledSiblingIds(runDir, contract, node) {
49
+ const ids = new Set();
50
+ for (const name of listNodeSnapshots(runDir)) {
51
+ const id = name.slice(0, -".json".length);
52
+ if (id === node.id) continue;
53
+ const sibling = contract.nodes.find((candidate) => candidate.id === id);
54
+ if (!sibling || !phaseTerminalNode(contract, sibling)) continue;
55
+ const snapshot = /** @type {{status?: string}} */ (readNodeSnapshot(runDir, id));
56
+ if (SETTLED.has(/** @type {string} */ (snapshot.status))) ids.add(id);
57
+ }
58
+ return ids;
59
+ }
60
+
31
61
  /**
32
62
  * The bounded `k/n · argv` shape a running node's status surfaces while a
33
63
  * verification command is in flight: the command's 1-based position among
@@ -108,7 +138,7 @@ export async function executeControllerVerification(contract, runDir, node, stat
108
138
  };
109
139
  writeNode(runDir, state, lock);
110
140
  const workspace = attemptWorkspace(state) ?? contract.cwd;
111
- const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node)];
141
+ const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node, settledSiblingIds(runDir, contract, node))];
112
142
  /** @param {VerificationAttempt} attempt @returns {VerificationProgress} */
113
143
  const progressFor = (attempt) => verificationProgress(attempt.commandIndex + 1, commands.length, /** @type {VerificationCommand|undefined} */ (commands[attempt.commandIndex])?.argv);
114
144
  try {
@@ -221,16 +251,37 @@ function argvEqual(a, b) {
221
251
  if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length) return false;
222
252
  return a.every((item, index) => item === b[index]);
223
253
  }
254
+ /**
255
+ * Marks whether the integration candidate's own verification pass is running
256
+ * right now, nested inside the attempt's own already-completed verification
257
+ * record rather than as a new node-snapshot field: that record's validator
258
+ * (`validateVerificationSnapshot`) accepts extra keys, where the node
259
+ * snapshot's own strict field list would reject one. Status surfaces read it
260
+ * to tell "still verifying the sealed candidate" apart from "waiting on the
261
+ * judge", which otherwise both read as whatever phase the attempt last
262
+ * dispatched under.
263
+ *
264
+ * @param {string} runDir
265
+ * @param {NodeSnapshot} state
266
+ * @param {LockHandle|null} lock
267
+ * @param {boolean} active
268
+ */
269
+ function markCandidateVerification(runDir, state, lock, active) {
270
+ state.verification = /** @type {VerificationState} */ ({ ...(state.verification ?? { passed: false, commands: [], completed: false }), candidate: active });
271
+ writeNode(runDir, state, lock);
272
+ }
224
273
  /**
225
274
  * @param {ValidatedContract} contract
226
275
  * @param {ValidatedNode} node
227
276
  * @param {NodeSnapshot} state
228
277
  * @param {string} runDir
229
278
  * @param {string} workspace
279
+ * @param {LockHandle|null} [lock]
230
280
  * @returns {Promise<import("../repo/integrate.mjs").CandidateEvidence>}
231
281
  */
232
- export async function verifyCandidateWorkspace(contract, node, state, runDir, workspace) {
233
- const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node)];
282
+ export async function verifyCandidateWorkspace(contract, node, state, runDir, workspace, lock = null) {
283
+ const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node, settledSiblingIds(runDir, contract, node))];
284
+ markCandidateVerification(runDir, state, lock, true);
234
285
  try {
235
286
  const result = await runVerification(commands, workspace, {
236
287
  logDir: join(runDir, "logs", `${node.id}.${state.attempt}.candidate-verification`),
@@ -252,5 +303,7 @@ export async function verifyCandidateWorkspace(contract, node, state, runDir, wo
252
303
  return settled.retried ? { ...compacted, retried: settled.retried } : compacted;
253
304
  } catch (error) {
254
305
  return { passed: false, error: boundedUtf8(errorMessage(error), 4 * 1024) };
306
+ } finally {
307
+ markCandidateVerification(runDir, state, lock, false);
255
308
  }
256
309
  }
@@ -503,7 +503,7 @@ export function promoteRun({ repo, runId, landBranch, runHead, baseSha, finalVer
503
503
  if (!head) throw refusal(`refusing to promote ${runId}: the run ref is unavailable`, "run_ref_missing");
504
504
  const checkedOutAt = worktreeCheckedOutAt(repo, landBranch);
505
505
  if (checkedOutAt) {
506
- throw refusal(`refusing to promote ${landBranch}: it is checked out in ${checkedOutAt}`, "land_branch_checked_out");
506
+ throw refusal(`refusing to promote ${landBranch}: it is checked out in ${checkedOutAt}; ${landBranchCheckedOutRemedy(checkedOutAt)}`, "land_branch_checked_out");
507
507
  }
508
508
  const branchRef = `refs/heads/${landBranch}`;
509
509
  const current = gitHead(repo, branchRef);
@@ -540,6 +540,11 @@ function refusal(message, code) {
540
540
  return Object.assign(new Error(message), { code });
541
541
  }
542
542
 
543
+ /** @param {string} checkedOutAt @returns {string} */
544
+ function landBranchCheckedOutRemedy(checkedOutAt) {
545
+ return `detach the checkout at ${checkedOutAt} or run the coordinator from a worktree`;
546
+ }
547
+
543
548
  /** @param {string} path @returns {string} */
544
549
  function readTextFile(path) {
545
550
  return readFileSync(path, "utf8");
@@ -26,7 +26,7 @@ const POINTER_ATTENTION_CHARS = 80;
26
26
  /** @typedef {{costUsd: number|null, costProvenance: CostProvenance, inputTokens: number, outputTokens: number, cacheReadInputTokens: number, pricedInvocations: number, unpricedInvocations: number}} RoleUsage */
27
27
  /** @typedef {{inputTokens: number|null, outputTokens: number|null, cacheReadInputTokens: number|null}} StatusPayloadUsage */
28
28
  /** @typedef {{index: number, total: number, argv: string}} VerificationProgress */
29
- /** @typedef {{id: string, status: NodeStatus, phase: string|null, executionPhase: string|null, runtime: string|null, workerRuntime: string|null, continuation: string, attempt: number, revisions: number, startedAt: string|null, updatedAt: string|null, usage: StatusPayloadUsage|null, costUsd: number|null, verdict: string|null, pendingHandoff: {runtime: string, reason: string}|null, note: string|null, scopeFindings: string[]|null, errorCode: string|null, blockedBy: string[], verificationProgress: VerificationProgress|null, declaredReadBytes: number|null}} StatusPayloadNode */
29
+ /** @typedef {{id: string, status: NodeStatus, phase: string|null, executionPhase: string|null, runtime: string|null, workerRuntime: string|null, continuation: string, attempt: number, revisions: number, startedAt: string|null, updatedAt: string|null, usage: StatusPayloadUsage|null, costUsd: number|null, verdict: string|null, gateOutcome: "passed"|"rejected"|null, pendingHandoff: {runtime: string, reason: string}|null, note: string|null, scopeFindings: string[]|null, errorCode: string|null, blockedBy: string[], verificationProgress: VerificationProgress|null, declaredReadBytes: number|null}} StatusPayloadNode */
30
30
  /** @typedef {{schemaVersion: 1, run: string, contractId: string, campaignId: string, goal: string, usage: {inputTokens: number, outputTokens: number, cacheReadInputTokens: number, costUsd: number|null}, roles: {worker: RoleUsage, judge: RoleUsage}, controller: JsonObject, identityWarnings: string[], summary: string, nodes: StatusPayloadNode[]}} StatusPayload */
31
31
 
32
32
  /** The glyph each terminal state prints in a status table. */
@@ -43,13 +43,11 @@ export const MARK = {
43
43
  };
44
44
 
45
45
  /**
46
- * `status <run-dir>`: everything the operator needs, in the same order as
47
- * the dashboard page (TECH-SPEC lean, section 4's last paragraph)
48
- * needs-you, now, nodes, cost. Every cell comes from the same payload
49
- * `status --json` and the per-run `status.json` file emit
50
- * (`buildStatusPayload`); `nodes` and `identityWarnings` from `loadRun` are
51
- * used only for the two facts the payload does not carry: throwing on an
52
- * invalid persisted snapshot, and `controllerStatus`'s lock read.
46
+ * `status <run-dir>`: needs-you, now, nodes, cost, in the dashboard's order
47
+ * (TECH-SPEC lean, section 4's last paragraph). Every cell comes from the same
48
+ * payload `status --json` emits (`buildStatusPayload`); `nodes` and
49
+ * `identityWarnings` from `loadRun` cover only what the payload does not:
50
+ * throwing on an invalid snapshot, and `controllerStatus`'s lock read.
53
51
  *
54
52
  * @param {string} runDir
55
53
  * @returns {string}
@@ -75,7 +73,7 @@ export function renderStatus(runDir) {
75
73
  const widths = [3, 24, 9, 3, 28, 8, 6, 6, 6, 10, 9, MAX_NOTE_LENGTH];
76
74
  /** @type {(cells: unknown[]) => string} */
77
75
  const row = (cells) => cells.map((cell, i) => fit(String(cell ?? ""), widths[i])).join(" ");
78
- lines.push("```", row(["", "NODE", "STATE", "TRY", "RUNTIME", "ELAPSED", "IN", "CACHE", "OUT", "USD", "VERDICT", "NOTE"]), row(widths.map((width) => "-".repeat(width))));
76
+ lines.push("```", row(["", "NODE", "STATE", "TRY", "RUNTIME", "ELAPSED", "IN", "CACHE", "OUT", "USD", "GATE", "NOTE"]), row(widths.map((width) => "-".repeat(width))));
79
77
  for (const node of payload.nodes) {
80
78
  lines.push(row([
81
79
  MARK[node.status] ?? "[?]",
@@ -88,7 +86,7 @@ export function renderStatus(runDir) {
88
86
  compactTokens(node.usage?.cacheReadInputTokens),
89
87
  compactTokens(node.usage?.outputTokens),
90
88
  compactCost(node.costUsd),
91
- node.verdict ?? "-",
89
+ node.gateOutcome ?? "-",
92
90
  node.note ?? "-",
93
91
  ]));
94
92
  }
@@ -98,12 +96,10 @@ export function renderStatus(runDir) {
98
96
  }
99
97
 
100
98
  /**
101
- * The node the operator should look at right now, formatted the same way
102
- * the dashboard's now strip is (TECH-SPEC lean, section 4, item 1): the
103
- * active node's elapsed time and cost so far, or an idle line once every
104
- * node has settled. A node awaiting a controller verification command
105
- * appends which one, `k/n`, and its argv, so a minutes-long suite is not
106
- * silent between ticks.
99
+ * The node the operator should look at right now, formatted like the
100
+ * dashboard's now strip (TECH-SPEC lean, section 4, item 1): elapsed time and
101
+ * cost so far, or idle once every node has settled. A node awaiting a
102
+ * controller verification command appends which one, `k/n`, and its argv.
107
103
  *
108
104
  * @param {StatusPayload} payload
109
105
  * @param {number} now epoch ms
@@ -114,7 +110,8 @@ function nowLine(payload, now) {
114
110
  if (active) {
115
111
  const progress = active.verificationProgress;
116
112
  const verification = progress ? ` · verification ${progress.index}/${progress.total} · ${progress.argv}` : "";
117
- return `now: ${active.id} ${active.status} (${formatElapsed(active, now)}) · ${active.runtime ?? "-"} · ${compactCost(active.costUsd)}${verification}`;
113
+ const candidate = !progress && active.executionPhase === "candidate" ? " · candidate verification" : "";
114
+ return `now: ${active.id} ${active.status} (${formatElapsed(active, now)}) · ${active.runtime ?? "-"} · ${compactCost(active.costUsd)}${verification}${candidate}`;
118
115
  }
119
116
  const allTerminal = payload.nodes.every((node) => SUCCESS.has(node.status));
120
117
  return allTerminal ? `now: idle · run done · ${compactCost(payload.usage.costUsd)}` : "now: idle";
@@ -134,6 +131,27 @@ function verificationProgress(node) {
134
131
  return verification?.progress ?? null;
135
132
  }
136
133
 
134
+ /** Set by `verifyCandidateWorkspace` (engine/verify.mjs) for the run of the sealed candidate's own re-verification pass. @param {NodeSnapshot} node @returns {boolean} */
135
+ function candidateVerificationActive(node) {
136
+ const verification = /** @type {{candidate?: boolean}|null|undefined} */ (node.verification);
137
+ return verification?.candidate === true;
138
+ }
139
+
140
+ /** `candidate` while re-verifying the sealed candidate, `verification` mid-attempt-verification, else the persisted phase; the two never overlap. @param {NodeSnapshot} node @returns {string|null} */
141
+ function executionPhaseOf(node) {
142
+ if (candidateVerificationActive(node)) return "candidate";
143
+ if (verificationProgress(node)) return "verification";
144
+ return node.phase;
145
+ }
146
+
147
+ /** What the gate decided (independent of `verdict`: an advisory review or a below-threshold fail verdict still accepts the node, engine/review.mjs `applyJudgeResult`). @param {NodeSnapshot} node @returns {"passed"|"rejected"|null} */
148
+ export function gateOutcome(node) {
149
+ if (!node.gate) return null;
150
+ if (SUCCESS.has(node.status)) return "passed";
151
+ if (node.status === "failed" || node.status === "exhausted") return "rejected";
152
+ return null;
153
+ }
154
+
137
155
  /**
138
156
  * A node's wall-clock elapsed time: `startedAt` to `updatedAt` once it has
139
157
  * settled into a terminal state, `startedAt` to `now` while it is still
@@ -216,7 +234,7 @@ function buildStatusPayload(runDir, contract, nodes, identityWarnings, usage) {
216
234
  // A running controller verification command does not move
217
235
  // `node.phase` (it stays `worker`, the phase that dispatched it), so
218
236
  // the surface that shows it is computed here rather than persisted.
219
- executionPhase: progress ? "verification" : node.phase,
237
+ executionPhase: executionPhaseOf(node),
220
238
  runtime: node.runtime ? `${node.runtime.harness}/${node.runtime.model}` : null,
221
239
  workerRuntime: workerRuntimeLabel(node),
222
240
  continuation: continuationMode(node),
@@ -227,6 +245,7 @@ function buildStatusPayload(runDir, contract, nodes, identityWarnings, usage) {
227
245
  usage: node.usage ? { inputTokens: node.usage.inputTokens ?? null, outputTokens: node.usage.outputTokens ?? null, cacheReadInputTokens: node.usage.cacheReadInputTokens ?? null } : null,
228
246
  costUsd: typeof node.costUsd === "number" ? node.costUsd : null,
229
247
  verdict: node.gate?.verdict ?? null,
248
+ gateOutcome: gateOutcome(node),
230
249
  pendingHandoff: pendingHandoff(node),
231
250
  note: statusNote(node),
232
251
  scopeFindings: node.scopeFindings?.unexpectedPaths ?? null,
@@ -256,16 +275,11 @@ function activeStatusNode(nodes) {
256
275
  }
257
276
 
258
277
  /**
259
- * The bounded pointer record written to `.runs/status.json`: enough for a
260
- * quick ambient read (statusline, a stale watcher) without opening the
261
- * per-run status.json. Bounded the same way heartbeat.json used to be, so a
262
- * cheap bounded read stays valid for any reader still built that way.
263
- * `generatedAt` is unix seconds, not ISO: the statusline's no-jq fallback has
264
- * no clock and only jq's builtin `now` can compute an age from a live clock,
265
- * and neither path needs a `date` process to read an integer. `elapsedSec`
266
- * is likewise precomputed here (as of `generatedAt`, not live) so the
267
- * statusline never has to subtract two timestamps to show it — it just
268
- * prints the integer, whichever reader it is.
278
+ * The bounded pointer record written to `.runs/status.json`, for a quick
279
+ * ambient read without opening the per-run status.json (bounded like the old
280
+ * heartbeat.json). `generatedAt` is unix seconds, not ISO, and `elapsedSec` is
281
+ * precomputed from it, since the no-jq statusline fallback has no clock to
282
+ * subtract two timestamps with.
269
283
  *
270
284
  * @param {JsonObject} payload the per-run status.json payload
271
285
  * @param {NodeSnapshot[]} nodes
@@ -340,10 +354,7 @@ export function writeStatusArtifacts(runDir, runsDir, contract, states) {
340
354
  */
341
355
  export function controllerStatus(runDir, nodes) {
342
356
  const lock = readLock(runDir);
343
- // The heartbeat's `at` is the live tick, so a working controller and a dead
344
- // one are distinguishable on disk. Before this it was hard-coded null here
345
- // and computed from node updates only once the lock was already stale, which
346
- // is why all 47 recorded status.json files reported a null lastTick.
357
+ // The heartbeat's `at` distinguishes a working controller from a dead one on disk.
347
358
  const heartbeat = readHeartbeat(runDir);
348
359
  const lastTick = typeof heartbeat?.at === "string" ? heartbeat.at : null;
349
360
  if (!lock || /** @type {{invalid?: true}} */ (lock).invalid) {
@@ -473,14 +484,11 @@ export function renderFindings(runDir) {
473
484
  }
474
485
 
475
486
  /**
476
- * A node the worker itself stopped on, rendered as the repair it asks for.
477
- *
478
- * `findings` used to answer only gate exhaustion, and a run whose nodes all
479
- * stopped on `blocked_context` reported nothing to act on while the workers
480
- * had each named exactly what they needed measured on a four-node campaign
481
- * where three nodes were blocked and the command printed one line saying so.
482
- * The question is already structured, so the repair is mechanical: put the
483
- * named paths in the packet's `readFiles` and take a new run id.
487
+ * A node the worker itself stopped on, rendered as the repair it asks for:
488
+ * `findings` used to answer only gate exhaustion, so a run whose nodes all
489
+ * stopped on `blocked_context` reported nothing to act on (measured on a
490
+ * four-node campaign with three nodes blocked). The repair is mechanical: put
491
+ * the named paths in the packet's `readFiles` and take a new run id.
484
492
  *
485
493
  * @param {NodeSnapshot} node
486
494
  * @returns {string|null}
@@ -553,16 +561,11 @@ function continuationMode(node) {
553
561
  }
554
562
 
555
563
  /**
556
- * Who produced this node's work, for the RUNTIME column of both tables.
557
- *
558
- * `state.runtime` is the last runtime *dispatched*, and the judge dispatch
559
- * overwrites the worker's: every gated node that reached its gate therefore
560
- * reported its judge as the runtime, and a six-node campaign whose workers
561
- * were five different harnesses rendered as though three of them had never
562
- * run (measured 2026-09-13). The invocation ledger keeps both roles, so the
563
- * label is derived from it: the worker that ran, falling back to the live
564
- * runtime for a node that has not dispatched one yet. The judge is not lost —
565
- * it owns the VERDICT column, and `nowLine` still names whatever is running.
564
+ * Who produced this node's work, for the RUNTIME column: `state.runtime` is
565
+ * the last runtime *dispatched*, and the judge dispatch overwrites the
566
+ * worker's (measured 2026-09-13: a six-harness campaign rendered as though
567
+ * three workers never ran), so this reads the invocation ledger for the
568
+ * worker that ran, falling back to the live runtime if none dispatched yet.
566
569
  *
567
570
  * @param {NodeSnapshot} node
568
571
  * @returns {string|null}
@@ -633,7 +636,7 @@ function boundedNote(segments, maxLength = MAX_NOTE_LENGTH) {
633
636
  export function statusNote(node) {
634
637
  const scope = scopeFindingsNote(node.scopeFindings);
635
638
  const review = reviewNote(node);
636
- const detail = node.gate?.summary ?? node.error?.message ?? node.blockedBy?.join(", ") ?? node.phase;
639
+ const detail = node.gate?.summary ?? node.error?.message ?? node.blockedBy?.join(", ") ?? (candidateVerificationActive(node) ? "candidate" : node.phase);
637
640
  const note = boundedNote([review, detail]);
638
641
  if (!scope) return note;
639
642
  return boundedNote([scope, note]);
@@ -657,13 +660,10 @@ function nodeNote(node) {
657
660
  /** @typedef {{costUsd: number|null, status: "known"|"estimated"|"ambiguous"}} CostProjection */
658
661
 
659
662
  /**
660
- * Per-role usage from the invocation ledger alone, with the provenance that
661
- * says whether the role's cost total is honest. A role whose invocations all
662
- * carry a provider cost is `priced` and its `costUsd` is the sum; a role with
663
- * any unpriced invocation is `partial` (some priced) or `unpriced` (none), and
664
- * its `costUsd` stays null rather than summing the known half and understating
665
- * it. A role with no invocation is `none`. Token totals are always carried, so
666
- * a role the provider would not price still reads as work.
663
+ * Per-role usage from the invocation ledger, with provenance for whether the
664
+ * cost total is honest: `priced` (all invocations costed), `partial`/`unpriced`
665
+ * (some/none costed, so `costUsd` stays null rather than understating), or
666
+ * `none` (no invocation). Token totals are always carried.
667
667
  *
668
668
  * @param {NodeSnapshot[]} nodes
669
669
  * @returns {{worker: RoleUsage, judge: RoleUsage}}
@@ -185,18 +185,18 @@ function tabBodyHtml(tab, detail) {
185
185
  function drawerHtml(drawer, tab) {
186
186
  if (!drawer) return `<section id="drawer"><h2>Run drawer</h2><div class="card" style="padding:16px 18px"><p class="empty">select a run above</p></div></section>`;
187
187
  const rows = drawer.nodes.map((node) => `<tr class="clickable ${node.id === drawer.selectedNodeId ? "selected" : ""}" data-node="${esc(node.id)}">
188
- <td>${pill(node.status)}</td>
188
+ <td>${pill(node.status)}${node.executionPhase === "candidate" ? ` <span class="dim">· candidate verification</span>` : node.executionPhase === "verification" ? ` <span class="dim">· verification</span>` : ""}</td>
189
189
  <td class="mono">${esc(node.id)}</td>
190
190
  <td class="num">${node.attempt}</td>
191
191
  <td class="mono">${esc(node.model ?? "–")}</td>
192
192
  <td class="num">${fmtDur(node.elapsedMs)}</td>
193
193
  <td class="num">${fmtUsd(node.costUsd)}</td>
194
- <td>${node.verdict ? pill(node.verdict) : ""}</td>
194
+ <td>${node.gateOutcome ? pill(node.gateOutcome) : ""}</td>
195
195
  <td class="goal" title="${esc(node.note ?? "")}">${esc(node.note ?? "")}</td>
196
196
  </tr>`).join("");
197
197
  const tabs = TABS.map(([key, label]) => `<button data-tab="${key}" class="${tab === key ? "active" : ""}">${esc(label)}</button>`).join("");
198
198
  return `<section id="drawer"><h2>Run drawer · ${esc(drawer.runId)}</h2><div class="card" style="padding:16px 18px">
199
- <table><thead><tr><th>state</th><th>node</th><th>try</th><th>model</th><th>elapsed</th><th>USD</th><th>verdict</th><th>note</th></tr></thead>
199
+ <table><thead><tr><th>state</th><th>node</th><th>try</th><th>model</th><th>elapsed</th><th>USD</th><th>gate</th><th>note</th></tr></thead>
200
200
  <tbody>${rows}</tbody></table>
201
201
  <div class="tabs">${tabs}</div>
202
202
  <div id="tabbody">${tabBodyHtml(tab, drawer.detail)}</div>
@@ -294,7 +294,8 @@ function nodeRow(runDir, statusNode) {
294
294
  model: node.runtime?.model ?? null,
295
295
  elapsedMs: elapsedMsBetween(node.startedAt, closed ? node.updatedAt : null),
296
296
  costUsd: typeof node.costUsd === "number" ? node.costUsd : null,
297
- verdict: node.gate?.verdict ?? null,
297
+ gateOutcome: statusNode.gateOutcome ?? null,
298
+ executionPhase: statusNode.executionPhase ?? null,
298
299
  note: statusNode.note ?? null,
299
300
  errorCode: statusNode.errorCode ?? null,
300
301
  blockedBy: statusNode.blockedBy ?? [],