faberun 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +193 -31
- package/package.json +1 -1
- package/src/campaign/chain.mjs +19 -0
- package/src/cli.mjs +4 -3
- package/src/contract/final-verification.mjs +41 -10
- package/src/engine/scheduler.mjs +199 -26
- package/src/engine/settle.mjs +1 -1
- package/src/engine/verify.mjs +57 -4
- package/src/repo/integrate.mjs +6 -1
- package/src/report/render.mjs +58 -58
- package/src/web/index.html +3 -3
- package/src/web/server.mjs +2 -1
package/README.md
CHANGED
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
<h1 align="center">faberun</h1>
|
|
4
4
|
|
|
5
|
-
<p align="center">From intent to running software.</p>
|
|
5
|
+
<p align="center"><strong>From intent to running software.</strong></p>
|
|
6
|
+
|
|
7
|
+
<p align="center">Graph engineering for coding agents.</p>
|
|
6
8
|
|
|
7
9
|
<p align="center">
|
|
8
10
|
<a href="https://github.com/feliperun/faberun/actions/workflows/ci.yml"><img src="https://github.com/feliperun/faberun/actions/workflows/ci.yml/badge.svg" alt="CI status"></a>
|
|
@@ -10,21 +12,162 @@
|
|
|
10
12
|
<a href="https://www.npmjs.com/package/faberun"><img src="https://img.shields.io/npm/v/faberun" alt="npm version"></a>
|
|
11
13
|
</p>
|
|
12
14
|
|
|
13
|
-
Faberun is a
|
|
14
|
-
|
|
15
|
-
dependencies,
|
|
16
|
-
|
|
17
|
-
|
|
15
|
+
Faberun is a graph-engineering runtime for building software with coding agents.
|
|
16
|
+
It sits above the models and the harnesses and turns an implementation plan into
|
|
17
|
+
a durable graph of tasks, dependencies, workers, proofs, reviews, retries and
|
|
18
|
+
integration gates.
|
|
19
|
+
|
|
20
|
+
Harnesses are execution environments: Claude Code, Codex, the Gemini, DeepSeek
|
|
21
|
+
and GLM shells, and a generic `exec-jsonl` door for whatever comes next. Models
|
|
22
|
+
are the engines inside them. Faberun owns the graph, the state and the
|
|
23
|
+
definition of done around both.
|
|
24
|
+
|
|
25
|
+
The goal is not to make a group of agents talk to each other. It is to use the
|
|
26
|
+
right model for each kind of work, keep the process independent of any single
|
|
27
|
+
model or harness, and require evidence before software counts as done.
|
|
28
|
+
|
|
29
|
+
## Why Faberun
|
|
30
|
+
|
|
31
|
+
Coding agents are already very good at changing repositories. The remaining
|
|
32
|
+
problem is everything around that execution.
|
|
33
|
+
|
|
34
|
+
A frontier model earns its cost when it reasons about architecture, challenges a
|
|
35
|
+
design, writes a specification or reviews a difficult decision. The same model
|
|
36
|
+
is usually unnecessary for a small, well-defined implementation task a much
|
|
37
|
+
cheaper one performs well. Without an orchestration layer, though, the model
|
|
38
|
+
driving the current session becomes the model doing everything.
|
|
39
|
+
|
|
40
|
+
Faberun separates those concerns. One campaign can use a frontier model to
|
|
41
|
+
reason about the problem and author a plan, a second strong model to attack that
|
|
42
|
+
plan, lower-cost models to execute well-defined implementation nodes, a model
|
|
43
|
+
from another vendor to judge the work that genuinely needs judgment, and plain
|
|
44
|
+
deterministic commands wherever no model is needed at all. The expensive
|
|
45
|
+
intelligence goes where it matters, and the rest becomes an execution problem.
|
|
46
|
+
|
|
47
|
+
It also takes the campaign out of the lifetime of one chat. The work has durable
|
|
48
|
+
state on disk, so Claude Code can start a campaign and Codex, DeepSeek, GLM or
|
|
49
|
+
any other adapted harness can continue it later without reconstructing the
|
|
50
|
+
process from a conversation history.
|
|
51
|
+
|
|
52
|
+
## Graph engineering
|
|
18
53
|
|
|
19
|
-
Faberun
|
|
20
|
-
|
|
21
|
-
models are engines; Faberun sits above them. It keeps the intent, coordinates
|
|
22
|
-
the work, tracks what was actually completed, validates the result and decides
|
|
23
|
-
what should happen next.
|
|
54
|
+
Faberun treats development as an executable graph rather than a long
|
|
55
|
+
conversation.
|
|
24
56
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
57
|
+
```text
|
|
58
|
+
intent
|
|
59
|
+
└─ campaign
|
|
60
|
+
└─ contract
|
|
61
|
+
├─ node A ── proof ── review ──┐
|
|
62
|
+
├─ node B ── proof ── review ──┼── integration ── promotion
|
|
63
|
+
└─ node C ── proof ── review ──┘
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
A campaign carries the durable intent. A contract turns part of that intent into
|
|
67
|
+
a schema-versioned DAG. Each node carries a closed packet: what the worker may
|
|
68
|
+
read, what it may change, what done means, and how that claim has to be proven.
|
|
69
|
+
The graph decides what can run, what must wait, what may retry, what needs a
|
|
70
|
+
human and what is allowed to advance. The models are replaceable participants
|
|
71
|
+
inside it. The graph remains.
|
|
72
|
+
|
|
73
|
+
## What is different
|
|
74
|
+
|
|
75
|
+
### Use the right model for the job
|
|
76
|
+
|
|
77
|
+
A runtime is one harness running one model. Workers, judges and planning
|
|
78
|
+
sessions need not share a provider or a capability tier, so high-cost reasoning
|
|
79
|
+
and low-cost execution live in the same campaign instead of one model owning the
|
|
80
|
+
whole process. Changing providers is a routing decision, not a rewrite of the
|
|
81
|
+
workflow.
|
|
82
|
+
|
|
83
|
+
### Proof before opinion
|
|
84
|
+
|
|
85
|
+
A worker saying "done" is not evidence that it is. Every definition-of-done item
|
|
86
|
+
declares how it can be proven: a command that must succeed, a path that must
|
|
87
|
+
exist, or — only when the criterion cannot be mechanised — the judgment of a
|
|
88
|
+
judge. Mechanical verification always runs first, and a node whose proofs are
|
|
89
|
+
all mechanical settles without spending a token on review. A gate may also
|
|
90
|
+
declare `skipWhen`, so a change that is green and small enough skips the judge
|
|
91
|
+
by policy rather than by accident.
|
|
92
|
+
|
|
93
|
+
### Independent review
|
|
94
|
+
|
|
95
|
+
When judgment is required, the worker does not grade itself. Validation refuses
|
|
96
|
+
a gated node whose worker and judge resolve to the same vendor. The judge reads
|
|
97
|
+
the recorded result and its evidence, never the reasoning that produced it, and
|
|
98
|
+
never re-runs the work. The point is not to make two agents agree; it is to keep
|
|
99
|
+
implementation and acceptance as separate responsibilities.
|
|
100
|
+
|
|
101
|
+
### Isolated execution
|
|
102
|
+
|
|
103
|
+
Each attempt runs in its own git worktree, so independent nodes run at the same
|
|
104
|
+
time without sharing a mutable working tree, and accepted work reaches the
|
|
105
|
+
repository through controlled refs instead of being copied into the operator's
|
|
106
|
+
checkout. The operator keeps working while the campaign runs.
|
|
107
|
+
|
|
108
|
+
### Durable campaigns
|
|
109
|
+
|
|
110
|
+
A run is not tied to the terminal that launched it. The controller runs
|
|
111
|
+
detached, state is persisted under `.runs/`, `supervise` resumes a controller
|
|
112
|
+
that died, and a campaign carries its intent across many runs and sessions. If a
|
|
113
|
+
provider goes down or a harness session ends, the work does not have to be
|
|
114
|
+
re-authored.
|
|
115
|
+
|
|
116
|
+
### Safe integration
|
|
117
|
+
|
|
118
|
+
Passing a worker's own checks is not the end of the process. An accepted attempt
|
|
119
|
+
is sealed, integrated onto the run ref and verified again in the candidate
|
|
120
|
+
state, and the phase-terminal node runs the contract's final verification. If
|
|
121
|
+
combining individually valid changes breaks the system, the campaign does not
|
|
122
|
+
promote them.
|
|
123
|
+
|
|
124
|
+
### Measurable model economics
|
|
125
|
+
|
|
126
|
+
Every invocation records its usage, routing and outcome, and a closed campaign
|
|
127
|
+
keeps that ledger in its own record under `docs/campaigns/<id>/ledger/`. It makes
|
|
128
|
+
questions like these measurable instead of anecdotal: which model handles this
|
|
129
|
+
class of task reliably, where a frontier model is actually worth its price,
|
|
130
|
+
which nodes close mechanically with no judge at all, how many revisions a worker
|
|
131
|
+
needs before acceptance, what one closed checkpoint costs, and how often a
|
|
132
|
+
fallback saves a campaign. The objective is not to spend less. It is to know
|
|
133
|
+
where expensive intelligence creates value.
|
|
134
|
+
|
|
135
|
+
## Campaigns, not synthetic teams
|
|
136
|
+
|
|
137
|
+
Faberun does not model an agent process as a human organisation. It needs no "AI
|
|
138
|
+
architect", "AI developer", "AI QA engineer" and "AI product manager" merely
|
|
139
|
+
because human teams carry those titles: models are general enough that those
|
|
140
|
+
boundaries are mostly artificial. The roles it does define are the ones that
|
|
141
|
+
stay useful for agents.
|
|
142
|
+
|
|
143
|
+
| role | responsibility |
|
|
144
|
+
| --- | --- |
|
|
145
|
+
| worker | produces a result inside its packet. |
|
|
146
|
+
| judge | decides independently whether a result that needs judgment satisfies its contract. |
|
|
147
|
+
| controller | advances the graph deterministically. |
|
|
148
|
+
| operator | supplies the intent and intervenes when a decision genuinely needs a human. |
|
|
149
|
+
|
|
150
|
+
Everything else belongs in the contract.
|
|
151
|
+
|
|
152
|
+
## From spec to contract
|
|
153
|
+
|
|
154
|
+
Planning is part of the product, and it is adversarial on purpose. `faberun spec
|
|
155
|
+
validate` checks a specification's front matter, its `R<n>` requirements and the
|
|
156
|
+
proof each one declares, and invokes no model to do it. `faberun plan` then runs
|
|
157
|
+
the debate in budgeted, isolated runs: an author drafts, a reviewer from another
|
|
158
|
+
vendor attacks the draft holding only the spec, the repository facts and the
|
|
159
|
+
plan under review, and a revision round follows while a critical finding
|
|
160
|
+
remains. A converged draft is sized, routed and frozen into a contract. Freezing
|
|
161
|
+
never launches, and a plan that never converges ends `contested`, with its open
|
|
162
|
+
findings recorded as a campaign question instead of a contract.
|
|
163
|
+
|
|
164
|
+
None of it is mandatory. The traceability rules are advisory unless you ask for
|
|
165
|
+
`--strict-traceability`, a document with no front matter validates as `legacy`
|
|
166
|
+
rather than being rejected, and you can skip planning altogether and author the
|
|
167
|
+
contract by hand. What Faberun needs is the executable part of the plan:
|
|
168
|
+
objective, nodes, dependencies, read and write scope, definitions of done,
|
|
169
|
+
verification, runtime policy and integration behaviour. Your specification
|
|
170
|
+
process stays yours.
|
|
28
171
|
|
|
29
172
|
## Install
|
|
30
173
|
|
|
@@ -92,29 +235,35 @@ The operator writes the intent into a campaign and an authored contract, a
|
|
|
92
235
|
schema-versioned DAG whose nodes each carry a closed task packet. The controller
|
|
93
236
|
schedules every dependency-ready node and dispatches its packet to a worker
|
|
94
237
|
inside an attempt worktree, where the worker sees only the files the packet
|
|
95
|
-
names.
|
|
96
|
-
|
|
97
|
-
A passing attempt is sealed and integrated onto the run ref, a campaign promotes
|
|
98
|
-
each run onto its landing branch, and the orchestrator lands that branch.
|
|
99
|
-
Campaigns, handoffs and the `supervise` watchdog carry the work across sessions,
|
|
100
|
-
so an interrupted run is continued in place instead of being re-authored.
|
|
238
|
+
names. Faberun then judges the result in stages, and a free slot dispatches the
|
|
239
|
+
next node while another one is still being verified.
|
|
101
240
|
|
|
102
241
|
```text
|
|
103
|
-
|
|
104
|
-
└─
|
|
105
|
-
└─
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
242
|
+
packet
|
|
243
|
+
└─ worker attempt in its own worktree
|
|
244
|
+
└─ mechanical proof ── failure ─→ retry · route · attention
|
|
245
|
+
└─ judge, when required ── rejection ─→ revision
|
|
246
|
+
└─ seal ─→ integration ref ─→ candidate verification
|
|
247
|
+
└─ promotion onto the campaign's landing branch
|
|
109
248
|
```
|
|
110
249
|
|
|
111
|
-
|
|
112
|
-
|
|
250
|
+
A judge reviews recorded evidence rather than re-running arbitrary work. A
|
|
251
|
+
passing attempt is sealed and integrated onto the run ref, a campaign promotes
|
|
252
|
+
each finished run onto its landing branch, and the orchestrator lands that
|
|
253
|
+
branch. Campaigns, handoffs and the `supervise` watchdog carry the work across
|
|
254
|
+
sessions, so an interrupted run is continued in place instead of being
|
|
255
|
+
re-authored.
|
|
113
256
|
|
|
114
|
-
|
|
257
|
+
The vocabulary is in [Concepts](docs/CONCEPTS.md), and the layers, process model
|
|
258
|
+
and gates are in [Architecture](docs/ARCHITECTURE.md).
|
|
115
259
|
|
|
116
|
-
|
|
117
|
-
|
|
260
|
+
## Harnesses and models
|
|
261
|
+
|
|
262
|
+
Faberun keeps three things apart. A **model** is the reasoning engine, a
|
|
263
|
+
**harness** is the environment that gives a model access to code and tools, and
|
|
264
|
+
a **runtime** is one configured harness-and-model pair Faberun can dispatch. Any
|
|
265
|
+
runtime can act as a worker or as a judge, according to the contract and the
|
|
266
|
+
routing policy.
|
|
118
267
|
|
|
119
268
|
| Harness | Default vendor | Example model |
|
|
120
269
|
| --- | --- | --- |
|
|
@@ -126,6 +275,19 @@ judge of another vendor reviews what it produced.
|
|
|
126
275
|
| `exec-jsonl` | declared per runtime | the model its command names |
|
|
127
276
|
| `replay` | declared per runtime | the recorded model |
|
|
128
277
|
|
|
278
|
+
Harness and model support are adapter concerns. The orchestration above them
|
|
279
|
+
depends on no single vendor.
|
|
280
|
+
|
|
281
|
+
## Philosophy
|
|
282
|
+
|
|
283
|
+
Software should be built, not merely generated. Generating plausible code is
|
|
284
|
+
becoming cheap; the valuable part is the system around the generation —
|
|
285
|
+
preserving intent, decomposing work, choosing the appropriate intelligence,
|
|
286
|
+
isolating effects, proving outcomes, recovering from failure, and recording why a
|
|
287
|
+
change was accepted. A craftsman does not depend on one hammer, and Faberun does
|
|
288
|
+
not depend on one model. Models change, harnesses change, providers change. The
|
|
289
|
+
intent, the graph, the evidence and the finished work remain.
|
|
290
|
+
|
|
129
291
|
## Documentation
|
|
130
292
|
|
|
131
293
|
- [Documentation map](docs/README.md): every document and the question it answers.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "faberun",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.0",
|
|
4
4
|
"description": "Faberun is a development orchestration system that turns intent into verified software: harness- and model-agnostic, it keeps the intent, coordinates the work, verifies the result and decides what happens next.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/src/campaign/chain.mjs
CHANGED
|
@@ -290,6 +290,25 @@ export function validateContractAgainstRef(raw, contractPath, context = {}) {
|
|
|
290
290
|
}
|
|
291
291
|
}
|
|
292
292
|
|
|
293
|
+
/**
|
|
294
|
+
* Validate a contract for a launch, the one path the CLI's `run` command and
|
|
295
|
+
* the scheduler's own re-validation both go through: against `baseRef` when
|
|
296
|
+
* one is given, the checkout otherwise. `repo` defaults to the contract's own
|
|
297
|
+
* `cwd` when the caller does not know a better one (the campaign chain always
|
|
298
|
+
* passes its own), since a contract's `cwd` is always inside the repo it
|
|
299
|
+
* names.
|
|
300
|
+
*
|
|
301
|
+
* @param {Record<string, unknown>} raw
|
|
302
|
+
* @param {string} contractPath
|
|
303
|
+
* @param {{repo?: string, baseRef?: string}} [context]
|
|
304
|
+
* @returns {ValidatedContract}
|
|
305
|
+
*/
|
|
306
|
+
export function validateContractForLaunch(raw, contractPath, context = {}) {
|
|
307
|
+
if (!context.baseRef) return validateContract(raw, contractPath);
|
|
308
|
+
const repo = context.repo ?? resolve(dirname(contractPath), typeof raw.cwd === "string" ? raw.cwd : ".");
|
|
309
|
+
return validateContractAgainstRef(raw, contractPath, { repo, baseRef: context.baseRef });
|
|
310
|
+
}
|
|
311
|
+
|
|
293
312
|
/**
|
|
294
313
|
* Reduce a run's progress to the one decision the chain makes. An in-flight run
|
|
295
314
|
* is `unfinished` even though `reduceRunOutcome` already names its running nodes
|
package/src/cli.mjs
CHANGED
|
@@ -24,6 +24,7 @@ import {
|
|
|
24
24
|
validBootstrapNonce,
|
|
25
25
|
} from "./run/lock.mjs";
|
|
26
26
|
import { renderRunHandoff } from "./campaign/index.mjs";
|
|
27
|
+
import { validateContractForLaunch } from "./campaign/chain.mjs";
|
|
27
28
|
import { campaignCli } from "./cli/campaign.mjs";
|
|
28
29
|
import { seatCli } from "./cli/seat.mjs";
|
|
29
30
|
import { initCommand } from "./cli/init.mjs";
|
|
@@ -291,9 +292,9 @@ async function main(argv) {
|
|
|
291
292
|
if (command === "run") {
|
|
292
293
|
warnIfNoTransport();
|
|
293
294
|
const absolute = resolve(target);
|
|
294
|
-
const contract = validateContract(JSON.parse(readFileSync(absolute, "utf8")), absolute);
|
|
295
|
-
const runDir = join(contract.cwd, ".runs", contract.id);
|
|
296
295
|
const baseRef = typeof values["base-ref"] === "string" && values["base-ref"] ? values["base-ref"] : undefined;
|
|
296
|
+
const contract = validateContractForLaunch(JSON.parse(readFileSync(absolute, "utf8")), absolute, { baseRef });
|
|
297
|
+
const runDir = join(contract.cwd, ".runs", contract.id);
|
|
297
298
|
setLaunchBaseRef(baseRef);
|
|
298
299
|
// The base is what every worktree is cut from; a dirty tree only blocks
|
|
299
300
|
// when the cwd HEAD *is* that base. A `--base-ref` elsewhere leaves the
|
|
@@ -314,7 +315,7 @@ async function main(argv) {
|
|
|
314
315
|
return;
|
|
315
316
|
}
|
|
316
317
|
for (const warning of [...contract.warnings, ...reusedDoneWarnings(contract)]) process.stdout.write(`${advisoryToken()} ${warning}\n`);
|
|
317
|
-
const result = await runContract(target, { detachedBootstrap: hasDetachedBootstrapNonce() });
|
|
318
|
+
const result = await runContract(target, { detachedBootstrap: hasDetachedBootstrapNonce(), baseRef });
|
|
318
319
|
if (!result.ok) process.exitCode = 1;
|
|
319
320
|
return;
|
|
320
321
|
}
|
|
@@ -3,11 +3,13 @@
|
|
|
3
3
|
* rule 12).
|
|
4
4
|
*
|
|
5
5
|
* `contract.finalVerification` is the contract-wide proof that the phase as a
|
|
6
|
-
* whole closes: the controller runs it before the judge on
|
|
7
|
-
* node (
|
|
8
|
-
* approved on partial verification
|
|
9
|
-
*
|
|
10
|
-
*
|
|
6
|
+
* whole closes: the controller runs it before the judge, once, on whichever
|
|
7
|
+
* phase-terminal node (a node no other node depends on) is the last of them
|
|
8
|
+
* to settle, so no final checkpoint is ever approved on partial verification
|
|
9
|
+
* and a flake on one phase-terminal node cannot also cost its siblings a
|
|
10
|
+
* revision. `contract.sharedVerification` carries the same command schema but
|
|
11
|
+
* is appended to every node's attempt and integration candidate, for the fast
|
|
12
|
+
* repository ratchets a node's write set can break.
|
|
11
13
|
* The persisted node-snapshot shape for verification evidence lives here too,
|
|
12
14
|
* next to the schema it records.
|
|
13
15
|
*/
|
|
@@ -58,20 +60,49 @@ export function sharedVerificationCommands(contract) {
|
|
|
58
60
|
return contract.sharedVerification ?? [];
|
|
59
61
|
}
|
|
60
62
|
|
|
63
|
+
/**
|
|
64
|
+
* Whether no other node in the contract depends on this one. `finalVerification`
|
|
65
|
+
* is a candidate to run on a phase-terminal node only; a node with a dependant
|
|
66
|
+
* never carries it, no matter which of its siblings settles last.
|
|
67
|
+
*
|
|
68
|
+
* @param {{nodes: {id: string, dependsOn: string[]}[]}} contract
|
|
69
|
+
* @param {{id: string}} node
|
|
70
|
+
* @returns {boolean}
|
|
71
|
+
*/
|
|
72
|
+
export function phaseTerminalNode(contract, node) {
|
|
73
|
+
return !contract.nodes.some((candidate) => candidate.dependsOn.includes(node.id));
|
|
74
|
+
}
|
|
75
|
+
|
|
61
76
|
/**
|
|
62
77
|
* The contract's `finalVerification` commands when this node is the one that
|
|
63
|
-
* closes the phase, otherwise none. A
|
|
64
|
-
*
|
|
78
|
+
* closes the phase, otherwise none. A phase can have several phase-terminal
|
|
79
|
+
* nodes (several nodes no other node depends on); the suite runs once for the
|
|
80
|
+
* phase, on whichever of them is the last to settle, not on every one of them.
|
|
81
|
+
*
|
|
82
|
+
* `settledIds` is the set of sibling node ids (any status a node does not
|
|
83
|
+
* leave on its own -- see `SETTLED` in `engine/prompts.mjs`) already reached
|
|
84
|
+
* at the moment this call is made. A phase-terminal node carries the suite
|
|
85
|
+
* exactly when every *other* phase-terminal node is already in that set --
|
|
86
|
+
* whichever node that is true for, in whatever order nodes actually settle,
|
|
87
|
+
* including a node retried after every sibling has already closed. Omitting
|
|
88
|
+
* `settledIds` (the scheduler's budget estimate, which has no run to inspect)
|
|
89
|
+
* falls back to the old per-node-shape answer: any phase-terminal node may
|
|
90
|
+
* still turn out to be the one that carries it, so the estimate stays an
|
|
91
|
+
* upper bound.
|
|
65
92
|
*
|
|
66
93
|
* @param {{finalVerification?: VerificationCommand[], nodes: {id: string, dependsOn: string[]}[]}} contract
|
|
67
94
|
* @param {{id: string}} node
|
|
95
|
+
* @param {Set<string>} [settledIds]
|
|
68
96
|
* @returns {VerificationCommand[]}
|
|
69
97
|
*/
|
|
70
|
-
export function finalVerificationCommands(contract, node) {
|
|
98
|
+
export function finalVerificationCommands(contract, node, settledIds) {
|
|
71
99
|
const commands = contract.finalVerification ?? [];
|
|
72
100
|
if (commands.length === 0) return [];
|
|
73
|
-
|
|
74
|
-
|
|
101
|
+
if (!phaseTerminalNode(contract, node)) return [];
|
|
102
|
+
if (!settledIds) return commands;
|
|
103
|
+
const siblings = contract.nodes.filter((candidate) => candidate.id !== node.id && phaseTerminalNode(contract, candidate));
|
|
104
|
+
const isLastToClose = siblings.every((sibling) => settledIds.has(sibling.id));
|
|
105
|
+
return isLastToClose ? commands : [];
|
|
75
106
|
}
|
|
76
107
|
|
|
77
108
|
/**
|
package/src/engine/scheduler.mjs
CHANGED
|
@@ -31,15 +31,15 @@ import {
|
|
|
31
31
|
terminalErrorCode,
|
|
32
32
|
} from "./lifecycle.mjs";
|
|
33
33
|
import { delay, errorCode } from "../util.mjs";
|
|
34
|
-
import { alreadyNotified, notifyQueueFor, notifyQueuesByRun, renderCampaignHandoffSafely } from "./notify-queue.mjs";
|
|
35
|
-
import { detectStalls, terminateProcess } from "./process.mjs";
|
|
34
|
+
import { alreadyNotified, emitNodeAdvisories, notifyQueueFor, notifyQueuesByRun, renderCampaignHandoffSafely } from "./notify-queue.mjs";
|
|
35
|
+
import { detectStalls, invocationAlive, terminateProcess } from "./process.mjs";
|
|
36
36
|
import { transition, writeNode } from "./state.mjs";
|
|
37
37
|
import { listNodeSnapshots, readNodeSnapshot } from "../run/node-store.mjs";
|
|
38
38
|
import { render, renderFinalReport, writeFindingsArtifact } from "../report/final.mjs";
|
|
39
39
|
import { operationNextState, providerReceipts, settleInvocation } from "../run/operations.mjs";
|
|
40
40
|
import { appendUsageRecord, invocationCost, invocationUsage, recordInvocationUsage } from "../run/usage.mjs";
|
|
41
41
|
import { captureNodeScopeBoundaries, checkWorkerScope, emptyScope } from "./scope.mjs";
|
|
42
|
-
import {
|
|
42
|
+
import { validateContractForLaunch } from "../campaign/chain.mjs";
|
|
43
43
|
import { validateNodeSnapshot } from "../contract/snapshot.mjs";
|
|
44
44
|
import { finalVerificationCommands, sharedVerificationCommands } from "../contract/final-verification.mjs";
|
|
45
45
|
import { startJudge, startWorker } from "./dispatch.mjs";
|
|
@@ -131,16 +131,22 @@ export function nodeBudgetBasisMs(contract, node) {
|
|
|
131
131
|
* never touches phase 2's parked `blocked`/`failed`/`exhausted`/`stalled`
|
|
132
132
|
* states, and never overwrites the `runtime_tier_exhausted` waiting shape.
|
|
133
133
|
*
|
|
134
|
+
* A node whose closed job is being settled in the background (its id is a key
|
|
135
|
+
* of `pendingSettlements`) is not this dead end either: its invocation has
|
|
136
|
+
* already exited and left `running`, but the settlement promise still owns
|
|
137
|
+
* deciding what happens to it, so it is left alone until that promise resolves.
|
|
138
|
+
*
|
|
134
139
|
* @param {string} runDir
|
|
135
140
|
* @param {Map<string, NodeSnapshot>} states
|
|
136
141
|
* @param {Map<string, Job>} running
|
|
137
142
|
* @param {LockHandle|null} lock
|
|
143
|
+
* @param {Map<string, Promise<void>>} [pendingSettlements]
|
|
138
144
|
* @returns {string[]} the node ids this pass parked
|
|
139
145
|
*/
|
|
140
|
-
export function enforceRunningInvariant(runDir, states, running, lock) {
|
|
146
|
+
export function enforceRunningInvariant(runDir, states, running, lock, pendingSettlements = new Map()) {
|
|
141
147
|
const parked = [];
|
|
142
148
|
for (const [nodeId, state] of states) {
|
|
143
|
-
if (state.status !== "running" || running.has(nodeId)) continue;
|
|
149
|
+
if (state.status !== "running" || running.has(nodeId) || pendingSettlements.has(nodeId)) continue;
|
|
144
150
|
transition(runDir, state, "blocked", {
|
|
145
151
|
phase: state.phase,
|
|
146
152
|
error: {
|
|
@@ -175,14 +181,17 @@ function renderFingerprint(states) {
|
|
|
175
181
|
|
|
176
182
|
/**
|
|
177
183
|
* @param {string} contractPath
|
|
178
|
-
* @param {{detachedBootstrap?: boolean}} [options]
|
|
179
|
-
* only by the CLI entry when this process is its
|
|
180
|
-
* makes the controller wait for the launcher's
|
|
184
|
+
* @param {{detachedBootstrap?: boolean, baseRef?: string}} [options]
|
|
185
|
+
* `detachedBootstrap` is set only by the CLI entry when this process is its
|
|
186
|
+
* own detached child, and makes the controller wait for the launcher's
|
|
187
|
+
* acknowledgement; `baseRef` is the CLI's own `--base-ref`, re-validated
|
|
188
|
+
* here so a launch and the run it starts agree about what the contract was
|
|
189
|
+
* checked against
|
|
181
190
|
* @returns {Promise<RunOutcome>}
|
|
182
191
|
*/
|
|
183
192
|
export async function runContract(contractPath, options = {}) {
|
|
184
193
|
const absoluteContractPath = resolve(contractPath);
|
|
185
|
-
const contract =
|
|
194
|
+
const contract = validateContractForLaunch(JSON.parse(readFileSync(absoluteContractPath, "utf8")), absoluteContractPath, { baseRef: options.baseRef });
|
|
186
195
|
const runDir = join(contract.cwd, ".runs", contract.id);
|
|
187
196
|
if (existsSync(runDir)) throw new Error(`run already exists: ${runDir}`);
|
|
188
197
|
mkdirSync(join(contract.cwd, ".runs"), { recursive: true });
|
|
@@ -319,12 +328,13 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
319
328
|
statusFingerprint = fingerprint;
|
|
320
329
|
render(runDir, runsDir, contract, states, renderLock);
|
|
321
330
|
};
|
|
322
|
-
//
|
|
323
|
-
//
|
|
324
|
-
//
|
|
325
|
-
//
|
|
326
|
-
//
|
|
327
|
-
// writes nothing
|
|
331
|
+
// A node's own settlement (its controller verification, its judge round) can
|
|
332
|
+
// be minutes long, and it now runs off the tick's critical path in
|
|
333
|
+
// `pendingSettlements` so dispatch never waits behind it -- but that whole
|
|
334
|
+
// time status.json would otherwise report whatever the last tick left it at.
|
|
335
|
+
// A timer renders between ticks too; it is cheap even when idle because
|
|
336
|
+
// `renderFingerprint` still change-detects, so a quiet run writes nothing
|
|
337
|
+
// extra.
|
|
328
338
|
const statusTimer = setInterval(() => renderStatusIfChanged(), contract.pollIntervalMs);
|
|
329
339
|
statusTimer.unref();
|
|
330
340
|
let handoffFingerprint = statesFingerprint(states);
|
|
@@ -364,14 +374,31 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
364
374
|
|
|
365
375
|
/** @type {Map<string, Job>} */
|
|
366
376
|
const running = new Map();
|
|
377
|
+
// One promise per node currently settling a closed job -- its controller
|
|
378
|
+
// verification, its candidate verification, its judge round -- kept off the
|
|
379
|
+
// tick's critical path so an eligible sibling still dispatches into a free
|
|
380
|
+
// slot while this node's own invocation has already exited. A node's own
|
|
381
|
+
// steps stay ordered because only one settlement per node is ever in flight
|
|
382
|
+
// (see the dispatch loop below); a *different* node's settlement is queued
|
|
383
|
+
// behind whichever one is already running (`settlementQueue`), not run
|
|
384
|
+
// alongside it -- `finalizeClosedJobs` shares state a concurrent second call
|
|
385
|
+
// would corrupt: the phase-continuation selection that picks at most one
|
|
386
|
+
// live node to carry a session forward, and `repo/integrate.mjs`'s one
|
|
387
|
+
// candidate ref and worktree per run. Only *dispatching a sibling* skips
|
|
388
|
+
// ahead of a node's settlement; two nodes' settlements never interleave.
|
|
389
|
+
/** @type {Map<string, Promise<void>>} */
|
|
390
|
+
const pendingSettlements = new Map();
|
|
391
|
+
/** @type {Promise<void>} */
|
|
392
|
+
let settlementQueue = Promise.resolve();
|
|
367
393
|
let canceled = false;
|
|
368
394
|
const cancel = () => { canceled = true; };
|
|
369
395
|
process.once("SIGINT", cancel);
|
|
370
396
|
process.once("SIGTERM", cancel);
|
|
371
397
|
process.once("SIGHUP", cancel);
|
|
372
398
|
// The heartbeat's `at` is owned by an unref'd timer inside this writer, never
|
|
373
|
-
// by the loop body below:
|
|
374
|
-
// critical path and
|
|
399
|
+
// by the loop body below: a node's settlement can still run for minutes off
|
|
400
|
+
// the tick's critical path (`pendingSettlements`), and the heartbeat must
|
|
401
|
+
// keep answering "the process is alive" while it does.
|
|
375
402
|
const heartbeat = createHeartbeat({ runDir, intervalMs: HEARTBEAT_INTERVAL_MS });
|
|
376
403
|
const heartbeatNodes = new Map(contract.nodes.map((node) => [node.id, node]));
|
|
377
404
|
let heartbeatFingerprint = statesFingerprint(states);
|
|
@@ -380,11 +407,89 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
380
407
|
const activeHeartbeatNodes = () => [...states.values()]
|
|
381
408
|
.filter((state) => state.status === "running" && heartbeatNodes.has(state.id))
|
|
382
409
|
.map((state) => ({ nodeId: state.id, budgetBasis: nodeBudgetBasisMs(contract, /** @type {ValidatedNode} */ (heartbeatNodes.get(state.id))) }));
|
|
410
|
+
// A programmer error surfacing inside a background settlement must still
|
|
411
|
+
// crash the whole run, exactly as an unguarded `await finalizeClosedJobs`
|
|
412
|
+
// used to: it is recorded here and thrown from the top of the loop on the
|
|
413
|
+
// very next tick, rather than immediately, so it cannot itself become the
|
|
414
|
+
// block a sibling's dispatch is waiting behind. A lost lock is not this --
|
|
415
|
+
// the loop's own `lock.assert()` calls surface that same condition on their
|
|
416
|
+
// own schedule, so it is left for them.
|
|
417
|
+
/** @type {unknown} */
|
|
418
|
+
let backgroundSettlementFailure = null;
|
|
419
|
+
// Mark every job that closed this tick as settling, and free its slot,
|
|
420
|
+
// without waiting for any of them: that alone is what lets an eligible
|
|
421
|
+
// sibling dispatch into the freed slot while this node's minutes-long
|
|
422
|
+
// controller verification or judge round is still running. The actual
|
|
423
|
+
// settlement work is chained onto `settlementQueue`, one node at a time in
|
|
424
|
+
// the order its job closed, so it still runs exactly as serialized against
|
|
425
|
+
// every *other* node's settlement as it did when this loop awaited
|
|
426
|
+
// `finalizeClosedJobs` directly -- `finalizeClosedJobs` runs against a
|
|
427
|
+
// one-entry map per node, so this is one call per node rather than the one
|
|
428
|
+
// batched call it used to be, but the chain still runs them one at a time.
|
|
429
|
+
// A settlement may itself dispatch the node's next phase (a judge, a
|
|
430
|
+
// revision) through the same `startJudge`/`startWorker` calls dispatch below
|
|
431
|
+
// uses; those land in `slot`, not the real `running`, so they are copied
|
|
432
|
+
// back into `running` in a `finally` -- unconditionally, win or lose, so a
|
|
433
|
+
// job a settlement started before failing (a lost lock, a programmer error)
|
|
434
|
+
// is still visible to the cleanup sweeps below rather than leaked. The
|
|
435
|
+
// node's own steps stay ordered by never starting a second settlement for a
|
|
436
|
+
// node whose first has not yet cleared `pendingSettlements`.
|
|
437
|
+
const settleClosedJobsInBackground = () => {
|
|
438
|
+
for (const [nodeId, job] of [...running]) {
|
|
439
|
+
if (pendingSettlements.has(nodeId) || !job.closed || invocationAlive(job.invocation)) continue;
|
|
440
|
+
running.delete(nodeId);
|
|
441
|
+
const slot = new Map([[nodeId, job]]);
|
|
442
|
+
const settlement = settlementQueue
|
|
443
|
+
.then(() => finalizeClosedJobs(contract, runDir, states, slot, lock, campaign.path))
|
|
444
|
+
.finally(() => {
|
|
445
|
+
for (const [settledId, settledJob] of slot) running.set(settledId, settledJob);
|
|
446
|
+
pendingSettlements.delete(nodeId);
|
|
447
|
+
});
|
|
448
|
+
// The queue itself must never reject -- a rejected settlement (a lost
|
|
449
|
+
// lock, a programmer error) would otherwise wedge every node queued
|
|
450
|
+
// behind it. The rejection still reaches whoever awaits the real
|
|
451
|
+
// `settlement` promise (`pendingSettlements`, below).
|
|
452
|
+
settlementQueue = settlement.catch(() => {});
|
|
453
|
+
// Handled here so an in-flight settlement never becomes an unhandled
|
|
454
|
+
// rejection when nobody happens to await `pendingSettlements` before the
|
|
455
|
+
// process exits; the original promise, still held below, carries the
|
|
456
|
+
// rejection to whichever checkpoint (cancel, shutdown) awaits it.
|
|
457
|
+
settlement.catch((error) => {
|
|
458
|
+
if (!(error instanceof LockLostError) && backgroundSettlementFailure === null) backgroundSettlementFailure = error;
|
|
459
|
+
});
|
|
460
|
+
pendingSettlements.set(nodeId, settlement);
|
|
461
|
+
}
|
|
462
|
+
};
|
|
463
|
+
// Captured once, before the loop, rather than re-derived every tick: it is
|
|
464
|
+
// "parked when this controller invocation started" (autoRetryParkedNodes's
|
|
465
|
+
// own contract), and a node a background settlement parks between two ticks
|
|
466
|
+
// -- rather than synchronously within one, as it always did before
|
|
467
|
+
// settlement moved off the tick's critical path -- must still read as newly
|
|
468
|
+
// parked whichever later tick first observes it, not as already-parked
|
|
469
|
+
// because a per-tick snapshot happened to be taken after it landed.
|
|
470
|
+
const parkedBefore = new Set([...states.values()].filter((state) => PARKED.has(state.status)).map((state) => state.id));
|
|
471
|
+
const anyUnsettled = () => [...states.values()].some((state) => !SETTLED.has(state.status));
|
|
383
472
|
try {
|
|
384
|
-
while
|
|
473
|
+
// A `while` that re-checked this at the very top of every tick would exit
|
|
474
|
+
// the instant a background settlement flips the run's last unsettled node
|
|
475
|
+
// straight to a terminal status between two ticks -- before the tick body
|
|
476
|
+
// that would have run `autoRetryParkedNodes` against that new status ever
|
|
477
|
+
// gets to. The entry guard skips the loop entirely when there is nothing
|
|
478
|
+
// to do at all (a resume of an already-settled run still does zero
|
|
479
|
+
// iterations); once inside, the exit check moves to the bottom, after the
|
|
480
|
+
// body, so that body always sees a freshly-parked node at least once
|
|
481
|
+
// before the loop is allowed to end.
|
|
482
|
+
if (anyUnsettled()) for (;;) {
|
|
385
483
|
lock.assert();
|
|
484
|
+
if (backgroundSettlementFailure !== null) throw backgroundSettlementFailure;
|
|
386
485
|
if (existsSync(join(runDir, "cancel.request.json"))) canceled = true;
|
|
387
486
|
if (canceled) {
|
|
487
|
+
// A settlement already in flight owns the one decision a cancellation
|
|
488
|
+
// must not race: what this node's own invocation resolved to. Let it
|
|
489
|
+
// land on its real terminal status (and, when it re-dispatched a judge
|
|
490
|
+
// or a revision, on the new job that landed in `running`) before this
|
|
491
|
+
// branch decides which nodes are merely canceled.
|
|
492
|
+
await Promise.all([...pendingSettlements.values()]);
|
|
388
493
|
const jobs = [...running.values()];
|
|
389
494
|
await Promise.all(jobs.map((job) => terminateProcess(job)));
|
|
390
495
|
const envelopes = new Map();
|
|
@@ -408,8 +513,14 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
408
513
|
break;
|
|
409
514
|
}
|
|
410
515
|
|
|
411
|
-
|
|
412
|
-
|
|
516
|
+
// Advisory spend lines are checked every tick, closed job or not: a
|
|
517
|
+
// crossing must be visible while the spend is happening on a node still
|
|
518
|
+
// running, not only once it closes. `finalizeClosedJobs` used to open
|
|
519
|
+
// with this same check; calling it once here, rather than once per
|
|
520
|
+
// node settled this tick, is what keeps it at one pass per tick now
|
|
521
|
+
// that settlement runs per node in `settleClosedJobsInBackground`.
|
|
522
|
+
await emitNodeAdvisories(contract, runDir, states);
|
|
523
|
+
settleClosedJobsInBackground();
|
|
413
524
|
await detectStalls(contract, running, async (job, status, error) => {
|
|
414
525
|
const envelope = recordInvocationUsage(job);
|
|
415
526
|
job.state.usage = invocationUsage(job.state);
|
|
@@ -450,8 +561,11 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
450
561
|
writeNode(runDir, job.state, lock);
|
|
451
562
|
});
|
|
452
563
|
// A node left `running` with no job is a dead end; park it before the
|
|
453
|
-
// dispatch pass so it cannot hide behind a healthy sibling.
|
|
454
|
-
|
|
564
|
+
// dispatch pass so it cannot hide behind a healthy sibling. A node
|
|
565
|
+
// whose closed job is settling in the background is not that dead end
|
|
566
|
+
// -- `pendingSettlements` is what tells this apart from one truly
|
|
567
|
+
// abandoned.
|
|
568
|
+
enforceRunningInvariant(runDir, states, running, lock, pendingSettlements);
|
|
455
569
|
// Before dependants are blocked, a node that parked on this tick gets
|
|
456
570
|
// its one automatic retry: it becomes pending, so `blockDependents` sees
|
|
457
571
|
// nothing to block and the dependants stay `pending`/`phase: "waiting"`
|
|
@@ -463,14 +577,37 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
463
577
|
if (slots > 0) {
|
|
464
578
|
const ready = contract.nodes.filter((node) => {
|
|
465
579
|
const state = states.get(node.id);
|
|
466
|
-
return state?.status === "pending" &&
|
|
580
|
+
return state?.status === "pending" && !pendingSettlements.has(node.id)
|
|
581
|
+
&& node.dependsOn.every((id) => states.get(id)?.status === "done");
|
|
467
582
|
});
|
|
468
583
|
for (const node of ready.slice(0, slots)) {
|
|
469
584
|
const state = states.get(node.id);
|
|
470
585
|
if (!state || routingBackoffActive(state, state.phase)) continue;
|
|
586
|
+
// A node recovered pending a re-ask judge (its own worker attempt
|
|
587
|
+
// already accepted, `state.result` durable) reaches `settleDone` /
|
|
588
|
+
// `integrateAttempt` exactly like a closed job's own settlement
|
|
589
|
+
// does, on the same per-run candidate ref and worktree
|
|
590
|
+
// `settlementQueue` exists to serialize -- so it is dispatched the
|
|
591
|
+
// same way: chained onto the queue rather than awaited here, using
|
|
592
|
+
// its own one-entry `slot` merged back into `running` in a
|
|
593
|
+
// `finally`. The node's own order is untouched (still one entry at
|
|
594
|
+
// a time, gated by `pendingSettlements`); only the tick stops
|
|
595
|
+
// waiting behind it.
|
|
471
596
|
if (state.phase === "judge" && state.result) {
|
|
472
|
-
|
|
473
|
-
|
|
597
|
+
const workerResult = state.result;
|
|
598
|
+
const slot = new Map();
|
|
599
|
+
const settlement = settlementQueue
|
|
600
|
+
.then(() => startJudge(contract, node, state, runDir, slot, workerResult, lock, states, campaign.path))
|
|
601
|
+
.then((round) => applyJudgeRound(round, contract, node, state, runDir, slot, lock, states, campaign.path, workerResult))
|
|
602
|
+
.finally(() => {
|
|
603
|
+
for (const [settledId, settledJob] of slot) running.set(settledId, settledJob);
|
|
604
|
+
pendingSettlements.delete(node.id);
|
|
605
|
+
});
|
|
606
|
+
settlementQueue = settlement.catch(() => {});
|
|
607
|
+
settlement.catch((error) => {
|
|
608
|
+
if (!(error instanceof LockLostError) && backgroundSettlementFailure === null) backgroundSettlementFailure = error;
|
|
609
|
+
});
|
|
610
|
+
pendingSettlements.set(node.id, settlement);
|
|
474
611
|
continue;
|
|
475
612
|
}
|
|
476
613
|
state.attempt += 1;
|
|
@@ -499,10 +636,46 @@ export async function driveRun(contract, runDir, states, campaign, lock, sourceI
|
|
|
499
636
|
}
|
|
500
637
|
}
|
|
501
638
|
heartbeat.setActive(activeHeartbeatNodes());
|
|
502
|
-
|
|
639
|
+
// A background settlement can flip a node straight to a parked
|
|
640
|
+
// terminal status during this same tick's own later awaits
|
|
641
|
+
// (`detectStalls`, `notifyStateChanges`), after the auto-retry pass
|
|
642
|
+
// above already ran and saw it as not-yet-parked. Running it once more
|
|
643
|
+
// here, immediately before the exit check, is what stops the loop from
|
|
644
|
+
// mistaking that freshly-parked node for settled and exiting before it
|
|
645
|
+
// ever got its automatic retry; a node it reopens dispatches on the
|
|
646
|
+
// next tick rather than this one, which `anyUnsettled()` below still
|
|
647
|
+
// correctly keeps the loop alive for.
|
|
648
|
+
autoRetryParkedNodes(contract, runDir, states, lock, parkedBefore);
|
|
649
|
+
if (!anyUnsettled()) break;
|
|
650
|
+
await delay(contract.pollIntervalMs);
|
|
503
651
|
}
|
|
652
|
+
// The loop above exits (by the bottom check above or the cancel branch's
|
|
653
|
+
// `break`) the instant every node's state looks settled, but a
|
|
654
|
+
// settlement's own trailing work -- sealing a candidate's acceptance,
|
|
655
|
+
// removing its worktree, enqueueing its terminal notification -- can
|
|
656
|
+
// still be running after the transition that made the state look
|
|
657
|
+
// terminal. Nothing past this point (the final render, the findings
|
|
658
|
+
// artifact, the run-terminal notification, releasing the lock) may run
|
|
659
|
+
// ahead of that trailing work.
|
|
660
|
+
await Promise.all([...pendingSettlements.values()]);
|
|
661
|
+
// A settlement's own `finally` deletes it from `pendingSettlements` the
|
|
662
|
+
// instant it settles, win or lose -- so a rejection recorded here can
|
|
663
|
+
// already be gone from the map above by the time this line runs, with
|
|
664
|
+
// nothing left to await it. The loop's own top-of-tick check
|
|
665
|
+
// (`if (backgroundSettlementFailure !== null) throw ...`) cannot save
|
|
666
|
+
// that case either: the loop has already exited. Checked again here,
|
|
667
|
+
// once, so a background settlement failure can never read as a clean
|
|
668
|
+
// finish just because it settled on the same tick the run's last node
|
|
669
|
+
// did.
|
|
670
|
+
if (backgroundSettlementFailure !== null) throw backgroundSettlementFailure;
|
|
504
671
|
} catch (error) {
|
|
505
672
|
if (!(error instanceof LockLostError)) throw error;
|
|
673
|
+
// A settlement still merges whatever it started into `running` in its own
|
|
674
|
+
// `finally`, win or lose, regardless of whether anything awaits it; wait
|
|
675
|
+
// for that to land (never rejecting itself, so a second lock-loss here
|
|
676
|
+
// cannot mask the one already being handled) before the termination sweep
|
|
677
|
+
// reads `running`.
|
|
678
|
+
await Promise.allSettled([...pendingSettlements.values()]);
|
|
506
679
|
await Promise.all([...running.values()].map((job) => terminateProcess(job)));
|
|
507
680
|
notifyQueuesByRun.delete(runDir);
|
|
508
681
|
return { runDir, states, ok: false, error };
|
package/src/engine/settle.mjs
CHANGED
|
@@ -165,7 +165,7 @@ export async function settleDone(contract, node, state, runDir, lock, states, ca
|
|
|
165
165
|
attemptSha: sealed.sha,
|
|
166
166
|
branch: state.worktree.branch,
|
|
167
167
|
verificationEvidence: state.verification,
|
|
168
|
-
verifyCandidate: (candidateWorkspace) => verifyCandidateWorkspace(contract, node, state, runDir, candidateWorkspace),
|
|
168
|
+
verifyCandidate: (candidateWorkspace) => verifyCandidateWorkspace(contract, node, state, runDir, candidateWorkspace, lock),
|
|
169
169
|
onAccepted: async (transaction) => {
|
|
170
170
|
const acceptedPath = state.worktree?.path ?? attemptWorktreePath(runDir, contract.id, node.id, transaction.attempt);
|
|
171
171
|
if (state.attempt === transaction.attempt && state.status !== "done") {
|
package/src/engine/verify.mjs
CHANGED
|
@@ -11,8 +11,10 @@
|
|
|
11
11
|
import { attemptWorkspace } from "../repo/worktree.mjs";
|
|
12
12
|
import { boundedUtf8, errorMessage } from "../util.mjs";
|
|
13
13
|
import { compactVerification } from "../contract/verification.mjs";
|
|
14
|
-
import { finalVerificationCommands, sharedVerificationCommands } from "../contract/final-verification.mjs";
|
|
14
|
+
import { finalVerificationCommands, phaseTerminalNode, sharedVerificationCommands } from "../contract/final-verification.mjs";
|
|
15
15
|
import { join } from "node:path";
|
|
16
|
+
import { listNodeSnapshots, readNodeSnapshot } from "../run/node-store.mjs";
|
|
17
|
+
import { SETTLED } from "./prompts.mjs";
|
|
16
18
|
import { terminateInvocation } from "./process.mjs";
|
|
17
19
|
import { writeNode } from "./state.mjs";
|
|
18
20
|
import { runVerification } from "./run-command.mjs";
|
|
@@ -28,6 +30,34 @@ import { runVerification } from "./run-command.mjs";
|
|
|
28
30
|
/** @typedef {import("../contract/index.mjs").VerificationState} VerificationState */
|
|
29
31
|
/** @typedef {{index: number, total: number, argv: string}} VerificationProgress */
|
|
30
32
|
|
|
33
|
+
/**
|
|
34
|
+
* The sibling phase-terminal node ids already settled, read straight off
|
|
35
|
+
* disk: every node snapshot is persisted from the run's first tick (see
|
|
36
|
+
* `runContract`), so a node still `pending` reads back honestly, not as
|
|
37
|
+
* missing. `finalVerificationCommands` uses this to decide, at the moment a
|
|
38
|
+
* phase-terminal node's own verification runs, whether it is the one that
|
|
39
|
+
* closes the phase -- recomputed fresh on every call, so a retried node sees
|
|
40
|
+
* its siblings' current status each time, not a decision frozen from an
|
|
41
|
+
* earlier attempt.
|
|
42
|
+
*
|
|
43
|
+
* @param {string} runDir
|
|
44
|
+
* @param {ValidatedContract} contract
|
|
45
|
+
* @param {ValidatedNode} node
|
|
46
|
+
* @returns {Set<string>}
|
|
47
|
+
*/
|
|
48
|
+
function settledSiblingIds(runDir, contract, node) {
|
|
49
|
+
const ids = new Set();
|
|
50
|
+
for (const name of listNodeSnapshots(runDir)) {
|
|
51
|
+
const id = name.slice(0, -".json".length);
|
|
52
|
+
if (id === node.id) continue;
|
|
53
|
+
const sibling = contract.nodes.find((candidate) => candidate.id === id);
|
|
54
|
+
if (!sibling || !phaseTerminalNode(contract, sibling)) continue;
|
|
55
|
+
const snapshot = /** @type {{status?: string}} */ (readNodeSnapshot(runDir, id));
|
|
56
|
+
if (SETTLED.has(/** @type {string} */ (snapshot.status))) ids.add(id);
|
|
57
|
+
}
|
|
58
|
+
return ids;
|
|
59
|
+
}
|
|
60
|
+
|
|
31
61
|
/**
|
|
32
62
|
* The bounded `k/n · argv` shape a running node's status surfaces while a
|
|
33
63
|
* verification command is in flight: the command's 1-based position among
|
|
@@ -108,7 +138,7 @@ export async function executeControllerVerification(contract, runDir, node, stat
|
|
|
108
138
|
};
|
|
109
139
|
writeNode(runDir, state, lock);
|
|
110
140
|
const workspace = attemptWorkspace(state) ?? contract.cwd;
|
|
111
|
-
const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node)];
|
|
141
|
+
const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node, settledSiblingIds(runDir, contract, node))];
|
|
112
142
|
/** @param {VerificationAttempt} attempt @returns {VerificationProgress} */
|
|
113
143
|
const progressFor = (attempt) => verificationProgress(attempt.commandIndex + 1, commands.length, /** @type {VerificationCommand|undefined} */ (commands[attempt.commandIndex])?.argv);
|
|
114
144
|
try {
|
|
@@ -221,16 +251,37 @@ function argvEqual(a, b) {
|
|
|
221
251
|
if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length) return false;
|
|
222
252
|
return a.every((item, index) => item === b[index]);
|
|
223
253
|
}
|
|
254
|
+
/**
|
|
255
|
+
* Marks whether the integration candidate's own verification pass is running
|
|
256
|
+
* right now, nested inside the attempt's own already-completed verification
|
|
257
|
+
* record rather than as a new node-snapshot field: that record's validator
|
|
258
|
+
* (`validateVerificationSnapshot`) accepts extra keys, where the node
|
|
259
|
+
* snapshot's own strict field list would reject one. Status surfaces read it
|
|
260
|
+
* to tell "still verifying the sealed candidate" apart from "waiting on the
|
|
261
|
+
* judge", which otherwise both read as whatever phase the attempt last
|
|
262
|
+
* dispatched under.
|
|
263
|
+
*
|
|
264
|
+
* @param {string} runDir
|
|
265
|
+
* @param {NodeSnapshot} state
|
|
266
|
+
* @param {LockHandle|null} lock
|
|
267
|
+
* @param {boolean} active
|
|
268
|
+
*/
|
|
269
|
+
function markCandidateVerification(runDir, state, lock, active) {
|
|
270
|
+
state.verification = /** @type {VerificationState} */ ({ ...(state.verification ?? { passed: false, commands: [], completed: false }), candidate: active });
|
|
271
|
+
writeNode(runDir, state, lock);
|
|
272
|
+
}
|
|
224
273
|
/**
|
|
225
274
|
* @param {ValidatedContract} contract
|
|
226
275
|
* @param {ValidatedNode} node
|
|
227
276
|
* @param {NodeSnapshot} state
|
|
228
277
|
* @param {string} runDir
|
|
229
278
|
* @param {string} workspace
|
|
279
|
+
* @param {LockHandle|null} [lock]
|
|
230
280
|
* @returns {Promise<import("../repo/integrate.mjs").CandidateEvidence>}
|
|
231
281
|
*/
|
|
232
|
-
export async function verifyCandidateWorkspace(contract, node, state, runDir, workspace) {
|
|
233
|
-
const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node)];
|
|
282
|
+
export async function verifyCandidateWorkspace(contract, node, state, runDir, workspace, lock = null) {
|
|
283
|
+
const commands = [...node.taskPacket.verification, ...sharedVerificationCommands(contract), ...finalVerificationCommands(contract, node, settledSiblingIds(runDir, contract, node))];
|
|
284
|
+
markCandidateVerification(runDir, state, lock, true);
|
|
234
285
|
try {
|
|
235
286
|
const result = await runVerification(commands, workspace, {
|
|
236
287
|
logDir: join(runDir, "logs", `${node.id}.${state.attempt}.candidate-verification`),
|
|
@@ -252,5 +303,7 @@ export async function verifyCandidateWorkspace(contract, node, state, runDir, wo
|
|
|
252
303
|
return settled.retried ? { ...compacted, retried: settled.retried } : compacted;
|
|
253
304
|
} catch (error) {
|
|
254
305
|
return { passed: false, error: boundedUtf8(errorMessage(error), 4 * 1024) };
|
|
306
|
+
} finally {
|
|
307
|
+
markCandidateVerification(runDir, state, lock, false);
|
|
255
308
|
}
|
|
256
309
|
}
|
package/src/repo/integrate.mjs
CHANGED
|
@@ -503,7 +503,7 @@ export function promoteRun({ repo, runId, landBranch, runHead, baseSha, finalVer
|
|
|
503
503
|
if (!head) throw refusal(`refusing to promote ${runId}: the run ref is unavailable`, "run_ref_missing");
|
|
504
504
|
const checkedOutAt = worktreeCheckedOutAt(repo, landBranch);
|
|
505
505
|
if (checkedOutAt) {
|
|
506
|
-
throw refusal(`refusing to promote ${landBranch}: it is checked out in ${checkedOutAt}`, "land_branch_checked_out");
|
|
506
|
+
throw refusal(`refusing to promote ${landBranch}: it is checked out in ${checkedOutAt}; ${landBranchCheckedOutRemedy(checkedOutAt)}`, "land_branch_checked_out");
|
|
507
507
|
}
|
|
508
508
|
const branchRef = `refs/heads/${landBranch}`;
|
|
509
509
|
const current = gitHead(repo, branchRef);
|
|
@@ -540,6 +540,11 @@ function refusal(message, code) {
|
|
|
540
540
|
return Object.assign(new Error(message), { code });
|
|
541
541
|
}
|
|
542
542
|
|
|
543
|
+
/** @param {string} checkedOutAt @returns {string} */
|
|
544
|
+
function landBranchCheckedOutRemedy(checkedOutAt) {
|
|
545
|
+
return `detach the checkout at ${checkedOutAt} or run the coordinator from a worktree`;
|
|
546
|
+
}
|
|
547
|
+
|
|
543
548
|
/** @param {string} path @returns {string} */
|
|
544
549
|
function readTextFile(path) {
|
|
545
550
|
return readFileSync(path, "utf8");
|
package/src/report/render.mjs
CHANGED
|
@@ -26,7 +26,7 @@ const POINTER_ATTENTION_CHARS = 80;
|
|
|
26
26
|
/** @typedef {{costUsd: number|null, costProvenance: CostProvenance, inputTokens: number, outputTokens: number, cacheReadInputTokens: number, pricedInvocations: number, unpricedInvocations: number}} RoleUsage */
|
|
27
27
|
/** @typedef {{inputTokens: number|null, outputTokens: number|null, cacheReadInputTokens: number|null}} StatusPayloadUsage */
|
|
28
28
|
/** @typedef {{index: number, total: number, argv: string}} VerificationProgress */
|
|
29
|
-
/** @typedef {{id: string, status: NodeStatus, phase: string|null, executionPhase: string|null, runtime: string|null, workerRuntime: string|null, continuation: string, attempt: number, revisions: number, startedAt: string|null, updatedAt: string|null, usage: StatusPayloadUsage|null, costUsd: number|null, verdict: string|null, pendingHandoff: {runtime: string, reason: string}|null, note: string|null, scopeFindings: string[]|null, errorCode: string|null, blockedBy: string[], verificationProgress: VerificationProgress|null, declaredReadBytes: number|null}} StatusPayloadNode */
|
|
29
|
+
/** @typedef {{id: string, status: NodeStatus, phase: string|null, executionPhase: string|null, runtime: string|null, workerRuntime: string|null, continuation: string, attempt: number, revisions: number, startedAt: string|null, updatedAt: string|null, usage: StatusPayloadUsage|null, costUsd: number|null, verdict: string|null, gateOutcome: "passed"|"rejected"|null, pendingHandoff: {runtime: string, reason: string}|null, note: string|null, scopeFindings: string[]|null, errorCode: string|null, blockedBy: string[], verificationProgress: VerificationProgress|null, declaredReadBytes: number|null}} StatusPayloadNode */
|
|
30
30
|
/** @typedef {{schemaVersion: 1, run: string, contractId: string, campaignId: string, goal: string, usage: {inputTokens: number, outputTokens: number, cacheReadInputTokens: number, costUsd: number|null}, roles: {worker: RoleUsage, judge: RoleUsage}, controller: JsonObject, identityWarnings: string[], summary: string, nodes: StatusPayloadNode[]}} StatusPayload */
|
|
31
31
|
|
|
32
32
|
/** The glyph each terminal state prints in a status table. */
|
|
@@ -43,13 +43,11 @@ export const MARK = {
|
|
|
43
43
|
};
|
|
44
44
|
|
|
45
45
|
/**
|
|
46
|
-
* `status <run-dir>`:
|
|
47
|
-
*
|
|
48
|
-
*
|
|
49
|
-
* `
|
|
50
|
-
*
|
|
51
|
-
* used only for the two facts the payload does not carry: throwing on an
|
|
52
|
-
* invalid persisted snapshot, and `controllerStatus`'s lock read.
|
|
46
|
+
* `status <run-dir>`: needs-you, now, nodes, cost, in the dashboard's order
|
|
47
|
+
* (TECH-SPEC lean, section 4's last paragraph). Every cell comes from the same
|
|
48
|
+
* payload `status --json` emits (`buildStatusPayload`); `nodes` and
|
|
49
|
+
* `identityWarnings` from `loadRun` cover only what the payload does not:
|
|
50
|
+
* throwing on an invalid snapshot, and `controllerStatus`'s lock read.
|
|
53
51
|
*
|
|
54
52
|
* @param {string} runDir
|
|
55
53
|
* @returns {string}
|
|
@@ -75,7 +73,7 @@ export function renderStatus(runDir) {
|
|
|
75
73
|
const widths = [3, 24, 9, 3, 28, 8, 6, 6, 6, 10, 9, MAX_NOTE_LENGTH];
|
|
76
74
|
/** @type {(cells: unknown[]) => string} */
|
|
77
75
|
const row = (cells) => cells.map((cell, i) => fit(String(cell ?? ""), widths[i])).join(" ");
|
|
78
|
-
lines.push("```", row(["", "NODE", "STATE", "TRY", "RUNTIME", "ELAPSED", "IN", "CACHE", "OUT", "USD", "
|
|
76
|
+
lines.push("```", row(["", "NODE", "STATE", "TRY", "RUNTIME", "ELAPSED", "IN", "CACHE", "OUT", "USD", "GATE", "NOTE"]), row(widths.map((width) => "-".repeat(width))));
|
|
79
77
|
for (const node of payload.nodes) {
|
|
80
78
|
lines.push(row([
|
|
81
79
|
MARK[node.status] ?? "[?]",
|
|
@@ -88,7 +86,7 @@ export function renderStatus(runDir) {
|
|
|
88
86
|
compactTokens(node.usage?.cacheReadInputTokens),
|
|
89
87
|
compactTokens(node.usage?.outputTokens),
|
|
90
88
|
compactCost(node.costUsd),
|
|
91
|
-
node.
|
|
89
|
+
node.gateOutcome ?? "-",
|
|
92
90
|
node.note ?? "-",
|
|
93
91
|
]));
|
|
94
92
|
}
|
|
@@ -98,12 +96,10 @@ export function renderStatus(runDir) {
|
|
|
98
96
|
}
|
|
99
97
|
|
|
100
98
|
/**
|
|
101
|
-
* The node the operator should look at right now, formatted the
|
|
102
|
-
*
|
|
103
|
-
*
|
|
104
|
-
*
|
|
105
|
-
* appends which one, `k/n`, and its argv, so a minutes-long suite is not
|
|
106
|
-
* silent between ticks.
|
|
99
|
+
* The node the operator should look at right now, formatted like the
|
|
100
|
+
* dashboard's now strip (TECH-SPEC lean, section 4, item 1): elapsed time and
|
|
101
|
+
* cost so far, or idle once every node has settled. A node awaiting a
|
|
102
|
+
* controller verification command appends which one, `k/n`, and its argv.
|
|
107
103
|
*
|
|
108
104
|
* @param {StatusPayload} payload
|
|
109
105
|
* @param {number} now epoch ms
|
|
@@ -114,7 +110,8 @@ function nowLine(payload, now) {
|
|
|
114
110
|
if (active) {
|
|
115
111
|
const progress = active.verificationProgress;
|
|
116
112
|
const verification = progress ? ` · verification ${progress.index}/${progress.total} · ${progress.argv}` : "";
|
|
117
|
-
|
|
113
|
+
const candidate = !progress && active.executionPhase === "candidate" ? " · candidate verification" : "";
|
|
114
|
+
return `now: ${active.id} ${active.status} (${formatElapsed(active, now)}) · ${active.runtime ?? "-"} · ${compactCost(active.costUsd)}${verification}${candidate}`;
|
|
118
115
|
}
|
|
119
116
|
const allTerminal = payload.nodes.every((node) => SUCCESS.has(node.status));
|
|
120
117
|
return allTerminal ? `now: idle · run done · ${compactCost(payload.usage.costUsd)}` : "now: idle";
|
|
@@ -134,6 +131,27 @@ function verificationProgress(node) {
|
|
|
134
131
|
return verification?.progress ?? null;
|
|
135
132
|
}
|
|
136
133
|
|
|
134
|
+
/** Set by `verifyCandidateWorkspace` (engine/verify.mjs) for the run of the sealed candidate's own re-verification pass. @param {NodeSnapshot} node @returns {boolean} */
|
|
135
|
+
function candidateVerificationActive(node) {
|
|
136
|
+
const verification = /** @type {{candidate?: boolean}|null|undefined} */ (node.verification);
|
|
137
|
+
return verification?.candidate === true;
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
/** `candidate` while re-verifying the sealed candidate, `verification` mid-attempt-verification, else the persisted phase; the two never overlap. @param {NodeSnapshot} node @returns {string|null} */
|
|
141
|
+
function executionPhaseOf(node) {
|
|
142
|
+
if (candidateVerificationActive(node)) return "candidate";
|
|
143
|
+
if (verificationProgress(node)) return "verification";
|
|
144
|
+
return node.phase;
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
/** What the gate decided (independent of `verdict`: an advisory review or a below-threshold fail verdict still accepts the node, engine/review.mjs `applyJudgeResult`). @param {NodeSnapshot} node @returns {"passed"|"rejected"|null} */
|
|
148
|
+
export function gateOutcome(node) {
|
|
149
|
+
if (!node.gate) return null;
|
|
150
|
+
if (SUCCESS.has(node.status)) return "passed";
|
|
151
|
+
if (node.status === "failed" || node.status === "exhausted") return "rejected";
|
|
152
|
+
return null;
|
|
153
|
+
}
|
|
154
|
+
|
|
137
155
|
/**
|
|
138
156
|
* A node's wall-clock elapsed time: `startedAt` to `updatedAt` once it has
|
|
139
157
|
* settled into a terminal state, `startedAt` to `now` while it is still
|
|
@@ -216,7 +234,7 @@ function buildStatusPayload(runDir, contract, nodes, identityWarnings, usage) {
|
|
|
216
234
|
// A running controller verification command does not move
|
|
217
235
|
// `node.phase` (it stays `worker`, the phase that dispatched it), so
|
|
218
236
|
// the surface that shows it is computed here rather than persisted.
|
|
219
|
-
executionPhase:
|
|
237
|
+
executionPhase: executionPhaseOf(node),
|
|
220
238
|
runtime: node.runtime ? `${node.runtime.harness}/${node.runtime.model}` : null,
|
|
221
239
|
workerRuntime: workerRuntimeLabel(node),
|
|
222
240
|
continuation: continuationMode(node),
|
|
@@ -227,6 +245,7 @@ function buildStatusPayload(runDir, contract, nodes, identityWarnings, usage) {
|
|
|
227
245
|
usage: node.usage ? { inputTokens: node.usage.inputTokens ?? null, outputTokens: node.usage.outputTokens ?? null, cacheReadInputTokens: node.usage.cacheReadInputTokens ?? null } : null,
|
|
228
246
|
costUsd: typeof node.costUsd === "number" ? node.costUsd : null,
|
|
229
247
|
verdict: node.gate?.verdict ?? null,
|
|
248
|
+
gateOutcome: gateOutcome(node),
|
|
230
249
|
pendingHandoff: pendingHandoff(node),
|
|
231
250
|
note: statusNote(node),
|
|
232
251
|
scopeFindings: node.scopeFindings?.unexpectedPaths ?? null,
|
|
@@ -256,16 +275,11 @@ function activeStatusNode(nodes) {
|
|
|
256
275
|
}
|
|
257
276
|
|
|
258
277
|
/**
|
|
259
|
-
* The bounded pointer record written to `.runs/status.json
|
|
260
|
-
*
|
|
261
|
-
*
|
|
262
|
-
*
|
|
263
|
-
*
|
|
264
|
-
* no clock and only jq's builtin `now` can compute an age from a live clock,
|
|
265
|
-
* and neither path needs a `date` process to read an integer. `elapsedSec`
|
|
266
|
-
* is likewise precomputed here (as of `generatedAt`, not live) so the
|
|
267
|
-
* statusline never has to subtract two timestamps to show it — it just
|
|
268
|
-
* prints the integer, whichever reader it is.
|
|
278
|
+
* The bounded pointer record written to `.runs/status.json`, for a quick
|
|
279
|
+
* ambient read without opening the per-run status.json (bounded like the old
|
|
280
|
+
* heartbeat.json). `generatedAt` is unix seconds, not ISO, and `elapsedSec` is
|
|
281
|
+
* precomputed from it, since the no-jq statusline fallback has no clock to
|
|
282
|
+
* subtract two timestamps with.
|
|
269
283
|
*
|
|
270
284
|
* @param {JsonObject} payload the per-run status.json payload
|
|
271
285
|
* @param {NodeSnapshot[]} nodes
|
|
@@ -340,10 +354,7 @@ export function writeStatusArtifacts(runDir, runsDir, contract, states) {
|
|
|
340
354
|
*/
|
|
341
355
|
export function controllerStatus(runDir, nodes) {
|
|
342
356
|
const lock = readLock(runDir);
|
|
343
|
-
// The heartbeat's `at`
|
|
344
|
-
// one are distinguishable on disk. Before this it was hard-coded null here
|
|
345
|
-
// and computed from node updates only once the lock was already stale, which
|
|
346
|
-
// is why all 47 recorded status.json files reported a null lastTick.
|
|
357
|
+
// The heartbeat's `at` distinguishes a working controller from a dead one on disk.
|
|
347
358
|
const heartbeat = readHeartbeat(runDir);
|
|
348
359
|
const lastTick = typeof heartbeat?.at === "string" ? heartbeat.at : null;
|
|
349
360
|
if (!lock || /** @type {{invalid?: true}} */ (lock).invalid) {
|
|
@@ -473,14 +484,11 @@ export function renderFindings(runDir) {
|
|
|
473
484
|
}
|
|
474
485
|
|
|
475
486
|
/**
|
|
476
|
-
* A node the worker itself stopped on, rendered as the repair it asks for
|
|
477
|
-
*
|
|
478
|
-
* `
|
|
479
|
-
*
|
|
480
|
-
*
|
|
481
|
-
* where three nodes were blocked and the command printed one line saying so.
|
|
482
|
-
* The question is already structured, so the repair is mechanical: put the
|
|
483
|
-
* named paths in the packet's `readFiles` and take a new run id.
|
|
487
|
+
* A node the worker itself stopped on, rendered as the repair it asks for:
|
|
488
|
+
* `findings` used to answer only gate exhaustion, so a run whose nodes all
|
|
489
|
+
* stopped on `blocked_context` reported nothing to act on (measured on a
|
|
490
|
+
* four-node campaign with three nodes blocked). The repair is mechanical: put
|
|
491
|
+
* the named paths in the packet's `readFiles` and take a new run id.
|
|
484
492
|
*
|
|
485
493
|
* @param {NodeSnapshot} node
|
|
486
494
|
* @returns {string|null}
|
|
@@ -553,16 +561,11 @@ function continuationMode(node) {
|
|
|
553
561
|
}
|
|
554
562
|
|
|
555
563
|
/**
|
|
556
|
-
* Who produced this node's work, for the RUNTIME column
|
|
557
|
-
*
|
|
558
|
-
*
|
|
559
|
-
*
|
|
560
|
-
*
|
|
561
|
-
* were five different harnesses rendered as though three of them had never
|
|
562
|
-
* run (measured 2026-09-13). The invocation ledger keeps both roles, so the
|
|
563
|
-
* label is derived from it: the worker that ran, falling back to the live
|
|
564
|
-
* runtime for a node that has not dispatched one yet. The judge is not lost —
|
|
565
|
-
* it owns the VERDICT column, and `nowLine` still names whatever is running.
|
|
564
|
+
* Who produced this node's work, for the RUNTIME column: `state.runtime` is
|
|
565
|
+
* the last runtime *dispatched*, and the judge dispatch overwrites the
|
|
566
|
+
* worker's (measured 2026-09-13: a six-harness campaign rendered as though
|
|
567
|
+
* three workers never ran), so this reads the invocation ledger for the
|
|
568
|
+
* worker that ran, falling back to the live runtime if none dispatched yet.
|
|
566
569
|
*
|
|
567
570
|
* @param {NodeSnapshot} node
|
|
568
571
|
* @returns {string|null}
|
|
@@ -633,7 +636,7 @@ function boundedNote(segments, maxLength = MAX_NOTE_LENGTH) {
|
|
|
633
636
|
export function statusNote(node) {
|
|
634
637
|
const scope = scopeFindingsNote(node.scopeFindings);
|
|
635
638
|
const review = reviewNote(node);
|
|
636
|
-
const detail = node.gate?.summary ?? node.error?.message ?? node.blockedBy?.join(", ") ?? node.phase;
|
|
639
|
+
const detail = node.gate?.summary ?? node.error?.message ?? node.blockedBy?.join(", ") ?? (candidateVerificationActive(node) ? "candidate" : node.phase);
|
|
637
640
|
const note = boundedNote([review, detail]);
|
|
638
641
|
if (!scope) return note;
|
|
639
642
|
return boundedNote([scope, note]);
|
|
@@ -657,13 +660,10 @@ function nodeNote(node) {
|
|
|
657
660
|
/** @typedef {{costUsd: number|null, status: "known"|"estimated"|"ambiguous"}} CostProjection */
|
|
658
661
|
|
|
659
662
|
/**
|
|
660
|
-
* Per-role usage from the invocation ledger
|
|
661
|
-
*
|
|
662
|
-
*
|
|
663
|
-
*
|
|
664
|
-
* its `costUsd` stays null rather than summing the known half and understating
|
|
665
|
-
* it. A role with no invocation is `none`. Token totals are always carried, so
|
|
666
|
-
* a role the provider would not price still reads as work.
|
|
663
|
+
* Per-role usage from the invocation ledger, with provenance for whether the
|
|
664
|
+
* cost total is honest: `priced` (all invocations costed), `partial`/`unpriced`
|
|
665
|
+
* (some/none costed, so `costUsd` stays null rather than understating), or
|
|
666
|
+
* `none` (no invocation). Token totals are always carried.
|
|
667
667
|
*
|
|
668
668
|
* @param {NodeSnapshot[]} nodes
|
|
669
669
|
* @returns {{worker: RoleUsage, judge: RoleUsage}}
|
package/src/web/index.html
CHANGED
|
@@ -185,18 +185,18 @@ function tabBodyHtml(tab, detail) {
|
|
|
185
185
|
function drawerHtml(drawer, tab) {
|
|
186
186
|
if (!drawer) return `<section id="drawer"><h2>Run drawer</h2><div class="card" style="padding:16px 18px"><p class="empty">select a run above</p></div></section>`;
|
|
187
187
|
const rows = drawer.nodes.map((node) => `<tr class="clickable ${node.id === drawer.selectedNodeId ? "selected" : ""}" data-node="${esc(node.id)}">
|
|
188
|
-
<td>${pill(node.status)}</td>
|
|
188
|
+
<td>${pill(node.status)}${node.executionPhase === "candidate" ? ` <span class="dim">· candidate verification</span>` : node.executionPhase === "verification" ? ` <span class="dim">· verification</span>` : ""}</td>
|
|
189
189
|
<td class="mono">${esc(node.id)}</td>
|
|
190
190
|
<td class="num">${node.attempt}</td>
|
|
191
191
|
<td class="mono">${esc(node.model ?? "–")}</td>
|
|
192
192
|
<td class="num">${fmtDur(node.elapsedMs)}</td>
|
|
193
193
|
<td class="num">${fmtUsd(node.costUsd)}</td>
|
|
194
|
-
<td>${node.
|
|
194
|
+
<td>${node.gateOutcome ? pill(node.gateOutcome) : ""}</td>
|
|
195
195
|
<td class="goal" title="${esc(node.note ?? "")}">${esc(node.note ?? "")}</td>
|
|
196
196
|
</tr>`).join("");
|
|
197
197
|
const tabs = TABS.map(([key, label]) => `<button data-tab="${key}" class="${tab === key ? "active" : ""}">${esc(label)}</button>`).join("");
|
|
198
198
|
return `<section id="drawer"><h2>Run drawer · ${esc(drawer.runId)}</h2><div class="card" style="padding:16px 18px">
|
|
199
|
-
<table><thead><tr><th>state</th><th>node</th><th>try</th><th>model</th><th>elapsed</th><th>USD</th><th>
|
|
199
|
+
<table><thead><tr><th>state</th><th>node</th><th>try</th><th>model</th><th>elapsed</th><th>USD</th><th>gate</th><th>note</th></tr></thead>
|
|
200
200
|
<tbody>${rows}</tbody></table>
|
|
201
201
|
<div class="tabs">${tabs}</div>
|
|
202
202
|
<div id="tabbody">${tabBodyHtml(tab, drawer.detail)}</div>
|
package/src/web/server.mjs
CHANGED
|
@@ -294,7 +294,8 @@ function nodeRow(runDir, statusNode) {
|
|
|
294
294
|
model: node.runtime?.model ?? null,
|
|
295
295
|
elapsedMs: elapsedMsBetween(node.startedAt, closed ? node.updatedAt : null),
|
|
296
296
|
costUsd: typeof node.costUsd === "number" ? node.costUsd : null,
|
|
297
|
-
|
|
297
|
+
gateOutcome: statusNode.gateOutcome ?? null,
|
|
298
|
+
executionPhase: statusNode.executionPhase ?? null,
|
|
298
299
|
note: statusNode.note ?? null,
|
|
299
300
|
errorCode: statusNode.errorCode ?? null,
|
|
300
301
|
blockedBy: statusNode.blockedBy ?? [],
|