@ctrl-spc/cs 0.5.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1988 @@
1
+ /**
2
+ * ═══ AGENT PANEL v3: the poll loop that answers a card. ═══
3
+ *
4
+ * THIS FILE BELONGS TO AGENT PANEL v3. Nothing outside `src/panel3/` may import
5
+ * it, WITH ONE NAMED EXCEPTION: `cli-v2/src/daemon.ts` imports `startPanel` from
6
+ * here and nothing else, which is how `cs start` runs the panel. It is written
7
+ * down in `conventions.md` and checked by `panel3-isolation.contract.test.mjs`;
8
+ * `startPanel`'s own comment at the foot of this file carries the reasoning.
9
+ *
10
+ * ---------------------------------------------------------------------------
11
+ * WHAT THIS IS. `cs3 say` puts a card and a user turn on the record and returns,
12
+ * because ux.md requires the acknowledgement never to wait for a spawn. This is
13
+ * the other half: the loop that notices the turn, leases it, writes a run row,
14
+ * starts a real agent, and puts what that agent said back on the card.
15
+ *
16
+ * ---------------------------------------------------------------------------
17
+ * ═══ THE DAEMON DOES NOT DECIDE WHAT IS TAKEABLE. THE DATABASE DOES. ═══
18
+ *
19
+ * Everything about exclusion lives in `panel3_take_turns()`: the lease, the
20
+ * "no live level 1 on this card" rule, and the skip-locked that keeps two
21
+ * daemons polling the same millisecond from both getting one card. There is no
22
+ * read-then-decide-then-write anywhere in this file, because that shape has a
23
+ * window between the read and the write and the window is exactly what a second
24
+ * daemon finds. This file honours what it was handed and nothing else.
25
+ *
26
+ * ---------------------------------------------------------------------------
27
+ * ═══ THE RUN ROW IS WRITTEN BEFORE THE SPAWN, OR THERE IS NO SPAWN. ═══
28
+ *
29
+ * plan.md's global constraint 8. A process with no row is a process nothing can
30
+ * find after a restart: it cannot be shown, cannot be recovered and cannot be
31
+ * ended, so the card would say working forever with an agent nobody can name.
32
+ *
33
+ * IT IS THE TAKE THAT WRITES IT, in the same statement that leases the turns,
34
+ * which is why this file has no ordering to be careful about: it cannot reach a
35
+ * spawn without a row, because it cannot reach a spawn without a take. The
36
+ * earlier version wrote the row here, two round trips after the lease, and both
37
+ * ends of that gap were holes — a daemon killed inside it stranded the card
38
+ * forever, and a turn arriving inside it could put a second agent on the same
39
+ * card. See 20260821140000.
40
+ *
41
+ * ---------------------------------------------------------------------------
42
+ * ═══ THE DAEMON WRITES THE AGENT TURN FROM WHAT THE PROCESS SAID. ═══
43
+ *
44
+ * The harness is one-shot: a prompt goes in, text comes out, it exits. That
45
+ * final text IS the answer, so this writes it as the card's `role='agent'` turn.
46
+ *
47
+ * ═══ AND IT IS THE SAME PATH AT EVERY LEVEL, WHICH IS ux.md's RULE, NOT A
48
+ * CONVENIENCE. ═══ "Whoever did the work writes the answer. The level 2
49
+ * agent writes it, straight into the card. It does not hand a summary up to
50
+ * level 1 to be re-written." A dispatched agent's text goes onto the card
51
+ * through the same `panel3_answer` its dispatcher's did, so there is nowhere in
52
+ * this file for one agent's words to pass through another's. The coordinator has
53
+ * usually ended long before.
54
+ *
55
+ * ═══ AND A WORKER'S WORDS DO NOT REACH THE CARD AT ALL, WHICH IS THE SAME RULE
56
+ * ONE LEVEL DOWN. ═══ "Each worker writes what it found into the record, and
57
+ * the level 2 agent reads them back." Three workers writing their own turns
58
+ * would be three answers to one request, before the one that covers them all
59
+ * arrives. `panel3_answer` puts a level 3 run's last words in its report instead,
60
+ * from the level on the row — decided there rather than here, because two owners
61
+ * for "does this reach the person" is one owner too many.
62
+ *
63
+ * ---------------------------------------------------------------------------
64
+ * ═══ A QUESTION MOVING, AND EVERYBODY YOU SENT FINISHING, ARE BOTH TAKEABLE
65
+ * WORK, AND THEY COME THROUGH THE SAME POLL. ═══
66
+ *
67
+ * ux.md: "Nothing new is built for this. The daemon already leases takeable work
68
+ * and spawns. What changes is the definition of takeable." So there is a second
69
+ * take beside `panel3_take_turns` and nothing else in this file changes shape:
70
+ * `panel3_take_rearms` hands back runs to start again, for either of the two
71
+ * reasons ux.md's table gives, and they are started, held and settled by the
72
+ * functions that already do those things.
73
+ *
74
+ * THE SECOND REASON IS THE ONE NOTHING ELSE COULD FIRE. An agent that split its
75
+ * work dispatched and ended, because it is forbidden to wait; by the time its
76
+ * workers finish there is no process anywhere holding what they found. Starting
77
+ * it again is what makes "it reads them back and writes the answer" a mechanism
78
+ * rather than a sentence.
79
+ *
80
+ * IT IS THE SAME MECHANISM AS RECOVERY, POINTED AT A DIFFERENT FACT. A run whose
81
+ * process died is started again as itself because the machine failed; a run that
82
+ * asked, or one whose workers are done, is started again as itself because the
83
+ * thing it was waiting on exists now. Same row, same id, same brief, its report
84
+ * carried — and the only difference is one paragraph at the top of the prompt.
85
+ *
86
+ * ---------------------------------------------------------------------------
87
+ * ═══ A DISPATCH IS A SPAWN THE DAEMON DOES ON BEHALF OF AN AGENT. ═══
88
+ *
89
+ * The tools server runs inside this process and `dispatch` is served from it,
90
+ * but the spawning is here: writing the row before the process, recording the
91
+ * pid, waiting for the answer, giving up when the process is gone, and being
92
+ * held open long enough for `--once` to be honest are all things this file
93
+ * already does once, and a second copy of them behind a tool would be a second
94
+ * owner of a child's whole life.
95
+ *
96
+ * WHAT IT DOES NOT DO IS WAIT. The tool call returns as soon as the process
97
+ * exists, and the child's answer settles here, long after the agent that asked
98
+ * for it has ended. That is ux.md's "level 1 never blocks waiting for it", and it
99
+ * is why a dispatched run's promise goes into `inFlight` rather than being
100
+ * awaited by its caller.
101
+ *
102
+ * ---------------------------------------------------------------------------
103
+ * ═══ A CARD MUST NOT SIT LIVE FOREVER. ═══
104
+ *
105
+ * If this daemon is killed mid-run its process dies with it, and the run row is
106
+ * left saying `running` with an end that never comes. `recoverStranded` is what
107
+ * closes that: on every poll it looks at the runs THIS machine owns and asks the
108
+ * operating system whether their processes are still there. A level 1 run's
109
+ * turns are handed back, so they are takeable again; a dispatched run holds no
110
+ * turns and is STARTED AGAIN AS ITSELF, with the brief it was sent and the
111
+ * report it wrote. Either way the user's message is answered late rather than
112
+ * lost.
113
+ *
114
+ * It is careful in three directions, and each is a way the sweep can lie: it
115
+ * never reaps a run this process holds, never reaps a run too young to have
116
+ * recorded a pid (which is another daemon's live work), and never believes a pid
117
+ * that predates this machine's boot (which is somebody else's process wearing a
118
+ * reused number).
119
+ *
120
+ * ═══ AND IT CAN ONLY EVER REACH THIS MACHINE'S RUNS, WHICH IS WHY A PERSON HAS
121
+ * THE OTHER MOVE. ═══
122
+ *
123
+ * A machine that is never coming back sweeps nothing, so its dispatched runs sit
124
+ * `running` forever. `panel3_hand_back` is the person's answer, pressed from the
125
+ * panel on a card nothing is listening for, and `takeHandedBack` is this
126
+ * daemon's half of it: the one read that looks past this machine's own id, so
127
+ * work another machine was holding is carried on from its brief and its report
128
+ * instead of being started over. The sweep is not loosened to do it, and which
129
+ * of the three cares above apply to that read is argued where it is written.
130
+ */
131
+ // The runtime import comes FIRST, deliberately: `tsc` elides a type-only import
132
+ // and takes the leading comment with it, so a file whose first statement is
133
+ // `import type` loses its v3 header in the published `dist/`.
134
+ import { out, returned, signedInClient } from './client.js';
135
+ import { answerPrompt, escalationPrompt, levelOnePrompt, readBackPrompt, resumePrompt, retryPrompt, } from './prompt.js';
136
+ import { attachmentLine, loadAttachments, loadOutputNames, outputOf } from './show.js';
137
+ import { forgetSecrets, redactSecrets } from './secrets.js';
138
+ import { sayListening, stopListening } from './presence.js';
139
+ import { checkoutForCodebase, hasCheckoutForCodebase } from './checkout.js';
140
+ import { harness, startAgent } from './spawn.js';
141
+ import { startToolsServer } from './tools.js';
142
+ import { getMachineIdentity, scratchDir } from '../config.js';
143
+ import { listCodebases } from '../codebases.js';
144
+ import { processIsAlive } from '../win-shell.js';
145
+ import { hostname, uptime } from 'node:os';
146
+ const USAGE = 'usage: cs3 run [--once]';
147
+ /** How long between takes. Short, because it is the whole delay between a user
148
+ * sending and a card showing an agent on it, and the take is one small indexed
149
+ * read against the user's own rows. */
150
+ const POLL_INTERVAL_MS = 2_000;
151
+ /**
152
+ * How long a run with no pid recorded is given before it counts as gone.
153
+ *
154
+ * ═══ IT EXISTS BECAUSE `mine` IS ONLY THIS PROCESS'S MEMORY. ═══ The take
155
+ * writes the run row and the pid lands a moment later, so between the two the
156
+ * row is indistinguishable from one whose daemon died — to this daemon. To a
157
+ * SECOND daemon on the same machine it is worse than indistinguishable: the run
158
+ * is not in its `mine` set at all, so without this it would hand back the turns
159
+ * of a run that started seconds ago and mark it failed while the agent was
160
+ * genuinely working. That configuration is not hypothetical; it is what the
161
+ * lease is proved with.
162
+ *
163
+ * Thirty seconds is far longer than the one round trip it has to cover and far
164
+ * shorter than any run, so it costs a genuinely dead run one extra poll cycle at
165
+ * most.
166
+ *
167
+ * ═══ AND `panel3_resume` HOLDS THE SAME NUMBER, SO IT IS EXPORTED AND CHECKED.
168
+ * ═══ The claim refuses to hand the same run out twice inside this window, which
169
+ * is this rule from the other side: a run that has just been claimed has no
170
+ * process yet and everything else has to leave it alone long enough for one to
171
+ * exist. Two comments saying they agree is not a check, and the two are in
172
+ * different languages in different files — so the contract test reads the
173
+ * interval out of the function definition and compares it to this. Disagreement
174
+ * either reaps a resume a second after it started, or makes a dead run wait out
175
+ * two windows.
176
+ */
177
+ export const PID_GRACE_MS = 30_000;
178
+ /**
179
+ * How many processes may be started for one run, counting the first.
180
+ *
181
+ * ═══ IT IS NOT A NEW NUMBER, AND THAT IS THE REASON FOR IT. ═══
182
+ * `orchestrator.ts:1107` `MAX_ATTEMPTS = 3` and `orchestrator.ts:4421`
183
+ * `MAX_REPLY_ATTEMPTS = 3` are the same bound, with the reason already written
184
+ * down at `orchestrator.ts:1102-1105`: three attempts is enough to ride out a
185
+ * genuinely transient failure and short enough that a deterministic one stops
186
+ * costing the user minutes of spawn time, and it is not configurable, because a
187
+ * knob here would mostly serve to hide a bug behind a bigger number. A fourth
188
+ * number invented here would be a dialect.
189
+ *
190
+ * ═══ AND THE BOUND ITSELF IS NOT THIS CONSTANT. ═══ `panel3_resume` counts the
191
+ * attempt and refuses past three IN THE SAME STATEMENT, because a bound whose
192
+ * counter can silently miss an attempt is not a bound
193
+ * (`20260806120000:1096-1099`). This is the number the daemon says out loud, and
194
+ * a contract test reads the literal back out of the function definition and
195
+ * compares the two, exactly as `PID_GRACE_MS` is compared against that
196
+ * function's 30 seconds.
197
+ */
198
+ export const MAX_ATTEMPTS = 3;
199
+ /** The level the daemon puts on a card of its own accord. Anything deeper exists
200
+ * because an agent asked for it, and its level comes off the row
201
+ * `panel3_dispatch` wrote rather than from anything here. */
202
+ const LEVEL = 1;
203
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
204
+ /**
205
+ * ═══ NOTHING IN THIS FILE STAMPS A COLUMN FROM THIS MACHINE'S CLOCK ANY MORE,
206
+ * AND THAT IS A RULE RATHER THAN A TIDY-UP. ═══
207
+ *
208
+ * `started_at`, `ended_at` and `resumed_at` are all written by the database's own
209
+ * `now()`, in the statements that end or claim a run, and `panel3_owed_read_back`
210
+ * COMPARES a parent's end with its children's. A second clock writing any of them
211
+ * makes that comparison meaningless in a way nothing reports: skew one way hides
212
+ * the child that genuinely finished last and loses the read-back, with every row
213
+ * individually plausible. `failRun` was the last place that did it and now calls
214
+ * `panel3_end_run`.
215
+ *
216
+ * ═══ THE TWO CLOCKS STILL MEET IN ONE PLACE, AND IT IS DELIBERATE. ═══
217
+ * `recoverStranded` reads `started_at`, which the database stamped, and measures
218
+ * it against `Date.now()` for the pid grace window and against this machine's
219
+ * boot time for the reused-pid check. It has no choice: liveness is a local fact
220
+ * and the boot time is only knowable here. Both directions of skew are real:
221
+ *
222
+ * THE DATABASE AHEAD OF LOCAL BY MORE THAN THE GRACE WINDOW makes every
223
+ * pid-null run look younger than it is, forever, so nothing is ever reaped —
224
+ * which is the permanent strand the recovery exists to prevent.
225
+ *
226
+ * THE DATABASE BEHIND LOCAL BY MORE THAN THIS MACHINE'S UPTIME makes a run
227
+ * that started moments ago look older than the boot, so a live agent is
228
+ * declared gone and its turns are handed to a second one.
229
+ *
230
+ * Considered rather than missed, and left as it is: both machines run NTP — here
231
+ * they are the same machine, since the local stack is a container on it — so the
232
+ * skew is milliseconds against a thirty-second window and an uptime measured in
233
+ * days. A product where the two could drift that far would have to stamp both
234
+ * clocks and compare like with like.
235
+ */
236
+ /* ═══ THE NAME A PERSON TYPED, WHICH IS THE ONLY NAME THEY KNOW. ═══ `cs start`
237
+ is the command in the product's copy everywhere else, and this is the one
238
+ prefix on the one stream a person actually reads. */
239
+ const said = (what) => console.error(`cs start: ${what}`);
240
+ /**
241
+ * The taken rows, grouped by card, each card's turns still in send order.
242
+ *
243
+ * ═══ THE GROUPING IS WHY THE TAKE RETURNS ROWS AND NOT ONE BODY. ═══ ux.md:
244
+ * "A turn takes every unaddressed input on that card, oldest first, and hands
245
+ * them all over. Not the newest. Not one chosen as 'the request'." There is
246
+ * nowhere in this function for the product to choose between them, which is the
247
+ * property that matters: v2 chose, and the message it did not choose was filed
248
+ * as background and never acted on.
249
+ */
250
+ function byCard(taken) {
251
+ const cards = new Map();
252
+ for (const row of taken) {
253
+ const existing = cards.get(row.card_id);
254
+ if (existing)
255
+ existing.push(row);
256
+ else
257
+ cards.set(row.card_id, [row]);
258
+ }
259
+ return cards;
260
+ }
261
+ /**
262
+ * ═══ WHAT THIS CARD HAS ALREADY PRODUCED, NAMED AS IT READS NOW. ═══
263
+ *
264
+ * A card's receipts are level 1's report: every turn is answered by a fresh
265
+ * agent that remembers nothing, and what it most needs from the last one is what
266
+ * that one made. So they are read here and handed over in the prompt, exactly as
267
+ * a dispatched run's report is, rather than being left behind a tool call — see
268
+ * `levelOnePrompt` for the measured failure this closes.
269
+ *
270
+ * THE NAME IS RESOLVED, NOT READ OFF THE RECEIPT, through the one function
271
+ * `cs3 show` uses for it. `label` is written when the thing is made and the
272
+ * coordinator may rename the thing on the next turn; printing the label then
273
+ * hands the agent the name of a different, real object. One owner for that,
274
+ * because two would eventually disagree about which name a card is showing.
275
+ */
276
+ async function receiptsFor(client, cardId) {
277
+ const outputs = await returned(client
278
+ .from('panel3_outputs')
279
+ .select('id, card_id, run_id, kind, ref_id, label, created_at')
280
+ .eq('card_id', cardId)
281
+ .order('created_at'), 'read', `what card ${cardId} has produced`);
282
+ const names = await loadOutputNames(client, outputs);
283
+ return outputs.map((o) => `${o.kind.padEnd(10)} ${outputOf(o, names)} id ${o.ref_id}`);
284
+ }
285
+ /**
286
+ * ═══ WHAT THE PERSON ATTACHED TO THIS CARD, FORMATTED THE ONE WAY THE PRODUCT
287
+ * NAMES AN ATTACHMENT. ═══
288
+ *
289
+ * The product applies the attachment (plan-work-items.md constraint 5): level 1
290
+ * is handed it here rather than asked to go and find it, exactly as `produced`
291
+ * is handed over rather than left behind `list_work_items`. Most cards have
292
+ * none, since Task 2 offers attaching only when a card is started, so this is
293
+ * usually an empty list, and `levelOnePrompt` says nothing about attachments at
294
+ * all when it is.
295
+ */
296
+ // Exported so a test can prove the wiring itself: that this reads the CALLER's
297
+ // own card through `loadAttachments` rather than a copy, the same reason
298
+ // `dispatch`'s equivalent read is proven in panel3-tools.contract.test.mjs
299
+ // instead of only the formatter it feeds.
300
+ export async function attachmentsFor(client, cardId) {
301
+ return (await loadAttachments(client, cardId)).map(attachmentLine);
302
+ }
303
+ /**
304
+ * What level 1 is sent, and also what is written to `panel3_runs.brief`: the
305
+ * column's own question is "what was this agent asked", answered with the exact
306
+ * text it was asked with rather than with a description of it.
307
+ *
308
+ * THE WORDS THEMSELVES ARE IN `prompt.ts`, which is where the rules that govern
309
+ * them are written down. This is only the join: the take hands over rows, the
310
+ * record says what the card has made, and a prompt is about a title, some
311
+ * messages and that list.
312
+ */
313
+ // Exported for the same reason as `attachmentsFor` above: the wiring test
314
+ // calls this with what that function actually returned, so what a coordinator
315
+ // is told is proven to come from the card's own attachment rows rather than
316
+ // only from `levelOnePrompt`'s own formatting tests in panel3-prompt.
317
+ export const briefFor = (title, turns, produced, attachments, codebases = null) => levelOnePrompt(title, turns.map((t) => t.body), produced, attachments, codebases?.map((codebase) => ({
318
+ id: codebase.id,
319
+ name: codebase.name,
320
+ identity: codebase.gitRemoteUrl,
321
+ located: hasCheckoutForCodebase(codebase),
322
+ })) ?? null);
323
+ // ---------------------------------------------------------------------------
324
+ // WRITES. Every one goes through the shared guard in `client.ts`, which returns
325
+ // the rows or throws. `.select()` on each is what makes that possible: a
326
+ // Supabase write with no select returns neither data nor an error, and the guard
327
+ // refuses that shape rather than reading it as success.
328
+ /**
329
+ * ═══ WHAT IS RUNNING A RUN, AND — AT LEVEL 1 — WHAT IT WAS SENT. ═══
330
+ *
331
+ * The run row already exists in both paths: the take wrote it for level 1 and
332
+ * `panel3_dispatch` wrote it for a dispatched agent, which is what makes
333
+ * constraint 8 unconditional rather than an ordering this file has to be careful
334
+ * about. So this is never the row's creation, only the facts that could not be
335
+ * known before the process existed.
336
+ *
337
+ * THE BRIEF IS PASSED ONLY WHERE IT IS STILL A PLACEHOLDER. The take cannot know
338
+ * what an agent will be sent, so level 1's row carries a placeholder until here;
339
+ * a dispatch knows the brief before the row exists and writes it there. Passing
340
+ * it again would be rewriting a brief that ux.md fixes at dispatch.
341
+ *
342
+ * A failure here is NOT fatal. The agent is already running and killing it over
343
+ * a bookkeeping write would cost the user the work. What is lost is precision:
344
+ * the run has no pid, so once this daemon is gone recovery treats it as dead —
345
+ * which errs towards answering again rather than stranding.
346
+ */
347
+ async function recordProcess(client, runId, pid, brief) {
348
+ await returned(client
349
+ .from('panel3_runs')
350
+ .update({ pid, ...(brief === undefined ? {} : { brief }) })
351
+ .eq('id', runId)
352
+ .select('id'), 'record what is running', `run ${runId}`);
353
+ }
354
+ /**
355
+ * ═══ A RUN GIVEN UP ON: ENDED, ITS TURNS HANDED BACK, ITS CARD IN HAND. ═══
356
+ *
357
+ * One statement, in `panel3_give_up`, for the reason 20260821170000 gives at
358
+ * length: as two round trips there was an instant in which the turns were
359
+ * takeable and the run still read `running`, and an answer landing in that
360
+ * instant was accepted onto a card another daemon was already free to take.
361
+ *
362
+ * Returns the number of turns handed back, or NULL when the run was no longer
363
+ * running — which is this function losing cleanly to a `panel3_answer` that got
364
+ * there first. It is a fact about the record rather than a failure, so it does
365
+ * not go through `returned()`, exactly as the answer's own null does not.
366
+ */
367
+ async function giveUp(client, runId, reason) {
368
+ const { data, error } = await client
369
+ .rpc('panel3_give_up', { p_run_id: runId, p_reason: reason });
370
+ if (error)
371
+ throw new Error(`could not give up run ${runId}: ${error.message}`);
372
+ /* THE RUN IS OVER, so whatever it read is dropped. Every ending does this —
373
+ here, `failRun` and `writeAnswer` — because a daemon stays up for days and
374
+ has no business holding Tuesday's secret. */
375
+ forgetSecrets(runId);
376
+ return data;
377
+ }
378
+ /**
379
+ * ═══ THE ANSWER, THE END OF THE RUN AND THE STATE OF THE CARD ARE ONE
380
+ * STATEMENT. ═══
381
+ *
382
+ * They are one event, and writing them as three round trips left a window where
383
+ * the answer was on the card while the run still read running: recovery then
384
+ * handed the turns back and a second agent answered the same question beside an
385
+ * answer that was already there. `panel3_answer` is where that window used to be.
386
+ * It also decides whether the card is DONE, which it is only when nothing else
387
+ * on it is still live — a coordinator that dispatched and answered must not
388
+ * settle a card whose work is still going. See 20260823090000.
389
+ *
390
+ * ═══ AND IT IS SHARED BY EVERY LEVEL ON PURPOSE. ═══ ux.md's "whoever did the
391
+ * work writes the answer" is only structurally true if there is exactly one way
392
+ * for text to reach a card. A second path for dispatched runs is where a
393
+ * summarising hop would eventually be added.
394
+ *
395
+ * ═══ THIS IS THE ONE v3 WRITE THAT DOES NOT GO THROUGH `returned()`. ═══ The
396
+ * shared guard treats "no error and no data" as a result it does not understand,
397
+ * which is right everywhere else and wrong here: a null from `panel3_answer` is a
398
+ * FACT about the record — the run was no longer running, so nothing was written
399
+ * — and it has its own sentence below. A failure is still a failure and still
400
+ * throws.
401
+ *
402
+ * The run's text is on the card as a turn and is deliberately NOT copied into
403
+ * `report` as well: one fact, one place. `report` carries the reason on the
404
+ * failure paths, where there is no turn and it is the only record.
405
+ *
406
+ * ═══ AND A RUN WITH NOTHING TO SAY PASSES NULL, WHICH IS NOT THE SAME AS
407
+ * SAYING NOTHING. ═══ ux.md's "outbound follows the work": a run started
408
+ * again only to settle a question from somebody it sent did no work for the
409
+ * person, and its words went to that agent through the answer. It is still
410
+ * ended here, and the card is still settled by the same statement, because such
411
+ * a run can be the last thing live on it — but no turn is written, exactly as
412
+ * none is written for a run that stopped to ask.
413
+ */
414
+ async function writeAnswer(client, runId, cardId, text) {
415
+ /* ═══ THE SECOND OF THE TWO CHOKEPOINTS THE CREDENTIAL RULE RESTS ON. ═══
416
+ This is the ONE path an agent's own words take to a hosted row — a turn on
417
+ the card at levels 1 and 2, and a level 3's `panel3_runs.report`, which
418
+ `panel3_answer` writes instead. Tool arguments are the other, redacted in
419
+ `buildServer`. Identity for a run that read no credential, so every existing
420
+ exact-string assertion is byte-identical. */
421
+ const { data: turnId, error } = await client
422
+ .rpc('panel3_answer', { p_run_id: runId, p_body: text === null ? null : redactSecrets(runId, text) });
423
+ if (error) {
424
+ /* Thrown, not settled as failed. The run stays `running` with a process
425
+ that has now exited, so recovery reaps it: at level 1 the turns are
426
+ answered again, so the user gets an answer late instead of a card marked
427
+ failed for a database hiccup. The caller prints the reason. */
428
+ throw new Error(`could not write the answer to card ${cardId}: ${error.message}`);
429
+ }
430
+ /* ═══ INCLUDING WHEN THE RUN STOPPED TO ASK AND WILL BE STARTED AGAIN. ═══
431
+ That reads like a mistake and is not: the process is gone, and the one that
432
+ takes its place is a fresh process with a fresh context which reads the
433
+ credential again through `get_credential`. What it is handed to start from —
434
+ its own report, and the card — went through this same substitution, so there
435
+ is nothing left for a stale entry to protect. */
436
+ forgetSecrets(runId);
437
+ if (turnId === null) {
438
+ /* NO TURN WAS WRITTEN, and there are FOUR reasons, which mean different
439
+ things and must not be printed as one line.
440
+
441
+ IT STOPPED TO ASK. Its last words were the question, which is already on
442
+ the card, and its end has just been stamped by the same statement that
443
+ refused the turn. Nothing is wrong and nothing is lost.
444
+
445
+ IT HAD NOTHING TO SAY TO THE PERSON. A run started only to settle a
446
+ question from somebody it sent is asked for exactly this, and it is the
447
+ daemon that asked.
448
+
449
+ IT IS A WORKER, and its words went into its report for whoever sent it.
450
+ ux.md: one request is one answer, written by the level that owns the card.
451
+
452
+ OR IT WAS NO LONGER RUNNING — recovery had given up on it and, at level 1,
453
+ its turns belong to another run now. Writing anyway would put a second
454
+ answer on the card. The process's work is lost, which is the honest cost
455
+ of having been declared dead.
456
+
457
+ ═══ AND THE STATE IS READ FIRST, WHICH IS THE FIX FOR A LINE THAT USED TO
458
+ LIE. ═══ The last of the four was reachable with a null body — a re-armed
459
+ run given up on while its process was still going — and it printed the
460
+ second one's sentence, "answered a question, so it wrote nothing here",
461
+ about a run that had been declared dead and written nothing anywhere. So
462
+ the state gates the two honest sentences rather than sitting under them.
463
+ It decides nothing except which sentence is true: whichever of the four
464
+ happened has already happened, in one statement, before this line runs. */
465
+ const run = await runNow(client, runId);
466
+ if (run.state === 'asked') {
467
+ out(`asked card ${cardId} run ${runId} stopped on a question, so it wrote no answer`);
468
+ }
469
+ else if (run.state !== 'finished') {
470
+ said(`run ${runId} was ${run.state}, so its answer was not written to card ${cardId}`);
471
+ }
472
+ else if (text === null) {
473
+ out(`settled card ${cardId} run ${runId} answered a question, so it wrote nothing here`);
474
+ }
475
+ else if (run.level === 3) {
476
+ out(`reported run ${runId} ${text.length} characters, to whoever sent it`);
477
+ }
478
+ else {
479
+ /* FINISHED, WITH SOMETHING TO SAY, AND NO TURN. It had already answered
480
+ once, which is the one shape left, and it is said rather than assumed
481
+ away. */
482
+ said(`run ${runId} had already ended, so its answer was not written to card ${cardId}`);
483
+ }
484
+ return;
485
+ }
486
+ out(`answered card ${cardId} run ${runId} ${text?.length ?? 0} characters`);
487
+ }
488
+ /** What the record says a run is now, and at what level. Read only to say the
489
+ * right sentence about something that has already happened; nothing branches on
490
+ * it that could have gone the other way. */
491
+ async function runNow(client, runId) {
492
+ const runs = await returned(client.from('panel3_runs').select('state, level').eq('id', runId), 'read', `the state of run ${runId}`);
493
+ return runs[0] ?? { state: 'no longer on the record', level: null };
494
+ }
495
+ /**
496
+ * A run whose process failed, ended with the reason ON ITS OWN COLUMN, and its
497
+ * turns left exactly where they are.
498
+ *
499
+ * ═══ NEVER INTO `report`, WHICH IS THE RUN'S OWN WORDS. ═══ `report` is what a
500
+ * respawn is handed, and this build starts settled runs again — an answered
501
+ * question does exactly that — so a reason written there would destroy the state
502
+ * the next attempt was going to resume from, and `cs3 show` would print the
503
+ * daemon's sentence under a heading that says the run wrote it. One owner per
504
+ * column: the run writes `report`, whoever ended it writes `failed_because`.
505
+ *
506
+ * ═══ AND IT IS A STATEMENT RATHER THAN AN UPDATE, BECAUSE `ended_at` IS
507
+ * COMPARED AGAINST OTHER ROWS' ENDS. ═══
508
+ *
509
+ * This wrote the stamp from THIS MACHINE'S clock, because a PostgREST update has
510
+ * nowhere to ask for the server's time. Everything else that ends a run —
511
+ * `panel3_answer`, `panel3_give_up` — uses the database's `now()`, and
512
+ * `panel3_owed_read_back` measures a parent's end against its children's. So the
513
+ * two stamps met, and skew between them loses a read-back SILENTLY: a machine
514
+ * clock running ahead writes a parent's end later than the child that genuinely
515
+ * outlived it, the comparison finds nobody who finished after it, and the
516
+ * synthesis never happens with every row individually plausible.
517
+ *
518
+ * The header of this file already argues that v3 must not compare two clocks.
519
+ * This was the one place left that did.
520
+ */
521
+ async function failRun(client, runId, reason) {
522
+ /* ═══ ONLY A RUN THAT HAS NOT ALREADY ENDED, AND THAT GUARD IS IN THE
523
+ STATEMENT. ═══ Unguarded, this wrote over a run that had been started again
524
+ in the meantime — which is not hypothetical now that an answered question
525
+ restarts a settled row — and marked a live agent failed. False is a fact the
526
+ caller reports, not a failure. */
527
+ const { data, error } = await client
528
+ .rpc('panel3_end_run', { p_run_id: runId, p_reason: reason });
529
+ if (error)
530
+ throw new Error(`could not end run ${runId}: ${error.message}`);
531
+ forgetSecrets(runId);
532
+ return data === true;
533
+ }
534
+ /**
535
+ * What the card is doing, written by the daemon.
536
+ *
537
+ * ═══ EXCEPT ON A CARD THE USER STOPPED, WHICH NOTHING HERE MAY OVERWRITE. ═══
538
+ * ux.md: the user's Stop is absolute. Its own effect is the reason this guard is
539
+ * needed rather than a precaution. `panel3_stop_card` ends the runs, this daemon
540
+ * kills their processes, and the level 1 agent then exits badly, which is the
541
+ * one path in this file that writes a card `failed`. Without the predicate the
542
+ * user presses Stop and the card reads Failed a second later, about a failure
543
+ * that never happened.
544
+ *
545
+ * Every other card write already carries the guard in SQL: `panel3_answer` and
546
+ * `panel3_give_up` both refuse a run that is not live, and a stopped run is not.
547
+ */
548
+ async function setCardState(client, cardId, state) {
549
+ await returned(client
550
+ .from('panel3_cards')
551
+ .update({ state })
552
+ .eq('id', cardId)
553
+ .neq('state', 'stopped')
554
+ .select('id'), 'set the state of', `card ${cardId}`);
555
+ }
556
+ // ---------------------------------------------------------------------------
557
+ /**
558
+ * One card, from the turns the take handed over to the answer on its thread.
559
+ *
560
+ * ═══ THERE IS NOTHING TO UNDO HERE, AND THAT IS THE POINT. ═══ The take wrote
561
+ * the run row in the same statement that leased the turns, so this function is
562
+ * never in the position of holding a lease it cannot honour: whatever it fails
563
+ * at, the row exists, `cs3 show` can see it, and recovery can reach it. The
564
+ * earlier version wrote the row two round trips later and had to hand the turns
565
+ * back on failure — which covered a thrown error and not a kill, and a kill in
566
+ * that window stranded the card forever.
567
+ *
568
+ * ═══ AND WHAT HAPPENS WHEN THE PROCESS IS OVER IS `settle`'s, NOT THIS
569
+ * FUNCTION'S. ═══ It used to be written out again here, and the two copies
570
+ * were already the same thing said twice: the agent's words onto the card
571
+ * through `writeAnswer`, or the run ended with its reason. recovery Slice 3 gave
572
+ * that moment a second job, starting a process that died badly all over again,
573
+ * and a second copy of THAT is two features owning one respawn, which is the
574
+ * failure recovery-1/ux.md names in as many words. So there is one owner, and
575
+ * the level is what tells it how a run of this shape ends.
576
+ */
577
+ async function answerCard(client, tools, machineId, cardId, turns) {
578
+ const runId = turns[0].run_id;
579
+ /* BEFORE THE SPAWN, AND ITS FAILURE IS THE SPAWN'S FAILURE. The receipts are
580
+ part of what the agent is sent, so a read that fails must not be papered
581
+ over with an empty list: that reads as a card that has made nothing, which
582
+ is how an agent creates a second epic beside the one it cannot see. The
583
+ attachments read carries the same rule: a failed read here must not read
584
+ as "nothing is attached", which is a different card than the one that was
585
+ actually sent. */
586
+ const brief = briefFor(turns[0].card_title, turns, await receiptsFor(client, cardId), await attachmentsFor(client, cardId), await codebasesForRun(client, runId));
587
+ /* ═══ THE RUN ID IS ON THE URL, AND THAT IS THE WHOLE OF WHAT THE AGENT IS
588
+ TOLD ABOUT ITS OWN STANDING. ═══ The tools server reads the level off the
589
+ run row this id names, so the daemon does not tell the child what it may do
590
+ and the child has nothing to claim. `LEVEL` below decides argv only — which
591
+ of the harness's own tools the process gets — and the two can never disagree
592
+ about the record, because only one of them consults it. */
593
+ /* AN EMPTY DIRECTORY OF THE USER'S OWN, because level 1 has no code tool to
594
+ use a real one with, and the daemon's inherited cwd under a launchd login
595
+ item is the filesystem root. */
596
+ const started = startAgent(brief, LEVEL, tools.urlFor(runId), scratchDir());
597
+ out(`run ${runId} level ${LEVEL} ${started.pid ? `pid ${started.pid}` : 'no process'}`);
598
+ try {
599
+ await recordProcess(client, runId, started.pid, brief);
600
+ }
601
+ catch (error) {
602
+ // Said, not fatal. See `recordProcess` for what this costs and why the
603
+ // agent is not killed over it.
604
+ said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
605
+ }
606
+ return settle(client, tools, machineId, LEVEL, runId, cardId, started);
607
+ }
608
+ /**
609
+ * WHICH PROJECT A RUN'S CARD IS FILED UNDER, or null when the card has none.
610
+ *
611
+ * ═══ ASKED OF THE RECORD AT EVERY SPAWN, RATHER THAN CARRIED. ═══ The card is
612
+ * the durable object and its project can change between one dispatch and the
613
+ * next; a value threaded through the daemon would be the state of the card when
614
+ * the daemon last looked. It is one indexed read on the path that is about to
615
+ * start a whole agent.
616
+ *
617
+ * ═══ AND A FAILED READ IS NOT "NO PROJECT". ═══ `returned` throws, which the
618
+ * caller turns into a dispatch that did not happen. Returning null here would
619
+ * quietly send the agent to the machine-wide fallback checkout, which is another
620
+ * project's code.
621
+ */
622
+ async function projectOfRun(client, runId) {
623
+ const rows = await returned(client.from('panel3_runs').select('card:panel3_cards(project_id)').eq('id', runId), 'read', 'which project this run belongs to');
624
+ return rows[0]?.card?.project_id ?? null;
625
+ }
626
+ async function codebasesForRun(client, runId) {
627
+ const projectId = await projectOfRun(client, runId);
628
+ return projectId ? listCodebases(client, projectId) : null;
629
+ }
630
+ async function codebaseOfRun(client, runId) {
631
+ const rows = await returned(client
632
+ .from('panel3_runs')
633
+ .select('codebase_id, card:panel3_cards(project_id)')
634
+ .eq('id', runId), 'read', `which codebase run ${runId} belongs to`);
635
+ const row = rows[0];
636
+ if (!row?.codebase_id || !row.card?.project_id) {
637
+ throw new Error('This code work does not name a registered project codebase.');
638
+ }
639
+ const codebase = (await listCodebases(client, row.card.project_id))
640
+ .find((candidate) => candidate.id === row.codebase_id);
641
+ if (!codebase) {
642
+ throw new Error('The selected codebase is no longer registered on this project.');
643
+ }
644
+ return codebase;
645
+ }
646
+ /**
647
+ * ═══ ONE AGENT, STARTED UNDER ANOTHER THAT IS STILL RUNNING. ═══
648
+ *
649
+ * Called from the `dispatch` tool, and it returns as soon as the process exists.
650
+ * The promise it hands back settles long afterwards, and the caller puts it
651
+ * where the daemon can wait on it rather than awaiting it itself: an agent that
652
+ * waited for the work it started would hold its own run open for the length of
653
+ * it, which is the thing ux.md forbids in as many words.
654
+ *
655
+ * ═══ THE ORDER IS THE WHOLE CORRECTNESS ARGUMENT. ═══ The working copy is
656
+ * resolved FIRST, before anything is written, so a machine that has none refuses
657
+ * the dispatch rather than leaving a run row no process ever corresponded to.
658
+ * Then the row, then the process, then the pid — constraint 8, in the only order
659
+ * that satisfies it.
660
+ */
661
+ async function startChild(client, tools, machineId, parentRunId, brief, codebase) {
662
+ const { data, error } = await client.rpc('panel3_dispatch', {
663
+ p_parent_run_id: parentRunId,
664
+ p_brief: brief,
665
+ p_machine_id: machineId,
666
+ p_codebase_id: codebase.id,
667
+ p_codebase_label: codebase.name,
668
+ });
669
+ if (error)
670
+ throw new Error(`could not start an agent under run ${parentRunId}: ${error.message}`);
671
+ const row = data?.[0];
672
+ if (!row) {
673
+ /* NOTHING WAS WRITTEN AND NOTHING IS RUNNING, and the two reasons are said
674
+ together because the caller cannot tell them apart from here and both mean
675
+ the same thing to it. */
676
+ throw new Error(`NO AGENT WAS STARTED and nothing was written: run ${parentRunId} has ended, or it is already `
677
+ + 'as deep as anything may be sent from.');
678
+ }
679
+ if (row.run_level !== 2 && row.run_level !== 3) {
680
+ /* UNREACHABLE, AND STILL SETTLED. `panel3_dispatch` writes `parent.level + 1`
681
+ from a parent it has just checked is below 3, so there is no level here
682
+ this cannot spawn. If that ever stops being true, the row exists and
683
+ nothing will ever start for it, and leaving it `running` would make a
684
+ recovery sweep wait out the pid grace window to conclude what is already
685
+ known. */
686
+ const why = `run ${row.run_id} was written at level ${row.run_level}, which cannot be spawned`;
687
+ await giveUp(client, row.run_id, why);
688
+ throw new Error(why);
689
+ }
690
+ const level = row.run_level;
691
+ let cwd;
692
+ try {
693
+ cwd = checkoutForCodebase(codebase, hostname());
694
+ }
695
+ catch (error) {
696
+ const why = error instanceof Error ? error.message : String(error);
697
+ await giveUp(client, row.run_id, why);
698
+ throw new Error(`NO AGENT IS RUNNING: ${why}`);
699
+ }
700
+ const started = startAgent(brief, level, tools.urlFor(row.run_id), cwd);
701
+ if (started.pid === null) {
702
+ /* THERE IS A ROW AND THERE IS NO PROCESS, which is the one shape the record
703
+ must never be left in quietly. The answer is already settled — nothing ran
704
+ — so the reason is read off it, the run is ended with that reason on it,
705
+ and the tool call fails saying no agent was started. */
706
+ const answer = await started.answered;
707
+ const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
708
+ await giveUp(client, row.run_id, reason);
709
+ throw new Error(`NO AGENT IS RUNNING: ${reason}`);
710
+ }
711
+ out(`dispatch run ${row.run_id} level ${level} under ${parentRunId} pid ${started.pid}`);
712
+ try {
713
+ await recordProcess(client, row.run_id, started.pid);
714
+ }
715
+ catch (error) {
716
+ // Said, not fatal, exactly as at level 1: the agent is running and killing
717
+ // it over a bookkeeping write would cost the user the work.
718
+ said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
719
+ }
720
+ return {
721
+ runId: row.run_id,
722
+ settled: settle(client, tools, machineId, level, row.run_id, row.run_card_id, started),
723
+ };
724
+ }
725
+ /**
726
+ * HOW MANY PROCESSES HAVE BEEN STARTED FOR THIS RUN, counting the first.
727
+ *
728
+ * ═══ IT SUPPLIES THE NUMBER IN THE SENTENCE, NOT THE BOUND. ═══ The bound is
729
+ * `panel3_resume`'s own predicate, counted and tested in the statement that
730
+ * grants the attempt, so a read here that is a moment out of date cannot grant
731
+ * an attempt the record refuses. The worst it can do is decline one the record
732
+ * would have allowed, which is a run that fails a spawn early rather than a
733
+ * fourth process nobody bounded.
734
+ *
735
+ * A row that is not there is not "one attempt". It is a run this daemon can
736
+ * neither retry nor fail, because there is nothing left to write to, and saying
737
+ * so is the caller's business rather than this function's to paper over.
738
+ */
739
+ async function attemptsSoFar(client, runId) {
740
+ const rows = await returned(client.from('panel3_runs').select('attempts').eq('id', runId), 'read', `how many times a process has been started for run ${runId}`);
741
+ const row = rows[0];
742
+ if (!row)
743
+ throw new Error(`run ${runId} is not there, so nothing can be recorded about it`);
744
+ return row.attempts;
745
+ }
746
+ /**
747
+ * ═══ ONE RUN, ENDED WITH ITS REASON, THE WAY A RUN OF ITS LEVEL ENDS. ═══
748
+ *
749
+ * There are two endings and the difference is not a preference, it is what the
750
+ * run is holding.
751
+ *
752
+ * A DISPATCHED RUN HOLDS NO LEASED TURNS. `panel3_give_up` hands nothing back
753
+ * and puts the card in `failed`, because nothing on that card will pick this
754
+ * piece of work up again unless a question outstanding somewhere else on it says
755
+ * work is genuinely coming back.
756
+ *
757
+ * A LEVEL 1 RUN HOLDS THE PERSON'S MESSAGE. Giving up hands it back for the next
758
+ * `panel3_take_turns` to lease to A BRAND NEW RUN, which is recovery's whole
759
+ * point when the MACHINE is gone and is the wrong ending everywhere else: a new
760
+ * run is a new `attempts`, so the bound below would never be spent, a harness
761
+ * that fails every time would be started again forever, and a coordinator that
762
+ * had already dispatched workers would dispatch them a second time. So level 1
763
+ * ends through `panel3_end_run`, which hands no turns back, and the card says
764
+ * failed.
765
+ *
766
+ * ═══ AND THE FORK LIVES HERE AND NOWHERE ELSE. ═══ FIVE places end a run this
767
+ * way: the settle, and the two failures after the claim in each of `resumeRun`
768
+ * and `startRearmed`. A copy of this rule in each is five places for it to
769
+ * drift, and it drifted the first time it could: those two functions hold the
770
+ * same pair of endings and only one pair was looked at.
771
+ *
772
+ * ALL FIVE CAN SEE LEVEL 1. `resumeRun` serving the retry is what opened its
773
+ * two. `startRearmed`'s were always reachable, because a re-arm has always
774
+ * served level 1, and were always ending one the way a dispatched run ends. The
775
+ * bound is what made that matter: a handed-back message is leased to a NEW run,
776
+ * whose attempts start again at one.
777
+ *
778
+ * ═══ AND THE GIVE-UPS THAT ARE NOT HERE ARE NOT OVERSIGHTS. ═══ `startChild`'s
779
+ * two end a dispatched run by construction, since a dispatch writes its parent's
780
+ * level plus one and there is no parent below level 1. The bad-level branches in
781
+ * `resumeRun` and `startRearmed` hold a level that is by definition none of the
782
+ * three, so there is no fork to make. And `recoverStranded` gives a level 1 run
783
+ * up ON PURPOSE, handing its message back, which is the whole of how another
784
+ * machine takes over a run whose machine is gone.
785
+ *
786
+ * Returns whether THIS call is what ended it. False is a fact about the record,
787
+ * not a failure: the run had already ended, and the caller says so.
788
+ */
789
+ async function endRun(client, level, runId, cardId, why) {
790
+ if (level !== 1)
791
+ return (await giveUp(client, runId, why)) !== null;
792
+ /* ═══ THE RUN FIRST, AND THE CARD ONLY IF THIS RUN WAS STILL THE CARD'S TO
793
+ FAIL. ═══
794
+ The card used to be written first, and the argument for that was a daemon
795
+ killed between the two: it left a card reading `failed` beside a run that
796
+ still read running with a dead process, which the record's own forbidden
797
+ state check reports and recovery corrects. THE PREEMPTION MAKES THAT ORDER A
798
+ LIE THAT NOTHING CORRECTS. A coordinator the person's next message replaced
799
+ is `stopped`, its process is killed by this same daemon, and it arrives here
800
+ having failed: writing the card `failed` then marks the card that a FRESH
801
+ coordinator is working on, about a run nobody is waiting for.
802
+ `panel3_end_run` is what knows the difference, because it refuses a run that
803
+ has already ended, so its answer is what the card write waits for.
804
+ WHAT THE ORDER COSTS is a daemon killed between the two, which leaves the
805
+ run `failed` and the card still reading `working` with nothing on it. The
806
+ product raises that as a forbidden state on the card itself, and the
807
+ person's next message takes the card again. That is a smaller and louder
808
+ wrong than a card marked failed under an agent that is working. */
809
+ const ended = await failRun(client, runId, why);
810
+ if (ended)
811
+ await setCardState(client, cardId, 'failed');
812
+ return ended;
813
+ }
814
+ /**
815
+ * ═══ THE END OF ONE RUN'S PROCESS, WHICHEVER SPAWN STARTED IT. ═══
816
+ *
817
+ * One owner for what happens when an agent's process exits, because there are
818
+ * four ways one comes to exist, being a card taken, a `dispatch`, a resume of a
819
+ * run whose process was found gone, and a re-arm, and they end identically. A
820
+ * second copy would be a second answer to "what does a dead agent mean", and the
821
+ * first thing to drift would be whether the card is told.
822
+ *
823
+ * ═══ `speaksToTheCard` IS THE ONE THING THAT DIFFERS ON THE WAY IN, AND IT IS
824
+ * ux.md's RULE RATHER THAN A SETTING. ═══ "Outbound follows the work." A run
825
+ * started again only to settle a question from somebody it sent did no work for
826
+ * the person: its words went to that agent, through the answer, and putting them
827
+ * on the card as well is a voice in a conversation the person did not ask to
828
+ * follow. It is still ended, and the card is still settled, because it can be
829
+ * the last thing live on it.
830
+ *
831
+ * ═══ AND `level` IS THE ONE THING THAT DIFFERS ON THE WAY OUT, WHICH IS
832
+ * `endRun`'s. ═══ It is above, with the whole argument: a dispatched run
833
+ * holds no leased turns and a level 1 run holds the person's message, so they
834
+ * cannot end the same way. What `answerCard` always did is what level 1 still
835
+ * does, and the only thing that has moved is where it is written.
836
+ */
837
+ function settle(client, tools, machineId, level, runId, cardId, started, speaksToTheCard = true) {
838
+ return started.answered.then(async (answer) => {
839
+ /* ═══ A RUN THAT STOPPED TO ASK DID NOT DIE, WHATEVER THE HARNESS PRINTED
840
+ ON ITS WAY OUT. ═══
841
+
842
+ ux.md: asking is TERMINAL for the agent that asks. It calls the tool, is
843
+ told "you have stopped", and exits — and it owes the person nothing more,
844
+ because its last words were the question and the question is already on
845
+ the card. So a harness that exits having said nothing to the person has
846
+ done exactly what it was told to.
847
+
848
+ `codexAnswer` reads that silence as a death (`spawn.ts`: "codex exited 0
849
+ without saying anything to the person"), which is the right reading of
850
+ every OTHER silent exit and the wrong one of this one. Claude happens to
851
+ emit a closing message after the tool call and Codex does not, so without
852
+ this the same question asked under the two harnesses ends two different
853
+ ways: answerable on one, and on the other retried twice and then FAILED
854
+ with the question still sitting open on a failed card.
855
+
856
+ ═══ SO THE RECORD DECIDES IT, NOT THE STDOUT. ═══ Whether a run asked is a
857
+ column, written by `panel3_ask` in the same statement that stopped the
858
+ run, and it is true or false identically for both harnesses. Read only on
859
+ the failure path, where it is the one thing that can turn a death back
860
+ into an ordinary ending, and then handed to `writeAnswer` — which already
861
+ owns this case and already prints its sentence — rather than to a second
862
+ ending written here beside it. */
863
+ if (!answer.ok && (await runNow(client, runId)).state === 'asked') {
864
+ await writeAnswer(client, runId, cardId, null);
865
+ return;
866
+ }
867
+ if (!answer.ok) {
868
+ /* ═══ A PROCESS THAT STARTED AND THEN DIED BADLY IS STARTED AGAIN. ═══
869
+ recovery-1/ux.md, Slice 3: "the agent's harness fails on something that
870
+ is nobody's fault: a rate limit, a dropped connection, a provider
871
+ outage. The product tries again rather than failing the card."
872
+
873
+ `started.pid` IS THE WHOLE OF THE DISTINCTION, and it is a fact rather
874
+ than a reading of a string. What reaches here as `{ ok: false }` is
875
+ never a verdict about the work. An agent that ran and concluded it
876
+ could not do the thing exits 0 having said so, and that is an ANSWER,
877
+ written to the card below. So what is left to separate is a process that
878
+ ran and died from one that never existed, and `Started.pid` answers it:
879
+ `could not start claude on this machine: ENOENT` and `claude is not
880
+ installed` are facts about this machine that will be identical in two
881
+ seconds and are fixed by a person, while `claude exited 1: API Error:
882
+ 529 Overloaded` is what a rate limit, a dropped connection and a
883
+ provider outage all look like from here. plan-slice-3.md section 1.2
884
+ measured the alternative and rejected it: the provider's own words reach
885
+ this daemon as a 500 character tail of one of two streams, absent
886
+ entirely for a rate limit, and different again next release.
887
+
888
+ WHAT THIS DELIBERATELY GIVES UP is that a genuinely deterministic
889
+ harness crash costs two extra spawns before the card fails. That is
890
+ bounded, and it is the price of not depending on a string.
891
+
892
+ THE RESPAWN IS `resumeRun`, WHICH ALREADY EXISTS. ux.md: "two features
893
+ owning one respawn is how two respawns are born." Nothing new starts a
894
+ process here; the failure calls the sweep's own respawn, fenced by the
895
+ pid this daemon watched exit, and AWAITS it, so the daemon's one
896
+ `inFlight` entry covers the whole chain and `--once` waits for the last
897
+ attempt rather than exiting mid-retry.
898
+
899
+ AND THE OLD RULING HERE IS AMENDED RATHER THAN DELETED. It read: "a
900
+ process that started and then failed is a failure of the work, not of
901
+ the machine: resuming it would put the same agent back on the same brief
902
+ to fail the same way, FOREVER." The half that survives is forever, and
903
+ the bound is what now provides it. */
904
+ const attempts = await attemptsSoFar(client, runId);
905
+ if (started.pid !== null && attempts < MAX_ATTEMPTS) {
906
+ out(`retry run ${runId} attempt ${attempts} of ${MAX_ATTEMPTS} died: ${answer.reason}`);
907
+ const again = await resumeRun(client, tools, machineId, runId, started.pid);
908
+ if (again)
909
+ return again.settled;
910
+ /* ═══ THE CLAIM MATCHED NOTHING, AND THAT IS NOT ALWAYS SOMEBODY ELSE
911
+ HOLDING THIS RUN, SO THE FAILURE IS STILL WRITTEN. ═══
912
+ Six predicates can refuse a retry and two of them mean NOTHING HAS
913
+ MOVED AT ALL. `pid is null` is the ordinary one, and not exotic:
914
+ `recordProcess` is caught and carried on from at all four spawn sites
915
+ precisely because that write is expected to be able to fail, and when
916
+ it does, the process still ran and still died with a reason nobody
917
+ else has. `attempts` reaching the bound between the read above and the
918
+ claim is the other. RETURNING HERE WOULD LEAVE THE RUN `running` WITH
919
+ A DEAD PROCESS AND THE CARD STILL SAYING WORKING, and the sentence a
920
+ person eventually read would be the next sweep's guess about a daemon
921
+ that never recorded a pid rather than the harness's own words.
922
+
923
+ SO IT FALLS THROUGH, AND THE ENDING IS THE ONE THIS PATH HAD BEFORE
924
+ THE RETRY EXISTED. A run that ended or stopped under this daemon is
925
+ refused by `panel3_end_run` and `panel3_give_up` on their own
926
+ predicates, and that refusal is said below rather than recorded. A run
927
+ another machine has taken over is not: neither ending is fenced by the
928
+ machine or the pid, so this daemon ends it, exactly as it did before
929
+ this slice. That is a property of the ending rather than of the retry,
930
+ it is narrower than it was (the claim is fenced, so no second PROCESS
931
+ starts), and closing it is a change to what a give-up may write. */
932
+ said(`run ${runId} could not be started again, so it is ending on: ${answer.reason}`);
933
+ }
934
+ /* ═══ AND WHEN THE BOUND IS SPENT THE CARD FAILS AND SAYS HOW MANY
935
+ ATTEMPTS IT TOOK. ═══ ux.md asks for that in as many words. It is said
936
+ once, here, into the `failed_because` column the panel and the terminal
937
+ both already read, so the count reaches a person from one place rather
938
+ than two. */
939
+ const why = attempts > 1 ? `${answer.reason} (after ${attempts} attempts)` : answer.reason;
940
+ const ended = await endRun(client, level, runId, cardId, why);
941
+ if (!ended) {
942
+ /* IT WAS ALREADY SETTLED, by recovery, which decided this process was
943
+ gone before it said so itself, or by the person's Stop. Nothing was
944
+ written here, and that is said rather than reported as a failure this
945
+ daemon recorded. */
946
+ said(`run ${runId} had already ended, so nothing was recorded: ${why}`);
947
+ return;
948
+ }
949
+ out(level === 1
950
+ ? `failed card ${cardId} run ${runId} ${why}`
951
+ : `failed run ${runId} ${why}`);
952
+ return;
953
+ }
954
+ /* ═══ ITS OWN WORDS, ONTO THE CARD, THROUGH THE SAME STATEMENT EVERY LEVEL
955
+ USES. ═══ Nothing reads them, nothing shortens them and nothing waits to
956
+ approve them: ux.md's "whoever did the work writes the answer". */
957
+ await writeAnswer(client, runId, cardId, speaksToTheCard ? answer.text : null);
958
+ });
959
+ }
960
+ /**
961
+ * ═══ ONE RUN, STARTED AGAIN AS ITSELF, WITH WHAT IT WAS SENT AND WHAT IT HAD
962
+ * GOT AS FAR AS. ═══
963
+ *
964
+ * ux.md: "a respawned agent reads three things: its brief, its own report, and
965
+ * whatever is new", and all three arrive in the prompt because "an agent cannot
966
+ * fail to read its own input".
967
+ *
968
+ * ═══ THE SAME ROW, WHICH IS THE PART THAT MATTERS. ═══ Its children point at
969
+ * this id. A new row would be a run whose children belong to a dead parent, and
970
+ * the agent would dispatch them again — the failure ux.md names as the single
971
+ * most expensive one observed. Nothing is inserted here for the same reason
972
+ * nothing needs to be: constraint 8 is satisfied by a row that already exists.
973
+ *
974
+ * ═══ TWO CALLERS, AND `afterPid` IS THE WHOLE OF WHAT SEPARATES THEM. ═══
975
+ *
976
+ * NULL is the sweep, which INFERS that a process is gone from a pid the
977
+ * operating system no longer knows: `recoverStranded` and `takeHandedBack`. It
978
+ * behaves exactly as it always has, level 1 refused and the claim window
979
+ * enforced, and it does not spend an attempt, because a laptop that slept is not
980
+ * a provider that failed.
981
+ *
982
+ * A PID is the retry, made by the daemon that WATCHED that process exit. It may
983
+ * resume any level, it is fenced by the pid rather than by the window, and the
984
+ * claim counts the attempt and refuses past three. See 20260909090000 for why
985
+ * each of those three predicates is about the sweep and not about a failure this
986
+ * daemon saw with its own eyes.
987
+ *
988
+ * recovery-1/ux.md is why there is one function and not two: "two features
989
+ * owning one respawn is how two respawns are born."
990
+ *
991
+ * ═══ AND THE ORDER IS THE SAME ARGUMENT `startChild` MAKES, ON THE SWEEP'S
992
+ * PATH. ═══ The working copy is resolved BEFORE the claim, so a machine that
993
+ * has none refuses without having taken a run it cannot start; then the claim,
994
+ * then the process, then the pid.
995
+ *
996
+ * ON THE RETRY'S PATH IT CANNOT BE, AND THAT IS `startRearmed`'s SITUATION
997
+ * EXACTLY. A level 1 run works in an empty directory of the user's own and
998
+ * anything deeper gets the working copy, and the level is not known until the
999
+ * claim returns. So the resolution happens after the claim, and a failure there
1000
+ * ends the run with its reason rather than leaving a claimed run sitting
1001
+ * `running` with no process. The sweep's ordering is untouched, because its
1002
+ * argument still holds there.
1003
+ *
1004
+ * ═══ THE CHILDREN ARE READ AFTER THE CLAIM AND AS LATE AS POSSIBLE. ═══ They
1005
+ * are the volatile half: one of them may finish while this function is running,
1006
+ * and the prompt says out loud that the list is a moment old and is re-readable.
1007
+ * Carrying them in the report instead is the thing ux.md forbids outright.
1008
+ *
1009
+ * Returns null when the claim matched nothing, which is a fact about the record
1010
+ * rather than a failure: nothing was started here. On the sweep's path that
1011
+ * means another daemon got there first. On the retry's it means the same thing
1012
+ * said by the pid having moved, OR that this run has had its three processes and
1013
+ * the record refuses a fourth, and the caller is the one that knows what to say
1014
+ * about the failure it was holding.
1015
+ */
1016
+ async function resumeRun(client, tools, machineId, runId, afterPid) {
1017
+ /* ═══ THE SWEEP RESOLVES ITS WORKING COPY BEFORE THE CLAIM AND THE RETRY
1018
+ CANNOT, AND `afterPid` IS THE WHOLE OF WHAT DECIDES IT. ═══ Said once, here.
1019
+ See the header for both halves of the argument: a machine with no checkout
1020
+ must not take a run it cannot start, and the level a retry needs the answer
1021
+ for is not known until the claim returns. */
1022
+ const sweptCwd = afterPid === null
1023
+ ? checkoutForCodebase(await codebaseOfRun(client, runId), hostname())
1024
+ : null;
1025
+ const { data, error } = await client.rpc('panel3_resume', {
1026
+ p_run_id: runId,
1027
+ p_machine_id: machineId,
1028
+ p_after_pid: afterPid,
1029
+ });
1030
+ if (error)
1031
+ throw new Error(`could not start run ${runId} again: ${error.message}`);
1032
+ const claimed = data?.[0];
1033
+ if (!claimed)
1034
+ return null;
1035
+ const level = claimed.run_level === 1 ? 1 : claimed.run_level === 2 ? 2 : 3;
1036
+ if (claimed.run_level !== level) {
1037
+ /* UNREACHABLE, AND STILL SETTLED, exactly as in `startChild`: the level
1038
+ column is checked at three, so there is no level here this cannot spawn.
1039
+ The claim has already happened, so leaving it would strand the run for a
1040
+ whole grace window before anything looked at it again. */
1041
+ const why = `run ${runId} is at level ${claimed.run_level}, which cannot be spawned`;
1042
+ await giveUp(client, runId, why);
1043
+ throw new Error(why);
1044
+ }
1045
+ /* NON-NULL EXACTLY WHEN THE SWEEP RESOLVED ONE, which is the rule stated at
1046
+ the assignment above. Branching on the VALUE rather than testing `afterPid`
1047
+ a second time is what keeps the resolution and its use from being able to
1048
+ disagree: whatever the sweep resolved is what the sweep runs in. */
1049
+ let cwd;
1050
+ if (sweptCwd !== null) {
1051
+ cwd = sweptCwd;
1052
+ }
1053
+ else {
1054
+ try {
1055
+ cwd = level === 1
1056
+ ? scratchDir()
1057
+ : checkoutForCodebase(await codebaseOfRun(client, runId), hostname());
1058
+ }
1059
+ catch (error) {
1060
+ /* ═══ THE REASON IS WRITTEN TO THE DATABASE, SO IT MAY NOT CARRY A PATH.
1061
+ ═══ Constraint 6, and `startRearmed`'s own handling of the same two
1062
+ calls: `workingCopy()` is already careful about this and says so;
1063
+ `scratchDir()` is not, because it is `mkdirSync`, whose EACCES and
1064
+ ENOTDIR messages name the directory they failed on. So the machine's own
1065
+ error is kept for stderr and the record is told only what is true and
1066
+ shareable. */
1067
+ const stderrOnly = error instanceof Error ? error.message : String(error);
1068
+ const why = level === 1
1069
+ ? 'this machine could not make the empty directory this runs in'
1070
+ : stderrOnly;
1071
+ /* ═══ AND IT ENDS THE WAY A RUN OF THIS LEVEL ENDS. ═══ It was
1072
+ `panel3_give_up` outright, which was right while only a dispatched run
1073
+ could reach this function and is a leak now that the retry brings level
1074
+ 1 here: handing the person's message back mints a new run with its
1075
+ attempts at one, which is the bound the retry is under, undone by the
1076
+ one path that could not start. See `endRun`. */
1077
+ await endRun(client, level, runId, claimed.run_card_id, why);
1078
+ throw new Error(`NO AGENT IS RUNNING: ${level === 1 ? stderrOnly : why}`);
1079
+ }
1080
+ }
1081
+ /* WHAT IT SENT OTHERS TO DO, FROM THE RECORD. Level 3 has no `dispatch`, so it
1082
+ has no children to have: null says that, where an empty list would say it
1083
+ chose to send nobody.
1084
+
1085
+ ═══ AND LEVEL 1 HAS THEM TOO, WHICH ONLY THE RETRY CAN REACH. ═══ This read
1086
+ used to be `level === 2`, which was correct only because nothing at level 1
1087
+ ever got here. Left alone it would hand a coordinator a null child list, and
1088
+ a coordinator that came back to no children dispatches its workers a second
1089
+ time: ux.md's single most expensive failure. `startRearmed` reads it the
1090
+ same way, for the same reason. */
1091
+ const children = level === 3 ? null : await childrenOf(client, runId);
1092
+ const started = startAgent(
1093
+ /* ═══ WHY IT DIED IS WHAT DIFFERS, AND IT IS TOLD THE TRUTH ABOUT IT. ═══
1094
+ `resumePrompt` opens by saying the machine went down, which is true of the
1095
+ sweep and false of a retry: the daemon that watched this harness exit is
1096
+ still running. See `retryPrompt`. */
1097
+ afterPid === null
1098
+ ? resumePrompt(claimed.run_brief, claimed.run_report, children)
1099
+ : retryPrompt(claimed.run_brief, claimed.run_report, children), level, tools.urlFor(runId), cwd);
1100
+ if (started.pid === null) {
1101
+ /* THE CLAIM HAPPENED AND NO PROCESS DID, which is the one shape the record
1102
+ must never be left in quietly. Same handling as a dispatch that could not
1103
+ start: the reason is read off the settled answer and the run is ended with
1104
+ it. */
1105
+ const answer = await started.answered;
1106
+ const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
1107
+ // The same level fork, for the same reason. See `endRun`.
1108
+ await endRun(client, level, runId, claimed.run_card_id, reason);
1109
+ throw new Error(`NO AGENT IS RUNNING: ${reason}`);
1110
+ }
1111
+ out(`resume run ${runId} level ${level} pid ${started.pid} `
1112
+ + `${claimed.run_report === null ? 'no report to carry' : 'carrying its report'}`);
1113
+ try {
1114
+ await recordProcess(client, runId, started.pid);
1115
+ }
1116
+ catch (error) {
1117
+ // Said, not fatal, as everywhere else: the agent is running and killing it
1118
+ // over a bookkeeping write would cost the user the work a second time.
1119
+ said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
1120
+ }
1121
+ return { settled: settle(client, tools, machineId, level, runId, claimed.run_card_id, started) };
1122
+ }
1123
+ /**
1124
+ * ═══ ONE RUN, STARTED AGAIN BECAUSE SOMETHING IT WAS WAITING ON EXISTS NOW. ═══
1125
+ *
1126
+ * The other half of `resumeRun`, and deliberately the same shape: same row, same
1127
+ * id, same brief, the report carried, the children read live. Two facts differ,
1128
+ * and both come off the row `panel3_take_rearms` handed over.
1129
+ *
1130
+ * WHAT IS NEW. The answer to this run's own question, a question from somebody
1131
+ * it sent that it has to answer or pass on, or everybody it sent having
1132
+ * finished. `ask_id` and `mine` say which, and they are the record's word
1133
+ * rather than an inference here.
1134
+ *
1135
+ * WHERE IT RUNS. A level 1 run is started in an empty directory of the user's
1136
+ * own, because level 1 has no code tool to use a real one with; anything deeper
1137
+ * gets the working copy. That is `answerCard` and `startChild`'s rule, and this
1138
+ * is the one path that can be handed either.
1139
+ *
1140
+ * ═══ THE CLAIM HAS ALREADY HAPPENED WHEN THIS IS CALLED, SO A FAILURE ENDS THE
1141
+ * RUN RATHER THAN LEAVING IT. ═══ `panel3_take_rearms` stamps the claim in the
1142
+ * same statement that makes it — the question delivered, and the run's end
1143
+ * cleared — which is what stops two daemons starting two processes for it and
1144
+ * what stops one question, or one set of finished workers, spawning forever. The cost is that this cannot decline afterwards: a run claimed and not
1145
+ * started would sit `running` with no process and its question already spent. So
1146
+ * every failure below ends the run with its reason, which is visible on the card
1147
+ * and in `cs3 show`, rather than quietly hoping the next poll finds it.
1148
+ */
1149
+ async function startRearmed(client, tools, machineId, row) {
1150
+ const level = row.run_level === 1 ? 1 : row.run_level === 2 ? 2 : 3;
1151
+ if (row.run_level !== level) {
1152
+ const why = `run ${row.run_id} is at level ${row.run_level}, which cannot be spawned`;
1153
+ await giveUp(client, row.run_id, why);
1154
+ throw new Error(why);
1155
+ }
1156
+ let cwd;
1157
+ try {
1158
+ cwd = level === 1
1159
+ ? scratchDir()
1160
+ : checkoutForCodebase(await codebaseOfRun(client, row.run_id), hostname());
1161
+ }
1162
+ catch (error) {
1163
+ /* ═══ THE REASON IS WRITTEN TO THE DATABASE, SO IT MAY NOT CARRY A PATH.
1164
+ ═══ Constraint 6. `workingCopy()` is already careful about this and says
1165
+ so; `scratchDir()` is not — it is `mkdirSync`, whose EACCES and ENOTDIR
1166
+ messages name the directory they failed on — and `failed_because` is a
1167
+ column `cs3 show` prints. So the machine's own error is kept for stderr
1168
+ and the record is told only what is true and shareable. */
1169
+ const said = error instanceof Error ? error.message : String(error);
1170
+ const why = level === 1
1171
+ ? 'this machine could not make the empty directory this runs in'
1172
+ : said;
1173
+ // The level fork, which this path needs for the same reason `resumeRun`'s
1174
+ // two do: a re-arm serves level 1, and a level 1 run holds the person's
1175
+ // message. See `endRun`.
1176
+ await endRun(client, level, row.run_id, row.run_card_id, why);
1177
+ throw new Error(`NO AGENT IS RUNNING: ${level === 1 ? said : why}`);
1178
+ }
1179
+ /* WHAT IT SENT OTHERS TO DO AND WHAT THEY WROTE, FROM THE RECORD, AS LATE AS
1180
+ POSSIBLE. Level 3 has no `dispatch`, so it has no children to have: null
1181
+ says that, where an empty list would say it chose to send nobody. */
1182
+ const children = level === 3 ? null : await childrenOf(client, row.run_id);
1183
+ /* ═══ THREE REASONS, AND THE ROW SAYS WHICH. ═══ No question is ux.md's third
1184
+ re-arm: everybody it sent has finished, and it is started to read them back.
1185
+ `children` cannot be null on that path — only a run with children is ever
1186
+ claimed for it — and the prompt takes the list rather than the maybe-list so
1187
+ that is a fact of the signature rather than of a comment. */
1188
+ const prompt = row.ask_id === null
1189
+ ? readBackPrompt(row.run_brief, row.run_report, children ?? [])
1190
+ : row.mine
1191
+ ? answerPrompt(row.run_brief, row.run_report, children, row.question ?? '', row.answer ?? '')
1192
+ : escalationPrompt(row.run_brief, row.run_report, children, row.ask_id, row.question ?? '');
1193
+ const started = startAgent(prompt, level, tools.urlFor(row.run_id), cwd);
1194
+ if (started.pid === null) {
1195
+ const answer = await started.answered;
1196
+ const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
1197
+ // The same level fork, for the same reason. See `endRun`.
1198
+ await endRun(client, level, row.run_id, row.run_card_id, reason);
1199
+ throw new Error(`NO AGENT IS RUNNING: ${reason}`);
1200
+ }
1201
+ out(`rearm run ${row.run_id} level ${level} pid ${started.pid} `
1202
+ + `${row.ask_id === null
1203
+ ? 'to read back everybody it sent'
1204
+ : row.mine
1205
+ ? 'with the answer to its own question'
1206
+ : 'with a question it has to settle'}`);
1207
+ try {
1208
+ await recordProcess(client, row.run_id, started.pid);
1209
+ }
1210
+ catch (error) {
1211
+ // Said, not fatal, as everywhere else: the agent is running and killing it
1212
+ // over a bookkeeping write would cost the user the work.
1213
+ said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
1214
+ }
1215
+ /* ═══ AND IT SPEAKS TO THE CARD ONLY IF IT IS DOING THE PERSON'S WORK. ═══
1216
+ Exactly the distinction ux.md draws: a run carrying on with its own work has
1217
+ something to say when it finishes, and a run started only to settle somebody
1218
+ else's question does not. A run started to read back everybody it sent is
1219
+ the first kind and the clearest case of it — that reply IS the answer to the
1220
+ request. See `settle`. */
1221
+ return {
1222
+ settled: settle(client, tools, machineId, level, row.run_id, row.run_card_id, started, row.ask_id === null || !!row.mine),
1223
+ };
1224
+ }
1225
+ /**
1226
+ * The runs one run dispatched, AND WHAT EACH OF THEM WROTE, in the words the
1227
+ * prompt prints them in. The same question `list_child_runs` and
1228
+ * `get_child_report` answer, asked here so the answer is IN the prompt rather
1229
+ * than waiting for the agent to think of asking.
1230
+ *
1231
+ * ═══ THE REPORTS ARE HERE BECAUSE OF WHAT THIS IS FOR. ═══ ux.md: "the level 2
1232
+ * agent does not hold its three workers' findings in its head while they run.
1233
+ * Each worker writes what it found into the record, and the level 2 agent reads
1234
+ * them back." The respawn that exists to do exactly that is fired by the record
1235
+ * and given the record; leaving the reports behind a tool call would make the
1236
+ * whole synthesis conditional on an agent choosing to make it, which is the
1237
+ * difference ux.md draws between a guarantee and a hope.
1238
+ *
1239
+ * AND THEY ARE NOT TRUNCATED. A report is rewritten state rather than an append
1240
+ * log, which is what keeps it bounded; cutting one here would hand the agent a
1241
+ * half-sentence and no way to tell it was cut.
1242
+ */
1243
+ async function childrenOf(client, runId) {
1244
+ const children = await returned(client
1245
+ .from('panel3_runs')
1246
+ .select('id, level, state, activity, report, failed_because, ended_at')
1247
+ .eq('parent_run_id', runId)
1248
+ .order('started_at'), 'read', `what run ${runId} sent others to do`);
1249
+ return children.flatMap((c) => [
1250
+ [
1251
+ c.id,
1252
+ `L${c.level}`,
1253
+ c.state === 'running' && !c.ended_at ? 'still going' : c.state,
1254
+ c.activity ?? '',
1255
+ ].join(' ').trimEnd(),
1256
+ /* ONE OWNER PER COLUMN, PRINTED AS TWO FACTS. `report` is what the run
1257
+ wrote and `failed_because` is why the product ended it, and running them
1258
+ together would have an agent read the daemon's sentence as its worker's
1259
+ finding. */
1260
+ ...(c.failed_because === null ? [] : [` it was ended: ${c.failed_because}`]),
1261
+ ...(c.report === null
1262
+ ? [' it wrote nothing down']
1263
+ : c.report.split('\n').map((l) => ` ${l}`)),
1264
+ ]);
1265
+ }
1266
+ // ---------------------------------------------------------------------------
1267
+ /**
1268
+ * ═══ THE TWO ENDS THE PERSON CAUSES, WHICH ARE THE TWO THIS DAEMON KILLS. ═══
1269
+ *
1270
+ * `stopped` is their Stop and `superseded` is the coordinator their next message
1271
+ * replaced. The record keeps them apart on purpose, because a stop is read
1272
+ * across the whole card and being replaced says nothing about anybody else's
1273
+ * work. HERE they are one thing and only one: a process nothing will ever read
1274
+ * from again.
1275
+ */
1276
+ const ENDED_BY_THE_PERSON = ['stopped', 'superseded'];
1277
+ /** When this machine came up. Anything that started before it did has no process
1278
+ * left, whatever a pid lookup says about that number today. */
1279
+ const bootedAt = () => Date.now() - uptime() * 1000;
1280
+ /**
1281
+ * ═══ IS THE PROCESS THIS RUN RECORDED STILL THERE. ═══
1282
+ *
1283
+ * `processIsAlive` is `kill(pid, 0)`, which answers "some process has this
1284
+ * number", not "the process I started has this number". `machine_id` is derived
1285
+ * from the hardware and survives a reboot, pids restart low, so after a restart
1286
+ * a run's recorded pid regularly matches something unrelated and live. A run
1287
+ * that started before this machine booted has no process left by definition,
1288
+ * whatever the number says.
1289
+ *
1290
+ * ONE ANSWER FOR BOTH READERS. Recovery asks it to decide whether to reap a run,
1291
+ * and the stop asks it to decide whether there is anything left to kill. The
1292
+ * second is the one where a wrong yes is worst, because it signals a pid this
1293
+ * machine no longer owns.
1294
+ */
1295
+ function runProcessIsAlive(run, booted) {
1296
+ if (run.pid === null)
1297
+ return false;
1298
+ // The attempt now running began at the resume, if there was one.
1299
+ if (new Date(run.resumed_at ?? run.started_at).getTime() < booted)
1300
+ return false;
1301
+ return processIsAlive(run.pid);
1302
+ }
1303
+ /**
1304
+ * ═══ THE USER STOPPED IT, OR THEIR NEXT MESSAGE REPLACED IT, SO THE PROCESS
1305
+ * STOPS. ═══
1306
+ *
1307
+ * ux.md: "The user's Stop is absolute, and nothing else stops work."
1308
+ * `panel3_stop_card` is the whole of the decision, ending the runs and the card
1309
+ * in one statement, and this is the effect.
1310
+ *
1311
+ * ═══ AND `superseded` IS READ OFF THE SAME QUERY, NOT SWEPT SEPARATELY. ═══ A
1312
+ * coordinator the person's next message replaced has a process that will never
1313
+ * be read from again, exactly like a stopped one, and that is the ONE thing the
1314
+ * two states share here. Two sweeps would be two owners of "kill what is not
1315
+ * coming back", and the second would be the one that fell behind. THE RECORD IS THE INTENT AND THE
1316
+ * KILL IS THE EFFECT, in that order, so a machine that is offline when the
1317
+ * button is pressed still honours it on the poll after it comes back.
1318
+ *
1319
+ * ONLY THIS MACHINE'S RUNS, for the same reason recovery only sweeps its own: a
1320
+ * pid is a local fact, and no other machine can act on this one's.
1321
+ *
1322
+ * ═══ A PROCESS THAT IS ALREADY GONE IS NOT AN ERROR. ═══ It is the ordinary
1323
+ * case, being an agent that finished in the second before the button or a
1324
+ * daemon that was restarted since, and there is nothing to report about it.
1325
+ *
1326
+ * ═══ AND THE PID IS CLEARED, WHICH IS WHAT KEEPS THIS SET FROM BECOMING A
1327
+ * MINEFIELD. ═══ A stopped run is stopped forever, so without this its pid
1328
+ * would be re-examined on every poll for the life of the machine, and pids are
1329
+ * reused. Sooner or later that number belongs to something else that is alive
1330
+ * and innocent, and this would signal it. Cleared, the row means what it says:
1331
+ * this machine has no process for this run. It is the same `pid = null` a resume
1332
+ * writes, for the same reason.
1333
+ *
1334
+ * AND ONLY WHEN THAT IS TRUE. A refused signal leaves the pid where it is, so
1335
+ * the next poll tries again: clearing it after an EPERM would say this machine
1336
+ * has no process for a run whose agent is still working.
1337
+ */
1338
+ async function killStopped(client, machineId) {
1339
+ const stopped = await returned(client
1340
+ .from('panel3_runs')
1341
+ .select('id, card_id, pid, started_at, resumed_at')
1342
+ .eq('machine_id', machineId)
1343
+ .in('state', ENDED_BY_THE_PERSON)
1344
+ .not('pid', 'is', null), 'read', 'the runs on this machine that are not coming back');
1345
+ if (stopped.length === 0)
1346
+ return;
1347
+ const booted = bootedAt();
1348
+ for (const run of stopped) {
1349
+ if (runProcessIsAlive(run, booted)) {
1350
+ try {
1351
+ process.kill(run.pid);
1352
+ out(`stopped run ${run.id} card ${run.card_id} killed pid ${run.pid}`);
1353
+ }
1354
+ catch (error) {
1355
+ said(`could not stop pid ${run.pid} for run ${run.id}: ${error instanceof Error ? error.message : String(error)}`);
1356
+ }
1357
+ /* ═══ SIGNALLED IS NOT DEAD, SO THE PID STAYS ON THE ROW. ═══ `kill`
1358
+ returning means the signal was delivered, not that the process acted on
1359
+ it: an agent mid-write, or one ignoring SIGTERM, is still there and
1360
+ still holding the working copy. Clearing the pid now would take the row
1361
+ out of this read for good and it would never be signalled again. The
1362
+ next poll finds it either gone, and clears it below, or still alive, and
1363
+ signals it again. The same line covers a refused signal, where the
1364
+ process is certainly still there. */
1365
+ continue;
1366
+ }
1367
+ try {
1368
+ await returned(client
1369
+ .from('panel3_runs')
1370
+ .update({ pid: null })
1371
+ // STILL NOT COMING BACK. Nothing starts either of these states, so
1372
+ // this cannot lose a race, and saying so in the statement is what
1373
+ // keeps that true if something one day does.
1374
+ .eq('id', run.id)
1375
+ .in('state', ENDED_BY_THE_PERSON)
1376
+ .select('id'), 'clear the process id of', `run ${run.id}`);
1377
+ }
1378
+ catch (error) {
1379
+ // Said, not fatal. The kill has already happened; this is bookkeeping, and
1380
+ // the next poll finds the row again and tries once more.
1381
+ said(`${error instanceof Error ? error.message : String(error)} (it was stopped all the same)`);
1382
+ }
1383
+ }
1384
+ }
1385
+ /**
1386
+ * ═══ A RUN WHOSE PROCESS IS GONE GOES BACK TO TAKEABLE. ═══
1387
+ *
1388
+ * Only this machine's runs, because liveness is a local fact: no other machine
1389
+ * can honestly say whether this one's pid is still there, and guessing from a
1390
+ * timestamp is how a run that is working gets declared dead.
1391
+ *
1392
+ * A run with NO PID RECORDED counts as gone — there is no process to be alive —
1393
+ * but only once it is neither this process's own nor inside `PID_GRACE_MS` of
1394
+ * starting. `mine` covers this daemon; the grace window covers a SECOND daemon
1395
+ * on the same machine, whose `mine` set does not and cannot contain the first
1396
+ * one's runs. Without it, starting a second daemon reaped the first daemon's
1397
+ * live work: turns handed back and a run marked failed while its agent was
1398
+ * genuinely running, producing a duplicate answer.
1399
+ *
1400
+ * ═══ AND A PID THAT PREDATES THIS BOOT IS NOT THIS RUN'S PID, which is
1401
+ * `runProcessIsAlive`'s rule and not this function's. ═══ What it costs here
1402
+ * if it is ever weakened: after a restart a stranded run's pid regularly matches
1403
+ * something unrelated and live, so the card would be skipped on every sweep,
1404
+ * forever, which is the exact failure this function exists to make impossible.
1405
+ *
1406
+ * ═══ ENDING THE RUN AND HANDING ITS TURNS BACK IS ONE STATEMENT. ═══ It used to
1407
+ * be two round trips, turns first so that a failure between them left the run
1408
+ * still `running` for the next sweep to redo. That ordering was safe against a
1409
+ * failure and not against an ANSWER: in the gap the turns were takeable while
1410
+ * the run still read running, so a `panel3_answer` landing there was accepted
1411
+ * onto a card another daemon was already free to take. `panel3_give_up` closes
1412
+ * it, and the loser of that race is now decided by the database rather than by
1413
+ * which write happened to be in flight.
1414
+ *
1415
+ * IT CANNOT UNDO AN ANSWER ANY MORE. A run that has answered is `finished` in
1416
+ * the same statement that wrote the answer (`panel3_answer`), so it is never in
1417
+ * this sweep's result at all. Before that, a daemon killed between the two left
1418
+ * an answered card whose run still read running, and this function blanked
1419
+ * `addressed_at` on turns with their answer sitting beside them.
1420
+ *
1421
+ * ---------------------------------------------------------------------------
1422
+ * ═══ AND A DISPATCHED RUN IS NOT HANDED BACK. IT IS STARTED AGAIN. ═══
1423
+ *
1424
+ * The two kinds of run come back by different routes because their durable
1425
+ * state lives in different places, and this is the fork between them.
1426
+ *
1427
+ * A LEVEL 1 RUN'S WORK IS ITS TURNS. Handing them back is a complete recovery:
1428
+ * the next taker is sent the same thing, built from the same turns.
1429
+ *
1430
+ * A DISPATCHED RUN HOLDS NO TURNS. What it was sent to do and how far it got
1431
+ * are on its own row, so ending it throws both away — and the person's request
1432
+ * dies with a card reading `failed`. `panel3_resume` claims the row instead and
1433
+ * the same agent is started again with its brief, its report and its children.
1434
+ *
1435
+ * ═══ WHICH IS ALSO WHY THE GRACE WINDOW READS `resumed_at`. ═══ A claim nulls
1436
+ * the pid, so a freshly resumed run looks exactly like one whose daemon died
1437
+ * before recording a pid. The window has to be measured from the attempt that is
1438
+ * running now, not from the first one, or every resume would be reaped by the
1439
+ * next sweep a second after it started.
1440
+ */
1441
+ async function recoverStranded(client, tools, machineId, mine, hold) {
1442
+ const live = await returned(client
1443
+ .from('panel3_runs')
1444
+ .select('id, card_id, parent_run_id, state, pid, started_at, resumed_at')
1445
+ .eq('machine_id', machineId)
1446
+ /* ═══ AND A RUN THAT STOPPED TO ASK IS SWEPT TOO, WHICH IS WHAT MAKES THE
1447
+ RE-ARM'S OWN PREDICATE SAFE. ═══ `panel3_take_rearms` will only start a run
1448
+ again once its last process is accounted for — `ended_at is not null` —
1449
+ so that a settle still in flight cannot land on the attempt that
1450
+ replaced it. The stamp arrives when the process exits; if the daemon
1451
+ dies first, nothing else would ever write it and the branch would wait
1452
+ forever on an answer that could never be delivered. This is where it
1453
+ arrives instead. The two predicates are one rule and may not be changed
1454
+ apart. */
1455
+ .in('state', ['running', 'asked'])
1456
+ .is('ended_at', null), 'read', "this machine's live runs");
1457
+ const booted = bootedAt();
1458
+ for (const run of live) {
1459
+ if (mine.has(run.id))
1460
+ continue;
1461
+ // WHEN THE ATTEMPT NOW RUNNING BEGAN, which is the first one until a resume
1462
+ // says otherwise. See the header.
1463
+ const startedAt = new Date(run.resumed_at ?? run.started_at).getTime();
1464
+ const rebooted = startedAt < booted;
1465
+ // A run too young to have recorded a pid is another daemon's live work. The
1466
+ // reused-pid check that used to sit beside this is `runProcessIsAlive`'s.
1467
+ if (!rebooted && run.pid === null && Date.now() - startedAt < PID_GRACE_MS)
1468
+ continue;
1469
+ if (runProcessIsAlive(run, booted))
1470
+ continue;
1471
+ if (run.state === 'asked') {
1472
+ /* ═══ ITS END IS STAMPED AND ITS TURNS ARE LEFT ALONE. ═══ This run is not
1473
+ handed back and is not started again from here: its continuation is the
1474
+ answer to what it asked, and `panel3_take_rearms` is what brings it back.
1475
+ So it must NOT go through `panel3_give_up`, which would hand back the
1476
+ turns a level 1 run took — putting the same message in front of a second
1477
+ coordinator while the person is still being asked about the first.
1478
+ What is missing is only the end stamp its process never got to write. */
1479
+ const ended = await failRun(client, run.id, 'the process that asked this question was gone before it could finish');
1480
+ out(ended
1481
+ ? `recover run ${run.id} card ${run.card_id} asked, and its process is gone: ended, `
1482
+ + 'and it comes back when its question is answered'
1483
+ : `recover run ${run.id} had already ended, left alone`);
1484
+ continue;
1485
+ }
1486
+ if (run.parent_run_id !== null) {
1487
+ /* STARTED AGAIN AS ITSELF, AND A FAILURE IS SAID RATHER THAN THROWN. What
1488
+ it must not do is take the sweep down mid-loop and leave every other
1489
+ stranded run unlooked-at.
1490
+
1491
+ WHAT IS LEFT BEHIND DEPENDS ON HOW FAR IT GOT, and each outcome is
1492
+ honest. A failure BEFORE the claim — this machine has no working copy —
1493
+ leaves the run exactly as it was, still `running` with a dead process,
1494
+ so the next sweep tries again: right for a checkout that is missing for
1495
+ a minute, and one line per poll for one that is missing for good. A run
1496
+ that was CLAIMED and could not be STARTED is ended inside `resumeRun`,
1497
+ because a claimed run with no process must not sit out a grace window
1498
+ pretending to be alive. A failure between the two — the read of its
1499
+ children — leaves it claimed, which costs it one window before the next
1500
+ sweep claims it again. */
1501
+ try {
1502
+ const resumed = await resumeRun(client, tools, machineId, run.id, null);
1503
+ if (resumed)
1504
+ hold(run.id, resumed.settled);
1505
+ else
1506
+ out(`resume run ${run.id} already claimed or started again, left alone`);
1507
+ }
1508
+ catch (error) {
1509
+ said(`could not start run ${run.id} again: ${error instanceof Error ? error.message : String(error)}`);
1510
+ }
1511
+ continue;
1512
+ }
1513
+ const back = await giveUp(client, run.id, rebooted
1514
+ ? 'this machine has restarted since the run started, so its process is gone'
1515
+ : run.pid === null
1516
+ ? 'the daemon that started this was gone before it recorded a process id'
1517
+ : `the process running this (pid ${run.pid}) is gone and it never ended`);
1518
+ if (back === null) {
1519
+ /* IT ANSWERED WHILE IT WAS BEING GIVEN UP. The two functions race for one
1520
+ run and `panel3_give_up` lost: the card is answered and complete, and
1521
+ nothing here has touched it. Said rather than silent, because a sweep
1522
+ that decided a run was dead and was wrong is worth seeing. */
1523
+ out(`recover run ${run.id} card ${run.card_id} answered as it was being given up, left alone`);
1524
+ continue;
1525
+ }
1526
+ out(`recover run ${run.id} card ${run.card_id} process gone, `
1527
+ + `${back} turn${back === 1 ? '' : 's'} takeable again`);
1528
+ }
1529
+ }
1530
+ /**
1531
+ * ═══ A RUN A PERSON HANDED BACK, TAKEN UP BY WHOEVER IS LISTENING. ═══
1532
+ *
1533
+ * The same act as `recoverStranded`'s dispatched branch, by the other route.
1534
+ * The sweep can only ever reach THIS machine's runs, because liveness is a local
1535
+ * fact and no machine can honestly say whether another one's pid is still there.
1536
+ * So a machine that is never coming back strands its dispatched runs forever:
1537
+ * nothing sweeps them, and `panel3_take_turns` supersedes level 1 only.
1538
+ *
1539
+ * `panel3_hand_back` is the person's answer to that, and it decided everything.
1540
+ * It ran under the card's own row lock, refused any run whose machine had
1541
+ * heartbeated inside the listening window, and stamped `handed_back_at` on the
1542
+ * dispatched ones. THIS IS ONLY THE DISCOVERY, which is the one thing that was
1543
+ * missing: the daemon never asked whether another machine's run was available.
1544
+ *
1545
+ * ═══ AND SO THE READ HAS NO `machine_id` FILTER, WHICH IS THE WHOLE POINT. ═══
1546
+ * RLS scopes it to the person's own account and nothing narrower, because the
1547
+ * machine that stamped the offer is by definition not the machine that should
1548
+ * serve it. `panel3_resume` has never had a machine predicate either, so the
1549
+ * claim was always cross-machine; only the looking was not.
1550
+ *
1551
+ * ---------------------------------------------------------------------------
1552
+ * ═══ WHICH OF THE SWEEP'S SAFETY CHECKS APPLY HERE, AND WHICH CANNOT. ═══
1553
+ *
1554
+ * The sweep refuses three things, and they are not decoration: they are what
1555
+ * stops a machine reaping its own live work. Taken one at a time, because
1556
+ * carrying them across without thinking and dropping them without saying are
1557
+ * the same mistake.
1558
+ *
1559
+ * `mine`, A RUN THIS DAEMON IS HOLDING RIGHT NOW: KEPT. Presence is coarser
1560
+ * than a pid, so a daemon that is alive but has missed the listening window,
1561
+ * meaning a closed lid, a paused VM or a network stall, has its runs offered
1562
+ * while its agents are genuinely working. When it comes back it finds its OWN
1563
+ * work on offer, and without this line it would claim it: `panel3_resume`
1564
+ * writes `pid = null` and a second process starts on the same run id and the
1565
+ * same working copy, while the first one's `settle` is still going to answer
1566
+ * the card.
1567
+ *
1568
+ * ═══ AND IT NARROWS THAT CLASS RATHER THAN CLOSING IT. ═══ A THIRD machine,
1569
+ * listening and healthy, can claim that same live run, because the guard the
1570
+ * database applied is presence and presence is what was wrong. That limit is
1571
+ * the migration header's own and is not this function's to fix; `mine` closes
1572
+ * one half that IS knowable here, which is a run THIS PROCESS is holding, and
1573
+ * the pid check below closes the other, which is a run ANOTHER PROCESS ON THIS
1574
+ * MACHINE is holding.
1575
+ *
1576
+ * `PID_GRACE_MS`, TOO YOUNG TO HAVE RECORDED A PID: KEPT, in its `pid is null`
1577
+ * half only, and it closes the same hole `mine` does one step earlier. A
1578
+ * FRESHLY DISPATCHED run is not in `inFlight` yet: `startChild` writes the row
1579
+ * through `panel3_dispatch` and the poll only calls `hold` once `startChild`
1580
+ * has RESOLVED, so in between the row reads `running`, `parent_run_id` is not
1581
+ * null, `pid` is null, and `mine` does not contain it. The tools server runs
1582
+ * off HTTP rather than off the poll, so a daemon stalled past the listening
1583
+ * window still serves a `dispatch` and still writes that row: the person
1584
+ * presses, the child is stamped, the stall ends, and this read would claim a
1585
+ * run whose own process is in the act of starting. Two agents, one run id, one
1586
+ * working copy, which is what `mine` is here to prevent.
1587
+ *
1588
+ * ═══ AND IT COSTS THE REAL CASE NOTHING. ═══ A genuinely stranded offer is by
1589
+ * construction older than the fifteen seconds the banner needed before a
1590
+ * person could even see it, so the window never fires for one.
1591
+ *
1592
+ * THE SWEEP'S OTHER HALF OF THAT LINE, `rebooted`, STAYS OUT. It is read off
1593
+ * this machine's uptime, and this is the one read that is not about this
1594
+ * machine's runs.
1595
+ *
1596
+ * `runProcessIsAlive`, THE PID PLUS A BOOT-TIME REFUSAL: APPLIED TO THIS
1597
+ * MACHINE'S OWN OFFERED RUNS, AND TO NOBODY ELSE'S. The two halves are
1598
+ * genuinely different questions and were once answered as one.
1599
+ *
1600
+ * ON ANOTHER MACHINE'S RUN IT WOULD BE A DEFECT. That pid belongs to a process
1601
+ * on a box this one cannot see, so `kill(pid, 0)` here asks about an unrelated
1602
+ * local process, and pids are reused: a collision would make this machine skip
1603
+ * a run nobody is working on, forever, which is the exact strand the feature
1604
+ * exists to end. What stands in for it there is the heartbeat guard already
1605
+ * spent inside `panel3_hand_back`, with the honest limit that migration's
1606
+ * header states.
1607
+ *
1608
+ * ON A RUN WHOSE `machine_id` IS THIS MACHINE IT IS EXACTLY AS TRUSTWORTHY AS
1609
+ * IT IS IN `recoverStranded`, and TWO DAEMONS ON ONE MACHINE IS NOT
1610
+ * HYPOTHETICAL: 20260903140000 exists to support one per harness, and
1611
+ * `PID_GRACE_MS`'s own header says so. Without this check, D1 holds a live
1612
+ * dispatched run with its pid recorded, the machine sleeps, both presence rows
1613
+ * go stale, a person hands the card back, the machine wakes and D2 polls
1614
+ * first. The sweep leaves the run alone, correctly, because the process really
1615
+ * is alive. This read would then claim the very same row: `mine` is D2's and
1616
+ * does not hold D1's work, the grace window does not fire because the pid is
1617
+ * there, and `panel3_resume` matches every predicate, because `machine_id`
1618
+ * moves from this machine to this machine. Two agents, one run id, one working
1619
+ * copy, and D1's answer is accepted afterwards by a `panel3_answer` fenced by
1620
+ * neither machine nor pid.
1621
+ *
1622
+ * ---------------------------------------------------------------------------
1623
+ * ═══ AND A FAILURE IS SAID RATHER THAN THROWN, AS IT IS IN THE SWEEP. ═══ The
1624
+ * ordinary one is a machine with no folder for the project: `resumeRun` resolves
1625
+ * the checkout BEFORE the claim, so such a machine never takes a run it cannot
1626
+ * start. It says why on its own stderr and leaves the offer standing for a
1627
+ * machine that has the codebase. That is not silence and it is not a success:
1628
+ * the run is exactly as it was, and the card goes on saying nothing is listening,
1629
+ * which is true until somebody who can serve it is.
1630
+ *
1631
+ * ═══ AND THAT HOLDS ONLY WHERE `CTRL_SPC_V3_WORKING_COPY` IS UNSET, WHICH IS
1632
+ * THIS TASK'S ONE REAL EXPOSURE. ═══
1633
+ *
1634
+ * `checkoutFor` resolves in three steps and the third is a machine-wide folder
1635
+ * named by that variable, which applies EVEN WHEN THE CARD HAS A PROJECT. So a
1636
+ * listening machine with it set resolves a checkout for any project at all,
1637
+ * passes the ordering guarantee above, claims the run and starts an agent in a
1638
+ * folder holding somebody else's code, which is the failure
1639
+ * `panel3-checkout.contract.test.mjs`'s own header names. Before this function
1640
+ * existed it was unreachable across machines, because nothing ever looked at
1641
+ * another machine's run; this read is what makes it reachable, and the
1642
+ * cross-machine walk in plan-slice-2.md section 8 is exactly the setup that sets
1643
+ * the variable, so that walk can produce a false pass.
1644
+ *
1645
+ * ═══ RECORDED, NOT FIXED HERE. ═══ Reordering that resolution is `../codebases-3`'s
1646
+ * assignment, and `checkoutFor` is shared by every spawn path with its own
1647
+ * contract suite: changing it from this slice would widen a fifty-line read into
1648
+ * a change to how every agent in the product finds its code.
1649
+ *
1650
+ * ═══ AND ONE OFFER NOBODY CAN CHECK OUT IS NOW RETRIED BY THE WHOLE FLEET. ═══
1651
+ * Nothing clears `handed_back_at` except a claim, so every listening machine
1652
+ * reads that run every poll and prints one line for it, forever, where
1653
+ * `recoverStranded`'s header accepts the same cost for one machine. Two reads and
1654
+ * a line per machine per poll is cheap and is the honest state of the record: the
1655
+ * work really is offered and really has no taker. Silencing it would mean an
1656
+ * expiry on the offer, which is a rule nothing has asked for and which would
1657
+ * throw the work away on a fleet that was merely all asleep.
1658
+ */
1659
+ export async function takeHandedBack(client, tools, machineId, mine, hold) {
1660
+ /* ITS OWN NARROW ROW, NOT `LiveRun`. `machine_id` is read for one question
1661
+ and one only: whether this row is THIS machine's, which is what decides
1662
+ whether its pid means anything here. See the header. */
1663
+ const offered = await returned(client
1664
+ .from('panel3_runs')
1665
+ .select('id, card_id, machine_id, pid, started_at, resumed_at')
1666
+ .not('handed_back_at', 'is', null)
1667
+ /* WHAT THE OFFER MEANS, RESTATED IN THE READ. `panel3_hand_back` only ever
1668
+ stamps a dispatched run that is still going, and `panel3_resume` refuses
1669
+ anything else. Saying it here too costs one line and means a row that
1670
+ somehow carried a stale stamp is passed over rather than sent to a
1671
+ function that would refuse it a round trip later. */
1672
+ .eq('state', 'running')
1673
+ .is('ended_at', null)
1674
+ .not('parent_run_id', 'is', null), 'read', 'the runs a person has handed back');
1675
+ // ONCE, OUTSIDE THE LOOP. It is a property of this machine, not of a row.
1676
+ const booted = bootedAt();
1677
+ for (const run of offered) {
1678
+ if (mine.has(run.id)) {
1679
+ /* THIS DAEMON'S OWN LIVE WORK, OFFERED WHILE IT WAS BUSY BEING QUIET. See
1680
+ the header. Said rather than passed over in silence, because a machine
1681
+ whose work was handed back out from under it is the one thing here worth
1682
+ seeing in a terminal. */
1683
+ out(`offered run ${run.id} card ${run.card_id} is this daemon's own live work, left alone`);
1684
+ continue;
1685
+ }
1686
+ /* ═══ AND THE OTHER DAEMON ON THIS MACHINE IS HOLDING IT. ═══ `mine` covers
1687
+ this process and cannot cover its sibling: one machine runs one daemon per
1688
+ harness on purpose (20260903140000), and a machine that slept has both
1689
+ presence rows stale, so `panel3_hand_back` offers work that is still
1690
+ genuinely running. THE PID ANSWERS THAT, AND ONLY FOR THIS MACHINE'S OWN
1691
+ ROWS: for anybody else's it would be a question about an unrelated local
1692
+ process. `recoverStranded` asks it the same way on the same rows and
1693
+ correctly leaves them alone, so without this the sweep declines and the
1694
+ very next read claims. See the header. */
1695
+ if (run.machine_id === machineId && runProcessIsAlive(run, booted)) {
1696
+ out(`offered run ${run.id} card ${run.card_id} is still running on this machine, left alone`);
1697
+ continue;
1698
+ }
1699
+ /* ═══ AND A RUN TOO YOUNG TO HAVE RECORDED A PID IS SOMEBODY'S LIVE WORK
1700
+ TOO, INCLUDING THIS DAEMON'S OWN. ═══ `mine` cannot hold a dispatch
1701
+ between the row `panel3_dispatch` wrote and the `hold` that follows the
1702
+ spawn, and that gap is reachable from here: see the header. This is
1703
+ `recoverStranded`'s own line without its `rebooted` half, which reads this
1704
+ machine's uptime and has no meaning for another machine's run. */
1705
+ if (run.pid === null
1706
+ && Date.now() - new Date(run.resumed_at ?? run.started_at).getTime() < PID_GRACE_MS) {
1707
+ out(`offered run ${run.id} card ${run.card_id} is too young to have a process yet, left alone`);
1708
+ continue;
1709
+ }
1710
+ try {
1711
+ const resumed = await resumeRun(client, tools, machineId, run.id, null);
1712
+ if (resumed)
1713
+ hold(run.id, resumed.settled);
1714
+ else
1715
+ out(`resume run ${run.id} already claimed or started again, left alone`);
1716
+ }
1717
+ catch (error) {
1718
+ said(`could not take run ${run.id}, which was handed back: `
1719
+ + `${error instanceof Error ? error.message : String(error)}`);
1720
+ }
1721
+ }
1722
+ }
1723
+ // ---------------------------------------------------------------------------
1724
+ /**
1725
+ * The poll loop: it takes what is waiting and answers it, forever. `--once`
1726
+ * polls a single time and exits when the work it took is done, which is what
1727
+ * makes the loop testable — two of them against one card is how the lease is
1728
+ * proved.
1729
+ *
1730
+ * ═══ ONE LOOP, TWO CALLERS, AND THAT IS THE WHOLE OF recovery-1 SLICE 4. ═══
1731
+ * `startPanel` below is `cs start`'s entry and hands in the session the
1732
+ * installed CLI already holds. `cs3 run` is the acceptance harness's entry and
1733
+ * signs in from the environment against its own isolated account. Neither is a
1734
+ * copy of the other: a second poll loop would be two things leasing the same
1735
+ * turns with two ideas of what recovery may reap.
1736
+ *
1737
+ * @param injected the client to poll and serve tools through, when the caller
1738
+ * already has one. Its presence is also what says THIS PROCESS IS NOT v3's, so
1739
+ * the signals belong to somebody else — see the handler block below.
1740
+ */
1741
+ export async function run(args, injected) {
1742
+ let once = false;
1743
+ for (const arg of args) {
1744
+ if (arg === '--once')
1745
+ once = true;
1746
+ else
1747
+ throw new Error(`unknown option ${arg}. ${USAGE}`);
1748
+ }
1749
+ /* ═══ THE ONE v3 CLIENT THAT OUTLIVES ITS ACCESS TOKEN, SO IT IS THE ONE THAT
1750
+ REFRESHES. ═══ This client polls for as long as the daemon runs AND is the
1751
+ client every tool call is served through, so past the token's TTL both stop
1752
+ working at once. See `client.ts` for why refreshing is safe here:
1753
+ `persistSession` stays false, so the rotated token lives in this process and
1754
+ no file on the machine is written.
1755
+
1756
+ ═══ AND UNDER `cs start` THERE IS NO SIGN-IN AT ALL, WHICH IS THE POINT. ═══
1757
+ The installed daemon's own client is handed in: it refreshes, it retries at
1758
+ the network layer, and it is the ONLY client in that process, so the rotated
1759
+ token has exactly one writer. Signing in again there would be the second
1760
+ refresh loop `client.ts`'s header exists to prevent, this time inside one
1761
+ process rather than across two. */
1762
+ const client = injected ?? await signedInClient(true);
1763
+ const machineId = getMachineIdentity().id;
1764
+ /* THE NAME AND THE HARNESS, RESOLVED ONCE, AT STARTUP, RATHER THAN PER POLL.
1765
+ Both are properties of this machine for the life of this process: the
1766
+ hostname does not change under it, and neither does `CTRL_SPC_V3_AGENT`.
1767
+ Resolving `harness()` here means a machine misconfigured with a name this
1768
+ build cannot spawn (constraint 7: reported, never chosen) fails before it
1769
+ ever says it is listening, rather than on its first poll. */
1770
+ const machineName = hostname();
1771
+ const machineHarness = harness();
1772
+ /* THE RUNS THIS PROCESS IS HOLDING RIGHT NOW, so recovery cannot declare its
1773
+ own live work dead in the moment before a pid is recorded. It covers THIS
1774
+ daemon only, which is why `PID_GRACE_MS` exists for the other ones. Keyed by
1775
+ run id, at every level: a dispatched agent is this daemon's to hold too, and
1776
+ leaving it out would have recovery reap the children of its own runs. */
1777
+ const inFlight = new Map();
1778
+ /** One dispatched run's promise, held where the daemon can wait on it and
1779
+ * recovery can see it. The same bookkeeping the poll does for a level 1 run,
1780
+ * in the one other place a process is started. */
1781
+ const hold = (runId, work) => {
1782
+ inFlight.set(runId, work
1783
+ .catch((error) => {
1784
+ said(`run ${runId}: ${error instanceof Error ? error.message : String(error)}`);
1785
+ })
1786
+ .finally(() => { inFlight.delete(runId); }));
1787
+ };
1788
+ /* ═══ THE TOOLS BEFORE THE FIRST TAKE, OR NO TAKES AT ALL. ═══ A daemon that
1789
+ took a card and then found it could not serve tools would spawn an agent
1790
+ with no hands and write its confusion to the card as an answer. It is one
1791
+ loopback listener on an ephemeral port; failing here is a daemon that never
1792
+ started, which is the honest outcome.
1793
+
1794
+ `tools` is referenced inside the callback it is being given, which is safe
1795
+ for the plain reason that the callback can only run once a request has
1796
+ arrived at a server that by then exists. */
1797
+ const tools = await startToolsServer(client, async (parentRunId, brief, codebase) => {
1798
+ const child = await startChild(client, tools, machineId, parentRunId, brief, codebase);
1799
+ hold(child.runId, child.settled);
1800
+ return { runId: child.runId };
1801
+ });
1802
+ out(`daemon machine ${machineId}`);
1803
+ out(`tools ${tools.urlFor('<run-id>')}`);
1804
+ out(once ? 'mode one poll' : `mode polling every ${POLL_INTERVAL_MS / 1000}s, Ctrl-C to stop`);
1805
+ /* ═══ AND THIS MACHINE STOPS SAYING IT IS LISTENING WHEN IT GOES. ═══ ux.md
1806
+ forbids a card reading `working` while nothing is working, and the freshness
1807
+ window is a backstop for the machine that CANNOT say goodbye — a crash, a
1808
+ lost network, a kill -9. A Ctrl-C can, so it does, and the panel tells the
1809
+ truth within a poll rather than within a window.
1810
+
1811
+ `once` never gets here: it returns below, and the same call is made there.
1812
+
1813
+ ═══ AND NOT WHEN A CLIENT WAS HANDED IN, BECAUSE THEN THE PROCESS IS NOT
1814
+ THIS LOOP'S TO EXIT. ═══ Under `cs start` this shares a process with the v2
1815
+ presence heartbeat, which has a shutdown of its own that marks
1816
+ `cliv2_agents` offline and then calls `process.exit`. Two handlers means two
1817
+ exits, and whichever finishes its write first kills the other's mid-flight:
1818
+ a Ctrl-C would leave the machine reading listening in one place and gone in
1819
+ the other, which is Slice 1's card lying in the direction Slice 4 exists to
1820
+ close. `daemon.ts` owns the signal there and does both writes before it
1821
+ goes. */
1822
+ if (!once && !injected) {
1823
+ for (const signal of ['SIGINT', 'SIGTERM']) {
1824
+ process.once(signal, () => {
1825
+ void stopListening(client, machineId, machineHarness).finally(() => {
1826
+ /* The exit code a signal is supposed to produce, and the reason it is
1827
+ said explicitly: `process.once` REPLACES node's default handler, so
1828
+ without this a Ctrl-C would leave the daemon polling forever. */
1829
+ process.exit(signal === 'SIGINT' ? 130 : 143);
1830
+ });
1831
+ });
1832
+ }
1833
+ }
1834
+ for (;;) {
1835
+ /* ═══ ONE POLL FAILING IS NOT THE DAEMON FAILING. ═══ Every read and write
1836
+ here throws on a network or database error, by design (constraint 7), and
1837
+ until Slice 4 that threw straight out of `cs3 run` and exited the process.
1838
+ Inside `cs start` that is no longer an honest outcome twice over: it would
1839
+ take the v2 presence heartbeat down with it, and it would leave a stranded
1840
+ card printing `cs start` at a person who IS running `cs start`, which is
1841
+ the exact lie this slice exists to end. A dropped connection is a poll
1842
+ that did not happen; the next one is two seconds away.
1843
+
1844
+ `--once` still fails loudly, because the acceptance harness reads the exit
1845
+ code and a swallowed failure there would make a broken suite look green. */
1846
+ try {
1847
+ /* ═══ THE FIRST THING EVERY POLL, BECAUSE IT IS WHAT MAKES THE OTHER THINGS
1848
+ LEGIBLE. ═══ A card with an untaken turn reads the same whether a daemon
1849
+ is two seconds away or nobody has one running; this row is the only place
1850
+ the difference exists. It is written before the takes rather than after
1851
+ so that a machine which is up but busy still reads as up. */
1852
+ await sayListening(client, machineId, machineName, machineHarness);
1853
+ /* ═══ THE USER'S STOP IS HONOURED BEFORE ANYTHING ELSE ON THE POLL. ═══ It
1854
+ is the only thing here that a person is waiting on, and the two takes
1855
+ below can spend the rest of the poll starting agents. Nothing else needs
1856
+ to run first: `panel3_stop_card` has already ended the runs, so recovery
1857
+ cannot see them and neither take can start them. */
1858
+ await killStopped(client, machineId);
1859
+ await recoverStranded(client, tools, machineId, new Set(inFlight.keys()), hold);
1860
+ /* ═══ AND THEN WHAT SOMEBODY ELSE'S MACHINE WAS HOLDING, IF A PERSON HANDED
1861
+ IT BACK. ═══ AFTER the sweep, deliberately: this machine settles its own
1862
+ runs on local, pid-accurate evidence before it looks at anybody's, and a
1863
+ run of its own that was offered while it was quiet is dealt with there on
1864
+ the better evidence: resumed if its process is really gone, which spends
1865
+ the offer, and skipped if it is not, which is what `mine` then honours
1866
+ here. The set is rebuilt rather than reused because the sweep adds to it.
1867
+ BEFORE both takes, for the reason the sweep is: work that already exists,
1868
+ with a brief and a report behind it, comes before work that has not
1869
+ started. */
1870
+ await takeHandedBack(client, tools, machineId, new Set(inFlight.keys()), hold);
1871
+ /* THE MACHINE ID GOES IN because the take writes the run row, and a run has
1872
+ to say where it is running: the exclusion is cross-machine and recovery is
1873
+ per-machine, so a row with nobody's machine on it could be neither. */
1874
+ const taken = await returned(client.rpc('panel3_take_turns', { p_machine_id: machineId, p_agent: machineHarness }), 'take', 'turns');
1875
+ /* ═══ THE OTHER KIND OF TAKEABLE WORK. ═══ ux.md's re-arm: an answered
1876
+ question makes the branch that asked it takeable again, and a question
1877
+ still walking up makes the run it reached takeable so that level gets its
1878
+ turn. Same poll, same machine id, same holding: only the reason a run is
1879
+ started differs, and the record decides that rather than this file.
1880
+
1881
+ AFTER THE TURNS ARE TAKEN, deliberately, because the take WRITES the run
1882
+ row it leases to. A person who answered and then typed something else has
1883
+ both waiting, and in this order the message's level 1 is already on the
1884
+ record, so the re-arm leaves that card's coordinator alone and comes back
1885
+ to it on a later poll rather than putting two of them on one card. The
1886
+ other order would decide the same question from a snapshot taken before
1887
+ the run existed. */
1888
+ const rearmed = await returned(client.rpc('panel3_take_rearms', { p_machine_id: machineId, p_agent: machineHarness }), 'take', 'runs that can carry on');
1889
+ for (const row of rearmed) {
1890
+ /* NOT AWAITED PAST THE SPAWN, exactly as a taken card is not: the work is a
1891
+ real agent and holding the poll open for it would put every other card
1892
+ behind it. A throw here is caught rather than taking the loop down,
1893
+ because one run that could not be started must not stop the others. */
1894
+ try {
1895
+ hold(row.run_id, (await startRearmed(client, tools, machineId, row)).settled);
1896
+ }
1897
+ catch (error) {
1898
+ said(`could not start run ${row.run_id} again: ${error instanceof Error ? error.message : String(error)}`);
1899
+ }
1900
+ }
1901
+ for (const [cardId, turns] of byCard(taken)) {
1902
+ const runId = turns[0].run_id;
1903
+ out(`took card ${cardId} ${turns.length} turn${turns.length === 1 ? '' : 's'} run ${runId}`);
1904
+ /* NOT AWAITED HERE. A run is a real agent doing real work, and awaiting it
1905
+ in the poll would put every other card behind it — v2's own measured
1906
+ defect (env.ts, I21/I24: "a second request typed seconds after the first
1907
+ sat untouched until the first finished"). Nothing is needed to stop the
1908
+ same card being taken twice, because that is the database's job and it
1909
+ has already been done by the take. */
1910
+ // The catch inside `hold` is the last resort. A run left `running` by a
1911
+ // throw is not lost: its process has exited, so the next sweep reaps it
1912
+ // and the turns are answered again. It is there so the reason is printed
1913
+ // rather than becoming an unhandled rejection.
1914
+ hold(runId, answerCard(client, tools, machineId, cardId, turns));
1915
+ }
1916
+ if (once) {
1917
+ /* ═══ UNTIL NOTHING IS LEFT, NOT ONCE OVER WHAT WAS THERE. ═══ A level 1
1918
+ run dispatches WHILE it is being waited on, so the child appears in
1919
+ `inFlight` after the first wait started. Closing then would kill the
1920
+ tools server under an agent that had just been started, and `--once`
1921
+ would report a card answered while the work on it was still going. */
1922
+ while (inFlight.size > 0)
1923
+ await Promise.all([...inFlight.values()]);
1924
+ // The listener would otherwise keep this process alive after its one poll
1925
+ // was done, which is the whole point of `--once` being testable.
1926
+ await tools.close();
1927
+ // This machine is not listening any more, and it knows that here rather
1928
+ // than fifteen seconds from now. Same reason as the signal handlers above.
1929
+ await stopListening(client, machineId, machineHarness);
1930
+ return;
1931
+ }
1932
+ }
1933
+ catch (error) {
1934
+ if (once)
1935
+ throw error;
1936
+ said(`this poll did not finish: ${error instanceof Error ? error.message : String(error)}`);
1937
+ }
1938
+ await sleep(POLL_INTERVAL_MS);
1939
+ }
1940
+ }
1941
+ // ---------------------------------------------------------------------------
1942
+ /**
1943
+ * ═══ THE ONE THING `cs start` REACHES INTO `panel3/` FOR. ═══
1944
+ *
1945
+ * recovery-1 Slice 4, and Lane's ruling behind it (2026-08-21): *the user
1946
+ * experience must never require them to launch or authenticate multiple CLIs.
1947
+ * They launch the CLI with `cs start`.* Slices 1 to 3 print `cs start` on a
1948
+ * stranded card and, until this function existed, that command started a daemon
1949
+ * which polled no cards at all — the sentence named the right machine and the
1950
+ * wrong command, and no rewording could fix it.
1951
+ *
1952
+ * ═══ IT IS A NAMED EXCEPTION TO THE ISOLATION RULE, NOT A HOLE IN IT. ═══
1953
+ * `conventions.md` forbids anything outside `panel3/` importing anything inside
1954
+ * it, and `panel3-isolation.contract.test.mjs` enforces it. `cli-v2/src/daemon.ts`
1955
+ * is listed there with exactly ONE allowed specifier, `./panel3/run.js`, which is
1956
+ * why this is a single entry point handing back a single closure rather than
1957
+ * three exports the daemon would have to assemble. A second specifier fails the
1958
+ * suite, which is the point: the rule is enumerated, never weakened.
1959
+ *
1960
+ * ═══ THE CLIENT IS THE CALLER'S, AND THAT IS THE WHOLE OF "ONE SIGN-IN". ═══
1961
+ * `startPresence()` hands over the client it built from the session `cs login`
1962
+ * stored. This must never take one of its own: two clients in one process is two
1963
+ * refresh loops on one rotating refresh token, both writing `session.json`
1964
+ * through `getClient`'s `onAuthStateChange`, which is the hazard `client.ts`
1965
+ * describes moved indoors.
1966
+ *
1967
+ * ═══ IT DOES NOT BLOCK, AND IT DOES NOT TAKE THE DAEMON DOWN. ═══ The loop
1968
+ * never returns, so awaiting it here would hang `cs start` before it printed a
1969
+ * line. A throw that escapes the per-poll guard inside it is the panel stopping,
1970
+ * which is said on the one stream a person reads and leaves v2 presence
1971
+ * heartbeating rather than killing the process around it.
1972
+ */
1973
+ export function startPanel(client) {
1974
+ /* Both resolved HERE as well as inside the loop, deliberately: `stop` has to
1975
+ name the same row the loop's heartbeat writes, and it must be able to do
1976
+ that after the loop has thrown. They are the two facts about this machine
1977
+ that cannot change under a running process — the hardware-derived id and
1978
+ `CTRL_SPC_V3_AGENT` — so reading them twice cannot disagree. */
1979
+ const machineId = getMachineIdentity().id;
1980
+ const machineHarness = harness();
1981
+ void run([], client).catch((error) => {
1982
+ said(`the agent panel stopped polling: ${error instanceof Error ? error.message : String(error)}`);
1983
+ });
1984
+ /* `stopListening` is already best-effort and swallows its own failure: a
1985
+ process on its way out must not hang or fail, and the freshness window
1986
+ covers every way a machine can leave without getting here. */
1987
+ return { stop: () => stopListening(client, machineId, machineHarness) };
1988
+ }