@ctrl-spc/cs 0.6.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/agents.js +70 -5
- package/dist/autostart.js +15 -1
- package/dist/browser.js +386 -0
- package/dist/codebases.js +15 -0
- package/dist/codex-home.js +562 -0
- package/dist/companion.js +9 -12
- package/dist/config.js +23 -0
- package/dist/daemon.js +33 -3
- package/dist/env.js +42 -0
- package/dist/failure-reason.js +98 -0
- package/dist/index.js +1 -1
- package/dist/mcp.js +7627 -298
- package/dist/orchestrator.js +6011 -0
- package/dist/panel3/answer.js +166 -0
- package/dist/panel3/checkout.js +29 -0
- package/dist/panel3/cli.js +83 -0
- package/dist/panel3/client.js +181 -0
- package/dist/panel3/coordinator.js +18 -0
- package/dist/panel3/presence.js +162 -0
- package/dist/panel3/prompt.js +945 -0
- package/dist/panel3/run.js +2516 -0
- package/dist/panel3/say.js +262 -0
- package/dist/panel3/secrets.js +98 -0
- package/dist/panel3/session.js +128 -0
- package/dist/panel3/show.js +997 -0
- package/dist/panel3/spawn.js +565 -0
- package/dist/panel3/tools.js +1906 -0
- package/dist/presence.js +178 -6
- package/dist/win-shell.js +162 -0
- package/dist/work-context.js +1484 -0
- package/dist/workflows.js +68 -0
- package/package.json +3 -2
|
@@ -0,0 +1,2516 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* ═══ AGENT PANEL v3: the poll loop that answers a card. ═══
|
|
3
|
+
*
|
|
4
|
+
* THIS FILE BELONGS TO AGENT PANEL v3. Nothing outside `src/panel3/` may import
|
|
5
|
+
* it, WITH ONE NAMED EXCEPTION: `cli-v2/src/daemon.ts` imports `startPanel` from
|
|
6
|
+
* here and nothing else, which is how `cs start` runs the panel. It is written
|
|
7
|
+
* down in `conventions.md` and checked by `panel3-isolation.contract.test.mjs`;
|
|
8
|
+
* `startPanel`'s own comment at the foot of this file carries the reasoning.
|
|
9
|
+
*
|
|
10
|
+
* ---------------------------------------------------------------------------
|
|
11
|
+
* WHAT THIS IS. `cs3 say` puts a card and a user turn on the record and returns,
|
|
12
|
+
* because ux.md requires the acknowledgement never to wait for a spawn. This is
|
|
13
|
+
* the other half: the loop that notices the turn, leases it, writes a run row,
|
|
14
|
+
* starts a real agent, and puts what that agent said back on the card.
|
|
15
|
+
*
|
|
16
|
+
* ---------------------------------------------------------------------------
|
|
17
|
+
* ═══ THE DAEMON DOES NOT DECIDE WHAT IS TAKEABLE. THE DATABASE DOES. ═══
|
|
18
|
+
*
|
|
19
|
+
* Everything about exclusion lives in `panel3_take_turns()`: the lease, the
|
|
20
|
+
* "no live level 1 on this card" rule, and the skip-locked that keeps two
|
|
21
|
+
* daemons polling the same millisecond from both getting one card. There is no
|
|
22
|
+
* read-then-decide-then-write anywhere in this file, because that shape has a
|
|
23
|
+
* window between the read and the write and the window is exactly what a second
|
|
24
|
+
* daemon finds. This file honours what it was handed and nothing else.
|
|
25
|
+
*
|
|
26
|
+
* ---------------------------------------------------------------------------
|
|
27
|
+
* ═══ THE RUN ROW IS WRITTEN BEFORE THE SPAWN, OR THERE IS NO SPAWN. ═══
|
|
28
|
+
*
|
|
29
|
+
* plan.md's global constraint 8. A process with no row is a process nothing can
|
|
30
|
+
* find after a restart: it cannot be shown, cannot be recovered and cannot be
|
|
31
|
+
* ended, so the card would say working forever with an agent nobody can name.
|
|
32
|
+
*
|
|
33
|
+
* IT IS THE TAKE THAT WRITES IT, in the same statement that leases the turns,
|
|
34
|
+
* which is why this file has no ordering to be careful about: it cannot reach a
|
|
35
|
+
* spawn without a row, because it cannot reach a spawn without a take. The
|
|
36
|
+
* earlier version wrote the row here, two round trips after the lease, and both
|
|
37
|
+
* ends of that gap were holes — a daemon killed inside it stranded the card
|
|
38
|
+
* forever, and a turn arriving inside it could put a second agent on the same
|
|
39
|
+
* card. See 20260821140000.
|
|
40
|
+
*
|
|
41
|
+
* ---------------------------------------------------------------------------
|
|
42
|
+
* ═══ THE DAEMON WRITES THE AGENT TURN FROM WHAT THE PROCESS SAID. ═══
|
|
43
|
+
*
|
|
44
|
+
* The harness is one-shot: a prompt goes in, text comes out, it exits. That
|
|
45
|
+
* final text IS the answer, so this writes it as the card's `role='agent'` turn.
|
|
46
|
+
*
|
|
47
|
+
* ═══ AND IT IS THE SAME PATH AT EVERY LEVEL, WHICH IS ux.md's RULE, NOT A
|
|
48
|
+
* CONVENIENCE. ═══ "Whoever did the work writes the answer. The level 2
|
|
49
|
+
* agent writes it, straight into the card. It does not hand a summary up to
|
|
50
|
+
* level 1 to be re-written." A dispatched agent's text goes onto the card
|
|
51
|
+
* through the same `panel3_answer` its dispatcher's did, so there is nowhere in
|
|
52
|
+
* this file for one agent's words to pass through another's. The coordinator has
|
|
53
|
+
* usually ended long before.
|
|
54
|
+
*
|
|
55
|
+
* ═══ AND A WORKER'S WORDS DO NOT REACH THE CARD AT ALL, WHICH IS THE SAME RULE
|
|
56
|
+
* ONE LEVEL DOWN. ═══ "Each worker writes what it found into the record, and
|
|
57
|
+
* the level 2 agent reads them back." Three workers writing their own turns
|
|
58
|
+
* would be three answers to one request, before the one that covers them all
|
|
59
|
+
* arrives. `panel3_answer` puts a level 3 run's last words in its report instead,
|
|
60
|
+
* from the level on the row — decided there rather than here, because two owners
|
|
61
|
+
* for "does this reach the person" is one owner too many.
|
|
62
|
+
*
|
|
63
|
+
* ---------------------------------------------------------------------------
|
|
64
|
+
* ═══ A QUESTION MOVING, AND EVERYBODY YOU SENT FINISHING, ARE BOTH TAKEABLE
|
|
65
|
+
* WORK, AND THEY COME THROUGH THE SAME POLL. ═══
|
|
66
|
+
*
|
|
67
|
+
* ux.md: "Nothing new is built for this. The daemon already leases takeable work
|
|
68
|
+
* and spawns. What changes is the definition of takeable." So there is a second
|
|
69
|
+
* take beside `panel3_take_turns` and nothing else in this file changes shape:
|
|
70
|
+
* `panel3_take_rearms` hands back runs to start again, for either of the two
|
|
71
|
+
* reasons ux.md's table gives, and they are started, held and settled by the
|
|
72
|
+
* functions that already do those things.
|
|
73
|
+
*
|
|
74
|
+
* THE SECOND REASON IS THE ONE NOTHING ELSE COULD FIRE. An agent that split its
|
|
75
|
+
* work dispatched and ended, because it is forbidden to wait; by the time its
|
|
76
|
+
* workers finish there is no process anywhere holding what they found. Starting
|
|
77
|
+
* it again is what makes "it reads them back and writes the answer" a mechanism
|
|
78
|
+
* rather than a sentence.
|
|
79
|
+
*
|
|
80
|
+
* IT IS THE SAME MECHANISM AS RECOVERY, POINTED AT A DIFFERENT FACT. A run whose
|
|
81
|
+
* process died is started again as itself because the machine failed; a run that
|
|
82
|
+
* asked, or one whose workers are done, is started again as itself because the
|
|
83
|
+
* thing it was waiting on exists now. Same row, same id, same brief, its report
|
|
84
|
+
* carried — and the only difference is one paragraph at the top of the prompt.
|
|
85
|
+
*
|
|
86
|
+
* ---------------------------------------------------------------------------
|
|
87
|
+
* ═══ A DISPATCH IS A SPAWN THE DAEMON DOES ON BEHALF OF AN AGENT. ═══
|
|
88
|
+
*
|
|
89
|
+
* The tools server runs inside this process and `dispatch` is served from it,
|
|
90
|
+
* but the spawning is here: writing the row before the process, recording the
|
|
91
|
+
* pid, waiting for the answer, giving up when the process is gone, and being
|
|
92
|
+
* held open long enough for `--once` to be honest are all things this file
|
|
93
|
+
* already does once, and a second copy of them behind a tool would be a second
|
|
94
|
+
* owner of a child's whole life.
|
|
95
|
+
*
|
|
96
|
+
* WHAT IT DOES NOT DO IS WAIT. The tool call returns as soon as the process
|
|
97
|
+
* exists, and the child's answer settles here, long after the agent that asked
|
|
98
|
+
* for it has ended. That is ux.md's "level 1 never blocks waiting for it", and it
|
|
99
|
+
* is why a dispatched run's promise goes into `inFlight` rather than being
|
|
100
|
+
* awaited by its caller.
|
|
101
|
+
*
|
|
102
|
+
* ---------------------------------------------------------------------------
|
|
103
|
+
* ═══ A CARD MUST NOT SIT LIVE FOREVER. ═══
|
|
104
|
+
*
|
|
105
|
+
* If this daemon is killed mid-run its process dies with it, and the run row is
|
|
106
|
+
* left saying `running` with an end that never comes. `recoverStranded` is what
|
|
107
|
+
* closes that: on every poll it looks at the runs THIS machine owns and asks the
|
|
108
|
+
* operating system whether their processes are still there. A level 1 run's
|
|
109
|
+
* turns are handed back, so they are takeable again; a dispatched run holds no
|
|
110
|
+
* turns and is STARTED AGAIN AS ITSELF, with the brief it was sent and the
|
|
111
|
+
* report it wrote. Either way the user's message is answered late rather than
|
|
112
|
+
* lost.
|
|
113
|
+
*
|
|
114
|
+
* It is careful in three directions, and each is a way the sweep can lie: it
|
|
115
|
+
* never reaps a run this process holds, never reaps a run too young to have
|
|
116
|
+
* recorded a pid (which is another daemon's live work), and never believes a pid
|
|
117
|
+
* that predates this machine's boot (which is somebody else's process wearing a
|
|
118
|
+
* reused number).
|
|
119
|
+
*
|
|
120
|
+
* ═══ AND IT CAN ONLY EVER REACH THIS MACHINE'S RUNS, WHICH IS WHY A PERSON HAS
|
|
121
|
+
* THE OTHER MOVE. ═══
|
|
122
|
+
*
|
|
123
|
+
* A machine that is never coming back sweeps nothing, so its dispatched runs sit
|
|
124
|
+
* `running` forever. `panel3_hand_back` is the person's answer, pressed from the
|
|
125
|
+
* panel on a card nothing is listening for, and `takeHandedBack` is this
|
|
126
|
+
* daemon's half of it: the one read that looks past this machine's own id, so
|
|
127
|
+
* work another machine was holding is carried on from its brief and its report
|
|
128
|
+
* instead of being started over. The sweep is not loosened to do it, and which
|
|
129
|
+
* of the three cares above apply to that read is argued where it is written.
|
|
130
|
+
*/
|
|
131
|
+
// The runtime import comes FIRST, deliberately: `tsc` elides a type-only import
|
|
132
|
+
// and takes the leading comment with it, so a file whose first statement is
|
|
133
|
+
// `import type` loses its v3 header in the published `dist/`.
|
|
134
|
+
import { out, returned, signedInClient } from './client.js';
|
|
135
|
+
import { answerPrompt, escalationPrompt, levelOnePrompt, ownerActivationPrompt, readBackPrompt, presentedArtifactAnswerContext, resumePrompt, retryPrompt, } from './prompt.js';
|
|
136
|
+
import { ASK_CONTENT_COLUMNS, attachmentLine, loadAttachments, loadOutputNames, outputOf, withAskContent, } from './show.js';
|
|
137
|
+
import { forgetSecrets, redactSecrets } from './secrets.js';
|
|
138
|
+
import { sayListening, stopListening } from './presence.js';
|
|
139
|
+
import { checkoutForCodebase, hasCheckoutForCodebase } from './checkout.js';
|
|
140
|
+
import { harness, startAgent } from './spawn.js';
|
|
141
|
+
import { establishOwnerSession, listOwnerSessionIds, OWNER_SESSION_GRACE_MS, readOwnerSession, removeOwnerSession, validSessionUuid, writeOwnerSession, } from './session.js';
|
|
142
|
+
import { startToolsServer } from './tools.js';
|
|
143
|
+
import { listPanel3CodexOwnerHomeIds, removePanel3CodexOwnerHome, } from '../codex-home.js';
|
|
144
|
+
import { getMachineIdentity, scratchDir } from '../config.js';
|
|
145
|
+
import { listCodebases } from '../codebases.js';
|
|
146
|
+
import { killTree, processIsAlive } from '../win-shell.js';
|
|
147
|
+
import { hostname, uptime } from 'node:os';
|
|
148
|
+
const USAGE = 'usage: cs3 run [--once]';
|
|
149
|
+
/** How long between takes. Short, because it is the whole delay between a user
|
|
150
|
+
* sending and a card showing an agent on it, and the take is one small indexed
|
|
151
|
+
* read against the user's own rows. */
|
|
152
|
+
const POLL_INTERVAL_MS = 2_000;
|
|
153
|
+
/**
|
|
154
|
+
* How long a run with no pid recorded is given before it counts as gone.
|
|
155
|
+
*
|
|
156
|
+
* ═══ IT EXISTS BECAUSE `mine` IS ONLY THIS PROCESS'S MEMORY. ═══ The take
|
|
157
|
+
* writes the run row and the pid lands a moment later, so between the two the
|
|
158
|
+
* row is indistinguishable from one whose daemon died — to this daemon. To a
|
|
159
|
+
* SECOND daemon on the same machine it is worse than indistinguishable: the run
|
|
160
|
+
* is not in its `mine` set at all, so without this it would hand back the turns
|
|
161
|
+
* of a run that started seconds ago and mark it failed while the agent was
|
|
162
|
+
* genuinely working. That configuration is not hypothetical; it is what the
|
|
163
|
+
* lease is proved with.
|
|
164
|
+
*
|
|
165
|
+
* Thirty seconds is far longer than the one round trip it has to cover and far
|
|
166
|
+
* shorter than any run, so it costs a genuinely dead run one extra poll cycle at
|
|
167
|
+
* most.
|
|
168
|
+
*
|
|
169
|
+
* ═══ AND `panel3_resume` HOLDS THE SAME NUMBER, SO IT IS EXPORTED AND CHECKED.
|
|
170
|
+
* ═══ The claim refuses to hand the same run out twice inside this window, which
|
|
171
|
+
* is this rule from the other side: a run that has just been claimed has no
|
|
172
|
+
* process yet and everything else has to leave it alone long enough for one to
|
|
173
|
+
* exist. Two comments saying they agree is not a check, and the two are in
|
|
174
|
+
* different languages in different files — so the contract test reads the
|
|
175
|
+
* interval out of the function definition and compares it to this. Disagreement
|
|
176
|
+
* either reaps a resume a second after it started, or makes a dead run wait out
|
|
177
|
+
* two windows.
|
|
178
|
+
*/
|
|
179
|
+
export const PID_GRACE_MS = 30_000;
|
|
180
|
+
/**
|
|
181
|
+
* How many processes may be started for one run, counting the first.
|
|
182
|
+
*
|
|
183
|
+
* ═══ IT IS NOT A NEW NUMBER, AND THAT IS THE REASON FOR IT. ═══
|
|
184
|
+
* `orchestrator.ts:1107` `MAX_ATTEMPTS = 3` and `orchestrator.ts:4421`
|
|
185
|
+
* `MAX_REPLY_ATTEMPTS = 3` are the same bound, with the reason already written
|
|
186
|
+
* down at `orchestrator.ts:1102-1105`: three attempts is enough to ride out a
|
|
187
|
+
* genuinely transient failure and short enough that a deterministic one stops
|
|
188
|
+
* costing the user minutes of spawn time, and it is not configurable, because a
|
|
189
|
+
* knob here would mostly serve to hide a bug behind a bigger number. A fourth
|
|
190
|
+
* number invented here would be a dialect.
|
|
191
|
+
*
|
|
192
|
+
* ═══ AND THE BOUND ITSELF IS NOT THIS CONSTANT. ═══ `panel3_resume` counts the
|
|
193
|
+
* attempt and refuses past three IN THE SAME STATEMENT, because a bound whose
|
|
194
|
+
* counter can silently miss an attempt is not a bound
|
|
195
|
+
* (`20260806120000:1096-1099`). This is the number the daemon says out loud, and
|
|
196
|
+
* a contract test reads the literal back out of the function definition and
|
|
197
|
+
* compares the two, exactly as `PID_GRACE_MS` is compared against that
|
|
198
|
+
* function's 30 seconds.
|
|
199
|
+
*/
|
|
200
|
+
export const MAX_ATTEMPTS = 3;
|
|
201
|
+
/** The level the daemon puts on a card of its own accord. Anything deeper exists
|
|
202
|
+
* because an agent asked for it, and its level comes off the row
|
|
203
|
+
* `panel3_dispatch` wrote rather than from anything here. */
|
|
204
|
+
const LEVEL = 1;
|
|
205
|
+
const artifactAnswer = (id, revision, selected) => {
|
|
206
|
+
if (!id || revision === null || revision === undefined || selected?.length !== 1)
|
|
207
|
+
return null;
|
|
208
|
+
const answer = selected[0]?.trim().toLowerCase();
|
|
209
|
+
return answer === 'approve' || answer === 'request changes'
|
|
210
|
+
? { id, revision, answer }
|
|
211
|
+
: null;
|
|
212
|
+
};
|
|
213
|
+
export function deliveredArtifactAnswer(events, delivered) {
|
|
214
|
+
if (!delivered.mine || delivered.ask_id === null)
|
|
215
|
+
return null;
|
|
216
|
+
const event = events.find((candidate) => candidate.kind === 'question' && candidate.id === delivered.ask_id);
|
|
217
|
+
const answer = artifactAnswer(event?.relatedArtifactId, event?.relatedArtifactRevision, event?.artifactAnswer ? [event.artifactAnswer] : null);
|
|
218
|
+
return answer === null ? null : { questionId: delivered.ask_id, ...answer };
|
|
219
|
+
}
|
|
220
|
+
async function rearmedArtifactAnswer(client, askId) {
|
|
221
|
+
const asks = await withAskContent(client, await returned(client.from('panel3_asks').select(`id, ${ASK_CONTENT_COLUMNS}`).eq('id', askId), 'read', `question ${askId}`));
|
|
222
|
+
const ask = asks[0];
|
|
223
|
+
return ask === undefined
|
|
224
|
+
? null
|
|
225
|
+
: artifactAnswer(ask.related_artifact_id, ask.related_artifact_revision, ask.selected_options);
|
|
226
|
+
}
|
|
227
|
+
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
|
228
|
+
/**
|
|
229
|
+
* ═══ NOTHING IN THIS FILE STAMPS A COLUMN FROM THIS MACHINE'S CLOCK ANY MORE,
|
|
230
|
+
* AND THAT IS A RULE RATHER THAN A TIDY-UP. ═══
|
|
231
|
+
*
|
|
232
|
+
* `started_at`, `ended_at` and `resumed_at` are all written by the database's own
|
|
233
|
+
* `now()`, in the statements that end or claim a run, and `panel3_owed_read_back`
|
|
234
|
+
* COMPARES a parent's end with its children's. A second clock writing any of them
|
|
235
|
+
* makes that comparison meaningless in a way nothing reports: skew one way hides
|
|
236
|
+
* the child that genuinely finished last and loses the read-back, with every row
|
|
237
|
+
* individually plausible. `failRun` was the last place that did it and now calls
|
|
238
|
+
* `panel3_end_run`.
|
|
239
|
+
*
|
|
240
|
+
* ═══ THE TWO CLOCKS STILL MEET IN ONE PLACE, AND IT IS DELIBERATE. ═══
|
|
241
|
+
* `recoverStranded` reads `started_at`, which the database stamped, and measures
|
|
242
|
+
* it against `Date.now()` for the pid grace window and against this machine's
|
|
243
|
+
* boot time for the reused-pid check. It has no choice: liveness is a local fact
|
|
244
|
+
* and the boot time is only knowable here. Both directions of skew are real:
|
|
245
|
+
*
|
|
246
|
+
* THE DATABASE AHEAD OF LOCAL BY MORE THAN THE GRACE WINDOW makes every
|
|
247
|
+
* pid-null run look younger than it is, forever, so nothing is ever reaped —
|
|
248
|
+
* which is the permanent strand the recovery exists to prevent.
|
|
249
|
+
*
|
|
250
|
+
* THE DATABASE BEHIND LOCAL BY MORE THAN THIS MACHINE'S UPTIME makes a run
|
|
251
|
+
* that started moments ago look older than the boot, so a live agent is
|
|
252
|
+
* declared gone and its turns are handed to a second one.
|
|
253
|
+
*
|
|
254
|
+
* Considered rather than missed, and left as it is: both machines run NTP — here
|
|
255
|
+
* they are the same machine, since the local stack is a container on it — so the
|
|
256
|
+
* skew is milliseconds against a thirty-second window and an uptime measured in
|
|
257
|
+
* days. A product where the two could drift that far would have to stamp both
|
|
258
|
+
* clocks and compare like with like.
|
|
259
|
+
*/
|
|
260
|
+
/* ═══ THE NAME A PERSON TYPED, WHICH IS THE ONLY NAME THEY KNOW. ═══ `cs start`
|
|
261
|
+
is the command in the product's copy everywhere else, and this is the one
|
|
262
|
+
prefix on the one stream a person actually reads. */
|
|
263
|
+
const said = (what) => console.error(`cs start: ${what}`);
|
|
264
|
+
/**
|
|
265
|
+
* The taken rows, grouped by card, each card's turns still in send order.
|
|
266
|
+
*
|
|
267
|
+
* ═══ THE GROUPING IS WHY THE TAKE RETURNS ROWS AND NOT ONE BODY. ═══ ux.md:
|
|
268
|
+
* "A turn takes every unaddressed input on that card, oldest first, and hands
|
|
269
|
+
* them all over. Not the newest. Not one chosen as 'the request'." There is
|
|
270
|
+
* nowhere in this function for the product to choose between them, which is the
|
|
271
|
+
* property that matters: v2 chose, and the message it did not choose was filed
|
|
272
|
+
* as background and never acted on.
|
|
273
|
+
*/
|
|
274
|
+
function byCard(taken) {
|
|
275
|
+
const cards = new Map();
|
|
276
|
+
for (const row of taken) {
|
|
277
|
+
const existing = cards.get(row.card_id);
|
|
278
|
+
if (existing)
|
|
279
|
+
existing.push(row);
|
|
280
|
+
else
|
|
281
|
+
cards.set(row.card_id, [row]);
|
|
282
|
+
}
|
|
283
|
+
return cards;
|
|
284
|
+
}
|
|
285
|
+
/**
|
|
286
|
+
* ═══ WHAT THIS CARD HAS ALREADY PRODUCED, NAMED AS IT READS NOW. ═══
|
|
287
|
+
*
|
|
288
|
+
* A card's receipts are level 1's report: every turn is answered by a fresh
|
|
289
|
+
* agent that remembers nothing, and what it most needs from the last one is what
|
|
290
|
+
* that one made. So they are read here and handed over in the prompt, exactly as
|
|
291
|
+
* a dispatched run's report is, rather than being left behind a tool call — see
|
|
292
|
+
* `levelOnePrompt` for the measured failure this closes.
|
|
293
|
+
*
|
|
294
|
+
* THE NAME IS RESOLVED, NOT READ OFF THE RECEIPT, through the one function
|
|
295
|
+
* `cs3 show` uses for it. `label` is written when the thing is made and the
|
|
296
|
+
* coordinator may rename the thing on the next turn; printing the label then
|
|
297
|
+
* hands the agent the name of a different, real object. One owner for that,
|
|
298
|
+
* because two would eventually disagree about which name a card is showing.
|
|
299
|
+
*/
|
|
300
|
+
async function receiptsFor(client, cardId) {
|
|
301
|
+
const outputs = await returned(client
|
|
302
|
+
.from('panel3_outputs')
|
|
303
|
+
.select('id, card_id, run_id, kind, ref_id, label, created_at')
|
|
304
|
+
.eq('card_id', cardId)
|
|
305
|
+
.order('created_at'), 'read', `what card ${cardId} has produced`);
|
|
306
|
+
const names = await loadOutputNames(client, outputs);
|
|
307
|
+
return outputs.map((o) => `${o.kind.padEnd(10)} ${outputOf(o, names)} id ${o.ref_id}`);
|
|
308
|
+
}
|
|
309
|
+
/**
|
|
310
|
+
* ═══ WHAT THE PERSON ATTACHED TO THIS CARD, FORMATTED THE ONE WAY THE PRODUCT
|
|
311
|
+
* NAMES AN ATTACHMENT. ═══
|
|
312
|
+
*
|
|
313
|
+
* The product applies the attachment (plan-work-items.md constraint 5): level 1
|
|
314
|
+
* is handed it here rather than asked to go and find it, exactly as `produced`
|
|
315
|
+
* is handed over rather than left behind `list_work_items`. Most cards have
|
|
316
|
+
* none, since Task 2 offers attaching only when a card is started, so this is
|
|
317
|
+
* usually an empty list, and `levelOnePrompt` says nothing about attachments at
|
|
318
|
+
* all when it is.
|
|
319
|
+
*/
|
|
320
|
+
// Exported so a test can prove the wiring itself: that this reads the CALLER's
|
|
321
|
+
// own card through `loadAttachments` rather than a copy, the same reason
|
|
322
|
+
// `dispatch`'s equivalent read is proven in panel3-tools.contract.test.mjs
|
|
323
|
+
// instead of only the formatter it feeds.
|
|
324
|
+
export async function attachmentsFor(client, cardId) {
|
|
325
|
+
return (await loadAttachments(client, cardId)).map(attachmentLine);
|
|
326
|
+
}
|
|
327
|
+
/**
|
|
328
|
+
* What level 1 is sent, and also what is written to `panel3_runs.brief`: the
|
|
329
|
+
* column's own question is "what was this agent asked", answered with the exact
|
|
330
|
+
* text it was asked with rather than with a description of it.
|
|
331
|
+
*
|
|
332
|
+
* THE WORDS THEMSELVES ARE IN `prompt.ts`, which is where the rules that govern
|
|
333
|
+
* them are written down. This is only the join: the take hands over rows, the
|
|
334
|
+
* record says what the card has made, and a prompt is about a title, some
|
|
335
|
+
* messages and that list.
|
|
336
|
+
*/
|
|
337
|
+
// Exported for the same reason as `attachmentsFor` above: the wiring test
|
|
338
|
+
// calls this with what that function actually returned, so what a coordinator
|
|
339
|
+
// is told is proven to come from the card's own attachment rows rather than
|
|
340
|
+
// only from `levelOnePrompt`'s own formatting tests in panel3-prompt.
|
|
341
|
+
export const briefFor = (title, turns, produced, attachments, codebases = null) => levelOnePrompt(title, turns.map((t) => t.body), produced, attachments, codebases?.map((codebase) => ({
|
|
342
|
+
id: codebase.id,
|
|
343
|
+
name: codebase.name,
|
|
344
|
+
identity: codebase.gitRemoteUrl,
|
|
345
|
+
located: hasCheckoutForCodebase(codebase),
|
|
346
|
+
})) ?? null);
|
|
347
|
+
// ---------------------------------------------------------------------------
|
|
348
|
+
// WRITES. Every one goes through the shared guard in `client.ts`, which returns
|
|
349
|
+
// the rows or throws. `.select()` on each is what makes that possible: a
|
|
350
|
+
// Supabase write with no select returns neither data nor an error, and the guard
|
|
351
|
+
// refuses that shape rather than reading it as success.
|
|
352
|
+
/**
|
|
353
|
+
* ═══ WHAT IS RUNNING A RUN, AND — AT LEVEL 1 — WHAT IT WAS SENT. ═══
|
|
354
|
+
*
|
|
355
|
+
* The run row already exists in both paths: the take wrote it for level 1 and
|
|
356
|
+
* `panel3_dispatch` wrote it for a dispatched agent, which is what makes
|
|
357
|
+
* constraint 8 unconditional rather than an ordering this file has to be careful
|
|
358
|
+
* about. So this is never the row's creation, only the facts that could not be
|
|
359
|
+
* known before the process existed.
|
|
360
|
+
*
|
|
361
|
+
* THE BRIEF IS PASSED ONLY WHERE IT IS STILL A PLACEHOLDER. The take cannot know
|
|
362
|
+
* what an agent will be sent, so level 1's row carries a placeholder until here;
|
|
363
|
+
* a dispatch knows the brief before the row exists and writes it there. Passing
|
|
364
|
+
* it again would be rewriting a brief that ux.md fixes at dispatch.
|
|
365
|
+
*
|
|
366
|
+
* A failure here is NOT fatal. The agent is already running and killing it over
|
|
367
|
+
* a bookkeeping write would cost the user the work. What is lost is precision:
|
|
368
|
+
* the run has no pid, so once this daemon is gone recovery treats it as dead —
|
|
369
|
+
* which errs towards answering again rather than stranding.
|
|
370
|
+
*/
|
|
371
|
+
async function recordProcess(client, runId, pid, brief, processToken) {
|
|
372
|
+
let query = client
|
|
373
|
+
.from('panel3_runs')
|
|
374
|
+
.update({ pid, ...(brief === undefined ? {} : { brief }) })
|
|
375
|
+
.eq('id', runId);
|
|
376
|
+
if (processToken !== undefined)
|
|
377
|
+
query = query.eq('process_token', processToken);
|
|
378
|
+
const written = await returned(query.select('id'), 'record what is running', `run ${runId}`);
|
|
379
|
+
if (written.length === 0)
|
|
380
|
+
throw new Error(`could not record what is running for run ${runId}: its activation has ended`);
|
|
381
|
+
}
|
|
382
|
+
/**
|
|
383
|
+
* ═══ A RUN GIVEN UP ON: ENDED, ITS TURNS HANDED BACK, ITS CARD IN HAND. ═══
|
|
384
|
+
*
|
|
385
|
+
* One statement, in `panel3_give_up`, for the reason 20260821170000 gives at
|
|
386
|
+
* length: as two round trips there was an instant in which the turns were
|
|
387
|
+
* takeable and the run still read `running`, and an answer landing in that
|
|
388
|
+
* instant was accepted onto a card another daemon was already free to take.
|
|
389
|
+
*
|
|
390
|
+
* Returns the number of turns handed back, or NULL when the run was no longer
|
|
391
|
+
* running — which is this function losing cleanly to a `panel3_answer` that got
|
|
392
|
+
* there first. It is a fact about the record rather than a failure, so it does
|
|
393
|
+
* not go through `returned()`, exactly as the answer's own null does not.
|
|
394
|
+
*/
|
|
395
|
+
async function giveUp(client, runId, reason, processToken) {
|
|
396
|
+
const { data, error } = await client
|
|
397
|
+
.rpc('panel3_give_up', {
|
|
398
|
+
p_run_id: runId,
|
|
399
|
+
p_reason: reason,
|
|
400
|
+
...(processToken === undefined ? {} : { p_process_token: processToken }),
|
|
401
|
+
});
|
|
402
|
+
if (error)
|
|
403
|
+
throw new Error(`could not give up run ${runId}: ${error.message}`);
|
|
404
|
+
/* THE RUN IS OVER, so whatever it read is dropped. Every ending does this —
|
|
405
|
+
here, `failRun` and `writeAnswer` — because a daemon stays up for days and
|
|
406
|
+
has no business holding Tuesday's secret. */
|
|
407
|
+
forgetSecrets(processToken === undefined ? runId : `${runId}:${processToken}`);
|
|
408
|
+
return data;
|
|
409
|
+
}
|
|
410
|
+
/**
|
|
411
|
+
* ═══ THE ANSWER, THE END OF THE RUN AND THE STATE OF THE CARD ARE ONE
|
|
412
|
+
* STATEMENT. ═══
|
|
413
|
+
*
|
|
414
|
+
* They are one event, and writing them as three round trips left a window where
|
|
415
|
+
* the answer was on the card while the run still read running: recovery then
|
|
416
|
+
* handed the turns back and a second agent answered the same question beside an
|
|
417
|
+
* answer that was already there. `panel3_answer` is where that window used to be.
|
|
418
|
+
* It also decides whether the card is DONE, which it is only when nothing else
|
|
419
|
+
* on it is still live — a coordinator that dispatched and answered must not
|
|
420
|
+
* settle a card whose work is still going. See 20260823090000.
|
|
421
|
+
*
|
|
422
|
+
* ═══ AND IT IS SHARED BY EVERY LEVEL ON PURPOSE. ═══ ux.md's "whoever did the
|
|
423
|
+
* work writes the answer" is only structurally true if there is exactly one way
|
|
424
|
+
* for text to reach a card. A second path for dispatched runs is where a
|
|
425
|
+
* summarising hop would eventually be added.
|
|
426
|
+
*
|
|
427
|
+
* ═══ THIS IS THE ONE v3 WRITE THAT DOES NOT GO THROUGH `returned()`. ═══ The
|
|
428
|
+
* shared guard treats "no error and no data" as a result it does not understand,
|
|
429
|
+
* which is right everywhere else and wrong here: a null from `panel3_answer` is a
|
|
430
|
+
* FACT about the record — the run was no longer running, so nothing was written
|
|
431
|
+
* — and it has its own sentence below. A failure is still a failure and still
|
|
432
|
+
* throws.
|
|
433
|
+
*
|
|
434
|
+
* The run's text is on the card as a turn and is deliberately NOT copied into
|
|
435
|
+
* `report` as well: one fact, one place. `report` carries the reason on the
|
|
436
|
+
* failure paths, where there is no turn and it is the only record.
|
|
437
|
+
*
|
|
438
|
+
* ═══ AND A RUN WITH NOTHING TO SAY PASSES NULL, WHICH IS NOT THE SAME AS
|
|
439
|
+
* SAYING NOTHING. ═══ ux.md's "outbound follows the work": a run started
|
|
440
|
+
* again only to settle a question from somebody it sent did no work for the
|
|
441
|
+
* person, and its words went to that agent through the answer. It is still
|
|
442
|
+
* ended here, and the card is still settled by the same statement, because such
|
|
443
|
+
* a run can be the last thing live on it — but no turn is written, exactly as
|
|
444
|
+
* none is written for a run that stopped to ask.
|
|
445
|
+
*/
|
|
446
|
+
async function writeAnswer(client, runId, cardId, text, processToken) {
|
|
447
|
+
/* ═══ THE SECOND OF THE TWO CHOKEPOINTS THE CREDENTIAL RULE RESTS ON. ═══
|
|
448
|
+
This is the ONE path an agent's own words take to a hosted row — a turn on
|
|
449
|
+
the card at levels 1 and 2, and a level 3's `panel3_runs.report`, which
|
|
450
|
+
`panel3_answer` writes instead. Tool arguments are the other, redacted in
|
|
451
|
+
`buildServer`. Identity for a run that read no credential, so every existing
|
|
452
|
+
exact-string assertion is byte-identical. */
|
|
453
|
+
const { data: turnId, error } = await client
|
|
454
|
+
.rpc('panel3_answer', {
|
|
455
|
+
p_run_id: runId,
|
|
456
|
+
p_body: text === null ? null : redactSecrets(processToken === undefined ? runId : `${runId}:${processToken}`, text),
|
|
457
|
+
...(processToken === undefined ? {} : { p_process_token: processToken }),
|
|
458
|
+
});
|
|
459
|
+
if (error) {
|
|
460
|
+
/* Thrown, not settled as failed. The run stays `running` with a process
|
|
461
|
+
that has now exited, so recovery reaps it: at level 1 the turns are
|
|
462
|
+
answered again, so the user gets an answer late instead of a card marked
|
|
463
|
+
failed for a database hiccup. The caller prints the reason. */
|
|
464
|
+
throw new Error(`could not write the answer to card ${cardId}: ${error.message}`);
|
|
465
|
+
}
|
|
466
|
+
/* ═══ INCLUDING WHEN THE RUN STOPPED TO ASK AND WILL BE STARTED AGAIN. ═══
|
|
467
|
+
That reads like a mistake and is not: the process is gone, and the one that
|
|
468
|
+
takes its place is a fresh process with a fresh context which reads the
|
|
469
|
+
credential again through `get_credential`. What it is handed to start from —
|
|
470
|
+
its own report, and the card — went through this same substitution, so there
|
|
471
|
+
is nothing left for a stale entry to protect. */
|
|
472
|
+
forgetSecrets(processToken === undefined ? runId : `${runId}:${processToken}`);
|
|
473
|
+
if (turnId === null) {
|
|
474
|
+
/* NO TURN WAS WRITTEN, and there are FOUR reasons, which mean different
|
|
475
|
+
things and must not be printed as one line.
|
|
476
|
+
|
|
477
|
+
IT STOPPED TO ASK. Its last words were the question, which is already on
|
|
478
|
+
the card, and its end has just been stamped by the same statement that
|
|
479
|
+
refused the turn. Nothing is wrong and nothing is lost.
|
|
480
|
+
|
|
481
|
+
IT HAD NOTHING TO SAY TO THE PERSON. A run started only to settle a
|
|
482
|
+
question from somebody it sent is asked for exactly this, and it is the
|
|
483
|
+
daemon that asked.
|
|
484
|
+
|
|
485
|
+
IT IS A WORKER, and its words went into its report for whoever sent it.
|
|
486
|
+
ux.md: one request is one answer, written by the level that owns the card.
|
|
487
|
+
|
|
488
|
+
OR IT WAS NO LONGER RUNNING — recovery had given up on it and, at level 1,
|
|
489
|
+
its turns belong to another run now. Writing anyway would put a second
|
|
490
|
+
answer on the card. The process's work is lost, which is the honest cost
|
|
491
|
+
of having been declared dead.
|
|
492
|
+
|
|
493
|
+
═══ AND THE STATE IS READ FIRST, WHICH IS THE FIX FOR A LINE THAT USED TO
|
|
494
|
+
LIE. ═══ The last of the four was reachable with a null body — a re-armed
|
|
495
|
+
run given up on while its process was still going — and it printed the
|
|
496
|
+
second one's sentence, "answered a question, so it wrote nothing here",
|
|
497
|
+
about a run that had been declared dead and written nothing anywhere. So
|
|
498
|
+
the state gates the two honest sentences rather than sitting under them.
|
|
499
|
+
It decides nothing except which sentence is true: whichever of the four
|
|
500
|
+
happened has already happened, in one statement, before this line runs. */
|
|
501
|
+
const run = await runNow(client, runId);
|
|
502
|
+
if (run.state === 'asked') {
|
|
503
|
+
out(`asked card ${cardId} run ${runId} stopped on a question, so it wrote no answer`);
|
|
504
|
+
}
|
|
505
|
+
else if (run.state !== 'finished') {
|
|
506
|
+
said(`run ${runId} was ${run.state}, so its answer was not written to card ${cardId}`);
|
|
507
|
+
}
|
|
508
|
+
else if (run.level === 1 && run.conversationOwnerId !== null) {
|
|
509
|
+
out(`settled card ${cardId} run ${runId} assigned the conversation, so it wrote nothing here`);
|
|
510
|
+
}
|
|
511
|
+
else if (text === null) {
|
|
512
|
+
out(`settled card ${cardId} run ${runId} answered a question, so it wrote nothing here`);
|
|
513
|
+
}
|
|
514
|
+
else if (run.level === 3) {
|
|
515
|
+
out(`reported run ${runId} ${text.length} characters, to whoever sent it`);
|
|
516
|
+
}
|
|
517
|
+
else {
|
|
518
|
+
/* FINISHED, WITH SOMETHING TO SAY, AND NO TURN. It had already answered
|
|
519
|
+
once, which is the one shape left, and it is said rather than assumed
|
|
520
|
+
away. */
|
|
521
|
+
said(`run ${runId} had already ended, so its answer was not written to card ${cardId}`);
|
|
522
|
+
}
|
|
523
|
+
return ownerSettlementAccepted(run, processToken);
|
|
524
|
+
}
|
|
525
|
+
out(`answered card ${cardId} run ${runId} ${text?.length ?? 0} characters`);
|
|
526
|
+
return true;
|
|
527
|
+
}
|
|
528
|
+
export function ownerSettlementAccepted(run, processToken) {
|
|
529
|
+
return (run.state === 'asked' || run.state === 'finished')
|
|
530
|
+
&& (processToken === undefined || run.processToken === processToken);
|
|
531
|
+
}
|
|
532
|
+
/** What the record says a run is now, and at what level. Read only to say the
|
|
533
|
+
* right sentence about something that has already happened; nothing branches on
|
|
534
|
+
* it that could have gone the other way. */
|
|
535
|
+
async function runNow(client, runId) {
|
|
536
|
+
const runs = await returned(client
|
|
537
|
+
.from('panel3_runs')
|
|
538
|
+
.select('state, level, process_token, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id)')
|
|
539
|
+
.eq('id', runId), 'read', `the state of run ${runId}`);
|
|
540
|
+
const run = runs[0];
|
|
541
|
+
return run
|
|
542
|
+
? {
|
|
543
|
+
state: run.state,
|
|
544
|
+
level: run.level,
|
|
545
|
+
conversationOwnerId: run.card?.conversation_run_id ?? null,
|
|
546
|
+
processToken: run.process_token,
|
|
547
|
+
}
|
|
548
|
+
: { state: 'no longer on the record', level: null, conversationOwnerId: null, processToken: null };
|
|
549
|
+
}
|
|
550
|
+
/**
|
|
551
|
+
* A run whose process failed, ended with the reason ON ITS OWN COLUMN, and its
|
|
552
|
+
* turns left exactly where they are.
|
|
553
|
+
*
|
|
554
|
+
* ═══ NEVER INTO `report`, WHICH IS THE RUN'S OWN WORDS. ═══ `report` is what a
|
|
555
|
+
* respawn is handed, and this build starts settled runs again — an answered
|
|
556
|
+
* question does exactly that — so a reason written there would destroy the state
|
|
557
|
+
* the next attempt was going to resume from, and `cs3 show` would print the
|
|
558
|
+
* daemon's sentence under a heading that says the run wrote it. One owner per
|
|
559
|
+
* column: the run writes `report`, whoever ended it writes `failed_because`.
|
|
560
|
+
*
|
|
561
|
+
* ═══ AND IT IS A STATEMENT RATHER THAN AN UPDATE, BECAUSE `ended_at` IS
|
|
562
|
+
* COMPARED AGAINST OTHER ROWS' ENDS. ═══
|
|
563
|
+
*
|
|
564
|
+
* This wrote the stamp from THIS MACHINE'S clock, because a PostgREST update has
|
|
565
|
+
* nowhere to ask for the server's time. Everything else that ends a run —
|
|
566
|
+
* `panel3_answer`, `panel3_give_up` — uses the database's `now()`, and
|
|
567
|
+
* `panel3_owed_read_back` measures a parent's end against its children's. So the
|
|
568
|
+
* two stamps met, and skew between them loses a read-back SILENTLY: a machine
|
|
569
|
+
* clock running ahead writes a parent's end later than the child that genuinely
|
|
570
|
+
* outlived it, the comparison finds nobody who finished after it, and the
|
|
571
|
+
* synthesis never happens with every row individually plausible.
|
|
572
|
+
*
|
|
573
|
+
* The header of this file already argues that v3 must not compare two clocks.
|
|
574
|
+
* This was the one place left that did.
|
|
575
|
+
*/
|
|
576
|
+
async function failRun(client, runId, reason) {
|
|
577
|
+
/* ═══ ONLY A RUN THAT HAS NOT ALREADY ENDED, AND THAT GUARD IS IN THE
|
|
578
|
+
STATEMENT. ═══ Unguarded, this wrote over a run that had been started again
|
|
579
|
+
in the meantime — which is not hypothetical now that an answered question
|
|
580
|
+
restarts a settled row — and marked a live agent failed. False is a fact the
|
|
581
|
+
caller reports, not a failure. */
|
|
582
|
+
const { data, error } = await client
|
|
583
|
+
.rpc('panel3_end_run', { p_run_id: runId, p_reason: reason });
|
|
584
|
+
if (error)
|
|
585
|
+
throw new Error(`could not end run ${runId}: ${error.message}`);
|
|
586
|
+
forgetSecrets(runId);
|
|
587
|
+
return data === true;
|
|
588
|
+
}
|
|
589
|
+
/**
|
|
590
|
+
* What the card is doing, written by the daemon.
|
|
591
|
+
*
|
|
592
|
+
* ═══ EXCEPT ON A CARD THE USER STOPPED, WHICH NOTHING HERE MAY OVERWRITE. ═══
|
|
593
|
+
* ux.md: the user's Stop is absolute. Its own effect is the reason this guard is
|
|
594
|
+
* needed rather than a precaution. `panel3_stop_card` ends the runs, this daemon
|
|
595
|
+
* kills their processes, and the level 1 agent then exits badly, which is the
|
|
596
|
+
* one path in this file that writes a card `failed`. Without the predicate the
|
|
597
|
+
* user presses Stop and the card reads Failed a second later, about a failure
|
|
598
|
+
* that never happened.
|
|
599
|
+
*
|
|
600
|
+
* Every other card write already carries the guard in SQL: `panel3_answer` and
|
|
601
|
+
* `panel3_give_up` both refuse a run that is not live, and a stopped run is not.
|
|
602
|
+
*/
|
|
603
|
+
async function setCardState(client, cardId, state) {
|
|
604
|
+
await returned(client
|
|
605
|
+
.from('panel3_cards')
|
|
606
|
+
.update({ state })
|
|
607
|
+
.eq('id', cardId)
|
|
608
|
+
.neq('state', 'stopped')
|
|
609
|
+
.select('id'), 'set the state of', `card ${cardId}`);
|
|
610
|
+
}
|
|
611
|
+
// ---------------------------------------------------------------------------
|
|
612
|
+
/**
|
|
613
|
+
* One card, from the turns the take handed over to the answer on its thread.
|
|
614
|
+
*
|
|
615
|
+
* ═══ THERE IS NOTHING TO UNDO HERE, AND THAT IS THE POINT. ═══ The take wrote
|
|
616
|
+
* the run row in the same statement that leased the turns, so this function is
|
|
617
|
+
* never in the position of holding a lease it cannot honour: whatever it fails
|
|
618
|
+
* at, the row exists, `cs3 show` can see it, and recovery can reach it. The
|
|
619
|
+
* earlier version wrote the row two round trips later and had to hand the turns
|
|
620
|
+
* back on failure — which covered a thrown error and not a kill, and a kill in
|
|
621
|
+
* that window stranded the card forever.
|
|
622
|
+
*
|
|
623
|
+
* ═══ AND WHAT HAPPENS WHEN THE PROCESS IS OVER IS `settle`'s, NOT THIS
|
|
624
|
+
* FUNCTION'S. ═══ It used to be written out again here, and the two copies
|
|
625
|
+
* were already the same thing said twice: the agent's words onto the card
|
|
626
|
+
* through `writeAnswer`, or the run ended with its reason. recovery Slice 3 gave
|
|
627
|
+
* that moment a second job, starting a process that died badly all over again,
|
|
628
|
+
* and a second copy of THAT is two features owning one respawn, which is the
|
|
629
|
+
* failure recovery-1/ux.md names in as many words. So there is one owner, and
|
|
630
|
+
* the level is what tells it how a run of this shape ends.
|
|
631
|
+
*/
|
|
632
|
+
async function answerCard(client, tools, machineId, cardId, turns) {
|
|
633
|
+
const runId = turns[0].run_id;
|
|
634
|
+
/* BEFORE THE SPAWN, AND ITS FAILURE IS THE SPAWN'S FAILURE. The receipts are
|
|
635
|
+
part of what the agent is sent, so a read that fails must not be papered
|
|
636
|
+
over with an empty list: that reads as a card that has made nothing, which
|
|
637
|
+
is how an agent creates a second epic beside the one it cannot see. The
|
|
638
|
+
attachments read carries the same rule: a failed read here must not read
|
|
639
|
+
as "nothing is attached", which is a different card than the one that was
|
|
640
|
+
actually sent. */
|
|
641
|
+
/* ═══ AND A FAILURE HERE ENDS THE RUN, RATHER THAN LEAVING IT RUNNING WITH NO
|
|
642
|
+
PROCESS. ═══ The take already wrote the run row in the statement that leased
|
|
643
|
+
the turns, so a throw between here and `startAgent` leaves a run reading
|
|
644
|
+
`running` with a null pid and nothing on stderr the person can see.
|
|
645
|
+
`recoverStranded` then reads that as a machine that went away, hands the
|
|
646
|
+
message back, and `panel3_take_turns` leases it to A BRAND NEW RUN whose
|
|
647
|
+
attempts start again at one — so a permanent failure, such as a read this
|
|
648
|
+
build cannot make against the current schema, repeats forever while the card
|
|
649
|
+
says Working and never says why. This is the sixth of `endRun`'s endings and
|
|
650
|
+
the last one that was missing: `resumeRun` and `startRearmed` already end
|
|
651
|
+
their two post-claim failures this way for exactly this reason.
|
|
652
|
+
THE REASON IS SHAREABLE. All three reads are `returned()` calls against the
|
|
653
|
+
database, whose messages name tables and columns and never a local path, so
|
|
654
|
+
constraint 6 is satisfied without a level fork here. */
|
|
655
|
+
let brief;
|
|
656
|
+
try {
|
|
657
|
+
brief = briefFor(turns[0].card_title, turns, await receiptsFor(client, cardId), await attachmentsFor(client, cardId), await codebasesForRun(client, runId));
|
|
658
|
+
}
|
|
659
|
+
catch (error) {
|
|
660
|
+
const why = error instanceof Error ? error.message : String(error);
|
|
661
|
+
await endRun(client, LEVEL, runId, cardId, why);
|
|
662
|
+
throw new Error(`NO AGENT IS RUNNING: ${why}`);
|
|
663
|
+
}
|
|
664
|
+
/* ═══ THE RUN ID IS ON THE URL, AND THAT IS THE WHOLE OF WHAT THE AGENT IS
|
|
665
|
+
TOLD ABOUT ITS OWN STANDING. ═══ The tools server reads the level off the
|
|
666
|
+
run row this id names, so the daemon does not tell the child what it may do
|
|
667
|
+
and the child has nothing to claim. `LEVEL` below decides argv only — which
|
|
668
|
+
of the harness's own tools the process gets — and the two can never disagree
|
|
669
|
+
about the record, because only one of them consults it. */
|
|
670
|
+
/* AN EMPTY DIRECTORY OF THE USER'S OWN, because level 1 has no code tool to
|
|
671
|
+
use a real one with, and the daemon's inherited cwd under a launchd login
|
|
672
|
+
item is the filesystem root. */
|
|
673
|
+
const started = startAgent(brief, LEVEL, tools.urlFor(runId), scratchDir());
|
|
674
|
+
out(`run ${runId} level ${LEVEL} ${started.pid ? `pid ${started.pid}` : 'no process'}`);
|
|
675
|
+
try {
|
|
676
|
+
await recordProcess(client, runId, started.pid, brief);
|
|
677
|
+
}
|
|
678
|
+
catch (error) {
|
|
679
|
+
// Said, not fatal. See `recordProcess` for what this costs and why the
|
|
680
|
+
// agent is not killed over it.
|
|
681
|
+
said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
|
|
682
|
+
}
|
|
683
|
+
return settle(client, tools, machineId, LEVEL, runId, cardId, started, false);
|
|
684
|
+
}
|
|
685
|
+
/**
|
|
686
|
+
* WHICH PROJECT A RUN'S CARD IS FILED UNDER, or null when the card has none.
|
|
687
|
+
*
|
|
688
|
+
* ═══ ASKED OF THE RECORD AT EVERY SPAWN, RATHER THAN CARRIED. ═══ The card is
|
|
689
|
+
* the durable object and its project can change between one dispatch and the
|
|
690
|
+
* next; a value threaded through the daemon would be the state of the card when
|
|
691
|
+
* the daemon last looked. It is one indexed read on the path that is about to
|
|
692
|
+
* start a whole agent.
|
|
693
|
+
*
|
|
694
|
+
* ═══ AND A FAILED READ IS NOT "NO PROJECT". ═══ `returned` throws, which the
|
|
695
|
+
* caller turns into a dispatch that did not happen. Returning null here would
|
|
696
|
+
* quietly send the agent to the machine-wide fallback checkout, which is another
|
|
697
|
+
* project's code.
|
|
698
|
+
*/
|
|
699
|
+
async function projectOfRun(client, runId) {
|
|
700
|
+
const rows = await returned(client.from('panel3_runs').select('card:panel3_cards!panel3_runs_card_id_fkey(project_id)').eq('id', runId), 'read', 'which project this run belongs to');
|
|
701
|
+
return rows[0]?.card?.project_id ?? null;
|
|
702
|
+
}
|
|
703
|
+
async function codebasesForRun(client, runId) {
|
|
704
|
+
const projectId = await projectOfRun(client, runId);
|
|
705
|
+
return projectId ? listCodebases(client, projectId) : null;
|
|
706
|
+
}
|
|
707
|
+
async function codebaseOfRun(client, runId) {
|
|
708
|
+
const rows = await returned(client
|
|
709
|
+
.from('panel3_runs')
|
|
710
|
+
.select('codebase_id, card:panel3_cards!panel3_runs_card_id_fkey(project_id)')
|
|
711
|
+
.eq('id', runId), 'read', `which codebase run ${runId} belongs to`);
|
|
712
|
+
const row = rows[0];
|
|
713
|
+
if (!row?.codebase_id || !row.card?.project_id) {
|
|
714
|
+
throw new Error('This code work does not name a registered project codebase.');
|
|
715
|
+
}
|
|
716
|
+
const codebase = (await listCodebases(client, row.card.project_id))
|
|
717
|
+
.find((candidate) => candidate.id === row.codebase_id);
|
|
718
|
+
if (!codebase) {
|
|
719
|
+
throw new Error('The selected codebase is no longer registered on this project.');
|
|
720
|
+
}
|
|
721
|
+
return codebase;
|
|
722
|
+
}
|
|
723
|
+
/** One directory rule for every spawn whose level and ownership are known. */
|
|
724
|
+
async function workingDirectory(client, runId, level, isOwner, knownCodebase) {
|
|
725
|
+
if (level === 1)
|
|
726
|
+
return scratchDir();
|
|
727
|
+
if (knownCodebase === null) {
|
|
728
|
+
if (level === 2 && isOwner)
|
|
729
|
+
return scratchDir();
|
|
730
|
+
throw new Error('This code work does not name a registered project codebase.');
|
|
731
|
+
}
|
|
732
|
+
return checkoutForCodebase(knownCodebase ?? await codebaseOfRun(client, runId), hostname());
|
|
733
|
+
}
|
|
734
|
+
/**
|
|
735
|
+
* ═══ ONE AGENT, STARTED UNDER ANOTHER THAT IS STILL RUNNING. ═══
|
|
736
|
+
*
|
|
737
|
+
* Called from the `dispatch` tool, and it returns as soon as the process exists.
|
|
738
|
+
* The promise it hands back settles long afterwards, and the caller puts it
|
|
739
|
+
* where the daemon can wait on it rather than awaiting it itself: an agent that
|
|
740
|
+
* waited for the work it started would hold its own run open for the length of
|
|
741
|
+
* it, which is the thing ux.md forbids in as many words.
|
|
742
|
+
*
|
|
743
|
+
* ═══ THE ORDER IS THE WHOLE CORRECTNESS ARGUMENT. ═══ The working copy is
|
|
744
|
+
* resolved FIRST, before anything is written, so a machine that has none refuses
|
|
745
|
+
* the dispatch rather than leaving a run row no process ever corresponded to.
|
|
746
|
+
* Then the row, then the process, then the pid — constraint 8, in the only order
|
|
747
|
+
* that satisfies it.
|
|
748
|
+
*/
|
|
749
|
+
async function startChild(client, tools, machineId, parentRunId, brief, codebase, parentProcessToken) {
|
|
750
|
+
const { data, error } = await client.rpc('panel3_dispatch', {
|
|
751
|
+
p_parent_run_id: parentRunId,
|
|
752
|
+
p_brief: brief,
|
|
753
|
+
p_machine_id: machineId,
|
|
754
|
+
p_codebase_id: codebase?.id ?? null,
|
|
755
|
+
p_codebase_label: codebase?.name ?? null,
|
|
756
|
+
...(parentProcessToken === undefined ? {} : { p_process_token: parentProcessToken }),
|
|
757
|
+
});
|
|
758
|
+
if (error)
|
|
759
|
+
throw new Error(`could not start an agent under run ${parentRunId}: ${error.message}`);
|
|
760
|
+
const row = data?.[0];
|
|
761
|
+
if (!row) {
|
|
762
|
+
/* NOTHING WAS WRITTEN AND NOTHING IS RUNNING, and the two reasons are said
|
|
763
|
+
together because the caller cannot tell them apart from here and both mean
|
|
764
|
+
the same thing to it. */
|
|
765
|
+
throw new Error(`NO AGENT WAS STARTED and nothing was written: run ${parentRunId} has ended, is already as `
|
|
766
|
+
+ 'deep as anything may be sent from, or this conversation already has its owner. Exit now.');
|
|
767
|
+
}
|
|
768
|
+
if (row.run_level !== 2 && row.run_level !== 3) {
|
|
769
|
+
/* UNREACHABLE, AND STILL SETTLED. `panel3_dispatch` writes `parent.level + 1`
|
|
770
|
+
from a parent it has just checked is below 3, so there is no level here
|
|
771
|
+
this cannot spawn. If that ever stops being true, the row exists and
|
|
772
|
+
nothing will ever start for it, and leaving it `running` would make a
|
|
773
|
+
recovery sweep wait out the pid grace window to conclude what is already
|
|
774
|
+
known. */
|
|
775
|
+
const why = `run ${row.run_id} was written at level ${row.run_level}, which cannot be spawned`;
|
|
776
|
+
await giveUp(client, row.run_id, why, row.process_token ?? undefined);
|
|
777
|
+
throw new Error(why);
|
|
778
|
+
}
|
|
779
|
+
const level = row.run_level;
|
|
780
|
+
let cwd;
|
|
781
|
+
let prompt;
|
|
782
|
+
try {
|
|
783
|
+
cwd = await workingDirectory(client, row.run_id, level, level === 2, codebase);
|
|
784
|
+
prompt = level === 2
|
|
785
|
+
? await initialOwnerPrompt(client, row.run_card_id, parentRunId, brief)
|
|
786
|
+
: brief;
|
|
787
|
+
}
|
|
788
|
+
catch (error) {
|
|
789
|
+
const why = error instanceof Error ? error.message : String(error);
|
|
790
|
+
await giveUp(client, row.run_id, why, row.process_token ?? undefined);
|
|
791
|
+
throw new Error(`NO AGENT IS RUNNING: ${why}`);
|
|
792
|
+
}
|
|
793
|
+
const processToken = row.process_token ?? undefined;
|
|
794
|
+
const isOwner = level === 2;
|
|
795
|
+
const started = startAgent(prompt, level, tools.urlFor(row.run_id, processToken), cwd, isOwner ? { ownerId: row.run_id } : undefined);
|
|
796
|
+
if (started.pid === null) {
|
|
797
|
+
/* THERE IS A ROW AND THERE IS NO PROCESS, which is the one shape the record
|
|
798
|
+
must never be left in quietly. The answer is already settled — nothing ran
|
|
799
|
+
— so the reason is read off it, the run is ended with that reason on it,
|
|
800
|
+
and the tool call fails saying no agent was started. */
|
|
801
|
+
const answer = await started.answered;
|
|
802
|
+
const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
|
|
803
|
+
await giveUp(client, row.run_id, reason, processToken);
|
|
804
|
+
throw new Error(`NO AGENT IS RUNNING: ${reason}`);
|
|
805
|
+
}
|
|
806
|
+
out(`dispatch run ${row.run_id} level ${level} under ${parentRunId} pid ${started.pid}`);
|
|
807
|
+
try {
|
|
808
|
+
await recordProcess(client, row.run_id, started.pid, undefined, processToken);
|
|
809
|
+
}
|
|
810
|
+
catch (error) {
|
|
811
|
+
// Said, not fatal, exactly as at level 1: the agent is running and killing
|
|
812
|
+
// it over a bookkeeping write would cost the user the work.
|
|
813
|
+
said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
|
|
814
|
+
}
|
|
815
|
+
return {
|
|
816
|
+
runId: row.run_id,
|
|
817
|
+
settled: settle(client, tools, machineId, level, row.run_id, row.run_card_id, started, true, processToken, isOwner && processToken
|
|
818
|
+
? ownerSessionLifecycle(started, row.run_id, harness(), processToken)
|
|
819
|
+
: undefined),
|
|
820
|
+
};
|
|
821
|
+
}
|
|
822
|
+
/**
|
|
823
|
+
* HOW MANY PROCESSES HAVE BEEN STARTED FOR THIS RUN, counting the first.
|
|
824
|
+
*
|
|
825
|
+
* ═══ IT SUPPLIES THE NUMBER IN THE SENTENCE, NOT THE BOUND. ═══ The bound is
|
|
826
|
+
* `panel3_resume`'s own predicate, counted and tested in the statement that
|
|
827
|
+
* grants the attempt, so a read here that is a moment out of date cannot grant
|
|
828
|
+
* an attempt the record refuses. The worst it can do is decline one the record
|
|
829
|
+
* would have allowed, which is a run that fails a spawn early rather than a
|
|
830
|
+
* fourth process nobody bounded.
|
|
831
|
+
*
|
|
832
|
+
* A row that is not there is not "one attempt". It is a run this daemon can
|
|
833
|
+
* neither retry nor fail, because there is nothing left to write to, and saying
|
|
834
|
+
* so is the caller's business rather than this function's to paper over.
|
|
835
|
+
*/
|
|
836
|
+
async function attemptsSoFar(client, runId) {
|
|
837
|
+
const rows = await returned(client.from('panel3_runs').select('attempts').eq('id', runId), 'read', `how many times a process has been started for run ${runId}`);
|
|
838
|
+
const row = rows[0];
|
|
839
|
+
if (!row)
|
|
840
|
+
throw new Error(`run ${runId} is not there, so nothing can be recorded about it`);
|
|
841
|
+
return row.attempts;
|
|
842
|
+
}
|
|
843
|
+
/**
|
|
844
|
+
* ═══ ONE RUN, ENDED WITH ITS REASON, THE WAY A RUN OF ITS LEVEL ENDS. ═══
|
|
845
|
+
*
|
|
846
|
+
* There are two endings and the difference is not a preference, it is what the
|
|
847
|
+
* run is holding.
|
|
848
|
+
*
|
|
849
|
+
* A DISPATCHED RUN HOLDS NO LEASED TURNS. `panel3_give_up` hands nothing back
|
|
850
|
+
* and puts the card in `failed`, because nothing on that card will pick this
|
|
851
|
+
* piece of work up again unless a question outstanding somewhere else on it says
|
|
852
|
+
* work is genuinely coming back.
|
|
853
|
+
*
|
|
854
|
+
* A LEVEL 1 RUN HOLDS THE PERSON'S MESSAGE. Giving up hands it back for the next
|
|
855
|
+
* `panel3_take_turns` to lease to A BRAND NEW RUN, which is recovery's whole
|
|
856
|
+
* point when the MACHINE is gone and is the wrong ending everywhere else: a new
|
|
857
|
+
* run is a new `attempts`, so the bound below would never be spent, a harness
|
|
858
|
+
* that fails every time would be started again forever, and a coordinator that
|
|
859
|
+
* had already dispatched workers would dispatch them a second time. So level 1
|
|
860
|
+
* ends through `panel3_end_run`, which hands no turns back, and the card says
|
|
861
|
+
* failed.
|
|
862
|
+
*
|
|
863
|
+
* ═══ AND THE FORK LIVES HERE AND NOWHERE ELSE. ═══ SIX places end a run this
|
|
864
|
+
* way: the settle, the two failures after the claim in each of `resumeRun` and
|
|
865
|
+
* `startRearmed`, and the pre-spawn failure in `answerCard`. A copy of this rule
|
|
866
|
+
* in each is six places for it to drift, and it drifted the first time it could:
|
|
867
|
+
* those two functions hold the same pair of endings and only one pair was looked
|
|
868
|
+
* at. `answerCard`'s was missing outright, which is how a build that could not
|
|
869
|
+
* read a card's project turned every take into a silent take/recover loop.
|
|
870
|
+
*
|
|
871
|
+
* ALL FIVE CAN SEE LEVEL 1. `resumeRun` serving the retry is what opened its
|
|
872
|
+
* two. `startRearmed`'s were always reachable, because a re-arm has always
|
|
873
|
+
* served level 1, and were always ending one the way a dispatched run ends. The
|
|
874
|
+
* bound is what made that matter: a handed-back message is leased to a NEW run,
|
|
875
|
+
* whose attempts start again at one.
|
|
876
|
+
*
|
|
877
|
+
* ═══ AND THE GIVE-UPS THAT ARE NOT HERE ARE NOT OVERSIGHTS. ═══ `startChild`'s
|
|
878
|
+
* two end a dispatched run by construction, since a dispatch writes its parent's
|
|
879
|
+
* level plus one and there is no parent below level 1. The bad-level branches in
|
|
880
|
+
* `resumeRun` and `startRearmed` hold a level that is by definition none of the
|
|
881
|
+
* three, so there is no fork to make. And `recoverStranded` gives a level 1 run
|
|
882
|
+
* up ON PURPOSE, handing its message back, which is the whole of how another
|
|
883
|
+
* machine takes over a run whose machine is gone.
|
|
884
|
+
*
|
|
885
|
+
* Returns whether THIS call is what ended it. False is a fact about the record,
|
|
886
|
+
* not a failure: the run had already ended, and the caller says so.
|
|
887
|
+
*/
|
|
888
|
+
async function endRun(client, level, runId, cardId, why) {
|
|
889
|
+
if (level !== 1)
|
|
890
|
+
return (await giveUp(client, runId, why)) !== null;
|
|
891
|
+
/* ═══ THE RUN FIRST, AND THE CARD ONLY IF THIS RUN WAS STILL THE CARD'S TO
|
|
892
|
+
FAIL. ═══
|
|
893
|
+
The card used to be written first, and the argument for that was a daemon
|
|
894
|
+
killed between the two: it left a card reading `failed` beside a run that
|
|
895
|
+
still read running with a dead process, which the record's own forbidden
|
|
896
|
+
state check reports and recovery corrects. THE PREEMPTION MAKES THAT ORDER A
|
|
897
|
+
LIE THAT NOTHING CORRECTS. A coordinator the person's next message replaced
|
|
898
|
+
is `stopped`, its process is killed by this same daemon, and it arrives here
|
|
899
|
+
having failed: writing the card `failed` then marks the card that a FRESH
|
|
900
|
+
coordinator is working on, about a run nobody is waiting for.
|
|
901
|
+
`panel3_end_run` is what knows the difference, because it refuses a run that
|
|
902
|
+
has already ended, so its answer is what the card write waits for.
|
|
903
|
+
WHAT THE ORDER COSTS is a daemon killed between the two, which leaves the
|
|
904
|
+
run `failed` and the card still reading `working` with nothing on it. The
|
|
905
|
+
product raises that as a forbidden state on the card itself, and the
|
|
906
|
+
person's next message takes the card again. That is a smaller and louder
|
|
907
|
+
wrong than a card marked failed under an agent that is working. */
|
|
908
|
+
const ended = await failRun(client, runId, why);
|
|
909
|
+
if (ended)
|
|
910
|
+
await setCardState(client, cardId, 'failed');
|
|
911
|
+
return ended;
|
|
912
|
+
}
|
|
913
|
+
/**
|
|
914
|
+
* ═══ THE END OF ONE RUN'S PROCESS, WHICHEVER SPAWN STARTED IT. ═══
|
|
915
|
+
*
|
|
916
|
+
* One owner for what happens when an agent's process exits, because there are
|
|
917
|
+
* four ways one comes to exist, being a card taken, a `dispatch`, a resume of a
|
|
918
|
+
* run whose process was found gone, and a re-arm, and they end identically. A
|
|
919
|
+
* second copy would be a second answer to "what does a dead agent mean", and the
|
|
920
|
+
* first thing to drift would be whether the card is told.
|
|
921
|
+
*
|
|
922
|
+
* ═══ `speaksToTheCard` IS THE ONE THING THAT DIFFERS ON THE WAY IN, AND IT IS
|
|
923
|
+
* ux.md's RULE RATHER THAN A SETTING. ═══ "Outbound follows the work." A run
|
|
924
|
+
* started again only to settle a question from somebody it sent did no work for
|
|
925
|
+
* the person: its words went to that agent, through the answer, and putting them
|
|
926
|
+
* on the card as well is a voice in a conversation the person did not ask to
|
|
927
|
+
* follow. It is still ended, and the card is still settled, because it can be
|
|
928
|
+
* the last thing live on it.
|
|
929
|
+
*
|
|
930
|
+
* ═══ AND `level` IS THE ONE THING THAT DIFFERS ON THE WAY OUT, WHICH IS
|
|
931
|
+
* `endRun`'s. ═══ It is above, with the whole argument: a dispatched run
|
|
932
|
+
* holds no leased turns and a level 1 run holds the person's message, so they
|
|
933
|
+
* cannot end the same way. What `answerCard` always did is what level 1 still
|
|
934
|
+
* does, and the only thing that has moved is where it is written.
|
|
935
|
+
*/
|
|
936
|
+
function ownerSessionLifecycle(started, ownerId, ownerHarness, processToken, expectedSessionId) {
|
|
937
|
+
const pending = started.session.then((nativeSessionId) => {
|
|
938
|
+
if (nativeSessionId === null)
|
|
939
|
+
return false;
|
|
940
|
+
if (expectedSessionId && nativeSessionId !== expectedSessionId) {
|
|
941
|
+
said(`the resumed harness returned a different conversation for owner ${ownerId}; it will not be reused`);
|
|
942
|
+
return false;
|
|
943
|
+
}
|
|
944
|
+
try {
|
|
945
|
+
writeOwnerSession({
|
|
946
|
+
ownerId,
|
|
947
|
+
harness: ownerHarness,
|
|
948
|
+
nativeSessionId,
|
|
949
|
+
processToken,
|
|
950
|
+
state: 'pending',
|
|
951
|
+
});
|
|
952
|
+
return true;
|
|
953
|
+
}
|
|
954
|
+
catch (error) {
|
|
955
|
+
said(`could not save the local conversation for owner ${ownerId}: ${error instanceof Error ? error.message : String(error)}`);
|
|
956
|
+
return false;
|
|
957
|
+
}
|
|
958
|
+
});
|
|
959
|
+
// Start observing immediately; Codex reports thread.started before its answer.
|
|
960
|
+
void pending;
|
|
961
|
+
return {
|
|
962
|
+
established: async () => {
|
|
963
|
+
if (await pending)
|
|
964
|
+
establishOwnerSession(ownerId, processToken);
|
|
965
|
+
},
|
|
966
|
+
failed: async () => {
|
|
967
|
+
// A failed or rejected generation stays pending and therefore cannot be
|
|
968
|
+
// resumed. Reconciliation is the sole owner of stable-state deletion.
|
|
969
|
+
await pending;
|
|
970
|
+
},
|
|
971
|
+
};
|
|
972
|
+
}
|
|
973
|
+
function settle(client, tools, machineId, level, runId, cardId, started, speaksToTheCard = true, processToken, ownerSession) {
|
|
974
|
+
return started.answered.then(async (answer) => {
|
|
975
|
+
/* ═══ A RUN THAT STOPPED TO ASK DID NOT DIE, WHATEVER THE HARNESS PRINTED
|
|
976
|
+
ON ITS WAY OUT. ═══
|
|
977
|
+
|
|
978
|
+
ux.md: asking is TERMINAL for the agent that asks. It calls the tool, is
|
|
979
|
+
told "you have stopped", and exits — and it owes the person nothing more,
|
|
980
|
+
because its last words were the question and the question is already on
|
|
981
|
+
the card. So a harness that exits having said nothing to the person has
|
|
982
|
+
done exactly what it was told to.
|
|
983
|
+
|
|
984
|
+
`codexAnswer` reads that silence as a death (`spawn.ts`: "codex exited 0
|
|
985
|
+
without saying anything to the person"), which is the right reading of
|
|
986
|
+
every OTHER silent exit and the wrong one of this one. Claude happens to
|
|
987
|
+
emit a closing message after the tool call and Codex does not, so without
|
|
988
|
+
this the same question asked under the two harnesses ends two different
|
|
989
|
+
ways: answerable on one, and on the other retried twice and then FAILED
|
|
990
|
+
with the question still sitting open on a failed card.
|
|
991
|
+
|
|
992
|
+
═══ SO THE RECORD DECIDES IT, NOT THE STDOUT. ═══ Whether a run asked is a
|
|
993
|
+
column, written by `panel3_ask` in the same statement that stopped the
|
|
994
|
+
run, and it is true or false identically for both harnesses. Read only on
|
|
995
|
+
the failure path, where it is the one thing that can turn a death back
|
|
996
|
+
into an ordinary ending, and then handed to `writeAnswer` — which already
|
|
997
|
+
owns this case and already prints its sentence — rather than to a second
|
|
998
|
+
ending written here beside it. */
|
|
999
|
+
if (!answer.ok) {
|
|
1000
|
+
const run = await runNow(client, runId);
|
|
1001
|
+
if (run.state === 'asked') {
|
|
1002
|
+
const accepted = await writeAnswer(client, runId, cardId, null, processToken);
|
|
1003
|
+
if (accepted)
|
|
1004
|
+
await ownerSession?.established();
|
|
1005
|
+
else
|
|
1006
|
+
await ownerSession?.failed();
|
|
1007
|
+
return;
|
|
1008
|
+
}
|
|
1009
|
+
/* Level 1's only successful result is the owner pointer written by
|
|
1010
|
+
dispatch. Its stdout is deliberately hidden, and Codex may therefore
|
|
1011
|
+
exit cleanly without an agent message after the tool succeeds. The
|
|
1012
|
+
record, not prose the launcher was told not to write, decides success. */
|
|
1013
|
+
if (level === 1 && run.conversationOwnerId !== null) {
|
|
1014
|
+
await writeAnswer(client, runId, cardId, null, processToken);
|
|
1015
|
+
return;
|
|
1016
|
+
}
|
|
1017
|
+
}
|
|
1018
|
+
if (!answer.ok) {
|
|
1019
|
+
await ownerSession?.failed();
|
|
1020
|
+
/* ═══ A PROCESS THAT STARTED AND THEN DIED BADLY IS STARTED AGAIN. ═══
|
|
1021
|
+
recovery-1/ux.md, Slice 3: "the agent's harness fails on something that
|
|
1022
|
+
is nobody's fault: a rate limit, a dropped connection, a provider
|
|
1023
|
+
outage. The product tries again rather than failing the card."
|
|
1024
|
+
|
|
1025
|
+
`started.pid` IS THE WHOLE OF THE DISTINCTION, and it is a fact rather
|
|
1026
|
+
than a reading of a string. What reaches here as `{ ok: false }` is
|
|
1027
|
+
never a verdict about the work. An agent that ran and concluded it
|
|
1028
|
+
could not do the thing exits 0 having said so, and that is an ANSWER,
|
|
1029
|
+
written to the card below. So what is left to separate is a process that
|
|
1030
|
+
ran and died from one that never existed, and `Started.pid` answers it:
|
|
1031
|
+
`could not start claude on this machine: ENOENT` and `claude is not
|
|
1032
|
+
installed` are facts about this machine that will be identical in two
|
|
1033
|
+
seconds and are fixed by a person, while `claude exited 1: API Error:
|
|
1034
|
+
529 Overloaded` is what a rate limit, a dropped connection and a
|
|
1035
|
+
provider outage all look like from here. plan-slice-3.md section 1.2
|
|
1036
|
+
measured the alternative and rejected it: the provider's own words reach
|
|
1037
|
+
this daemon as a 500 character tail of one of two streams, absent
|
|
1038
|
+
entirely for a rate limit, and different again next release.
|
|
1039
|
+
|
|
1040
|
+
WHAT THIS DELIBERATELY GIVES UP is that a genuinely deterministic
|
|
1041
|
+
harness crash costs two extra spawns before the card fails. That is
|
|
1042
|
+
bounded, and it is the price of not depending on a string.
|
|
1043
|
+
|
|
1044
|
+
THE RESPAWN IS `resumeRun`, WHICH ALREADY EXISTS. ux.md: "two features
|
|
1045
|
+
owning one respawn is how two respawns are born." Nothing new starts a
|
|
1046
|
+
process here; the failure calls the sweep's own respawn, fenced by the
|
|
1047
|
+
pid this daemon watched exit, and AWAITS it, so the daemon's one
|
|
1048
|
+
`inFlight` entry covers the whole chain and `--once` waits for the last
|
|
1049
|
+
attempt rather than exiting mid-retry.
|
|
1050
|
+
|
|
1051
|
+
AND THE OLD RULING HERE IS AMENDED RATHER THAN DELETED. It read: "a
|
|
1052
|
+
process that started and then failed is a failure of the work, not of
|
|
1053
|
+
the machine: resuming it would put the same agent back on the same brief
|
|
1054
|
+
to fail the same way, FOREVER." The half that survives is forever, and
|
|
1055
|
+
the bound is what now provides it. */
|
|
1056
|
+
const attempts = await attemptsSoFar(client, runId);
|
|
1057
|
+
if (started.pid !== null && attempts < MAX_ATTEMPTS) {
|
|
1058
|
+
out(`retry run ${runId} attempt ${attempts} of ${MAX_ATTEMPTS} died: ${answer.reason}`);
|
|
1059
|
+
const again = processToken === undefined
|
|
1060
|
+
? await resumeRun(client, tools, machineId, runId, started.pid)
|
|
1061
|
+
: await activateOwner(client, tools, machineId, runId, processToken, started.pid);
|
|
1062
|
+
if (again)
|
|
1063
|
+
return again.settled;
|
|
1064
|
+
/* ═══ THE CLAIM MATCHED NOTHING, AND THAT IS NOT ALWAYS SOMEBODY ELSE
|
|
1065
|
+
HOLDING THIS RUN, SO THE FAILURE IS STILL WRITTEN. ═══
|
|
1066
|
+
Six predicates can refuse a retry and two of them mean NOTHING HAS
|
|
1067
|
+
MOVED AT ALL. `pid is null` is the ordinary one, and not exotic:
|
|
1068
|
+
`recordProcess` is caught and carried on from at all four spawn sites
|
|
1069
|
+
precisely because that write is expected to be able to fail, and when
|
|
1070
|
+
it does, the process still ran and still died with a reason nobody
|
|
1071
|
+
else has. `attempts` reaching the bound between the read above and the
|
|
1072
|
+
claim is the other. RETURNING HERE WOULD LEAVE THE RUN `running` WITH
|
|
1073
|
+
A DEAD PROCESS AND THE CARD STILL SAYING WORKING, and the sentence a
|
|
1074
|
+
person eventually read would be the next sweep's guess about a daemon
|
|
1075
|
+
that never recorded a pid rather than the harness's own words.
|
|
1076
|
+
|
|
1077
|
+
SO IT FALLS THROUGH, AND THE ENDING IS THE ONE THIS PATH HAD BEFORE
|
|
1078
|
+
THE RETRY EXISTED. A run that ended or stopped under this daemon is
|
|
1079
|
+
refused by `panel3_end_run` and `panel3_give_up` on their own
|
|
1080
|
+
predicates, and that refusal is said below rather than recorded. A run
|
|
1081
|
+
another machine has taken over is not: neither ending is fenced by the
|
|
1082
|
+
machine or the pid, so this daemon ends it, exactly as it did before
|
|
1083
|
+
this slice. That is a property of the ending rather than of the retry,
|
|
1084
|
+
it is narrower than it was (the claim is fenced, so no second PROCESS
|
|
1085
|
+
starts), and closing it is a change to what a give-up may write. */
|
|
1086
|
+
said(`run ${runId} could not be started again, so it is ending on: ${answer.reason}`);
|
|
1087
|
+
}
|
|
1088
|
+
/* ═══ AND WHEN THE BOUND IS SPENT THE CARD FAILS AND SAYS HOW MANY
|
|
1089
|
+
ATTEMPTS IT TOOK. ═══ ux.md asks for that in as many words. It is said
|
|
1090
|
+
once, here, into the `failed_because` column the panel and the terminal
|
|
1091
|
+
both already read, so the count reaches a person from one place rather
|
|
1092
|
+
than two. */
|
|
1093
|
+
const why = attempts > 1 ? `${answer.reason} (after ${attempts} attempts)` : answer.reason;
|
|
1094
|
+
const ended = processToken === undefined
|
|
1095
|
+
? await endRun(client, level, runId, cardId, why)
|
|
1096
|
+
: (await giveUp(client, runId, why, processToken)) !== null;
|
|
1097
|
+
if (!ended) {
|
|
1098
|
+
/* IT WAS ALREADY SETTLED, by recovery, which decided this process was
|
|
1099
|
+
gone before it said so itself, or by the person's Stop. Nothing was
|
|
1100
|
+
written here, and that is said rather than reported as a failure this
|
|
1101
|
+
daemon recorded. */
|
|
1102
|
+
said(`run ${runId} had already ended, so nothing was recorded: ${why}`);
|
|
1103
|
+
return;
|
|
1104
|
+
}
|
|
1105
|
+
out(level === 1
|
|
1106
|
+
? `failed card ${cardId} run ${runId} ${why}`
|
|
1107
|
+
: `failed run ${runId} ${why}`);
|
|
1108
|
+
return;
|
|
1109
|
+
}
|
|
1110
|
+
/* ═══ ITS OWN WORDS, ONTO THE CARD, THROUGH THE SAME STATEMENT EVERY LEVEL
|
|
1111
|
+
USES. ═══ Nothing reads them, nothing shortens them and nothing waits to
|
|
1112
|
+
approve them: ux.md's "whoever did the work writes the answer". */
|
|
1113
|
+
const accepted = await writeAnswer(client, runId, cardId, speaksToTheCard ? answer.text : null, processToken);
|
|
1114
|
+
if (accepted)
|
|
1115
|
+
await ownerSession?.established();
|
|
1116
|
+
else
|
|
1117
|
+
await ownerSession?.failed();
|
|
1118
|
+
});
|
|
1119
|
+
}
|
|
1120
|
+
/**
|
|
1121
|
+
* ═══ ONE RUN, STARTED AGAIN AS ITSELF, WITH WHAT IT WAS SENT AND WHAT IT HAD
|
|
1122
|
+
* GOT AS FAR AS. ═══
|
|
1123
|
+
*
|
|
1124
|
+
* ux.md: "a respawned agent reads three things: its brief, its own report, and
|
|
1125
|
+
* whatever is new", and all three arrive in the prompt because "an agent cannot
|
|
1126
|
+
* fail to read its own input".
|
|
1127
|
+
*
|
|
1128
|
+
* ═══ THE SAME ROW, WHICH IS THE PART THAT MATTERS. ═══ Its children point at
|
|
1129
|
+
* this id. A new row would be a run whose children belong to a dead parent, and
|
|
1130
|
+
* the agent would dispatch them again — the failure ux.md names as the single
|
|
1131
|
+
* most expensive one observed. Nothing is inserted here for the same reason
|
|
1132
|
+
* nothing needs to be: constraint 8 is satisfied by a row that already exists.
|
|
1133
|
+
*
|
|
1134
|
+
* ═══ TWO CALLERS, AND `afterPid` IS THE WHOLE OF WHAT SEPARATES THEM. ═══
|
|
1135
|
+
*
|
|
1136
|
+
* NULL is the sweep, which INFERS that a process is gone from a pid the
|
|
1137
|
+
* operating system no longer knows: `recoverStranded` and `takeHandedBack`. It
|
|
1138
|
+
* behaves exactly as it always has, level 1 refused and the claim window
|
|
1139
|
+
* enforced, and it does not spend an attempt, because a laptop that slept is not
|
|
1140
|
+
* a provider that failed.
|
|
1141
|
+
*
|
|
1142
|
+
* A PID is the retry, made by the daemon that WATCHED that process exit. It may
|
|
1143
|
+
* resume any level, it is fenced by the pid rather than by the window, and the
|
|
1144
|
+
* claim counts the attempt and refuses past three. See 20260909090000 for why
|
|
1145
|
+
* each of those three predicates is about the sweep and not about a failure this
|
|
1146
|
+
* daemon saw with its own eyes.
|
|
1147
|
+
*
|
|
1148
|
+
* recovery-1/ux.md is why there is one function and not two: "two features
|
|
1149
|
+
* owning one respawn is how two respawns are born."
|
|
1150
|
+
*
|
|
1151
|
+
* ═══ AND THE ORDER IS THE SAME ARGUMENT `startChild` MAKES, ON THE SWEEP'S
|
|
1152
|
+
* PATH. ═══ The working copy is resolved BEFORE the claim, so a machine that
|
|
1153
|
+
* has none refuses without having taken a run it cannot start; then the claim,
|
|
1154
|
+
* then the process, then the pid.
|
|
1155
|
+
*
|
|
1156
|
+
* ON THE RETRY'S PATH IT CANNOT BE, AND THAT IS `startRearmed`'s SITUATION
|
|
1157
|
+
* EXACTLY. A level 1 run works in an empty directory of the user's own and
|
|
1158
|
+
* anything deeper gets the working copy, and the level is not known until the
|
|
1159
|
+
* claim returns. So the resolution happens after the claim, and a failure there
|
|
1160
|
+
* ends the run with its reason rather than leaving a claimed run sitting
|
|
1161
|
+
* `running` with no process. The sweep's ordering is untouched, because its
|
|
1162
|
+
* argument still holds there.
|
|
1163
|
+
*
|
|
1164
|
+
* ═══ THE CHILDREN ARE READ AFTER THE CLAIM AND AS LATE AS POSSIBLE. ═══ They
|
|
1165
|
+
* are the volatile half: one of them may finish while this function is running,
|
|
1166
|
+
* and the prompt says out loud that the list is a moment old and is re-readable.
|
|
1167
|
+
* Carrying them in the report instead is the thing ux.md forbids outright.
|
|
1168
|
+
*
|
|
1169
|
+
* Returns null when the claim matched nothing, which is a fact about the record
|
|
1170
|
+
* rather than a failure: nothing was started here. On the sweep's path that
|
|
1171
|
+
* means another daemon got there first. On the retry's it means the same thing
|
|
1172
|
+
* said by the pid having moved, OR that this run has had its three processes and
|
|
1173
|
+
* the record refuses a fourth, and the caller is the one that knows what to say
|
|
1174
|
+
* about the failure it was holding.
|
|
1175
|
+
*/
|
|
1176
|
+
async function resumeRun(client, tools, machineId, runId, afterPid) {
|
|
1177
|
+
/* ═══ THE SWEEP RESOLVES ITS WORKING COPY BEFORE THE CLAIM AND THE RETRY
|
|
1178
|
+
CANNOT, AND `afterPid` IS THE WHOLE OF WHAT DECIDES IT. ═══ Said once, here.
|
|
1179
|
+
See the header for both halves of the argument: a machine with no checkout
|
|
1180
|
+
must not take a run it cannot start, and the level a retry needs the answer
|
|
1181
|
+
for is not known until the claim returns. */
|
|
1182
|
+
const sweptCwd = afterPid === null
|
|
1183
|
+
? checkoutForCodebase(await codebaseOfRun(client, runId), hostname())
|
|
1184
|
+
: null;
|
|
1185
|
+
const { data, error } = await client.rpc('panel3_resume', {
|
|
1186
|
+
p_run_id: runId,
|
|
1187
|
+
p_machine_id: machineId,
|
|
1188
|
+
p_after_pid: afterPid,
|
|
1189
|
+
});
|
|
1190
|
+
if (error)
|
|
1191
|
+
throw new Error(`could not start run ${runId} again: ${error.message}`);
|
|
1192
|
+
const claimed = data?.[0];
|
|
1193
|
+
if (!claimed)
|
|
1194
|
+
return null;
|
|
1195
|
+
const level = claimed.run_level === 1 ? 1 : claimed.run_level === 2 ? 2 : 3;
|
|
1196
|
+
if (claimed.run_level !== level) {
|
|
1197
|
+
/* UNREACHABLE, AND STILL SETTLED, exactly as in `startChild`: the level
|
|
1198
|
+
column is checked at three, so there is no level here this cannot spawn.
|
|
1199
|
+
The claim has already happened, so leaving it would strand the run for a
|
|
1200
|
+
whole grace window before anything looked at it again. */
|
|
1201
|
+
const why = `run ${runId} is at level ${claimed.run_level}, which cannot be spawned`;
|
|
1202
|
+
await giveUp(client, runId, why);
|
|
1203
|
+
throw new Error(why);
|
|
1204
|
+
}
|
|
1205
|
+
/* NON-NULL EXACTLY WHEN THE SWEEP RESOLVED ONE, which is the rule stated at
|
|
1206
|
+
the assignment above. Branching on the VALUE rather than testing `afterPid`
|
|
1207
|
+
a second time is what keeps the resolution and its use from being able to
|
|
1208
|
+
disagree: whatever the sweep resolved is what the sweep runs in. */
|
|
1209
|
+
let cwd;
|
|
1210
|
+
if (sweptCwd !== null) {
|
|
1211
|
+
cwd = sweptCwd;
|
|
1212
|
+
}
|
|
1213
|
+
else {
|
|
1214
|
+
try {
|
|
1215
|
+
cwd = await workingDirectory(client, runId, level, false);
|
|
1216
|
+
}
|
|
1217
|
+
catch (error) {
|
|
1218
|
+
/* ═══ THE REASON IS WRITTEN TO THE DATABASE, SO IT MAY NOT CARRY A PATH.
|
|
1219
|
+
═══ Constraint 6, and `startRearmed`'s own handling of the same two
|
|
1220
|
+
calls: `workingCopy()` is already careful about this and says so;
|
|
1221
|
+
`scratchDir()` is not, because it is `mkdirSync`, whose EACCES and
|
|
1222
|
+
ENOTDIR messages name the directory they failed on. So the machine's own
|
|
1223
|
+
error is kept for stderr and the record is told only what is true and
|
|
1224
|
+
shareable. */
|
|
1225
|
+
const stderrOnly = error instanceof Error ? error.message : String(error);
|
|
1226
|
+
const why = level === 1
|
|
1227
|
+
? 'this machine could not make the empty directory this runs in'
|
|
1228
|
+
: stderrOnly;
|
|
1229
|
+
/* ═══ AND IT ENDS THE WAY A RUN OF THIS LEVEL ENDS. ═══ It was
|
|
1230
|
+
`panel3_give_up` outright, which was right while only a dispatched run
|
|
1231
|
+
could reach this function and is a leak now that the retry brings level
|
|
1232
|
+
1 here: handing the person's message back mints a new run with its
|
|
1233
|
+
attempts at one, which is the bound the retry is under, undone by the
|
|
1234
|
+
one path that could not start. See `endRun`. */
|
|
1235
|
+
await endRun(client, level, runId, claimed.run_card_id, why);
|
|
1236
|
+
throw new Error(`NO AGENT IS RUNNING: ${level === 1 ? stderrOnly : why}`);
|
|
1237
|
+
}
|
|
1238
|
+
}
|
|
1239
|
+
/* WHAT IT SENT OTHERS TO DO, FROM THE RECORD. Level 3 has no `dispatch`, so it
|
|
1240
|
+
has no children to have: null says that, where an empty list would say it
|
|
1241
|
+
chose to send nobody.
|
|
1242
|
+
|
|
1243
|
+
═══ AND LEVEL 1 HAS THEM TOO, WHICH ONLY THE RETRY CAN REACH. ═══ This read
|
|
1244
|
+
used to be `level === 2`, which was correct only because nothing at level 1
|
|
1245
|
+
ever got here. Left alone it would hand a coordinator a null child list, and
|
|
1246
|
+
a coordinator that came back to no children dispatches its workers a second
|
|
1247
|
+
time: ux.md's single most expensive failure. `startRearmed` reads it the
|
|
1248
|
+
same way, for the same reason. */
|
|
1249
|
+
const children = level === 3 ? null : await childrenOf(client, runId);
|
|
1250
|
+
const started = startAgent(
|
|
1251
|
+
/* ═══ WHY IT DIED IS WHAT DIFFERS, AND IT IS TOLD THE TRUTH ABOUT IT. ═══
|
|
1252
|
+
`resumePrompt` opens by saying the machine went down, which is true of the
|
|
1253
|
+
sweep and false of a retry: the daemon that watched this harness exit is
|
|
1254
|
+
still running. See `retryPrompt`. */
|
|
1255
|
+
afterPid === null
|
|
1256
|
+
? resumePrompt(claimed.run_brief, claimed.run_report, children)
|
|
1257
|
+
: retryPrompt(claimed.run_brief, claimed.run_report, children), level, tools.urlFor(runId), cwd);
|
|
1258
|
+
if (started.pid === null) {
|
|
1259
|
+
/* THE CLAIM HAPPENED AND NO PROCESS DID, which is the one shape the record
|
|
1260
|
+
must never be left in quietly. Same handling as a dispatch that could not
|
|
1261
|
+
start: the reason is read off the settled answer and the run is ended with
|
|
1262
|
+
it. */
|
|
1263
|
+
const answer = await started.answered;
|
|
1264
|
+
const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
|
|
1265
|
+
// The same level fork, for the same reason. See `endRun`.
|
|
1266
|
+
await endRun(client, level, runId, claimed.run_card_id, reason);
|
|
1267
|
+
throw new Error(`NO AGENT IS RUNNING: ${reason}`);
|
|
1268
|
+
}
|
|
1269
|
+
out(`resume run ${runId} level ${level} pid ${started.pid} `
|
|
1270
|
+
+ `${claimed.run_report === null ? 'no report to carry' : 'carrying its report'}`);
|
|
1271
|
+
try {
|
|
1272
|
+
await recordProcess(client, runId, started.pid);
|
|
1273
|
+
}
|
|
1274
|
+
catch (error) {
|
|
1275
|
+
// Said, not fatal, as everywhere else: the agent is running and killing it
|
|
1276
|
+
// over a bookkeeping write would cost the user the work a second time.
|
|
1277
|
+
said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
|
|
1278
|
+
}
|
|
1279
|
+
return { settled: settle(client, tools, machineId, level, runId, claimed.run_card_id, started) };
|
|
1280
|
+
}
|
|
1281
|
+
/**
|
|
1282
|
+
* ═══ ONE RUN, STARTED AGAIN BECAUSE SOMETHING IT WAS WAITING ON EXISTS NOW. ═══
|
|
1283
|
+
*
|
|
1284
|
+
* The other half of `resumeRun`, and deliberately the same shape: same row, same
|
|
1285
|
+
* id, same brief, the report carried, the children read live. Two facts differ,
|
|
1286
|
+
* and both come off the row `panel3_take_rearms` handed over.
|
|
1287
|
+
*
|
|
1288
|
+
* WHAT IS NEW. The answer to this run's own question, a question from somebody
|
|
1289
|
+
* it sent that it has to answer or pass on, or everybody it sent having
|
|
1290
|
+
* finished. `ask_id` and `mine` say which, and they are the record's word
|
|
1291
|
+
* rather than an inference here.
|
|
1292
|
+
*
|
|
1293
|
+
* WHERE IT RUNS. A level 1 run is started in an empty directory of the user's
|
|
1294
|
+
* own, because level 1 has no code tool to use a real one with; anything deeper
|
|
1295
|
+
* gets the working copy. That is `answerCard` and `startChild`'s rule, and this
|
|
1296
|
+
* is the one path that can be handed either.
|
|
1297
|
+
*
|
|
1298
|
+
* ═══ THE CLAIM HAS ALREADY HAPPENED WHEN THIS IS CALLED, SO A FAILURE ENDS THE
|
|
1299
|
+
* RUN RATHER THAN LEAVING IT. ═══ `panel3_take_rearms` stamps the claim in the
|
|
1300
|
+
* same statement that makes it — the question delivered, and the run's end
|
|
1301
|
+
* cleared — which is what stops two daemons starting two processes for it and
|
|
1302
|
+
* what stops one question, or one set of finished workers, spawning forever. The cost is that this cannot decline afterwards: a run claimed and not
|
|
1303
|
+
* started would sit `running` with no process and its question already spent. So
|
|
1304
|
+
* every failure below ends the run with its reason, which is visible on the card
|
|
1305
|
+
* and in `cs3 show`, rather than quietly hoping the next poll finds it.
|
|
1306
|
+
*/
|
|
1307
|
+
async function startRearmed(client, tools, machineId, row) {
|
|
1308
|
+
const level = row.run_level === 1 ? 1 : row.run_level === 2 ? 2 : 3;
|
|
1309
|
+
if (row.run_level !== level) {
|
|
1310
|
+
const why = `run ${row.run_id} is at level ${row.run_level}, which cannot be spawned`;
|
|
1311
|
+
await giveUp(client, row.run_id, why);
|
|
1312
|
+
throw new Error(why);
|
|
1313
|
+
}
|
|
1314
|
+
let cwd;
|
|
1315
|
+
try {
|
|
1316
|
+
cwd = await workingDirectory(client, row.run_id, level, false);
|
|
1317
|
+
}
|
|
1318
|
+
catch (error) {
|
|
1319
|
+
/* ═══ THE REASON IS WRITTEN TO THE DATABASE, SO IT MAY NOT CARRY A PATH.
|
|
1320
|
+
═══ Constraint 6. `workingCopy()` is already careful about this and says
|
|
1321
|
+
so; `scratchDir()` is not — it is `mkdirSync`, whose EACCES and ENOTDIR
|
|
1322
|
+
messages name the directory they failed on — and `failed_because` is a
|
|
1323
|
+
column `cs3 show` prints. So the machine's own error is kept for stderr
|
|
1324
|
+
and the record is told only what is true and shareable. */
|
|
1325
|
+
const said = error instanceof Error ? error.message : String(error);
|
|
1326
|
+
const why = level === 1
|
|
1327
|
+
? 'this machine could not make the empty directory this runs in'
|
|
1328
|
+
: said;
|
|
1329
|
+
// The level fork, which this path needs for the same reason `resumeRun`'s
|
|
1330
|
+
// two do: a re-arm serves level 1, and a level 1 run holds the person's
|
|
1331
|
+
// message. See `endRun`.
|
|
1332
|
+
await endRun(client, level, row.run_id, row.run_card_id, why);
|
|
1333
|
+
throw new Error(`NO AGENT IS RUNNING: ${level === 1 ? said : why}`);
|
|
1334
|
+
}
|
|
1335
|
+
/* WHAT IT SENT OTHERS TO DO AND WHAT THEY WROTE, FROM THE RECORD, AS LATE AS
|
|
1336
|
+
POSSIBLE. Level 3 has no `dispatch`, so it has no children to have: null
|
|
1337
|
+
says that, where an empty list would say it chose to send nobody. */
|
|
1338
|
+
const children = level === 3 ? null : await childrenOf(client, row.run_id);
|
|
1339
|
+
const deliveredArtifact = row.ask_id !== null && row.mine
|
|
1340
|
+
? await rearmedArtifactAnswer(client, row.ask_id)
|
|
1341
|
+
: null;
|
|
1342
|
+
/* ═══ THREE REASONS, AND THE ROW SAYS WHICH. ═══ No question is ux.md's third
|
|
1343
|
+
re-arm: everybody it sent has finished, and it is started to read them back.
|
|
1344
|
+
`children` cannot be null on that path — only a run with children is ever
|
|
1345
|
+
claimed for it — and the prompt takes the list rather than the maybe-list so
|
|
1346
|
+
that is a fact of the signature rather than of a comment. */
|
|
1347
|
+
const prompt = row.ask_id === null
|
|
1348
|
+
? readBackPrompt(row.run_brief, row.run_report, children ?? [])
|
|
1349
|
+
: row.mine
|
|
1350
|
+
? answerPrompt(row.run_brief, row.run_report, children, row.question ?? '', row.answer ?? '', deliveredArtifact)
|
|
1351
|
+
: escalationPrompt(row.run_brief, row.run_report, children, row.ask_id, row.question ?? '');
|
|
1352
|
+
const started = startAgent(prompt, level, tools.urlFor(row.run_id), cwd);
|
|
1353
|
+
if (started.pid === null) {
|
|
1354
|
+
const answer = await started.answered;
|
|
1355
|
+
const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
|
|
1356
|
+
// The same level fork, for the same reason. See `endRun`.
|
|
1357
|
+
await endRun(client, level, row.run_id, row.run_card_id, reason);
|
|
1358
|
+
throw new Error(`NO AGENT IS RUNNING: ${reason}`);
|
|
1359
|
+
}
|
|
1360
|
+
out(`rearm run ${row.run_id} level ${level} pid ${started.pid} `
|
|
1361
|
+
+ `${row.ask_id === null
|
|
1362
|
+
? 'to read back everybody it sent'
|
|
1363
|
+
: row.mine
|
|
1364
|
+
? 'with the answer to its own question'
|
|
1365
|
+
: 'with a question it has to settle'}`);
|
|
1366
|
+
try {
|
|
1367
|
+
await recordProcess(client, row.run_id, started.pid);
|
|
1368
|
+
}
|
|
1369
|
+
catch (error) {
|
|
1370
|
+
// Said, not fatal, as everywhere else: the agent is running and killing it
|
|
1371
|
+
// over a bookkeeping write would cost the user the work.
|
|
1372
|
+
said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
|
|
1373
|
+
}
|
|
1374
|
+
/* ═══ AND IT SPEAKS TO THE CARD ONLY IF IT IS DOING THE PERSON'S WORK. ═══
|
|
1375
|
+
Exactly the distinction ux.md draws: a run carrying on with its own work has
|
|
1376
|
+
something to say when it finishes, and a run started only to settle somebody
|
|
1377
|
+
else's question does not. A run started to read back everybody it sent is
|
|
1378
|
+
the first kind and the clearest case of it — that reply IS the answer to the
|
|
1379
|
+
request. See `settle`. */
|
|
1380
|
+
return {
|
|
1381
|
+
settled: settle(client, tools, machineId, level, row.run_id, row.run_card_id, started, row.ask_id === null || !!row.mine),
|
|
1382
|
+
};
|
|
1383
|
+
}
|
|
1384
|
+
/**
|
|
1385
|
+
* The runs one run dispatched, AND WHAT EACH OF THEM WROTE, in the words the
|
|
1386
|
+
* prompt prints them in. The same question `list_child_runs` and
|
|
1387
|
+
* `get_child_report` answer, asked here so the answer is IN the prompt rather
|
|
1388
|
+
* than waiting for the agent to think of asking.
|
|
1389
|
+
*
|
|
1390
|
+
* ═══ THE REPORTS ARE HERE BECAUSE OF WHAT THIS IS FOR. ═══ ux.md: "the level 2
|
|
1391
|
+
* agent does not hold its three workers' findings in its head while they run.
|
|
1392
|
+
* Each worker writes what it found into the record, and the level 2 agent reads
|
|
1393
|
+
* them back." The respawn that exists to do exactly that is fired by the record
|
|
1394
|
+
* and given the record; leaving the reports behind a tool call would make the
|
|
1395
|
+
* whole synthesis conditional on an agent choosing to make it, which is the
|
|
1396
|
+
* difference ux.md draws between a guarantee and a hope.
|
|
1397
|
+
*
|
|
1398
|
+
* AND THEY ARE NOT TRUNCATED. A report is rewritten state rather than an append
|
|
1399
|
+
* log, which is what keeps it bounded; cutting one here would hand the agent a
|
|
1400
|
+
* half-sentence and no way to tell it was cut.
|
|
1401
|
+
*/
|
|
1402
|
+
async function childrenOf(client, runId) {
|
|
1403
|
+
const children = await returned(client
|
|
1404
|
+
.from('panel3_runs')
|
|
1405
|
+
.select('id, level, state, activity, report, failed_because, ended_at')
|
|
1406
|
+
.eq('parent_run_id', runId)
|
|
1407
|
+
.order('started_at'), 'read', `what run ${runId} sent others to do`);
|
|
1408
|
+
return children.flatMap((c) => [
|
|
1409
|
+
[
|
|
1410
|
+
c.id,
|
|
1411
|
+
`L${c.level}`,
|
|
1412
|
+
c.state === 'running' && !c.ended_at ? 'still going' : c.state,
|
|
1413
|
+
c.activity ?? '',
|
|
1414
|
+
].join(' ').trimEnd(),
|
|
1415
|
+
/* ONE OWNER PER COLUMN, PRINTED AS TWO FACTS. `report` is what the run
|
|
1416
|
+
wrote and `failed_because` is why the product ended it, and running them
|
|
1417
|
+
together would have an agent read the daemon's sentence as its worker's
|
|
1418
|
+
finding. */
|
|
1419
|
+
...(c.failed_because === null ? [] : [` it was ended: ${c.failed_because}`]),
|
|
1420
|
+
...(c.report === null
|
|
1421
|
+
? [' it wrote nothing down']
|
|
1422
|
+
: c.report.split('\n').map((l) => ` ${l}`)),
|
|
1423
|
+
]);
|
|
1424
|
+
}
|
|
1425
|
+
/** Private owner context: enough to continue or stop work, without hierarchy or activity text. */
|
|
1426
|
+
async function ownerChildren(client, runId) {
|
|
1427
|
+
const children = await returned(client.from('panel3_runs').select('id, state, report, failed_because')
|
|
1428
|
+
.eq('parent_run_id', runId).order('started_at'), 'read', `what owner ${runId} sent others to do`);
|
|
1429
|
+
return children.map((child) => [
|
|
1430
|
+
`${child.id} ${child.state}`,
|
|
1431
|
+
...(child.failed_because ? [`ended: ${child.failed_because}`] : []),
|
|
1432
|
+
child.report ?? 'no report',
|
|
1433
|
+
].join('\n'));
|
|
1434
|
+
}
|
|
1435
|
+
async function ownerConversation(client, cardId, leased, initialLeaseRunId) {
|
|
1436
|
+
const [turns, rawAsks] = await Promise.all([
|
|
1437
|
+
returned(client.from('panel3_turns').select('id, role, body, created_at, lease_id')
|
|
1438
|
+
.eq('card_id', cardId).order('created_at'), 'read', `the visible conversation on card ${cardId}`),
|
|
1439
|
+
returned(client.from('panel3_asks').select(`id, created_at, ${ASK_CONTENT_COLUMNS}`)
|
|
1440
|
+
.eq('card_id', cardId).not('decision_id', 'is', null).order('created_at'), 'read', `the visible questions on card ${cardId}`),
|
|
1441
|
+
]);
|
|
1442
|
+
const asks = await withAskContent(client, rawAsks);
|
|
1443
|
+
const events = [
|
|
1444
|
+
...turns.map((turn) => ({
|
|
1445
|
+
at: turn.created_at,
|
|
1446
|
+
kind: turn.role,
|
|
1447
|
+
id: turn.id,
|
|
1448
|
+
body: turn.body,
|
|
1449
|
+
needsReply: turn.role === 'user'
|
|
1450
|
+
&& (leased.has(turn.id) || turn.lease_id === initialLeaseRunId),
|
|
1451
|
+
})),
|
|
1452
|
+
...asks.map((ask) => ({
|
|
1453
|
+
at: ask.created_at,
|
|
1454
|
+
kind: 'question',
|
|
1455
|
+
id: ask.id,
|
|
1456
|
+
body: ask.question ?? '(question unavailable)',
|
|
1457
|
+
answer: ask.answer,
|
|
1458
|
+
relatedArtifactId: ask.related_artifact_id,
|
|
1459
|
+
relatedArtifactRevision: ask.related_artifact_revision,
|
|
1460
|
+
artifactAnswer: artifactAnswer(ask.related_artifact_id, ask.related_artifact_revision, ask.selected_options)?.answer ?? null,
|
|
1461
|
+
})),
|
|
1462
|
+
];
|
|
1463
|
+
return events.sort((a, b) => a.at.localeCompare(b.at) || a.id.localeCompare(b.id));
|
|
1464
|
+
}
|
|
1465
|
+
/** The first owner gets the same durable context as every later activation. */
|
|
1466
|
+
export async function initialOwnerPrompt(client, cardId, launcherRunId, brief) {
|
|
1467
|
+
const events = await ownerConversation(client, cardId, new Set(), launcherRunId);
|
|
1468
|
+
return ownerActivationPrompt(brief, null, events, [], null);
|
|
1469
|
+
}
|
|
1470
|
+
/** Only facts delivered by this activation enter an already-resumed native
|
|
1471
|
+
* conversation. Older visible turns are already in that conversation. */
|
|
1472
|
+
export function ownerContinuationPrompt(brief, report, events, children, delivered) {
|
|
1473
|
+
const messages = events.filter((event) => event.kind === 'user' && event.needsReply);
|
|
1474
|
+
return [
|
|
1475
|
+
'CONTINUE THE CTRL+SPC CONVERSATION YOU ALREADY OWN.',
|
|
1476
|
+
'Use only the new facts below as the new turn. Keep coordinating the whole card and speak as CTRL+SPC.',
|
|
1477
|
+
'',
|
|
1478
|
+
'NEW USER MESSAGES',
|
|
1479
|
+
...(messages.length === 0
|
|
1480
|
+
? ['None.']
|
|
1481
|
+
: messages.flatMap((event) => [`Turn ${event.id}`, event.body])),
|
|
1482
|
+
...(delivered === null ? [] : delivered.mine ? [
|
|
1483
|
+
'',
|
|
1484
|
+
'ANSWER TO YOUR QUESTION',
|
|
1485
|
+
`Question ${delivered.id}: ${delivered.question}`,
|
|
1486
|
+
`Answer: ${delivered.answer ?? '(no answer text)'}`,
|
|
1487
|
+
...presentedArtifactAnswerContext(delivered.artifactAnswer),
|
|
1488
|
+
] : [
|
|
1489
|
+
'',
|
|
1490
|
+
'A WORKER NEEDS YOUR DECISION',
|
|
1491
|
+
`Question ${delivered.id}: ${delivered.question}`,
|
|
1492
|
+
'Answer it with answer_escalation if you can. If the person must decide, ask them directly.',
|
|
1493
|
+
]),
|
|
1494
|
+
'',
|
|
1495
|
+
'CURRENT PRIVATE WORKER SNAPSHOT — THIS REPLACES THE PREVIOUS SNAPSHOT',
|
|
1496
|
+
...(children.length === 0 ? ['You have sent nobody.'] : children),
|
|
1497
|
+
'',
|
|
1498
|
+
'YOUR CURRENT DURABLE REPORT',
|
|
1499
|
+
report ?? 'Nothing yet.',
|
|
1500
|
+
'',
|
|
1501
|
+
'YOUR IMMUTABLE RESPONSIBILITY',
|
|
1502
|
+
brief,
|
|
1503
|
+
].join('\n');
|
|
1504
|
+
}
|
|
1505
|
+
async function ownerCandidate(client, runId) {
|
|
1506
|
+
const candidates = await returned(client.from('panel3_runs')
|
|
1507
|
+
.select('id, card_id, codebase_id, machine_id, harness, state, ended_at, attempts, pid, started_at, resumed_at, process_token, handed_back_at, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id, state)')
|
|
1508
|
+
.eq('id', runId), 'read', `conversation owner ${runId}`);
|
|
1509
|
+
const candidate = candidates[0];
|
|
1510
|
+
return candidate?.card?.conversation_run_id === candidate?.id ? candidate : null;
|
|
1511
|
+
}
|
|
1512
|
+
async function ownerDirectory(client, candidate) {
|
|
1513
|
+
return workingDirectory(client, candidate.id, 2, true, candidate.codebase_id === null ? null : undefined);
|
|
1514
|
+
}
|
|
1515
|
+
export function resumableOwnerSessionId(candidate, machineId, machineHarness) {
|
|
1516
|
+
const local = readOwnerSession(candidate.id);
|
|
1517
|
+
return candidate.machine_id === machineId
|
|
1518
|
+
&& candidate.harness === machineHarness
|
|
1519
|
+
&& local?.state === 'established'
|
|
1520
|
+
&& local.harness === machineHarness
|
|
1521
|
+
&& local.processToken === candidate.process_token
|
|
1522
|
+
? local.nativeSessionId
|
|
1523
|
+
: undefined;
|
|
1524
|
+
}
|
|
1525
|
+
async function activateOwner(client, tools, machineId, runId, afterProcessToken = null, afterPid = null) {
|
|
1526
|
+
const candidate = await ownerCandidate(client, runId);
|
|
1527
|
+
if (!candidate)
|
|
1528
|
+
return null;
|
|
1529
|
+
const machineHarness = harness();
|
|
1530
|
+
const resumeSessionId = resumableOwnerSessionId(candidate, machineId, machineHarness);
|
|
1531
|
+
const cwd = await ownerDirectory(client, candidate);
|
|
1532
|
+
const { data, error } = await client.rpc('panel3_take_owner_activation', {
|
|
1533
|
+
p_run_id: runId,
|
|
1534
|
+
p_machine_id: machineId,
|
|
1535
|
+
p_agent: machineHarness,
|
|
1536
|
+
p_after_process_token: afterProcessToken,
|
|
1537
|
+
p_after_pid: afterPid,
|
|
1538
|
+
});
|
|
1539
|
+
if (error)
|
|
1540
|
+
throw new Error(`could not activate conversation owner ${runId}: ${error.message}`);
|
|
1541
|
+
const claimed = data?.[0];
|
|
1542
|
+
if (!claimed)
|
|
1543
|
+
return null;
|
|
1544
|
+
const [events, children] = await Promise.all([
|
|
1545
|
+
ownerConversation(client, claimed.run_card_id, new Set(claimed.turn_ids ?? [])),
|
|
1546
|
+
ownerChildren(client, runId),
|
|
1547
|
+
]);
|
|
1548
|
+
const currentArtifactAnswer = deliveredArtifactAnswer(events, claimed);
|
|
1549
|
+
const delivered = claimed.ask_id === null ? null : {
|
|
1550
|
+
id: claimed.ask_id,
|
|
1551
|
+
question: claimed.question ?? '(question unavailable)',
|
|
1552
|
+
answer: claimed.answer,
|
|
1553
|
+
mine: claimed.mine === true,
|
|
1554
|
+
artifactAnswer: currentArtifactAnswer,
|
|
1555
|
+
};
|
|
1556
|
+
const prompt = resumeSessionId
|
|
1557
|
+
? ownerContinuationPrompt(claimed.run_brief, claimed.run_report, events, children, delivered)
|
|
1558
|
+
: ownerActivationPrompt(claimed.run_brief, claimed.run_report, events, children, delivered !== null && !delivered.mine
|
|
1559
|
+
? { id: delivered.id, question: delivered.question }
|
|
1560
|
+
: null, currentArtifactAnswer,
|
|
1561
|
+
/* ═══ A PROCESS OF ITS OWN ENDED BEFORE IT FINISHED. ═══ `afterPid` is the
|
|
1562
|
+
fact, and it is non-null on all three paths that follow one: a harness
|
|
1563
|
+
that crashed, a machine that went down, and now a person's correction.
|
|
1564
|
+
The sentences it adds say what to do and never why, because those three
|
|
1565
|
+
are not the same event and `prompt.ts` exists to stop an agent being
|
|
1566
|
+
told an untrue reason for its own restart. */
|
|
1567
|
+
afterPid !== null);
|
|
1568
|
+
const started = startAgent(prompt, 2, tools.urlFor(runId, claimed.process_token), cwd, { ownerId: runId, ...(resumeSessionId ? { resumeSessionId } : {}) });
|
|
1569
|
+
if (started.pid === null) {
|
|
1570
|
+
const answer = await started.answered;
|
|
1571
|
+
const reason = answer.ok ? 'the process ended before it could be identified' : answer.reason;
|
|
1572
|
+
await giveUp(client, runId, reason, claimed.process_token);
|
|
1573
|
+
throw new Error(`NO AGENT IS RUNNING: ${reason}`);
|
|
1574
|
+
}
|
|
1575
|
+
try {
|
|
1576
|
+
await recordProcess(client, runId, started.pid, undefined, claimed.process_token);
|
|
1577
|
+
}
|
|
1578
|
+
catch (error) {
|
|
1579
|
+
said(`${error instanceof Error ? error.message : String(error)} (the run is still going)`);
|
|
1580
|
+
}
|
|
1581
|
+
return {
|
|
1582
|
+
settled: settle(client, tools, machineId, 2, runId, claimed.run_card_id, started, true, claimed.process_token, ownerSessionLifecycle(started, runId, machineHarness, claimed.process_token, resumeSessionId)),
|
|
1583
|
+
};
|
|
1584
|
+
}
|
|
1585
|
+
/** The only owner rows the normal poll may try, including an explicit hand-back move. */
|
|
1586
|
+
export function ownerPollClaim(candidate) {
|
|
1587
|
+
if (candidate.state === 'running') {
|
|
1588
|
+
if (candidate.handed_back_at === null || candidate.process_token === null)
|
|
1589
|
+
return null;
|
|
1590
|
+
return { afterProcessToken: candidate.process_token, afterPid: null };
|
|
1591
|
+
}
|
|
1592
|
+
if (candidate.state === 'finished' || candidate.state === 'asked') {
|
|
1593
|
+
return { afterProcessToken: null, afterPid: null };
|
|
1594
|
+
}
|
|
1595
|
+
return null;
|
|
1596
|
+
}
|
|
1597
|
+
/**
|
|
1598
|
+
* ═══ THE PROCESS A PERSON'S CORRECTION CANNOT HAVE REACHED. ═══
|
|
1599
|
+
*
|
|
1600
|
+
* redirect-5/ux.md: stop ends work, and changing its direction mid-flight did
|
|
1601
|
+
* not exist. ux.md's mechanism is "redirect is stop plus respawn", and the
|
|
1602
|
+
* respawn half is already built: a process that dies is started again as itself
|
|
1603
|
+
* by `settle`, and the claim it goes through leases every unaddressed person
|
|
1604
|
+
* turn on its way. **So the only thing missing was the ending**, and this is it.
|
|
1605
|
+
*
|
|
1606
|
+
* ═══ IT IS A MIRROR OF THE RETRY CLAIM, AND MUST BE CHANGED WITH IT. ═══
|
|
1607
|
+
* `20260914090000`'s retry branch requires the token, `running`, `ended_at` null,
|
|
1608
|
+
* the pid, and `attempts < 3`. The kill is the ONE irreversible act in this file:
|
|
1609
|
+
* every other predicate here pre-filters an RPC that can refuse harmlessly, and
|
|
1610
|
+
* a claim refused AFTER a kill falls through `settle` to `giveUp`, which turns a
|
|
1611
|
+
* person's correction into a FAILED CARD. So every condition below is either
|
|
1612
|
+
* that mirror, or a product rule with a named cost.
|
|
1613
|
+
*
|
|
1614
|
+
* Two things are NOT here, because neither is a property of the row: whether
|
|
1615
|
+
* this run owns its card, and whether this daemon holds its process. Both are
|
|
1616
|
+
* the caller's, and `mine` is what makes the respawn possible at all.
|
|
1617
|
+
*/
|
|
1618
|
+
export function redirectedProcess(candidate, machineId,
|
|
1619
|
+
/** The newest unaddressed person turn per card. MAX, never first-seen: a turn
|
|
1620
|
+
* left unaddressed while a question was open would otherwise pin its card
|
|
1621
|
+
* below `resumed_at` for good and nothing on it could ever redirect. */
|
|
1622
|
+
waiting, cardId, booted) {
|
|
1623
|
+
// The claim's own five, mirrored.
|
|
1624
|
+
if (candidate.state !== 'running' || candidate.ended_at !== null)
|
|
1625
|
+
return null;
|
|
1626
|
+
if (candidate.attempts >= MAX_ATTEMPTS)
|
|
1627
|
+
return null;
|
|
1628
|
+
// The person moved this card to another machine. Killing here makes `settle`
|
|
1629
|
+
// retry locally and the claim clears `handed_back_at`, silently reversing them.
|
|
1630
|
+
if (candidate.handed_back_at !== null)
|
|
1631
|
+
return null;
|
|
1632
|
+
// `mine` is not enough: a takeover moves the row while this daemon still holds
|
|
1633
|
+
// a promise for it, and the claim then refuses on the machine predicate.
|
|
1634
|
+
if (candidate.machine_id !== machineId)
|
|
1635
|
+
return null;
|
|
1636
|
+
/* ═══ A CARD THAT NEEDS THE PERSON IS NOT A CARD THEY ARE REDIRECTING. ═══
|
|
1637
|
+
The claim refuses to lease turns while a person's question is open, so a
|
|
1638
|
+
kill here would destroy the work and NOT deliver the correction. The card's
|
|
1639
|
+
state is written from a character-identical predicate by the
|
|
1640
|
+
`panel3_prepare_user_turn` trigger on every person message, so it is the
|
|
1641
|
+
same fact, one read earlier. */
|
|
1642
|
+
if (candidate.card_state !== 'working')
|
|
1643
|
+
return null;
|
|
1644
|
+
/* ═══ AND IT HAS TO BE A MESSAGE THIS PROCESS CANNOT HAVE SEEN. ═══ The rule
|
|
1645
|
+
`panel3_take_turns` already applies one level up. Measured against the
|
|
1646
|
+
attempt that is running NOW, which is also what stops a replacement being
|
|
1647
|
+
killed by the very message that caused it. */
|
|
1648
|
+
const said = waiting.get(cardId);
|
|
1649
|
+
if (said === undefined)
|
|
1650
|
+
return null;
|
|
1651
|
+
if (new Date(said).getTime() <= new Date(candidate.resumed_at ?? candidate.started_at).getTime()) {
|
|
1652
|
+
return null;
|
|
1653
|
+
}
|
|
1654
|
+
return runProcessIsAlive(candidate, booted) ? candidate.pid : null;
|
|
1655
|
+
}
|
|
1656
|
+
export function pendingOwnerSessionWithinGrace(candidate, now = Date.now()) {
|
|
1657
|
+
const local = readOwnerSession(candidate.id);
|
|
1658
|
+
return !!local
|
|
1659
|
+
&& local.state === 'pending'
|
|
1660
|
+
&& local.harness === candidate.harness
|
|
1661
|
+
&& local.processToken === candidate.process_token
|
|
1662
|
+
&& now - new Date(local.updatedAt).getTime() < OWNER_SESSION_GRACE_MS;
|
|
1663
|
+
}
|
|
1664
|
+
async function takeOwnerActivations(client, tools, machineId, mine, hold) {
|
|
1665
|
+
/* ═══ THE MESSAGES NOBODY HAS TAKEN, READ BEFORE THE RUNS, DELIBERATELY. ═══
|
|
1666
|
+
A claim landing between these two reads stamps a `resumed_at` NEWER than
|
|
1667
|
+
every turn it just leased, so the redirect below refuses and nothing is
|
|
1668
|
+
killed. In the other order the same claim leaves a stale `resumed_at` beside
|
|
1669
|
+
a turn that now reads unaddressed, and the daemon kills the replacement it
|
|
1670
|
+
has just started. One narrow indexed read (`panel3_turns_takeable_idx`),
|
|
1671
|
+
unconditional because that ordering is the whole point of it. */
|
|
1672
|
+
const outstanding = await returned(client.from('panel3_turns')
|
|
1673
|
+
.select('card_id, created_at')
|
|
1674
|
+
.eq('role', 'user')
|
|
1675
|
+
.is('addressed_at', null), 'read', 'the messages nobody has taken yet');
|
|
1676
|
+
const waiting = new Map();
|
|
1677
|
+
for (const turn of outstanding) {
|
|
1678
|
+
// THE NEWEST PER CARD. See `redirectedProcess`: first-wins would pin a card
|
|
1679
|
+
// below its own `resumed_at` for good.
|
|
1680
|
+
const held = waiting.get(turn.card_id);
|
|
1681
|
+
if (held === undefined || turn.created_at > held)
|
|
1682
|
+
waiting.set(turn.card_id, turn.created_at);
|
|
1683
|
+
}
|
|
1684
|
+
const candidates = await returned(client.from('panel3_runs')
|
|
1685
|
+
.select('id, card_id, codebase_id, machine_id, harness, state, ended_at, attempts, pid, started_at, resumed_at, process_token, handed_back_at, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id, state)')
|
|
1686
|
+
.in('state', ['finished', 'asked', 'running']), 'read', 'conversation owners that may be ready');
|
|
1687
|
+
for (const candidate of candidates) {
|
|
1688
|
+
if (candidate.card?.conversation_run_id !== candidate.id)
|
|
1689
|
+
continue;
|
|
1690
|
+
/* ═══ A RUN THIS DAEMON IS HOLDING IS THE ONLY ONE A CORRECTION CAN
|
|
1691
|
+
REDIRECT. ═══ Not because of the claim, which is not made here at all, but
|
|
1692
|
+
because of what happens AFTER the kill: the respawn is the dying process's
|
|
1693
|
+
own `settle`, and only the daemon that started it has one. Kill anything
|
|
1694
|
+
else and there is a dead process, a row still reading `running` and a card
|
|
1695
|
+
still saying `working`, with nothing local to retry it. */
|
|
1696
|
+
if (mine.has(candidate.id)) {
|
|
1697
|
+
const pid = redirectedProcess({ ...candidate, card_state: candidate.card?.state ?? null }, machineId, waiting, candidate.card_id, bootedAt());
|
|
1698
|
+
if (pid !== null) {
|
|
1699
|
+
killTree({ pid, kill: (signal) => process.kill(pid, signal) });
|
|
1700
|
+
/* ═══ SIGNALLED, NOT KILLED, AND THE WORD IS THE POINT. ═══ `killTree`
|
|
1701
|
+
swallows a refused signal on both platforms, so saying "killed" would
|
|
1702
|
+
claim a death this daemon never observed. A process that survives is
|
|
1703
|
+
still running, still held, and its turn is still unaddressed and still
|
|
1704
|
+
newer than `resumed_at`, so the next poll signals it again — the same
|
|
1705
|
+
rule `killStopped` states two hundred lines below. */
|
|
1706
|
+
out(`redirect card ${candidate.card_id} run ${candidate.id} signalled pid ${pid}`);
|
|
1707
|
+
}
|
|
1708
|
+
continue;
|
|
1709
|
+
}
|
|
1710
|
+
if (pendingOwnerSessionWithinGrace(candidate))
|
|
1711
|
+
continue;
|
|
1712
|
+
if (candidate.handed_back_at !== null
|
|
1713
|
+
&& candidate.machine_id === machineId
|
|
1714
|
+
&& (runProcessIsAlive(candidate, bootedAt())
|
|
1715
|
+
|| (candidate.pid === null
|
|
1716
|
+
&& Date.now() - new Date(candidate.resumed_at ?? candidate.started_at).getTime() < PID_GRACE_MS)))
|
|
1717
|
+
continue;
|
|
1718
|
+
const claim = ownerPollClaim(candidate);
|
|
1719
|
+
if (!claim)
|
|
1720
|
+
continue;
|
|
1721
|
+
try {
|
|
1722
|
+
const activated = await activateOwner(client, tools, machineId, candidate.id, claim.afterProcessToken, claim.afterPid);
|
|
1723
|
+
if (activated)
|
|
1724
|
+
hold(candidate.id, activated.settled);
|
|
1725
|
+
}
|
|
1726
|
+
catch (error) {
|
|
1727
|
+
said(`could not activate owner ${candidate.id}: ${error instanceof Error ? error.message : String(error)}`);
|
|
1728
|
+
}
|
|
1729
|
+
}
|
|
1730
|
+
}
|
|
1731
|
+
// ---------------------------------------------------------------------------
|
|
1732
|
+
/**
|
|
1733
|
+
* ═══ THE TWO ENDS THE PERSON CAUSES, WHICH ARE THE TWO THIS DAEMON KILLS. ═══
|
|
1734
|
+
*
|
|
1735
|
+
* `stopped` is their Stop and `superseded` is the coordinator their next message
|
|
1736
|
+
* replaced. The record keeps them apart on purpose, because a stop is read
|
|
1737
|
+
* across the whole card and being replaced says nothing about anybody else's
|
|
1738
|
+
* work. HERE they are one thing and only one: a process nothing will ever read
|
|
1739
|
+
* from again.
|
|
1740
|
+
*/
|
|
1741
|
+
const ENDED_BY_THE_PERSON = ['stopped', 'superseded'];
|
|
1742
|
+
/** When this machine came up. Anything that started before it did has no process
|
|
1743
|
+
* left, whatever a pid lookup says about that number today. */
|
|
1744
|
+
const bootedAt = () => Date.now() - uptime() * 1000;
|
|
1745
|
+
/**
|
|
1746
|
+
* ═══ IS THE PROCESS THIS RUN RECORDED STILL THERE. ═══
|
|
1747
|
+
*
|
|
1748
|
+
* `processIsAlive` is `kill(pid, 0)`, which answers "some process has this
|
|
1749
|
+
* number", not "the process I started has this number". `machine_id` is derived
|
|
1750
|
+
* from the hardware and survives a reboot, pids restart low, so after a restart
|
|
1751
|
+
* a run's recorded pid regularly matches something unrelated and live. A run
|
|
1752
|
+
* that started before this machine booted has no process left by definition,
|
|
1753
|
+
* whatever the number says.
|
|
1754
|
+
*
|
|
1755
|
+
* ONE ANSWER FOR BOTH READERS. Recovery asks it to decide whether to reap a run,
|
|
1756
|
+
* and the stop asks it to decide whether there is anything left to kill. The
|
|
1757
|
+
* second is the one where a wrong yes is worst, because it signals a pid this
|
|
1758
|
+
* machine no longer owns.
|
|
1759
|
+
*/
|
|
1760
|
+
function runProcessIsAlive(run, booted) {
|
|
1761
|
+
if (run.pid === null)
|
|
1762
|
+
return false;
|
|
1763
|
+
// The attempt now running began at the resume, if there was one.
|
|
1764
|
+
if (new Date(run.resumed_at ?? run.started_at).getTime() < booted)
|
|
1765
|
+
return false;
|
|
1766
|
+
return processIsAlive(run.pid);
|
|
1767
|
+
}
|
|
1768
|
+
/** Reconcile only CTRL+SPC-owned local session state. Native Claude and Windows
|
|
1769
|
+
* Codex transcripts live in harness-owned locations and are deliberately not
|
|
1770
|
+
* touched here. A current owner row, live PID, in-flight process, or fresh
|
|
1771
|
+
* pid-null claim always defers cleanup. Stable state is removed only after the
|
|
1772
|
+
* owner row disappears, becomes terminal, or no longer owns its card. */
|
|
1773
|
+
export async function reconcileOwnerSessions(client, machineId, _machineHarness, inFlightOwnerIds = new Set()) {
|
|
1774
|
+
const mappingIds = listOwnerSessionIds();
|
|
1775
|
+
const homeIds = listPanel3CodexOwnerHomeIds();
|
|
1776
|
+
const all = [...new Set([...mappingIds, ...homeIds])];
|
|
1777
|
+
const invalid = all.filter((id) => !validSessionUuid(id));
|
|
1778
|
+
for (const id of invalid)
|
|
1779
|
+
removePanel3CodexOwnerHome(id);
|
|
1780
|
+
const ids = all.filter(validSessionUuid);
|
|
1781
|
+
if (ids.length === 0)
|
|
1782
|
+
return;
|
|
1783
|
+
const rows = await returned(client.from('panel3_runs')
|
|
1784
|
+
.select('id, machine_id, harness, state, pid, started_at, resumed_at, ended_at, process_token, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id)')
|
|
1785
|
+
.in('id', ids), 'read', 'which local conversation sessions still have an owner');
|
|
1786
|
+
const byId = new Map(rows.map((row) => [row.id, row]));
|
|
1787
|
+
for (const id of ids) {
|
|
1788
|
+
if (inFlightOwnerIds.has(id))
|
|
1789
|
+
continue;
|
|
1790
|
+
const row = byId.get(id);
|
|
1791
|
+
const localProcessInUse = !!row
|
|
1792
|
+
&& row.machine_id === machineId
|
|
1793
|
+
&& ((row.pid !== null && processIsAlive(row.pid))
|
|
1794
|
+
|| (row.state === 'running'
|
|
1795
|
+
&& row.ended_at === null
|
|
1796
|
+
&& row.pid === null
|
|
1797
|
+
&& Date.now() - new Date(row.resumed_at ?? row.started_at).getTime() < PID_GRACE_MS));
|
|
1798
|
+
const rowStillOwnsConversation = !!row
|
|
1799
|
+
&& row.card?.conversation_run_id === id
|
|
1800
|
+
&& ['running', 'asked', 'finished'].includes(row.state);
|
|
1801
|
+
if (localProcessInUse || rowStillOwnsConversation)
|
|
1802
|
+
continue;
|
|
1803
|
+
removeOwnerSession(id);
|
|
1804
|
+
if (homeIds.includes(id))
|
|
1805
|
+
removePanel3CodexOwnerHome(id);
|
|
1806
|
+
}
|
|
1807
|
+
}
|
|
1808
|
+
/**
|
|
1809
|
+
* ═══ THE USER STOPPED IT, OR THEIR NEXT MESSAGE REPLACED IT, SO THE PROCESS
|
|
1810
|
+
* STOPS. ═══
|
|
1811
|
+
*
|
|
1812
|
+
* ux.md: "The user's Stop is absolute, and nothing else stops work."
|
|
1813
|
+
* `panel3_stop_card` is the whole of the decision, ending the runs and the card
|
|
1814
|
+
* in one statement, and this is the effect.
|
|
1815
|
+
*
|
|
1816
|
+
* ═══ AND `superseded` IS READ OFF THE SAME QUERY, NOT SWEPT SEPARATELY. ═══ A
|
|
1817
|
+
* coordinator the person's next message replaced has a process that will never
|
|
1818
|
+
* be read from again, exactly like a stopped one, and that is the ONE thing the
|
|
1819
|
+
* two states share here. Two sweeps would be two owners of "kill what is not
|
|
1820
|
+
* coming back", and the second would be the one that fell behind. THE RECORD IS THE INTENT AND THE
|
|
1821
|
+
* KILL IS THE EFFECT, in that order, so a machine that is offline when the
|
|
1822
|
+
* button is pressed still honours it on the poll after it comes back.
|
|
1823
|
+
*
|
|
1824
|
+
* ONLY THIS MACHINE'S RUNS, for the same reason recovery only sweeps its own: a
|
|
1825
|
+
* pid is a local fact, and no other machine can act on this one's.
|
|
1826
|
+
*
|
|
1827
|
+
* ═══ A PROCESS THAT IS ALREADY GONE IS NOT AN ERROR. ═══ It is the ordinary
|
|
1828
|
+
* case, being an agent that finished in the second before the button or a
|
|
1829
|
+
* daemon that was restarted since, and there is nothing to report about it.
|
|
1830
|
+
*
|
|
1831
|
+
* ═══ AND THE PID IS CLEARED, WHICH IS WHAT KEEPS THIS SET FROM BECOMING A
|
|
1832
|
+
* MINEFIELD. ═══ A stopped run is stopped forever, so without this its pid
|
|
1833
|
+
* would be re-examined on every poll for the life of the machine, and pids are
|
|
1834
|
+
* reused. Sooner or later that number belongs to something else that is alive
|
|
1835
|
+
* and innocent, and this would signal it. Cleared, the row means what it says:
|
|
1836
|
+
* this machine has no process for this run. It is the same `pid = null` a resume
|
|
1837
|
+
* writes, for the same reason.
|
|
1838
|
+
*
|
|
1839
|
+
* AND ONLY WHEN THAT IS TRUE. A refused signal leaves the pid where it is, so
|
|
1840
|
+
* the next poll tries again: clearing it after an EPERM would say this machine
|
|
1841
|
+
* has no process for a run whose agent is still working.
|
|
1842
|
+
*/
|
|
1843
|
+
async function killStopped(client, machineId) {
|
|
1844
|
+
const stopped = await returned(client
|
|
1845
|
+
.from('panel3_runs')
|
|
1846
|
+
.select('id, card_id, pid, started_at, resumed_at')
|
|
1847
|
+
.eq('machine_id', machineId)
|
|
1848
|
+
.in('state', ENDED_BY_THE_PERSON)
|
|
1849
|
+
.not('pid', 'is', null), 'read', 'the runs on this machine that are not coming back');
|
|
1850
|
+
if (stopped.length === 0)
|
|
1851
|
+
return;
|
|
1852
|
+
const booted = bootedAt();
|
|
1853
|
+
for (const run of stopped) {
|
|
1854
|
+
if (runProcessIsAlive(run, booted)) {
|
|
1855
|
+
try {
|
|
1856
|
+
process.kill(run.pid);
|
|
1857
|
+
out(`stopped run ${run.id} card ${run.card_id} killed pid ${run.pid}`);
|
|
1858
|
+
}
|
|
1859
|
+
catch (error) {
|
|
1860
|
+
said(`could not stop pid ${run.pid} for run ${run.id}: ${error instanceof Error ? error.message : String(error)}`);
|
|
1861
|
+
}
|
|
1862
|
+
/* ═══ SIGNALLED IS NOT DEAD, SO THE PID STAYS ON THE ROW. ═══ `kill`
|
|
1863
|
+
returning means the signal was delivered, not that the process acted on
|
|
1864
|
+
it: an agent mid-write, or one ignoring SIGTERM, is still there and
|
|
1865
|
+
still holding the working copy. Clearing the pid now would take the row
|
|
1866
|
+
out of this read for good and it would never be signalled again. The
|
|
1867
|
+
next poll finds it either gone, and clears it below, or still alive, and
|
|
1868
|
+
signals it again. The same line covers a refused signal, where the
|
|
1869
|
+
process is certainly still there. */
|
|
1870
|
+
continue;
|
|
1871
|
+
}
|
|
1872
|
+
try {
|
|
1873
|
+
await returned(client
|
|
1874
|
+
.from('panel3_runs')
|
|
1875
|
+
.update({ pid: null })
|
|
1876
|
+
// STILL NOT COMING BACK. Nothing starts either of these states, so
|
|
1877
|
+
// this cannot lose a race, and saying so in the statement is what
|
|
1878
|
+
// keeps that true if something one day does.
|
|
1879
|
+
.eq('id', run.id)
|
|
1880
|
+
.in('state', ENDED_BY_THE_PERSON)
|
|
1881
|
+
.select('id'), 'clear the process id of', `run ${run.id}`);
|
|
1882
|
+
}
|
|
1883
|
+
catch (error) {
|
|
1884
|
+
// Said, not fatal. The kill has already happened; this is bookkeeping, and
|
|
1885
|
+
// the next poll finds the row again and tries once more.
|
|
1886
|
+
said(`${error instanceof Error ? error.message : String(error)} (it was stopped all the same)`);
|
|
1887
|
+
}
|
|
1888
|
+
}
|
|
1889
|
+
}
|
|
1890
|
+
/**
|
|
1891
|
+
* ═══ A RUN WHOSE PROCESS IS GONE GOES BACK TO TAKEABLE. ═══
|
|
1892
|
+
*
|
|
1893
|
+
* Only this machine's runs, because liveness is a local fact: no other machine
|
|
1894
|
+
* can honestly say whether this one's pid is still there, and guessing from a
|
|
1895
|
+
* timestamp is how a run that is working gets declared dead.
|
|
1896
|
+
*
|
|
1897
|
+
* A run with NO PID RECORDED counts as gone — there is no process to be alive —
|
|
1898
|
+
* but only once it is neither this process's own nor inside `PID_GRACE_MS` of
|
|
1899
|
+
* starting. `mine` covers this daemon; the grace window covers a SECOND daemon
|
|
1900
|
+
* on the same machine, whose `mine` set does not and cannot contain the first
|
|
1901
|
+
* one's runs. Without it, starting a second daemon reaped the first daemon's
|
|
1902
|
+
* live work: turns handed back and a run marked failed while its agent was
|
|
1903
|
+
* genuinely running, producing a duplicate answer.
|
|
1904
|
+
*
|
|
1905
|
+
* ═══ AND A PID THAT PREDATES THIS BOOT IS NOT THIS RUN'S PID, which is
|
|
1906
|
+
* `runProcessIsAlive`'s rule and not this function's. ═══ What it costs here
|
|
1907
|
+
* if it is ever weakened: after a restart a stranded run's pid regularly matches
|
|
1908
|
+
* something unrelated and live, so the card would be skipped on every sweep,
|
|
1909
|
+
* forever, which is the exact failure this function exists to make impossible.
|
|
1910
|
+
*
|
|
1911
|
+
* ═══ ENDING THE RUN AND HANDING ITS TURNS BACK IS ONE STATEMENT. ═══ It used to
|
|
1912
|
+
* be two round trips, turns first so that a failure between them left the run
|
|
1913
|
+
* still `running` for the next sweep to redo. That ordering was safe against a
|
|
1914
|
+
* failure and not against an ANSWER: in the gap the turns were takeable while
|
|
1915
|
+
* the run still read running, so a `panel3_answer` landing there was accepted
|
|
1916
|
+
* onto a card another daemon was already free to take. `panel3_give_up` closes
|
|
1917
|
+
* it, and the loser of that race is now decided by the database rather than by
|
|
1918
|
+
* which write happened to be in flight.
|
|
1919
|
+
*
|
|
1920
|
+
* IT CANNOT UNDO AN ANSWER ANY MORE. A run that has answered is `finished` in
|
|
1921
|
+
* the same statement that wrote the answer (`panel3_answer`), so it is never in
|
|
1922
|
+
* this sweep's result at all. Before that, a daemon killed between the two left
|
|
1923
|
+
* an answered card whose run still read running, and this function blanked
|
|
1924
|
+
* `addressed_at` on turns with their answer sitting beside them.
|
|
1925
|
+
*
|
|
1926
|
+
* ---------------------------------------------------------------------------
|
|
1927
|
+
* ═══ AND A DISPATCHED RUN IS NOT HANDED BACK. IT IS STARTED AGAIN. ═══
|
|
1928
|
+
*
|
|
1929
|
+
* The two kinds of run come back by different routes because their durable
|
|
1930
|
+
* state lives in different places, and this is the fork between them.
|
|
1931
|
+
*
|
|
1932
|
+
* A LEVEL 1 RUN'S WORK IS ITS TURNS. Handing them back is a complete recovery:
|
|
1933
|
+
* the next taker is sent the same thing, built from the same turns.
|
|
1934
|
+
*
|
|
1935
|
+
* A DISPATCHED RUN HOLDS NO TURNS. What it was sent to do and how far it got
|
|
1936
|
+
* are on its own row, so ending it throws both away — and the person's request
|
|
1937
|
+
* dies with a card reading `failed`. `panel3_resume` claims the row instead and
|
|
1938
|
+
* the same agent is started again with its brief, its report and its children.
|
|
1939
|
+
*
|
|
1940
|
+
* ═══ WHICH IS ALSO WHY THE GRACE WINDOW READS `resumed_at`. ═══ A claim nulls
|
|
1941
|
+
* the pid, so a freshly resumed run looks exactly like one whose daemon died
|
|
1942
|
+
* before recording a pid. The window has to be measured from the attempt that is
|
|
1943
|
+
* running now, not from the first one, or every resume would be reaped by the
|
|
1944
|
+
* next sweep a second after it started.
|
|
1945
|
+
*/
|
|
1946
|
+
async function recoverStranded(client, tools, machineId, mine, hold) {
|
|
1947
|
+
const live = await returned(client
|
|
1948
|
+
.from('panel3_runs')
|
|
1949
|
+
.select('id, card_id, parent_run_id, state, pid, started_at, resumed_at, process_token, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id)')
|
|
1950
|
+
.eq('machine_id', machineId)
|
|
1951
|
+
/* ═══ AND A RUN THAT STOPPED TO ASK IS SWEPT TOO, WHICH IS WHAT MAKES THE
|
|
1952
|
+
RE-ARM'S OWN PREDICATE SAFE. ═══ `panel3_take_rearms` will only start a run
|
|
1953
|
+
again once its last process is accounted for — `ended_at is not null` —
|
|
1954
|
+
so that a settle still in flight cannot land on the attempt that
|
|
1955
|
+
replaced it. The stamp arrives when the process exits; if the daemon
|
|
1956
|
+
dies first, nothing else would ever write it and the branch would wait
|
|
1957
|
+
forever on an answer that could never be delivered. This is where it
|
|
1958
|
+
arrives instead. The two predicates are one rule and may not be changed
|
|
1959
|
+
apart. */
|
|
1960
|
+
.in('state', ['running', 'asked'])
|
|
1961
|
+
.is('ended_at', null), 'read', "this machine's live runs");
|
|
1962
|
+
const booted = bootedAt();
|
|
1963
|
+
for (const run of live) {
|
|
1964
|
+
if (mine.has(run.id))
|
|
1965
|
+
continue;
|
|
1966
|
+
// WHEN THE ATTEMPT NOW RUNNING BEGAN, which is the first one until a resume
|
|
1967
|
+
// says otherwise. See the header.
|
|
1968
|
+
const startedAt = new Date(run.resumed_at ?? run.started_at).getTime();
|
|
1969
|
+
const rebooted = startedAt < booted;
|
|
1970
|
+
// A run too young to have recorded a pid is another daemon's live work. The
|
|
1971
|
+
// reused-pid check that used to sit beside this is `runProcessIsAlive`'s.
|
|
1972
|
+
if (!rebooted && run.pid === null && Date.now() - startedAt < PID_GRACE_MS)
|
|
1973
|
+
continue;
|
|
1974
|
+
if (runProcessIsAlive(run, booted))
|
|
1975
|
+
continue;
|
|
1976
|
+
const isOwner = run.card?.conversation_run_id === run.id && run.process_token !== null;
|
|
1977
|
+
if (isOwner) {
|
|
1978
|
+
try {
|
|
1979
|
+
if (run.state === 'asked') {
|
|
1980
|
+
await writeAnswer(client, run.id, run.card_id, null, run.process_token ?? undefined);
|
|
1981
|
+
}
|
|
1982
|
+
else {
|
|
1983
|
+
const resumed = await activateOwner(client, tools, machineId, run.id, run.process_token, run.pid);
|
|
1984
|
+
if (resumed)
|
|
1985
|
+
hold(run.id, resumed.settled);
|
|
1986
|
+
}
|
|
1987
|
+
}
|
|
1988
|
+
catch (error) {
|
|
1989
|
+
said(`could not recover owner ${run.id}: ${error instanceof Error ? error.message : String(error)}`);
|
|
1990
|
+
}
|
|
1991
|
+
continue;
|
|
1992
|
+
}
|
|
1993
|
+
if (run.state === 'asked') {
|
|
1994
|
+
/* ═══ ITS END IS STAMPED AND ITS TURNS ARE LEFT ALONE. ═══ This run is not
|
|
1995
|
+
handed back and is not started again from here: its continuation is the
|
|
1996
|
+
answer to what it asked, and `panel3_take_rearms` is what brings it back.
|
|
1997
|
+
So it must NOT go through `panel3_give_up`, which would hand back the
|
|
1998
|
+
turns a level 1 run took — putting the same message in front of a second
|
|
1999
|
+
coordinator while the person is still being asked about the first.
|
|
2000
|
+
What is missing is only the end stamp its process never got to write. */
|
|
2001
|
+
const ended = await failRun(client, run.id, 'the process that asked this question was gone before it could finish');
|
|
2002
|
+
out(ended
|
|
2003
|
+
? `recover run ${run.id} card ${run.card_id} asked, and its process is gone: ended, `
|
|
2004
|
+
+ 'and it comes back when its question is answered'
|
|
2005
|
+
: `recover run ${run.id} had already ended, left alone`);
|
|
2006
|
+
continue;
|
|
2007
|
+
}
|
|
2008
|
+
if (run.parent_run_id !== null) {
|
|
2009
|
+
/* STARTED AGAIN AS ITSELF, AND A FAILURE IS SAID RATHER THAN THROWN. What
|
|
2010
|
+
it must not do is take the sweep down mid-loop and leave every other
|
|
2011
|
+
stranded run unlooked-at.
|
|
2012
|
+
|
|
2013
|
+
WHAT IS LEFT BEHIND DEPENDS ON HOW FAR IT GOT, and each outcome is
|
|
2014
|
+
honest. A failure BEFORE the claim — this machine has no working copy —
|
|
2015
|
+
leaves the run exactly as it was, still `running` with a dead process,
|
|
2016
|
+
so the next sweep tries again: right for a checkout that is missing for
|
|
2017
|
+
a minute, and one line per poll for one that is missing for good. A run
|
|
2018
|
+
that was CLAIMED and could not be STARTED is ended inside `resumeRun`,
|
|
2019
|
+
because a claimed run with no process must not sit out a grace window
|
|
2020
|
+
pretending to be alive. A failure between the two — the read of its
|
|
2021
|
+
children — leaves it claimed, which costs it one window before the next
|
|
2022
|
+
sweep claims it again. */
|
|
2023
|
+
try {
|
|
2024
|
+
const resumed = await resumeRun(client, tools, machineId, run.id, null);
|
|
2025
|
+
if (resumed)
|
|
2026
|
+
hold(run.id, resumed.settled);
|
|
2027
|
+
else
|
|
2028
|
+
out(`resume run ${run.id} already claimed or started again, left alone`);
|
|
2029
|
+
}
|
|
2030
|
+
catch (error) {
|
|
2031
|
+
said(`could not start run ${run.id} again: ${error instanceof Error ? error.message : String(error)}`);
|
|
2032
|
+
}
|
|
2033
|
+
continue;
|
|
2034
|
+
}
|
|
2035
|
+
const back = await giveUp(client, run.id, rebooted
|
|
2036
|
+
? 'this machine has restarted since the run started, so its process is gone'
|
|
2037
|
+
: run.pid === null
|
|
2038
|
+
? 'the daemon that started this was gone before it recorded a process id'
|
|
2039
|
+
: `the process running this (pid ${run.pid}) is gone and it never ended`);
|
|
2040
|
+
if (back === null) {
|
|
2041
|
+
/* IT ANSWERED WHILE IT WAS BEING GIVEN UP. The two functions race for one
|
|
2042
|
+
run and `panel3_give_up` lost: the card is answered and complete, and
|
|
2043
|
+
nothing here has touched it. Said rather than silent, because a sweep
|
|
2044
|
+
that decided a run was dead and was wrong is worth seeing. */
|
|
2045
|
+
out(`recover run ${run.id} card ${run.card_id} answered as it was being given up, left alone`);
|
|
2046
|
+
continue;
|
|
2047
|
+
}
|
|
2048
|
+
out(`recover run ${run.id} card ${run.card_id} process gone, `
|
|
2049
|
+
+ `${back} turn${back === 1 ? '' : 's'} takeable again`);
|
|
2050
|
+
}
|
|
2051
|
+
}
|
|
2052
|
+
/**
|
|
2053
|
+
* ═══ A RUN A PERSON HANDED BACK, TAKEN UP BY WHOEVER IS LISTENING. ═══
|
|
2054
|
+
*
|
|
2055
|
+
* The same act as `recoverStranded`'s dispatched branch, by the other route.
|
|
2056
|
+
* The sweep can only ever reach THIS machine's runs, because liveness is a local
|
|
2057
|
+
* fact and no machine can honestly say whether another one's pid is still there.
|
|
2058
|
+
* So a machine that is never coming back strands its dispatched runs forever:
|
|
2059
|
+
* nothing sweeps them, and `panel3_take_turns` supersedes level 1 only.
|
|
2060
|
+
*
|
|
2061
|
+
* `panel3_hand_back` is the person's answer to that, and it decided everything.
|
|
2062
|
+
* It ran under the card's own row lock, refused any run whose machine had
|
|
2063
|
+
* heartbeated inside the listening window, and stamped `handed_back_at` on the
|
|
2064
|
+
* dispatched ones. THIS IS ONLY THE DISCOVERY, which is the one thing that was
|
|
2065
|
+
* missing: the daemon never asked whether another machine's run was available.
|
|
2066
|
+
*
|
|
2067
|
+
* ═══ AND SO THE READ HAS NO `machine_id` FILTER, WHICH IS THE WHOLE POINT. ═══
|
|
2068
|
+
* RLS scopes it to the person's own account and nothing narrower, because the
|
|
2069
|
+
* machine that stamped the offer is by definition not the machine that should
|
|
2070
|
+
* serve it. `panel3_resume` has never had a machine predicate either, so the
|
|
2071
|
+
* claim was always cross-machine; only the looking was not.
|
|
2072
|
+
*
|
|
2073
|
+
* ---------------------------------------------------------------------------
|
|
2074
|
+
* ═══ WHICH OF THE SWEEP'S SAFETY CHECKS APPLY HERE, AND WHICH CANNOT. ═══
|
|
2075
|
+
*
|
|
2076
|
+
* The sweep refuses three things, and they are not decoration: they are what
|
|
2077
|
+
* stops a machine reaping its own live work. Taken one at a time, because
|
|
2078
|
+
* carrying them across without thinking and dropping them without saying are
|
|
2079
|
+
* the same mistake.
|
|
2080
|
+
*
|
|
2081
|
+
* `mine`, A RUN THIS DAEMON IS HOLDING RIGHT NOW: KEPT. Presence is coarser
|
|
2082
|
+
* than a pid, so a daemon that is alive but has missed the listening window,
|
|
2083
|
+
* meaning a closed lid, a paused VM or a network stall, has its runs offered
|
|
2084
|
+
* while its agents are genuinely working. When it comes back it finds its OWN
|
|
2085
|
+
* work on offer, and without this line it would claim it: `panel3_resume`
|
|
2086
|
+
* writes `pid = null` and a second process starts on the same run id and the
|
|
2087
|
+
* same working copy, while the first one's `settle` is still going to answer
|
|
2088
|
+
* the card.
|
|
2089
|
+
*
|
|
2090
|
+
* ═══ AND IT NARROWS THAT CLASS RATHER THAN CLOSING IT. ═══ A THIRD machine,
|
|
2091
|
+
* listening and healthy, can claim that same live run, because the guard the
|
|
2092
|
+
* database applied is presence and presence is what was wrong. That limit is
|
|
2093
|
+
* the migration header's own and is not this function's to fix; `mine` closes
|
|
2094
|
+
* one half that IS knowable here, which is a run THIS PROCESS is holding, and
|
|
2095
|
+
* the pid check below closes the other, which is a run ANOTHER PROCESS ON THIS
|
|
2096
|
+
* MACHINE is holding.
|
|
2097
|
+
*
|
|
2098
|
+
* `PID_GRACE_MS`, TOO YOUNG TO HAVE RECORDED A PID: KEPT, in its `pid is null`
|
|
2099
|
+
* half only, and it closes the same hole `mine` does one step earlier. A
|
|
2100
|
+
* FRESHLY DISPATCHED run is not in `inFlight` yet: `startChild` writes the row
|
|
2101
|
+
* through `panel3_dispatch` and the poll only calls `hold` once `startChild`
|
|
2102
|
+
* has RESOLVED, so in between the row reads `running`, `parent_run_id` is not
|
|
2103
|
+
* null, `pid` is null, and `mine` does not contain it. The tools server runs
|
|
2104
|
+
* off HTTP rather than off the poll, so a daemon stalled past the listening
|
|
2105
|
+
* window still serves a `dispatch` and still writes that row: the person
|
|
2106
|
+
* presses, the child is stamped, the stall ends, and this read would claim a
|
|
2107
|
+
* run whose own process is in the act of starting. Two agents, one run id, one
|
|
2108
|
+
* working copy, which is what `mine` is here to prevent.
|
|
2109
|
+
*
|
|
2110
|
+
* ═══ AND IT COSTS THE REAL CASE NOTHING. ═══ A genuinely stranded offer is by
|
|
2111
|
+
* construction older than the fifteen seconds the banner needed before a
|
|
2112
|
+
* person could even see it, so the window never fires for one.
|
|
2113
|
+
*
|
|
2114
|
+
* THE SWEEP'S OTHER HALF OF THAT LINE, `rebooted`, STAYS OUT. It is read off
|
|
2115
|
+
* this machine's uptime, and this is the one read that is not about this
|
|
2116
|
+
* machine's runs.
|
|
2117
|
+
*
|
|
2118
|
+
* `runProcessIsAlive`, THE PID PLUS A BOOT-TIME REFUSAL: APPLIED TO THIS
|
|
2119
|
+
* MACHINE'S OWN OFFERED RUNS, AND TO NOBODY ELSE'S. The two halves are
|
|
2120
|
+
* genuinely different questions and were once answered as one.
|
|
2121
|
+
*
|
|
2122
|
+
* ON ANOTHER MACHINE'S RUN IT WOULD BE A DEFECT. That pid belongs to a process
|
|
2123
|
+
* on a box this one cannot see, so `kill(pid, 0)` here asks about an unrelated
|
|
2124
|
+
* local process, and pids are reused: a collision would make this machine skip
|
|
2125
|
+
* a run nobody is working on, forever, which is the exact strand the feature
|
|
2126
|
+
* exists to end. What stands in for it there is the heartbeat guard already
|
|
2127
|
+
* spent inside `panel3_hand_back`, with the honest limit that migration's
|
|
2128
|
+
* header states.
|
|
2129
|
+
*
|
|
2130
|
+
* ON A RUN WHOSE `machine_id` IS THIS MACHINE IT IS EXACTLY AS TRUSTWORTHY AS
|
|
2131
|
+
* IT IS IN `recoverStranded`, and TWO DAEMONS ON ONE MACHINE IS NOT
|
|
2132
|
+
* HYPOTHETICAL: 20260903140000 exists to support one per harness, and
|
|
2133
|
+
* `PID_GRACE_MS`'s own header says so. Without this check, D1 holds a live
|
|
2134
|
+
* dispatched run with its pid recorded, the machine sleeps, both presence rows
|
|
2135
|
+
* go stale, a person hands the card back, the machine wakes and D2 polls
|
|
2136
|
+
* first. The sweep leaves the run alone, correctly, because the process really
|
|
2137
|
+
* is alive. This read would then claim the very same row: `mine` is D2's and
|
|
2138
|
+
* does not hold D1's work, the grace window does not fire because the pid is
|
|
2139
|
+
* there, and `panel3_resume` matches every predicate, because `machine_id`
|
|
2140
|
+
* moves from this machine to this machine. Two agents, one run id, one working
|
|
2141
|
+
* copy, and D1's answer is accepted afterwards by a `panel3_answer` fenced by
|
|
2142
|
+
* neither machine nor pid.
|
|
2143
|
+
*
|
|
2144
|
+
* ---------------------------------------------------------------------------
|
|
2145
|
+
* ═══ AND A FAILURE IS SAID RATHER THAN THROWN, AS IT IS IN THE SWEEP. ═══ The
|
|
2146
|
+
* ordinary one is a machine with no folder for the project: `resumeRun` resolves
|
|
2147
|
+
* the checkout BEFORE the claim, so such a machine never takes a run it cannot
|
|
2148
|
+
* start. It says why on its own stderr and leaves the offer standing for a
|
|
2149
|
+
* machine that has the codebase. That is not silence and it is not a success:
|
|
2150
|
+
* the run is exactly as it was, and the card goes on saying nothing is listening,
|
|
2151
|
+
* which is true until somebody who can serve it is.
|
|
2152
|
+
*
|
|
2153
|
+
* ═══ AND THAT HOLDS ONLY WHERE `CTRL_SPC_V3_WORKING_COPY` IS UNSET, WHICH IS
|
|
2154
|
+
* THIS TASK'S ONE REAL EXPOSURE. ═══
|
|
2155
|
+
*
|
|
2156
|
+
* `checkoutFor` resolves in three steps and the third is a machine-wide folder
|
|
2157
|
+
* named by that variable, which applies EVEN WHEN THE CARD HAS A PROJECT. So a
|
|
2158
|
+
* listening machine with it set resolves a checkout for any project at all,
|
|
2159
|
+
* passes the ordering guarantee above, claims the run and starts an agent in a
|
|
2160
|
+
* folder holding somebody else's code, which is the failure
|
|
2161
|
+
* `panel3-checkout.contract.test.mjs`'s own header names. Before this function
|
|
2162
|
+
* existed it was unreachable across machines, because nothing ever looked at
|
|
2163
|
+
* another machine's run; this read is what makes it reachable, and the
|
|
2164
|
+
* cross-machine walk in plan-slice-2.md section 8 is exactly the setup that sets
|
|
2165
|
+
* the variable, so that walk can produce a false pass.
|
|
2166
|
+
*
|
|
2167
|
+
* ═══ RECORDED, NOT FIXED HERE. ═══ Reordering that resolution is `../codebases-3`'s
|
|
2168
|
+
* assignment, and `checkoutFor` is shared by every spawn path with its own
|
|
2169
|
+
* contract suite: changing it from this slice would widen a fifty-line read into
|
|
2170
|
+
* a change to how every agent in the product finds its code.
|
|
2171
|
+
*
|
|
2172
|
+
* ═══ AND ONE OFFER NOBODY CAN CHECK OUT IS NOW RETRIED BY THE WHOLE FLEET. ═══
|
|
2173
|
+
* Nothing clears `handed_back_at` except a claim, so every listening machine
|
|
2174
|
+
* reads that run every poll and prints one line for it, forever, where
|
|
2175
|
+
* `recoverStranded`'s header accepts the same cost for one machine. Two reads and
|
|
2176
|
+
* a line per machine per poll is cheap and is the honest state of the record: the
|
|
2177
|
+
* work really is offered and really has no taker. Silencing it would mean an
|
|
2178
|
+
* expiry on the offer, which is a rule nothing has asked for and which would
|
|
2179
|
+
* throw the work away on a fleet that was merely all asleep.
|
|
2180
|
+
*/
|
|
2181
|
+
export async function takeHandedBack(client, tools, machineId, mine, hold) {
|
|
2182
|
+
/* ITS OWN NARROW ROW, NOT `LiveRun`. `machine_id` is read for one question
|
|
2183
|
+
and one only: whether this row is THIS machine's, which is what decides
|
|
2184
|
+
whether its pid means anything here. See the header. */
|
|
2185
|
+
const offered = await returned(client
|
|
2186
|
+
.from('panel3_runs')
|
|
2187
|
+
.select('id, card_id, machine_id, pid, started_at, resumed_at, process_token, card:panel3_cards!panel3_runs_card_id_fkey(conversation_run_id)')
|
|
2188
|
+
.not('handed_back_at', 'is', null)
|
|
2189
|
+
/* WHAT THE OFFER MEANS, RESTATED IN THE READ. `panel3_hand_back` only ever
|
|
2190
|
+
stamps a dispatched run that is still going, and `panel3_resume` refuses
|
|
2191
|
+
anything else. Saying it here too costs one line and means a row that
|
|
2192
|
+
somehow carried a stale stamp is passed over rather than sent to a
|
|
2193
|
+
function that would refuse it a round trip later. */
|
|
2194
|
+
.eq('state', 'running')
|
|
2195
|
+
.is('ended_at', null)
|
|
2196
|
+
.not('parent_run_id', 'is', null), 'read', 'the runs a person has handed back');
|
|
2197
|
+
// ONCE, OUTSIDE THE LOOP. It is a property of this machine, not of a row.
|
|
2198
|
+
const booted = bootedAt();
|
|
2199
|
+
for (const run of offered) {
|
|
2200
|
+
if (mine.has(run.id)) {
|
|
2201
|
+
/* THIS DAEMON'S OWN LIVE WORK, OFFERED WHILE IT WAS BUSY BEING QUIET. See
|
|
2202
|
+
the header. Said rather than passed over in silence, because a machine
|
|
2203
|
+
whose work was handed back out from under it is the one thing here worth
|
|
2204
|
+
seeing in a terminal. */
|
|
2205
|
+
out(`offered run ${run.id} card ${run.card_id} is this daemon's own live work, left alone`);
|
|
2206
|
+
continue;
|
|
2207
|
+
}
|
|
2208
|
+
/* ═══ AND THE OTHER DAEMON ON THIS MACHINE IS HOLDING IT. ═══ `mine` covers
|
|
2209
|
+
this process and cannot cover its sibling: one machine runs one daemon per
|
|
2210
|
+
harness on purpose (20260903140000), and a machine that slept has both
|
|
2211
|
+
presence rows stale, so `panel3_hand_back` offers work that is still
|
|
2212
|
+
genuinely running. THE PID ANSWERS THAT, AND ONLY FOR THIS MACHINE'S OWN
|
|
2213
|
+
ROWS: for anybody else's it would be a question about an unrelated local
|
|
2214
|
+
process. `recoverStranded` asks it the same way on the same rows and
|
|
2215
|
+
correctly leaves them alone, so without this the sweep declines and the
|
|
2216
|
+
very next read claims. See the header. */
|
|
2217
|
+
if (run.machine_id === machineId && runProcessIsAlive(run, booted)) {
|
|
2218
|
+
out(`offered run ${run.id} card ${run.card_id} is still running on this machine, left alone`);
|
|
2219
|
+
continue;
|
|
2220
|
+
}
|
|
2221
|
+
/* ═══ AND A RUN TOO YOUNG TO HAVE RECORDED A PID IS SOMEBODY'S LIVE WORK
|
|
2222
|
+
TOO, INCLUDING THIS DAEMON'S OWN. ═══ `mine` cannot hold a dispatch
|
|
2223
|
+
between the row `panel3_dispatch` wrote and the `hold` that follows the
|
|
2224
|
+
spawn, and that gap is reachable from here: see the header. This is
|
|
2225
|
+
`recoverStranded`'s own line without its `rebooted` half, which reads this
|
|
2226
|
+
machine's uptime and has no meaning for another machine's run. */
|
|
2227
|
+
if (run.pid === null
|
|
2228
|
+
&& Date.now() - new Date(run.resumed_at ?? run.started_at).getTime() < PID_GRACE_MS) {
|
|
2229
|
+
out(`offered run ${run.id} card ${run.card_id} is too young to have a process yet, left alone`);
|
|
2230
|
+
continue;
|
|
2231
|
+
}
|
|
2232
|
+
try {
|
|
2233
|
+
const resumed = run.card?.conversation_run_id === run.id && run.process_token !== null
|
|
2234
|
+
? await activateOwner(client, tools, machineId, run.id, run.process_token, null)
|
|
2235
|
+
: await resumeRun(client, tools, machineId, run.id, null);
|
|
2236
|
+
if (resumed)
|
|
2237
|
+
hold(run.id, resumed.settled);
|
|
2238
|
+
else
|
|
2239
|
+
out(`resume run ${run.id} already claimed or started again, left alone`);
|
|
2240
|
+
}
|
|
2241
|
+
catch (error) {
|
|
2242
|
+
said(`could not take run ${run.id}, which was handed back: `
|
|
2243
|
+
+ `${error instanceof Error ? error.message : String(error)}`);
|
|
2244
|
+
}
|
|
2245
|
+
}
|
|
2246
|
+
}
|
|
2247
|
+
// ---------------------------------------------------------------------------
|
|
2248
|
+
/**
|
|
2249
|
+
* The poll loop: it takes what is waiting and answers it, forever. `--once`
|
|
2250
|
+
* polls a single time and exits when the work it took is done, which is what
|
|
2251
|
+
* makes the loop testable — two of them against one card is how the lease is
|
|
2252
|
+
* proved.
|
|
2253
|
+
*
|
|
2254
|
+
* ═══ ONE LOOP, TWO CALLERS, AND THAT IS THE WHOLE OF recovery-1 SLICE 4. ═══
|
|
2255
|
+
* `startPanel` below is `cs start`'s entry and hands in the session the
|
|
2256
|
+
* installed CLI already holds. `cs3 run` is the acceptance harness's entry and
|
|
2257
|
+
* signs in from the environment against its own isolated account. Neither is a
|
|
2258
|
+
* copy of the other: a second poll loop would be two things leasing the same
|
|
2259
|
+
* turns with two ideas of what recovery may reap.
|
|
2260
|
+
*
|
|
2261
|
+
* @param injected the client to poll and serve tools through, when the caller
|
|
2262
|
+
* already has one. Its presence is also what says THIS PROCESS IS NOT v3's, so
|
|
2263
|
+
* the signals belong to somebody else — see the handler block below.
|
|
2264
|
+
*/
|
|
2265
|
+
export async function run(args, injected) {
|
|
2266
|
+
let once = false;
|
|
2267
|
+
for (const arg of args) {
|
|
2268
|
+
if (arg === '--once')
|
|
2269
|
+
once = true;
|
|
2270
|
+
else
|
|
2271
|
+
throw new Error(`unknown option ${arg}. ${USAGE}`);
|
|
2272
|
+
}
|
|
2273
|
+
/* ═══ THE ONE v3 CLIENT THAT OUTLIVES ITS ACCESS TOKEN, SO IT IS THE ONE THAT
|
|
2274
|
+
REFRESHES. ═══ This client polls for as long as the daemon runs AND is the
|
|
2275
|
+
client every tool call is served through, so past the token's TTL both stop
|
|
2276
|
+
working at once. See `client.ts` for why refreshing is safe here:
|
|
2277
|
+
`persistSession` stays false, so the rotated token lives in this process and
|
|
2278
|
+
no file on the machine is written.
|
|
2279
|
+
|
|
2280
|
+
═══ AND UNDER `cs start` THERE IS NO SIGN-IN AT ALL, WHICH IS THE POINT. ═══
|
|
2281
|
+
The installed daemon's own client is handed in: it refreshes, it retries at
|
|
2282
|
+
the network layer, and it is the ONLY client in that process, so the rotated
|
|
2283
|
+
token has exactly one writer. Signing in again there would be the second
|
|
2284
|
+
refresh loop `client.ts`'s header exists to prevent, this time inside one
|
|
2285
|
+
process rather than across two. */
|
|
2286
|
+
const client = injected ?? await signedInClient(true);
|
|
2287
|
+
const machineId = getMachineIdentity().id;
|
|
2288
|
+
/* THE NAME AND THE HARNESS, RESOLVED ONCE, AT STARTUP, RATHER THAN PER POLL.
|
|
2289
|
+
Both are properties of this machine for the life of this process: the
|
|
2290
|
+
hostname does not change under it, and neither does `CTRL_SPC_V3_AGENT`.
|
|
2291
|
+
Resolving `harness()` here means a machine misconfigured with a name this
|
|
2292
|
+
build cannot spawn (constraint 7: reported, never chosen) fails before it
|
|
2293
|
+
ever says it is listening, rather than on its first poll. */
|
|
2294
|
+
const machineName = hostname();
|
|
2295
|
+
const machineHarness = harness();
|
|
2296
|
+
/* THE RUNS THIS PROCESS IS HOLDING RIGHT NOW, so recovery cannot declare its
|
|
2297
|
+
own live work dead in the moment before a pid is recorded. It covers THIS
|
|
2298
|
+
daemon only, which is why `PID_GRACE_MS` exists for the other ones. Keyed by
|
|
2299
|
+
run id, at every level: a dispatched agent is this daemon's to hold too, and
|
|
2300
|
+
leaving it out would have recovery reap the children of its own runs. */
|
|
2301
|
+
const inFlight = new Map();
|
|
2302
|
+
/** One dispatched run's promise, held where the daemon can wait on it and
|
|
2303
|
+
* recovery can see it. The same bookkeeping the poll does for a level 1 run,
|
|
2304
|
+
* in the one other place a process is started. */
|
|
2305
|
+
const hold = (runId, work) => {
|
|
2306
|
+
inFlight.set(runId, work
|
|
2307
|
+
.catch((error) => {
|
|
2308
|
+
said(`run ${runId}: ${error instanceof Error ? error.message : String(error)}`);
|
|
2309
|
+
})
|
|
2310
|
+
.finally(() => { inFlight.delete(runId); }));
|
|
2311
|
+
};
|
|
2312
|
+
/* ═══ THE TOOLS BEFORE THE FIRST TAKE, OR NO TAKES AT ALL. ═══ A daemon that
|
|
2313
|
+
took a card and then found it could not serve tools would spawn an agent
|
|
2314
|
+
with no hands and write its confusion to the card as an answer. It is one
|
|
2315
|
+
loopback listener on an ephemeral port; failing here is a daemon that never
|
|
2316
|
+
started, which is the honest outcome.
|
|
2317
|
+
|
|
2318
|
+
`tools` is referenced inside the callback it is being given, which is safe
|
|
2319
|
+
for the plain reason that the callback can only run once a request has
|
|
2320
|
+
arrived at a server that by then exists. */
|
|
2321
|
+
const tools = await startToolsServer(client, async (parentRunId, brief, codebase, processToken) => {
|
|
2322
|
+
const child = await startChild(client, tools, machineId, parentRunId, brief, codebase, processToken);
|
|
2323
|
+
hold(child.runId, child.settled);
|
|
2324
|
+
return { runId: child.runId };
|
|
2325
|
+
});
|
|
2326
|
+
out(`daemon machine ${machineId}`);
|
|
2327
|
+
out(`tools ${tools.urlFor('<run-id>')}`);
|
|
2328
|
+
out(once ? 'mode one poll' : `mode polling every ${POLL_INTERVAL_MS / 1000}s, Ctrl-C to stop`);
|
|
2329
|
+
/* ═══ AND THIS MACHINE STOPS SAYING IT IS LISTENING WHEN IT GOES. ═══ ux.md
|
|
2330
|
+
forbids a card reading `working` while nothing is working, and the freshness
|
|
2331
|
+
window is a backstop for the machine that CANNOT say goodbye — a crash, a
|
|
2332
|
+
lost network, a kill -9. A Ctrl-C can, so it does, and the panel tells the
|
|
2333
|
+
truth within a poll rather than within a window.
|
|
2334
|
+
|
|
2335
|
+
`once` never gets here: it returns below, and the same call is made there.
|
|
2336
|
+
|
|
2337
|
+
═══ AND NOT WHEN A CLIENT WAS HANDED IN, BECAUSE THEN THE PROCESS IS NOT
|
|
2338
|
+
THIS LOOP'S TO EXIT. ═══ Under `cs start` this shares a process with the v2
|
|
2339
|
+
presence heartbeat, which has a shutdown of its own that marks
|
|
2340
|
+
`cliv2_agents` offline and then calls `process.exit`. Two handlers means two
|
|
2341
|
+
exits, and whichever finishes its write first kills the other's mid-flight:
|
|
2342
|
+
a Ctrl-C would leave the machine reading listening in one place and gone in
|
|
2343
|
+
the other, which is Slice 1's card lying in the direction Slice 4 exists to
|
|
2344
|
+
close. `daemon.ts` owns the signal there and does both writes before it
|
|
2345
|
+
goes. */
|
|
2346
|
+
if (!once && !injected) {
|
|
2347
|
+
for (const signal of ['SIGINT', 'SIGTERM']) {
|
|
2348
|
+
process.once(signal, () => {
|
|
2349
|
+
void stopListening(client, machineId, machineHarness).finally(() => {
|
|
2350
|
+
/* The exit code a signal is supposed to produce, and the reason it is
|
|
2351
|
+
said explicitly: `process.once` REPLACES node's default handler, so
|
|
2352
|
+
without this a Ctrl-C would leave the daemon polling forever. */
|
|
2353
|
+
process.exit(signal === 'SIGINT' ? 130 : 143);
|
|
2354
|
+
});
|
|
2355
|
+
});
|
|
2356
|
+
}
|
|
2357
|
+
}
|
|
2358
|
+
for (;;) {
|
|
2359
|
+
/* ═══ ONE POLL FAILING IS NOT THE DAEMON FAILING. ═══ Every read and write
|
|
2360
|
+
here throws on a network or database error, by design (constraint 7), and
|
|
2361
|
+
until Slice 4 that threw straight out of `cs3 run` and exited the process.
|
|
2362
|
+
Inside `cs start` that is no longer an honest outcome twice over: it would
|
|
2363
|
+
take the v2 presence heartbeat down with it, and it would leave a stranded
|
|
2364
|
+
card printing `cs start` at a person who IS running `cs start`, which is
|
|
2365
|
+
the exact lie this slice exists to end. A dropped connection is a poll
|
|
2366
|
+
that did not happen; the next one is two seconds away.
|
|
2367
|
+
|
|
2368
|
+
`--once` still fails loudly, because the acceptance harness reads the exit
|
|
2369
|
+
code and a swallowed failure there would make a broken suite look green. */
|
|
2370
|
+
try {
|
|
2371
|
+
/* ═══ THE FIRST THING EVERY POLL, BECAUSE IT IS WHAT MAKES THE OTHER THINGS
|
|
2372
|
+
LEGIBLE. ═══ A card with an untaken turn reads the same whether a daemon
|
|
2373
|
+
is two seconds away or nobody has one running; this row is the only place
|
|
2374
|
+
the difference exists. It is written before the takes rather than after
|
|
2375
|
+
so that a machine which is up but busy still reads as up. */
|
|
2376
|
+
await sayListening(client, machineId, machineName, machineHarness);
|
|
2377
|
+
/* ═══ THE USER'S STOP IS HONOURED BEFORE ANYTHING ELSE ON THE POLL. ═══ It
|
|
2378
|
+
is the only thing here that a person is waiting on, and the two takes
|
|
2379
|
+
below can spend the rest of the poll starting agents. Nothing else needs
|
|
2380
|
+
to run first: `panel3_stop_card` has already ended the runs, so recovery
|
|
2381
|
+
cannot see them and neither take can start them. */
|
|
2382
|
+
await killStopped(client, machineId);
|
|
2383
|
+
await reconcileOwnerSessions(client, machineId, machineHarness, new Set(inFlight.keys()));
|
|
2384
|
+
await recoverStranded(client, tools, machineId, new Set(inFlight.keys()), hold);
|
|
2385
|
+
/* ═══ AND THEN WHAT SOMEBODY ELSE'S MACHINE WAS HOLDING, IF A PERSON HANDED
|
|
2386
|
+
IT BACK. ═══ AFTER the sweep, deliberately: this machine settles its own
|
|
2387
|
+
runs on local, pid-accurate evidence before it looks at anybody's, and a
|
|
2388
|
+
run of its own that was offered while it was quiet is dealt with there on
|
|
2389
|
+
the better evidence: resumed if its process is really gone, which spends
|
|
2390
|
+
the offer, and skipped if it is not, which is what `mine` then honours
|
|
2391
|
+
here. The set is rebuilt rather than reused because the sweep adds to it.
|
|
2392
|
+
BEFORE both takes, for the reason the sweep is: work that already exists,
|
|
2393
|
+
with a brief and a report behind it, comes before work that has not
|
|
2394
|
+
started. */
|
|
2395
|
+
await takeHandedBack(client, tools, machineId, new Set(inFlight.keys()), hold);
|
|
2396
|
+
/* Every reason a conversational owner may continue is claimed together.
|
|
2397
|
+
The legacy turn and re-arm takes exclude named owners in the database. */
|
|
2398
|
+
await takeOwnerActivations(client, tools, machineId, new Set(inFlight.keys()), hold);
|
|
2399
|
+
/* THE MACHINE ID GOES IN because the take writes the run row, and a run has
|
|
2400
|
+
to say where it is running: the exclusion is cross-machine and recovery is
|
|
2401
|
+
per-machine, so a row with nobody's machine on it could be neither. */
|
|
2402
|
+
const taken = await returned(client.rpc('panel3_take_turns', { p_machine_id: machineId, p_agent: machineHarness }), 'take', 'turns');
|
|
2403
|
+
/* ═══ THE OTHER KIND OF TAKEABLE WORK. ═══ ux.md's re-arm: an answered
|
|
2404
|
+
question makes the branch that asked it takeable again, and a question
|
|
2405
|
+
still walking up makes the run it reached takeable so that level gets its
|
|
2406
|
+
turn. Same poll, same machine id, same holding: only the reason a run is
|
|
2407
|
+
started differs, and the record decides that rather than this file.
|
|
2408
|
+
|
|
2409
|
+
AFTER THE TURNS ARE TAKEN, deliberately, because the take WRITES the run
|
|
2410
|
+
row it leases to. A person who answered and then typed something else has
|
|
2411
|
+
both waiting, and in this order the message's level 1 is already on the
|
|
2412
|
+
record, so the re-arm leaves that card's coordinator alone and comes back
|
|
2413
|
+
to it on a later poll rather than putting two of them on one card. The
|
|
2414
|
+
other order would decide the same question from a snapshot taken before
|
|
2415
|
+
the run existed. */
|
|
2416
|
+
const rearmed = await returned(client.rpc('panel3_take_rearms', { p_machine_id: machineId, p_agent: machineHarness }), 'take', 'runs that can carry on');
|
|
2417
|
+
for (const row of rearmed) {
|
|
2418
|
+
/* NOT AWAITED PAST THE SPAWN, exactly as a taken card is not: the work is a
|
|
2419
|
+
real agent and holding the poll open for it would put every other card
|
|
2420
|
+
behind it. A throw here is caught rather than taking the loop down,
|
|
2421
|
+
because one run that could not be started must not stop the others. */
|
|
2422
|
+
try {
|
|
2423
|
+
hold(row.run_id, (await startRearmed(client, tools, machineId, row)).settled);
|
|
2424
|
+
}
|
|
2425
|
+
catch (error) {
|
|
2426
|
+
said(`could not start run ${row.run_id} again: ${error instanceof Error ? error.message : String(error)}`);
|
|
2427
|
+
}
|
|
2428
|
+
}
|
|
2429
|
+
for (const [cardId, turns] of byCard(taken)) {
|
|
2430
|
+
const runId = turns[0].run_id;
|
|
2431
|
+
out(`took card ${cardId} ${turns.length} turn${turns.length === 1 ? '' : 's'} run ${runId}`);
|
|
2432
|
+
/* NOT AWAITED HERE. A run is a real agent doing real work, and awaiting it
|
|
2433
|
+
in the poll would put every other card behind it — v2's own measured
|
|
2434
|
+
defect (env.ts, I21/I24: "a second request typed seconds after the first
|
|
2435
|
+
sat untouched until the first finished"). Nothing is needed to stop the
|
|
2436
|
+
same card being taken twice, because that is the database's job and it
|
|
2437
|
+
has already been done by the take. */
|
|
2438
|
+
// The catch inside `hold` is the last resort. A run left `running` by a
|
|
2439
|
+
// throw is not lost: its process has exited, so the next sweep reaps it
|
|
2440
|
+
// and the turns are answered again. It is there so the reason is printed
|
|
2441
|
+
// rather than becoming an unhandled rejection.
|
|
2442
|
+
hold(runId, answerCard(client, tools, machineId, cardId, turns));
|
|
2443
|
+
}
|
|
2444
|
+
if (once) {
|
|
2445
|
+
/* ═══ UNTIL NOTHING IS LEFT, NOT ONCE OVER WHAT WAS THERE. ═══ A level 1
|
|
2446
|
+
run dispatches WHILE it is being waited on, so the child appears in
|
|
2447
|
+
`inFlight` after the first wait started. Closing then would kill the
|
|
2448
|
+
tools server under an agent that had just been started, and `--once`
|
|
2449
|
+
would report a card answered while the work on it was still going. */
|
|
2450
|
+
while (inFlight.size > 0)
|
|
2451
|
+
await Promise.all([...inFlight.values()]);
|
|
2452
|
+
// The listener would otherwise keep this process alive after its one poll
|
|
2453
|
+
// was done, which is the whole point of `--once` being testable.
|
|
2454
|
+
await tools.close();
|
|
2455
|
+
// This machine is not listening any more, and it knows that here rather
|
|
2456
|
+
// than fifteen seconds from now. Same reason as the signal handlers above.
|
|
2457
|
+
await stopListening(client, machineId, machineHarness);
|
|
2458
|
+
return;
|
|
2459
|
+
}
|
|
2460
|
+
}
|
|
2461
|
+
catch (error) {
|
|
2462
|
+
if (once)
|
|
2463
|
+
throw error;
|
|
2464
|
+
said(`this poll did not finish: ${error instanceof Error ? error.message : String(error)}`);
|
|
2465
|
+
}
|
|
2466
|
+
await sleep(POLL_INTERVAL_MS);
|
|
2467
|
+
}
|
|
2468
|
+
}
|
|
2469
|
+
// ---------------------------------------------------------------------------
|
|
2470
|
+
/**
|
|
2471
|
+
* ═══ THE ONE THING `cs start` REACHES INTO `panel3/` FOR. ═══
|
|
2472
|
+
*
|
|
2473
|
+
* recovery-1 Slice 4, and Lane's ruling behind it (2026-08-21): *the user
|
|
2474
|
+
* experience must never require them to launch or authenticate multiple CLIs.
|
|
2475
|
+
* They launch the CLI with `cs start`.* Slices 1 to 3 print `cs start` on a
|
|
2476
|
+
* stranded card and, until this function existed, that command started a daemon
|
|
2477
|
+
* which polled no cards at all — the sentence named the right machine and the
|
|
2478
|
+
* wrong command, and no rewording could fix it.
|
|
2479
|
+
*
|
|
2480
|
+
* ═══ IT IS A NAMED EXCEPTION TO THE ISOLATION RULE, NOT A HOLE IN IT. ═══
|
|
2481
|
+
* `conventions.md` forbids anything outside `panel3/` importing anything inside
|
|
2482
|
+
* it, and `panel3-isolation.contract.test.mjs` enforces it. `cli-v2/src/daemon.ts`
|
|
2483
|
+
* is listed there with exactly ONE allowed specifier, `./panel3/run.js`, which is
|
|
2484
|
+
* why this is a single entry point handing back a single closure rather than
|
|
2485
|
+
* three exports the daemon would have to assemble. A second specifier fails the
|
|
2486
|
+
* suite, which is the point: the rule is enumerated, never weakened.
|
|
2487
|
+
*
|
|
2488
|
+
* ═══ THE CLIENT IS THE CALLER'S, AND THAT IS THE WHOLE OF "ONE SIGN-IN". ═══
|
|
2489
|
+
* `startPresence()` hands over the client it built from the session `cs login`
|
|
2490
|
+
* stored. This must never take one of its own: two clients in one process is two
|
|
2491
|
+
* refresh loops on one rotating refresh token, both writing `session.json`
|
|
2492
|
+
* through `getClient`'s `onAuthStateChange`, which is the hazard `client.ts`
|
|
2493
|
+
* describes moved indoors.
|
|
2494
|
+
*
|
|
2495
|
+
* ═══ IT DOES NOT BLOCK, AND IT DOES NOT TAKE THE DAEMON DOWN. ═══ The loop
|
|
2496
|
+
* never returns, so awaiting it here would hang `cs start` before it printed a
|
|
2497
|
+
* line. A throw that escapes the per-poll guard inside it is the panel stopping,
|
|
2498
|
+
* which is said on the one stream a person reads and leaves v2 presence
|
|
2499
|
+
* heartbeating rather than killing the process around it.
|
|
2500
|
+
*/
|
|
2501
|
+
export function startPanel(client) {
|
|
2502
|
+
/* Both resolved HERE as well as inside the loop, deliberately: `stop` has to
|
|
2503
|
+
name the same row the loop's heartbeat writes, and it must be able to do
|
|
2504
|
+
that after the loop has thrown. They are the two facts about this machine
|
|
2505
|
+
that cannot change under a running process — the hardware-derived id and
|
|
2506
|
+
`CTRL_SPC_V3_AGENT` — so reading them twice cannot disagree. */
|
|
2507
|
+
const machineId = getMachineIdentity().id;
|
|
2508
|
+
const machineHarness = harness();
|
|
2509
|
+
void run([], client).catch((error) => {
|
|
2510
|
+
said(`the agent panel stopped polling: ${error instanceof Error ? error.message : String(error)}`);
|
|
2511
|
+
});
|
|
2512
|
+
/* `stopListening` is already best-effort and swallows its own failure: a
|
|
2513
|
+
process on its way out must not hang or fail, and the freshness window
|
|
2514
|
+
covers every way a machine can leave without getting here. */
|
|
2515
|
+
return { stop: () => stopListening(client, machineId, machineHarness) };
|
|
2516
|
+
}
|