@jbkjr/relay 1.2.2 → 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -12
- package/adapters/claude/relay/.claude-plugin/plugin.json +1 -1
- package/adapters/claude/relay/skills/discuss/SKILL.md +3 -0
- package/adapters/codex/AGENTS.snippet.md +2 -0
- package/adapters/codex/relay/.codex-plugin/plugin.json +1 -1
- package/adapters/codex/relay/skills/relay-discuss/SKILL.md +3 -0
- package/adapters/generate.mjs +6 -1
- package/codex-pin.json +2 -2
- package/dist/build-info.json +2 -2
- package/dist/cli.js +160 -2
- package/dist/confine/canary.js +19 -4
- package/dist/confine/probe.js +11 -3
- package/dist/core/approvals.js +29 -4
- package/dist/core/db.js +44 -2
- package/dist/core/identity.js +43 -3
- package/dist/core/ops.js +9 -9
- package/dist/core/retention.js +66 -12
- package/dist/core/room-clipboard.js +86 -0
- package/dist/core/room-policy.js +17 -0
- package/dist/core/room-terminal.js +227 -0
- package/dist/core/room-types.js +130 -0
- package/dist/core/rooms.js +839 -0
- package/dist/core/secrets.js +4 -0
- package/dist/core/shown.js +19 -3
- package/dist/core/storage.js +2 -2
- package/dist/install/doctor.js +1 -0
- package/dist/mcp/elicit.js +81 -0
- package/dist/mcp/host.js +121 -1
- package/dist/mcp/server.js +159 -11
- package/docs/UPGRADING.md +88 -0
- package/docs/pinned-codex.md +3 -1
- package/npm-shrinkwrap.json +2 -2
- package/package.json +1 -1
- package/scripts/codex-qualify/codex-qualifier.mjs +22 -8
- package/scripts/codex-qualify/codex-stage2.mjs +57 -5
- package/scripts/codex-qualify/stage2-parallel.mjs +200 -0
- package/test/fixtures/stable-surfaces/registry.json +25 -1
- package/test/fixtures/stable-surfaces/room.json +681 -0
- package/test/fixtures/stable-surfaces/tools-desktop.json +57 -0
- package/test/fixtures/stable-surfaces/tools.json +54 -0
- package/test/fixtures/stable-surfaces/usage.txt +11 -1
- package/test/fixtures/stage2-parallel/fake-driver.mjs +34 -0
- package/test/fixtures/stage2-parallel/harness.mjs +8 -0
- package/test/helpers.ts +20 -1
- package/test/never-policy.test.ts +4 -4
- package/test/stable-surfaces.test.ts +125 -1
- package/test/wp4-interlock.test.ts +1 -1
package/docs/UPGRADING.md
CHANGED
|
@@ -106,6 +106,94 @@ What restoring loses: the registry's record of whatever ran under the newer rele
|
|
|
106
106
|
Newest first. Relay 1.1.4 and earlier came before the first public release; their notes are kept
|
|
107
107
|
because later notes and the reference docs point into them.
|
|
108
108
|
|
|
109
|
+
## Relay 1.3.1
|
|
110
|
+
|
|
111
|
+
A fix release. No config or registry migration is needed (the registry stays at schema 5), and Codex
|
|
112
|
+
pins in your store keep working.
|
|
113
|
+
|
|
114
|
+
- **A sandbox check that stalls no longer hides an escape.** Before a confined launch, relay runs a
|
|
115
|
+
canary inside the sandbox that tries what the sandbox must refuse. On a heavily loaded Mac one of
|
|
116
|
+
its steps could hang until relay's 30-second limit, and the launch was then refused as "canary"
|
|
117
|
+
with no results, so a step that did get through went unnamed. The canary now reports at its own
|
|
118
|
+
20-second deadline: a step still running then counts as "timeout", which is never taken as a
|
|
119
|
+
refusal, and every other step's result is kept. When one step escaped, the refusal names that step
|
|
120
|
+
first. A stalled step still refuses the launch, as before.
|
|
121
|
+
|
|
122
|
+
## Relay 1.3.0
|
|
123
|
+
|
|
124
|
+
Rooms: a new `room` tool and `relay room` commands that let two conversations you already have
|
|
125
|
+
open, a Claude Code session and a Codex session say, exchange messages until they agree or state
|
|
126
|
+
a crux ([Rooms](usage.md#rooms)). No config key or host registration changes. **The registry moves
|
|
127
|
+
from schema 4 to schema 5, and schema 5 needs every relay server stopped before the upgrade and
|
|
128
|
+
restarted afterwards**, whether or not you use rooms (below). What changes:
|
|
129
|
+
|
|
130
|
+
- **Rooms.** Create one in your own terminal with `relay room new`, which prints one invitation
|
|
131
|
+
prompt per conversation and, on macOS, offers each on the clipboard in turn; paste each into the
|
|
132
|
+
conversation you mean. In interactive Claude Code in a terminal and the Codex CLI, a conversation
|
|
133
|
+
can instead ask you to open a room, in that host's own dialog; on your yes it takes the first
|
|
134
|
+
seat, and the second seat's prompt goes to your clipboard, never to the model. On those hosts a
|
|
135
|
+
join asks you first too. A room carries information only, at most 24 messages, for at most 24
|
|
136
|
+
hours. Relay launches nobody, and each conversation keeps its own runs and permissions.
|
|
137
|
+
- **`relay room`** lists rooms (`list`), prints a transcript (`<id> --print`), follows one and adds
|
|
138
|
+
terminal lines (`<id>`, `--say`), and re-seats or closes one (`--reseat member-1|member-2`,
|
|
139
|
+
`--close`). Writes need your own interactive terminal.
|
|
140
|
+
- **An approval change for conversations that have read room text** (below).
|
|
141
|
+
- **`relay clean` and daily housekeeping** also expire rooms, and delete an ended room's transcript
|
|
142
|
+
`keep_records_days` after it ended (never, when the setting is unset).
|
|
143
|
+
- **`relay doctor`** has a "rooms" line saying how seats survive restarts and where a conversation
|
|
144
|
+
can open a room.
|
|
145
|
+
- **A question relay is putting to you stays yours while `ps` briefly fails.** When the machine
|
|
146
|
+
refused to start `ps` (under heavy load, say), relay took the server asking the question for gone:
|
|
147
|
+
a waiting call could then hand the agent a request you were being asked about, without
|
|
148
|
+
`user_is_being_asked`, and another relay server could ask you the same question again. Relay now
|
|
149
|
+
counts the asker gone only when it can show that it is.
|
|
150
|
+
|
|
151
|
+
### Upgrading to 1.3.0: stop every relay server first
|
|
152
|
+
|
|
153
|
+
Schema 5 adds six tables (`rooms`, `room_members`, `room_bindings`, `room_invites`,
|
|
154
|
+
`room_messages` and `room_exposures`) and keeps every existing row. As with any migration, relay
|
|
155
|
+
backs the registry up first ([The registry](#the-registry)). An older relay refuses a schema-5
|
|
156
|
+
file when it opens it, but a relay server that is already running keeps its open connection
|
|
157
|
+
without checking the version again, so it cannot be relied on to refuse. So:
|
|
158
|
+
|
|
159
|
+
1. **Stop all of relay.** Let work finish or cancel it, keeping the branches and worktrees of
|
|
160
|
+
finished write jobs you want. Quit every session that uses relay, and the Claude desktop app
|
|
161
|
+
(both of its shared relay processes), and make sure no relay process is left:
|
|
162
|
+
`pgrep -fl 'dist/cli.js (mcp|worker)'` prints nothing. Do this before the new release's
|
|
163
|
+
installer, or any command of the new release, opens the registry.
|
|
164
|
+
2. **Install 1.3.0 and run `relay install <host> --write`** for each host you use, as in
|
|
165
|
+
[How to upgrade](#how-to-upgrade). The first one migrates the registry, backing it up as
|
|
166
|
+
`relay.db.backup-v4-<time>-<pid>`, and prints and records the backup's path; keep it until you
|
|
167
|
+
are sure you will not go back. `sqlite3 "file:$HOME/.local/state/relay/relay.db?mode=rw" 'PRAGMA user_version'`
|
|
168
|
+
then prints 5. A new registry needs no backup.
|
|
169
|
+
3. **Restart every session**, and reopen the desktop app, so that every relay server runs 1.3.0.
|
|
170
|
+
|
|
171
|
+
A room seat held by a Codex or Claude Code session belongs to that session's relay server: when
|
|
172
|
+
the server restarts, the seat is lost, and `relay room <id> --reseat member-1` (or `member-2`)
|
|
173
|
+
prints a replacement prompt. A seat held by a desktop chat or Cowork session continues with its
|
|
174
|
+
session token, in the same relay home.
|
|
175
|
+
|
|
176
|
+
### The approval change
|
|
177
|
+
|
|
178
|
+
Once a conversation's relay owner has received room text from the other member or from a
|
|
179
|
+
terminal, relay records that (as a hash of the owner, kept after the transcript is deleted), and
|
|
180
|
+
`user_confirmed: true` on that owner's `approve` then counts for nothing: it no longer allows a
|
|
181
|
+
request you are being asked about (`asking`) or one whose question you closed (`dismissed`), and no
|
|
182
|
+
longer earns the audit label of a confirmation the agent claimed. Answer those requests yourself,
|
|
183
|
+
through relay's dialog, `confirm` or `relay approve` in a terminal. Everything else `approve` could
|
|
184
|
+
do stays: the root's own judgments where they were allowed, denials and grants; person-only
|
|
185
|
+
requests keep their existing restrictions. This is not a ban on approvals or grants in a
|
|
186
|
+
conversation that has been in a room, it does not make terminal text your consent, and it does not
|
|
187
|
+
follow text copied to another conversation.
|
|
188
|
+
|
|
189
|
+
### Rollback from 1.3.0
|
|
190
|
+
|
|
191
|
+
Follow [Rollback](#rollback), stopping every relay server as for the upgrade, and also close any
|
|
192
|
+
`relay room <id>` you are following in a terminal, which holds the registry open as well:
|
|
193
|
+
`pgrep -fl 'dist/cli.js (mcp|worker|room)'` must print nothing. Then restore the schema-4 backup the
|
|
194
|
+
upgrade made (`PRAGMA user_version` on it prints 4). Rooms and their transcripts, like everything
|
|
195
|
+
else recorded after the backup, are lost from the registry.
|
|
196
|
+
|
|
109
197
|
## Relay 1.2.2
|
|
110
198
|
|
|
111
199
|
A fix release. No config or registry migration is needed, and Codex pins in your store keep working.
|
package/docs/pinned-codex.md
CHANGED
|
@@ -109,7 +109,7 @@ the maintainer's qualification of one Codex release with exactly that relay buil
|
|
|
109
109
|
{
|
|
110
110
|
"format": "relay-shipped-codex-pin-v1",
|
|
111
111
|
"codex": { "version": "0.169.0", "npmIntegrity": "sha512-…" },
|
|
112
|
-
"relay": { "scheme": "relay-build-v2", "version": "1.
|
|
112
|
+
"relay": { "scheme": "relay-build-v2", "version": "1.3.1", "sha256": "…" }
|
|
113
113
|
}
|
|
114
114
|
```
|
|
115
115
|
|
|
@@ -270,6 +270,8 @@ The lock is an exclusive directory with a PID and random owner token. A live PID
|
|
|
270
270
|
|
|
271
271
|
Stage 2 has a two-hour watchdog (`timeoutMs`, the most it can be set to): a full qualification includes one probe that runs for just over an hour (3601 seconds) and takes about 85 minutes. The runner sets `RELAY_TEST_PUBLIC_NET=1` and `RELAY_TEST_ESCALATION_LONG=1` for the driver, since a part left "not run" fails the verdict. Its driver starts in its own process group. The watchdog inventories descendants using macOS `ps -E` (uppercase: include environments), tracks PID/start identities and groups whose leaders have exited, and finds detached/reparented processes by the unique scratch paths in their environments. On timeout or SIGTERM/SIGINT it repeatedly freezes newly found identities until discovery converges, then kills the frozen processes and their groups. It preserves the parent chain until discovery finishes and never resumes frozen children before killing them. It looks again after the kill, stops any newly discovered survivors, and requires a final empty look; zombies have already exited. Inspection failure, nonconvergent discovery or surviving processes fail cleanup and prevent promotion. This is for this trusted model-free harness, not a general hostile-process containment boundary. Real macOS process-tree cleanup and logout behavior require the host checks below; fixture tests alone do not establish them.
|
|
272
272
|
|
|
273
|
+
**Parallel stage 2 (opt-in).** `--parallel` (`codex-qualifier.mjs <config> --run --parallel`) runs stage 2 through `scripts/codex-qualify/stage2-parallel.mjs` instead of the driver alone. Item 30's hour-long call runs in one driver process, started first. Every other item runs in a second driver process, started once item 30's broker has received the call (so the second driver's port probes cannot race the first's startup). A full qualification then takes about an hour instead of about 85 minutes. The coordinator is the one command the watchdog above runs. Each driver has its own scratch directory (`stage2/i30` and `stage2/main`, neither a prefix of the other), relay home, registry and log (`stage2/i30.log`, `stage2/main.log`, copied into `stage2.log` at the end), and runs under a supervisor of its own. Item 30's driver runs only item 30, so no other item's process-global test seam is active in it. The coordinator writes `stage2/stage2-verdict.json`, judged once over all items, with `"mode": "parallel"` (a serial run's says `"serial"`), but only when each driver produced a verdict for exactly its own items. It exits non-zero on any driver, watchdog or cleanup failure, and a driver that fails stops the other. SIGTERM, SIGINT or SIGHUP to the coordinator freezes and kills both drivers' trees. The pin manifest records `stage2Mode`. The parallel layout nests scratch one level deeper, so its `scratchRoot` must be 5 bytes shorter. The serial run stays the default: the LaunchAgent runs it unless `--parallel` is added to its arguments.
|
|
274
|
+
|
|
273
275
|
Failure text goes to `runs/.../runner.log`, `failure.json`, `last-failure.json`, and launchd's stderr/stdout logs. Notifications use `/usr/bin/osascript` with a constant `display notification` script and the message as an argument, never interpolated AppleScript. Notification failure is logged too. Evidence and scratch directories are retained deliberately; monitor disk use and prune only inactive scratch/evidence that no manifest or transaction references. No automatic version eviction runs.
|
|
274
276
|
|
|
275
277
|
**Release gates run the suite's real-Codex tests on the pin too.** The sandbox tests that run a real Codex (`codex sandbox (installed Codex)` in `test/codex-permissions.test.ts`, and C1 in `test/security-boundary.test.ts`) take the absolute path in `RELAY_TEST_CODEX`, as the red team does. A gate sets it to the pinned executable (`versions/<version>-<sha>/bin/codex`, the one it qualifies and red-teams with) and sets `RELAY_REQUIRE_REAL_CODEX=1`, under which a missing `RELAY_TEST_CODEX` fails those tests: they neither skip nor fall back to PATH's or Homebrew's Codex, which update by themselves. Outside gates, with neither set, they use the first `codex` on PATH (Homebrew's for C1) and skip, saying so, when there is none.
|
package/npm-shrinkwrap.json
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jbkjr/relay",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.1",
|
|
4
4
|
"lockfileVersion": 3,
|
|
5
5
|
"requires": true,
|
|
6
6
|
"packages": {
|
|
7
7
|
"": {
|
|
8
8
|
"name": "@jbkjr/relay",
|
|
9
|
-
"version": "1.
|
|
9
|
+
"version": "1.3.1",
|
|
10
10
|
"license": "MIT",
|
|
11
11
|
"dependencies": {
|
|
12
12
|
"@modelcontextprotocol/sdk": "1.32.1",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jbkjr/relay",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.1",
|
|
4
4
|
"description": "Lets coding agents of different model families (Claude Code, OpenAI Codex CLI) call each other: delegate, consult, discuss and review across families, over one MCP server",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Jack Koch",
|
|
@@ -20,11 +20,14 @@ export const NOTIFICATION_SCRIPT = 'on run argv\ndisplay notification (item 1 of
|
|
|
20
20
|
|
|
21
21
|
// The longest socket path the driver's relay turns make, below a scratch root. macOS allows 103 bytes.
|
|
22
22
|
const LONGEST_SOCKET = "/run-XXXXXX/stage2/home/.relay-t/s/relay-probe-XXXXXX/relay.sock";
|
|
23
|
+
// The parallel run (stage2-parallel.mjs) puts each driver one level deeper, the longest at stage2/main.
|
|
24
|
+
const LONGEST_PARALLEL_SOCKET = LONGEST_SOCKET.replace("/stage2/", "/stage2/main/");
|
|
23
25
|
|
|
24
|
-
|
|
26
|
+
/** `parallel`: whether stage 2 runs as two drivers at once (stage2-parallel.mjs), which nests its scratch one level deeper. */
|
|
27
|
+
export function validateConfig(input, { parallel = false } = {}) {
|
|
25
28
|
const cfg = { store: path.join(os.homedir(), ".local/share/relay-codex-pins"), scratchRoot: path.join(os.homedir(), ".rsm/q"), timeoutMs: 2 * 60 * 60_000, ...input };
|
|
26
29
|
for (const key of ["repo", "store", "scratchRoot"]) if (!path.isAbsolute(cfg[key] ?? "")) throw new Error(`${key} must be absolute`);
|
|
27
|
-
if (Buffer.byteLength(cfg.scratchRoot + LONGEST_SOCKET) > 103) throw new Error(`scratchRoot ${cfg.scratchRoot} is too long: relay's unix sockets below it would pass macOS's 103-byte limit`);
|
|
30
|
+
if (Buffer.byteLength(cfg.scratchRoot + (parallel ? LONGEST_PARALLEL_SOCKET : LONGEST_SOCKET)) > 103) throw new Error(`scratchRoot ${cfg.scratchRoot} is too long: relay's unix sockets below it would pass macOS's 103-byte limit`);
|
|
28
31
|
if (cfg.oldBin !== undefined && !path.isAbsolute(cfg.oldBin)) throw new Error("oldBin must be absolute");
|
|
29
32
|
if (!Number.isSafeInteger(cfg.timeoutMs) || cfg.timeoutMs < 1000 || cfg.timeoutMs > 2 * 60 * 60_000) throw new Error("timeoutMs must be 1000..7200000");
|
|
30
33
|
return cfg;
|
|
@@ -117,9 +120,13 @@ function failedQualificationPath(store, attempt) {
|
|
|
117
120
|
return path.join(store, "failed-qualifications", `${key}.json`);
|
|
118
121
|
}
|
|
119
122
|
|
|
120
|
-
/**
|
|
121
|
-
|
|
122
|
-
|
|
123
|
+
/**
|
|
124
|
+
* Holds one updater/rollback lock. Durable evidence is retained on every failure. `parallel` runs
|
|
125
|
+
* stage 2 through stage2-parallel.mjs (item 30's hour-long call beside the other items, in two
|
|
126
|
+
* driver processes) instead of codex-stage2.mjs alone; off unless asked for.
|
|
127
|
+
*/
|
|
128
|
+
export async function runQualifier(input, { services: supplied, supervisor: suppliedSupervisor, version: requested, force = false, signals = true, aqua = true, parallel = false } = {}) {
|
|
129
|
+
const cfg = validateConfig(input, { parallel });
|
|
123
130
|
initStore(cfg.store);
|
|
124
131
|
const release = acquireLock(cfg.store);
|
|
125
132
|
if (!release) return { state: "busy" };
|
|
@@ -181,11 +188,15 @@ export async function runQualifier(input, { services: supplied, supervisor: supp
|
|
|
181
188
|
copyBuild(cfg.repo, tree);
|
|
182
189
|
const driver = path.join(tree, "scripts/codex-qualify/codex-stage2.mjs");
|
|
183
190
|
const stage2Sha256 = sha256(driver);
|
|
191
|
+
// The parallel coordinator is the one command the supervisor runs; it runs two drivers below it.
|
|
192
|
+
const coordinator = path.join(tree, "scripts/codex-qualify/stage2-parallel.mjs");
|
|
193
|
+
const mode = parallel ? "parallel" : "serial";
|
|
184
194
|
const out = path.join(scratch, "stage2");
|
|
195
|
+
log(`Running stage 2 (${mode})`);
|
|
185
196
|
const logFd = fs.openSync(path.join(runDir, "stage2.log"), "wx", 0o600);
|
|
186
197
|
let driverError;
|
|
187
198
|
try {
|
|
188
|
-
await services.command(process.execPath, [driver, tree, out, version, candidate.bin, "--source", source.source, "--old-bin", oldBin], {
|
|
199
|
+
await services.command(process.execPath, [parallel ? coordinator : driver, tree, out, version, candidate.bin, "--source", source.source, "--old-bin", oldBin], {
|
|
189
200
|
cwd: tree, timeoutMs: cfg.timeoutMs, logFd, inspect: true,
|
|
190
201
|
// Items 28 and 30 stay "not run" without these, and judgeVerdict refuses a verdict with any part not run.
|
|
191
202
|
env: { RELAY_TEST_PUBLIC_NET: "1", RELAY_TEST_ESCALATION_LONG: "1" },
|
|
@@ -197,6 +208,8 @@ export async function runQualifier(input, { services: supplied, supervisor: supp
|
|
|
197
208
|
const verdict = JSON.parse(fs.readFileSync(scratchVerdict, "utf8"));
|
|
198
209
|
const verdictPath = path.join(runDir, "stage2-verdict.json");
|
|
199
210
|
atomicJson(verdictPath, verdict);
|
|
211
|
+
// A parallel run's verdict is the coordinator's, never one driver's alone.
|
|
212
|
+
if (parallel && verdict.mode !== "parallel") throw driverError ?? new Error(`stage 2 ran in parallel but its verdict says mode ${verdict.mode}`);
|
|
200
213
|
// A candidate cannot opt into a historical schema: judge against the current one.
|
|
201
214
|
const judged = judgeVerdict(verdict, version, CURRENT_VERDICT_SCHEMA);
|
|
202
215
|
if (!judged.ok) {
|
|
@@ -222,6 +235,7 @@ export async function runQualifier(input, { services: supplied, supervisor: supp
|
|
|
222
235
|
qualifiedAt: new Date().toISOString(), verdictPath, verdictSchema: CURRENT_VERDICT_SCHEMA,
|
|
223
236
|
verdictSha256: sha256(verdictPath), sourceCommit: source.sourceCommit,
|
|
224
237
|
sourceArchiveSha256: source.sourceArchiveSha256, relayBuild, stage2Sha256, nodeVersion: process.version,
|
|
238
|
+
stage2Mode: mode, ...(parallel ? { stage2ParallelSha256: sha256(coordinator) } : {}),
|
|
225
239
|
};
|
|
226
240
|
const target = publishTree(cfg.store, candidate.vendor, manifest);
|
|
227
241
|
const transaction = promote(cfg.store, target, "promote", () => supervisor.check());
|
|
@@ -263,12 +277,12 @@ if (isMain(import.meta.url)) {
|
|
|
263
277
|
const [configFile, action, ...rest] = process.argv.slice(2);
|
|
264
278
|
let options;
|
|
265
279
|
try {
|
|
266
|
-
const { values } = parseArgs({ args: rest, options: { version: { type: "string" }, force: { type: "boolean", default: false } } });
|
|
280
|
+
const { values } = parseArgs({ args: rest, options: { version: { type: "string" }, force: { type: "boolean", default: false }, parallel: { type: "boolean", default: false } } });
|
|
267
281
|
if (!configFile || !["--run", "--rollback"].includes(action) || (action === "--rollback" && rest.length) ||
|
|
268
282
|
(values.version !== undefined && !/^\d+\.\d+\.\d+$/.test(values.version))) throw new Error("invalid arguments");
|
|
269
283
|
options = values;
|
|
270
284
|
} catch {
|
|
271
|
-
console.error("Usage: node codex-qualifier.mjs /absolute/qualifier.json --run [--version X.Y.Z] [--force] | --rollback"); process.exitCode = 2;
|
|
285
|
+
console.error("Usage: node codex-qualifier.mjs /absolute/qualifier.json --run [--version X.Y.Z] [--force] [--parallel] | --rollback"); process.exitCode = 2;
|
|
272
286
|
}
|
|
273
287
|
if (options) {
|
|
274
288
|
try {
|
|
@@ -23,7 +23,8 @@
|
|
|
23
23
|
// --source <dir> a codex-rs checkout at this version's tag, for its bundled model catalog;
|
|
24
24
|
// without it the catalog is fetched from GitHub at the tag
|
|
25
25
|
// --items 1,5,10 only these items (the rest are recorded "not run")
|
|
26
|
-
// Writes <scratch dir>/stage2-verdict.json: { ok, state, why, verdictSchema, version, items }.
|
|
26
|
+
// Writes <scratch dir>/stage2-verdict.json: { ok, state, why, verdictSchema, mode, version, items }.
|
|
27
|
+
// stage2-parallel.mjs runs item 30 and the other items as two of these drivers at once (opt-in).
|
|
27
28
|
import crypto from "node:crypto";
|
|
28
29
|
import fs from "node:fs";
|
|
29
30
|
import http from "node:http";
|
|
@@ -703,7 +704,7 @@ async function runCase(ctx, o) {
|
|
|
703
704
|
...(url ? { environments: [{ environmentId: "relay", cwd: d.wt }] } : {}),
|
|
704
705
|
});
|
|
705
706
|
await o.beforeTurn?.(e);
|
|
706
|
-
await e.rpc.request("turn/start", { threadId: e.thread.thread.id, input: [{ type: "text", text: o.prompt ?? "go", text_elements: [] }] });
|
|
707
|
+
e.turn = await e.rpc.request("turn/start", { threadId: e.thread.thread.id, input: [{ type: "text", text: o.prompt ?? "go", text_elements: [] }] });
|
|
707
708
|
e.broker?.arm();
|
|
708
709
|
e.done = await e.rpc.until((n) => ownTurnCompleted(n, e.thread.thread.id), o.timeoutMs ?? 60_000);
|
|
709
710
|
if (o.during) await o.during(e);
|
|
@@ -1861,6 +1862,7 @@ async function qualificationBroker(ctx, d, o = {}) {
|
|
|
1861
1862
|
},
|
|
1862
1863
|
toolHandler: async () => {
|
|
1863
1864
|
diag.calls++; const at = performance.now();
|
|
1865
|
+
o.onCall?.();
|
|
1864
1866
|
while (performance.now() - at < (o.delayMs ?? 0)) await delay(Math.max(1, o.delayMs - (performance.now() - at)), undefined, { signal: delayStop.signal });
|
|
1865
1867
|
diag.durations.push(performance.now() - at);
|
|
1866
1868
|
// relayServer decorates status by parsing its handler's text as JSON.
|
|
@@ -2262,6 +2264,51 @@ async function item22(ctx) {
|
|
|
2262
2264
|
/** For a run by hand of one case (debugging). */
|
|
2263
2265
|
export const internal = { runCase, relayTurn, stage2Flags, BARE };
|
|
2264
2266
|
|
|
2267
|
+
/** Item 30's long call: conversation 0's second call (a tool search, then the call). */
|
|
2268
|
+
export const LONG_CALL = "c0-1";
|
|
2269
|
+
/**
|
|
2270
|
+
* The file item 30 writes in its scratch directory once its broker has received the long call: the
|
|
2271
|
+
* fake model, the broker, the executor and the app server are all listening by then. The parallel
|
|
2272
|
+
* coordinator (stage2-parallel.mjs) starts the other items only after it appears.
|
|
2273
|
+
*/
|
|
2274
|
+
export const ITEM30_IN_FLIGHT = "item30-in-flight.json";
|
|
2275
|
+
|
|
2276
|
+
/**
|
|
2277
|
+
* What keeps item 30's evidence from showing one result delivered exactly once beyond the hour:
|
|
2278
|
+
* each failed condition, in words (none when it passed). `done` is the turn/completed notification
|
|
2279
|
+
* (null when none came in time), `turnId` the turn turn/start started, `requests` the fake model's
|
|
2280
|
+
* records, `diag` the broker's diagnostics.
|
|
2281
|
+
*/
|
|
2282
|
+
export function item30Problems({ done, turnId, requests, diag, delayMs, toolTimeoutS }) {
|
|
2283
|
+
const p = [];
|
|
2284
|
+
if (!(toolTimeoutS * 1000 > delayMs)) p.push(`relay's tool timeout (${toolTimeoutS} s) is not beyond the call's ${delayMs / 1000} s: the check cannot tell`);
|
|
2285
|
+
if (!done) p.push("the turn did not complete");
|
|
2286
|
+
else {
|
|
2287
|
+
const t = done.params?.turn;
|
|
2288
|
+
if (t?.status !== "completed") p.push(`the turn ended ${t?.status ?? "with no status"}${t?.error?.message ? `: ${String(t.error.message).slice(0, 300)}` : ""}`);
|
|
2289
|
+
if (!turnId || t?.id !== turnId) p.push(`the turn that completed (${t?.id ?? "no id"}) is not the one turn/start started (${turnId ?? "no id"})`);
|
|
2290
|
+
}
|
|
2291
|
+
if (diag.calls !== 1) p.push(`the broker received ${diag.calls} calls, not 1`);
|
|
2292
|
+
if (diag.durations.length !== 1 || !(diag.durations[0] >= delayMs)) p.push(`the broker's answers took ${JSON.stringify(diag.durations.map(Math.round))} ms, not one of at least ${delayMs} ms`);
|
|
2293
|
+
if (diag.fatal.length) p.push(`the broker reported a fatal failure: ${diag.fatal.join("; ")}`);
|
|
2294
|
+
const errors = requests.filter((r) => r.scriptError || r.hookError);
|
|
2295
|
+
if (errors.length) p.push(`the fake model failed: ${errors.map((r) => r.scriptError ?? r.hookError).join("; ")}`);
|
|
2296
|
+
const issued = requests.filter((r) => r.conv === 0 && r.step === 1 && r.sent !== "message").length;
|
|
2297
|
+
if (issued !== 1) p.push(`the fake model issued the long call ${issued} times, not once`);
|
|
2298
|
+
const outputs = (r) => (r.outputs ?? []).filter((o) => o.call_id === LONG_CALL);
|
|
2299
|
+
const carrying = requests.filter((r) => outputs(r).length);
|
|
2300
|
+
if (!carrying.length) p.push("no request to the model carried the long call's result");
|
|
2301
|
+
else {
|
|
2302
|
+
// Each later request repeats the conversation, so it must hold the same single output.
|
|
2303
|
+
const counts = carrying.map((r) => outputs(r).length);
|
|
2304
|
+
if (counts.some((n) => n !== 1)) p.push(`the requests that carried the long call's result held ${JSON.stringify(counts)} outputs for it, not one each`);
|
|
2305
|
+
const first = outputs(carrying[0])[0].text;
|
|
2306
|
+
if (!brokerOK(first)) p.push(`the long call's result is not the broker's answer: ${JSON.stringify(String(first ?? "").slice(0, 300))}`);
|
|
2307
|
+
if (carrying.some((r) => outputs(r).some((o) => o.text !== first))) p.push("the long call's result changed between requests");
|
|
2308
|
+
}
|
|
2309
|
+
return p;
|
|
2310
|
+
}
|
|
2311
|
+
|
|
2265
2312
|
// RELAY_TEST_ESCALATION_LONG=readback, for release gates: the readback alone, with the hour-long
|
|
2266
2313
|
// call recorded as not run, so the verdict can never promote a pin. The full call runs at pin
|
|
2267
2314
|
// qualification and at a gate whose change touches the broker or the tool timeout.
|
|
@@ -2279,9 +2326,12 @@ async function item30(ctx) {
|
|
|
2279
2326
|
await runCase(ctx, { name: "i30-readback", env: "relay", broker: { delayMs, toolTimeoutS }, appEnv: relayEnv(ctx), noTurn: true, beforeThread: readback, conversations: () => [] });
|
|
2280
2327
|
return { readback: part("passed", [], { toolTimeoutS }), long: part("not run", "readback only (RELAY_TEST_ESCALATION_LONG=readback): the hour-long call was not made") };
|
|
2281
2328
|
}
|
|
2282
|
-
const
|
|
2329
|
+
const inFlight = () => fs.writeFileSync(path.join(ctx.dir, ITEM30_IN_FLIGHT), `${JSON.stringify({ at: new Date().toISOString(), pid: process.pid })}\n`);
|
|
2330
|
+
const w = await runCase(ctx, { name: "i30-long", env: "relay", broker: { delayMs, toolTimeoutS, onCall: inFlight }, appEnv: relayEnv(ctx), timeoutMs: delayMs + 120_000,
|
|
2283
2331
|
beforeThread: readback, conversations: () => [{ calls: [brokerSearch(), brokerCall()] }] });
|
|
2284
|
-
|
|
2332
|
+
const g = w.broker.diag;
|
|
2333
|
+
const problems = item30Problems({ done: w.done, turnId: w.turn?.turn?.id, requests: w.model.requests, diag: g, delayMs, toolTimeoutS });
|
|
2334
|
+
return { long: judged(problems, { toolTimeoutS, calls: g.calls, durationMs: g.durations[0], turn: w.done?.params?.turn?.status ?? null, fatal: [...g.fatal], refusals: [...g.refusals] }) };
|
|
2285
2335
|
}
|
|
2286
2336
|
|
|
2287
2337
|
const RUN = { 1: item1, 2: item2, 3: item3, 4: item4, 5: item5, 6: item6, 7: item7, 8: item8, 9: item9, 10: item10, 11: item11, 12: item12, 13: item13, 14: item14, 15: item15, 16: item16, 17: item17, 18: item18, 19: item19, 20: item20, 21: item21, 22: item22, ...Object.fromEntries(Object.keys(ESCALATION_CASES).map(id => [id, async ctx => escalationParts(await runEscalationItem(ctx, id))])), 30: item30 };
|
|
@@ -2367,7 +2417,9 @@ export async function qualify({ checkout, dir, version, bin, oldBin = null, sour
|
|
|
2367
2417
|
const v = verdictOf(items);
|
|
2368
2418
|
if (left.length) v.why.push(`left running after the items, stopped by the sweep: ${left.join(", ")}`);
|
|
2369
2419
|
if (still.length) { v.state = "failed"; v.ok = false; v.why.push(`still running after the sweep: ${still.join(", ")}`); }
|
|
2370
|
-
|
|
2420
|
+
// `mode`: this driver ran its items one after another (stage2-parallel.mjs writes "parallel").
|
|
2421
|
+
// `sweep`: what the sweep found, which the parallel coordinator carries into its own verdict.
|
|
2422
|
+
const verdict = { ...v, mode: "serial", sweep: { left, still }, version, bin, oldBin, model: ctx.model, relayPath: ctx.relayMissing === undefined ? "not tried" : ctx.relayMissing === null ? "present" : ctx.relayMissing.pending ? `pending: ${ctx.relayMissing.pending}` : ctx.relayMissing.noSession ? `not run: no login session (${ctx.relayMissing.noSession})` : `failed: ${ctx.relayMissing.failed}`, items: v.items };
|
|
2371
2423
|
fs.writeFileSync(path.join(dir, "stage2-verdict.json"), `${JSON.stringify(verdict, null, 2)}\n`);
|
|
2372
2424
|
return verdict;
|
|
2373
2425
|
}
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Stage 2 in parallel (opt-in: codex-qualifier.mjs --parallel). Item 30's hour-long call runs in a
|
|
3
|
+
// driver process of its own while a second driver process runs every other item, so a full run takes
|
|
4
|
+
// about an hour instead of about 85 minutes. Same arguments as codex-stage2.mjs, without --items.
|
|
5
|
+
//
|
|
6
|
+
// The two drivers share nothing: each is a separate `node codex-stage2.mjs` process with its own
|
|
7
|
+
// scratch directory (<dir>/i30 and <dir>/main, neither a prefix of the other, so neither driver's
|
|
8
|
+
// `pgrep -f <dir>` sweep or exec-server cwd sweep reaches the other's processes), and so its own
|
|
9
|
+
// HOME, CODEX_HOME, TMPDIR, relay home, registry, config and log. Each runs under its own
|
|
10
|
+
// ProcessSupervisor rooted at its directory, the containment the qualifier uses for the serial
|
|
11
|
+
// driver. Item 30's driver runs only item 30, so no other item's process-global test seam (item 14's
|
|
12
|
+
// stage-2 dependencies, item 15's process.kill) is ever installed in it.
|
|
13
|
+
//
|
|
14
|
+
// Item 30's driver starts first. The other driver starts only once item 30's broker has received
|
|
15
|
+
// the long call (the driver writes ITEM30_IN_FLIGHT then): its fake model, broker, executor and app
|
|
16
|
+
// server are all listening by then, so the second driver's port probes cannot race their startup.
|
|
17
|
+
//
|
|
18
|
+
// The coordinator joins both, checks that each produced a verdict for exactly the items it was
|
|
19
|
+
// given, and judges all items once with codex-stage2.mjs's verdictOf. It writes
|
|
20
|
+
// <dir>/stage2-verdict.json (mode "parallel") only when both drivers wrote such a verdict. A driver
|
|
21
|
+
// that fails (non-zero exit, a signal, a timeout, descendants left behind, a cleanup failure) makes
|
|
22
|
+
// the coordinator exit non-zero, as a failed serial driver makes the qualifier fail; a driver that
|
|
23
|
+
// fails before its verdict, a verdict for the wrong items, and an interruption leave no verdict at
|
|
24
|
+
// all, so the qualifier never records them as a rejected candidate. SIGTERM, SIGINT and SIGHUP
|
|
25
|
+
// cancel both drivers' supervisors, which freeze and kill everything under their directories before
|
|
26
|
+
// the coordinator exits; the qualifier's own watchdog sees every process here by its scratch root.
|
|
27
|
+
import fs from "node:fs";
|
|
28
|
+
import path from "node:path";
|
|
29
|
+
import { setTimeout as delay } from "node:timers/promises";
|
|
30
|
+
import { ProcessSupervisor } from "./codex-qualifier-process.mjs";
|
|
31
|
+
import { ITEMS, ITEM30_IN_FLIGHT, parseArgs, verdictOf } from "./codex-stage2.mjs";
|
|
32
|
+
import { CURRENT_VERDICT_SCHEMA } from "./codex-pin-store.mjs";
|
|
33
|
+
import { isMain } from "./main-module.mjs";
|
|
34
|
+
|
|
35
|
+
export const DRIVER = path.join(import.meta.dirname, "codex-stage2.mjs");
|
|
36
|
+
const NOT_SELECTED = "not selected (--items)";
|
|
37
|
+
/** The scratch variables each driver's supervisor sets below its own directory. */
|
|
38
|
+
const SCRATCH_KEYS = ["HOME", "CODEX_HOME", "TMPDIR", "RELAY_HOME", "RELAY_CONFIG"];
|
|
39
|
+
|
|
40
|
+
/** The two drivers: item 30 alone, and every other item. */
|
|
41
|
+
export const ROLES = Object.freeze({
|
|
42
|
+
i30: Object.freeze({ dir: "i30", items: Object.freeze(["30"]), what: "item 30's driver" }),
|
|
43
|
+
main: Object.freeze({ dir: "main", items: Object.freeze(Object.keys(ITEMS).filter((id) => id !== "30")), what: "the driver of the other items" }),
|
|
44
|
+
});
|
|
45
|
+
|
|
46
|
+
/**
|
|
47
|
+
* What is wrong with a driver's verdict for its role: missing, for another version, binary or
|
|
48
|
+
* schema, an item of its own not run or missing, an item of the other driver's run, or processes
|
|
49
|
+
* its own sweep could not stop. Empty when it is that role's complete verdict.
|
|
50
|
+
*/
|
|
51
|
+
export function childProblems(verdict, { items, version, bin, what = "the driver" }) {
|
|
52
|
+
if (!verdict || typeof verdict !== "object") return [`${what} wrote no verdict`];
|
|
53
|
+
const p = [];
|
|
54
|
+
if (verdict.version !== version) p.push(`${what}'s verdict is for version ${verdict.version}, not ${version}`);
|
|
55
|
+
if (verdict.bin !== bin) p.push(`${what}'s verdict is for ${verdict.bin}, not ${bin}`);
|
|
56
|
+
if (verdict.verdictSchema !== CURRENT_VERDICT_SCHEMA) p.push(`${what}'s verdict has schema ${verdict.verdictSchema}, not ${CURRENT_VERDICT_SCHEMA}`);
|
|
57
|
+
for (const id of Object.keys(ITEMS)) {
|
|
58
|
+
const it = verdict.items?.[id];
|
|
59
|
+
const unselected = it?.state === "not run" && Object.keys(it.parts ?? {}).length === 1 && it.parts.run?.why?.length === 1 && it.parts.run.why[0] === NOT_SELECTED;
|
|
60
|
+
if (items.includes(id)) {
|
|
61
|
+
if (!it) p.push(`${what} recorded no result for item ${id}`);
|
|
62
|
+
else if (unselected) p.push(`${what} did not run item ${id}`);
|
|
63
|
+
} else if (!unselected) p.push(`${what} ran item ${id}, which is not its own`);
|
|
64
|
+
}
|
|
65
|
+
return p;
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/**
|
|
69
|
+
* The run's verdict from both drivers' verdicts (each already checked by childProblems): every item
|
|
70
|
+
* from the driver that ran it, judged once by verdictOf, failed where either driver's own sweep left
|
|
71
|
+
* processes running (as the serial driver's verdict is), with `mode: "parallel"` and each driver's
|
|
72
|
+
* timings and failure. `drivers` is { i30, main }, each { verdict, elapsedMs, error, ... }.
|
|
73
|
+
*/
|
|
74
|
+
export function mergeVerdicts({ version, bin, oldBin, drivers }) {
|
|
75
|
+
const items = {};
|
|
76
|
+
for (const [role, { items: ids }] of Object.entries(ROLES)) for (const id of ids) items[id] = drivers[role].verdict.items[id];
|
|
77
|
+
const v = verdictOf(items);
|
|
78
|
+
const why = [...v.why];
|
|
79
|
+
let failed = false;
|
|
80
|
+
for (const [role, d] of Object.entries(drivers)) {
|
|
81
|
+
const what = ROLES[role].what;
|
|
82
|
+
const sweep = d.verdict.sweep ?? {};
|
|
83
|
+
if (sweep.left?.length) why.push(`${what}: left running after the items, stopped by the sweep: ${sweep.left.join(", ")}`);
|
|
84
|
+
if (sweep.still?.length) { failed = true; why.push(`${what}: still running after the sweep: ${sweep.still.join(", ")}`); }
|
|
85
|
+
if (d.error) why.push(`${what} failed: ${d.error}`);
|
|
86
|
+
}
|
|
87
|
+
if (drivers.i30.verdict.model !== drivers.main.verdict.model) { failed = true; why.push(`the drivers ran different models (${drivers.i30.verdict.model}, ${drivers.main.verdict.model})`); }
|
|
88
|
+
const state = failed ? "failed" : v.state;
|
|
89
|
+
return {
|
|
90
|
+
ok: state === "passed", state, why, verdictSchema: v.verdictSchema, mode: "parallel", version, bin, oldBin,
|
|
91
|
+
model: drivers.main.verdict.model, relayPath: drivers.main.verdict.relayPath,
|
|
92
|
+
drivers: Object.fromEntries(Object.entries(drivers).map(([role, { verdict, ...d }]) => [role, d])),
|
|
93
|
+
items: v.items,
|
|
94
|
+
};
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
const readVerdict = (file) => { try { return JSON.parse(fs.readFileSync(file, "utf8")); } catch { return null; } };
|
|
98
|
+
const message = (e) => String(e?.message ?? e).slice(0, 2000);
|
|
99
|
+
|
|
100
|
+
/**
|
|
101
|
+
* Runs both drivers and writes the run's verdict. `o`: parseArgs's options (no `only`). `opts`:
|
|
102
|
+
* `driver` (the item driver to run, codex-stage2.mjs), `timeoutMs` (each driver's watchdog),
|
|
103
|
+
* `inFlightTimeoutMs` (how long item 30's call may take to reach its broker), `pollMs` (the
|
|
104
|
+
* supervisors' look), `signals` (whether SIGTERM, SIGINT and SIGHUP cancel), `log`.
|
|
105
|
+
* Returns { code, verdict, report }: code 0 only when both drivers ran cleanly and a verdict exists.
|
|
106
|
+
*/
|
|
107
|
+
export async function coordinate(o, { driver = DRIVER, timeoutMs = 2 * 60 * 60_000, inFlightTimeoutMs = 15 * 60_000, pollMs = 1000, signals = true, log = (m) => console.log(m) } = {}) {
|
|
108
|
+
if (o.only?.length) throw new Error("the parallel run runs every item: it takes no --items");
|
|
109
|
+
const started = Date.now();
|
|
110
|
+
fs.rmSync(o.dir, { recursive: true, force: true });
|
|
111
|
+
fs.mkdirSync(o.dir, { recursive: true, mode: 0o700 });
|
|
112
|
+
const dir = fs.realpathSync(o.dir);
|
|
113
|
+
const env = Object.fromEntries(Object.entries(process.env).filter(([k]) => !SCRATCH_KEYS.includes(k)));
|
|
114
|
+
const drivers = {};
|
|
115
|
+
for (const [role, r] of Object.entries(ROLES)) {
|
|
116
|
+
const own = path.join(dir, r.dir);
|
|
117
|
+
fs.mkdirSync(own, { mode: 0o700 });
|
|
118
|
+
drivers[role] = { role, what: r.what, dir: own, log: path.join(dir, `${r.dir}.log`), sup: new ProcessSupervisor(own, { pollMs }), done: false, error: null };
|
|
119
|
+
}
|
|
120
|
+
let interrupted = null;
|
|
121
|
+
const interrupt = (why) => { interrupted ??= why; for (const d of Object.values(drivers)) d.sup.cancel(why); };
|
|
122
|
+
const handlers = { SIGTERM: () => interrupt("the coordinator was interrupted by SIGTERM"), SIGINT: () => interrupt("the coordinator was interrupted by SIGINT"), SIGHUP: () => interrupt("the coordinator was interrupted by SIGHUP") };
|
|
123
|
+
if (signals) for (const [s, f] of Object.entries(handlers)) process.on(s, f);
|
|
124
|
+
const start = (d, extra) => {
|
|
125
|
+
d.startedAt = Date.now();
|
|
126
|
+
const fd = fs.openSync(d.log, "wx", 0o600);
|
|
127
|
+
const args = [driver, o.checkout, d.dir, o.version, o.bin, ...(o.source ? ["--source", o.source] : []), ...extra, "--items", ROLES[d.role].items.join(",")];
|
|
128
|
+
d.run = d.sup.run(process.execPath, args, { cwd: o.checkout, env, timeoutMs, logFd: fd, inspect: true })
|
|
129
|
+
.then(() => {}, (e) => { d.error = message(e); })
|
|
130
|
+
.finally(() => { fs.closeSync(fd); d.done = true; d.elapsedMs = Date.now() - d.startedAt; });
|
|
131
|
+
log(`stage2 parallel: started ${d.what} (items ${ROLES[d.role].items.join(",")}) in ${d.dir}`);
|
|
132
|
+
};
|
|
133
|
+
try {
|
|
134
|
+
const { i30, main } = drivers;
|
|
135
|
+
start(i30, []);
|
|
136
|
+
const marker = path.join(i30.dir, ITEM30_IN_FLIGHT);
|
|
137
|
+
const deadline = Date.now() + inFlightTimeoutMs;
|
|
138
|
+
while (!fs.existsSync(marker) && !i30.done && !interrupted && Date.now() < deadline) await delay(Math.min(250, pollMs));
|
|
139
|
+
if (fs.existsSync(marker)) { i30.inFlightAfterMs = Date.now() - i30.startedAt; log(`stage2 parallel: item 30's call reached its broker after ${Math.round(i30.inFlightAfterMs / 1000)} s`); }
|
|
140
|
+
else if (!i30.done && !interrupted) { i30.sup.cancel(`item 30's long call did not reach its broker within ${Math.round(inFlightTimeoutMs / 1000)} s`); await i30.run; }
|
|
141
|
+
// Item 30's driver may also have finished first, with item 30 failed early: its verdict counts.
|
|
142
|
+
if (!interrupted && !(i30.done && i30.error)) start(main, o.oldBin ? ["--old-bin", o.oldBin] : []);
|
|
143
|
+
// A driver that fails stops the other: the run cannot pass, and an hour need not be spent.
|
|
144
|
+
const running = Object.values(drivers).filter((d) => d.run);
|
|
145
|
+
await Promise.all(running.map((d) => d.run.then(() => { if (d.error) for (const x of running) if (x !== d) x.sup.cancel(`${d.what} failed: ${d.error}`); })));
|
|
146
|
+
// Each supervisor checked its own tree; one last look covers anything named by either directory.
|
|
147
|
+
const all = new ProcessSupervisor(dir, { pollMs });
|
|
148
|
+
let left = [];
|
|
149
|
+
try { left = all.scan(); if (left.length) await all.terminate(); }
|
|
150
|
+
catch (e) { left.push({ pid: `(${message(e)})` }); }
|
|
151
|
+
const report = {
|
|
152
|
+
mode: "parallel", dir, elapsedMs: Date.now() - started, interrupted,
|
|
153
|
+
leftAfterDrivers: left.map((r) => r.pid),
|
|
154
|
+
drivers: Object.fromEntries(Object.entries(drivers).map(([role, d]) => [role, { started: Boolean(d.run), elapsedMs: d.elapsedMs ?? null, inFlightAfterMs: d.inFlightAfterMs ?? null, error: d.error, log: d.log }])),
|
|
155
|
+
};
|
|
156
|
+
for (const d of Object.values(drivers)) {
|
|
157
|
+
if (!d.run) continue;
|
|
158
|
+
d.verdict = readVerdict(path.join(d.dir, "stage2-verdict.json"));
|
|
159
|
+
d.problems = childProblems(d.verdict, { items: ROLES[d.role].items, version: o.version, bin: o.bin, what: d.what });
|
|
160
|
+
report.drivers[d.role].problems = d.problems;
|
|
161
|
+
}
|
|
162
|
+
const complete = !interrupted && Object.values(drivers).every((d) => d.run && !d.problems.length);
|
|
163
|
+
const failures = [
|
|
164
|
+
...(interrupted ? [interrupted] : []),
|
|
165
|
+
...Object.values(drivers).filter((d) => !d.run).map((d) => `${d.what} was not started`),
|
|
166
|
+
...Object.values(drivers).flatMap((d) => [...(d.error ? [`${d.what} failed: ${d.error}`] : []), ...(d.problems ?? [])]),
|
|
167
|
+
...(left.length ? [`left running after both drivers: ${left.map((r) => r.pid).join(", ")}`] : []),
|
|
168
|
+
];
|
|
169
|
+
report.failures = failures;
|
|
170
|
+
fs.writeFileSync(path.join(dir, "parallel-report.json"), `${JSON.stringify(report, null, 2)}\n`);
|
|
171
|
+
let verdict = null;
|
|
172
|
+
if (complete) {
|
|
173
|
+
verdict = mergeVerdicts({
|
|
174
|
+
version: o.version, bin: o.bin, oldBin: o.oldBin,
|
|
175
|
+
drivers: Object.fromEntries(Object.entries(drivers).map(([role, d]) => [role, { verdict: d.verdict, elapsedMs: d.elapsedMs, inFlightAfterMs: d.inFlightAfterMs ?? null, error: d.error }])),
|
|
176
|
+
});
|
|
177
|
+
verdict.elapsedMs = report.elapsedMs;
|
|
178
|
+
if (left.length) verdict.why.push(`left running after both drivers, stopped: ${left.map((r) => r.pid).join(", ")}`);
|
|
179
|
+
fs.writeFileSync(path.join(dir, "stage2-verdict.json"), `${JSON.stringify(verdict, null, 2)}\n`);
|
|
180
|
+
}
|
|
181
|
+
for (const d of Object.values(drivers)) {
|
|
182
|
+
if (!d.run) continue;
|
|
183
|
+
let text = ""; try { text = fs.readFileSync(d.log, "utf8"); } catch {}
|
|
184
|
+
log(`----- ${d.what} (${d.log}) -----\n${text.trimEnd()}`);
|
|
185
|
+
}
|
|
186
|
+
for (const f of failures) process.stderr.write(`stage2 parallel: ${f}\n`);
|
|
187
|
+
return { code: complete && !failures.length ? 0 : 1, verdict, report };
|
|
188
|
+
} finally {
|
|
189
|
+
if (signals) for (const [s, f] of Object.entries(handlers)) process.off(s, f);
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
if (isMain(import.meta.url)) {
|
|
194
|
+
let o;
|
|
195
|
+
try { o = parseArgs(process.argv.slice(2)); if (o.only.length) throw new Error("the parallel run runs every item: it takes no --items"); }
|
|
196
|
+
catch (err) { console.error(err.message); process.exit(2); }
|
|
197
|
+
const { code, verdict } = await coordinate(o);
|
|
198
|
+
if (verdict) console.log(JSON.stringify({ state: verdict.state, mode: verdict.mode, relayPath: verdict.relayPath, items: Object.fromEntries(Object.entries(verdict.items).map(([k, i]) => [k, i.state])), why: verdict.why }, null, 2));
|
|
199
|
+
process.exit(code);
|
|
200
|
+
}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"version":
|
|
2
|
+
"version": 5,
|
|
3
3
|
"tables": [
|
|
4
4
|
{
|
|
5
5
|
"name": "calls",
|
|
@@ -33,6 +33,30 @@
|
|
|
33
33
|
"name": "requests",
|
|
34
34
|
"sql": "CREATE TABLE requests (\n id TEXT PRIMARY KEY, run_id TEXT NOT NULL, node_id TEXT NOT NULL, kind TEXT NOT NULL,\n summary TEXT NOT NULL, reason TEXT, detail_json TEXT, status TEXT NOT NULL,\n decided_by TEXT, note TEXT, created_at INTEGER NOT NULL, decided_at INTEGER\n, scope TEXT, turn_id INTEGER)"
|
|
35
35
|
},
|
|
36
|
+
{
|
|
37
|
+
"name": "room_bindings",
|
|
38
|
+
"sql": "CREATE TABLE room_bindings (\n owner_hash TEXT PRIMARY KEY, room_id TEXT NOT NULL, member_id TEXT NOT NULL,\n generation INTEGER NOT NULL, UNIQUE(room_id,member_id)\n)"
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
"name": "room_exposures",
|
|
42
|
+
"sql": "CREATE TABLE room_exposures (\n owner_hash TEXT PRIMARY KEY, given_at INTEGER NOT NULL\n)"
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
"name": "room_invites",
|
|
46
|
+
"sql": "CREATE TABLE room_invites (\n code_hash TEXT PRIMARY KEY, room_id TEXT NOT NULL, member_id TEXT NOT NULL,\n generation INTEGER NOT NULL, expires_at INTEGER NOT NULL,\n consumed_at INTEGER, consumed_owner_hash TEXT, UNIQUE(room_id,member_id,generation)\n)"
|
|
47
|
+
},
|
|
48
|
+
{
|
|
49
|
+
"name": "room_members",
|
|
50
|
+
"sql": "CREATE TABLE room_members (\n room_id TEXT NOT NULL, id TEXT NOT NULL, ordinal INTEGER NOT NULL,\n generation INTEGER NOT NULL DEFAULT 1, owner_hash TEXT, owner_kind TEXT,\n pid INTEGER, pid_start TEXT, joined_at INTEGER, joined_sender_json TEXT,\n last_call_at INTEGER, call_no INTEGER NOT NULL DEFAULT 0,\n wait_pid INTEGER, wait_start TEXT, wait_nonce TEXT,\n read_times_json TEXT NOT NULL DEFAULT '[]', reseat_times_json TEXT NOT NULL DEFAULT '[]',\n PRIMARY KEY(room_id,id), UNIQUE(room_id,ordinal)\n)"
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"name": "room_messages",
|
|
54
|
+
"sql": "CREATE TABLE room_messages (\n room_id TEXT NOT NULL, seq INTEGER NOT NULL, member_id TEXT, sender_owner_hash TEXT,\n sender_json TEXT NOT NULL, created_at INTEGER NOT NULL, text TEXT NOT NULL,\n stance TEXT CHECK(stance IN ('continue','agree','crux')), request_id TEXT, args_hash TEXT,\n PRIMARY KEY(room_id,seq), UNIQUE(room_id,member_id,request_id)\n)"
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"name": "rooms",
|
|
58
|
+
"sql": "CREATE TABLE rooms (\n id TEXT PRIMARY KEY, state TEXT NOT NULL CHECK(state IN ('waiting','open','ended')),\n created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, ended_at INTEGER,\n revision INTEGER NOT NULL DEFAULT 1, next_ordinal INTEGER NOT NULL DEFAULT 1,\n turns INTEGER NOT NULL DEFAULT 0, max_turns INTEGER NOT NULL DEFAULT 24, ending_json TEXT,\n origin TEXT NOT NULL DEFAULT 'unknown' CHECK(origin IN ('terminal','open','unknown')), purpose TEXT\n)"
|
|
59
|
+
},
|
|
36
60
|
{
|
|
37
61
|
"name": "runs",
|
|
38
62
|
"sql": "CREATE TABLE runs (\n id TEXT PRIMARY KEY, created_at INTEGER NOT NULL, root_cwd TEXT NOT NULL,\n config_json TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'active'\n, root_pid INTEGER, root_start TEXT, root_token TEXT)"
|