omp-conductor 0.15.4 → 0.15.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -5
- package/package.json +1 -1
- package/src/board.ts +6 -28
- package/src/fleet.ts +154 -16
- package/src/setup-host.ts +21 -2
- package/src/upgrade.ts +174 -54
package/README.md
CHANGED
|
@@ -251,10 +251,20 @@ command exits without changing anything when all three surfaces already use that
|
|
|
251
251
|
release, the Herdr plugin is pinned to its exact `gitHead`, and the brief is
|
|
252
252
|
current.
|
|
253
253
|
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
254
|
+
The upgrade is host-wide, because everything it replaces is: one daemon serves
|
|
255
|
+
every configured project, so its restart lands on all of them at once. A bare
|
|
256
|
+
`omp-conductor upgrade` therefore drains **every** project's workers and
|
|
257
|
+
refreshes **every** project's brief, and pauses them with the fleet-wide
|
|
258
|
+
sentinel — which leaves any per-project `hold` you set standing when it
|
|
259
|
+
restores dispatch.
|
|
260
|
+
|
|
261
|
+
`--project` narrows that only when it is truthful to do so. When a live daemon
|
|
262
|
+
recorded a single project, the command drains and restarts that project, and an
|
|
263
|
+
explicit `--project` naming a different one is rejected before pause or
|
|
264
|
+
installation. When the daemon serves every configured project and there is more
|
|
265
|
+
than one, `--project` is rejected too: draining one queue and then restarting
|
|
266
|
+
the shared daemon would kill another queue's workers without ever counting
|
|
267
|
+
them. Re-run without the flag.
|
|
258
268
|
|
|
259
269
|
Ticks remain in their existing armed or disarmed state, so an ordinary update
|
|
260
270
|
does not halt the exact pane or require another Telegram arm challenge. Progress
|
|
@@ -2334,7 +2344,7 @@ omp-conductor help
|
|
|
2334
2344
|
| `start` | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then spawn the dispatch loop in the background and wait until it answers `GET /healthz` on `:8787`. Without systemd or that unit it keeps the standalone daemon behaviour. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; if the process dies or never serves, it cleans up and quotes the tail of `daemon.log`. |
|
|
2335
2345
|
| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
|
|
2336
2346
|
| `restart [--now] [--timeout SECONDS] [--port N] [--project NAME]` | Drains the fleet by default: pause new claims, wait until live workers reach `0 / N` (bounded by `--timeout SECONDS`, default 1800 = 30 min), restart, then restore the prior dispatch state. Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. `--now` skips the drain and restarts immediately, orphaning any live runs (old behaviour). A drain that hits `--timeout` restarts nothing and leaves dispatch paused — `omp-conductor resume` lifts it, or re-run `restart` to keep waiting. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
|
|
2337
|
-
| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
|
|
2347
|
+
| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. Host-wide by default: one daemon serves every configured project, so a bare run drains all of them and refreshes every brief. `--project` is rejected when the live daemon serves several projects — draining one queue and restarting the shared daemon would kill another's workers. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
|
|
2338
2348
|
| `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. Active-run lines overlay cooperative worker `paused`/`pausing` from `/healthz` without changing SQLite `running` state or the live worker count. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
|
|
2339
2349
|
| `ledger [--issue N] [--limit N]` | The action audit: every [mediated-verb](#the-mediated-verbs-126) mutation and every next-attempt turn budget. Verb entries include the arguments, decision, named refusal, and resulting SHA. Turn-budget entries remain after an override is replaced or consumed. Reads (`conductor_pr_status`) are absent so polling cannot bury the signal. `--issue` narrows both histories; `--limit` defaults to 50. Recent verb refusals and pending turn overrides also appear in `status`. |
|
|
2340
2350
|
| `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.15.
|
|
3
|
+
"version": "0.15.6",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
package/src/board.ts
CHANGED
|
@@ -4,6 +4,7 @@ import { emitKeypressEvents } from "node:readline";
|
|
|
4
4
|
import { findProject, loadConfig, resolveCaps } from "./config.ts";
|
|
5
5
|
import { statusSnapshotFromStore, type StatusSnapshot } from "./daemon.ts";
|
|
6
6
|
import {
|
|
7
|
+
classifyDaemonProjectHealth,
|
|
7
8
|
codeGraphFromHealthz,
|
|
8
9
|
fleetLayers,
|
|
9
10
|
probeTelegramHealth,
|
|
@@ -926,39 +927,16 @@ export function renderBoard(
|
|
|
926
927
|
async function probeBoardHealth(project: ProjectConfig): Promise<BoardHealthProbe> {
|
|
927
928
|
const layers = fleetLayers(project.name);
|
|
928
929
|
const record = livingDaemon();
|
|
930
|
+
// Same gate as status (#379): never health-check a record pinned to another
|
|
931
|
+
// project, and never invent a second membership convention beside
|
|
932
|
+
// classifyDaemonProjectHealth.
|
|
929
933
|
const wrongRecord = record?.project !== undefined && record.project !== project.name;
|
|
930
934
|
const [telegram, health] = await Promise.all([
|
|
931
935
|
probeTelegramHealth(project.name),
|
|
932
936
|
record === undefined || wrongRecord ? undefined : healthCheck(record.port),
|
|
933
937
|
]);
|
|
934
|
-
|
|
935
|
-
|
|
936
|
-
else if (wrongRecord) daemon = "other-project";
|
|
937
|
-
else if (health?.ok !== true) daemon = "unreachable";
|
|
938
|
-
else {
|
|
939
|
-
try {
|
|
940
|
-
const payload = JSON.parse(health.body ?? "null") as unknown;
|
|
941
|
-
if (payload === null || typeof payload !== "object") {
|
|
942
|
-
daemon = "unreachable";
|
|
943
|
-
} else if (Reflect.get(payload, "project") === project.name) {
|
|
944
|
-
daemon = "ok";
|
|
945
|
-
} else {
|
|
946
|
-
const projects = Reflect.get(payload, "projects");
|
|
947
|
-
daemon =
|
|
948
|
-
Array.isArray(projects) &&
|
|
949
|
-
projects.some(
|
|
950
|
-
(entry) =>
|
|
951
|
-
entry !== null &&
|
|
952
|
-
typeof entry === "object" &&
|
|
953
|
-
Reflect.get(entry, "project") === project.name,
|
|
954
|
-
)
|
|
955
|
-
? "ok"
|
|
956
|
-
: "other-project";
|
|
957
|
-
}
|
|
958
|
-
} catch {
|
|
959
|
-
daemon = "unreachable";
|
|
960
|
-
}
|
|
961
|
-
}
|
|
938
|
+
const classified = classifyDaemonProjectHealth(record, health, project.name);
|
|
939
|
+
const daemon: DaemonBoardState = classified.kind;
|
|
962
940
|
const cachedGraph = daemon === "ok" ? codeGraphFromHealthz(health?.body, project.name) : undefined;
|
|
963
941
|
return {
|
|
964
942
|
health: {
|
package/src/fleet.ts
CHANGED
|
@@ -55,8 +55,10 @@ import {
|
|
|
55
55
|
healthCheck,
|
|
56
56
|
isAlive,
|
|
57
57
|
livingDaemon,
|
|
58
|
+
probeUnit,
|
|
58
59
|
stopDaemon,
|
|
59
60
|
type StopResult,
|
|
61
|
+
type UnitOwnership,
|
|
60
62
|
SYSTEMD_UNIT,
|
|
61
63
|
} from "./lifecycle.ts";
|
|
62
64
|
import { formatRss, rssBytesFromHealthz } from "./host.ts";
|
|
@@ -99,7 +101,7 @@ export const ARM_CHALLENGE_TIMEOUT_MS = 300_000;
|
|
|
99
101
|
const LEGACY_HERDR_SESSION = "fleet";
|
|
100
102
|
|
|
101
103
|
export const LEGACY_HERDR_SESSION_HINT =
|
|
102
|
-
'herdr session "fleet" found — rename it to "conductor" or
|
|
104
|
+
'herdr session "fleet" found — rename it to "conductor" or run omp-conductor setup host to pin HERDR_SESSION=fleet (remove this bridge next minor)';
|
|
103
105
|
|
|
104
106
|
let legacyHerdrSessionHintPrinted = false;
|
|
105
107
|
|
|
@@ -1270,10 +1272,132 @@ export function formatCodeGraphHealth(graph: CodeGraphHealth, now = Date.now()):
|
|
|
1270
1272
|
}
|
|
1271
1273
|
|
|
1272
1274
|
|
|
1275
|
+
/**
|
|
1276
|
+
* Project-scoped reading of a living daemon record + `/healthz` body.
|
|
1277
|
+
*
|
|
1278
|
+
* Board and `status` share this so a host running one daemon for several
|
|
1279
|
+
* projects cannot be called healthy for a project it does not serve (#379).
|
|
1280
|
+
* A third interpretation convention is forbidden.
|
|
1281
|
+
*/
|
|
1282
|
+
export type DaemonProjectHealth =
|
|
1283
|
+
| { kind: "stopped" }
|
|
1284
|
+
| { kind: "ok" }
|
|
1285
|
+
| { kind: "unreachable" }
|
|
1286
|
+
| { kind: "other-project"; serves?: string };
|
|
1287
|
+
|
|
1288
|
+
/**
|
|
1289
|
+
* Facts `formatFleetStatus` needs about the living daemon. Pure input — the
|
|
1290
|
+
* caller probes; the formatter never shells out (#379).
|
|
1291
|
+
*/
|
|
1292
|
+
export type FleetDaemonProbe = {
|
|
1293
|
+
project: DaemonProjectHealth;
|
|
1294
|
+
/** `/healthz` body when the probe is project-ok (rss / overlays). */
|
|
1295
|
+
body?: string;
|
|
1296
|
+
/** systemd ownership of the living record pid; omit when unprobed. */
|
|
1297
|
+
unit?: UnitOwnership;
|
|
1298
|
+
};
|
|
1299
|
+
|
|
1300
|
+
/**
|
|
1301
|
+
* Decide whether a living record + optional `/healthz` answer serve `project`.
|
|
1302
|
+
*
|
|
1303
|
+
* Mirrors the board gate: a record pinned to another project is never probed
|
|
1304
|
+
* further; a multi-project payload is ok only when `project` appears in
|
|
1305
|
+
* `projects[]` (or as the top-level single-project `project` field).
|
|
1306
|
+
*/
|
|
1307
|
+
export function classifyDaemonProjectHealth(
|
|
1308
|
+
record: { project?: string } | undefined,
|
|
1309
|
+
health: { ok: boolean; body?: string } | undefined,
|
|
1310
|
+
project: string,
|
|
1311
|
+
): DaemonProjectHealth {
|
|
1312
|
+
if (record === undefined) return { kind: "stopped" };
|
|
1313
|
+
// Record project is authoritative when set — a single-project leftover from
|
|
1314
|
+
// #376 must not be presented as this project's daemon just because its port
|
|
1315
|
+
// still answers.
|
|
1316
|
+
if (record.project !== undefined && record.project !== project) {
|
|
1317
|
+
return { kind: "other-project", serves: record.project };
|
|
1318
|
+
}
|
|
1319
|
+
if (health?.ok !== true) return { kind: "unreachable" };
|
|
1320
|
+
try {
|
|
1321
|
+
const payload = JSON.parse(health.body ?? "null") as unknown;
|
|
1322
|
+
if (payload === null || typeof payload !== "object") {
|
|
1323
|
+
return { kind: "unreachable" };
|
|
1324
|
+
}
|
|
1325
|
+
if (Reflect.get(payload, "project") === project) return { kind: "ok" };
|
|
1326
|
+
const projects = Reflect.get(payload, "projects");
|
|
1327
|
+
if (
|
|
1328
|
+
Array.isArray(projects) &&
|
|
1329
|
+
projects.some(
|
|
1330
|
+
(entry) =>
|
|
1331
|
+
entry !== null &&
|
|
1332
|
+
typeof entry === "object" &&
|
|
1333
|
+
Reflect.get(entry, "project") === project,
|
|
1334
|
+
)
|
|
1335
|
+
) {
|
|
1336
|
+
return { kind: "ok" };
|
|
1337
|
+
}
|
|
1338
|
+
return { kind: "other-project", serves: servedProjectName(payload) };
|
|
1339
|
+
} catch {
|
|
1340
|
+
return { kind: "unreachable" };
|
|
1341
|
+
}
|
|
1342
|
+
}
|
|
1343
|
+
|
|
1344
|
+
/** Best-effort name of who a foreign `/healthz` payload is actually serving. */
|
|
1345
|
+
function servedProjectName(payload: object): string | undefined {
|
|
1346
|
+
const top = Reflect.get(payload, "project");
|
|
1347
|
+
if (typeof top === "string" && top.length > 0) return top;
|
|
1348
|
+
const projects = Reflect.get(payload, "projects");
|
|
1349
|
+
if (!Array.isArray(projects)) return undefined;
|
|
1350
|
+
for (const entry of projects) {
|
|
1351
|
+
if (entry === null || typeof entry !== "object") continue;
|
|
1352
|
+
const name = Reflect.get(entry, "project");
|
|
1353
|
+
if (typeof name === "string" && name.length > 0) return name;
|
|
1354
|
+
}
|
|
1355
|
+
return undefined;
|
|
1356
|
+
}
|
|
1357
|
+
|
|
1358
|
+
function formatDaemonHealthz(probe: FleetDaemonProbe | undefined): string {
|
|
1359
|
+
if (probe === undefined) return "unprobed";
|
|
1360
|
+
switch (probe.project.kind) {
|
|
1361
|
+
case "stopped":
|
|
1362
|
+
return "stopped";
|
|
1363
|
+
case "ok":
|
|
1364
|
+
return "ok";
|
|
1365
|
+
case "unreachable":
|
|
1366
|
+
return "unreachable — the process is up but not serving";
|
|
1367
|
+
case "other-project":
|
|
1368
|
+
return probe.project.serves === undefined
|
|
1369
|
+
? "other-project"
|
|
1370
|
+
: `other-project — serves ${probe.project.serves}`;
|
|
1371
|
+
}
|
|
1372
|
+
}
|
|
1373
|
+
|
|
1374
|
+
/**
|
|
1375
|
+
* Unit line for a living record. The bare unit name alone used to imply
|
|
1376
|
+
* systemd ownership that was never checked (#379); every branch states the
|
|
1377
|
+
* probe result, and `unknown` is never rendered as owned.
|
|
1378
|
+
*/
|
|
1379
|
+
function formatDaemonUnit(unit: UnitOwnership | undefined, recordPid: number): string {
|
|
1380
|
+
if (unit === undefined) return ` unit ${SYSTEMD_UNIT} unprobed`;
|
|
1381
|
+
switch (unit.kind) {
|
|
1382
|
+
case "active":
|
|
1383
|
+
if (unit.pid === recordPid) {
|
|
1384
|
+
return ` unit ${SYSTEMD_UNIT} systemd-owned`;
|
|
1385
|
+
}
|
|
1386
|
+
// Live unit, different MainPID — the record pid is not systemd's.
|
|
1387
|
+
return ` unit ${SYSTEMD_UNIT} unmanaged — MainPID ${unit.pid}`;
|
|
1388
|
+
case "failed":
|
|
1389
|
+
return ` unit ${SYSTEMD_UNIT} unmanaged — unit failed`;
|
|
1390
|
+
case "inactive":
|
|
1391
|
+
return ` unit ${SYSTEMD_UNIT} inactive`;
|
|
1392
|
+
case "unknown":
|
|
1393
|
+
return ` unit ${SYSTEMD_UNIT} unknown — ${unit.reason}`;
|
|
1394
|
+
}
|
|
1395
|
+
}
|
|
1396
|
+
|
|
1273
1397
|
export function formatFleetStatus(
|
|
1274
1398
|
s: StatusSnapshot,
|
|
1275
1399
|
layers: FleetLayers,
|
|
1276
|
-
|
|
1400
|
+
daemon: FleetDaemonProbe | undefined = undefined,
|
|
1277
1401
|
telegram: TelegramHealth = { kind: "unprobed" },
|
|
1278
1402
|
now = Date.now(),
|
|
1279
1403
|
codeGraph: CodeGraphHealth = { configured: false },
|
|
@@ -1315,20 +1439,17 @@ export function formatFleetStatus(
|
|
|
1315
1439
|
if (!layers.daemon.running || layers.daemon.pid === undefined) {
|
|
1316
1440
|
daemonBlock = "daemon not running";
|
|
1317
1441
|
} else {
|
|
1318
|
-
|
|
1319
|
-
|
|
1320
|
-
|
|
1321
|
-
|
|
1322
|
-
? "ok"
|
|
1323
|
-
: "unreachable — the process is up but not serving";
|
|
1324
|
-
const rss = rssBytesFromHealthz(daemonHealth?.body);
|
|
1442
|
+
// Only trust rss from a project-ok body — a foreign payload's bytes are
|
|
1443
|
+
// not this project's daemon facts (#379).
|
|
1444
|
+
const rss =
|
|
1445
|
+
daemon?.project.kind === "ok" ? rssBytesFromHealthz(daemon.body) : undefined;
|
|
1325
1446
|
daemonBlock = [
|
|
1326
1447
|
"daemon",
|
|
1327
1448
|
` pid ${layers.daemon.pid}`,
|
|
1328
1449
|
` port ${layers.daemon.port ?? "?"}`,
|
|
1329
1450
|
...(rss === undefined ? [] : [` rss ${formatRss(rss)}`]),
|
|
1330
|
-
` healthz ${
|
|
1331
|
-
|
|
1451
|
+
` healthz ${formatDaemonHealthz(daemon)}`,
|
|
1452
|
+
formatDaemonUnit(daemon?.unit, layers.daemon.pid),
|
|
1332
1453
|
].join("\n");
|
|
1333
1454
|
}
|
|
1334
1455
|
|
|
@@ -1504,8 +1625,11 @@ export async function renderStatus(projectName?: string): Promise<string> {
|
|
|
1504
1625
|
const layers = fleetLayers(projectName);
|
|
1505
1626
|
const project = findProject(loadConfig(), projectName);
|
|
1506
1627
|
const rec = livingDaemon();
|
|
1507
|
-
|
|
1508
|
-
|
|
1628
|
+
// Gate the port probe on record membership first — a leftover single-project
|
|
1629
|
+
// daemon from #376 must not be health-checked as if it served us (#379).
|
|
1630
|
+
const wrongRecord = rec?.project !== undefined && rec.project !== project.name;
|
|
1631
|
+
const [rawHealth, telegram, planUsage, github] = await Promise.all([
|
|
1632
|
+
rec === undefined || wrongRecord ? undefined : healthCheck(rec.port),
|
|
1509
1633
|
probeTelegramHealth(projectName),
|
|
1510
1634
|
// Read here rather than in `statusSnapshot`, which is synchronous and used
|
|
1511
1635
|
// by callers that must not shell out. An unmetered project never spawns
|
|
@@ -1516,13 +1640,27 @@ export async function renderStatus(projectName?: string): Promise<string> {
|
|
|
1516
1640
|
// broken report (#188).
|
|
1517
1641
|
fetchRateLimit(),
|
|
1518
1642
|
]);
|
|
1519
|
-
const
|
|
1643
|
+
const projectHealth = classifyDaemonProjectHealth(rec, rawHealth, project.name);
|
|
1644
|
+
// Ownership is probed, never assumed from the unit name constant (#379).
|
|
1645
|
+
const unit = rec === undefined ? undefined : probeUnit();
|
|
1646
|
+
const daemon: FleetDaemonProbe | undefined =
|
|
1647
|
+
rec === undefined
|
|
1648
|
+
? undefined
|
|
1649
|
+
: {
|
|
1650
|
+
project: projectHealth,
|
|
1651
|
+
...(projectHealth.kind === "ok" && rawHealth?.body !== undefined
|
|
1652
|
+
? { body: rawHealth.body }
|
|
1653
|
+
: {}),
|
|
1654
|
+
...(unit === undefined ? {} : { unit }),
|
|
1655
|
+
};
|
|
1656
|
+
const healthBody = projectHealth.kind === "ok" ? rawHealth?.body : undefined;
|
|
1657
|
+
const cached = codeGraphFromHealthz(healthBody, project.name);
|
|
1520
1658
|
const codeGraph = cached ?? (await probeCodeGraph(project));
|
|
1521
|
-
const workerPhases = workerPhasesFromHealthz(
|
|
1659
|
+
const workerPhases = workerPhasesFromHealthz(healthBody, project.name);
|
|
1522
1660
|
return formatFleetStatus(
|
|
1523
1661
|
{ ...s, planUsage, github },
|
|
1524
1662
|
layers,
|
|
1525
|
-
|
|
1663
|
+
daemon,
|
|
1526
1664
|
telegram,
|
|
1527
1665
|
Date.now(),
|
|
1528
1666
|
codeGraph,
|
package/src/setup-host.ts
CHANGED
|
@@ -4,7 +4,7 @@ import { homedir, userInfo } from "node:os";
|
|
|
4
4
|
import { dirname, join } from "node:path";
|
|
5
5
|
import { configPath, loadConfig, resolveCaps, stateDir } from "./config.ts";
|
|
6
6
|
import { isPaused, runDaemon, statusSnapshot, type StatusSnapshot } from "./daemon.ts";
|
|
7
|
-
import { DEFAULT_FLEET_AGENT_NAME } from "./fleet.ts";
|
|
7
|
+
import { DEFAULT_FLEET_AGENT_NAME, DEFAULT_HERDR_SESSION, resolveHerdrSessionWithBridge } from "./fleet.ts";
|
|
8
8
|
import {
|
|
9
9
|
DEFAULT_PORT,
|
|
10
10
|
healthCheck,
|
|
@@ -65,6 +65,12 @@ export interface ServiceRuntime {
|
|
|
65
65
|
packageCli: string;
|
|
66
66
|
conductorHome: string;
|
|
67
67
|
telegramStateDir: string;
|
|
68
|
+
/**
|
|
69
|
+
* Non-default Herdr session to pin into the unit (#394). Omitted when the
|
|
70
|
+
* host uses {@link DEFAULT_HERDR_SESSION}: an always-present line would show
|
|
71
|
+
* as drift on every existing host and add a redundant default.
|
|
72
|
+
*/
|
|
73
|
+
herdrSession?: string;
|
|
68
74
|
}
|
|
69
75
|
|
|
70
76
|
function systemdQuote(value: string): string {
|
|
@@ -237,7 +243,12 @@ function refusal(
|
|
|
237
243
|
}
|
|
238
244
|
|
|
239
245
|
|
|
240
|
-
function defaultServiceRuntime(
|
|
246
|
+
function defaultServiceRuntime(
|
|
247
|
+
telegramStateDir: string,
|
|
248
|
+
// Bridge-resolved session name. Passed in so tests stay hermetic (no herdr
|
|
249
|
+
// spawn) and the renderer never shells out — same threading as telegramStateDir.
|
|
250
|
+
herdrSession: string = resolveHerdrSessionWithBridge(),
|
|
251
|
+
): ServiceRuntime {
|
|
241
252
|
const home = homedir();
|
|
242
253
|
const bun = process.execPath;
|
|
243
254
|
const globalCli = Bun.which("omp-conductor");
|
|
@@ -253,6 +264,9 @@ function defaultServiceRuntime(telegramStateDir: string): ServiceRuntime {
|
|
|
253
264
|
packageCli: join(import.meta.dir, "cli.ts"),
|
|
254
265
|
conductorHome: dirname(configPath()),
|
|
255
266
|
telegramStateDir,
|
|
267
|
+
// Only pin a non-default session. Default hosts keep a unit with no
|
|
268
|
+
// HERDR_SESSION line so setup/upgrade do not invent drift (#394).
|
|
269
|
+
...(herdrSession !== DEFAULT_HERDR_SESSION ? { herdrSession } : {}),
|
|
256
270
|
};
|
|
257
271
|
}
|
|
258
272
|
|
|
@@ -283,6 +297,11 @@ export function renderDaemonService(runtime: ServiceRuntime, totalWorkers: numbe
|
|
|
283
297
|
`Environment=${systemdQuote(`PATH=${runtime.path}`)}`,
|
|
284
298
|
`Environment=${systemdQuote(`OMP_CONDUCTOR_HOME=${runtime.conductorHome}`)}`,
|
|
285
299
|
`Environment=${systemdQuote(`OMP_TELEGRAM_STATE_DIR=${runtime.telegramStateDir}`)}`,
|
|
300
|
+
// Escape hatch for hosts still on the pre-#320 "fleet" session (#394). Only
|
|
301
|
+
// when set: an always-on line would drift every default host on every setup.
|
|
302
|
+
...(runtime.herdrSession === undefined
|
|
303
|
+
? []
|
|
304
|
+
: [`Environment=${systemdQuote(`HERDR_SESSION=${runtime.herdrSession}`)}`]),
|
|
286
305
|
`WorkingDirectory=${systemdPath(stateDir())}`,
|
|
287
306
|
`ExecStart=${command.map(systemdQuote).join(" ")}`,
|
|
288
307
|
"Restart=on-failure",
|
package/src/upgrade.ts
CHANGED
|
@@ -52,6 +52,13 @@ export interface UpgradeDeps {
|
|
|
52
52
|
snapshot(project?: string): { liveWorkers: number };
|
|
53
53
|
layers(project?: string): FleetLayers;
|
|
54
54
|
brief(project?: string): { kind: BriefLayout["kind"]; current: boolean };
|
|
55
|
+
/**
|
|
56
|
+
* Every configured project name, in config order. One daemon serves all of
|
|
57
|
+
* them (the generated unit's ExecStart carries no `--project`), so a
|
|
58
|
+
* transaction that restarts that daemon has to know the whole set rather
|
|
59
|
+
* than the one project a caller happened to name (#389).
|
|
60
|
+
*/
|
|
61
|
+
projectNames(): readonly string[];
|
|
55
62
|
/**
|
|
56
63
|
* The daemon the transaction targets, when one is running. `generation`
|
|
57
64
|
* identifies the exact instance (pid + start time) so a later restart cannot
|
|
@@ -103,6 +110,7 @@ export const DEFAULT_DEPS: UpgradeDeps = {
|
|
|
103
110
|
const live = readFileSync(layout.orchestratorPath, "utf8");
|
|
104
111
|
return { kind: layout.kind, current: live === renderBriefForProject(project, policy) };
|
|
105
112
|
},
|
|
113
|
+
projectNames: () => loadConfig().projects.map((p) => p.name),
|
|
106
114
|
daemonIdentity: () => {
|
|
107
115
|
const daemon = livingDaemon();
|
|
108
116
|
if (daemon === undefined) return { running: false };
|
|
@@ -268,9 +276,95 @@ function previousHerdrInstall(source: string): readonly [string, readonly string
|
|
|
268
276
|
];
|
|
269
277
|
}
|
|
270
278
|
|
|
279
|
+
/**
|
|
280
|
+
* The projects one lifecycle transaction covers, and the single pause sentinel
|
|
281
|
+
* that represents them.
|
|
282
|
+
*
|
|
283
|
+
* Everything an upgrade or a draining restart replaces is host-wide: the
|
|
284
|
+
* Bun-global CLI, the omp plugin, the Herdr plugin, and the one dispatch
|
|
285
|
+
* daemon. That daemon serves *every* configured project — the generated unit's
|
|
286
|
+
* ExecStart carries no `--project` — so its restart lands on all of them at
|
|
287
|
+
* once. The scope therefore names every project whose workers must drain and
|
|
288
|
+
* whose brief must be refreshed, instead of the single project a caller
|
|
289
|
+
* happened to name (#389).
|
|
290
|
+
*
|
|
291
|
+
* `selectors` are what the rest of the module passes back to the project-aware
|
|
292
|
+
* deps. A selector of `undefined` means "the one configured project", which is
|
|
293
|
+
* how a single-project host has always addressed itself: keeping it verbatim
|
|
294
|
+
* means the single-project path still emits exactly the same `brief-upgrade`
|
|
295
|
+
* command and the same one `snapshot` call it did before.
|
|
296
|
+
*
|
|
297
|
+
* `pauseKey` is the sentinel this transaction owns. It is `undefined` for a
|
|
298
|
+
* host-wide scope, which selects the legacy *global* sentinel — and that is
|
|
299
|
+
* precisely the right instrument: `isPaused(name)` consults the global file
|
|
300
|
+
* for every project, so one write pauses the whole host, and removing it lifts
|
|
301
|
+
* exactly this transaction's pause while leaving any per-project operator hold
|
|
302
|
+
* standing.
|
|
303
|
+
*/
|
|
304
|
+
interface UpgradeScope {
|
|
305
|
+
/** Project selectors to address the project-aware deps with, never empty. */
|
|
306
|
+
selectors: readonly (string | undefined)[];
|
|
307
|
+
/** The sentinel this transaction pauses and resumes. */
|
|
308
|
+
pauseKey: string | undefined;
|
|
309
|
+
}
|
|
310
|
+
|
|
311
|
+
/**
|
|
312
|
+
* Decides which projects a transaction covers, and refuses the requests that
|
|
313
|
+
* cannot be honoured truthfully.
|
|
314
|
+
*
|
|
315
|
+
* Two refusals, both about a caller naming a project the daemon does not agree
|
|
316
|
+
* with:
|
|
317
|
+
*
|
|
318
|
+
* - the daemon records a *different* project — it serves that one, so a
|
|
319
|
+
* request aimed at another is simply about a different fleet;
|
|
320
|
+
* - the daemon records *no* project while several are configured. That is not
|
|
321
|
+
* "unknown", it is the multi-project daemon: it serves them all. Narrowing
|
|
322
|
+
* is refused rather than obeyed, because draining one project and then
|
|
323
|
+
* restarting the shared daemon would kill the other projects' live workers
|
|
324
|
+
* without ever counting them (#389, and the shared-daemon rule in #378).
|
|
325
|
+
*
|
|
326
|
+
* A single-configured-project host keeps the historical bare selector, so its
|
|
327
|
+
* behaviour is unchanged whether or not the daemon recorded its name.
|
|
328
|
+
*/
|
|
329
|
+
function resolveScope(deps: UpgradeDeps, verb: "upgrade" | "restart", project?: string): UpgradeScope {
|
|
330
|
+
let configured: readonly string[] = [];
|
|
331
|
+
try {
|
|
332
|
+
configured = deps.projectNames();
|
|
333
|
+
} catch {
|
|
334
|
+
// No readable config is not this function's error to raise: the caller
|
|
335
|
+
// reaches a project-aware dep moments later and fails with the real
|
|
336
|
+
// problem. Treat it as the historical bare case.
|
|
337
|
+
}
|
|
338
|
+
const daemon = deps.daemonIdentity();
|
|
339
|
+
|
|
340
|
+
if (project !== undefined) {
|
|
341
|
+
if (daemon.running && daemon.project !== undefined && daemon.project !== project) {
|
|
342
|
+
throw new Error(
|
|
343
|
+
`active daemon serves ${daemon.project}; refusing to ${verb} --project ${project} while it is running`,
|
|
344
|
+
);
|
|
345
|
+
}
|
|
346
|
+
if (daemon.running && daemon.project === undefined && configured.length > 1) {
|
|
347
|
+
throw new Error(
|
|
348
|
+
`the running daemon serves all ${String(configured.length)} configured projects ` +
|
|
349
|
+
`(${configured.join(", ")}); ${verb} replaces host-wide packages and restarts that one daemon, ` +
|
|
350
|
+
`so it cannot be narrowed to --project ${project} — re-run \`omp-conductor ${verb}\` without --project`,
|
|
351
|
+
);
|
|
352
|
+
}
|
|
353
|
+
return { selectors: [project], pauseKey: project };
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
// A daemon that recorded its project serves that project alone; the rest of
|
|
357
|
+
// the config is somebody else's fleet as far as this restart is concerned.
|
|
358
|
+
if (daemon.running && daemon.project !== undefined) {
|
|
359
|
+
return { selectors: [daemon.project], pauseKey: daemon.project };
|
|
360
|
+
}
|
|
361
|
+
if (configured.length > 1) return { selectors: configured, pauseKey: undefined };
|
|
362
|
+
return { selectors: [undefined], pauseKey: undefined };
|
|
363
|
+
}
|
|
364
|
+
|
|
271
365
|
async function waitForDrain(
|
|
272
366
|
deps: UpgradeDeps,
|
|
273
|
-
|
|
367
|
+
scope: UpgradeScope,
|
|
274
368
|
deadlineAt?: number,
|
|
275
369
|
stale?: () => string | undefined,
|
|
276
370
|
): Promise<void> {
|
|
@@ -282,7 +376,10 @@ async function waitForDrain(
|
|
|
282
376
|
// and nothing it may stop (#377).
|
|
283
377
|
const cancelled = stale?.();
|
|
284
378
|
if (cancelled !== undefined) throw new Error(cancelled);
|
|
285
|
-
|
|
379
|
+
// Every project the scope covers: a host-wide drain that counted only one
|
|
380
|
+
// would report zero while another project's worker was still writing to a
|
|
381
|
+
// worktree the restart is about to orphan (#389).
|
|
382
|
+
const workers = scope.selectors.reduce((n, s) => n + deps.snapshot(s).liveWorkers, 0);
|
|
286
383
|
if (workers === 0) return;
|
|
287
384
|
if (deadlineAt !== undefined && Date.now() >= deadlineAt) {
|
|
288
385
|
throw new Error(
|
|
@@ -321,13 +418,13 @@ interface RestartBegun {
|
|
|
321
418
|
*/
|
|
322
419
|
function restartFenceProblem(
|
|
323
420
|
deps: UpgradeDeps,
|
|
324
|
-
|
|
421
|
+
scope: UpgradeScope,
|
|
325
422
|
begun: RestartBegun,
|
|
326
423
|
): string | undefined {
|
|
327
|
-
if (!deps.layers(
|
|
424
|
+
if (!deps.layers(scope.pauseKey).paused) {
|
|
328
425
|
return "restart cancelled: dispatch was resumed while the drain was in progress — nothing was restarted";
|
|
329
426
|
}
|
|
330
|
-
const owned = deps.pauseState(
|
|
427
|
+
const owned = deps.pauseState(scope.pauseKey);
|
|
331
428
|
if (
|
|
332
429
|
owned === undefined ||
|
|
333
430
|
owned.since !== begun.pauseToken.since ||
|
|
@@ -370,13 +467,17 @@ export async function drainAndRestart(
|
|
|
370
467
|
deps: UpgradeDeps,
|
|
371
468
|
o: { project?: string; timeoutMs: number },
|
|
372
469
|
): Promise<void> {
|
|
373
|
-
|
|
374
|
-
|
|
470
|
+
// Host-wide by default (#389): the daemon this restarts serves every
|
|
471
|
+
// configured project, so the drain counts every project's workers and the
|
|
472
|
+
// pause covers all of them.
|
|
473
|
+
const scope = resolveScope(deps, "restart", o.project);
|
|
474
|
+
const initial = deps.layers(scope.pauseKey);
|
|
475
|
+
if (!initial.paused) deps.setPaused(true, scope.pauseKey);
|
|
375
476
|
// The lock must be provable NOW. A pause that cannot be read as an instance
|
|
376
477
|
// — sentinel malformed or unreadable — would make the whole fence fail open
|
|
377
478
|
// (nothing to compare against), so the transaction refuses before waiting
|
|
378
479
|
// on anything it cannot act upon either way.
|
|
379
|
-
const pauseToken = deps.pauseState(
|
|
480
|
+
const pauseToken = deps.pauseState(scope.pauseKey);
|
|
380
481
|
if (pauseToken === undefined) {
|
|
381
482
|
throw new Error(
|
|
382
483
|
"restart cancelled: cannot prove the restart-owned pause — the active pause sentinel is unreadable or malformed; nothing was restarted",
|
|
@@ -386,14 +487,14 @@ export async function drainAndRestart(
|
|
|
386
487
|
pauseToken,
|
|
387
488
|
daemon: deps.daemonIdentity(),
|
|
388
489
|
};
|
|
389
|
-
const stale = () => restartFenceProblem(deps,
|
|
390
|
-
await waitForDrain(deps,
|
|
490
|
+
const stale = () => restartFenceProblem(deps, scope, begun);
|
|
491
|
+
await waitForDrain(deps, scope, Date.now() + o.timeoutMs, stale);
|
|
391
492
|
// The drain completed; the world may have moved on while it did. Re-prove
|
|
392
493
|
// the pause and the generation an instant before the destructive call.
|
|
393
494
|
const cancelled = stale();
|
|
394
495
|
if (cancelled !== undefined) throw new Error(cancelled);
|
|
395
496
|
await deps.restartDaemon();
|
|
396
|
-
if (!initial.paused) deps.setPaused(false,
|
|
497
|
+
if (!initial.paused) deps.setPaused(false, scope.pauseKey);
|
|
397
498
|
}
|
|
398
499
|
|
|
399
500
|
function recoveryProblem(
|
|
@@ -419,19 +520,32 @@ function recoveryProblem(
|
|
|
419
520
|
async function waitForRecovery(
|
|
420
521
|
deps: UpgradeDeps,
|
|
421
522
|
initial: FleetLayers,
|
|
422
|
-
|
|
523
|
+
scope: UpgradeScope,
|
|
423
524
|
): Promise<void> {
|
|
424
525
|
let problem = "recovery did not settle";
|
|
425
526
|
for (let attempt = 0; attempt < RECOVERY_ATTEMPTS; attempt += 1) {
|
|
426
|
-
|
|
527
|
+
const live = scope.selectors.reduce((n, s) => n + deps.snapshot(s).liveWorkers, 0);
|
|
528
|
+
problem = recoveryProblem(deps.layers(scope.pauseKey), initial, live) ?? "";
|
|
427
529
|
if (problem === "") return;
|
|
428
530
|
await deps.sleep(RECOVERY_POLL_MS);
|
|
429
531
|
}
|
|
430
532
|
throw new Error(`upgrade verification failed: ${problem}`);
|
|
431
533
|
}
|
|
432
534
|
|
|
535
|
+
/** One project's brief, as the transaction found it. */
|
|
536
|
+
interface ScopedBrief {
|
|
537
|
+
/** The selector to address the project-aware deps and the CLI with. */
|
|
538
|
+
selector: string | undefined;
|
|
539
|
+
kind: BriefLayout["kind"];
|
|
540
|
+
current: boolean;
|
|
541
|
+
}
|
|
542
|
+
|
|
433
543
|
/**
|
|
434
|
-
* Brings
|
|
544
|
+
* Brings every in-scope brief up with the freshly installed package.
|
|
545
|
+
*
|
|
546
|
+
* Each project owns its own ORCHESTRATOR.md, so a host-wide upgrade migrates
|
|
547
|
+
* them one at a time rather than leaving every project but one on a floor the
|
|
548
|
+
* installed package no longer emits (#389).
|
|
435
549
|
*
|
|
436
550
|
* `brief-upgrade --migrate --apply` is the cross-version ABI: this code runs
|
|
437
551
|
* from the *old* CLI while the new one is already installed, so the verb has to
|
|
@@ -445,29 +559,27 @@ async function waitForRecovery(
|
|
|
445
559
|
* fleet on a single-file brief whose floor no longer tracks the package —
|
|
446
560
|
* the exact drift this verb exists to end — so it fails loudly and rolls back.
|
|
447
561
|
*/
|
|
448
|
-
async function
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
const msg = err instanceof Error ? err.message : String(err);
|
|
463
|
-
deps.log(`brief: brief-upgrade unavailable in the target CLI (${msg}) — overlay recomposes each tick, continuing`);
|
|
562
|
+
async function upgradeBriefs(deps: UpgradeDeps, briefs: readonly ScopedBrief[]): Promise<void> {
|
|
563
|
+
for (const brief of briefs) {
|
|
564
|
+
const selected = brief.selector === undefined ? [] : ["--project", brief.selector];
|
|
565
|
+
if (brief.kind === "legacy-handwritten") {
|
|
566
|
+
// Ungated on purpose: a hand-written brief cannot be migrated without it.
|
|
567
|
+
await mustRun(deps, "omp-conductor", ["brief-upgrade", "--retrofit", "--apply", ...selected]);
|
|
568
|
+
}
|
|
569
|
+
try {
|
|
570
|
+
await mustRun(deps, "omp-conductor", ["brief-upgrade", "--migrate", "--apply", ...selected]);
|
|
571
|
+
} catch (err) {
|
|
572
|
+
if (brief.kind !== "overlay") throw err;
|
|
573
|
+
const msg = err instanceof Error ? err.message : String(err);
|
|
574
|
+
deps.log(`brief: brief-upgrade unavailable in the target CLI (${msg}) — overlay recomposes each tick, continuing`);
|
|
575
|
+
}
|
|
464
576
|
}
|
|
465
577
|
}
|
|
466
578
|
|
|
467
579
|
async function rollbackUpgrade(
|
|
468
580
|
deps: UpgradeDeps,
|
|
469
581
|
previous: InstalledSurfaces,
|
|
470
|
-
|
|
582
|
+
scope: UpgradeScope,
|
|
471
583
|
installTouched: boolean,
|
|
472
584
|
briefChanged: boolean,
|
|
473
585
|
herdrReloadStarted: boolean,
|
|
@@ -577,7 +689,10 @@ async function rollbackUpgrade(
|
|
|
577
689
|
if (briefChanged) {
|
|
578
690
|
try {
|
|
579
691
|
deps.log("rollback: ORCHESTRATOR.md package floor");
|
|
580
|
-
await
|
|
692
|
+
await upgradeBriefs(
|
|
693
|
+
deps,
|
|
694
|
+
scope.selectors.map((selector) => ({ selector, kind: "overlay" as const, current: false })),
|
|
695
|
+
);
|
|
581
696
|
} catch (err) {
|
|
582
697
|
failures.push(err instanceof Error ? err.message : String(err));
|
|
583
698
|
}
|
|
@@ -626,23 +741,21 @@ export async function upgradeConductor(
|
|
|
626
741
|
}
|
|
627
742
|
|
|
628
743
|
const requested = options.version === undefined ? `${PACKAGE}@latest` : `${PACKAGE}@${options.version}`;
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
throw new Error(
|
|
633
|
-
`active daemon serves ${active}; refusing to upgrade --project ${options.project} while it is running`,
|
|
634
|
-
);
|
|
635
|
-
}
|
|
636
|
-
const project = options.project ?? daemon.project;
|
|
744
|
+
// Host-wide by default (#389): one daemon serves every configured project,
|
|
745
|
+
// so an upgrade that restarts it drains them all and refreshes every brief.
|
|
746
|
+
const scope = resolveScope(deps, "upgrade", options.project);
|
|
637
747
|
|
|
638
748
|
const release = parseRegistry(
|
|
639
749
|
(await mustRun(deps, "npm", ["view", requested, "version", "gitHead", "--json"])).stdout,
|
|
640
750
|
);
|
|
641
|
-
const initial = deps.layers(
|
|
751
|
+
const initial = deps.layers(scope.pauseKey);
|
|
642
752
|
const surfaces = await inspectSurfaces(deps);
|
|
643
|
-
const
|
|
753
|
+
const briefs: ScopedBrief[] = scope.selectors.map((selector) => ({
|
|
754
|
+
selector,
|
|
755
|
+
...deps.brief(selector),
|
|
756
|
+
}));
|
|
644
757
|
const installNeeded = !surfacesCurrent(surfaces, release.version, release.gitHead);
|
|
645
|
-
if (!installNeeded &&
|
|
758
|
+
if (!installNeeded && briefs.every((b) => b.current)) {
|
|
646
759
|
return {
|
|
647
760
|
previousVersion: surfaces.cliVersion,
|
|
648
761
|
...release,
|
|
@@ -673,17 +786,24 @@ export async function upgradeConductor(
|
|
|
673
786
|
// either — the in-memory restore below still covers a clean rollback.
|
|
674
787
|
}
|
|
675
788
|
|
|
676
|
-
|
|
789
|
+
const missing = briefs.find((b) => b.kind === "missing");
|
|
790
|
+
if (missing !== undefined) {
|
|
791
|
+
throw new Error(
|
|
792
|
+
missing.selector === undefined
|
|
793
|
+
? "no ORCHESTRATOR.md exists for the configured project"
|
|
794
|
+
: `no ORCHESTRATOR.md exists for project ${missing.selector}`,
|
|
795
|
+
);
|
|
796
|
+
}
|
|
677
797
|
if (initial.herdr === "unknown") throw new Error("cannot determine whether herdr-fleet.service is active");
|
|
678
798
|
|
|
679
799
|
deps.log(`target release: omp-conductor@${release.version} (${release.gitHead})`);
|
|
680
800
|
if (!initial.paused) {
|
|
681
801
|
deps.log("safety: pausing new issue claims");
|
|
682
|
-
deps.setPaused(true,
|
|
802
|
+
deps.setPaused(true, scope.pauseKey);
|
|
683
803
|
}
|
|
684
804
|
deps.log("drain: waiting for live omp worker sessions");
|
|
685
805
|
try {
|
|
686
|
-
await waitForDrain(deps,
|
|
806
|
+
await waitForDrain(deps, scope);
|
|
687
807
|
} catch (err) {
|
|
688
808
|
const failure = err instanceof Error ? err.message : String(err);
|
|
689
809
|
throw new Error(
|
|
@@ -721,7 +841,7 @@ export async function upgradeConductor(
|
|
|
721
841
|
|
|
722
842
|
deps.log("brief: refreshing managed ORCHESTRATOR.md floor while preserving POLICY.md");
|
|
723
843
|
briefChanged = true;
|
|
724
|
-
await
|
|
844
|
+
await upgradeBriefs(deps, briefs);
|
|
725
845
|
|
|
726
846
|
if (initial.herdr === "active") {
|
|
727
847
|
herdrReloadStarted = true;
|
|
@@ -746,13 +866,13 @@ export async function upgradeConductor(
|
|
|
746
866
|
const active = await mustRun(deps, "systemctl", ["is-active", HERDR_UNIT]);
|
|
747
867
|
if (active.stdout.trim() !== "active") throw new Error(`${HERDR_UNIT} is not active after restart`);
|
|
748
868
|
}
|
|
749
|
-
await waitForRecovery(deps, initial,
|
|
869
|
+
await waitForRecovery(deps, initial, scope);
|
|
750
870
|
deps.log("verify 2/2: recovered fleet remains stable");
|
|
751
871
|
await deps.sleep(1_000);
|
|
752
|
-
await waitForRecovery(deps, initial,
|
|
872
|
+
await waitForRecovery(deps, initial, scope);
|
|
753
873
|
|
|
754
|
-
if (!initial.paused) deps.setPaused(false,
|
|
755
|
-
const restoredDispatch = deps.layers(
|
|
874
|
+
if (!initial.paused) deps.setPaused(false, scope.pauseKey);
|
|
875
|
+
const restoredDispatch = deps.layers(scope.pauseKey).dispatch;
|
|
756
876
|
if (restoredDispatch !== initial.dispatch) {
|
|
757
877
|
throw new Error(
|
|
758
878
|
`dispatch state is ${restoredDispatch}; expected to restore ${initial.dispatch}`,
|
|
@@ -760,9 +880,9 @@ export async function upgradeConductor(
|
|
|
760
880
|
}
|
|
761
881
|
} catch (err) {
|
|
762
882
|
try {
|
|
763
|
-
if (!deps.layers(
|
|
883
|
+
if (!deps.layers(scope.pauseKey).paused) deps.setPaused(true, scope.pauseKey);
|
|
764
884
|
} catch {
|
|
765
|
-
deps.setPaused(true,
|
|
885
|
+
deps.setPaused(true, scope.pauseKey);
|
|
766
886
|
}
|
|
767
887
|
const failure = err instanceof Error ? err.message : String(err);
|
|
768
888
|
deps.log(`upgrade failed: ${failure}`);
|
|
@@ -770,7 +890,7 @@ export async function upgradeConductor(
|
|
|
770
890
|
await rollbackUpgrade(
|
|
771
891
|
deps,
|
|
772
892
|
surfaces,
|
|
773
|
-
|
|
893
|
+
scope,
|
|
774
894
|
installTouched,
|
|
775
895
|
briefChanged,
|
|
776
896
|
herdrReloadStarted,
|