omp-conductor 0.4.0 → 0.4.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "omp-conductor",
3
- "version": "0.4.0",
3
+ "version": "0.4.2",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
@@ -513,6 +513,62 @@ lose the substituted coordinates and the overwrite confirmation.
513
513
 
514
514
  ---
515
515
 
516
+ ## Step 5b — offer the credential boundary, and be honest about its cost
517
+
518
+ Ask this once, plainly, because the default is not the safe answer — it is the
519
+ *non-breaking* one:
520
+
521
+ > Should worker and orchestrator sessions run under their own OS accounts, so a
522
+ > session that goes rogue in bash holds no credential capable of merging,
523
+ > pushing, releasing or reading your `gh` config?
524
+
525
+ `credentials.isolation` is `none` unless someone says otherwise, and `status`
526
+ reports such a fleet **unprotected** on every tick. That is deliberate: a fleet
527
+ that refuses to dispatch the moment it upgrades is worse than one that is
528
+ honestly unprotected.
529
+
530
+ Say what it costs before they answer:
531
+
532
+ - **Linux only** for the real thing. macOS dev hosts get `sandbox-exec`, which
533
+ is Apple-deprecated but is still the only stock mechanism that denies a *read*.
534
+ - **Root, once**, to create the accounts and groups.
535
+ - **A service restart**, because supplementary group membership is fixed when a
536
+ process starts.
537
+ - Worktrees and mirrors move to `/var/lib/omp-conductor`. Uninstall becomes two
538
+ paths instead of one.
539
+
540
+ If they say yes, the order matters and getting it wrong looks like a bug:
541
+
542
+ ```bash
543
+ omp-conductor boundary-setup --slots <maxConcurrentWorkers> # read it first
544
+ omp-conductor boundary-setup --slots <maxConcurrentWorkers> | sudo bash
545
+ sudo systemctl restart omp-conductor.service
546
+ ```
547
+
548
+ **Then open a fresh shell before running the wizard.** The setup probe reads its
549
+ *own* process credentials, so a stale login will refuse to offer `per-run` and
550
+ that reads like a broken install rather than an un-refreshed session.
551
+
552
+ Only then run `/conductor setup` and choose `per-run`. That single amend writes
553
+ the isolation, relocates the roots, and regenerates the systemd unit *with* the
554
+ capabilities the launcher needs — the unit is not optional, and a fleet
555
+ configured `per-run` whose unit lacks them probes down to `group-mode` and
556
+ refuses every issue. Install the unit with the commands the wizard prints, then
557
+ confirm:
558
+
559
+ ```bash
560
+ omp-conductor status # boundary: per-run — uid-pool
561
+ ```
562
+
563
+ `status` reports the **daemon's** boundary, read from `/healthz`, not a probe of
564
+ whatever shell you are standing in — capabilities belong to the process, and an
565
+ interactive shell has none of the unit's. If no daemon is running it falls back
566
+ to a local probe and says so; that line is not the fleet's answer.
567
+
568
+ If it says anything else, the reason names the missing capability, group or
569
+ path. Do not guess at it, and never "fix" it by widening the state directory —
570
+ that is where `conductor.db` lives.
571
+
516
572
  ## Step 6 — check the worker brief's assumptions against reality
517
573
 
518
574
  The worker brief makes concrete claims to a session that has no other context. If
@@ -23,6 +23,33 @@ drain, exact-version installation across the Bun-global CLI, omp plugin, and
23
23
  Herdr plugin, managed-brief recomposition, process reload, pane recovery,
24
24
  two-pass verification, and restoration of the previous dispatch state.
25
25
 
26
+ ## When it fails on the systemd unit
27
+
28
+ Since 0.4.0 the command also verifies the unit systemd actually booted from, not
29
+ just the staged copy. A fleet configured `credentials.isolation: "per-run"`
30
+ whose installed unit predates the capability grant probes down to `group-mode`
31
+ and then refuses every issue — and package identities, services, pane and ticks
32
+ all look perfectly healthy around it, which is why it needed its own check.
33
+
34
+ The error names the missing directives. Fix it by re-staging rather than by
35
+ hand-editing the live unit:
36
+
37
+ ```bash
38
+ # in an omp session
39
+ /conductor setup # amend nothing; it re-renders the unit
40
+ # then run the install commands it prints, and:
41
+ sudo systemctl daemon-reload && sudo systemctl restart omp-conductor.service
42
+ omp-conductor upgrade # re-run; it now verifies clean
43
+ ```
44
+
45
+ Only directives this configuration *requires* are checked, so an operator's own
46
+ `MemoryMax`, `After=` or extra `Environment=` never fails an upgrade.
47
+
48
+ What the command deliberately does **not** do is create the accounts and groups
49
+ the boundary needs. That is `omp-conductor boundary-setup`, it needs root, and
50
+ the daemon runs unprivileged on purpose — so it detects and instructs rather
51
+ than escalating.
52
+
26
53
  Report the command's result. Do not reproduce its lifecycle as an AI checklist,
27
54
  edit installed files, publish npm, or substitute separate install/restart steps.
28
55
  If it fails, report the exact error and leave dispatch paused as the command
package/src/config.ts CHANGED
@@ -219,6 +219,18 @@ export function loadConfig(): ConductorConfig {
219
219
  * a truncated one. Mode 0600 because a config carries chat ids and clone URLs.
220
220
  */
221
221
  export function saveConfig(c: ConductorConfig): void {
222
+ writeConfigFile(c);
223
+ }
224
+
225
+ /**
226
+ * Atomic write of whatever object is handed in.
227
+ *
228
+ * Takes `unknown` rather than `ConductorConfig` so the credentials migration
229
+ * can persist the operator's own parsed file with one key added, instead of a
230
+ * value that has been through the loader and back out. That round-trip is what
231
+ * rewrote a live fleet's config into a dialect the previous release rejected.
232
+ */
233
+ export function writeConfigFile(c: unknown): void {
222
234
  const dir = stateDir();
223
235
  const created = mkdirSync(dir, { recursive: true, mode: 0o700 });
224
236
  // mkdir's mode is masked by umask; chmod only what we just created so an
@@ -931,9 +943,39 @@ export function migrateCredentialsOnDisk(): { migrated: string[]; path: string }
931
943
  }
932
944
  const migrated = projectsMissingCredentials(parsed);
933
945
  if (migrated.length === 0) return { migrated: [], path };
934
- // Through the loader and back out, so the rewrite is the same normalisation
935
- // every other key already gets rather than a second, divergent writer.
936
- saveConfig(loadConfig());
946
+
947
+ // A SURGICAL edit: add the one missing key to the projects that lack it and
948
+ // leave every other byte exactly as the operator wrote it.
949
+ //
950
+ // This was `saveConfig(loadConfig())` once, and that round-trip took a live
951
+ // fleet down. Normalising on the way out rewrites the whole file into the
952
+ // current dialect — `defaults.planUsage`, `releasePolicy` as a per-shape map
953
+ // — and the previous release rejects both. The daemon performing the
954
+ // migration is started *by* an upgrade still running the previous CLI, so
955
+ // that CLI's next `loadConfig` failed, and its rollback reinstalled a version
956
+ // which then could not read the file either. A migration must never make a
957
+ // config unreadable by the release you may have to roll back to.
958
+ // Verified against the real previous release, not just against this parser:
959
+ // omp-conductor@0.3.25 loads a config carrying `credentials` and prints
960
+ // `status` normally, because it ignores unknown *project* keys while
961
+ // rejecting unknown `defaults` keys and a non-string `releasePolicy` — the
962
+ // two things the old round-trip wrote. Re-check that with `npm install
963
+ // omp-conductor@<previous>` before widening what this writes.
964
+ if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return { migrated: [], path };
965
+ const projects = (parsed as Raw)["projects"];
966
+ if (!Array.isArray(projects)) return { migrated: [], path };
967
+ for (const entry of projects) {
968
+ if (typeof entry !== "object" || entry === null || Array.isArray(entry)) continue;
969
+ const raw = entry as Raw;
970
+ const credentials = raw["credentials"];
971
+ const explicit =
972
+ typeof credentials === "object" &&
973
+ credentials !== null &&
974
+ !Array.isArray(credentials) &&
975
+ (credentials as Raw)["isolation"] !== undefined;
976
+ if (!explicit) (raw as Record<string, unknown>)["credentials"] = { isolation: "none" };
977
+ }
978
+ writeConfigFile(parsed);
937
979
  return { migrated, path };
938
980
  }
939
981
 
package/src/daemon.ts CHANGED
@@ -60,7 +60,7 @@ import type {
60
60
  Tracker,
61
61
  VerbLedgerEntry,
62
62
  } from "./types.ts";
63
- import { CONDUCTOR_GROUPS } from "./types.ts";
63
+ import { CONDUCTOR_GROUPS, type IsolationMechanism } from "./types.ts";
64
64
  import { type KilledBy, type WorkerResult, renderBrief, runWorker } from "./worker.ts";
65
65
  import {
66
66
  addRunRepo,
@@ -2353,6 +2353,20 @@ export interface DaemonHealthSnapshot {
2353
2353
  rssBytes: number;
2354
2354
  dispatch?: DispatchSummary;
2355
2355
  codeGraph?: CodeGraphHealth;
2356
+ /**
2357
+ * The boundary THIS daemon resolved at startup.
2358
+ *
2359
+ * Published because capabilities are a property of the process, not the host:
2360
+ * `omp-conductor status` runs in an interactive shell with none of the unit's
2361
+ * ambient capabilities and no conductor groups, so re-probing there reports
2362
+ * `group-mode` on a fleet that is genuinely `uid-pool`. The operator's
2363
+ * verification step would say the boundary failed while it was working.
2364
+ *
2365
+ * It is also the only answer that survives a `daemon-reload` without a
2366
+ * restart: the config on disk can say anything, this is what the running
2367
+ * process actually got.
2368
+ */
2369
+ boundary?: { isolation: CredentialIsolation; mechanism: IsolationMechanism; headline: string };
2356
2370
  }
2357
2371
 
2358
2372
  export function daemonHealthSnapshot(
@@ -2361,6 +2375,7 @@ export function daemonHealthSnapshot(
2361
2375
  paused = isPaused(),
2362
2376
  rssBytes = process.memoryUsage().rss,
2363
2377
  codeGraph?: CodeGraphHealth,
2378
+ boundary?: DaemonHealthSnapshot["boundary"],
2364
2379
  ): DaemonHealthSnapshot {
2365
2380
  const dispatch = store.latestDispatch(project);
2366
2381
  return {
@@ -2371,6 +2386,7 @@ export function daemonHealthSnapshot(
2371
2386
  rssBytes,
2372
2387
  ...(dispatch === undefined ? {} : { dispatch }),
2373
2388
  ...(codeGraph?.configured === true ? { codeGraph } : {}),
2389
+ ...(boundary === undefined ? {} : { boundary }),
2374
2390
  };
2375
2391
  }
2376
2392
 
@@ -3174,7 +3190,12 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
3174
3190
  project: project.name,
3175
3191
  store,
3176
3192
  turnLimits,
3177
- health: () => daemonHealthSnapshot(store, project.name, undefined, undefined, codeGraph),
3193
+ health: () =>
3194
+ daemonHealthSnapshot(store, project.name, undefined, undefined, codeGraph, {
3195
+ isolation: boundary.isolation,
3196
+ mechanism: boundary.probe.mechanism,
3197
+ headline: describeBoundary(boundary.isolation, boundary.probe).headline,
3198
+ }),
3178
3199
  }),
3179
3200
  });
3180
3201
  log(`serving /healthz on :${server.port}, project ${project.name}`);
package/src/fleet.ts CHANGED
@@ -1112,11 +1112,32 @@ export async function renderStatus(projectName?: string): Promise<string> {
1112
1112
  ]);
1113
1113
  const cached = codeGraphFromHealthz(health?.body, project.name);
1114
1114
  const codeGraph = cached ?? (await probeCodeGraph(project));
1115
- // Probed here, per `status` call, rather than read off the daemon: an
1116
- // operator asking whether their fleet is protected must get the answer for
1117
- // the host they are standing on, including when no daemon is running.
1115
+ // The daemon's own answer wins, and this is not an optimisation.
1116
+ //
1117
+ // Capabilities are a property of the PROCESS, not the host. `status` runs in
1118
+ // an interactive shell with none of the unit's ambient capabilities and none
1119
+ // of the conductor groups, so re-probing here reports `group-mode` — or
1120
+ // "REFUSING DISPATCH" — on a fleet whose daemon is genuinely `uid-pool`. An
1121
+ // operator following the documented verification step would conclude the
1122
+ // boundary had failed while it was working perfectly.
1123
+ //
1124
+ // It is also the only answer that survives a `daemon-reload` without a
1125
+ // restart, which no config check can see: this is what the running process
1126
+ // actually got at startup.
1127
+ //
1128
+ // Falling back to a local probe when no daemon is running is still right —
1129
+ // there is a real question to answer then — but it is labelled, because it is
1130
+ // a different question.
1118
1131
  const credentials = resolveCredentials(project);
1119
- const probe = await probeHost({ slots: s.caps.maxConcurrentWorkers });
1132
+ const live = boundaryFromHealthz(health?.body);
1133
+ const boundary =
1134
+ live ?? {
1135
+ ...describeBoundary(credentials.isolation, await probeHost({ slots: s.caps.maxConcurrentWorkers })),
1136
+ detail: [
1137
+ "No daemon is running, so this is THIS SHELL's view, not the fleet's — an interactive shell holds",
1138
+ "none of the unit's capabilities. Start the daemon and re-read before concluding anything.",
1139
+ ],
1140
+ };
1120
1141
  return formatFleetStatus(
1121
1142
  { ...s, planUsage },
1122
1143
  layers,
@@ -1125,7 +1146,7 @@ export async function renderStatus(projectName?: string): Promise<string> {
1125
1146
  Date.now(),
1126
1147
  codeGraph,
1127
1148
  confinementRefusalsToday(),
1128
- describeBoundary(credentials.isolation, probe),
1149
+ boundary,
1129
1150
  );
1130
1151
  }
1131
1152
 
@@ -1419,3 +1440,21 @@ function probeOmpPane(
1419
1440
  return { kind: "unknown", reason: err instanceof Error ? err.message : String(err) };
1420
1441
  }
1421
1442
  }
1443
+
1444
+ /**
1445
+ * The boundary the running daemon resolved, out of its `/healthz` body.
1446
+ *
1447
+ * Same posture as `codeGraphFromHealthz`: an older daemon that predates the
1448
+ * field simply does not answer, and the caller falls back to a local probe with
1449
+ * that difference stated rather than papered over.
1450
+ */
1451
+ export function boundaryFromHealthz(
1452
+ body: unknown,
1453
+ ): { headline: string; detail: string[] } | undefined {
1454
+ if (typeof body !== "object" || body === null) return undefined;
1455
+ const boundary = (body as { boundary?: unknown }).boundary;
1456
+ if (typeof boundary !== "object" || boundary === null) return undefined;
1457
+ const headline = (boundary as { headline?: unknown }).headline;
1458
+ if (typeof headline !== "string" || headline.length === 0) return undefined;
1459
+ return { headline, detail: [] };
1460
+ }
package/src/setup-host.ts CHANGED
@@ -110,11 +110,14 @@ export function renderDaemonService(
110
110
  `Environment=${systemdQuote(`PATH=${runtime.path}`)}`,
111
111
  `Environment=${systemdQuote(`OMP_CONDUCTOR_HOME=${runtime.conductorHome}`)}`,
112
112
  `Environment=${systemdQuote(`OMP_TELEGRAM_STATE_DIR=${runtime.telegramStateDir}`)}`,
113
- // Baked in so a custom shared root survives systemd's clean environment.
114
- // Without it the daemon resolves the platform default while the operator
115
- // provisioned somewhere else, and every per-run dispatch is refused for a
116
- // path nobody chose (#125).
117
- `Environment=${systemdQuote(`OMP_CONDUCTOR_SHARED=${sharedRoot()}`)}`,
113
+ // Both of these are per-run only, and grouped so the managed set stays one
114
+ // idea. The shared root is baked in so a custom location survives systemd's
115
+ // clean environment without it the daemon resolves the platform default
116
+ // while the operator provisioned somewhere else, and every dispatch is
117
+ // refused for a path nobody chose. It is meaningless under `none`, where
118
+ // nothing resolves it, and rendering it there would make every unit
119
+ // installed before 0.4.0 look drifted for no reason (#125).
120
+ //
118
121
  // The capabilities exist to be DROPPED INTO run children by the
119
122
  // privilege-dropping launcher, never inherited by them — see `setprivArgv`.
120
123
  // Rendered only for `per-run`, because that is the only isolation that
@@ -124,6 +127,7 @@ export function renderDaemonService(
124
127
  // failure an operator following setup would otherwise hit first.
125
128
  ...(resolveCredentials(project).isolation === "per-run"
126
129
  ? [
130
+ `Environment=${systemdQuote(`OMP_CONDUCTOR_SHARED=${sharedRoot()}`)}`,
127
131
  "AmbientCapabilities=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP",
128
132
  "CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_SETPCAP",
129
133
  ]
@@ -301,3 +305,142 @@ export async function runSetupSmoke(
301
305
  await deps.stop();
302
306
  }
303
307
  }
308
+
309
+ /** Absolute path of the unit systemd actually reads. */
310
+ export function installedUnitPath(): string {
311
+ return join(SYSTEMD_UNIT_DIR, STAGED_SERVICE_NAME);
312
+ }
313
+
314
+ /**
315
+ * Why the *live* systemd unit no longer matches what this version would render.
316
+ *
317
+ * `planHostRuntime` compares against the **staged** copy under the state dir,
318
+ * which says nothing about the file systemd booted from. That gap is not
319
+ * cosmetic since 0.4.0: a fleet configured `per-run` whose installed unit
320
+ * predates the capability grant probes down to `group-mode` and then refuses
321
+ * every dispatch — and `upgrade` would happily verify package identities,
322
+ * services, pane and ticks around it, because none of those look at the unit.
323
+ *
324
+ * Checked by directive rather than by whole-file equality on purpose. An
325
+ * operator is entitled to hand-tune `MemoryMax`, add `After=`, or set an
326
+ * `Environment=` of their own, and failing an upgrade over that would teach
327
+ * them to stop running upgrades. What is reported is the set of directives this
328
+ * configuration *requires* and the live unit lacks.
329
+ *
330
+ * `undefined` means either "no drift that matters" or "no installed unit at
331
+ * all" — a host running `omp-conductor start` without systemd is not broken,
332
+ * and must not be told it is.
333
+ */
334
+ /**
335
+ * What systemd has actually loaded for the unit, as `systemctl show` reports it.
336
+ *
337
+ * Deliberately not the file on disk. A unit installed without `daemon-reload`
338
+ * can match the rendered text byte for byte while the manager still runs the
339
+ * previous configuration, and a drop-in under `…service.d/` changes what runs
340
+ * without touching the file at all. Both read clean from the filesystem.
341
+ */
342
+ export interface EffectiveUnit {
343
+ /** `systemctl show -p AmbientCapabilities`; empty string when unset. */
344
+ ambientCapabilities: string;
345
+ capabilityBoundingSet: string;
346
+ /** `systemctl show -p Environment`, newline- or space-joined. */
347
+ environment: string;
348
+ needDaemonReload: boolean;
349
+ }
350
+
351
+ /** Capability names as a comparable set: systemd reports them lowercased, `cap_`-prefixed and reordered. */
352
+ function capabilitySet(text: string): Set<string> {
353
+ return new Set(
354
+ text
355
+ .split(/[\s,]+/)
356
+ .map((name) => name.trim().toLowerCase().replace(/^cap_/, ""))
357
+ .filter((name) => name.length > 0),
358
+ );
359
+ }
360
+
361
+ function renderedDirective(rendered: string, key: string): string | undefined {
362
+ for (const line of rendered.split("\n")) {
363
+ const trimmed = line.trim();
364
+ if (trimmed.startsWith(`${key}=`)) return trimmed.slice(key.length + 1);
365
+ }
366
+ return undefined;
367
+ }
368
+
369
+ /**
370
+ * Why what systemd loaded no longer matches what this version would render.
371
+ *
372
+ * Three asymmetries, each for a reason:
373
+ *
374
+ * - **Ambient capabilities are compared exactly, both ways.** Missing them
375
+ * breaks a per-run fleet; *leftover* ones are worse. Roll back to `none` and
376
+ * there is no `setpriv` launcher at all, so ambient capabilities on the unit
377
+ * are inherited straight into model-executed code, which can then `setuid()`
378
+ * to any account. `systemctl` reports an empty string when unset, so the
379
+ * rollback case is unambiguous.
380
+ * - **The bounding set is checked only when this version sets it.** systemd
381
+ * reports the full default set when a unit does not, so comparing both ways
382
+ * would flag every ordinary host.
383
+ * - **Only directives this configuration manages are considered**, so an
384
+ * operator's own `MemoryMax`, `After=` or drop-in never fails an upgrade.
385
+ */
386
+ export function unitDriftReason(rendered: string, effective: EffectiveUnit | undefined): string | undefined {
387
+ if (effective === undefined) return undefined;
388
+ const problems: string[] = [];
389
+
390
+ const wantAmbient = capabilitySet(renderedDirective(rendered, "AmbientCapabilities") ?? "");
391
+ const haveAmbient = capabilitySet(effective.ambientCapabilities);
392
+ const missingAmbient = [...wantAmbient].filter((c) => !haveAmbient.has(c));
393
+ const extraAmbient = [...haveAmbient].filter((c) => !wantAmbient.has(c));
394
+ if (missingAmbient.length > 0) {
395
+ problems.push(`AmbientCapabilities is missing ${missingAmbient.join(", ")}`);
396
+ }
397
+ if (extraAmbient.length > 0) {
398
+ problems.push(
399
+ `AmbientCapabilities grants ${extraAmbient.join(", ")} that this configuration does not — with ` +
400
+ `isolation off there is no privilege-dropping launcher, so these are inherited by model-executed code`,
401
+ );
402
+ }
403
+
404
+ const wantBounding = renderedDirective(rendered, "CapabilityBoundingSet");
405
+ if (wantBounding !== undefined) {
406
+ const missing = [...capabilitySet(wantBounding)].filter(
407
+ (c) => !capabilitySet(effective.capabilityBoundingSet).has(c),
408
+ );
409
+ if (missing.length > 0) problems.push(`CapabilityBoundingSet is missing ${missing.join(", ")}`);
410
+ }
411
+
412
+ const wantShared = rendered
413
+ .split("\n")
414
+ .map((l) => l.trim())
415
+ .find((l) => l.includes("OMP_CONDUCTOR_SHARED="));
416
+ if (wantShared !== undefined) {
417
+ const value = wantShared.slice(wantShared.indexOf("OMP_CONDUCTOR_SHARED=")).replace(/"$/, "");
418
+ if (!effective.environment.includes(value)) {
419
+ problems.push(`Environment is missing ${value}`);
420
+ }
421
+ }
422
+
423
+ if (effective.needDaemonReload) {
424
+ problems.push("systemd reports the unit needs a daemon-reload, so what runs is not what is on disk");
425
+ }
426
+
427
+ if (problems.length === 0) return undefined;
428
+ return (
429
+ `what systemd loaded for ${STAGED_SERVICE_NAME} does not match this configuration: ${problems.join("; ")}.`
430
+ );
431
+ }
432
+
433
+ /** Parse `systemctl show -p …` key=value output into an {@link EffectiveUnit}. */
434
+ export function parseEffectiveUnit(stdout: string): EffectiveUnit {
435
+ const values = new Map<string, string>();
436
+ for (const line of stdout.split("\n")) {
437
+ const at = line.indexOf("=");
438
+ if (at > 0) values.set(line.slice(0, at).trim(), line.slice(at + 1).trim());
439
+ }
440
+ return {
441
+ ambientCapabilities: values.get("AmbientCapabilities") ?? "",
442
+ capabilityBoundingSet: values.get("CapabilityBoundingSet") ?? "",
443
+ environment: values.get("Environment") ?? "",
444
+ needDaemonReload: (values.get("NeedDaemonReload") ?? "no") === "yes",
445
+ };
446
+ }
package/src/upgrade.ts CHANGED
@@ -1,10 +1,18 @@
1
- import { readFileSync } from "node:fs";
1
+ import { existsSync, readFileSync } from "node:fs";
2
2
  import { inspectBriefLayout, type BriefLayout } from "./brief-upgrade.ts";
3
3
  import { setPaused, statusSnapshot } from "./daemon.ts";
4
- import { fleetLayers, type DispatchLayer, type FleetLayers } from "./fleet.ts";
4
+ import { fleetLayers, telegramStateDir, type DispatchLayer, type FleetLayers } from "./fleet.ts";
5
5
  import { livingDaemon, restartDaemon } from "./lifecycle.ts";
6
- import { findProject, loadConfig } from "./config.ts";
6
+ import { configPath, findProject, loadConfig, resolveCaps, writeConfigFile } from "./config.ts";
7
7
  import { renderBriefForProject } from "./setup.ts";
8
+ import {
9
+ STAGED_SERVICE_NAME,
10
+ parseEffectiveUnit,
11
+ planHostRuntime,
12
+ unitDriftReason,
13
+ writeHostRuntime,
14
+ type EffectiveUnit,
15
+ } from "./setup-host.ts";
8
16
 
9
17
  const PACKAGE = "omp-conductor";
10
18
  const HERDR_PLUGIN = "herdr-conductor";
@@ -33,6 +41,15 @@ export interface UpgradeResult {
33
41
  dispatch: DispatchLayer;
34
42
  }
35
43
 
44
+ /** What the unit check reads, and how it repairs what it can. */
45
+ export interface UnitFiles {
46
+ rendered: string;
47
+ /** What systemd loaded, not what is on disk. Undefined means no systemd here. */
48
+ effective: EffectiveUnit | undefined;
49
+ /** Writes the corrected unit to the state dir; returns the commands root must run. */
50
+ stage?(): readonly string[];
51
+ }
52
+
36
53
  export interface UpgradeDeps {
37
54
  run(command: string, args: readonly string[]): Promise<UpgradeCommandResult>;
38
55
  snapshot(project?: string): { liveWorkers: number };
@@ -44,6 +61,12 @@ export interface UpgradeDeps {
44
61
  sleep(ms: number): Promise<void>;
45
62
  env: NodeJS.ProcessEnv;
46
63
  log(message: string): void;
64
+ /**
65
+ * What this version would render as the unit, and what systemd actually
66
+ * booted from. Injected so the drift check is testable without a systemd host
67
+ * — and defaulted, so no production caller has to know it exists.
68
+ */
69
+ unitFiles?(): Promise<UnitFiles | undefined> | UnitFiles | undefined;
47
70
  }
48
71
 
49
72
  async function runCommand(command: string, args: readonly string[]): Promise<UpgradeCommandResult> {
@@ -273,8 +296,30 @@ async function rollbackUpgrade(
273
296
  briefChanged: boolean,
274
297
  herdrReloadStarted: boolean,
275
298
  daemonReloadStarted: boolean,
299
+ configBefore?: string,
276
300
  ): Promise<void> {
277
301
  const failures: string[] = [];
302
+
303
+ // Restored FIRST, before the packages that have to read it.
304
+ //
305
+ // Rolling back binaries alone is not a rollback. The new daemon starts during
306
+ // the upgrade and may migrate config.json; the reinstated older release then
307
+ // cannot parse its own config and will not boot — which is exactly how an
308
+ // upgrade failure turned into an outage. The snapshot is taken before the
309
+ // first install, and restoring it is unconditional: writing back a file that
310
+ // never changed is harmless, and detecting "did it change" is one more thing
311
+ // to get wrong while the fleet is down.
312
+ if (configBefore !== undefined) {
313
+ try {
314
+ const path = configPath();
315
+ if (readFileSync(path, "utf8") !== configBefore) {
316
+ deps.log("rollback: conductor config.json");
317
+ writeConfigFile(JSON.parse(configBefore));
318
+ }
319
+ } catch (err) {
320
+ failures.push(`could not restore config.json: ${err instanceof Error ? err.message : String(err)}`);
321
+ }
322
+ }
278
323
  const restore = async (label: string, command: string, args: readonly string[]): Promise<void> => {
279
324
  deps.log(`rollback: ${label}`);
280
325
  try {
@@ -381,6 +426,49 @@ async function rollbackUpgrade(
381
426
  if (failures.length > 0) throw new Error(failures.join("; "));
382
427
  }
383
428
 
429
+ /**
430
+ * The rendered-vs-installed pair for {@link unitDriftReason}.
431
+ *
432
+ * Best-effort by design: a host with no config, no project, or no systemd is
433
+ * not a host with a broken unit, and an upgrade must not fail because it could
434
+ * not answer a question that does not apply there.
435
+ */
436
+ async function defaultUnitFiles(deps: UpgradeDeps): Promise<UnitFiles | undefined> {
437
+ try {
438
+ const cfg = loadConfig();
439
+ const project = cfg.projects[0];
440
+ if (project === undefined) return undefined;
441
+ const plan = planHostRuntime(project, resolveCaps(project, cfg.defaults), telegramStateDir());
442
+ const shown = await deps.run("systemctl", [
443
+ "show",
444
+ STAGED_SERVICE_NAME,
445
+ "-p",
446
+ "AmbientCapabilities",
447
+ "-p",
448
+ "CapabilityBoundingSet",
449
+ "-p",
450
+ "Environment",
451
+ "-p",
452
+ "NeedDaemonReload",
453
+ ]);
454
+ return {
455
+ rendered: plan.service.content,
456
+ effective: shown.code === 0 ? parseEffectiveUnit(shown.stdout) : undefined,
457
+ // Staging needs no privilege, so the repair is reduced to the two lines
458
+ // that genuinely do. Installing the unit is root's, and this command runs
459
+ // as the fleet user by design — so it goes as far as it can and then says
460
+ // exactly what is left, rather than sending the operator back through a
461
+ // wizard to regenerate a file it could write itself.
462
+ stage: () => {
463
+ writeHostRuntime(plan);
464
+ return plan.installCommands;
465
+ },
466
+ };
467
+ } catch {
468
+ return undefined;
469
+ }
470
+ }
471
+
384
472
  export async function upgradeConductor(
385
473
  options: UpgradeOptions = {},
386
474
  overrides: Partial<UpgradeDeps> = {},
@@ -407,6 +495,22 @@ export async function upgradeConductor(
407
495
  const surfaces = await inspectSurfaces(deps);
408
496
  const brief = deps.brief(project);
409
497
  const installNeeded = !surfacesCurrent(surfaces, release.version, release.gitHead);
498
+ // Computed BEFORE the no-op decision, not inside verification, and that
499
+ // placement is the whole point. The upgrade that installs a version is run by
500
+ // the *previous* CLI, so a check living only in the new code never executes
501
+ // for the release that introduces it — and re-running afterwards would take
502
+ // the already-current early return and skip it forever. A fleet whose unit is
503
+ // stale is not current, whatever its package identities say.
504
+ const files = await (deps.unitFiles === undefined ? defaultUnitFiles(deps) : deps.unitFiles());
505
+ const drift = files === undefined ? undefined : unitDriftReason(files.rendered, files.effective);
506
+ if (drift !== undefined) {
507
+ const commands = files?.stage?.() ?? [];
508
+ throw new Error(
509
+ commands.length === 0
510
+ ? drift
511
+ : `${drift}\n\nThe corrected unit has been staged. Run:\n${commands.map((c) => ` ${c}`).join("\n")}`,
512
+ );
513
+ }
410
514
  if (!installNeeded && brief.current) {
411
515
  return {
412
516
  previousVersion: surfaces.cliVersion,
@@ -416,6 +520,18 @@ export async function upgradeConductor(
416
520
  };
417
521
  }
418
522
 
523
+ // Snapshotted before anything is installed, because the new daemon starts
524
+ // during this transaction and may migrate the file — see the restore in
525
+ // `rollbackUpgrade`. Read as bytes, not through the loader: the point is to
526
+ // put back exactly what was there, dialect and all.
527
+ let configBefore: string | undefined;
528
+ try {
529
+ configBefore = readFileSync(configPath(), "utf8");
530
+ } catch {
531
+ // No readable config is not a reason to refuse an upgrade; there is simply
532
+ // nothing to put back.
533
+ }
534
+
419
535
  if (brief.kind === "missing") throw new Error("no ORCHESTRATOR.md exists for the configured project");
420
536
  if (initial.herdr === "unknown") throw new Error("cannot determine whether herdr-fleet.service is active");
421
537
 
@@ -489,6 +605,17 @@ export async function upgradeConductor(
489
605
  const active = await mustRun(deps, "systemctl", ["is-active", HERDR_UNIT]);
490
606
  if (active.stdout.trim() !== "active") throw new Error(`${HERDR_UNIT} is not active after restart`);
491
607
  }
608
+ // The unit is the one surface `upgrade` never looked at, and since 0.4.0 it
609
+ // is the difference between a working per-run fleet and one that refuses
610
+ // every issue. Fatal rather than a warning: the whole point of this command
611
+ // is that a fleet is either upgraded or explicitly left paused, and a
612
+ // "verified" fleet that cannot dispatch is the worse outcome.
613
+ // Re-read rather than trust the earlier pass: the daemon has restarted
614
+ // since, and an operator who installed a unit mid-upgrade should not get a
615
+ // green verification for a file nobody checked.
616
+ const after = await (deps.unitFiles === undefined ? defaultUnitFiles(deps) : deps.unitFiles());
617
+ const stillDrifted = after === undefined ? undefined : unitDriftReason(after.rendered, after.effective);
618
+ if (stillDrifted !== undefined) throw new Error(stillDrifted);
492
619
  await waitForRecovery(deps, initial, project);
493
620
  deps.log("verify 2/2: recovered fleet remains stable");
494
621
  await deps.sleep(1_000);
@@ -518,6 +645,7 @@ export async function upgradeConductor(
518
645
  briefChanged,
519
646
  herdrReloadStarted,
520
647
  daemonReloadStarted,
648
+ configBefore,
521
649
  );
522
650
  } catch (rollbackErr) {
523
651
  const rollback = rollbackErr instanceof Error ? rollbackErr.message : String(rollbackErr);