omp-conductor 0.15.4 → 0.15.5

package/README.md

@@ -251,10 +251,20 @@ command exits without changing anything when all three surfaces already use that
 release, the Herdr plugin is pinned to its exact `gitHead`, and the brief is
 current.
 
-When a daemon is live, the command drains and restarts its recorded project.
-An explicit `--project` that names a different project is rejected before pause
-or installation, so an operator cannot drain one queue and kill another queue's
-workers.
+The upgrade is host-wide, because everything it replaces is: one daemon serves
+every configured project, so its restart lands on all of them at once. A bare
+`omp-conductor upgrade` therefore drains **every** project's workers and
+refreshes **every** project's brief, and pauses them with the fleet-wide
+sentinel — which leaves any per-project `hold` you set standing when it
+restores dispatch.
+
+`--project` narrows that only when it is truthful to do so. When a live daemon
+recorded a single project, the command drains and restarts that project, and an
+explicit `--project` naming a different one is rejected before pause or
+installation. When the daemon serves every configured project and there is more
+than one, `--project` is rejected too: draining one queue and then restarting
+the shared daemon would kill another queue's workers without ever counting
+them. Re-run without the flag.
 
 Ticks remain in their existing armed or disarmed state, so an ordinary update
 does not halt the exact pane or require another Telegram arm challenge. Progress
@@ -2334,7 +2344,7 @@ omp-conductor help
 | `start` | Start `herdr-fleet.service` when that optional unit is installed, clearing a previous pane-recovery pin, then spawn the dispatch loop in the background and wait until it answers `GET /healthz` on `:8787`. Without systemd or that unit it keeps the standalone daemon behaviour. It never clears pause or arms ticks. Refuses if a daemon is already live, naming its pid; if the process dies or never serves, it cleans up and quotes the tail of `daemon.log`. |
 | `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
 | `restart [--now] [--timeout SECONDS] [--port N] [--project NAME]` | Drains the fleet by default: pause new claims, wait until live workers reach `0 / N` (bounded by `--timeout SECONDS`, default 1800 = 30 min), restart, then restore the prior dispatch state. Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. `--now` skips the drain and restarts immediately, orphaning any live runs (old behaviour). A drain that hits `--timeout` restarts nothing and leaves dispatch paused — `omp-conductor resume` lifts it, or re-run `restart` to keep waiting. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
-| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
+| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. Host-wide by default: one daemon serves every configured project, so a bare run drains all of them and refreshes every brief. `--project` is rejected when the live daemon serves several projects — draining one queue and restarting the shared daemon would kill another's workers. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
 | `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. Active-run lines overlay cooperative worker `paused`/`pausing` from `/healthz` without changing SQLite `running` state or the live worker count. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
 | `ledger [--issue N] [--limit N]` | The action audit: every [mediated-verb](#the-mediated-verbs-126) mutation and every next-attempt turn budget. Verb entries include the arguments, decision, named refusal, and resulting SHA. Turn-budget entries remain after an override is replaced or consumed. Reads (`conductor_pr_status`) are absent so polling cannot bury the signal. `--issue` narrows both histories; `--limit` defaults to 50. Recent verb refusals and pending turn overrides also appear in `status`. |
 | `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omp-conductor",
-  "version": "0.15.4",
+  "version": "0.15.5",
   "type": "module",
   "license": "MIT",
   "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",

package/src/upgrade.ts

@@ -52,6 +52,13 @@ export interface UpgradeDeps {
   snapshot(project?: string): { liveWorkers: number };
   layers(project?: string): FleetLayers;
   brief(project?: string): { kind: BriefLayout["kind"]; current: boolean };
+  /**
+   * Every configured project name, in config order. One daemon serves all of
+   * them (the generated unit's ExecStart carries no `--project`), so a
+   * transaction that restarts that daemon has to know the whole set rather
+   * than the one project a caller happened to name (#389).
+   */
+  projectNames(): readonly string[];
   /**
    * The daemon the transaction targets, when one is running. `generation`
    * identifies the exact instance (pid + start time) so a later restart cannot
@@ -103,6 +110,7 @@ export const DEFAULT_DEPS: UpgradeDeps = {
     const live = readFileSync(layout.orchestratorPath, "utf8");
     return { kind: layout.kind, current: live === renderBriefForProject(project, policy) };
   },
+  projectNames: () => loadConfig().projects.map((p) => p.name),
   daemonIdentity: () => {
     const daemon = livingDaemon();
     if (daemon === undefined) return { running: false };
@@ -268,9 +276,95 @@ function previousHerdrInstall(source: string): readonly [string, readonly string
   ];
 }
 
+/**
+ * The projects one lifecycle transaction covers, and the single pause sentinel
+ * that represents them.
+ *
+ * Everything an upgrade or a draining restart replaces is host-wide: the
+ * Bun-global CLI, the omp plugin, the Herdr plugin, and the one dispatch
+ * daemon. That daemon serves *every* configured project — the generated unit's
+ * ExecStart carries no `--project` — so its restart lands on all of them at
+ * once. The scope therefore names every project whose workers must drain and
+ * whose brief must be refreshed, instead of the single project a caller
+ * happened to name (#389).
+ *
+ * `selectors` are what the rest of the module passes back to the project-aware
+ * deps. A selector of `undefined` means "the one configured project", which is
+ * how a single-project host has always addressed itself: keeping it verbatim
+ * means the single-project path still emits exactly the same `brief-upgrade`
+ * command and the same one `snapshot` call it did before.
+ *
+ * `pauseKey` is the sentinel this transaction owns. It is `undefined` for a
+ * host-wide scope, which selects the legacy *global* sentinel — and that is
+ * precisely the right instrument: `isPaused(name)` consults the global file
+ * for every project, so one write pauses the whole host, and removing it lifts
+ * exactly this transaction's pause while leaving any per-project operator hold
+ * standing.
+ */
+interface UpgradeScope {
+  /** Project selectors to address the project-aware deps with, never empty. */
+  selectors: readonly (string | undefined)[];
+  /** The sentinel this transaction pauses and resumes. */
+  pauseKey: string | undefined;
+}
+
+/**
+ * Decides which projects a transaction covers, and refuses the requests that
+ * cannot be honoured truthfully.
+ *
+ * Two refusals, both about a caller naming a project the daemon does not agree
+ * with:
+ *
+ *  - the daemon records a *different* project — it serves that one, so a
+ *    request aimed at another is simply about a different fleet;
+ *  - the daemon records *no* project while several are configured. That is not
+ *    "unknown", it is the multi-project daemon: it serves them all. Narrowing
+ *    is refused rather than obeyed, because draining one project and then
+ *    restarting the shared daemon would kill the other projects' live workers
+ *    without ever counting them (#389, and the shared-daemon rule in #378).
+ *
+ * A single-configured-project host keeps the historical bare selector, so its
+ * behaviour is unchanged whether or not the daemon recorded its name.
+ */
+function resolveScope(deps: UpgradeDeps, verb: "upgrade" | "restart", project?: string): UpgradeScope {
+  let configured: readonly string[] = [];
+  try {
+    configured = deps.projectNames();
+  } catch {
+    // No readable config is not this function's error to raise: the caller
+    // reaches a project-aware dep moments later and fails with the real
+    // problem. Treat it as the historical bare case.
+  }
+  const daemon = deps.daemonIdentity();
+
+  if (project !== undefined) {
+    if (daemon.running && daemon.project !== undefined && daemon.project !== project) {
+      throw new Error(
+        `active daemon serves ${daemon.project}; refusing to ${verb} --project ${project} while it is running`,
+      );
+    }
+    if (daemon.running && daemon.project === undefined && configured.length > 1) {
+      throw new Error(
+        `the running daemon serves all ${String(configured.length)} configured projects ` +
+          `(${configured.join(", ")}); ${verb} replaces host-wide packages and restarts that one daemon, ` +
+          `so it cannot be narrowed to --project ${project} — re-run \`omp-conductor ${verb}\` without --project`,
+      );
+    }
+    return { selectors: [project], pauseKey: project };
+  }
+
+  // A daemon that recorded its project serves that project alone; the rest of
+  // the config is somebody else's fleet as far as this restart is concerned.
+  if (daemon.running && daemon.project !== undefined) {
+    return { selectors: [daemon.project], pauseKey: daemon.project };
+  }
+  if (configured.length > 1) return { selectors: configured, pauseKey: undefined };
+  return { selectors: [undefined], pauseKey: undefined };
+}
+
 async function waitForDrain(
   deps: UpgradeDeps,
-  project?: string,
+  scope: UpgradeScope,
   deadlineAt?: number,
   stale?: () => string | undefined,
 ): Promise<void> {
@@ -282,7 +376,10 @@ async function waitForDrain(
     // and nothing it may stop (#377).
     const cancelled = stale?.();
     if (cancelled !== undefined) throw new Error(cancelled);
-    const workers = deps.snapshot(project).liveWorkers;
+    // Every project the scope covers: a host-wide drain that counted only one
+    // would report zero while another project's worker was still writing to a
+    // worktree the restart is about to orphan (#389).
+    const workers = scope.selectors.reduce((n, s) => n + deps.snapshot(s).liveWorkers, 0);
     if (workers === 0) return;
     if (deadlineAt !== undefined && Date.now() >= deadlineAt) {
       throw new Error(
@@ -321,13 +418,13 @@ interface RestartBegun {
  */
 function restartFenceProblem(
   deps: UpgradeDeps,
-  project: string | undefined,
+  scope: UpgradeScope,
   begun: RestartBegun,
 ): string | undefined {
-  if (!deps.layers(project).paused) {
+  if (!deps.layers(scope.pauseKey).paused) {
     return "restart cancelled: dispatch was resumed while the drain was in progress — nothing was restarted";
   }
-  const owned = deps.pauseState(project);
+  const owned = deps.pauseState(scope.pauseKey);
   if (
     owned === undefined ||
     owned.since !== begun.pauseToken.since ||
@@ -370,13 +467,17 @@ export async function drainAndRestart(
   deps: UpgradeDeps,
   o: { project?: string; timeoutMs: number },
 ): Promise<void> {
-  const initial = deps.layers(o.project);
-  if (!initial.paused) deps.setPaused(true, o.project);
+  // Host-wide by default (#389): the daemon this restarts serves every
+  // configured project, so the drain counts every project's workers and the
+  // pause covers all of them.
+  const scope = resolveScope(deps, "restart", o.project);
+  const initial = deps.layers(scope.pauseKey);
+  if (!initial.paused) deps.setPaused(true, scope.pauseKey);
   // The lock must be provable NOW. A pause that cannot be read as an instance
   // — sentinel malformed or unreadable — would make the whole fence fail open
   // (nothing to compare against), so the transaction refuses before waiting
   // on anything it cannot act upon either way.
-  const pauseToken = deps.pauseState(o.project);
+  const pauseToken = deps.pauseState(scope.pauseKey);
   if (pauseToken === undefined) {
     throw new Error(
       "restart cancelled: cannot prove the restart-owned pause — the active pause sentinel is unreadable or malformed; nothing was restarted",
@@ -386,14 +487,14 @@ export async function drainAndRestart(
     pauseToken,
     daemon: deps.daemonIdentity(),
   };
-  const stale = () => restartFenceProblem(deps, o.project, begun);
-  await waitForDrain(deps, o.project, Date.now() + o.timeoutMs, stale);
+  const stale = () => restartFenceProblem(deps, scope, begun);
+  await waitForDrain(deps, scope, Date.now() + o.timeoutMs, stale);
   // The drain completed; the world may have moved on while it did. Re-prove
   // the pause and the generation an instant before the destructive call.
   const cancelled = stale();
   if (cancelled !== undefined) throw new Error(cancelled);
   await deps.restartDaemon();
-  if (!initial.paused) deps.setPaused(false, o.project);
+  if (!initial.paused) deps.setPaused(false, scope.pauseKey);
 }
 
 function recoveryProblem(
@@ -419,19 +520,32 @@ function recoveryProblem(
 async function waitForRecovery(
   deps: UpgradeDeps,
   initial: FleetLayers,
-  project?: string,
+  scope: UpgradeScope,
 ): Promise<void> {
   let problem = "recovery did not settle";
   for (let attempt = 0; attempt < RECOVERY_ATTEMPTS; attempt += 1) {
-    problem = recoveryProblem(deps.layers(project), initial, deps.snapshot(project).liveWorkers) ?? "";
+    const live = scope.selectors.reduce((n, s) => n + deps.snapshot(s).liveWorkers, 0);
+    problem = recoveryProblem(deps.layers(scope.pauseKey), initial, live) ?? "";
     if (problem === "") return;
     await deps.sleep(RECOVERY_POLL_MS);
   }
   throw new Error(`upgrade verification failed: ${problem}`);
 }
 
+/** One project's brief, as the transaction found it. */
+interface ScopedBrief {
+  /** The selector to address the project-aware deps and the CLI with. */
+  selector: string | undefined;
+  kind: BriefLayout["kind"];
+  current: boolean;
+}
+
 /**
- * Brings the brief up with the freshly installed package.
+ * Brings every in-scope brief up with the freshly installed package.
+ *
+ * Each project owns its own ORCHESTRATOR.md, so a host-wide upgrade migrates
+ * them one at a time rather than leaving every project but one on a floor the
+ * installed package no longer emits (#389).
  *
  * `brief-upgrade --migrate --apply` is the cross-version ABI: this code runs
  * from the *old* CLI while the new one is already installed, so the verb has to
@@ -445,29 +559,27 @@ async function waitForRecovery(
  *    fleet on a single-file brief whose floor no longer tracks the package —
  *    the exact drift this verb exists to end — so it fails loudly and rolls back.
  */
-async function upgradeBrief(
-  deps: UpgradeDeps,
-  kind: BriefLayout["kind"],
-  project?: string,
-): Promise<void> {
-  const selected = project === undefined ? [] : ["--project", project];
-  if (kind === "legacy-handwritten") {
-    // Ungated on purpose: a hand-written brief cannot be migrated without it.
-    await mustRun(deps, "omp-conductor", ["brief-upgrade", "--retrofit", "--apply", ...selected]);
-  }
-  try {
-    await mustRun(deps, "omp-conductor", ["brief-upgrade", "--migrate", "--apply", ...selected]);
-  } catch (err) {
-    if (kind !== "overlay") throw err;
-    const msg = err instanceof Error ? err.message : String(err);
-    deps.log(`brief: brief-upgrade unavailable in the target CLI (${msg}) — overlay recomposes each tick, continuing`);
+async function upgradeBriefs(deps: UpgradeDeps, briefs: readonly ScopedBrief[]): Promise<void> {
+  for (const brief of briefs) {
+    const selected = brief.selector === undefined ? [] : ["--project", brief.selector];
+    if (brief.kind === "legacy-handwritten") {
+      // Ungated on purpose: a hand-written brief cannot be migrated without it.
+      await mustRun(deps, "omp-conductor", ["brief-upgrade", "--retrofit", "--apply", ...selected]);
+    }
+    try {
+      await mustRun(deps, "omp-conductor", ["brief-upgrade", "--migrate", "--apply", ...selected]);
+    } catch (err) {
+      if (brief.kind !== "overlay") throw err;
+      const msg = err instanceof Error ? err.message : String(err);
+      deps.log(`brief: brief-upgrade unavailable in the target CLI (${msg}) — overlay recomposes each tick, continuing`);
+    }
   }
 }
 
 async function rollbackUpgrade(
   deps: UpgradeDeps,
   previous: InstalledSurfaces,
-  project: string | undefined,
+  scope: UpgradeScope,
   installTouched: boolean,
   briefChanged: boolean,
   herdrReloadStarted: boolean,
@@ -577,7 +689,10 @@ async function rollbackUpgrade(
   if (briefChanged) {
     try {
       deps.log("rollback: ORCHESTRATOR.md package floor");
-      await upgradeBrief(deps, "overlay", project);
+      await upgradeBriefs(
+        deps,
+        scope.selectors.map((selector) => ({ selector, kind: "overlay" as const, current: false })),
+      );
     } catch (err) {
       failures.push(err instanceof Error ? err.message : String(err));
     }
@@ -626,23 +741,21 @@ export async function upgradeConductor(
   }
 
   const requested = options.version === undefined ? `${PACKAGE}@latest` : `${PACKAGE}@${options.version}`;
-  const daemon = deps.daemonIdentity();
-  if (daemon.running && options.project !== undefined && daemon.project !== options.project) {
-    const active = daemon.project === undefined ? "an unrecorded project" : daemon.project;
-    throw new Error(
-      `active daemon serves ${active}; refusing to upgrade --project ${options.project} while it is running`,
-    );
-  }
-  const project = options.project ?? daemon.project;
+  // Host-wide by default (#389): one daemon serves every configured project,
+  // so an upgrade that restarts it drains them all and refreshes every brief.
+  const scope = resolveScope(deps, "upgrade", options.project);
 
   const release = parseRegistry(
     (await mustRun(deps, "npm", ["view", requested, "version", "gitHead", "--json"])).stdout,
   );
-  const initial = deps.layers(project);
+  const initial = deps.layers(scope.pauseKey);
   const surfaces = await inspectSurfaces(deps);
-  const brief = deps.brief(project);
+  const briefs: ScopedBrief[] = scope.selectors.map((selector) => ({
+    selector,
+    ...deps.brief(selector),
+  }));
   const installNeeded = !surfacesCurrent(surfaces, release.version, release.gitHead);
-  if (!installNeeded && brief.current) {
+  if (!installNeeded && briefs.every((b) => b.current)) {
     return {
       previousVersion: surfaces.cliVersion,
       ...release,
@@ -673,17 +786,24 @@ export async function upgradeConductor(
     // either — the in-memory restore below still covers a clean rollback.
   }
 
-  if (brief.kind === "missing") throw new Error("no ORCHESTRATOR.md exists for the configured project");
+  const missing = briefs.find((b) => b.kind === "missing");
+  if (missing !== undefined) {
+    throw new Error(
+      missing.selector === undefined
+        ? "no ORCHESTRATOR.md exists for the configured project"
+        : `no ORCHESTRATOR.md exists for project ${missing.selector}`,
+    );
+  }
   if (initial.herdr === "unknown") throw new Error("cannot determine whether herdr-fleet.service is active");
 
   deps.log(`target release: omp-conductor@${release.version} (${release.gitHead})`);
   if (!initial.paused) {
     deps.log("safety: pausing new issue claims");
-    deps.setPaused(true, project);
+    deps.setPaused(true, scope.pauseKey);
   }
   deps.log("drain: waiting for live omp worker sessions");
   try {
-    await waitForDrain(deps, project);
+    await waitForDrain(deps, scope);
   } catch (err) {
     const failure = err instanceof Error ? err.message : String(err);
     throw new Error(
@@ -721,7 +841,7 @@ export async function upgradeConductor(
 
     deps.log("brief: refreshing managed ORCHESTRATOR.md floor while preserving POLICY.md");
     briefChanged = true;
-    await upgradeBrief(deps, brief.kind, project);
+    await upgradeBriefs(deps, briefs);
 
     if (initial.herdr === "active") {
       herdrReloadStarted = true;
@@ -746,13 +866,13 @@ export async function upgradeConductor(
       const active = await mustRun(deps, "systemctl", ["is-active", HERDR_UNIT]);
       if (active.stdout.trim() !== "active") throw new Error(`${HERDR_UNIT} is not active after restart`);
     }
-    await waitForRecovery(deps, initial, project);
+    await waitForRecovery(deps, initial, scope);
     deps.log("verify 2/2: recovered fleet remains stable");
     await deps.sleep(1_000);
-    await waitForRecovery(deps, initial, project);
+    await waitForRecovery(deps, initial, scope);
 
-    if (!initial.paused) deps.setPaused(false, project);
-    const restoredDispatch = deps.layers(project).dispatch;
+    if (!initial.paused) deps.setPaused(false, scope.pauseKey);
+    const restoredDispatch = deps.layers(scope.pauseKey).dispatch;
     if (restoredDispatch !== initial.dispatch) {
       throw new Error(
         `dispatch state is ${restoredDispatch}; expected to restore ${initial.dispatch}`,
@@ -760,9 +880,9 @@ export async function upgradeConductor(
     }
   } catch (err) {
     try {
-      if (!deps.layers(project).paused) deps.setPaused(true, project);
+      if (!deps.layers(scope.pauseKey).paused) deps.setPaused(true, scope.pauseKey);
     } catch {
-      deps.setPaused(true, project);
+      deps.setPaused(true, scope.pauseKey);
     }
     const failure = err instanceof Error ? err.message : String(err);
     deps.log(`upgrade failed: ${failure}`);
@@ -770,7 +890,7 @@ export async function upgradeConductor(
       await rollbackUpgrade(
         deps,
         surfaces,
-        project,
+        scope,
         installTouched,
         briefChanged,
         herdrReloadStarted,