omp-conductor 0.3.7 → 0.3.11

package/README.md

@@ -355,8 +355,11 @@ a yes over Telegram, which is the same protocol it uses for any other amendment.
    omp-conductor daemon --once
    ```
 
-The loop ticks every 5 minutes. `omp-conductor stop` sends `SIGTERM`, and the loop
-shuts down after the current tick rather than mid-run.
+The loop ticks every 5 minutes. `omp-conductor stop` shuts the loop down after
+the current tick rather than mid-run. When the live process is the MainPID of
+`omp-conductor.service`, stop goes through `systemctl stop` so a unit with
+`Restart=on-failure` cannot bring it straight back; otherwise it is a raw
+`SIGTERM` (then `SIGKILL` after 10 seconds).
 
 ## How one tick works
 
@@ -454,19 +457,48 @@ A `claimed` or `running` row is a promise that a worker process exists, and a
 daemon that just started knows that promise is broken: its workers died with the
 previous process. At startup — unless another daemon is alive, so a foreground
 `daemon --once` cannot orphan a running daemon's real workers — every such row is
-moved to `orphaned`, with a log line naming the issue, the attempt and the
-worktree. That frees the slots immediately; a fleet must never resume as
-deadlocked as it crashed.
-
-Only the rows change. The issue keeps `agent:in-progress` — the label is the
-crash guard against double-dispatch — and deciding what the dead worker's remains
-are worth is the orchestrator's drain-duty judgement, spelled out in its brief:
-an open green PR goes to the merge path, a dirty tree is reported before anything
-destroys it (uncommitted edits have no other copy; unpushed *commits* are safe on
-the run's branch in the mirror, which a retry deliberately reattaches), and a
-clean orphan has its label released so the next tick re-claims it. Orphaned
-attempts still count toward `maxAttemptsPerIssue`, so a crash loop escalates
-instead of redispatching forever.
+**salvaged first** (dirty tree → `wip(#N): attempt N killed by a daemon restart —
+auto-salvaged` on the run's branch, same path as a turns-cap kill), then moved to
+`orphaned`, with a log line naming the issue, the attempt and the worktree. That
+frees the slots immediately; a fleet must never resume as deadlocked as it
+crashed, and uncommitted edits must not wait for a human with `bun -e`.
+
+Only the rows change after salvage. The issue keeps `agent:in-progress` — the
+label is the crash guard against double-dispatch — and deciding what the dead
+worker's remains are worth is the orchestrator's drain-duty judgement, spelled
+out in its brief: an open green PR goes to the merge path, a salvaged sha is a
+continuation hand-off, and a clean orphan has its label released so the next
+tick re-claims it. Orphaned attempts still count toward `maxAttemptsPerIssue`,
+so a crash loop escalates instead of redispatching forever.
+
+### Deploying a new package onto a busy fleet
+
+`systemctl restart` / `omp-conductor restart` is safe for **work product** once
+this version is installed: startup salvage commits dirty trees before orphaning
+rows, and salvage rewrites the mirror's managed `info/exclude` to the package's
+current list before `git add` so a narrowed ignore cannot hide deliverables.
+
+It is still disruptive for **in-flight sessions** (the worker process dies; the
+attempt is spent). Prefer draining when you can wait:
+
+1. `omp-conductor pause` — stop new claims; live workers finish.
+2. Wait until `omp-conductor status` shows `workers 0 / N` (no `deploy` hint line).
+3. Install the new package (`bun add -g omp-conductor@…`, `omp plugin install …`).
+4. `systemctl restart omp-conductor` (or `omp-conductor restart`).
+5. `omp-conductor resume` if you left it paused.
+
+If you cannot wait:
+
+1. `omp-conductor pause` (optional but keeps new claims off during the swap).
+2. Install.
+3. Restart — salvage runs on boot for every live worktree, then rows go `orphaned`.
+4. Resume; the pane orchestrator triages `agent:in-progress` orphans (continuation
+   / merge / release label). Status prints a `deploy` line while live workers > 0
+   so you can see the risk before you restart.
+
+Do **not** edit files under the running install and expect the daemon to keep
+dispatching — the integrity tripwire pauses and pages. Install, then restart, so
+the new process records a fresh baseline.
 
 ### What settles a green PR
 
@@ -496,6 +528,17 @@ merged and whose issues were all closed, through two daemon restarts — and bec
 the active set *is* the busy set, those three issues were permanently unclaimable.
 A status page that has stopped being evidence is worse than no status page.
 
+### Continuation runs
+
+When a worktree is provisioned onto a branch that already exists in the mirror
+(reattach after a prior attempt, orphan, or turns-cap auto-requeue), the worker
+brief includes a **Continuation** section: read `git log` / `git diff` against
+the default branch first, and do not recreate work already on the branch.
+
+A **turns-cap kill with attempts remaining** salvages the tree, puts the queue
+label back on, and skips the failed label so the next tick reclaims as a
+continuation automatically.
+
 ### Branch names
 
 `<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
@@ -539,14 +582,14 @@ rest. `0` is a real value (a hard stop), not "unset".
 | Cap | Default | What it protects |
 | --- | --- | --- |
 | `maxConcurrentWorkers` | `2` | Parallel omp sessions. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. Raise it only if you actually have the runners. |
-| `dailySpendUsd` | `25` | Rolling-day spend ceiling. The one cap that stops the fleet rather than deferring work. |
+| `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
 | `workerMaxTurns` | `120` | Turn ceiling for one worker. Catches a session looping without converging. |
 | `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
 | `maxAttemptsPerIssue` | `2` | Retries per issue before it escalates. One clean retry recovers from flaky CI; a third attempt almost always means the issue itself is underspecified. |
 
 Days are counted from **local midnight**, matching how a human reads "today".
 
-Hitting `dailySpendUsd` is not the same as hitting the other caps. A concurrency
+Set `dailySpendUsd` to `null` (wizard: blank) for no money gate — turns and wall-clock still apply. Hitting a numeric `dailySpendUsd` is not the same as hitting the other caps. A concurrency
 limit simply defers work to a later tick. The spend cap **pauses the daemon and
 pages at Tier 2**: a loop that is burning money has to halt itself, because
 waiting for someone to notice tomorrow is how a runaway becomes expensive.
@@ -1065,9 +1108,9 @@ omp-conductor help
 | Command | Behaviour |
 | --- | --- |
 | `start` | Spawn the loop in the background, detached, and wait until it answers `GET /healthz` on `:8787`. Refuses if one is already live, naming its pid. If the process dies or never serves, `start` cleans up after it and quotes the tail of `daemon.log`. |
-| `stop` | `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop. |
-| `restart` | `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them — a restart that quietly moved to the default port would leave every existing health check pointing at nothing. |
-| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. A `.conductor-stalled` marker in the state directory adds an `orchestrator   STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
+| `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
+| `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows — see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
+| `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. While live workers > 0, prints a `deploy` line naming the count so a busy restart is visible before you take it. A `.conductor-stalled` marker in the state directory adds an `orchestrator   STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
 | `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
 | `unblock <issue>` | Remove that issue's `blocked` and `failed` state labels through the tracker, so the next tick can claim it again. This is the supported way back for an escalation you answered: eligibility disqualifies any issue carrying a state label, so an answered issue that keeps one is never re-claimed and the answer is inert. Removing a label the issue does not carry is a no-op, so both are always cleared and neither has to be looked up first. `agent:in-progress` is deliberately not touched — it means a worker process exists, which is not something an answer changes. The run history is left exactly as it is: an answered block still spent a worker, so it still counts toward `maxAttemptsPerIssue`, and the output says how many attempts remain — or warns that the next tick will escalate instead of dispatching, when none do. Exits `2` with `unblock needs an issue number` on a missing or malformed positional. |
 | `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. This is what `start` launches, and what a systemd unit should call. Writes the pidfile itself, and refuses with `another daemon is alive (pid N); stop it first` rather than becoming a second dispatcher. |
@@ -1187,10 +1230,14 @@ Known and deliberate in this version:
   retried rather than corrupted.
 - **Mirrors grow one branch ref per run.** Unpushed work is never discarded, so
   refs accumulate until you reap them.
-- **`stop` is a deadline, not a clean drain.** `SIGTERM` asks the loop to finish the
-  tick it is on, and a tick with a worker in flight can run for that worker's whole
-  wall clock; after 10 seconds it is `SIGKILL`. There is no "stop once the current
-  worker lands".
+- **`stop` is a deadline, not a clean drain.** Whether it goes through
+  `systemctl` or a raw signal, the loop is asked to finish the tick it is on, and
+  a tick with a worker in flight can run for that worker's whole wall clock; the
+  signal path escalates to `SIGKILL` after 10 seconds. There is no "stop once the
+  current worker lands". A unit that supervises the daemon should set
+  `SuccessExitStatus=0 143` (belt-and-braces for a handled `SIGTERM` exit) and
+  operators should prefer `omp-conductor stop` / `systemctl stop` over a raw
+  `kill`, so `Restart=on-failure` cannot misread a deliberate stop as a crash.
 - **A failed orchestrator degrades quietly.** The daemon logs a warning and keeps
   running, but tier-1 escalations then land in issue comments — which is exactly the
   "nobody reads it until morning" path the orchestrator exists to avoid. The warning

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omp-conductor",
-  "version": "0.3.7",
+  "version": "0.3.11",
   "type": "module",
   "license": "MIT",
   "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",

package/src/briefs/orchestrator.md

@@ -18,9 +18,11 @@ written out as the one you chose — ask an omp session to read
 
 ---
 
-You are the orchestrator for **{{PROJECT}}**. You do not write product code and
-you do not touch a worker's branch. You keep the queue moving, and you are the
-first responder when a worker gets stuck.
+You are the orchestrator for **{{PROJECT}}**. You do not write product code —
+**Hard boundaries** below names the exact acts that are out (checkout / commit /
+push inside a worker's worktree; inventing a commit identity) and the one that
+is in (`gh pr update-branch` for a green PR that fell behind). You keep the
+queue moving, and you are the first responder when a worker gets stuck.
 
 You are prompted on a timer. Each tick: do the three duties below, then stop.
 
@@ -100,8 +102,11 @@ checked in this order:
 - **Commits — pushed or not — or a PR that is not green.** Safe either way:
   pushed work lives on the remote, and unpushed commits live on the run's branch
   in the mirror, which a re-claim deliberately reattaches so the next worker
-  starts from them. Note what exists and release the label; the attempt counter
-  still bounds a loop of deaths.
+  starts from them with a **continuation brief** (read the log/diff first; do
+  not recreate existing work). Note what exists and release the label; the
+  attempt counter still bounds a loop of deaths. A turns-cap kill with attempts
+  left is re-queued automatically by the daemon — you should still notice it on
+  drain, but you do not have to invent the continuation prompt.
 - **Genuinely nothing** (clean tree, no commits, no PR). Release the label and let
   the next tick re-claim it clean.
 
@@ -222,8 +227,12 @@ The protocol, in order:
      big for that, send the one-sentence version of each change and say the
      full text lands in the file on yes — the diff stays in your transcript for
      anyone who wants it verbatim.
-3. **On yes, apply it** by editing this file yourself. On no, or on no answer at
-   all, drop it and do not re-ask that amendment.
+3. **On yes, apply it** by editing this file yourself. **On explicit no, drop
+   it** forever and do not re-ask that amendment. **On cancel, timeout, or no
+   answer**, park it — that means "not now", not "never": mention it once in the
+   next report as `pending amendment: <one-liner> — say 'apply it' or 'drop it'`,
+   never re-open the yes/no dialog, and drop it if still unanswered after 7 days.
+   A cancelled dialog is not a permanent rejection.
 4. **Log it.** Append one line to **Amendments** at the bottom of this file: the
    date, what triggered it, a one-sentence summary.
 5. **Offer general fixes upstream.** Ask one question of the amendment you just

package/src/briefs/worker.md

@@ -28,7 +28,7 @@ Then read the repo's own guidance before writing anything: `AGENTS.md`,
 `CLAUDE.md`, `CONTEXT.md`, and any `docs/adr/` entry the issue touches. Those
 files are canonical; your priors are not.
 
-## Acceptance criteria
+{{CONTINUATION}}## Acceptance criteria
 
 {{ACCEPTANCE_CRITERIA}}
 

package/src/cli.ts

@@ -16,6 +16,7 @@ import {
   DEFAULT_PORT,
   healthCheck,
   livingDaemon,
+  restartDaemon,
   startDaemon,
   stopDaemon,
   writeRecord,
@@ -46,9 +47,14 @@ usage:
   start    run the dispatch loop in the background and wait until it answers
            GET /healthz on :8787 (override with --port). Refuses if one is
            already running.
-  stop     signal the running daemon and wait for it to exit.
+  stop     stop the running daemon. Uses systemctl when the omp-conductor
+           unit owns the process (so Restart=on-failure cannot bring it back);
+           otherwise SIGTERM then SIGKILL.
   restart  stop then start, keeping the running daemon's port and project
-           unless a flag overrides them.
+           unless a flag overrides them. On boot the new process salvages any
+           dirty live worktrees before orphaning those rows — see README
+           "Deploying a new package onto a busy fleet". Goes through systemctl
+           when the unit owns the live pid.
   status   show pause state, caps, active runs, today's usage, and whether a
            daemon is alive.
   tail     follow the newest run for <issue>: the worker's assistant text and
@@ -398,27 +404,35 @@ try {
     }
 
     case "stop": {
-      // Read the pid before it stops existing, so the confirmation can name it.
-      const pid = livingDaemon()?.pid;
       const result = await stopDaemon();
-      process.stdout.write(result === "stopped" ? `stopped — pid ${pid ?? "?"}\n` : "not running\n");
+      if (result.kind === "not-running") {
+        process.stdout.write("not running\n");
+      } else {
+        process.stdout.write(
+          `stopped — pid ${result.pid}${result.via === "systemctl" ? " (via systemctl)" : ""}\n`,
+        );
+      }
       break;
     }
 
     case "restart": {
       // Inherit the running daemon's port and project: a restart that quietly
       // moved to the default port would leave every existing health check
-      // pointing at nothing.
-      const previous = livingDaemon();
-      const result = await stopDaemon();
-      if (result === "stopped") process.stdout.write(`stopped — pid ${previous?.pid ?? "?"}\n`);
-      const rec = await startDaemon({
-        port: portFlag(argv) ?? previous?.port,
-        project: flag(argv, "project") ?? previous?.project,
+      // pointing at nothing. When the unit owns the live pid, restartDaemon
+      // goes through systemctl so the replacement stays supervised.
+      const { previous, record, via } = await restartDaemon({
+        port: portFlag(argv),
+        project: flag(argv, "project"),
       });
+      if (previous !== undefined) {
+        process.stdout.write(
+          `stopped — pid ${previous.pid}${via === "systemctl" ? " (via systemctl)" : ""}\n`,
+        );
+      }
       process.stdout.write(
-        `started — pid ${rec.pid}, /healthz on :${rec.port}` +
-          `${rec.project === undefined ? "" : `, project ${rec.project}`}\nlog ${rec.logFile}\n`,
+        `started — pid ${record.pid}, /healthz on :${record.port}` +
+          `${record.project === undefined ? "" : `, project ${record.project}`}` +
+          `${via === "systemctl" ? " (via systemctl)" : ""}\nlog ${record.logFile}\n`,
       );
       break;
     }

package/src/config.ts

@@ -139,15 +139,16 @@ export function saveConfig(c: ConductorConfig): void {
 /**
  * Layers a project's overrides on the global defaults, field by field, so a
  * project that pins one cap still inherits the other five. `??` not `||`: a
- * deliberate `dailySpendUsd: 0` is a hard stop, not "unset". Spelled out per
- * field so adding a `Caps` member fails to compile here instead of resolving
- * to `undefined` at a call site.
+ * deliberate `dailySpendUsd: 0` is a hard stop, not "unset", and `null` is a
+ * deliberate "no spend gate" that must not fall through to the default.
+ * Spelled out per field so adding a `Caps` member fails to compile here.
  */
 export function resolveCaps(p: ProjectConfig, defaults: Caps): Caps {
   const o: Partial<Caps> = p.caps ?? {};
   return {
     maxConcurrentWorkers: o.maxConcurrentWorkers ?? defaults.maxConcurrentWorkers,
-    dailySpendUsd: o.dailySpendUsd ?? defaults.dailySpendUsd,
+    // nullish only — `null` means off; do not coalesce it to the default.
+    dailySpendUsd: o.dailySpendUsd !== undefined ? o.dailySpendUsd : defaults.dailySpendUsd,
     workerMaxTurns: o.workerMaxTurns ?? defaults.workerMaxTurns,
     workerWallClockMs: o.workerWallClockMs ?? defaults.workerWallClockMs,
     maxAttemptsPerIssue: o.maxAttemptsPerIssue ?? defaults.maxAttemptsPerIssue,
@@ -520,8 +521,18 @@ function coerceCaps(parsed: unknown, label: string, problems: string[], legacy:
   for (const key of CAP_KEYS) {
     const v = raw[key];
     if (v === undefined) continue;
+    // Spend is the only cap that may be null (= no gate). Every other ceiling
+    // is a non-negative number; 0 remains a hard stop where it already was.
+    if (key === "dailySpendUsd" && v === null) {
+      out.dailySpendUsd = null;
+      continue;
+    }
     if (typeof v !== "number" || !Number.isFinite(v) || v < 0) {
-      problems.push(`${label}.${key} must be a non-negative finite number, found ${JSON.stringify(v)}`);
+      problems.push(
+        key === "dailySpendUsd"
+          ? `${label}.${key} must be a non-negative finite number or null (no cap), found ${JSON.stringify(v)}`
+          : `${label}.${key} must be a non-negative finite number, found ${JSON.stringify(v)}`,
+      );
       continue;
     }
     out[key] = v;