omp-conductor 0.3.6 → 0.3.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -355,8 +355,11 @@ a yes over Telegram, which is the same protocol it uses for any other amendment.
355
355
  omp-conductor daemon --once
356
356
  ```
357
357
 
358
- The loop ticks every 5 minutes. `omp-conductor stop` sends `SIGTERM`, and the loop
359
- shuts down after the current tick rather than mid-run.
358
+ The loop ticks every 5 minutes. `omp-conductor stop` shuts the loop down after
359
+ the current tick rather than mid-run. When the live process is the MainPID of
360
+ `omp-conductor.service`, stop goes through `systemctl stop` so a unit with
361
+ `Restart=on-failure` cannot bring it straight back; otherwise it is a raw
362
+ `SIGTERM` (then `SIGKILL` after 10 seconds).
360
363
 
361
364
  ## How one tick works
362
365
 
@@ -454,19 +457,48 @@ A `claimed` or `running` row is a promise that a worker process exists, and a
454
457
  daemon that just started knows that promise is broken: its workers died with the
455
458
  previous process. At startup — unless another daemon is alive, so a foreground
456
459
  `daemon --once` cannot orphan a running daemon's real workers — every such row is
457
- moved to `orphaned`, with a log line naming the issue, the attempt and the
458
- worktree. That frees the slots immediately; a fleet must never resume as
459
- deadlocked as it crashed.
460
-
461
- Only the rows change. The issue keeps `agent:in-progress` the label is the
462
- crash guard against double-dispatch — and deciding what the dead worker's remains
463
- are worth is the orchestrator's drain-duty judgement, spelled out in its brief:
464
- an open green PR goes to the merge path, a dirty tree is reported before anything
465
- destroys it (uncommitted edits have no other copy; unpushed *commits* are safe on
466
- the run's branch in the mirror, which a retry deliberately reattaches), and a
467
- clean orphan has its label released so the next tick re-claims it. Orphaned
468
- attempts still count toward `maxAttemptsPerIssue`, so a crash loop escalates
469
- instead of redispatching forever.
460
+ **salvaged first** (dirty tree `wip(#N): attempt N killed by a daemon restart
461
+ auto-salvaged` on the run's branch, same path as a turns-cap kill), then moved to
462
+ `orphaned`, with a log line naming the issue, the attempt and the worktree. That
463
+ frees the slots immediately; a fleet must never resume as deadlocked as it
464
+ crashed, and uncommitted edits must not wait for a human with `bun -e`.
465
+
466
+ Only the rows change after salvage. The issue keeps `agent:in-progress` the
467
+ label is the crash guard against double-dispatch and deciding what the dead
468
+ worker's remains are worth is the orchestrator's drain-duty judgement, spelled
469
+ out in its brief: an open green PR goes to the merge path, a salvaged sha is a
470
+ continuation hand-off, and a clean orphan has its label released so the next
471
+ tick re-claims it. Orphaned attempts still count toward `maxAttemptsPerIssue`,
472
+ so a crash loop escalates instead of redispatching forever.
473
+
474
+ ### Deploying a new package onto a busy fleet
475
+
476
+ `systemctl restart` / `omp-conductor restart` is safe for **work product** once
477
+ this version is installed: startup salvage commits dirty trees before orphaning
478
+ rows, and salvage rewrites the mirror's managed `info/exclude` to the package's
479
+ current list before `git add` so a narrowed ignore cannot hide deliverables.
480
+
481
+ It is still disruptive for **in-flight sessions** (the worker process dies; the
482
+ attempt is spent). Prefer draining when you can wait:
483
+
484
+ 1. `omp-conductor pause` — stop new claims; live workers finish.
485
+ 2. Wait until `omp-conductor status` shows `workers 0 / N` (no `deploy` hint line).
486
+ 3. Install the new package (`bun add -g omp-conductor@…`, `omp plugin install …`).
487
+ 4. `systemctl restart omp-conductor` (or `omp-conductor restart`).
488
+ 5. `omp-conductor resume` if you left it paused.
489
+
490
+ If you cannot wait:
491
+
492
+ 1. `omp-conductor pause` (optional but keeps new claims off during the swap).
493
+ 2. Install.
494
+ 3. Restart — salvage runs on boot for every live worktree, then rows go `orphaned`.
495
+ 4. Resume; the pane orchestrator triages `agent:in-progress` orphans (continuation
496
+ / merge / release label). Status prints a `deploy` line while live workers > 0
497
+ so you can see the risk before you restart.
498
+
499
+ Do **not** edit files under the running install and expect the daemon to keep
500
+ dispatching — the integrity tripwire pauses and pages. Install, then restart, so
501
+ the new process records a fresh baseline.
470
502
 
471
503
  ### What settles a green PR
472
504
 
@@ -496,6 +528,17 @@ merged and whose issues were all closed, through two daemon restarts — and bec
496
528
  the active set *is* the busy set, those three issues were permanently unclaimable.
497
529
  A status page that has stopped being evidence is worse than no status page.
498
530
 
531
+ ### Continuation runs
532
+
533
+ When a worktree is provisioned onto a branch that already exists in the mirror
534
+ (reattach after a prior attempt, orphan, or turns-cap auto-requeue), the worker
535
+ brief includes a **Continuation** section: read `git log` / `git diff` against
536
+ the default branch first, and do not recreate work already on the branch.
537
+
538
+ A **turns-cap kill with attempts remaining** salvages the tree, puts the queue
539
+ label back on, and skips the failed label so the next tick reclaims as a
540
+ continuation automatically.
541
+
499
542
  ### Branch names
500
543
 
501
544
  `<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
@@ -539,14 +582,14 @@ rest. `0` is a real value (a hard stop), not "unset".
539
582
  | Cap | Default | What it protects |
540
583
  | --- | --- | --- |
541
584
  | `maxConcurrentWorkers` | `2` | Parallel omp sessions. Two, because **CI runner slots, not model tokens, are the usual throughput ceiling** — a third worker would starve its own PR checks on a small self-hosted runner pool. Raise it only if you actually have the runners. |
542
- | `dailySpendUsd` | `25` | Rolling-day spend ceiling. The one cap that stops the fleet rather than deferring work. |
585
+ | `dailySpendUsd` | `25` | Rolling-day spend ceiling in USD, or `null` for no spend gate. `0` is a hard stop. Metered from assistant `usage.cost.total`. |
543
586
  | `workerMaxTurns` | `120` | Turn ceiling for one worker. Catches a session looping without converging. |
544
587
  | `workerWallClockMs` | `5400000` (90 minutes) | Wall-clock ceiling for one worker. A session that is merely stuck spends no turns, so turns alone cannot detect it. |
545
588
  | `maxAttemptsPerIssue` | `2` | Retries per issue before it escalates. One clean retry recovers from flaky CI; a third attempt almost always means the issue itself is underspecified. |
546
589
 
547
590
  Days are counted from **local midnight**, matching how a human reads "today".
548
591
 
549
- Hitting `dailySpendUsd` is not the same as hitting the other caps. A concurrency
592
+ Set `dailySpendUsd` to `null` (wizard: blank) for no money gate — turns and wall-clock still apply. Hitting a numeric `dailySpendUsd` is not the same as hitting the other caps. A concurrency
550
593
  limit simply defers work to a later tick. The spend cap **pauses the daemon and
551
594
  pages at Tier 2**: a loop that is burning money has to halt itself, because
552
595
  waiting for someone to notice tomorrow is how a runaway becomes expensive.
@@ -1065,9 +1108,9 @@ omp-conductor help
1065
1108
  | Command | Behaviour |
1066
1109
  | --- | --- |
1067
1110
  | `start` | Spawn the loop in the background, detached, and wait until it answers `GET /healthz` on `:8787`. Refuses if one is already live, naming its pid. If the process dies or never serves, `start` cleans up after it and quotes the tail of `daemon.log`. |
1068
- | `stop` | `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop. |
1069
- | `restart` | `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them a restart that quietly moved to the default port would leave every existing health check pointing at nothing. |
1070
- | `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
1111
+ | `stop` | Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is the live daemon — systemd then owns the stop and will not schedule a restart for the exit it just requested. Otherwise `SIGTERM`, then `SIGKILL` after a 10-second grace period. Prints `not running` when there is nothing to stop, and tags the confirmation with `(via systemctl)` when the unit path was used. |
1112
+ | `restart` | Prefer `systemctl restart` when the unit owns the live pid so the replacement stays supervised; otherwise `stop` then `start`, inheriting the running daemon's port and project unless a flag overrides them. The new process **salvages dirty live worktrees before orphaning** those rows see [Deploying a new package onto a busy fleet](#deploying-a-new-package-onto-a-busy-fleet). |
1113
+ | `status [--project NAME]` | Pause state, config and state paths, resolved caps, active runs and today's usage, plus a `daemon` block: pid, uptime, port, project, `/healthz` result and log path. While live workers > 0, prints a `deploy` line naming the count so a busy restart is visible before you take it. A `.conductor-stalled` marker in the state directory adds an `orchestrator STALLED since …` line — see [the stall marker](#a-wedged-session-and-the-marker-that-notices). Reads while a daemon in another process writes. |
1071
1114
  | `tail <issue>` | Follow the newest run for that issue: the worker's assistant text as `assistant: …` and each tool it calls as `tool: <name>`, printed as they land. Workers are omp sessions inside the daemon rather than terminals, so this is the only way to watch one live — a herdr pane running it becomes an observation window. Starts from the top of the transcript, not the end, so attaching to a run that is already ten turns in shows those ten turns. Exits `1` with `no run recorded for #N` when the issue has never been dispatched, or `no transcript yet (state: …)` when the attempt has not opened one. Otherwise it runs until `Ctrl-C`, or until the run has finished and its transcript has been silent for five seconds, and prints `run ended: <state>`. |
1072
1115
  | `unblock <issue>` | Remove that issue's `blocked` and `failed` state labels through the tracker, so the next tick can claim it again. This is the supported way back for an escalation you answered: eligibility disqualifies any issue carrying a state label, so an answered issue that keeps one is never re-claimed and the answer is inert. Removing a label the issue does not carry is a no-op, so both are always cleared and neither has to be looked up first. `agent:in-progress` is deliberately not touched — it means a worker process exists, which is not something an answer changes. The run history is left exactly as it is: an answered block still spent a worker, so it still counts toward `maxAttemptsPerIssue`, and the output says how many attempts remain — or warns that the next tick will escalate instead of dispatching, when none do. Exits `2` with `unblock needs an issue number` on a missing or malformed positional. |
1073
1116
  | `daemon` | Run the loop in the **foreground**, ticking every 5 minutes and serving `/healthz`. This is what `start` launches, and what a systemd unit should call. Writes the pidfile itself, and refuses with `another daemon is alive (pid N); stop it first` rather than becoming a second dispatcher. |
@@ -1187,10 +1230,14 @@ Known and deliberate in this version:
1187
1230
  retried rather than corrupted.
1188
1231
  - **Mirrors grow one branch ref per run.** Unpushed work is never discarded, so
1189
1232
  refs accumulate until you reap them.
1190
- - **`stop` is a deadline, not a clean drain.** `SIGTERM` asks the loop to finish the
1191
- tick it is on, and a tick with a worker in flight can run for that worker's whole
1192
- wall clock; after 10 seconds it is `SIGKILL`. There is no "stop once the current
1193
- worker lands".
1233
+ - **`stop` is a deadline, not a clean drain.** Whether it goes through
1234
+ `systemctl` or a raw signal, the loop is asked to finish the tick it is on, and
1235
+ a tick with a worker in flight can run for that worker's whole wall clock; the
1236
+ signal path escalates to `SIGKILL` after 10 seconds. There is no "stop once the
1237
+ current worker lands". A unit that supervises the daemon should set
1238
+ `SuccessExitStatus=0 143` (belt-and-braces for a handled `SIGTERM` exit) and
1239
+ operators should prefer `omp-conductor stop` / `systemctl stop` over a raw
1240
+ `kill`, so `Restart=on-failure` cannot misread a deliberate stop as a crash.
1194
1241
  - **A failed orchestrator degrades quietly.** The daemon logs a warning and keeps
1195
1242
  running, but tier-1 escalations then land in issue comments — which is exactly the
1196
1243
  "nobody reads it until morning" path the orchestrator exists to avoid. The warning
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "omp-conductor",
3
- "version": "0.3.6",
3
+ "version": "0.3.9",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
@@ -100,8 +100,11 @@ checked in this order:
100
100
  - **Commits — pushed or not — or a PR that is not green.** Safe either way:
101
101
  pushed work lives on the remote, and unpushed commits live on the run's branch
102
102
  in the mirror, which a re-claim deliberately reattaches so the next worker
103
- starts from them. Note what exists and release the label; the attempt counter
104
- still bounds a loop of deaths.
103
+ starts from them with a **continuation brief** (read the log/diff first; do
104
+ not recreate existing work). Note what exists and release the label; the
105
+ attempt counter still bounds a loop of deaths. A turns-cap kill with attempts
106
+ left is re-queued automatically by the daemon — you should still notice it on
107
+ drain, but you do not have to invent the continuation prompt.
105
108
  - **Genuinely nothing** (clean tree, no commits, no PR). Release the label and let
106
109
  the next tick re-claim it clean.
107
110
 
@@ -28,7 +28,7 @@ Then read the repo's own guidance before writing anything: `AGENTS.md`,
28
28
  `CLAUDE.md`, `CONTEXT.md`, and any `docs/adr/` entry the issue touches. Those
29
29
  files are canonical; your priors are not.
30
30
 
31
- ## Acceptance criteria
31
+ {{CONTINUATION}}## Acceptance criteria
32
32
 
33
33
  {{ACCEPTANCE_CRITERIA}}
34
34
 
package/src/cli.ts CHANGED
@@ -16,6 +16,7 @@ import {
16
16
  DEFAULT_PORT,
17
17
  healthCheck,
18
18
  livingDaemon,
19
+ restartDaemon,
19
20
  startDaemon,
20
21
  stopDaemon,
21
22
  writeRecord,
@@ -46,9 +47,14 @@ usage:
46
47
  start run the dispatch loop in the background and wait until it answers
47
48
  GET /healthz on :8787 (override with --port). Refuses if one is
48
49
  already running.
49
- stop signal the running daemon and wait for it to exit.
50
+ stop stop the running daemon. Uses systemctl when the omp-conductor
51
+ unit owns the process (so Restart=on-failure cannot bring it back);
52
+ otherwise SIGTERM then SIGKILL.
50
53
  restart stop then start, keeping the running daemon's port and project
51
- unless a flag overrides them.
54
+ unless a flag overrides them. On boot the new process salvages any
55
+ dirty live worktrees before orphaning those rows — see README
56
+ "Deploying a new package onto a busy fleet". Goes through systemctl
57
+ when the unit owns the live pid.
52
58
  status show pause state, caps, active runs, today's usage, and whether a
53
59
  daemon is alive.
54
60
  tail follow the newest run for <issue>: the worker's assistant text and
@@ -398,27 +404,35 @@ try {
398
404
  }
399
405
 
400
406
  case "stop": {
401
- // Read the pid before it stops existing, so the confirmation can name it.
402
- const pid = livingDaemon()?.pid;
403
407
  const result = await stopDaemon();
404
- process.stdout.write(result === "stopped" ? `stopped — pid ${pid ?? "?"}\n` : "not running\n");
408
+ if (result.kind === "not-running") {
409
+ process.stdout.write("not running\n");
410
+ } else {
411
+ process.stdout.write(
412
+ `stopped — pid ${result.pid}${result.via === "systemctl" ? " (via systemctl)" : ""}\n`,
413
+ );
414
+ }
405
415
  break;
406
416
  }
407
417
 
408
418
  case "restart": {
409
419
  // Inherit the running daemon's port and project: a restart that quietly
410
420
  // moved to the default port would leave every existing health check
411
- // pointing at nothing.
412
- const previous = livingDaemon();
413
- const result = await stopDaemon();
414
- if (result === "stopped") process.stdout.write(`stopped — pid ${previous?.pid ?? "?"}\n`);
415
- const rec = await startDaemon({
416
- port: portFlag(argv) ?? previous?.port,
417
- project: flag(argv, "project") ?? previous?.project,
421
+ // pointing at nothing. When the unit owns the live pid, restartDaemon
422
+ // goes through systemctl so the replacement stays supervised.
423
+ const { previous, record, via } = await restartDaemon({
424
+ port: portFlag(argv),
425
+ project: flag(argv, "project"),
418
426
  });
427
+ if (previous !== undefined) {
428
+ process.stdout.write(
429
+ `stopped — pid ${previous.pid}${via === "systemctl" ? " (via systemctl)" : ""}\n`,
430
+ );
431
+ }
419
432
  process.stdout.write(
420
- `started — pid ${rec.pid}, /healthz on :${rec.port}` +
421
- `${rec.project === undefined ? "" : `, project ${rec.project}`}\nlog ${rec.logFile}\n`,
433
+ `started — pid ${record.pid}, /healthz on :${record.port}` +
434
+ `${record.project === undefined ? "" : `, project ${record.project}`}` +
435
+ `${via === "systemctl" ? " (via systemctl)" : ""}\nlog ${record.logFile}\n`,
422
436
  );
423
437
  break;
424
438
  }
package/src/config.ts CHANGED
@@ -139,15 +139,16 @@ export function saveConfig(c: ConductorConfig): void {
139
139
  /**
140
140
  * Layers a project's overrides on the global defaults, field by field, so a
141
141
  * project that pins one cap still inherits the other five. `??` not `||`: a
142
- * deliberate `dailySpendUsd: 0` is a hard stop, not "unset". Spelled out per
143
- * field so adding a `Caps` member fails to compile here instead of resolving
144
- * to `undefined` at a call site.
142
+ * deliberate `dailySpendUsd: 0` is a hard stop, not "unset", and `null` is a
143
+ * deliberate "no spend gate" that must not fall through to the default.
144
+ * Spelled out per field so adding a `Caps` member fails to compile here.
145
145
  */
146
146
  export function resolveCaps(p: ProjectConfig, defaults: Caps): Caps {
147
147
  const o: Partial<Caps> = p.caps ?? {};
148
148
  return {
149
149
  maxConcurrentWorkers: o.maxConcurrentWorkers ?? defaults.maxConcurrentWorkers,
150
- dailySpendUsd: o.dailySpendUsd ?? defaults.dailySpendUsd,
150
+ // nullish only — `null` means off; do not coalesce it to the default.
151
+ dailySpendUsd: o.dailySpendUsd !== undefined ? o.dailySpendUsd : defaults.dailySpendUsd,
151
152
  workerMaxTurns: o.workerMaxTurns ?? defaults.workerMaxTurns,
152
153
  workerWallClockMs: o.workerWallClockMs ?? defaults.workerWallClockMs,
153
154
  maxAttemptsPerIssue: o.maxAttemptsPerIssue ?? defaults.maxAttemptsPerIssue,
@@ -520,8 +521,18 @@ function coerceCaps(parsed: unknown, label: string, problems: string[], legacy:
520
521
  for (const key of CAP_KEYS) {
521
522
  const v = raw[key];
522
523
  if (v === undefined) continue;
524
+ // Spend is the only cap that may be null (= no gate). Every other ceiling
525
+ // is a non-negative number; 0 remains a hard stop where it already was.
526
+ if (key === "dailySpendUsd" && v === null) {
527
+ out.dailySpendUsd = null;
528
+ continue;
529
+ }
523
530
  if (typeof v !== "number" || !Number.isFinite(v) || v < 0) {
524
- problems.push(`${label}.${key} must be a non-negative finite number, found ${JSON.stringify(v)}`);
531
+ problems.push(
532
+ key === "dailySpendUsd"
533
+ ? `${label}.${key} must be a non-negative finite number or null (no cap), found ${JSON.stringify(v)}`
534
+ : `${label}.${key} must be a non-negative finite number, found ${JSON.stringify(v)}`,
535
+ );
525
536
  continue;
526
537
  }
527
538
  out[key] = v;
package/src/daemon.ts CHANGED
@@ -444,10 +444,35 @@ export async function buildBrief(
444
444
  r: Routed,
445
445
  branch: string,
446
446
  worktree: string,
447
+ opts: { continuation?: boolean; defaultBranch?: string } = {},
447
448
  ): Promise<string> {
448
449
  // Read per dispatch rather than caching: editing the brief then takes effect
449
450
  // on the next issue instead of needing a daemon restart.
450
451
  const template = await Bun.file(BRIEF_TEMPLATE_PATH).text();
452
+ const defaultBranch = opts.defaultBranch ?? r.repo.defaultBranch;
453
+ const continuation =
454
+ opts.continuation === true
455
+ ? [
456
+ "",
457
+ "## Continuation — do not start from zero",
458
+ "",
459
+ `You are **resuming** issue #${r.issue.number}. Branch \`${branch}\` already exists`,
460
+ "and was reattached with prior commits (and possibly a salvaged WIP tip).",
461
+ "Before writing anything:",
462
+ "",
463
+ "```bash",
464
+ "git log --oneline origin/" + defaultBranch + "..HEAD",
465
+ "git diff --stat origin/" + defaultBranch + "...HEAD",
466
+ "git status --porcelain",
467
+ "```",
468
+ "",
469
+ "Read that history. **Do not recreate work that already exists.** Finish",
470
+ "what remains against the same acceptance criteria. If a prior attempt",
471
+ "left a `wip(#…): … auto-salvaged` commit, treat it as your starting",
472
+ "point, not as trash to rewrite from scratch.",
473
+ "",
474
+ ].join("\n")
475
+ : "";
451
476
  return renderBrief(template, {
452
477
  ISSUE_NUMBER: String(r.issue.number),
453
478
  ISSUE_TITLE: r.issue.title,
@@ -461,6 +486,7 @@ export async function buildBrief(
461
486
  // placeholder sits flush against the next list item in the template, so an
462
487
  // unconfigured render leaves no blank line where a hint would have gone.
463
488
  GRAPH_HINT: graphHint(r.repo),
489
+ CONTINUATION: continuation,
464
490
  });
465
491
  }
466
492
 
@@ -513,14 +539,14 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
513
539
  // would cost a second network fetch per attempt.
514
540
  const mirrorPath = mirrorPathFor(r.repo, project.mirrorRoot);
515
541
  await removeWorktree(mirrorPath, worktreePathFor(project.workspaceRoot, issue));
516
-
517
- worktreePath = await addWorktree(
542
+ const provisioned = await addWorktree(
518
543
  r.repo,
519
544
  project.mirrorRoot,
520
545
  project.workspaceRoot,
521
546
  issue,
522
547
  branch,
523
548
  );
549
+ worktreePath = provisioned.path;
524
550
 
525
551
  // The SDK names the transcript itself, so the daemon supplies the parent
526
552
  // directory and learns the real path back from the result. Inventing one
@@ -529,10 +555,16 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
529
555
  mkdirSync(sessionDir, { recursive: true });
530
556
  store.updateRun(runId, { worktree: worktreePath, state: "running" });
531
557
 
532
- log(`#${issue} attempt ${attempt} → ${r.repo.name} ${branch}`);
558
+ log(
559
+ `#${issue} attempt ${attempt} → ${r.repo.name} ${branch}` +
560
+ (provisioned.reattached ? " (continuation: reattached existing branch)" : ""),
561
+ );
533
562
 
534
563
  const result = await runWorker({
535
- brief: await buildBrief(project, r, branch, worktreePath),
564
+ brief: await buildBrief(project, r, branch, worktreePath, {
565
+ continuation: provisioned.reattached,
566
+ defaultBranch: r.repo.defaultBranch,
567
+ }),
536
568
  cwd: worktreePath,
537
569
  caps,
538
570
  sessionDir,
@@ -572,30 +604,61 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
572
604
  detail: [`${r.issue.title}`, r.issue.url, "", result.report].join("\n"),
573
605
  });
574
606
  } else if (result.state === "failed" || result.state === "killed") {
575
- await swapLabel(tracker, issue, inProgress, project.stateLabels.failed);
576
- // Before the escalation is composed, so it can say where the work went —
577
- // and long before the next attempt provisions over this tree.
607
+ // Turns-cap with attempts left: salvage, put the issue back on the queue,
608
+ // and skip the failed label so the next tick reclaims as a continuation
609
+ // instead of burning a human triage cycle (#50 / #21).
610
+ const continueTurns =
611
+ result.killedBy === "turns" && attempt < caps.maxAttemptsPerIssue;
578
612
  const salvaged = await salvage(issue, attempt, endedBy(result.killedBy), worktreePath);
579
- await safeEscalate(d, {
580
- tier: 1,
581
- project: project.name,
582
- issue,
583
- runId,
584
- // The dedup key includes the summary, so the attempt number is what
585
- // lets a genuine second failure page again while a tick that keeps
586
- // seeing the same dead issue stays quiet.
587
- summary: result.killedBy
588
- ? `#${issue} was killed on attempt ${attempt} by the ${result.killedBy} cap`
589
- : `#${issue} failed on attempt ${attempt}`,
590
- detail: [
591
- `${r.issue.title}`,
592
- r.issue.url,
593
- ...salvaged,
594
- `Session: ${result.sessionFile ?? "(no transcript)"}`,
595
- "",
596
- result.report,
597
- ].join("\n"),
598
- });
613
+
614
+ if (continueTurns) {
615
+ await tracker.removeLabel(issue, inProgress);
616
+ await tracker.addLabel(issue, project.queueLabel);
617
+ log(
618
+ `#${issue} turns-cap on attempt ${attempt}/${caps.maxAttemptsPerIssue} ` +
619
+ `salvaged and re-queued for continuation`,
620
+ );
621
+ await safeEscalate(d, {
622
+ tier: 1,
623
+ project: project.name,
624
+ issue,
625
+ runId,
626
+ summary: `#${issue} hit the turns cap on attempt ${attempt} — auto-requeued for continuation`,
627
+ detail: [
628
+ `${r.issue.title}`,
629
+ r.issue.url,
630
+ ...salvaged,
631
+ `Session: ${result.sessionFile ?? "(no transcript)"}`,
632
+ "",
633
+ "The queue label is back on; the next tick should reattach the branch",
634
+ "and open a continuation brief. No failed label was applied.",
635
+ "",
636
+ result.report,
637
+ ].join("\n"),
638
+ });
639
+ } else {
640
+ await swapLabel(tracker, issue, inProgress, project.stateLabels.failed);
641
+ await safeEscalate(d, {
642
+ tier: 1,
643
+ project: project.name,
644
+ issue,
645
+ runId,
646
+ // The dedup key includes the summary, so the attempt number is what
647
+ // lets a genuine second failure page again while a tick that keeps
648
+ // seeing the same dead issue stays quiet.
649
+ summary: result.killedBy
650
+ ? `#${issue} was killed on attempt ${attempt} by the ${result.killedBy} cap`
651
+ : `#${issue} failed on attempt ${attempt}`,
652
+ detail: [
653
+ `${r.issue.title}`,
654
+ r.issue.url,
655
+ ...salvaged,
656
+ `Session: ${result.sessionFile ?? "(no transcript)"}`,
657
+ "",
658
+ result.report,
659
+ ].join("\n"),
660
+ });
661
+ }
599
662
  } else {
600
663
  // pushed-green: the PR belongs to a human now. The in-progress label
601
664
  // stays on until the merge closes the issue, which is also what keeps
@@ -923,9 +986,10 @@ async function tick(d: Deps): Promise<void> {
923
986
 
924
987
  // Spend is the one cap that stops the fleet instead of merely deferring work.
925
988
  // A loop that is burning money has to halt itself; waiting for a human to
926
- // notice tomorrow is how a runaway becomes expensive.
989
+ // notice tomorrow is how a runaway becomes expensive. `null` means the
990
+ // operator opted out — turns and wall-clock still brake every run (#46).
927
991
  const spent = store.spendSince(project.name, since);
928
- if (spent >= caps.dailySpendUsd) {
992
+ if (caps.dailySpendUsd !== null && spent >= caps.dailySpendUsd) {
929
993
  setPaused(true);
930
994
  await safeEscalate(d, {
931
995
  tier: 2,
@@ -1011,7 +1075,9 @@ export function formatStatus(s: StatusSnapshot): string {
1011
1075
  "caps",
1012
1076
  ` workers ${s.liveWorkers} / ${s.caps.maxConcurrentWorkers}`,
1013
1077
  ` issues today ${s.runsToday}`,
1014
- ` spend today $${s.spendTodayUsd.toFixed(2)} / $${s.caps.dailySpendUsd.toFixed(2)}`,
1078
+ s.caps.dailySpendUsd === null
1079
+ ? ` spend today $${s.spendTodayUsd.toFixed(2)} (no daily cap)`
1080
+ : ` spend today $${s.spendTodayUsd.toFixed(2)} / $${s.caps.dailySpendUsd.toFixed(2)}`,
1015
1081
  ` worker max turns ${s.caps.workerMaxTurns}`,
1016
1082
  ` worker wall clock ${Math.round(s.caps.workerWallClockMs / 60_000)}m`,
1017
1083
  ` attempts per issue ${s.caps.maxAttemptsPerIssue}`,
@@ -1029,6 +1095,15 @@ export function formatStatus(s: StatusSnapshot): string {
1029
1095
  );
1030
1096
  }
1031
1097
  }
1098
+ // Deploy hint: a restart while workers are live orphans them (salvage runs
1099
+ // first — #35). Prefer pause + drain to zero live workers when you can wait.
1100
+ if (s.liveWorkers > 0) {
1101
+ lines.push(
1102
+ "",
1103
+ `deploy ${s.liveWorkers} live worker(s) — restart salvages dirty trees then orphans the rows; ` +
1104
+ `pause and wait for workers 0/${s.caps.maxConcurrentWorkers} when you can drain instead`,
1105
+ );
1106
+ }
1032
1107
  return lines.join("\n");
1033
1108
  }
1034
1109
 
@@ -1085,18 +1160,18 @@ export function armConductor(): void {
1085
1160
  }
1086
1161
 
1087
1162
  /**
1088
- * Settles the runs a previous daemon process left in flight.
1163
+ * Settles `claimed`/`running` rows left by a dead daemon process and, before
1164
+ * marking each one `orphaned`, salvages any dirty worktree.
1089
1165
  *
1090
- * A `claimed` or `running` row is a promise that a worker exists in *some*
1091
- * process. This is called from a freshly started daemon, so when no other
1092
- * daemon is alive every such row is a worker that died with the previous
1093
- * process. Left "active", those rows deadlock admission forever: the slot
1094
- * count reads full while nothing runs, and the fleet looks busy doing nothing
1095
- * (found live, after a host restart killed two workers mid-run).
1166
+ * Found live after a host restart killed two workers mid-run, and again on
1167
+ * every package deploy that restarted while workers were live (#35): without
1168
+ * the salvage call the next attempt's `worktree remove --force` destroyed
1169
+ * uncommitted edits that had no other copy. Cap-kills already salvaged (#27);
1170
+ * this is the same call site for the restart path.
1096
1171
  *
1097
1172
  * Only the rows change. The issue keeps its in-progress label — that label is
1098
1173
  * the crash guard against double-dispatch, and deciding what a dead worker's
1099
- * remains are worth (an open PR? unpushed commits? a dirty tree?) is the
1174
+ * remains are worth (an open PR? a salvaged sha? a clean tree?) is the
1100
1175
  * orchestrator's drain-duty judgement, not something to automate here. The
1101
1176
  * rows also keep counting toward `maxAttemptsPerIssue`, so a loop of deaths
1102
1177
  * still escalates instead of retrying forever.
@@ -1107,12 +1182,21 @@ export function armConductor(): void {
1107
1182
  * them is {@link settlePushedGreen}, on the tick, by asking the tracker what
1108
1183
  * became of the PR — the one question a restart cannot answer by inference.
1109
1184
  */
1110
- export function reconcileOrphanedRuns(store: Store, project: string): RunRecord[] {
1185
+ export async function reconcileOrphanedRuns(
1186
+ store: Store,
1187
+ project: string,
1188
+ ): Promise<RunRecord[]> {
1111
1189
  // Live runs only: `pushed-green` holds no process, so it cannot be orphaned by
1112
1190
  // a process dying — it is finished work waiting on a human merge.
1113
1191
  const stale = store.liveRuns(project);
1114
1192
  const endedAt = Date.now();
1115
1193
  for (const r of stale) {
1194
+ // Salvage before the row flips: the worktree path is on the record, and
1195
+ // salvageWip is a no-op for a missing/clean tree. Reason string matches
1196
+ // the cap-kill wording so triage reads the same either way.
1197
+ if (r.worktree !== "") {
1198
+ await salvage(r.issue, r.attempt, "a daemon restart", r.worktree);
1199
+ }
1116
1200
  store.updateRun(r.id, { state: "orphaned", endedAt });
1117
1201
  }
1118
1202
  return stale;
@@ -1141,7 +1225,7 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
1141
1225
  // daemon must not orphan that daemon's real, live workers).
1142
1226
  const alive = livingDaemon();
1143
1227
  if (alive === undefined || alive.pid === process.pid) {
1144
- for (const r of reconcileOrphanedRuns(store, project.name)) {
1228
+ for (const r of await reconcileOrphanedRuns(store, project.name)) {
1145
1229
  log(
1146
1230
  `#${r.issue} orphaned by a previous daemon (attempt ${r.attempt}, was ${r.state}, worktree ${r.worktree}) — ` +
1147
1231
  `slot freed; the ${project.stateLabels.inProgress} label stays until the orchestrator triages what the worker left`,
@@ -1283,5 +1367,11 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
1283
1367
  await orchestrator?.dispose();
1284
1368
  store.close();
1285
1369
  log("stopped");
1370
+ // A handled SIGTERM still leaves some runtimes with a non-zero default
1371
+ // (historically 128+signal). Under systemd `Restart=on-failure` that looks
1372
+ // like a crash and the unit comes straight back — the exact failure mode
1373
+ // `omp-conductor stop` hit on the reference fleet. Force success so a
1374
+ // graceful drain is not a restart.
1375
+ process.exitCode = 0;
1286
1376
  }
1287
1377
  }
package/src/lifecycle.ts CHANGED
@@ -10,16 +10,30 @@
10
10
  * operator real time are `start` refusing against a ghost and `status`
11
11
  * reporting a daemon that died hours ago.
12
12
  *
13
+ * When the live pid is the MainPID of the `omp-conductor.service` unit,
14
+ * `stop`/`restart` go through `systemctl` rather than a raw `SIGTERM`. A
15
+ * raw signal against a unit with `Restart=on-failure` is read as a crash and
16
+ * the unit comes straight back — the bug that made `omp-conductor stop` look
17
+ * like a no-op on a systemd-managed host.
18
+ *
13
19
  * Deliberately free of every other module in this package: nothing here opens
14
20
  * the store, loads the config or talks to `gh`, so `stop` and `status` keep
15
21
  * working when the config is the very thing that is broken.
16
22
  */
17
23
 
18
- import { spawn } from "node:child_process";
24
+ import { spawn, spawnSync } from "node:child_process";
19
25
  import { chmodSync, closeSync, mkdirSync, openSync, readFileSync, readSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs";
20
26
  import { homedir } from "node:os";
21
27
  import { join } from "node:path";
22
28
 
29
+ /**
30
+ * The systemd unit name operators are expected to install for a supervised
31
+ * daemon. Hardcoded rather than configured: one name on the host is the
32
+ * whole point of a unit, and a wrong name would silently fall back to the
33
+ * SIGTERM path and reintroduce the restart loop this module exists to avoid.
34
+ */
35
+ export const SYSTEMD_UNIT = "omp-conductor.service";
36
+
23
37
  /**
24
38
  * Mirrors `DEFAULT_PORT` in ./daemon.ts. Duplicated rather than imported so
25
39
  * this module stays free of the dispatcher's dependency tree; exported so the
@@ -271,23 +285,67 @@ export async function startDaemon(o: { port?: number; project?: string } = {}):
271
285
  }
272
286
 
273
287
  /**
274
- * Stops the daemon: `SIGTERM`, wait, then `SIGKILL`.
288
+ * How the last stop actually landed. Callers print this so an operator can
289
+ * tell a supervised stop from a bare SIGTERM without reading the journal.
290
+ */
291
+ export type StopResult =
292
+ | { kind: "stopped"; pid: number; via: "systemctl" | "signal" }
293
+ | { kind: "not-running" };
294
+
295
+ /**
296
+ * Stops the daemon.
275
297
  *
276
- * `SIGTERM` asks the loop to finish the tick it is on, and a tick that has a
277
- * worker in flight can run for the worker's whole wall clock. The grace period
278
- * is therefore a deadline, not a promise of a clean drain.
279
- * // ponytail: no way to say "stop when the current worker lands". If that
280
- * // matters, take a `--timeout` on the CLI verb, or poll `activeRuns` before
281
- * // escalating.
298
+ * Prefer `systemctl stop omp-conductor.service` when that unit's MainPID is
299
+ * the live daemon: systemd then owns the stop and will not schedule a restart
300
+ * for the exit it just requested. A raw `SIGTERM` against a unit with
301
+ * `Restart=on-failure` is what made `omp-conductor stop` look like a no-op on
302
+ * the reference fleet exit 143 was a "failure", and the unit came back five
303
+ * seconds later.
304
+ *
305
+ * Falls back to `SIGTERM` then `SIGKILL` when there is no unit, the unit is
306
+ * not running, or its MainPID is somebody else (a hand-started daemon next to
307
+ * a stopped unit). The grace period is a deadline, not a clean drain: a tick
308
+ * with a worker in flight can run for that worker's whole wall clock.
282
309
  */
283
- export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<"stopped" | "not-running"> {
310
+ export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<StopResult> {
284
311
  const rec = livingDaemon();
285
312
  if (rec === undefined) {
286
313
  // `livingDaemon` already cleared a stale file; this covers the unparseable
287
- // one it refused to read.
314
+ // one it refused to read. Still ask systemd: a unit can be running with
315
+ // no pidfile (boot race, wiped runtime dir) and stop must still land.
316
+ const unitPid = systemdMainPid();
317
+ if (unitPid !== undefined && (await stopViaSystemd(undefined))) {
318
+ const deadline = Date.now() + (o.timeoutMs ?? STOP_TIMEOUT_MS);
319
+ while (isAlive(unitPid) && Date.now() < deadline) await sleep(100);
320
+ if (isAlive(unitPid)) {
321
+ throw new Error(
322
+ `daemon pid ${unitPid} still alive after systemctl stop ${SYSTEMD_UNIT} — ` +
323
+ `check \`systemctl status ${SYSTEMD_UNIT}\``,
324
+ );
325
+ }
326
+ clearRecord();
327
+ return { kind: "stopped", pid: unitPid, via: "systemctl" };
328
+ }
288
329
  clearRecord();
289
- return "not-running";
330
+ return { kind: "not-running" };
290
331
  }
332
+
333
+ if (await stopViaSystemd(rec.pid)) {
334
+ // Wait out the unit: systemctl stop is synchronous for Type=simple, but
335
+ // a slow drain still holds the old pid briefly and the next start would
336
+ // refuse against it.
337
+ const deadline = Date.now() + (o.timeoutMs ?? STOP_TIMEOUT_MS);
338
+ while (isAlive(rec.pid) && Date.now() < deadline) await sleep(100);
339
+ if (isAlive(rec.pid)) {
340
+ throw new Error(
341
+ `daemon pid ${rec.pid} still alive after systemctl stop ${SYSTEMD_UNIT} — ` +
342
+ `check \`systemctl status ${SYSTEMD_UNIT}\``,
343
+ );
344
+ }
345
+ clearRecord();
346
+ return { kind: "stopped", pid: rec.pid, via: "systemctl" };
347
+ }
348
+
291
349
  const gone = await terminate(rec.pid, o.timeoutMs ?? STOP_TIMEOUT_MS);
292
350
  if (!gone) {
293
351
  // The record stays: something is still holding that pid, and forgetting
@@ -295,13 +353,146 @@ export async function stopDaemon(o: { timeoutMs?: number } = {}): Promise<"stopp
295
353
  throw new Error(`daemon pid ${rec.pid} survived SIGTERM and SIGKILL — it may belong to another user`);
296
354
  }
297
355
  clearRecord();
298
- return "stopped";
356
+ return { kind: "stopped", pid: rec.pid, via: "signal" };
357
+ }
358
+
359
+ /**
360
+ * Restarts the daemon.
361
+ *
362
+ * Same ownership rule as {@link stopDaemon}: when the unit owns the live pid,
363
+ * `systemctl restart` keeps systemd in charge of the replacement process so
364
+ * the new MainPID is still the unit's. Falling back to stop+start for a
365
+ * hand-started daemon would leave the unit dead and the new process
366
+ * unsupervised — fine for a laptop, wrong for the host that installed a unit
367
+ * specifically so a crash comes back.
368
+ *
369
+ * Returns the record of the process that is now answering `/healthz`.
370
+ */
371
+ export async function restartDaemon(
372
+ o: { port?: number; project?: string; timeoutMs?: number } = {},
373
+ ): Promise<{ previous: DaemonRecord | undefined; record: DaemonRecord; via: "systemctl" | "cli" }> {
374
+ const previous = livingDaemon();
375
+ const unitPid = systemdMainPid();
376
+ const unitOwns =
377
+ unitPid !== undefined && (previous === undefined || previous.pid === unitPid);
378
+
379
+ if (unitOwns) {
380
+ const ran = systemctl(["restart", SYSTEMD_UNIT]);
381
+ if (ran.ok) {
382
+ // systemctl restart returns once the new MainPID is up; the pidfile is
383
+ // written by the daemon itself on boot, so wait for that rather than
384
+ // inventing a record from the unit alone.
385
+ const deadline = Date.now() + READY_TIMEOUT_MS;
386
+ for (;;) {
387
+ const rec = livingDaemon();
388
+ if (rec !== undefined) {
389
+ const health = await healthCheck(rec.port);
390
+ if (health.ok) return { previous, record: rec, via: "systemctl" };
391
+ }
392
+ if (Date.now() >= deadline) break;
393
+ await sleep(READY_POLL_MS);
394
+ }
395
+ throw new Error(
396
+ `systemctl restart ${SYSTEMD_UNIT} returned, but the daemon never answered /healthz`,
397
+ );
398
+ }
399
+ // Unit exists and owns the pid but systemctl refused (permissions, dbus
400
+ // down). Fall through to the signal path rather than stranding the
401
+ // operator with "restart failed" and a still-running daemon they cannot
402
+ // reach through the unit.
403
+ }
404
+
405
+ await stopDaemon({ timeoutMs: o.timeoutMs });
406
+ const record = await startDaemon({ port: o.port ?? previous?.port, project: o.project ?? previous?.project });
407
+ return { previous, record, via: "cli" };
408
+ }
409
+
410
+ /**
411
+ * The MainPID of {@link SYSTEMD_UNIT}, or `undefined` when systemd is absent,
412
+ * the unit is unknown, or it is not running. Never throws: a missing binary
413
+ * or a dbus blip is "no unit", and stop falls back to SIGTERM.
414
+ */
415
+ export function systemdMainPid(unit = SYSTEMD_UNIT): number | undefined {
416
+ const ran = systemctl(["show", unit, "--property=MainPID", "--property=ActiveState", "--value"]);
417
+ if (!ran.ok) return undefined;
418
+ // `systemctl show --value` prints one property per line, MainPID then
419
+ // ActiveState, in the order requested. Tolerate either order and blank
420
+ // lines so a future systemctl rearrange does not silently disable the path.
421
+ const lines = ran.stdout
422
+ .split("\n")
423
+ .map((l) => l.trim())
424
+ .filter((l) => l.length > 0);
425
+ let pid: number | undefined;
426
+ let active: string | undefined;
427
+ for (const line of lines) {
428
+ if (/^\d+$/.test(line)) {
429
+ const n = Number(line);
430
+ if (Number.isInteger(n) && n > 1) pid = n;
431
+ } else {
432
+ active = line;
433
+ }
434
+ }
435
+ if (active !== undefined && active !== "active" && active !== "reactivating") return undefined;
436
+ return pid;
299
437
  }
300
438
 
301
439
  // ---------------------------------------------------------------------------
302
440
  // internals
303
441
  // ---------------------------------------------------------------------------
304
442
 
443
+ /**
444
+ * Ask systemd to stop the unit, but only when it actually owns `pid`.
445
+ *
446
+ * `pid === undefined` means "no pidfile" — still stop the unit if it is
447
+ * active, because that is the only process that could be the daemon. A unit
448
+ * whose MainPID is some other process is left alone: stopping it would take
449
+ * down a neighbour, and the signal path handles *our* pid.
450
+ */
451
+ async function stopViaSystemd(pid: number | undefined): Promise<boolean> {
452
+ const main = systemdMainPid();
453
+ if (main === undefined) return false;
454
+ if (pid !== undefined && main !== pid) return false;
455
+ const ran = systemctl(["stop", SYSTEMD_UNIT]);
456
+ return ran.ok;
457
+ }
458
+
459
+ /**
460
+ * Run one `systemctl` invocation. Captures output and never throws: absence
461
+ * of the binary, a missing unit, or a permission error are all "not ok", and
462
+ * the caller decides whether to fall back.
463
+ *
464
+ * The default shells out. Tests replace it with {@link setSystemctlForTest}
465
+ * so the ownership decision is exercised without a real systemd.
466
+ */
467
+ export type SystemctlFn = (args: string[]) => { ok: boolean; stdout: string; stderr: string };
468
+
469
+ function defaultSystemctl(args: string[]): { ok: boolean; stdout: string; stderr: string } {
470
+ try {
471
+ const res = spawnSync("systemctl", args, {
472
+ encoding: "utf8",
473
+ // A hung dbus is not worth blocking stop on; the signal path is right there.
474
+ timeout: 15_000,
475
+ env: process.env,
476
+ });
477
+ if (res.error) return { ok: false, stdout: "", stderr: res.error.message };
478
+ return {
479
+ ok: res.status === 0,
480
+ stdout: res.stdout ?? "",
481
+ stderr: res.stderr ?? "",
482
+ };
483
+ } catch (err) {
484
+ return { ok: false, stdout: "", stderr: err instanceof Error ? err.message : String(err) };
485
+ }
486
+ }
487
+
488
+ let systemctl: SystemctlFn = defaultSystemctl;
489
+
490
+ /** Test-only: replace the `systemctl` runner. Pass `undefined` to restore. */
491
+ export function setSystemctlForTest(fn: SystemctlFn | undefined): void {
492
+ systemctl = fn ?? defaultSystemctl;
493
+ }
494
+
495
+
305
496
  function sleep(ms: number): Promise<void> {
306
497
  return new Promise<void>((resolve) => {
307
498
  const t = setTimeout(resolve, ms);
package/src/omp.ts CHANGED
@@ -162,6 +162,11 @@ export async function createSession(opts: {
162
162
  // sharing it fails to start. The daemon runs `maxConcurrentWorkers`
163
163
  // (2 by default) workers at once, which makes this the normal path.
164
164
  agentRegistry: new mod.AgentRegistry(),
165
+ // Default true in the SDK, but pass it explicitly so a harness change
166
+ // cannot silently strip MCP from workers. Discovery walks cwd + user
167
+ // agentDir (~/.omp/agent/mcp.json) — without this, workers grep-only and
168
+ // burn the turns cap on discovery (#29).
169
+ enableMCP: true,
165
170
  });
166
171
  const raw = asRawSession(created);
167
172
  // Surfaced rather than swallowed: this is how a quiet downgrade to a weaker
package/src/plugin.ts CHANGED
@@ -193,6 +193,30 @@ async function askNumber(ctx: CommandContext, title: string, fallback: number):
193
193
  return value;
194
194
  }
195
195
 
196
+ /**
197
+ * Daily spend ceiling. Blank / "none" / "off" → null (no gate). Unparseable
198
+ * non-empty input keeps the current value. Distinct from askNumber so operators
199
+ * can turn the money brake off without writing a magic 0 (which is a hard stop).
200
+ */
201
+ async function askSpendCap(
202
+ ctx: CommandContext,
203
+ title: string,
204
+ fallback: number | null,
205
+ ): Promise<number | null> {
206
+ const seed = fallback === null ? "" : String(fallback);
207
+ const raw = (await ask(ctx, title, seed)).trim().toLowerCase();
208
+ if (raw === "" || raw === "none" || raw === "off" || raw === "null") return null;
209
+ const value = Number(raw);
210
+ if (!Number.isFinite(value) || value < 0) {
211
+ ctx.ui.notify(
212
+ `"${raw}" is not a non-negative number or blank — keeping ${fallback === null ? "no cap" : fallback}.`,
213
+ "warning",
214
+ );
215
+ return fallback;
216
+ }
217
+ return value;
218
+ }
219
+
196
220
  /**
197
221
  * Pre-push gates as one comma-separated line, `cmd @ cwd` for a subdirectory:
198
222
  * `bun run check, bun test @ server`. Shown through `formatGates`, the same
@@ -500,10 +524,12 @@ const askGraph: AreaAsker = async (ctx, a) => {
500
524
  * answer for anyone who has not measured their own runners. */
501
525
  const askCaps: AreaAsker = async (ctx, a) => {
502
526
  const caps: Partial<Caps> = { ...a.caps };
527
+ const spendLabel =
528
+ DEFAULT_CAPS.dailySpendUsd === null ? "no spend cap" : `$${DEFAULT_CAPS.dailySpendUsd}/day`;
503
529
  const tuneCaps = await ctx.ui.confirm(
504
530
  "Caps",
505
531
  `Defaults: ${DEFAULT_CAPS.maxConcurrentWorkers} workers, ` +
506
- `$${DEFAULT_CAPS.dailySpendUsd}/day, ${DEFAULT_CAPS.workerMaxTurns} turns and ` +
532
+ `${spendLabel}, ${DEFAULT_CAPS.workerMaxTurns} turns and ` +
507
533
  `${Math.round(DEFAULT_CAPS.workerWallClockMs / 60000)} min per worker, ` +
508
534
  `${DEFAULT_CAPS.maxAttemptsPerIssue} attempts per issue. Change them?`,
509
535
  );
@@ -516,7 +542,11 @@ const askCaps: AreaAsker = async (ctx, a) => {
516
542
  "Max concurrent workers",
517
543
  caps.maxConcurrentWorkers ?? DEFAULT_CAPS.maxConcurrentWorkers,
518
544
  );
519
- caps.dailySpendUsd = await askNumber(ctx, "Spend ceiling per rolling day (USD)", caps.dailySpendUsd ?? DEFAULT_CAPS.dailySpendUsd);
545
+ caps.dailySpendUsd = await askSpendCap(
546
+ ctx,
547
+ "Spend ceiling per rolling day (USD) — blank = no spend cap",
548
+ caps.dailySpendUsd !== undefined ? caps.dailySpendUsd : DEFAULT_CAPS.dailySpendUsd,
549
+ );
520
550
  caps.workerMaxTurns = await askNumber(ctx, "Turn ceiling per worker", caps.workerMaxTurns ?? DEFAULT_CAPS.workerMaxTurns);
521
551
  caps.workerWallClockMs = await askNumber(
522
552
  ctx,
package/src/setup.ts CHANGED
@@ -954,9 +954,11 @@ export const AMEND_AREAS: {
954
954
  describe: (p) => {
955
955
  const c = resolveCaps(p, DEFAULT_CAPS);
956
956
  const answered = Object.keys(p.caps).length > 0;
957
+ const spend =
958
+ c.dailySpendUsd === null ? "no spend cap" : `$${c.dailySpendUsd}/day`;
957
959
  return (
958
960
  `${c.maxConcurrentWorkers} workers, ${c.workerMaxTurns} turns, ` +
959
- `${Math.round(c.workerWallClockMs / 60000)}m, $${c.dailySpendUsd}/day, ` +
961
+ `${Math.round(c.workerWallClockMs / 60000)}m, ${spend}, ` +
960
962
  `${c.maxAttemptsPerIssue} attempts${answered ? "" : " (all defaults)"} — ` +
961
963
  `${p.workerModel === undefined ? "harness default model" : `model ${p.workerModel}`}`
962
964
  );
package/src/types.ts CHANGED
@@ -16,8 +16,11 @@ export interface Caps {
16
16
  /** Parallel omp sessions. Two by default: on a small self-hosted runner pool
17
17
  * a third worker would starve its own PR checks. */
18
18
  maxConcurrentWorkers: number;
19
- /** Rolling-day spend ceiling; the loop stops claiming work once it is hit. */
20
- dailySpendUsd: number;
19
+ /**
20
+ * Rolling-day spend ceiling. `null` means no spend gate (turns + wall-clock
21
+ * still apply). `0` is a hard stop — deliberate, not "unset".
22
+ */
23
+ dailySpendUsd: number | null;
21
24
  /** Turn ceiling for one worker — catches loops that are burning tokens
22
25
  * without converging. */
23
26
  workerMaxTurns: number;
package/src/worker.ts CHANGED
@@ -216,15 +216,22 @@ export async function runWorker(
216
216
  // is its report, whether it finished cleanly or was cut off.
217
217
  const text = reportText(field(message, "content"));
218
218
  if (text !== "") report = text;
219
+
220
+ // Real cost lives on assistant messages as `usage.cost.total` (live hermes
221
+ // transcripts, 2026-08-07). The earlier agent_end.telemetry path never
222
+ // fired, so every run recorded $0 and the daily cap was theater (#46).
223
+ const cost = costUsdFromMessage(message);
224
+ if (cost !== undefined) spendUsd += cost;
219
225
  });
220
226
 
221
227
  session.on("agent_end", (event) => {
222
- // ponytail: cost only arrives when the harness run carries telemetry, so
223
- // spend can legitimately read 0 and the daily-spend cap then leans on the
224
- // turn and wall-clock ceilings. Upgrade path: pass a telemetry config
225
- // through `createSession` once the harness exposes it on the SDK options.
228
+ // Fallback for harnesses that only attach cost on the terminal event.
226
229
  const estimated = field(field(field(event, "telemetry"), "cost"), "estimatedUsd");
227
- if (typeof estimated === "number" && Number.isFinite(estimated)) spendUsd += estimated;
230
+ if (typeof estimated === "number" && Number.isFinite(estimated) && estimated > 0) {
231
+ // Prefer message totals when both exist — do not double-count a run that
232
+ // already accumulated per-message costs.
233
+ if (spendUsd === 0) spendUsd += estimated;
234
+ }
228
235
 
229
236
  // Anything that is not literally `false` — including garbage or nothing at
230
237
  // all — is a finished run.
@@ -282,6 +289,31 @@ export async function runWorker(
282
289
  );
283
290
  }
284
291
 
292
+ /**
293
+ * USD cost from one assistant message's `usage.cost` block.
294
+ *
295
+ * Prefer `total` when present; otherwise sum the component fields the live
296
+ * harness emits (input/output/cacheRead/cacheWrite). Exported so a unit test
297
+ * can pin the shape without standing up a session.
298
+ */
299
+ export function costUsdFromMessage(message: unknown): number | undefined {
300
+ const usage = field(message, "usage");
301
+ const cost = field(usage, "cost");
302
+ if (cost === null || typeof cost !== "object") return undefined;
303
+ const total = field(cost, "total");
304
+ if (typeof total === "number" && Number.isFinite(total) && total >= 0) return total;
305
+ let sum = 0;
306
+ let any = false;
307
+ for (const key of ["input", "output", "cacheRead", "cacheWrite"] as const) {
308
+ const v = field(cost, key);
309
+ if (typeof v === "number" && Number.isFinite(v) && v >= 0) {
310
+ sum += v;
311
+ any = true;
312
+ }
313
+ }
314
+ return any ? sum : undefined;
315
+ }
316
+
285
317
  /**
286
318
  * Read one property off an unvalidated harness event. The event union lives in
287
319
  * the peer dependency, so the worker narrows the handful of fields it reads
package/src/worktree.ts CHANGED
@@ -102,15 +102,27 @@ const EXCLUDE_END = "# <<< omp-conductor";
102
102
 
103
103
  /**
104
104
  * Appended to every mirror's `info/exclude`, and so in force in every worktree
105
- * cut from it. Deliberately the same shapes salvage therefore skips, and
106
- * for the same reason a worker's own scaffolding is not the repo's business.
105
+ * cut from it: a worker's own scaffolding is not the repo's business, and
106
+ * keeping it out of `git status` keeps it out of the worker's own `git add -A`
107
+ * as well as out of salvage.
107
108
  *
108
- * Two layers because they catch different moments: this one keeps scratch out
109
- * of a worker's own `git add -A` and out of its `git status`, which salvage
110
- * never observes; the salvage list catches whatever a worker created before
111
- * this landed, or wrote past an ignore with `add -f`.
109
+ * **Only the directory form of `.scratch*`; the slash is the design.** An ignore
110
+ * here applies to every repo this fleet touches, and an ignored *new* file is
111
+ * invisible to salvage `git add -A` skips it, and a tree holding only such
112
+ * files reports `nothing`. So a name that could plausibly be a deliverable must
113
+ * never appear in this list. The first version also carried `.env.local` and
114
+ * `*.local.sh`; its first correction still carried an unqualified `.scratch*`.
115
+ * All three make the same mistake: repos can legitimately ship an `.env.local`
116
+ * template, `bootstrap.local.sh`, `.scratchrc`, or `.scratchpad`, and a worker
117
+ * asked to add one would watch it vanish.
118
+ *
119
+ * The directory-only pattern survives because its trailing slash limits it to a
120
+ * directory whose name announces that its contents are disposable. That was
121
+ * the 2026-08-07 incident's exact shape (`.scratch82/env.sh`). Broader
122
+ * conventions belong in a repo's own `.gitignore`, where its operator chooses
123
+ * them, rather than being imposed by whatever dispatcher happens to be driving.
112
124
  */
113
- const LOCAL_EXCLUDE = [".scratch*/", ".scratch*", ".env.local", "*.local.sh"];
125
+ const LOCAL_EXCLUDE = [".scratch*/"];
114
126
 
115
127
  /**
116
128
  * Adds the managed block to an `info/exclude`, preserving everything else.
@@ -182,6 +194,40 @@ async function configureMirror(mirrorPath: string): Promise<void> {
182
194
  writeFileSync(exclude, mergeExclude(existsSync(exclude) ? readFileSync(exclude, "utf8") : ""));
183
195
  }
184
196
 
197
+ /**
198
+ * Rewrites this worktree's common `info/exclude` managed block to the package's
199
+ * current list. Salvage must do this itself: it is the path that runs after a
200
+ * package swap, when the mirror may still carry a previous release's patterns,
201
+ * and `git add -A` would otherwise silently skip legitimate new files the old
202
+ * list happened to name (dogfood 2026-08-07 / #44).
203
+ *
204
+ * Failures are swallowed — a missing common dir is "no exclude to heal", and
205
+ * the salvage dirty check still runs. Never throws into the salvage outcome.
206
+ */
207
+ function refreshManagedExclude(worktree: string): void {
208
+ try {
209
+ // `--git-common-dir` is relative for linked worktrees; resolve against the
210
+ // tree so bare-mirror layouts and plain clones both land on info/exclude.
211
+ const common = Bun.spawnSync(["git", "rev-parse", "--git-common-dir"], {
212
+ cwd: worktree,
213
+ stdin: "ignore",
214
+ stdout: "pipe",
215
+ stderr: "pipe",
216
+ env: { ...process.env, GIT_TERMINAL_PROMPT: "0" },
217
+ });
218
+ if (common.exitCode !== 0) return;
219
+ const raw = common.stdout.toString().trim();
220
+ if (raw === "") return;
221
+ const commonDir = raw.startsWith("/") ? raw : join(worktree, raw);
222
+ const exclude = join(commonDir, "info", "exclude");
223
+ mkdirSync(dirname(exclude), { recursive: true });
224
+ writeFileSync(exclude, mergeExclude(existsSync(exclude) ? readFileSync(exclude, "utf8") : ""));
225
+ } catch {
226
+ // ponytail: exclude heal is best-effort; salvage still prefers a commit of
227
+ // whatever git can see over failing the whole orphan path.
228
+ }
229
+ }
230
+
185
231
  /**
186
232
  * Returns the path of the bare mirror for `repo`, cloning it on first use and
187
233
  * refreshing it otherwise.
@@ -239,7 +285,7 @@ export async function addWorktree(
239
285
  workspaceRoot: string,
240
286
  issue: number,
241
287
  branch: string,
242
- ): Promise<string> {
288
+ ): Promise<{ path: string; reattached: boolean }> {
243
289
  const mirrorPath = await ensureMirror(repo, mirrorRoot);
244
290
  mkdirSync(workspaceRoot, { recursive: true });
245
291
 
@@ -323,7 +369,7 @@ export async function addWorktree(
323
369
  }
324
370
  }
325
371
 
326
- return worktreePath;
372
+ return { path: worktreePath, reattached: branchExists };
327
373
  }
328
374
 
329
375
  /**
@@ -394,6 +440,12 @@ export async function salvageWip(
394
440
  // code, and "no tree" is not a salvage failure worth alarming anyone with.
395
441
  if (!existsSync(worktree)) return { kind: "nothing" };
396
442
 
443
+ // Heal the managed ignore *before* status/add. A mirror left by an older
444
+ // build may still list patterns this release dropped; without this, an
445
+ // untracked deliverable matching the stale list is invisible to salvage
446
+ // and dies with the next `worktree remove --force` (#44).
447
+ refreshManagedExclude(worktree);
448
+
397
449
  if ((await git(["status", "--porcelain"], worktree)) === "") return { kind: "nothing" };
398
450
 
399
451
  // The tree's own branch, not one the caller believes it should be on: this