omp-conductor 0.2.1 → 0.2.2

package/README.md

@@ -317,10 +317,14 @@ Per tick, for the daemon's project:
    the configured repo names. These are never dispatched.
 5. **Check spend.** If spend since local midnight has reached `dailySpendUsd`, the
    daemon **pauses itself**, pages at Tier 2, and returns.
-6. **Check capacity.** `maxConcurrentWorkers` minus active runs gives the free
-   slots. If none are free, the tick logs and returns.
+6. **Check capacity.** `maxConcurrentWorkers` minus *live* workers (runs in
+   `claimed` or `running`) gives the free slots. A green PR awaiting a human
+   merge occupies its issue but not a slot: its worker is finished, and counting
+   it would let two green PRs stop the fleet. If no slot is free, the tick logs
+   and returns.
 7. **Admit issues** up to the free slots, skipping any issue that already has
-   an active run. An issue that has used `maxAttemptsPerIssue` escalates at Tier 1
+   an active run — including a green PR, so a second attempt can never land on a
+   live PR. An issue that has used `maxAttemptsPerIssue` escalates at Tier 1
    instead of being admitted.
 8. **Dispatch** the admitted issues concurrently.
 
@@ -330,9 +334,9 @@ Then, per admitted issue:
    This ordering is the whole crash-safety story: the label, not the local
    database, is the guard against dispatching the same issue twice. If the process
    dies at any later point, the next daemon sees the label, eligibility filters the
-   issue out, and a human decides what to do with the orphan. A store that is lost
-   can be rebuilt from the tracker; a label that was written too late cannot undo a
-   duplicate PR.
+   issue out, and the orchestrator's drain duty triages the orphan (see below). A
+   store that is lost can be rebuilt from the tracker; a label that was written too
+   late cannot undo a duplicate PR.
 2. Create the run row (`claimed`).
 3. Clear any stale tree for this issue, then add a fresh worktree at
    `<workspaceRoot>/<issue>` cut from the bare mirror at `<mirrorRoot>/<repo>.git`,
@@ -354,6 +358,26 @@ Then, per admitted issue:
    leaves a window in which the issue carries no state label at all, which is
    exactly the shape eligibility reads as fresh work.
 
+### What a restart does to runs that were in flight
+
+A `claimed` or `running` row is a promise that a worker process exists, and a
+daemon that just started knows that promise is broken: its workers died with the
+previous process. At startup — unless another daemon is alive, so a foreground
+`daemon --once` cannot orphan a running daemon's real workers — every such row is
+moved to `orphaned`, with a log line naming the issue, the attempt and the
+worktree. That frees the slots immediately; a fleet must never resume as
+deadlocked as it crashed.
+
+Only the rows change. The issue keeps `agent:in-progress` — the label is the
+crash guard against double-dispatch — and deciding what the dead worker's remains
+are worth is the orchestrator's drain-duty judgement, spelled out in its brief:
+an open green PR goes to the merge path, a dirty tree is reported before anything
+destroys it (uncommitted edits have no other copy; unpushed *commits* are safe on
+the run's branch in the mirror, which a retry deliberately reattaches), and a
+clean orphan has its label released so the next tick re-claims it. Orphaned
+attempts still count toward `maxAttemptsPerIssue`, so a crash loop escalates
+instead of redispatching forever.
+
 ### Branch names
 
 `<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
@@ -766,10 +790,10 @@ Known and deliberate in this version:
   (the orchestrator's) no matter how many workers are running, and no amount of
   `maxConcurrentWorkers` changes that.
 
-  The cap does work. The admission loop (`src/daemon.ts:405-445`) computes
-  `slots = maxConcurrentWorkers - active runs`, admits at most that many issues per
-  tick, and dispatches them together. To see them, read `omp-conductor status`,
-  which lists every active run, or follow `daemon.log`.
+  The cap does work. The admission loop (`src/daemon.ts:410-448`) computes
+  `slots = maxConcurrentWorkers - live workers`, admits at most that many issues
+  per tick, and dispatches them together. To see them, read `omp-conductor
+  status`, which lists every occupied issue, or follow `daemon.log`.
 - **Merges, releases and deploys are human-only, by design.** The conductor
   produces green PRs and stops.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omp-conductor",
-  "version": "0.2.1",
+  "version": "0.2.2",
   "type": "module",
   "license": "MIT",
   "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",

package/src/briefs/orchestrator.md

@@ -62,15 +62,28 @@ a slot that no longer exists. Compare the in-progress labels against the active
 runs `omp-conductor status` just showed you: **an in-progress issue with no
 matching active run is an orphan.**
 
-For an orphan, look at what the dead worker left — its branch, any commits, an
-open PR — then pick one:
-
-- **Real progress exists** (commits or an open PR). Note the issue, the branch and
-  what state it reached, and remove the in-progress label so the loop can re-claim
-  it. The next worker starts from the branch's actual state rather than from
-  nothing, and the attempt counter still protects against a loop of deaths.
-- **Nothing useful exists.** Remove the in-progress label and let the next tick
-  re-claim it clean.
+For an orphan, inspect what the dead worker left before touching the label. Read
+the issue itself (`gh issue view <n> --json labels` — the label-filtered *list*
+reads GitHub's eventually-consistent search index and lags label writes in both
+directions), then the worktree (`git status --porcelain`, `git log
+origin/main..HEAD`) and any PR. Four cases, checked in this order:
+
+- **An open PR that is green.** That worker finished; it just never got to report.
+  This is the "already done" case above: note it with the link and move on. Never
+  release-and-re-claim it — a fresh worker would duplicate a finished run.
+- **A dirty tree** (uncommitted edits in the worktree). This is the one thing a
+  re-claim destroys: the conductor removes and reattaches worktrees with `--force`
+  on every attempt, and uncommitted edits have no other copy. Do not release the
+  label yet — report what exists and where, and let your operator decide whether
+  it is worth salvaging. Uncommitted edits are work too; "nothing committed" is
+  not "nothing there".
+- **Commits — pushed or not — or a PR that is not green.** Safe either way:
+  pushed work lives on the remote, and unpushed commits live on the run's branch
+  in the mirror, which a re-claim deliberately reattaches so the next worker
+  starts from them. Note what exists and release the label; the attempt counter
+  still bounds a loop of deaths.
+- **Genuinely nothing** (clean tree, no commits, no PR). Release the label and let
+  the next tick re-claim it clean.
 
 Never leave an orphan holding a slot "to be safe": a label nobody is working under
 is not safety, it is a deadlocked fleet that looks busy.
@@ -147,13 +160,32 @@ trigger an amendment:
 The protocol, in order:
 
 1. **Draft the exact replacement.** Quote the lines as they stand, then the lines
-   you propose. A diff, not a description of one.
-2. **Ask, once.** Send it as a single yes/no question over the escalation channel
-   (the `ask` tool — it reaches your operator's Telegram).
+   you propose. A diff, not a description of one. This full text is what you
+   *apply* on a yes — it is not what you send.
+2. **Ask, once — a single yes/no question, written for a phone.** It goes over
+   the escalation channel (the `ask` tool — it reaches your operator's Telegram),
+   and Telegram renders none of your markdown: asterisks and backticks arrive as
+   literal characters, and a pasted section becomes an unreadable wall. So:
+   - Lead with one plain sentence: what changes, and why, in your own words.
+   - Then show only the lines that actually change, compact, under two short
+     labels like "now:" and "proposed:". Never paste whole sections around a
+     two-line change.
+   - Keep the whole proposal readable on one phone screen. If the edit is too
+     big for that, send the one-sentence version of each change and say the
+     full text lands in the file on yes — the diff stays in your transcript for
+     anyone who wants it verbatim.
 3. **On yes, apply it** by editing this file yourself. On no, or on no answer at
    all, drop it and do not re-ask that amendment.
 4. **Log it.** Append one line to **Amendments** at the bottom of this file: the
    date, what triggered it, a one-sentence summary.
+5. **Offer general fixes upstream.** Ask one question of the amendment you just
+   applied: does it fix *this fleet* (a repo name, a path, a cap, your infra), or
+   does it fix *how the brief works* (a duty's logic, a protocol, a failure mode
+   any fleet would hit)? The second kind belongs in the shipped template, or
+   every other operator re-learns it the hard way. Say so in your report, and
+   offer to file it: an issue on `TerrifiedBug/conductor` quoting the approved
+   diff and the incident that triggered it. File it only when your operator says
+   yes — it is their name on the account.
 
 Two limits. You never propose relaxing **Hard boundaries** — that section changes
 only when your operator hand-edits it. And at most one proposal per tick: an

package/src/daemon.ts

@@ -11,6 +11,7 @@ import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { configPath, findProject, loadConfig, resolveCaps, stateDir } from "./config.ts";
 import { createEscalator } from "./escalate.ts";
+import { livingDaemon } from "./lifecycle.ts";
 import { startOrchestrator } from "./orchestrator.ts";
 import type { OrchestratorHandle } from "./orchestrator.ts";
 import { branchName, route } from "./routing.ts";
@@ -402,16 +403,18 @@ async function tick(d: Deps): Promise<void> {
     return;
   }
 
-  const active = store.activeRuns(project.name);
-  const slots = caps.maxConcurrentWorkers - active.length;
+  // Two different questions, deliberately two queries. Capacity counts worker
+  // *processes*, so a green PR awaiting a human merge must not consume a slot —
+  // two of those would otherwise stop the fleet. The busy set protects *issues*,
+  // so that same green PR must be in it, or a second attempt lands on a live PR.
+  const live = store.liveRuns(project.name);
+  const slots = caps.maxConcurrentWorkers - live.length;
   if (slots <= 0) {
-    log(`at capacity: ${active.length}/${caps.maxConcurrentWorkers} workers`);
+    log(`at capacity: ${live.length}/${caps.maxConcurrentWorkers} workers`);
     return;
   }
 
-  // activeRuns includes pushed-green work that is still waiting on a human
-  // merge, so this also stops a second attempt landing on a live PR.
-  const busy = new Set(active.map((r) => r.issue));
+  const busy = new Set(store.activeRuns(project.name).map((r) => r.issue));
 
   const admitted: { r: Routed; attempt: number }[] = [];
   for (const r of routed) {
@@ -453,7 +456,10 @@ export interface StatusSnapshot {
   stateDir: string;
   paused: boolean;
   caps: Caps;
+  /** Occupied issues: live workers plus green PRs awaiting a human merge. */
   activeRuns: RunRecord[];
+  /** Runs backed by a worker process — the number capacity compares against. */
+  liveWorkers: number;
   runsToday: number;
   spendTodayUsd: number;
 }
@@ -474,6 +480,7 @@ export function statusSnapshot(project?: string): StatusSnapshot {
       paused: isPaused(),
       caps: resolveCaps(p, cfg.defaults),
       activeRuns: store.activeRuns(p.name),
+      liveWorkers: store.liveRuns(p.name).length,
       runsToday: store.runsStartedSince(p.name, since),
       spendTodayUsd: store.spendSince(p.name, since),
     };
@@ -489,7 +496,7 @@ export function formatStatus(s: StatusSnapshot): string {
     `state     ${s.stateDir}`,
     "",
     "caps",
-    `  workers            ${s.activeRuns.length} / ${s.caps.maxConcurrentWorkers}`,
+    `  workers            ${s.liveWorkers} / ${s.caps.maxConcurrentWorkers}`,
     `  issues today       ${s.runsToday}`,
     `  spend today        $${s.spendTodayUsd.toFixed(2)} / $${s.caps.dailySpendUsd.toFixed(2)}`,
     `  worker max turns   ${s.caps.workerMaxTurns}`,
@@ -564,6 +571,38 @@ export function armConductor(): void {
   setPaused(false);
 }
 
+/**
+ * Settles the runs a previous daemon process left in flight.
+ *
+ * A `claimed` or `running` row is a promise that a worker exists in *some*
+ * process. This is called from a freshly started daemon, so when no other
+ * daemon is alive every such row is a worker that died with the previous
+ * process. Left "active", those rows deadlock admission forever: the slot
+ * count reads full while nothing runs, and the fleet looks busy doing nothing
+ * (found live, after a host restart killed two workers mid-run).
+ *
+ * Only the rows change. The issue keeps its in-progress label — that label is
+ * the crash guard against double-dispatch, and deciding what a dead worker's
+ * remains are worth (an open PR? unpushed commits? a dirty tree?) is the
+ * orchestrator's drain-duty judgement, not something to automate here. The
+ * rows also keep counting toward `maxAttemptsPerIssue`, so a loop of deaths
+ * still escalates instead of retrying forever.
+ *
+ * `pushed-green` rows are deliberately left alone: they hold no process — they
+ * are finished work waiting on a human merge, and they must keep occupying the
+ * issue so a second attempt cannot land on a live PR.
+ */
+export function reconcileOrphanedRuns(store: Store, project: string): RunRecord[] {
+  // Live runs only: `pushed-green` holds no process, so it cannot be orphaned by
+  // a process dying — it is finished work waiting on a human merge.
+  const stale = store.liveRuns(project);
+  const endedAt = Date.now();
+  for (const r of stale) {
+    store.updateRun(r.id, { state: "orphaned", endedAt });
+  }
+  return stale;
+}
+
 // ------------------------------------------------------------------- the daemon
 
 export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
@@ -573,6 +612,21 @@ export async function runDaemon(o: DaemonOpts = {}): Promise<void> {
   const store = openStore(dbPath());
   const tracker = makeTracker(project);
 
+  // Before the first tick, settle what the last process left behind — unless
+  // another daemon is alive (a foreground `daemon --once` beside a running
+  // daemon must not orphan that daemon's real, live workers).
+  const alive = livingDaemon();
+  if (alive === undefined || alive.pid === process.pid) {
+    for (const r of reconcileOrphanedRuns(store, project.name)) {
+      log(
+        `#${r.issue} orphaned by a previous daemon (attempt ${r.attempt}, was ${r.state}, worktree ${r.worktree}) — ` +
+          `slot freed; the ${project.stateLabels.inProgress} label stays until the orchestrator triages what the worker left`,
+      );
+    }
+  } else {
+    log(`skipping orphan reconciliation: daemon pid ${alive.pid} is alive and owns the active runs`);
+  }
+
   // Standing orders. The orchestrator holds none of this file's context, so
   // everything it needs to act — which tracker, which labels, what the fleet
   // does — has to be said once, in words.

package/src/store.ts

@@ -16,12 +16,21 @@ import { dirname } from "node:path";
 import type { RunRecord, RunState, Store } from "./types.ts";
 
 /**
- * States that consume a worker slot. `pushed-green` counts: the branch is
- * pushed and CI is green but nothing is merged, so the worktree, the branch
- * and the issue claim are all still held.
+ * States backed by a worker process. These are what worker capacity counts:
+ * a slot is a process, and only a claimed or running attempt has one.
  */
-const ACTIVE_STATES: readonly RunState[] = ["claimed", "running", "pushed-green"];
+const LIVE_STATES: readonly RunState[] = ["claimed", "running"];
 
+/**
+ * States that keep an *issue* occupied. `pushed-green` belongs here but not in
+ * {@link LIVE_STATES}: its worker is finished and its worktree already removed,
+ * so it must not consume a slot — two green PRs awaiting a human merge would
+ * otherwise stop the whole fleet — but its issue has a live PR that a second
+ * attempt must not land on.
+ */
+const ACTIVE_STATES: readonly RunState[] = [...LIVE_STATES, "pushed-green"];
+
+const LIVE_PLACEHOLDERS = LIVE_STATES.map(() => "?").join(", ");
 const ACTIVE_PLACEHOLDERS = ACTIVE_STATES.map(() => "?").join(", ");
 
 /**
@@ -165,6 +174,11 @@ export function openStore(dbPath: string): Store {
       WHERE project = ? AND state IN (${ACTIVE_PLACEHOLDERS})
       ORDER BY startedAt ASC`,
   );
+  const selectLive = db.query<RunRow, SqlValue[]>(
+    `SELECT * FROM runs
+      WHERE project = ? AND state IN (${LIVE_PLACEHOLDERS})
+      ORDER BY startedAt ASC`,
+  );
   const countAttempts = db.query<{ n: number }, [string, number]>(
     `SELECT COUNT(*) AS n FROM runs WHERE project = ? AND issue = ?`,
   );
@@ -233,6 +247,10 @@ export function openStore(dbPath: string): Store {
       return selectActive.all(project, ...ACTIVE_STATES).map(toRecord);
     },
 
+    liveRuns(project: string): RunRecord[] {
+      return selectLive.all(project, ...LIVE_STATES).map(toRecord);
+    },
+
     attemptsFor(project: string, issue: number): number {
       return countAttempts.get(project, issue)?.n ?? 0;
     },

package/src/types.ts

@@ -170,7 +170,9 @@ export type RunState =
   | "merged"
   | "blocked"
   | "failed"
-  | "killed";
+  | "killed"
+  /** In flight when its daemon process died; reconciled at the next startup. */
+  | "orphaned";
 
 /**
  * One attempt at one issue. Persisted so a daemon restart can reconcile
@@ -207,7 +209,10 @@ export interface Store {
   createRun(r: Omit<RunRecord, "id">): RunRecord;
   updateRun(id: string, patch: Partial<RunRecord>): void;
   getRun(id: string): RunRecord | undefined;
+  /** Runs whose issue is occupied: a live worker, or a green PR awaiting merge. */
   activeRuns(project: string): RunRecord[];
+  /** Runs backed by a worker process — what capacity counts. Subset of {@link Store.activeRuns}. */
+  liveRuns(project: string): RunRecord[];
   attemptsFor(project: string, issue: number): number;
   runsStartedSince(project: string, sinceEpochMs: number): number;
   spendSince(project: string, sinceEpochMs: number): number;