omp-conductor 0.3.1 → 0.3.3

package/README.md

@@ -312,25 +312,32 @@ Per tick, for the daemon's project:
 1. **Paused?** If the pause sentinel exists, the tick claims nothing and returns.
    Pause is checked first, so `omp-conductor pause` takes effect on the next tick
    without signalling the process.
-2. **List the queue.** Open issues in `tracker.repo` labelled `queueLabel`.
-3. **Filter and route.** An issue is eligible only if it carries the queue label
+2. **Settle the green PRs.** For every run in `pushed-green`, ask the tracker
+   what became of its PR. Merged → the row becomes `merged`; closed without
+   merging → `failed`, with the PR named in the row's `lastError`. Still open, or
+   an answer that could not be obtained at all → the row is left exactly as it
+   is. Bounded by the number of green PRs awaiting a merge, which is a handful by
+   construction. This runs above admission so a row settled here frees its issue
+   in the same tick. See [what settles a green PR](#what-settles-a-green-pr).
+3. **List the queue.** Open issues in `tracker.repo` labelled `queueLabel`.
+4. **Filter and route.** An issue is eligible only if it carries the queue label
    and none of the three state labels (`inProgress`, `blocked`, `failed`). Eligible
    issues are partitioned into routable and unroutable.
-4. **Escalate the unroutable** at Tier 1, quoting the repo labels actually seen and
+5. **Escalate the unroutable** at Tier 1, quoting the repo labels actually seen and
    the configured repo names. These are never dispatched.
-5. **Check spend.** If spend since local midnight has reached `dailySpendUsd`, the
+6. **Check spend.** If spend since local midnight has reached `dailySpendUsd`, the
    daemon **pauses itself**, pages at Tier 2, and returns.
-6. **Check capacity.** `maxConcurrentWorkers` minus *live* workers (runs in
+7. **Check capacity.** `maxConcurrentWorkers` minus *live* workers (runs in
    `claimed` or `running`) gives the free slots. A green PR awaiting a human
    merge occupies its issue but not a slot: its worker is finished, and counting
    it would let two green PRs stop the fleet. If no slot is free, the tick logs
    and returns.
-7. **Admit issues** up to the free slots, skipping any issue that already has
+8. **Admit issues** up to the free slots, skipping any issue that already has
    an active run — including a green PR, so a second attempt can never land on a
    live PR. An issue that has used `maxAttemptsPerIssue` escalates at Tier 1
    instead of being admitted.
-8. **Ask the tracker whether the work already exists.** For each candidate that
-   survived step 7 — so at most one API call per free slot, never one per queued
+9. **Ask the tracker whether the work already exists.** For each candidate that
+   survived step 8 — so at most one API call per free slot, never one per queued
    issue — the daemon asks whether an **open** PR already closes the issue. If one
    does, the issue is skipped with its PR named in the log. Drafts count: a draft
    PR's branch still holds the only copy of the work. This is the guard the store
@@ -340,7 +347,7 @@ Per tick, for the daemon's project:
    retried next tick: the cost of holding is five minutes, the cost of admitting
    on an unknown is a burned attempt and a duplicate PR. Only that candidate is
    held, so a flaky API cannot stall the rest of the queue.
-9. **Dispatch** the admitted issues concurrently.
+10. **Dispatch** the admitted issues concurrently.
 
 Then, per admitted issue:
 
@@ -368,6 +375,9 @@ Then, per admitted issue:
    | `failed` / `killed` | swapped to `agent:failed` | dirty tree committed to the branch, then **kept** as evidence | Tier 1 |
    | unexpected error | swapped to `agent:failed` | same | Tier 1 |
 
+   `pushed-green` is not the end of the row: a later tick settles it once the PR
+   resolves. See [what settles a green PR](#what-settles-a-green-pr).
+
    Label swaps add the new label before removing the old one: the reverse order
    leaves a window in which the issue carries no state label at all, which is
    exactly the shape eligibility reads as fresh work.
@@ -405,6 +415,34 @@ clean orphan has its label released so the next tick re-claims it. Orphaned
 attempts still count toward `maxAttemptsPerIssue`, so a crash loop escalates
 instead of redispatching forever.
 
+### What settles a green PR
+
+`pushed-green` means the worker finished, pushed, and watched the checks go green.
+What happens next is a human's decision, taken minutes to days later and never
+announced to the daemon — so every tick asks the tracker about every green PR it
+is still holding, and settles the ones that resolved:
+
+| PR | Row becomes | Why |
+| --- | --- | --- |
+| merged | `merged` | The work landed. This is the state `merged` was reserved for. |
+| closed without merging | `failed`, with the PR in `lastError` | A human read the work and said no. Leaving it `pushed-green` strands the issue forever behind a PR nobody will merge, and calling it `merged` is a lie about code that is not on the base branch. `failed` is true, and it releases the issue so a re-queue can be attempted again. |
+| still open | unchanged | The normal steady state. Its issue must stay occupied, or a second attempt lands on the live PR. |
+| could not be determined | unchanged | A flaky network, a revoked token, a deleted PR. An unknown answer never settles a row; the next tick asks again for free. |
+
+The attempt counter is untouched either way, because both were real attempts. Run
+rows are the only thing that changes: a merge normally closes the issue, and a
+human who closed a PR is already looking at it, so what an issue's labels should
+say next is the orchestrator's drain-duty judgement — the same division of labour
+as a restart, above. One unreachable PR costs its own row and nothing else; the
+rest of the sweep still settles.
+
+Until this existed, nothing ever revisited a `pushed-green` row: the startup
+reconciler only settles rows that held a process, and `merged` went unwritten. On
+2026-08-07 the reference fleet reported three active runs whose PRs were all
+merged and whose issues were all closed, through two daemon restarts — and because
+the active set *is* the busy set, those three issues were permanently unclaimable.
+A status page that has stopped being evidence is worse than no status page.
+
 ### Branch names
 
 `<type>/<slug>`, where the type is `fix` when any label's last segment (after `:`
@@ -769,23 +807,30 @@ prompt was never consumed, and the extension:
 Both escapes deliberately leave the session, because a loop that cannot drain
 its queue cannot report on itself — that is the whole failure.
 
-**Two things read it.** A marker nobody consumes is an artifact, not an alert:
-
-- **The daemon**, on its own five-minute tick, and *before* its pause check. The
-  orchestrator is a different process and can be wedged while the fleet is
-  deliberately paused — precisely the state the dogfood fleet was in when this
-  happened. One tier-2 page per stall, dated, re-armed when the marker clears,
-  and latched only once the page is confirmed delivered, so an escalation
-  channel that fails on the one tick that noticed does not buy permanent
-  silence. It restarts nothing: a wedge lands mid-turn, and this process cannot
-  tell a half-applied edit from an idle loop.
-- **herdr-conductor's recovery**, which treats marker-present as *not live*. Its
-  liveness test — agent listed AND a non-shell foreground process — passes
-  straight through a wedge, so without this a recovery run certifies a stuck
-  session as healthy and returns "nothing to do".
-
-The daemon is what covers a wedge beginning *between* herdr's lifecycle events,
-since a session that stays alive and stops working emits none of them.
+**The daemon reads it.** A marker nobody consumes is an artifact, not an alert,
+so the dispatch daemon checks it on its own five-minute tick — and *before* its
+pause check. The orchestrator is a different process and can be wedged while
+the fleet is deliberately paused, which is precisely the state the dogfood
+fleet was in when this happened. One tier-2 page per stall, keyed on the
+marker's own timestamp so a second wedge the same day is not swallowed as a
+repeat, re-armed when the marker clears, and latched only once the page is
+confirmed delivered — an escalation channel that fails on the one tick that
+noticed must not buy permanent silence.
+
+It restarts nothing. A wedge lands mid-turn, and no other process can tell a
+half-applied edit from an idle loop; the operator attaches, looks, and decides.
+
+**herdr-conductor deliberately does not read it**, though its liveness test
+(agent listed AND a non-shell foreground process) passes straight through a
+wedge. That plugin only runs on `startup`, `pane.exited` and
+`pane.agent_detected`, and a session that stays alive and stops working emits
+none of them — so the check could never fire during the wedge itself. What it
+*would* catch is the recovery afterwards: the marker survives a restart until
+the new session consumes a tick, so every operator SIGTERM-and-resume would
+page about the healthy session they just fixed. Telling those apart needs the
+process start time against the marker's, and herdr's `pane process-info`
+reports pids, not start times. The daemon gives up at most one tick of
+coverage and never cries wolf.
 
 The first tick that actually sends clears the counter and deletes the marker,
 and it deletes one it did not write: recovery normally arrives as a fresh
@@ -863,6 +908,11 @@ Any other path or method returns `404`. Note that `ok` reports only that the
 process is serving. It does not report that the fleet is doing work: read
 `paused` to tell those apart.
 
+`activeRuns` counts occupied issues — live workers plus green PRs still awaiting a
+merge — and each of those is settled by the tick once its PR resolves, so the
+number goes back down on its own. A count that only ever grows is the bug this
+used to have: read [what settles a green PR](#what-settles-a-green-pr).
+
 ## What a worker may and may not do
 
 Each worker gets one brief, one worktree, one branch, and no knowledge of the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omp-conductor",
-  "version": "0.3.1",
+  "version": "0.3.3",
   "type": "module",
   "license": "MIT",
   "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",

package/src/briefs/orchestrator.md

@@ -66,14 +66,28 @@ by design — it is the guard that stops the next tick double-dispatching — bu
 nothing removes it, so a dead worker's issue sits "in progress" forever, occupying
 a slot that no longer exists. Compare the in-progress labels against the active
 runs `omp-conductor status` just showed you: **an in-progress issue with no
-matching active run is an orphan.**
+matching active run is unattended — nobody is working under that label, and only
+you can say why.**
 
-For an orphan, inspect what the dead worker left before touching the label. Read
+For each, inspect what is actually there before touching the label. Read
 the issue itself (`gh issue view <n> --json labels` — the label-filtered *list*
 reads GitHub's eventually-consistent search index and lags label writes in both
 directions), then the worktree (`git status --porcelain`, `git log
-origin/main..HEAD`) and any PR. Four cases, checked in this order:
-
+origin/main..HEAD`) and any PR. Not every one of these is a dead worker: the
+conductor settles a green run's row once its PR resolves, so a resolved PR whose
+issue nobody relabelled looks identical to an orphan from the outside. Six cases,
+checked in this order:
+
+- **A merged PR.** Nothing died — the work landed and the row already says
+  `merged`. Never release the label: the open-PR guard only objects to an *open*
+  PR, so a re-claim would re-implement a PR that is already on the base branch.
+  Close the issue if the merge satisfied it (a merge with no closing keyword
+  leaves it open), take the queue label off, and remove the in-progress label last.
+- **A PR closed without merging.** A human read the work and said no; the row says
+  `failed`. Read the rejection before you touch anything — most of the time a
+  review comment is a spec change. Fold what it says into the issue, then release
+  the label so the next tick can attempt it again; the attempt counter still bounds
+  it. If the answer was "this should not be built", take it off the queue instead.
 - **An open PR that is green.** That worker finished; it just never got to report.
   This is the "already done" case above — handle it exactly the same way. Never
   release-and-re-claim it — a fresh worker would duplicate a finished run.

package/src/daemon.ts

@@ -23,6 +23,7 @@ import { makeTracker } from "./tracker/github.ts";
 import type {
   Caps,
   Escalation,
+  PrState,
   ProjectConfig,
   ReadyIssue,
   RepoTarget,
@@ -151,10 +152,13 @@ async function watchOrchestrator(d: Deps): Promise<void> {
     tier: 2,
     project: d.project.name,
     issue: NO_ISSUE,
-    // Dated like the other tier-2 summaries: the dedup key is the summary, and
-    // a second wedge next month must not read as a repeat of this one.
+    // Keyed on the marker's own timestamp, not the date. The dedup ledger keys
+    // on this summary, and two wedges in one day is not a hypothetical — the
+    // failure mode is a session that gets stuck, gets restarted, and gets stuck
+    // again on the same cause an hour later. A day-keyed summary would report
+    // the first and silently swallow every one after it.
     summary:
-      `Orchestrator session wedged on ${new Date().toISOString().slice(0, 10)} — ` +
+      `Orchestrator session wedged (${verdict.since ?? `marker at ${marker}`}) — ` +
       `it has stopped reading its queue (${d.project.name})`,
     detail: [
       verdict.since ?? "Marker present with no readable timestamp.",
@@ -626,6 +630,102 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
   }
 }
 
+// ------------------------------------------------------------------- settlement
+
+/** What a resolved PR turns its `pushed-green` row into. */
+export interface Settlement {
+  state: "merged" | "failed";
+  /** The log line after `#<n> settled: `, and — for a rejection — the row's own
+   *  `lastError`, because a `failed` row whose worker succeeded has to say so. */
+  reason: string;
+}
+
+/**
+ * What one `pushed-green` row becomes now its PR has an answer, or undefined to
+ * leave the row exactly as it is.
+ *
+ * A `pushed-green` row is the only one nothing ever revisited: the worker is
+ * finished, `reconcileOrphanedRuns` only settles rows that held a process, and
+ * `merged` went unwritten from day one. So they accumulated — three of them on
+ * the reference fleet on 2026-08-07, every PR merged and every issue closed,
+ * with `/healthz` still reporting three active runs and their issues
+ * permanently unclaimable, because the busy set *is* the active set (#18).
+ *
+ * The mapping, and why each answer is the only honest one:
+ *
+ * - `merged` — the work landed. That is what `merged` was reserved for.
+ * - `closed` — a human read the work and said no. Leaving it `pushed-green`
+ *   strands the issue forever behind a PR nobody will ever merge, and calling it
+ *   `merged` is simply a lie about work that does not exist on the base branch.
+ *   `failed` is true — the attempt did not land — and it releases the busy guard,
+ *   so an issue a human re-queues can be attempted again. The attempt counter is
+ *   untouched either way: this row was a real attempt, and pretending otherwise
+ *   would let a rejected issue cycle past `maxAttemptsPerIssue`.
+ * - `open`, and undefined — nothing changes. Undefined is "could not tell": a
+ *   flaky network, a revoked token, a deleted PR. Settling on it would record a
+ *   merge that never happened, and the next tick asks again for free. An
+ *   ambiguous answer must never settle a row.
+ */
+export function settlementFor(pr: PrState | undefined, prUrl: string): Settlement | undefined {
+  if (pr === "merged") return { state: "merged", reason: `${prUrl} merged` };
+  if (pr === "closed") return { state: "failed", reason: `${prUrl} closed without merging` };
+  return undefined;
+}
+
+/**
+ * Asks the tracker about every `pushed-green` PR and settles the ones that
+ * resolved.
+ *
+ * Effects at the call site, decision in {@link settlementFor} — the same split
+ * as `checkIntegrity`/`watchOrchestrator`. Exported like `admitCandidates`
+ * rather than kept private, because half of what has to hold is about the sweep
+ * and not the mapping: that a row without a PR costs no API call, and that one
+ * unreachable PR does not stop the others from settling.
+ *
+ * Tracker labels are deliberately not touched, exactly as
+ * {@link reconcileOrphanedRuns} does not touch them. A merge normally closes the
+ * issue, and a human who closed a PR is already looking at it; deciding what an
+ * issue's labels should say next is the orchestrator's drain duty, which reads
+ * these very rows through `omp-conductor status`.
+ */
+export async function settlePushedGreen(
+  d: Pick<Deps, "project" | "tracker" | "store">,
+): Promise<void> {
+  const { project, tracker, store } = d;
+  // Filtered from the active set rather than asked for with a new query: active
+  // is live workers plus these, so the list is bounded by the worker cap plus
+  // the number of PRs awaiting a merge — a handful, by construction. A fleet
+  // where that is not a handful has a merge problem, not a dispatch one.
+  const pending = store.activeRuns(project.name).filter((r) => r.state === "pushed-green");
+
+  for (const run of pending) {
+    // Nothing to ask about. A green push means a PR, so this row should not
+    // exist; if one ever does, it must not buy a `gh` call every five minutes
+    // forever to be told nothing.
+    if (run.prUrl === undefined) continue;
+
+    let pr: PrState | undefined;
+    try {
+      pr = await tracker.prState(run.prUrl);
+    } catch (err) {
+      // Per row, like admission's held candidate. The GitHub adapter already
+      // answers undefined instead of throwing, so this catch is the port's
+      // contract rather than that adapter's behaviour — and a tracker that does
+      // throw must cost its own row, not the whole sweep.
+      log(`#${run.issue} not settled: PR state lookup failed (${errText(err)}) — retrying next tick`);
+      continue;
+    }
+
+    const settlement = settlementFor(pr, run.prUrl);
+    if (settlement === undefined) continue;
+
+    const patch: Partial<RunRecord> = { state: settlement.state, endedAt: Date.now() };
+    if (settlement.state === "failed") patch.lastError = settlement.reason;
+    store.updateRun(run.id, patch);
+    log(`#${run.issue} settled: ${settlement.reason}`);
+  }
+}
+
 // -------------------------------------------------------------------- admission
 
 /** A candidate cleared for dispatch, with the attempt number it will run as. */
@@ -731,6 +831,15 @@ async function tick(d: Deps): Promise<void> {
   // attributable. A legitimate deploy never trips it, because installing a new
   // build and restarting the unit re-records the baseline from the new files;
   // only an edit *underneath* a live daemon diverges from it.
+  //
+  // Below the pause gate on purpose, unlike the stall watch above. The property
+  // being defended is that no work is dispatched under a package the operator
+  // did not install — and a paused fleet dispatches nothing, so nothing needs
+  // attributing yet. Tampering during a pause is not missed, only deferred: the
+  // baseline is boot's, so the first tick after `resume` compares against it and
+  // pauses again before claiming anything. Checking above the gate instead would
+  // page on every legitimate build an operator deploys into a parked fleet,
+  // which is exactly when they deploy them.
   const integrity = checkIntegrity(d.integrity, packageManifest());
   if (integrity.pause) {
     const shown = integrity.diff.slice(0, INTEGRITY_SAMPLE);
@@ -766,6 +875,13 @@ async function tick(d: Deps): Promise<void> {
     return;
   }
 
+  // Above admission on purpose: a row settled this tick frees its issue for
+  // this same tick, so a merge and a re-queue no longer cost five minutes each.
+  // Above the spend cap too, which returns early — settling is bookkeeping about
+  // work already paid for, and a fleet that halts itself is exactly when an
+  // operator reads `status` and needs it to be true.
+  await settlePushedGreen(d);
+
   // route() filters the queue through isEligible() itself, so anything already
   // carrying a state label is gone before it gets here.
   const { routed, unroutable } = route(await d.tracker.listReady(), project);
@@ -973,7 +1089,9 @@ export function armConductor(): void {
  *
  * `pushed-green` rows are deliberately left alone: they hold no process — they
  * are finished work waiting on a human merge, and they must keep occupying the
- * issue so a second attempt cannot land on a live PR.
+ * issue so a second attempt cannot land on a live PR. What eventually settles
+ * them is {@link settlePushedGreen}, on the tick, by asking the tracker what
+ * became of the PR — the one question a restart cannot answer by inference.
  */
 export function reconcileOrphanedRuns(store: Store, project: string): RunRecord[] {
   // Live runs only: `pushed-green` holds no process, so it cannot be orphaned by

package/src/tracker/github.ts

@@ -10,10 +10,10 @@
  * ~200-400ms) and failure classification by matching human-readable stderr
  * instead of reading a status code. Upgrade path when either bites: replace the
  * body of `gh()` with `fetch("https://api.github.com/...")` using a token from
- * `gh auth token`; the seven Tracker methods above it stay untouched.
+ * `gh auth token`; the eight Tracker methods above it stay untouched.
  */
 
-import type { ProjectConfig, ReadyIssue, Tracker } from "../types.ts";
+import type { PrState, ProjectConfig, ReadyIssue, Tracker } from "../types.ts";
 
 /** The subset of `gh issue list --json` output this adapter reads. Fields the
  *  API can return as null are typed as such so the mapping has to handle it. */
@@ -154,6 +154,41 @@ export function firstOpenCloser(raw: string): string | undefined {
   return nodes.find((n) => n !== null && n.state === "OPEN")?.url;
 }
 
+/**
+ * A pull request URL this adapter is willing to hand to `gh`.
+ *
+ * Load-bearing, not defensive: `gh pr view` also accepts a bare number or a
+ * *branch name*, and resolves those against whatever repository the current
+ * directory belongs to. So a `prUrl` that is not a URL would not fail — it would
+ * be answered, confidently, about some other repository's pull request. A
+ * settle sweep that acted on that answer would mark the wrong run merged.
+ */
+const PR_URL = /^https?:\/\/[^\s/]+\/[^\s/]+\/[^\s/]+\/pull\/\d+/;
+
+/**
+ * GitHub's PR state spelling mapped onto {@link PrState}.
+ *
+ * Split from the call so the mapping is pinned without a network round trip,
+ * and unrecognised text answers undefined rather than being coerced: this
+ * function's caller settles a run row on the answer, and every wrong answer
+ * here rewrites history for work that has no other record. `MERGED`, `CLOSED`
+ * and `OPEN` are the only three `gh pr view --json state` emits (verified on
+ * gh 2.97.0); anything else means the CLI changed under us, which is a reason
+ * to leave the row alone and let a human look.
+ */
+export function prStateFrom(raw: string): PrState | undefined {
+  switch (raw.trim()) {
+    case "MERGED":
+      return "merged";
+    case "CLOSED":
+      return "closed";
+    case "OPEN":
+      return "open";
+    default:
+      return undefined;
+  }
+}
+
 export function makeTracker(p: ProjectConfig): Tracker {
   const repo = p.tracker.repo;
 
@@ -248,5 +283,24 @@ export function makeTracker(p: ProjectConfig): Tracker {
 
       return firstOpenCloser(raw);
     },
+
+    async prState(url: string): Promise<PrState | undefined> {
+      // No `--repo`: a full URL is self-locating, and verified so on gh 2.97.0
+      // from a directory that is not a git repository at all — which is exactly
+      // where the daemon runs (its state directory), while the PR lives in one
+      // of the routed repos. Deriving `--repo` from the URL would only re-state
+      // what the URL already says.
+      if (!PR_URL.test(url)) return undefined;
+      try {
+        return prStateFrom(await gh(["pr", "view", url, "--json", "state", "--jq", ".state"]));
+      } catch {
+        // Never throws, per the port's contract. A deleted PR, a revoked token
+        // and a flaky network all mean "could not tell", and the caller's whole
+        // job is to leave the row alone on that — so classifying them here would
+        // buy nothing but a way to get the classification wrong. The next tick
+        // asks again for free.
+        return undefined;
+      }
+    },
   };
 }

package/src/types.ts

@@ -180,6 +180,15 @@ export interface ReadyIssue {
   updatedAt: string;
 }
 
+/**
+ * How a pull request ended, in tracker-agnostic terms. Lowercase because the
+ * loop's vocabulary is lowercase; mapping GitHub's `MERGED`/`CLOSED`/`OPEN` onto
+ * it is the adapter's job.
+ *
+ * `closed` is closed *without* merging — a human looked at the work and said no.
+ */
+export type PrState = "merged" | "closed" | "open";
+
 /**
  * Deliberately narrow so a Gitea or local-file tracker can drop in later.
  * Nothing here is GitHub-shaped; the GitHub adapter owns `gh` entirely.
@@ -205,6 +214,19 @@ export interface Tracker {
    * only party that remembers across all of those.
    */
   openCloserFor(issue: number): Promise<string | undefined>;
+  /**
+   * The state of one specific pull request, or undefined when this adapter
+   * could not tell — a network failure, a deleted PR, a URL it cannot parse.
+   * Undefined never means "no".
+   *
+   * Deliberately separate from {@link Tracker.openCloserFor}, which answers a
+   * different question: that one asks "should this candidate be admitted", and
+   * answers undefined for merged, closed-unmerged, no PR at all and a deleted
+   * PR alike, because admission treats all four the same. This one asks "did
+   * *this* run's PR land", where those answers are opposites. Collapsing them
+   * is how a PR a human rejected gets recorded as merged.
+   */
+  prState(url: string): Promise<PrState | undefined>;
 }
 
 /**
@@ -216,6 +238,8 @@ export type RunState =
   | "claimed"
   | "running"
   | "pushed-green"
+  /** Its PR landed. Written by the tick's settle sweep, never by a worker: only
+   *  the tracker knows, and it knows minutes to days after the run ended. */
   | "merged"
   | "blocked"
   | "failed"