omp-conductor 0.9.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -538,7 +538,8 @@ add waits with it.
538
538
 
539
539
  `omp-conductor board [--project NAME]` opens the same facts as a live terminal
540
540
  kanban instead of a scrolling wall of status text. Its columns are Queue,
541
- Claimed, Running, Green, Blocked, Failed, Orphaned, Merged, and History.
541
+ Claimed, Running, Green, Blocked, Failed, Orphaned, Merged, Settled, and
542
+ Parked.
542
543
 
543
544
  Every column is a claim about an issue's state **now**, and each issue appears
544
545
  in exactly one of them. Claimed, Running, and Green come from the store, because
@@ -549,10 +550,17 @@ label, Blocked is the blocked state label, and Orphaned is an issue the
549
550
  in-progress label still claims with no worker running it — a protected orphan
550
551
  whose worktree is kept because it may hold uncommitted work. So a requeued issue
551
552
  is queue work even when its last attempt failed, and a closed issue is not
552
- active work at all. Merged and History are the two backward-looking columns,
553
- both bounded to the last 24 hours and neither one an action list: History holds
553
+ active work at all. Merged and Settled are the two backward-looking columns,
554
+ both bounded to the last 24 hours and neither one an action list: Settled holds
554
555
  recently finished runs that no current label claims, so the board stays
555
- operational rather than becoming an analytics archive.
556
+ operational rather than becoming an analytics archive. Parked is the issue's
557
+ last word when the tracker has **not** confirmed it closed — a terminal run
558
+ whose issue wears no state or queue label (still open, or a label read that
559
+ failed), so nothing dispatches it until a human labels it. Unlike Merged and
560
+ Settled it is not bounded to 24 hours, because parked work does not stop being
561
+ the issue's only remaining record; Settled alone asserts the issue is closed,
562
+ so an unreadable probe parks rather than letting a run masquerade as finished
563
+ work.
556
564
 
557
565
  The board refreshes run, spend, turn, and dispatch values from SQLite every
558
566
  second. It refreshes the slower daemon, Herdr, Telegram, code-graph, and tracker
@@ -1458,6 +1466,7 @@ persists the class on the row, and performs the one recovery that class names.
1458
1466
  | `admin-kill` | killed *below* its own ceiling — a restart or a drain | requeue | none |
1459
1467
  | `ci-infra` | PR open, every unresolved check cancelled / timed out / stale | re-run the failed jobs | none |
1460
1468
  | `ci-deterministic` | PR open, a check genuinely reports `FAILURE` | escalate with the failing check names and links | failed attempt |
1469
+ | `provider-credit` | the provider refused the run for credit (HTTP 402, or its own out-of-credit text read off the transcript) | pause the fleet and require `omp-conductor resume` once the provider has credit | none |
1461
1470
  | `unknown` | anything unrecognised | escalate | as recorded |
1462
1471
 
1463
1472
  **Unknown escalates; it never silently retries.** A shape this table does not
@@ -1466,11 +1475,14 @@ cause nobody has named — the behaviour this exists to end.
1466
1475
 
1467
1476
  ### The budgets follow the cause
1468
1477
 
1469
- `failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck`
1470
- and `env-start-failure`; `continuationsFor` excludes `admin-kill`,
1471
- `settlement-stuck` and `env-start-failure`. An environment fault charges neither:
1472
- the session never started, so nothing about the issue was attempted. A merge conflict *is* charged as a continuation, because a
1473
- rebase is real work just never as a failed implementation attempt.
1478
+ `failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck`,
1479
+ `env-start-failure` and `provider-credit`; `continuationsFor` excludes
1480
+ `admin-kill`, `settlement-stuck`, `env-start-failure` and `provider-credit`. An
1481
+ environment fault charges neither: the session never started, so nothing about
1482
+ the issue was attempted, and nor does a provider-credit refusal, which is a
1483
+ billing state rather than an implementation failure. A merge conflict *is*
1484
+ charged as a continuation, because a rebase is real work — just never as a
1485
+ failed implementation attempt.
1474
1486
 
1475
1487
  An **unclassified** row (every row written before 0.4.3) counts exactly as it
1476
1488
  did before classification existed. Upgrading therefore changes no existing
@@ -1971,7 +1983,7 @@ omp-conductor help
1971
1983
  | `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
1972
1984
  | `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
1973
1985
  | `ledger [--issue N] [--limit N]` | Every [mediated-verb](#the-mediated-verbs-126) call and how the daemon decided it: the verb, the arguments it was given, allow or refuse, the named refusal reason, and any resulting SHA. Sessions cannot push, open a PR, merge, label or release except through those verbs, so this is the record of what they *tried* as well as what they did — the thing an escalation cites. Reads (`conductor_pr_status`) are deliberately absent: a poll every thirty seconds would bury the refusals. `--issue` narrows to one issue's run; `--limit` defaults to 50. The newest few also appear in `status`. |
1974
- | `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, and the last 24 hours of Merged and History. Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
1986
+ | `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
1975
1987
  | `hold [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this over `pause` when the intent is "stop the conductor" without killing processes. See [Stop the conductor](#stop-the-conductor-hold--halt). |
1976
1988
  | `halt [--pane] [--project NAME]` | `hold`, then stop the dispatch daemon (systemctl-aware). Pane stays up unless `--pane` is passed. `halt --pane` also pins herdr-conductor recovery off for the conductor agent only — it does **not** stop `herdr-fleet.service` or any other herdr session. Fail-closed: exits nonzero unless the agent is proven gone. |
1977
1989
  | `arm [--project NAME]` | Proof-gated: send a Telegram challenge and write the arm marker only after your reply appears as a user turn in the orchestrator transcript. Never auto-armed by `resume` / `hold`. |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "omp-conductor",
3
- "version": "0.9.0",
3
+ "version": "0.10.0",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
package/src/board.ts CHANGED
@@ -72,9 +72,9 @@ const COLUMN_DEFS = [
72
72
  { key: "blocked", title: "BLOCKED" },
73
73
  { key: "failed", title: "FAILED" },
74
74
  { key: "orphaned", title: "ORPHANED" },
75
- { key: "parked", title: "PARKED" },
76
75
  { key: "merged", title: "MERGED" },
77
- { key: "history", title: "HISTORY" },
76
+ { key: "settled", title: "SETTLED" },
77
+ { key: "parked", title: "PARKED" },
78
78
  ] as const satisfies readonly { key: string; title: string }[];
79
79
 
80
80
  export type BoardLane = (typeof COLUMN_DEFS)[number]["key"];
@@ -151,11 +151,15 @@ export interface BoardLabels {
151
151
  inProgress: ReadonlySet<number>;
152
152
  blocked: ReadonlySet<number>;
153
153
  failed: ReadonlySet<number>;
154
- /** Issues whose newest run is terminal and unlabelled but that are still open
155
- * on the tracker the PARKED lane (#173), probed on the same 10s cadence as
156
- * the label sets. An issue missing from this set (closed, or the probe could
157
- * not tell) keeps today's HISTORY/undefined behaviour rather than parking. */
158
- open: ReadonlySet<number>;
154
+ /**
155
+ * Issues the tracker reports CLOSED, derived only from a *successful* label
156
+ * snapshot on the same 10s cadence as the other sets. This is the one fact
157
+ * that separates SETTLED (finished work, issue closed) from PARKED (a human
158
+ * still has to look): absence from this set means open **or** the probe could
159
+ * not tell, and either way the run parks rather than falsely asserting
160
+ * closure. On a failed read the last good set survives behind {@link error}.
161
+ */
162
+ closed: ReadonlySet<number>;
159
163
  /** Epoch ms of the last read that succeeded; 0 when none has. */
160
164
  readAt: number;
161
165
  /** Message from the most recent failed read. The sets above are then the last
@@ -171,7 +175,7 @@ const UNREAD_LABELS: BoardLabels = {
171
175
  inProgress: new Set<number>(),
172
176
  blocked: new Set<number>(),
173
177
  failed: new Set<number>(),
174
- open: new Set<number>(),
178
+ closed: new Set<number>(),
175
179
  readAt: 0,
176
180
  };
177
181
 
@@ -225,7 +229,7 @@ function ansiColor(key: BoardLane): string {
225
229
  return YELLOW;
226
230
  case "failed":
227
231
  return RED;
228
- case "history":
232
+ case "settled":
229
233
  return DIM;
230
234
  default:
231
235
  return MAGENTA;
@@ -285,10 +289,11 @@ function humanDuration(ms: number): string {
285
289
  * parent issues a human had already cleared (#81, #86) — every one of them a
286
290
  * historical terminal run, not one of them failed work.
287
291
  *
288
- * So: the store decides the live lanes, because a running worker is the most
292
+ * so: the store decides the live lanes, because a running worker is the most
289
293
  * current fact there is; the tracker's current labels decide every stopped
290
294
  * lane; each issue matches exactly once, first branch wins; and last-run
291
- * history is answered in HISTORY, under its own name.
295
+ * history is answered in SETTLED (issue closed) and PARKED (issue still open,
296
+ * labels cleared), under their own names.
292
297
  */
293
298
  function laneOf(
294
299
  snapshot: BoardSnapshot,
@@ -322,17 +327,19 @@ function laneOf(
322
327
  if (queued.has(issue)) return "queue";
323
328
  if (run?.state === "merged") return "merged";
324
329
  if (run === undefined) return undefined;
325
- // PARKED (#173): a terminal run row with no state or queue label left on it,
326
- // on an issue the tracker says is still open. A human cleared the labels but
327
- // has not acted the blank lane the four labels used to imply is a lie, and
328
- // closed or unreadable issues keep the HISTORY/undefined fallthrough below.
329
- // Deliberately not subject to MERGED_HISTORY_MS: parked work does not stop
330
- // being the issue's last word after 24 hours.
331
- if (PARKED_STATES.has(run.state) && labels.open.has(issue)) return "parked";
332
- // What is left is a terminal run row with no current claim on it — a closed
333
- // issue, or one a human already cleared. Recent rows stay readable under
334
- // HISTORY for the same 24 hours MERGED uses; older ones are not news.
335
- return (run.endedAt ?? run.startedAt) >= snapshot.now - MERGED_HISTORY_MS ? "history" : undefined;
330
+ // SETTLED vs PARKED, decided by the one fact the board can prove: whether the
331
+ // tracker confirmed the issue closed. Absence from `closed` means open OR the
332
+ // probe could not tell, and either way the run parks a failed label read
333
+ // must render the issue as "a human has to look", never as finished work
334
+ // (#218). PARKED is deliberately not subject to MERGED_HISTORY_MS: parked work
335
+ // does not stop being the issue's last word after 24 hours.
336
+ if (PARKED_STATES.has(run.state) && !labels.closed.has(issue)) return "parked";
337
+ // What is left is a confirmed-closed issue with a terminal run row. Recent
338
+ // rows stay readable under SETTLED for the same 24 hours MERGED uses; older
339
+ // ones are not news. Named for what it means the run settled and the issue
340
+ // is closed not for the vague archive HISTORY read as, which is how one lane
341
+ // came to look like two different things (#218).
342
+ return (run.endedAt ?? run.startedAt) >= snapshot.now - MERGED_HISTORY_MS ? "settled" : undefined;
336
343
  }
337
344
 
338
345
  /** Description lines for an issue the store has no run row for. Only the lanes
@@ -467,7 +474,7 @@ export function boardLabelsEqual(a: BoardLabels, b: BoardLabels): boolean {
467
474
  sameIssueSet(a.inProgress, b.inProgress) &&
468
475
  sameIssueSet(a.blocked, b.blocked) &&
469
476
  sameIssueSet(a.failed, b.failed) &&
470
- sameIssueSet(a.open, b.open) &&
477
+ sameIssueSet(a.closed, b.closed) &&
471
478
  (a.error === undefined) === (b.error === undefined)
472
479
  );
473
480
  }
@@ -551,7 +558,7 @@ function normalizeCursor(snapshot: BoardSnapshot, cursor: BoardCursor): void {
551
558
  cursor.transcriptOffset = Math.max(0, cursor.transcriptOffset);
552
559
  }
553
560
 
554
- function runCardLines(run: RunRecord, snapshot: BoardSnapshot): string[] {
561
+ function runCardLines(run: RunRecord, snapshot: BoardSnapshot, lane: BoardLane): string[] {
555
562
  const endedAt = run.endedAt ?? snapshot.now;
556
563
  const duration = humanDuration(endedAt - run.startedAt);
557
564
  const lines = [
@@ -564,30 +571,39 @@ function runCardLines(run: RunRecord, snapshot: BoardSnapshot): string[] {
564
571
  `attempt ${run.attempt} · ${run.turns}/${run.maxTurns}t`,
565
572
  `$${run.spendUsd.toFixed(2)} · ${duration}`,
566
573
  ];
567
- // #173: a live re-verification of a pushed row, when the 60s probe has one.
568
- // Guarded on the pushed states: a stale entry must not follow a row that
569
- // settled into another lane and keep claiming "now: green" about a PR that
570
- // already merged or failed.
571
- const live =
572
- run.state === "pushed-green" || run.state === "pushed-pending" ? snapshot.pr.get(run.id) : undefined;
573
- if (live !== undefined) {
574
- const reason = live.reason.split(/\s+/, 1)[0] ?? "";
575
- lines.push(
576
- live.status === "failed" ? `now: red ${reason === "" ? "failed" : reason}` : `now: ${prLabel(live.status)}`,
577
- );
578
- }
579
- // #173: a terminal blocked/failed run whose state label is gone reads as the
580
- // lane it sits in — which the label being absent made PARKED, not the truth.
581
- if (isLastRun(run, snapshot.labels)) lines.push(`last run: ${run.state}`);
582
- // Ordered by what needs a human first. An unsalvaged tree outranks even a
583
- // last error: the error describes a run that is over, the tree is work that
584
- // is still at risk and an issue that will not dispatch (#118).
574
+ // Exactly one fourth line, so a card stays within the four content lines that
575
+ // renderColumn's slot arithmetic (`slots = floor((height - 2) / 5)`) allocates
576
+ // for it. Every fact a card can carry folds into this single priority:
577
+ // an unsalvaged tree outranks even a live verdict (#118) — the error describes
578
+ // a run that is over, the tree is work still at risk — then the live probe for
579
+ // a pushed row, then the word the lane name cannot carry, then the run's own
580
+ // evidence.
581
+ //
582
+ // `isLastRun` is deliberately not a card line: it belongs in the detail header
583
+ // (renderDetail's `stateShown`), where a full width can spell out `last run:
584
+ // blocked`. Putting it here too would blow the budget and repeat the #218
585
+ // conflation on the two-line boundary.
585
586
  if (run.salvageError !== undefined && run.salvageAckAt === undefined) lines.push("UNSALVAGED WIP");
586
- else if (run.state === "pushed-pending") lines.push("checks pending");
587
- else if (run.salvageSha !== undefined) lines.push(`wip @ ${run.salvageSha.slice(0, 7)}`);
588
- else if (run.lastError !== undefined) lines.push(run.lastError.replace(/\s+/g, " "));
589
- else if (run.prUrl !== undefined) lines.push(run.prUrl.replace(/^https?:\/\//, ""));
590
- else lines.push(run.branch);
587
+ else {
588
+ // #173: a live re-verification of a pushed row, when the 60s probe has one.
589
+ // Guarded on the pushed states: a stale entry must not follow a row that
590
+ // settled into another lane and keep claiming "now: green" about a PR that
591
+ // already merged or failed.
592
+ const live =
593
+ run.state === "pushed-green" || run.state === "pushed-pending" ? snapshot.pr.get(run.id) : undefined;
594
+ if (live !== undefined) {
595
+ const reason = live.reason.split(/\s+/, 1)[0] ?? "";
596
+ lines.push(
597
+ live.status === "failed" ? `now: red — ${reason === "" ? "failed" : reason}` : `now: ${prLabel(live.status)}`,
598
+ );
599
+ } else if (lane === "settled") lines.push("issue closed");
600
+ else if (lane === "parked") lines.push("needs a label");
601
+ else if (run.state === "pushed-pending") lines.push("checks pending");
602
+ else if (run.salvageSha !== undefined) lines.push(`wip @ ${run.salvageSha.slice(0, 7)}`);
603
+ else if (run.lastError !== undefined) lines.push(run.lastError.replace(/\s+/g, " "));
604
+ else if (run.prUrl !== undefined) lines.push(run.prUrl.replace(/^https?:\/\//, ""));
605
+ else lines.push(run.branch);
606
+ }
591
607
  return lines;
592
608
  }
593
609
 
@@ -616,7 +632,7 @@ function renderColumn(
616
632
  lines.push(styledCell(" (empty)", width, DIM));
617
633
  } else {
618
634
  for (const [offset, card] of shown.entries()) {
619
- const raw = card.kind === "run" ? runCardLines(card.run, snapshot) : [`#${card.issue}`, card.reason, card.note];
635
+ const raw = card.kind === "run" ? runCardLines(card.run, snapshot, key) : [`#${card.issue}`, card.reason, card.note];
620
636
  while (raw.length < 4) raw.push("");
621
637
  const selected = selectedColumn && start + offset === selectedCard;
622
638
  for (const [lineIndex, value] of raw.entries()) {
@@ -834,6 +850,8 @@ function renderHelp(width: number, height: number): string[] {
834
850
  "? close help",
835
851
  "Esc back, then quit",
836
852
  "q / Ctrl-C back, then quit",
853
+ "",
854
+ "PARKED = issue open, no label — nothing dispatches it until you label it",
837
855
  ];
838
856
  const top = Math.max(0, Math.floor((height - lines.length) / 2));
839
857
  return [
@@ -916,7 +934,7 @@ async function probeBoardHealth(project: ProjectConfig): Promise<BoardHealth> {
916
934
  *
917
935
  * One open-issue snapshot per pass, never one list call per label (#203): the
918
936
  * four lifecycle sets are derived in memory from it by {@link boardLabelSets},
919
- * and PARKED openness is read off the same snapshot instead of a per-candidate
937
+ * and SETTLED closure is read off the same snapshot instead of a per-candidate
920
938
  * `issueState` fan-out. The single read rides the conditional-request cache, so
921
939
  * an idle board costs no primary rate-limit budget at all. The cadence stays
922
940
  * the ten-second health refresh, never the one-second redraw path.
@@ -947,11 +965,19 @@ export async function probeBoardLabels(
947
965
  const parkedCandidates = [...newestRun.values()].filter(
948
966
  (run) => PARKED_STATES.has(run.state) && !labelled.has(run.issue),
949
967
  );
950
- // The snapshot IS the set of open issues, so openness needs no per-issue
951
- // lookup: a candidate whose issue appears in it is open.
968
+ // The snapshot IS the set of open issues, so closure needs no per-issue
969
+ // lookup: a candidate whose issue does NOT appear in it is closed, *because
970
+ // this read succeeded*. That last clause is the whole distinction on the
971
+ // failed path — `closed` is only ever derived here, inside the try, so an
972
+ // unreadable probe survives as the previous set (or empty) and an issue the
973
+ // board could not confirm closed keeps parking rather than falsely reading
974
+ // as SETTLED (#218). Absence from `closed` is therefore open OR unknown,
975
+ // and both mean PARKED.
952
976
  const openNumbers = new Set(open.map((issue) => issue.number));
953
- const openSet = new Set(parkedCandidates.map((run) => run.issue).filter((n) => openNumbers.has(n)));
954
- return { queued, inProgress, blocked, failed, open: openSet, readAt: Date.now() };
977
+ const closed = new Set(
978
+ parkedCandidates.map((run) => run.issue).filter((n) => !openNumbers.has(n)),
979
+ );
980
+ return { queued, inProgress, blocked, failed, closed, readAt: Date.now() };
955
981
  } catch (err) {
956
982
  return {
957
983
  ...(previous ?? UNREAD_LABELS),
@@ -191,6 +191,34 @@ Keep the queue worth draining.
191
191
  every attempt.
192
192
  - An issue that has exhausted its attempts is not a retry candidate. Diagnose it,
193
193
  split it, or hand it back to a human.
194
+ - **Grooming is throughput-bound, so delegate the finding.** N workers drain the
195
+ queue in parallel while you refill it alone: on a three-worker fleet grooming
196
+ has to produce roughly three well-specced issues in the time one worker takes
197
+ to finish one. Auditing candidates serially cannot keep up, and the board then
198
+ reads "0 ready / workers idle" while you are doing exactly what this duty asks.
199
+ When the queue is below the grooming trigger, fan out read-only `scout`
200
+ subagents over backlog clusters **in one batch** rather than auditing one issue
201
+ at a time. Scouts do the finding; you still do the deciding and you still write
202
+ the brief. The quality bar above does not move.
203
+ - **Give every scout the same return contract**, or it comes back with prose
204
+ nobody can act on:
205
+ - verdict — `ALREADY DONE` / `PROMOTABLE` / `NEEDS DECOMPOSITION` / `BLOCKED` /
206
+ `NEEDS PRODUCT DECISION`
207
+ - routing — exactly one repo, or `MULTI` with the split
208
+ - evidence — the file or symbol proving `ALREADY DONE`, never a title match
209
+ - entry points — the 3-6 files to change or read first
210
+ - existing tests covering the behaviour, by path
211
+ - the one thing most likely to be silently faked
212
+ - **Disqualifying an issue is a successful grooming outcome.** Measured on this
213
+ package's own fleet: four scouts over sixteen backlog issues promoted four and
214
+ *disqualified six* that looked promotable from their titles — four written
215
+ against a tenancy model a later epic retired, two blocked behind open
216
+ prerequisites named in their own bodies. Each would have burned a worker's whole
217
+ attempt to discover. On anything old, check first whether a later epic has
218
+ invalidated the issue's premise; that check is mechanical, read-only, and
219
+ exactly what a scout is cheap at and you are expensive at.
220
+ - Examine widely, promote narrowly. Sixteen examined and four promoted is the
221
+ shape to aim for — the cap is on what you **promote**, never on what you look at.
194
222
 
195
223
  ## Duty 3 — report
196
224
 
package/src/daemon.ts CHANGED
@@ -35,7 +35,7 @@ import { recordReleaseBlock } from "./release-policy.ts";
35
35
  import { branchName, effectiveLabels, route } from "./routing.ts";
36
36
  import type { Routed, UnroutableReason } from "./routing.ts";
37
37
  import { evaluateDecisionConditions, probeNpmVersion, probeRateLimitReset } from "./decisions.ts";
38
- import { classifyRun, type ClassifyFacts } from "./failure-class.ts";
38
+ import { classifyRun, providerCreditRefusal, type ClassifyFacts } from "./failure-class.ts";
39
39
  import { projectLabels } from "./label-projection.ts";
40
40
  import { dbPath, openStore, utcDay } from "./store.ts";
41
41
  import { makeTracker, type RateLimitStatus } from "./tracker/github.ts";
@@ -1108,6 +1108,16 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
1108
1108
  : { state: result.state };
1109
1109
  const state = verified.state;
1110
1110
 
1111
+ // Read before the row is written so `lastError` carries the provider's own
1112
+ // message — it names the fix and the URL, which no classification can.
1113
+ const providerCredit =
1114
+ state === "failed" || state === "killed"
1115
+ ? (() => {
1116
+ const err = readSessionError(result.sessionFile);
1117
+ return err === undefined ? undefined : providerCreditRefusal(err);
1118
+ })()
1119
+ : undefined;
1120
+
1111
1121
  // The other half of not believing a worker about its own run (#128). The
1112
1122
  // claim being audited is `state: pushed-green`, so the audit runs on the
1113
1123
  // worker's claim rather than on what verification made of it: a claim that
@@ -1192,7 +1202,11 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
1192
1202
  // push: the report of a killed attempt is exactly the one a later
1193
1203
  // continuation must pool its disclosures from (#199).
1194
1204
  report: result.report,
1195
- ...(verified.reason === undefined ? {} : { lastError: verified.reason }),
1205
+ ...(providerCredit !== undefined
1206
+ ? { lastError: providerCredit }
1207
+ : verified.reason === undefined
1208
+ ? {}
1209
+ : { lastError: verified.reason }),
1196
1210
  ...settlement?.patch,
1197
1211
  ...(audit === undefined || audit.flags.length === 0
1198
1212
  ? {}
@@ -1224,7 +1238,32 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
1224
1238
  maxContinuations: caps.maxContinuationsPerIssue,
1225
1239
  });
1226
1240
 
1227
- if (continueTurns) {
1241
+ if (providerCredit !== undefined) {
1242
+ // Pause here rather than at classification: the sweep runs on the tick,
1243
+ // and three issues each burned an attempt in the fifteen minutes between
1244
+ // the first 402 and a human noticing (#220).
1245
+ setPaused(true, { source: "provider-credit", reason: providerCredit });
1246
+ log(`#${issue} provider refused for credit — dispatch paused: ${providerCredit}`);
1247
+ swapLabel(store, project.name, issue, inProgress, project.stateLabels.failed);
1248
+ // Fleet-scoped and run-independent on purpose. The notification ledger
1249
+ // dedupes on `project:issue:tier:summary`, so `NO_ISSUE` plus a summary
1250
+ // carrying no run or attempt is what makes this page once for the fleet
1251
+ // instead of once per affected run.
1252
+ await safeEscalate(d, {
1253
+ tier: 2,
1254
+ project: project.name,
1255
+ issue: NO_ISSUE,
1256
+ summary: `Model provider refused for credit — ${project.name} is paused`,
1257
+ detail: [
1258
+ providerCredit,
1259
+ "",
1260
+ "No implementation attempt was charged: this is a billing state, not a",
1261
+ "failed implementation. Each affected issue keeps its queue label and",
1262
+ "re-dispatches on `omp-conductor resume` once the provider has credit.",
1263
+ `Session: ${result.sessionFile ?? "(no transcript)"}`,
1264
+ ].join("\n"),
1265
+ });
1266
+ } else if (continueTurns) {
1228
1267
  // Requeue as one ordered pair: the in-progress removal before the
1229
1268
  // queue add, exactly the order the projector will apply them in (#201).
1230
1269
  store.enqueueLabelOps(project.name, [
@@ -2505,6 +2544,12 @@ export interface StatusSnapshot {
2505
2544
  configPath: string;
2506
2545
  stateDir: string;
2507
2546
  paused: boolean;
2547
+ /**
2548
+ * Why the fleet is paused, when whatever paused it said. Rendered beside
2549
+ * `(PAUSED)` so a pause an operator did not issue names itself instead of
2550
+ * reading like a mistake (#220).
2551
+ */
2552
+ pauseReason?: string;
2508
2553
  caps: Caps;
2509
2554
  /**
2510
2555
  * The effective per-shape release grants. On the snapshot rather than re-read
@@ -2579,11 +2624,15 @@ export function statusSnapshotFromStore(
2579
2624
  const dispatch = store.latestDispatch(p.name);
2580
2625
  const labelOpsPending = store.countPendingLabelOps(p.name);
2581
2626
  const oldestLabelOpAt = store.oldestPendingLabelOpAt(p.name);
2627
+ // Read once: the provenance read touches the filesystem, and the renderer
2628
+ // should never pay for it twice per status.
2629
+ const reason = pauseProvenance()?.reason;
2582
2630
  return {
2583
2631
  project: p.name,
2584
2632
  configPath: configPath(),
2585
2633
  stateDir: stateDir(),
2586
2634
  paused: isPaused(),
2635
+ ...(reason === undefined ? {} : { pauseReason: reason }),
2587
2636
  caps,
2588
2637
  releaseGrants: resolveReleaseGrants(p),
2589
2638
  activeRuns: store.activeRuns(p.name),
@@ -2857,6 +2906,57 @@ export function lastToolCalls(sessionFile: string | undefined, limit = SPIN_EVID
2857
2906
  return names.slice(-limit);
2858
2907
  }
2859
2908
 
2909
+ /** A provider refusal a session recorded before dying, or undefined for none. */
2910
+ export interface SessionError {
2911
+ status?: number;
2912
+ message: string;
2913
+ }
2914
+
2915
+ /**
2916
+ * The last error a transcript recorded, or undefined when it recorded none.
2917
+ *
2918
+ * The harness writes `{"stopReason":"error","errorStatus":402,"errorId":402,
2919
+ * "errorMessage":"402 This request requires more credits, ..."}`. The daemon
2920
+ * read none of it, so three runs died `unknown` with an empty `lastError` and
2921
+ * charged an attempt each for a billing state (#220).
2922
+ *
2923
+ * Scanned newest-first: a session that recovered from an early error and then
2924
+ * died of something else must report the something else, and a session that
2925
+ * recovered from its only error and finished cleanly reports the error anyway
2926
+ * because there is no terminal verdict to outrank it (#220).
2927
+ */
2928
+ export function readSessionError(sessionFile: string | undefined): SessionError | undefined {
2929
+ if (sessionFile === undefined) return undefined;
2930
+ let text: string;
2931
+ try {
2932
+ text = readFileSync(sessionFile, "utf8");
2933
+ } catch {
2934
+ return undefined;
2935
+ }
2936
+ const lines = text.split("\n");
2937
+ for (let i = lines.length - 1; i >= 0; i -= 1) {
2938
+ const line = lines[i];
2939
+ if (line === undefined || line.length === 0) continue;
2940
+ let row: unknown;
2941
+ try {
2942
+ row = JSON.parse(line) as unknown;
2943
+ } catch {
2944
+ continue;
2945
+ }
2946
+ if (row === null || typeof row !== "object") continue;
2947
+ const rec = row as { readonly [key: string]: unknown };
2948
+ if (rec["stopReason"] !== "error") continue;
2949
+ const message = rec["errorMessage"];
2950
+ if (typeof message !== "string" || message.trim() === "") continue;
2951
+ const status = rec["errorStatus"];
2952
+ return {
2953
+ ...(typeof status === "number" && Number.isFinite(status) ? { status } : {}),
2954
+ message: message.trim(),
2955
+ };
2956
+ }
2957
+ return undefined;
2958
+ }
2959
+
2860
2960
  /**
2861
2961
  * Classify every unclassified terminal run, persist the verdict, and perform the
2862
2962
  * one recovery its class names (#132).
@@ -113,6 +113,37 @@ export function startFailure(lastError: string | undefined): string | undefined
113
113
  return hit === undefined ? undefined : lastError.split("\n")[0]?.trim();
114
114
  }
115
115
 
116
+ /**
117
+ * Provider messages that mean "this key is out of money", matched on the text
118
+ * the provider itself prints.
119
+ */
120
+ const CREDIT_REFUSAL_SIGNATURES = [
121
+ "requires more credits",
122
+ "insufficient credits",
123
+ "insufficient_quota",
124
+ "exceeded your current quota",
125
+ "credit balance is too low",
126
+ ] as const;
127
+
128
+ /**
129
+ * Evidence that a model provider refused this run for credit, or `undefined`
130
+ * when it did not.
131
+ *
132
+ * Two independent signals, either sufficient: HTTP 402, which means exactly
133
+ * this and nothing else, and the refusal text providers print when a key is out
134
+ * of allowance without using 402. Deliberately narrow — a generic 429 is rate
135
+ * limiting, which is a different failure with a different remedy.
136
+ */
137
+ export function providerCreditRefusal(error: {
138
+ status?: number;
139
+ message: string;
140
+ }): string | undefined {
141
+ const text = error.message.toLowerCase();
142
+ const named = CREDIT_REFUSAL_SIGNATURES.some((signature) => text.includes(signature));
143
+ if (error.status !== 402 && !named) return undefined;
144
+ return error.message.split("\n")[0]?.trim() ?? error.message;
145
+ }
146
+
116
147
  /**
117
148
  * Evidence that this run never started, or `undefined` when something did happen.
118
149
  *
@@ -201,6 +232,22 @@ export function classifyRun(run: RunRecord, facts: ClassifyFacts): Classificatio
201
232
  };
202
233
  }
203
234
 
235
+ // A billing state, not an implementation failure. Its own class because it is
236
+ // the one outage an operator fixes with a card rather than a diagnosis, and
237
+ // because burying a self-describing provider error in `unknown` erodes what
238
+ // `unknown` means (#220). Ahead of `neverStarted` on purpose: a 402 on the
239
+ // first request produces a turn-0 row with no artifacts, which
240
+ // `env-start-failure` would otherwise absorb and lose the cause.
241
+ if (run.state === "failed" || run.state === "killed") {
242
+ const credit =
243
+ run.lastError === undefined
244
+ ? undefined
245
+ : providerCreditRefusal({ message: run.lastError });
246
+ if (credit !== undefined) {
247
+ return { cls: "provider-credit", recovery: "requeue", evidence: credit };
248
+ }
249
+ }
250
+
204
251
  // A run that never started is the most classifiable failure there is, and the
205
252
  // least deserving of an implementation attempt: the session did not get as far
206
253
  // as reading the issue. See {@link neverStarted} for the two shapes and why the
package/src/fleet.ts CHANGED
@@ -1062,6 +1062,7 @@ export function formatFleetStatus(
1062
1062
  function formatProjectBody(s: StatusSnapshot): string {
1063
1063
  const lines = [
1064
1064
  `project ${s.project}${s.paused ? " (PAUSED)" : ""}`,
1065
+ ...(s.pauseReason === undefined ? [] : [`paused ${s.pauseReason}`]),
1065
1066
  `config ${s.configPath}`,
1066
1067
  `state ${s.stateDir}`,
1067
1068
  "",
package/src/store.ts CHANGED
@@ -902,12 +902,12 @@ export function openStore(dbPath: string): Store {
902
902
  const countFailures = db.query<{ n: number }, [string, number]>(
903
903
  `SELECT COUNT(*) AS n FROM runs
904
904
  WHERE project = ? AND issue = ? AND state = 'failed'
905
- AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck', 'env-start-failure', 'dispatch-infra'))`,
905
+ AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck', 'env-start-failure', 'dispatch-infra', 'provider-credit'))`,
906
906
  );
907
907
  const countContinuations = db.query<{ n: number }, [string, number]>(
908
908
  `SELECT COUNT(*) AS n FROM runs
909
909
  WHERE project = ? AND issue = ? AND state IN ('killed', 'orphaned', 'blocked')
910
- AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck', 'env-start-failure', 'dispatch-infra'))`,
910
+ AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck', 'env-start-failure', 'dispatch-infra', 'provider-credit'))`,
911
911
  );
912
912
  // How many times one issue reached a given class. Recovery uses it to bound a
913
913
  // retry loop whose cause is persistent (e.g. a mirror that will not refresh):
package/src/types.ts CHANGED
@@ -831,6 +831,7 @@ export const FAILURE_CLASSES = [
831
831
  "orphan-clean",
832
832
  "orphan-dirty",
833
833
  "settlement-stuck",
834
+ "provider-credit",
834
835
  "unknown",
835
836
  ] as const;
836
837
 
package/src/unblock.ts CHANGED
@@ -73,11 +73,11 @@ export interface UnblockOutcome {
73
73
  requeueSkipped?: true;
74
74
  /** Set when the tracker refused one or more of this verb's label ops, so
75
75
  * the label sync is owed rather than done: the intended label state is
76
- * durable in the store and the daemon retries it (#201). Safety is
77
- * preserved either way, but the issue is only claimable once the queue
78
- * label itself has landed `listReady` reads the tracker by label, so a
79
- * queue-label add that has not been applied is invisible to dispatch. The
80
- * number is what is still pending. */
76
+ * durable in the store and the daemon retries it (#201). This says nothing
77
+ * about claimability `listReady` reads the tracker by label, so an issue
78
+ * is dispatchable only once the queue label is physically present, which a
79
+ * pending add has not delivered and `--no-requeue` never enqueues at all.
80
+ * The number is what is still pending. */
81
81
  labelSyncQueued?: number;
82
82
  }
83
83
 
@@ -268,7 +268,7 @@ export function formatUnblock(
268
268
  const labelsPending = o.labelSyncQueued !== undefined;
269
269
  if (o.labelSyncQueued !== undefined) {
270
270
  lines.push(
271
- ` label sync queued (${o.labelSyncQueued} pending) — the daemon retries; the intended labels are durable, and the issue is claimable once they land`,
271
+ ` label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`,
272
272
  );
273
273
  }
274
274
  if (o.forced === true) {
@@ -321,7 +321,7 @@ export function formatUnblock(
321
321
  lines.push(
322
322
  latest.prUrl === undefined
323
323
  ? labelsPending
324
- ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries; eligibility is recorded in the store, not yet on the tracker`
324
+ ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`
325
325
  : o.requeued === true
326
326
  ? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` +
327
327
  "the dispatcher still applies its open-PR check at claim time)"
@@ -352,7 +352,7 @@ export function formatUnblock(
352
352
  );
353
353
  if (labelsPending) {
354
354
  lines.push(
355
- ` queue label sync queued (${o.labelSyncQueued} pending) — the daemon retries; the intended labels are durable, and the issue is claimable once they land`,
355
+ ` queue label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`,
356
356
  );
357
357
  } else if (o.requeued === true) {
358
358
  lines.push(` queue "${project.queueLabel}" restored (no-op if it was already present)`);
@@ -361,7 +361,7 @@ export function formatUnblock(
361
361
  } else {
362
362
  lines.push(
363
363
  labelsPending
364
- ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries; the intended labels are durable, and the issue is claimable once they land`
364
+ ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`
365
365
  : o.requeued === true
366
366
  ? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` +
367
367
  "the dispatcher still applies its open-PR check at claim time)"
package/src/worker.ts CHANGED
@@ -19,6 +19,9 @@ const HEAD_SHA_PATTERN = /^head:\s*([0-9a-f]{40})\s*$/im;
19
19
  const PUSHED_GREEN_PATTERN = /^state:\s*pushed-green\s*$/im;
20
20
  const BLOCKED_PATTERN = /^state:\s*blocked\s*$/im;
21
21
 
22
+ /** Any explicit verdict line, whatever it claims. */
23
+ const STATE_LINE_PATTERN = /^state:\s*\S+\s*$/im;
24
+
22
25
  /** `{{KEY}}` placeholders in a brief template. */
23
26
  const PLACEHOLDER_PATTERN = /\{\{([A-Za-z0-9_]+)\}\}/g;
24
27
 
@@ -160,6 +163,19 @@ export function deriveResult(report: string): {
160
163
  };
161
164
  }
162
165
 
166
+ /**
167
+ * Did this report state a verdict at all?
168
+ *
169
+ * The discriminator between "the worker said something about its outcome" and
170
+ * "the worker said something else entirely". A finished worker woken by its own
171
+ * stale polling timers emits neither `state: pushed-green` nor `state: blocked`
172
+ * — it emits prose — and that is the only case where an earlier verdict is
173
+ * still the run's real answer.
174
+ */
175
+ export function hasVerdictLine(report: string): boolean {
176
+ return STATE_LINE_PATTERN.test(report);
177
+ }
178
+
163
179
  /**
164
180
  * Is this `agent_end` the end of the run?
165
181
  *
@@ -224,6 +240,14 @@ export async function runWorker(
224
240
  let turns = 0;
225
241
  let spendUsd = 0;
226
242
  let report = "";
243
+ // The newest COMPLETE `pushed-green` verdict this session emitted. Tracked
244
+ // apart from `report` because `report` is deliberately the newest non-empty
245
+ // text — a run cut off mid-sentence must still report what it said last —
246
+ // and a worker that finishes, then wakes on its own stale polling timers,
247
+ // overwrites its own verdict with "nothing to resume" chatter. veltro#406
248
+ // burned 180/180 turns that way and charged an attempt against a PR that
249
+ // merged (#217).
250
+ let claim: { prUrl: string; headSha: string } | undefined;
227
251
  let killedBy: KilledBy | undefined;
228
252
  // Bun's global timer handle; cleared on every exit path below.
229
253
  let timer: Timer | undefined;
@@ -265,7 +289,13 @@ export async function runWorker(
265
289
  // Keep the newest non-empty assistant text: whatever the worker said last
266
290
  // is its report, whether it finished cleanly or was cut off.
267
291
  const text = reportText(field(message, "content"));
268
- if (text !== "") report = text;
292
+ if (text !== "") {
293
+ report = text;
294
+ const stated = deriveResult(text);
295
+ if (stated.state === "pushed-green" && stated.prUrl !== undefined && stated.headSha !== undefined) {
296
+ claim = { prUrl: stated.prUrl, headSha: stated.headSha };
297
+ }
298
+ }
269
299
 
270
300
  // Real cost lives on assistant messages as `usage.cost.total` (live hermes
271
301
  // transcripts, 2026-08-07). The earlier agent_end.telemetry path never
@@ -337,9 +367,26 @@ export async function runWorker(
337
367
  }
338
368
 
339
369
  if (killedBy !== undefined) {
340
- return withSessionFacts({ state: "killed", turns, spendUsd, report, killedBy });
370
+ // The PR and head are facts the session already established, so they
371
+ // survive the kill. Without them `shouldContinueAfterTurnsCap` sees no
372
+ // artifacts and charges an implementation attempt for a cap kill that had
373
+ // real work to continue from.
374
+ return withSessionFacts({
375
+ state: "killed",
376
+ turns,
377
+ spendUsd,
378
+ report,
379
+ killedBy,
380
+ ...(claim === undefined ? {} : { prUrl: claim.prUrl, headSha: claim.headSha }),
381
+ });
341
382
  }
342
383
 
384
+ // An explicit later verdict always wins: a worker that pushed green and then
385
+ // stopped to ask a question means the question. The earlier claim is only
386
+ // restored when the last thing said was not a verdict at all.
387
+ if (claim !== undefined && !hasVerdictLine(report)) {
388
+ return withSessionFacts({ state: "pushed-green", ...claim, turns, spendUsd, report });
389
+ }
343
390
  return withSessionFacts({
344
391
  ...deriveResult(report),
345
392
  turns,