omp-conductor 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -10
- package/package.json +1 -1
- package/src/board.ts +79 -53
- package/src/briefs/orchestrator.md +28 -0
- package/src/daemon.ts +103 -3
- package/src/failure-class.ts +47 -0
- package/src/fleet.ts +1 -0
- package/src/store.ts +2 -2
- package/src/types.ts +1 -0
- package/src/unblock.ts +9 -9
- package/src/worker.ts +49 -2
package/README.md
CHANGED
|
@@ -538,7 +538,8 @@ add waits with it.
|
|
|
538
538
|
|
|
539
539
|
`omp-conductor board [--project NAME]` opens the same facts as a live terminal
|
|
540
540
|
kanban instead of a scrolling wall of status text. Its columns are Queue,
|
|
541
|
-
Claimed, Running, Green, Blocked, Failed, Orphaned, Merged, and
|
|
541
|
+
Claimed, Running, Green, Blocked, Failed, Orphaned, Merged, Settled, and
|
|
542
|
+
Parked.
|
|
542
543
|
|
|
543
544
|
Every column is a claim about an issue's state **now**, and each issue appears
|
|
544
545
|
in exactly one of them. Claimed, Running, and Green come from the store, because
|
|
@@ -549,10 +550,17 @@ label, Blocked is the blocked state label, and Orphaned is an issue the
|
|
|
549
550
|
in-progress label still claims with no worker running it — a protected orphan
|
|
550
551
|
whose worktree is kept because it may hold uncommitted work. So a requeued issue
|
|
551
552
|
is queue work even when its last attempt failed, and a closed issue is not
|
|
552
|
-
active work at all. Merged and
|
|
553
|
-
both bounded to the last 24 hours and neither one an action list:
|
|
553
|
+
active work at all. Merged and Settled are the two backward-looking columns,
|
|
554
|
+
both bounded to the last 24 hours and neither one an action list: Settled holds
|
|
554
555
|
recently finished runs that no current label claims, so the board stays
|
|
555
|
-
operational rather than becoming an analytics archive.
|
|
556
|
+
operational rather than becoming an analytics archive. Parked is the issue's
|
|
557
|
+
last word when the tracker has **not** confirmed it closed — a terminal run
|
|
558
|
+
whose issue wears no state or queue label (still open, or a label read that
|
|
559
|
+
failed), so nothing dispatches it until a human labels it. Unlike Merged and
|
|
560
|
+
Settled it is not bounded to 24 hours, because parked work does not stop being
|
|
561
|
+
the issue's only remaining record; Settled alone asserts the issue is closed,
|
|
562
|
+
so an unreadable probe parks rather than letting a run masquerade as finished
|
|
563
|
+
work.
|
|
556
564
|
|
|
557
565
|
The board refreshes run, spend, turn, and dispatch values from SQLite every
|
|
558
566
|
second. It refreshes the slower daemon, Herdr, Telegram, code-graph, and tracker
|
|
@@ -1458,6 +1466,7 @@ persists the class on the row, and performs the one recovery that class names.
|
|
|
1458
1466
|
| `admin-kill` | killed *below* its own ceiling — a restart or a drain | requeue | none |
|
|
1459
1467
|
| `ci-infra` | PR open, every unresolved check cancelled / timed out / stale | re-run the failed jobs | none |
|
|
1460
1468
|
| `ci-deterministic` | PR open, a check genuinely reports `FAILURE` | escalate with the failing check names and links | failed attempt |
|
|
1469
|
+
| `provider-credit` | the provider refused the run for credit (HTTP 402, or its own out-of-credit text read off the transcript) | pause the fleet and require `omp-conductor resume` once the provider has credit | none |
|
|
1461
1470
|
| `unknown` | anything unrecognised | escalate | as recorded |
|
|
1462
1471
|
|
|
1463
1472
|
**Unknown escalates; it never silently retries.** A shape this table does not
|
|
@@ -1466,11 +1475,14 @@ cause nobody has named — the behaviour this exists to end.
|
|
|
1466
1475
|
|
|
1467
1476
|
### The budgets follow the cause
|
|
1468
1477
|
|
|
1469
|
-
`failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck
|
|
1470
|
-
|
|
1471
|
-
`settlement-stuck
|
|
1472
|
-
the session never started, so nothing about
|
|
1473
|
-
|
|
1478
|
+
`failuresFor` (implementation attempts) excludes `ci-infra`, `settlement-stuck`,
|
|
1479
|
+
`env-start-failure` and `provider-credit`; `continuationsFor` excludes
|
|
1480
|
+
`admin-kill`, `settlement-stuck`, `env-start-failure` and `provider-credit`. An
|
|
1481
|
+
environment fault charges neither: the session never started, so nothing about
|
|
1482
|
+
the issue was attempted, and nor does a provider-credit refusal, which is a
|
|
1483
|
+
billing state rather than an implementation failure. A merge conflict *is*
|
|
1484
|
+
charged as a continuation, because a rebase is real work — just never as a
|
|
1485
|
+
failed implementation attempt.
|
|
1474
1486
|
|
|
1475
1487
|
An **unclassified** row (every row written before 0.4.3) counts exactly as it
|
|
1476
1488
|
did before classification existed. Upgrading therefore changes no existing
|
|
@@ -1971,7 +1983,7 @@ omp-conductor help
|
|
|
1971
1983
|
| `upgrade [--to VERSION] [--project NAME]` | Deterministically update the Bun-global CLI, omp plugin, Herdr recovery plugin, and managed brief as one release. Resolves the npm version and exact `gitHead`, pauses only new claims, drains active workers, installs all surfaces, reloads Herdr and the daemon, waits for pane recovery, verifies identities and fleet health twice, then restores the original dispatch state. A no-op when already current. Failure leaves dispatch paused. Must run outside a Herdr-managed session. |
|
|
1972
1984
|
| `status [--project NAME]` | Layered fleet report first: `dispatch` / `ticks` / next scheduled tick / `pane` / `recovery` / `herdr` / `telegram` / `brief` / `decisions` / optional `failure classes` and `code graph` / `daemon`, then the project body. The project body includes the latest completed dispatch timestamp, ready/routed/admitted counts, bounded hold groups, and the GitHub API budget (`graphql` / `core` remaining and reset, in the caps block); API failures are marked `DEGRADED` so queue starvation cannot look idle. The next tick comes from the live heartbeat process, not a guess from log timestamps. Telegram health uses `getMe` to prove API authentication without sending a message and reports inbound bridge configuration separately. Configured graphs report prerequisites, indexed repos, timer state, and refresh freshness without blocking dispatch. A `reports` block lists everything the outbox has not delivered, with its age, and prints `pending` (nobody has it) differently from `SENDING` (outcome unknown, it may already have arrived) — see [Report delivery](#report-delivery-the-outbox). The daemon block includes `rss` from `/healthz`; live workers add a busy-deploy warning. A `.conductor-stalled` marker adds an `orchestrator STALLED since …` line. |
|
|
1973
1985
|
| `ledger [--issue N] [--limit N]` | Every [mediated-verb](#the-mediated-verbs-126) call and how the daemon decided it: the verb, the arguments it was given, allow or refuse, the named refusal reason, and any resulting SHA. Sessions cannot push, open a PR, merge, label or release except through those verbs, so this is the record of what they *tried* as well as what they did — the thing an escalation cites. Reads (`conductor_pr_status`) are deliberately absent: a poll every thirty seconds would bury the refusals. `--issue` narrows to one issue's run; `--limit` defaults to 50. The newest few also appear in `status`. |
|
|
1974
|
-
| `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned,
|
|
1986
|
+
| `board [--project NAME]` | Live keyboard-driven kanban over the same SQLite and `/healthz` truth as `status`, plus the tracker's current labels: Queue, Claimed, Running, Green, Blocked, Failed, Orphaned, the last 24 hours of Merged and Settled, and Parked (an issue the tracker has not confirmed closed — still open, or a label read that failed — so nothing dispatches it until a human labels it). Columns are mutually exclusive and describe current state, not the newest run row, so a requeued issue is queued rather than failed and a closed issue is neither. Refreshes run/spend/turn values every second, and health plus the label read every ten seconds. `Enter` follows the selected transcript in place; `u` invokes the existing unblock workflow on a Blocked, Failed, or Orphaned card; `i` / `p` open the issue / PR; `r` refreshes health; `?` shows all keys. Requires an interactive terminal of at least 50×20. |
|
|
1975
1987
|
| `hold [--project NAME]` | Soft stop: pause claiming **and** disarm ticks. Daemon and pane stay up. Prefer this over `pause` when the intent is "stop the conductor" without killing processes. See [Stop the conductor](#stop-the-conductor-hold--halt). |
|
|
1976
1988
|
| `halt [--pane] [--project NAME]` | `hold`, then stop the dispatch daemon (systemctl-aware). Pane stays up unless `--pane` is passed. `halt --pane` also pins herdr-conductor recovery off for the conductor agent only — it does **not** stop `herdr-fleet.service` or any other herdr session. Fail-closed: exits nonzero unless the agent is proven gone. |
|
|
1977
1989
|
| `arm [--project NAME]` | Proof-gated: send a Telegram challenge and write the arm marker only after your reply appears as a user turn in the orchestrator transcript. Never auto-armed by `resume` / `hold`. |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "omp-conductor",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"description": "A 24/7 dispatcher that takes ready GitHub issues to green, mergeable PRs using omp coding sessions, with tiered escalation first to an orchestrator session and then to a human.",
|
package/src/board.ts
CHANGED
|
@@ -72,9 +72,9 @@ const COLUMN_DEFS = [
|
|
|
72
72
|
{ key: "blocked", title: "BLOCKED" },
|
|
73
73
|
{ key: "failed", title: "FAILED" },
|
|
74
74
|
{ key: "orphaned", title: "ORPHANED" },
|
|
75
|
-
{ key: "parked", title: "PARKED" },
|
|
76
75
|
{ key: "merged", title: "MERGED" },
|
|
77
|
-
{ key: "
|
|
76
|
+
{ key: "settled", title: "SETTLED" },
|
|
77
|
+
{ key: "parked", title: "PARKED" },
|
|
78
78
|
] as const satisfies readonly { key: string; title: string }[];
|
|
79
79
|
|
|
80
80
|
export type BoardLane = (typeof COLUMN_DEFS)[number]["key"];
|
|
@@ -151,11 +151,15 @@ export interface BoardLabels {
|
|
|
151
151
|
inProgress: ReadonlySet<number>;
|
|
152
152
|
blocked: ReadonlySet<number>;
|
|
153
153
|
failed: ReadonlySet<number>;
|
|
154
|
-
/**
|
|
155
|
-
*
|
|
156
|
-
*
|
|
157
|
-
*
|
|
158
|
-
|
|
154
|
+
/**
|
|
155
|
+
* Issues the tracker reports CLOSED, derived only from a *successful* label
|
|
156
|
+
* snapshot on the same 10s cadence as the other sets. This is the one fact
|
|
157
|
+
* that separates SETTLED (finished work, issue closed) from PARKED (a human
|
|
158
|
+
* still has to look): absence from this set means open **or** the probe could
|
|
159
|
+
* not tell, and either way the run parks rather than falsely asserting
|
|
160
|
+
* closure. On a failed read the last good set survives behind {@link error}.
|
|
161
|
+
*/
|
|
162
|
+
closed: ReadonlySet<number>;
|
|
159
163
|
/** Epoch ms of the last read that succeeded; 0 when none has. */
|
|
160
164
|
readAt: number;
|
|
161
165
|
/** Message from the most recent failed read. The sets above are then the last
|
|
@@ -171,7 +175,7 @@ const UNREAD_LABELS: BoardLabels = {
|
|
|
171
175
|
inProgress: new Set<number>(),
|
|
172
176
|
blocked: new Set<number>(),
|
|
173
177
|
failed: new Set<number>(),
|
|
174
|
-
|
|
178
|
+
closed: new Set<number>(),
|
|
175
179
|
readAt: 0,
|
|
176
180
|
};
|
|
177
181
|
|
|
@@ -225,7 +229,7 @@ function ansiColor(key: BoardLane): string {
|
|
|
225
229
|
return YELLOW;
|
|
226
230
|
case "failed":
|
|
227
231
|
return RED;
|
|
228
|
-
case "
|
|
232
|
+
case "settled":
|
|
229
233
|
return DIM;
|
|
230
234
|
default:
|
|
231
235
|
return MAGENTA;
|
|
@@ -285,10 +289,11 @@ function humanDuration(ms: number): string {
|
|
|
285
289
|
* parent issues a human had already cleared (#81, #86) — every one of them a
|
|
286
290
|
* historical terminal run, not one of them failed work.
|
|
287
291
|
*
|
|
288
|
-
*
|
|
292
|
+
* so: the store decides the live lanes, because a running worker is the most
|
|
289
293
|
* current fact there is; the tracker's current labels decide every stopped
|
|
290
294
|
* lane; each issue matches exactly once, first branch wins; and last-run
|
|
291
|
-
* history is answered in
|
|
295
|
+
* history is answered in SETTLED (issue closed) and PARKED (issue still open,
|
|
296
|
+
* labels cleared), under their own names.
|
|
292
297
|
*/
|
|
293
298
|
function laneOf(
|
|
294
299
|
snapshot: BoardSnapshot,
|
|
@@ -322,17 +327,19 @@ function laneOf(
|
|
|
322
327
|
if (queued.has(issue)) return "queue";
|
|
323
328
|
if (run?.state === "merged") return "merged";
|
|
324
329
|
if (run === undefined) return undefined;
|
|
325
|
-
//
|
|
326
|
-
//
|
|
327
|
-
//
|
|
328
|
-
//
|
|
329
|
-
//
|
|
330
|
-
// being the issue's last word after 24 hours.
|
|
331
|
-
if (PARKED_STATES.has(run.state) && labels.
|
|
332
|
-
// What is left is a
|
|
333
|
-
//
|
|
334
|
-
//
|
|
335
|
-
|
|
330
|
+
// SETTLED vs PARKED, decided by the one fact the board can prove: whether the
|
|
331
|
+
// tracker confirmed the issue closed. Absence from `closed` means open OR the
|
|
332
|
+
// probe could not tell, and either way the run parks — a failed label read
|
|
333
|
+
// must render the issue as "a human has to look", never as finished work
|
|
334
|
+
// (#218). PARKED is deliberately not subject to MERGED_HISTORY_MS: parked work
|
|
335
|
+
// does not stop being the issue's last word after 24 hours.
|
|
336
|
+
if (PARKED_STATES.has(run.state) && !labels.closed.has(issue)) return "parked";
|
|
337
|
+
// What is left is a confirmed-closed issue with a terminal run row. Recent
|
|
338
|
+
// rows stay readable under SETTLED for the same 24 hours MERGED uses; older
|
|
339
|
+
// ones are not news. Named for what it means — the run settled and the issue
|
|
340
|
+
// is closed — not for the vague archive HISTORY read as, which is how one lane
|
|
341
|
+
// came to look like two different things (#218).
|
|
342
|
+
return (run.endedAt ?? run.startedAt) >= snapshot.now - MERGED_HISTORY_MS ? "settled" : undefined;
|
|
336
343
|
}
|
|
337
344
|
|
|
338
345
|
/** Description lines for an issue the store has no run row for. Only the lanes
|
|
@@ -467,7 +474,7 @@ export function boardLabelsEqual(a: BoardLabels, b: BoardLabels): boolean {
|
|
|
467
474
|
sameIssueSet(a.inProgress, b.inProgress) &&
|
|
468
475
|
sameIssueSet(a.blocked, b.blocked) &&
|
|
469
476
|
sameIssueSet(a.failed, b.failed) &&
|
|
470
|
-
sameIssueSet(a.
|
|
477
|
+
sameIssueSet(a.closed, b.closed) &&
|
|
471
478
|
(a.error === undefined) === (b.error === undefined)
|
|
472
479
|
);
|
|
473
480
|
}
|
|
@@ -551,7 +558,7 @@ function normalizeCursor(snapshot: BoardSnapshot, cursor: BoardCursor): void {
|
|
|
551
558
|
cursor.transcriptOffset = Math.max(0, cursor.transcriptOffset);
|
|
552
559
|
}
|
|
553
560
|
|
|
554
|
-
function runCardLines(run: RunRecord, snapshot: BoardSnapshot): string[] {
|
|
561
|
+
function runCardLines(run: RunRecord, snapshot: BoardSnapshot, lane: BoardLane): string[] {
|
|
555
562
|
const endedAt = run.endedAt ?? snapshot.now;
|
|
556
563
|
const duration = humanDuration(endedAt - run.startedAt);
|
|
557
564
|
const lines = [
|
|
@@ -564,30 +571,39 @@ function runCardLines(run: RunRecord, snapshot: BoardSnapshot): string[] {
|
|
|
564
571
|
`attempt ${run.attempt} · ${run.turns}/${run.maxTurns}t`,
|
|
565
572
|
`$${run.spendUsd.toFixed(2)} · ${duration}`,
|
|
566
573
|
];
|
|
567
|
-
//
|
|
568
|
-
//
|
|
569
|
-
//
|
|
570
|
-
//
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
// #173: a terminal blocked/failed run whose state label is gone reads as the
|
|
580
|
-
// lane it sits in — which the label being absent made PARKED, not the truth.
|
|
581
|
-
if (isLastRun(run, snapshot.labels)) lines.push(`last run: ${run.state}`);
|
|
582
|
-
// Ordered by what needs a human first. An unsalvaged tree outranks even a
|
|
583
|
-
// last error: the error describes a run that is over, the tree is work that
|
|
584
|
-
// is still at risk and an issue that will not dispatch (#118).
|
|
574
|
+
// Exactly one fourth line, so a card stays within the four content lines that
|
|
575
|
+
// renderColumn's slot arithmetic (`slots = floor((height - 2) / 5)`) allocates
|
|
576
|
+
// for it. Every fact a card can carry folds into this single priority:
|
|
577
|
+
// an unsalvaged tree outranks even a live verdict (#118) — the error describes
|
|
578
|
+
// a run that is over, the tree is work still at risk — then the live probe for
|
|
579
|
+
// a pushed row, then the word the lane name cannot carry, then the run's own
|
|
580
|
+
// evidence.
|
|
581
|
+
//
|
|
582
|
+
// `isLastRun` is deliberately not a card line: it belongs in the detail header
|
|
583
|
+
// (renderDetail's `stateShown`), where a full width can spell out `last run:
|
|
584
|
+
// blocked`. Putting it here too would blow the budget and repeat the #218
|
|
585
|
+
// conflation on the two-line boundary.
|
|
585
586
|
if (run.salvageError !== undefined && run.salvageAckAt === undefined) lines.push("UNSALVAGED WIP");
|
|
586
|
-
else
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
587
|
+
else {
|
|
588
|
+
// #173: a live re-verification of a pushed row, when the 60s probe has one.
|
|
589
|
+
// Guarded on the pushed states: a stale entry must not follow a row that
|
|
590
|
+
// settled into another lane and keep claiming "now: green" about a PR that
|
|
591
|
+
// already merged or failed.
|
|
592
|
+
const live =
|
|
593
|
+
run.state === "pushed-green" || run.state === "pushed-pending" ? snapshot.pr.get(run.id) : undefined;
|
|
594
|
+
if (live !== undefined) {
|
|
595
|
+
const reason = live.reason.split(/\s+/, 1)[0] ?? "";
|
|
596
|
+
lines.push(
|
|
597
|
+
live.status === "failed" ? `now: red — ${reason === "" ? "failed" : reason}` : `now: ${prLabel(live.status)}`,
|
|
598
|
+
);
|
|
599
|
+
} else if (lane === "settled") lines.push("issue closed");
|
|
600
|
+
else if (lane === "parked") lines.push("needs a label");
|
|
601
|
+
else if (run.state === "pushed-pending") lines.push("checks pending");
|
|
602
|
+
else if (run.salvageSha !== undefined) lines.push(`wip @ ${run.salvageSha.slice(0, 7)}`);
|
|
603
|
+
else if (run.lastError !== undefined) lines.push(run.lastError.replace(/\s+/g, " "));
|
|
604
|
+
else if (run.prUrl !== undefined) lines.push(run.prUrl.replace(/^https?:\/\//, ""));
|
|
605
|
+
else lines.push(run.branch);
|
|
606
|
+
}
|
|
591
607
|
return lines;
|
|
592
608
|
}
|
|
593
609
|
|
|
@@ -616,7 +632,7 @@ function renderColumn(
|
|
|
616
632
|
lines.push(styledCell(" (empty)", width, DIM));
|
|
617
633
|
} else {
|
|
618
634
|
for (const [offset, card] of shown.entries()) {
|
|
619
|
-
const raw = card.kind === "run" ? runCardLines(card.run, snapshot) : [`#${card.issue}`, card.reason, card.note];
|
|
635
|
+
const raw = card.kind === "run" ? runCardLines(card.run, snapshot, key) : [`#${card.issue}`, card.reason, card.note];
|
|
620
636
|
while (raw.length < 4) raw.push("");
|
|
621
637
|
const selected = selectedColumn && start + offset === selectedCard;
|
|
622
638
|
for (const [lineIndex, value] of raw.entries()) {
|
|
@@ -834,6 +850,8 @@ function renderHelp(width: number, height: number): string[] {
|
|
|
834
850
|
"? close help",
|
|
835
851
|
"Esc back, then quit",
|
|
836
852
|
"q / Ctrl-C back, then quit",
|
|
853
|
+
"",
|
|
854
|
+
"PARKED = issue open, no label — nothing dispatches it until you label it",
|
|
837
855
|
];
|
|
838
856
|
const top = Math.max(0, Math.floor((height - lines.length) / 2));
|
|
839
857
|
return [
|
|
@@ -916,7 +934,7 @@ async function probeBoardHealth(project: ProjectConfig): Promise<BoardHealth> {
|
|
|
916
934
|
*
|
|
917
935
|
* One open-issue snapshot per pass, never one list call per label (#203): the
|
|
918
936
|
* four lifecycle sets are derived in memory from it by {@link boardLabelSets},
|
|
919
|
-
* and
|
|
937
|
+
* and SETTLED closure is read off the same snapshot instead of a per-candidate
|
|
920
938
|
* `issueState` fan-out. The single read rides the conditional-request cache, so
|
|
921
939
|
* an idle board costs no primary rate-limit budget at all. The cadence stays
|
|
922
940
|
* the ten-second health refresh, never the one-second redraw path.
|
|
@@ -947,11 +965,19 @@ export async function probeBoardLabels(
|
|
|
947
965
|
const parkedCandidates = [...newestRun.values()].filter(
|
|
948
966
|
(run) => PARKED_STATES.has(run.state) && !labelled.has(run.issue),
|
|
949
967
|
);
|
|
950
|
-
// The snapshot IS the set of open issues, so
|
|
951
|
-
// lookup: a candidate whose issue
|
|
968
|
+
// The snapshot IS the set of open issues, so closure needs no per-issue
|
|
969
|
+
// lookup: a candidate whose issue does NOT appear in it is closed, *because
|
|
970
|
+
// this read succeeded*. That last clause is the whole distinction on the
|
|
971
|
+
// failed path — `closed` is only ever derived here, inside the try, so an
|
|
972
|
+
// unreadable probe survives as the previous set (or empty) and an issue the
|
|
973
|
+
// board could not confirm closed keeps parking rather than falsely reading
|
|
974
|
+
// as SETTLED (#218). Absence from `closed` is therefore open OR unknown,
|
|
975
|
+
// and both mean PARKED.
|
|
952
976
|
const openNumbers = new Set(open.map((issue) => issue.number));
|
|
953
|
-
const
|
|
954
|
-
|
|
977
|
+
const closed = new Set(
|
|
978
|
+
parkedCandidates.map((run) => run.issue).filter((n) => !openNumbers.has(n)),
|
|
979
|
+
);
|
|
980
|
+
return { queued, inProgress, blocked, failed, closed, readAt: Date.now() };
|
|
955
981
|
} catch (err) {
|
|
956
982
|
return {
|
|
957
983
|
...(previous ?? UNREAD_LABELS),
|
|
@@ -191,6 +191,34 @@ Keep the queue worth draining.
|
|
|
191
191
|
every attempt.
|
|
192
192
|
- An issue that has exhausted its attempts is not a retry candidate. Diagnose it,
|
|
193
193
|
split it, or hand it back to a human.
|
|
194
|
+
- **Grooming is throughput-bound, so delegate the finding.** N workers drain the
|
|
195
|
+
queue in parallel while you refill it alone: on a three-worker fleet grooming
|
|
196
|
+
has to produce roughly three well-specced issues in the time one worker takes
|
|
197
|
+
to finish one. Auditing candidates serially cannot keep up, and the board then
|
|
198
|
+
reads "0 ready / workers idle" while you are doing exactly what this duty asks.
|
|
199
|
+
When the queue is below the grooming trigger, fan out read-only `scout`
|
|
200
|
+
subagents over backlog clusters **in one batch** rather than auditing one issue
|
|
201
|
+
at a time. Scouts do the finding; you still do the deciding and you still write
|
|
202
|
+
the brief. The quality bar above does not move.
|
|
203
|
+
- **Give every scout the same return contract**, or it comes back with prose
|
|
204
|
+
nobody can act on:
|
|
205
|
+
- verdict — `ALREADY DONE` / `PROMOTABLE` / `NEEDS DECOMPOSITION` / `BLOCKED` /
|
|
206
|
+
`NEEDS PRODUCT DECISION`
|
|
207
|
+
- routing — exactly one repo, or `MULTI` with the split
|
|
208
|
+
- evidence — the file or symbol proving `ALREADY DONE`, never a title match
|
|
209
|
+
- entry points — the 3-6 files to change or read first
|
|
210
|
+
- existing tests covering the behaviour, by path
|
|
211
|
+
- the one thing most likely to be silently faked
|
|
212
|
+
- **Disqualifying an issue is a successful grooming outcome.** Measured on this
|
|
213
|
+
package's own fleet: four scouts over sixteen backlog issues promoted four and
|
|
214
|
+
*disqualified six* that looked promotable from their titles — four written
|
|
215
|
+
against a tenancy model a later epic retired, two blocked behind open
|
|
216
|
+
prerequisites named in their own bodies. Each would have burned a worker's whole
|
|
217
|
+
attempt to discover. On anything old, check first whether a later epic has
|
|
218
|
+
invalidated the issue's premise; that check is mechanical, read-only, and
|
|
219
|
+
exactly what a scout is cheap at and you are expensive at.
|
|
220
|
+
- Examine widely, promote narrowly. Sixteen examined and four promoted is the
|
|
221
|
+
shape to aim for — the cap is on what you **promote**, never on what you look at.
|
|
194
222
|
|
|
195
223
|
## Duty 3 — report
|
|
196
224
|
|
package/src/daemon.ts
CHANGED
|
@@ -35,7 +35,7 @@ import { recordReleaseBlock } from "./release-policy.ts";
|
|
|
35
35
|
import { branchName, effectiveLabels, route } from "./routing.ts";
|
|
36
36
|
import type { Routed, UnroutableReason } from "./routing.ts";
|
|
37
37
|
import { evaluateDecisionConditions, probeNpmVersion, probeRateLimitReset } from "./decisions.ts";
|
|
38
|
-
import { classifyRun, type ClassifyFacts } from "./failure-class.ts";
|
|
38
|
+
import { classifyRun, providerCreditRefusal, type ClassifyFacts } from "./failure-class.ts";
|
|
39
39
|
import { projectLabels } from "./label-projection.ts";
|
|
40
40
|
import { dbPath, openStore, utcDay } from "./store.ts";
|
|
41
41
|
import { makeTracker, type RateLimitStatus } from "./tracker/github.ts";
|
|
@@ -1108,6 +1108,16 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
1108
1108
|
: { state: result.state };
|
|
1109
1109
|
const state = verified.state;
|
|
1110
1110
|
|
|
1111
|
+
// Read before the row is written so `lastError` carries the provider's own
|
|
1112
|
+
// message — it names the fix and the URL, which no classification can.
|
|
1113
|
+
const providerCredit =
|
|
1114
|
+
state === "failed" || state === "killed"
|
|
1115
|
+
? (() => {
|
|
1116
|
+
const err = readSessionError(result.sessionFile);
|
|
1117
|
+
return err === undefined ? undefined : providerCreditRefusal(err);
|
|
1118
|
+
})()
|
|
1119
|
+
: undefined;
|
|
1120
|
+
|
|
1111
1121
|
// The other half of not believing a worker about its own run (#128). The
|
|
1112
1122
|
// claim being audited is `state: pushed-green`, so the audit runs on the
|
|
1113
1123
|
// worker's claim rather than on what verification made of it: a claim that
|
|
@@ -1192,7 +1202,11 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
1192
1202
|
// push: the report of a killed attempt is exactly the one a later
|
|
1193
1203
|
// continuation must pool its disclosures from (#199).
|
|
1194
1204
|
report: result.report,
|
|
1195
|
-
...(
|
|
1205
|
+
...(providerCredit !== undefined
|
|
1206
|
+
? { lastError: providerCredit }
|
|
1207
|
+
: verified.reason === undefined
|
|
1208
|
+
? {}
|
|
1209
|
+
: { lastError: verified.reason }),
|
|
1196
1210
|
...settlement?.patch,
|
|
1197
1211
|
...(audit === undefined || audit.flags.length === 0
|
|
1198
1212
|
? {}
|
|
@@ -1224,7 +1238,32 @@ async function handleIssue(d: Deps, r: Routed, attempt: number): Promise<void> {
|
|
|
1224
1238
|
maxContinuations: caps.maxContinuationsPerIssue,
|
|
1225
1239
|
});
|
|
1226
1240
|
|
|
1227
|
-
if (
|
|
1241
|
+
if (providerCredit !== undefined) {
|
|
1242
|
+
// Pause here rather than at classification: the sweep runs on the tick,
|
|
1243
|
+
// and three issues each burned an attempt in the fifteen minutes between
|
|
1244
|
+
// the first 402 and a human noticing (#220).
|
|
1245
|
+
setPaused(true, { source: "provider-credit", reason: providerCredit });
|
|
1246
|
+
log(`#${issue} provider refused for credit — dispatch paused: ${providerCredit}`);
|
|
1247
|
+
swapLabel(store, project.name, issue, inProgress, project.stateLabels.failed);
|
|
1248
|
+
// Fleet-scoped and run-independent on purpose. The notification ledger
|
|
1249
|
+
// dedupes on `project:issue:tier:summary`, so `NO_ISSUE` plus a summary
|
|
1250
|
+
// carrying no run or attempt is what makes this page once for the fleet
|
|
1251
|
+
// instead of once per affected run.
|
|
1252
|
+
await safeEscalate(d, {
|
|
1253
|
+
tier: 2,
|
|
1254
|
+
project: project.name,
|
|
1255
|
+
issue: NO_ISSUE,
|
|
1256
|
+
summary: `Model provider refused for credit — ${project.name} is paused`,
|
|
1257
|
+
detail: [
|
|
1258
|
+
providerCredit,
|
|
1259
|
+
"",
|
|
1260
|
+
"No implementation attempt was charged: this is a billing state, not a",
|
|
1261
|
+
"failed implementation. Each affected issue keeps its queue label and",
|
|
1262
|
+
"re-dispatches on `omp-conductor resume` once the provider has credit.",
|
|
1263
|
+
`Session: ${result.sessionFile ?? "(no transcript)"}`,
|
|
1264
|
+
].join("\n"),
|
|
1265
|
+
});
|
|
1266
|
+
} else if (continueTurns) {
|
|
1228
1267
|
// Requeue as one ordered pair: the in-progress removal before the
|
|
1229
1268
|
// queue add, exactly the order the projector will apply them in (#201).
|
|
1230
1269
|
store.enqueueLabelOps(project.name, [
|
|
@@ -2505,6 +2544,12 @@ export interface StatusSnapshot {
|
|
|
2505
2544
|
configPath: string;
|
|
2506
2545
|
stateDir: string;
|
|
2507
2546
|
paused: boolean;
|
|
2547
|
+
/**
|
|
2548
|
+
* Why the fleet is paused, when whatever paused it said. Rendered beside
|
|
2549
|
+
* `(PAUSED)` so a pause an operator did not issue names itself instead of
|
|
2550
|
+
* reading like a mistake (#220).
|
|
2551
|
+
*/
|
|
2552
|
+
pauseReason?: string;
|
|
2508
2553
|
caps: Caps;
|
|
2509
2554
|
/**
|
|
2510
2555
|
* The effective per-shape release grants. On the snapshot rather than re-read
|
|
@@ -2579,11 +2624,15 @@ export function statusSnapshotFromStore(
|
|
|
2579
2624
|
const dispatch = store.latestDispatch(p.name);
|
|
2580
2625
|
const labelOpsPending = store.countPendingLabelOps(p.name);
|
|
2581
2626
|
const oldestLabelOpAt = store.oldestPendingLabelOpAt(p.name);
|
|
2627
|
+
// Read once: the provenance read touches the filesystem, and the renderer
|
|
2628
|
+
// should never pay for it twice per status.
|
|
2629
|
+
const reason = pauseProvenance()?.reason;
|
|
2582
2630
|
return {
|
|
2583
2631
|
project: p.name,
|
|
2584
2632
|
configPath: configPath(),
|
|
2585
2633
|
stateDir: stateDir(),
|
|
2586
2634
|
paused: isPaused(),
|
|
2635
|
+
...(reason === undefined ? {} : { pauseReason: reason }),
|
|
2587
2636
|
caps,
|
|
2588
2637
|
releaseGrants: resolveReleaseGrants(p),
|
|
2589
2638
|
activeRuns: store.activeRuns(p.name),
|
|
@@ -2857,6 +2906,57 @@ export function lastToolCalls(sessionFile: string | undefined, limit = SPIN_EVID
|
|
|
2857
2906
|
return names.slice(-limit);
|
|
2858
2907
|
}
|
|
2859
2908
|
|
|
2909
|
+
/** A provider refusal a session recorded before dying, or undefined for none. */
|
|
2910
|
+
export interface SessionError {
|
|
2911
|
+
status?: number;
|
|
2912
|
+
message: string;
|
|
2913
|
+
}
|
|
2914
|
+
|
|
2915
|
+
/**
|
|
2916
|
+
* The last error a transcript recorded, or undefined when it recorded none.
|
|
2917
|
+
*
|
|
2918
|
+
* The harness writes `{"stopReason":"error","errorStatus":402,"errorId":402,
|
|
2919
|
+
* "errorMessage":"402 This request requires more credits, ..."}`. The daemon
|
|
2920
|
+
* read none of it, so three runs died `unknown` with an empty `lastError` and
|
|
2921
|
+
* charged an attempt each for a billing state (#220).
|
|
2922
|
+
*
|
|
2923
|
+
* Scanned newest-first: a session that recovered from an early error and then
|
|
2924
|
+
* died of something else must report the something else, and a session that
|
|
2925
|
+
* recovered from its only error and finished cleanly reports the error anyway
|
|
2926
|
+
* because there is no terminal verdict to outrank it (#220).
|
|
2927
|
+
*/
|
|
2928
|
+
export function readSessionError(sessionFile: string | undefined): SessionError | undefined {
|
|
2929
|
+
if (sessionFile === undefined) return undefined;
|
|
2930
|
+
let text: string;
|
|
2931
|
+
try {
|
|
2932
|
+
text = readFileSync(sessionFile, "utf8");
|
|
2933
|
+
} catch {
|
|
2934
|
+
return undefined;
|
|
2935
|
+
}
|
|
2936
|
+
const lines = text.split("\n");
|
|
2937
|
+
for (let i = lines.length - 1; i >= 0; i -= 1) {
|
|
2938
|
+
const line = lines[i];
|
|
2939
|
+
if (line === undefined || line.length === 0) continue;
|
|
2940
|
+
let row: unknown;
|
|
2941
|
+
try {
|
|
2942
|
+
row = JSON.parse(line) as unknown;
|
|
2943
|
+
} catch {
|
|
2944
|
+
continue;
|
|
2945
|
+
}
|
|
2946
|
+
if (row === null || typeof row !== "object") continue;
|
|
2947
|
+
const rec = row as { readonly [key: string]: unknown };
|
|
2948
|
+
if (rec["stopReason"] !== "error") continue;
|
|
2949
|
+
const message = rec["errorMessage"];
|
|
2950
|
+
if (typeof message !== "string" || message.trim() === "") continue;
|
|
2951
|
+
const status = rec["errorStatus"];
|
|
2952
|
+
return {
|
|
2953
|
+
...(typeof status === "number" && Number.isFinite(status) ? { status } : {}),
|
|
2954
|
+
message: message.trim(),
|
|
2955
|
+
};
|
|
2956
|
+
}
|
|
2957
|
+
return undefined;
|
|
2958
|
+
}
|
|
2959
|
+
|
|
2860
2960
|
/**
|
|
2861
2961
|
* Classify every unclassified terminal run, persist the verdict, and perform the
|
|
2862
2962
|
* one recovery its class names (#132).
|
package/src/failure-class.ts
CHANGED
|
@@ -113,6 +113,37 @@ export function startFailure(lastError: string | undefined): string | undefined
|
|
|
113
113
|
return hit === undefined ? undefined : lastError.split("\n")[0]?.trim();
|
|
114
114
|
}
|
|
115
115
|
|
|
116
|
+
/**
|
|
117
|
+
* Provider messages that mean "this key is out of money", matched on the text
|
|
118
|
+
* the provider itself prints.
|
|
119
|
+
*/
|
|
120
|
+
const CREDIT_REFUSAL_SIGNATURES = [
|
|
121
|
+
"requires more credits",
|
|
122
|
+
"insufficient credits",
|
|
123
|
+
"insufficient_quota",
|
|
124
|
+
"exceeded your current quota",
|
|
125
|
+
"credit balance is too low",
|
|
126
|
+
] as const;
|
|
127
|
+
|
|
128
|
+
/**
|
|
129
|
+
* Evidence that a model provider refused this run for credit, or `undefined`
|
|
130
|
+
* when it did not.
|
|
131
|
+
*
|
|
132
|
+
* Two independent signals, either sufficient: HTTP 402, which means exactly
|
|
133
|
+
* this and nothing else, and the refusal text providers print when a key is out
|
|
134
|
+
* of allowance without using 402. Deliberately narrow — a generic 429 is rate
|
|
135
|
+
* limiting, which is a different failure with a different remedy.
|
|
136
|
+
*/
|
|
137
|
+
export function providerCreditRefusal(error: {
|
|
138
|
+
status?: number;
|
|
139
|
+
message: string;
|
|
140
|
+
}): string | undefined {
|
|
141
|
+
const text = error.message.toLowerCase();
|
|
142
|
+
const named = CREDIT_REFUSAL_SIGNATURES.some((signature) => text.includes(signature));
|
|
143
|
+
if (error.status !== 402 && !named) return undefined;
|
|
144
|
+
return error.message.split("\n")[0]?.trim() ?? error.message;
|
|
145
|
+
}
|
|
146
|
+
|
|
116
147
|
/**
|
|
117
148
|
* Evidence that this run never started, or `undefined` when something did happen.
|
|
118
149
|
*
|
|
@@ -201,6 +232,22 @@ export function classifyRun(run: RunRecord, facts: ClassifyFacts): Classificatio
|
|
|
201
232
|
};
|
|
202
233
|
}
|
|
203
234
|
|
|
235
|
+
// A billing state, not an implementation failure. Its own class because it is
|
|
236
|
+
// the one outage an operator fixes with a card rather than a diagnosis, and
|
|
237
|
+
// because burying a self-describing provider error in `unknown` erodes what
|
|
238
|
+
// `unknown` means (#220). Ahead of `neverStarted` on purpose: a 402 on the
|
|
239
|
+
// first request produces a turn-0 row with no artifacts, which
|
|
240
|
+
// `env-start-failure` would otherwise absorb and lose the cause.
|
|
241
|
+
if (run.state === "failed" || run.state === "killed") {
|
|
242
|
+
const credit =
|
|
243
|
+
run.lastError === undefined
|
|
244
|
+
? undefined
|
|
245
|
+
: providerCreditRefusal({ message: run.lastError });
|
|
246
|
+
if (credit !== undefined) {
|
|
247
|
+
return { cls: "provider-credit", recovery: "requeue", evidence: credit };
|
|
248
|
+
}
|
|
249
|
+
}
|
|
250
|
+
|
|
204
251
|
// A run that never started is the most classifiable failure there is, and the
|
|
205
252
|
// least deserving of an implementation attempt: the session did not get as far
|
|
206
253
|
// as reading the issue. See {@link neverStarted} for the two shapes and why the
|
package/src/fleet.ts
CHANGED
|
@@ -1062,6 +1062,7 @@ export function formatFleetStatus(
|
|
|
1062
1062
|
function formatProjectBody(s: StatusSnapshot): string {
|
|
1063
1063
|
const lines = [
|
|
1064
1064
|
`project ${s.project}${s.paused ? " (PAUSED)" : ""}`,
|
|
1065
|
+
...(s.pauseReason === undefined ? [] : [`paused ${s.pauseReason}`]),
|
|
1065
1066
|
`config ${s.configPath}`,
|
|
1066
1067
|
`state ${s.stateDir}`,
|
|
1067
1068
|
"",
|
package/src/store.ts
CHANGED
|
@@ -902,12 +902,12 @@ export function openStore(dbPath: string): Store {
|
|
|
902
902
|
const countFailures = db.query<{ n: number }, [string, number]>(
|
|
903
903
|
`SELECT COUNT(*) AS n FROM runs
|
|
904
904
|
WHERE project = ? AND issue = ? AND state = 'failed'
|
|
905
|
-
AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck', 'env-start-failure', 'dispatch-infra'))`,
|
|
905
|
+
AND (failureClass IS NULL OR failureClass NOT IN ('ci-infra', 'settlement-stuck', 'env-start-failure', 'dispatch-infra', 'provider-credit'))`,
|
|
906
906
|
);
|
|
907
907
|
const countContinuations = db.query<{ n: number }, [string, number]>(
|
|
908
908
|
`SELECT COUNT(*) AS n FROM runs
|
|
909
909
|
WHERE project = ? AND issue = ? AND state IN ('killed', 'orphaned', 'blocked')
|
|
910
|
-
AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck', 'env-start-failure', 'dispatch-infra'))`,
|
|
910
|
+
AND (failureClass IS NULL OR failureClass NOT IN ('admin-kill', 'settlement-stuck', 'env-start-failure', 'dispatch-infra', 'provider-credit'))`,
|
|
911
911
|
);
|
|
912
912
|
// How many times one issue reached a given class. Recovery uses it to bound a
|
|
913
913
|
// retry loop whose cause is persistent (e.g. a mirror that will not refresh):
|
package/src/types.ts
CHANGED
package/src/unblock.ts
CHANGED
|
@@ -73,11 +73,11 @@ export interface UnblockOutcome {
|
|
|
73
73
|
requeueSkipped?: true;
|
|
74
74
|
/** Set when the tracker refused one or more of this verb's label ops, so
|
|
75
75
|
* the label sync is owed rather than done: the intended label state is
|
|
76
|
-
* durable in the store and the daemon retries it (#201).
|
|
77
|
-
*
|
|
78
|
-
*
|
|
79
|
-
*
|
|
80
|
-
* number is what is still pending. */
|
|
76
|
+
* durable in the store and the daemon retries it (#201). This says nothing
|
|
77
|
+
* about claimability — `listReady` reads the tracker by label, so an issue
|
|
78
|
+
* is dispatchable only once the queue label is physically present, which a
|
|
79
|
+
* pending add has not delivered and `--no-requeue` never enqueues at all.
|
|
80
|
+
* The number is what is still pending. */
|
|
81
81
|
labelSyncQueued?: number;
|
|
82
82
|
}
|
|
83
83
|
|
|
@@ -268,7 +268,7 @@ export function formatUnblock(
|
|
|
268
268
|
const labelsPending = o.labelSyncQueued !== undefined;
|
|
269
269
|
if (o.labelSyncQueued !== undefined) {
|
|
270
270
|
lines.push(
|
|
271
|
-
` label sync queued (${o.labelSyncQueued} pending) — the daemon retries
|
|
271
|
+
` label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`,
|
|
272
272
|
);
|
|
273
273
|
}
|
|
274
274
|
if (o.forced === true) {
|
|
@@ -321,7 +321,7 @@ export function formatUnblock(
|
|
|
321
321
|
lines.push(
|
|
322
322
|
latest.prUrl === undefined
|
|
323
323
|
? labelsPending
|
|
324
|
-
? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries
|
|
324
|
+
? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`
|
|
325
325
|
: o.requeued === true
|
|
326
326
|
? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` +
|
|
327
327
|
"the dispatcher still applies its open-PR check at claim time)"
|
|
@@ -352,7 +352,7 @@ export function formatUnblock(
|
|
|
352
352
|
);
|
|
353
353
|
if (labelsPending) {
|
|
354
354
|
lines.push(
|
|
355
|
-
` queue label sync queued (${o.labelSyncQueued} pending) — the daemon retries
|
|
355
|
+
` queue label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`,
|
|
356
356
|
);
|
|
357
357
|
} else if (o.requeued === true) {
|
|
358
358
|
lines.push(` queue "${project.queueLabel}" restored (no-op if it was already present)`);
|
|
@@ -361,7 +361,7 @@ export function formatUnblock(
|
|
|
361
361
|
} else {
|
|
362
362
|
lines.push(
|
|
363
363
|
labelsPending
|
|
364
|
-
? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries
|
|
364
|
+
? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`
|
|
365
365
|
: o.requeued === true
|
|
366
366
|
? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` +
|
|
367
367
|
"the dispatcher still applies its open-PR check at claim time)"
|
package/src/worker.ts
CHANGED
|
@@ -19,6 +19,9 @@ const HEAD_SHA_PATTERN = /^head:\s*([0-9a-f]{40})\s*$/im;
|
|
|
19
19
|
const PUSHED_GREEN_PATTERN = /^state:\s*pushed-green\s*$/im;
|
|
20
20
|
const BLOCKED_PATTERN = /^state:\s*blocked\s*$/im;
|
|
21
21
|
|
|
22
|
+
/** Any explicit verdict line, whatever it claims. */
|
|
23
|
+
const STATE_LINE_PATTERN = /^state:\s*\S+\s*$/im;
|
|
24
|
+
|
|
22
25
|
/** `{{KEY}}` placeholders in a brief template. */
|
|
23
26
|
const PLACEHOLDER_PATTERN = /\{\{([A-Za-z0-9_]+)\}\}/g;
|
|
24
27
|
|
|
@@ -160,6 +163,19 @@ export function deriveResult(report: string): {
|
|
|
160
163
|
};
|
|
161
164
|
}
|
|
162
165
|
|
|
166
|
+
/**
|
|
167
|
+
* Did this report state a verdict at all?
|
|
168
|
+
*
|
|
169
|
+
* The discriminator between "the worker said something about its outcome" and
|
|
170
|
+
* "the worker said something else entirely". A finished worker woken by its own
|
|
171
|
+
* stale polling timers emits neither `state: pushed-green` nor `state: blocked`
|
|
172
|
+
* — it emits prose — and that is the only case where an earlier verdict is
|
|
173
|
+
* still the run's real answer.
|
|
174
|
+
*/
|
|
175
|
+
export function hasVerdictLine(report: string): boolean {
|
|
176
|
+
return STATE_LINE_PATTERN.test(report);
|
|
177
|
+
}
|
|
178
|
+
|
|
163
179
|
/**
|
|
164
180
|
* Is this `agent_end` the end of the run?
|
|
165
181
|
*
|
|
@@ -224,6 +240,14 @@ export async function runWorker(
|
|
|
224
240
|
let turns = 0;
|
|
225
241
|
let spendUsd = 0;
|
|
226
242
|
let report = "";
|
|
243
|
+
// The newest COMPLETE `pushed-green` verdict this session emitted. Tracked
|
|
244
|
+
// apart from `report` because `report` is deliberately the newest non-empty
|
|
245
|
+
// text — a run cut off mid-sentence must still report what it said last —
|
|
246
|
+
// and a worker that finishes, then wakes on its own stale polling timers,
|
|
247
|
+
// overwrites its own verdict with "nothing to resume" chatter. veltro#406
|
|
248
|
+
// burned 180/180 turns that way and charged an attempt against a PR that
|
|
249
|
+
// merged (#217).
|
|
250
|
+
let claim: { prUrl: string; headSha: string } | undefined;
|
|
227
251
|
let killedBy: KilledBy | undefined;
|
|
228
252
|
// Bun's global timer handle; cleared on every exit path below.
|
|
229
253
|
let timer: Timer | undefined;
|
|
@@ -265,7 +289,13 @@ export async function runWorker(
|
|
|
265
289
|
// Keep the newest non-empty assistant text: whatever the worker said last
|
|
266
290
|
// is its report, whether it finished cleanly or was cut off.
|
|
267
291
|
const text = reportText(field(message, "content"));
|
|
268
|
-
if (text !== "")
|
|
292
|
+
if (text !== "") {
|
|
293
|
+
report = text;
|
|
294
|
+
const stated = deriveResult(text);
|
|
295
|
+
if (stated.state === "pushed-green" && stated.prUrl !== undefined && stated.headSha !== undefined) {
|
|
296
|
+
claim = { prUrl: stated.prUrl, headSha: stated.headSha };
|
|
297
|
+
}
|
|
298
|
+
}
|
|
269
299
|
|
|
270
300
|
// Real cost lives on assistant messages as `usage.cost.total` (live hermes
|
|
271
301
|
// transcripts, 2026-08-07). The earlier agent_end.telemetry path never
|
|
@@ -337,9 +367,26 @@ export async function runWorker(
|
|
|
337
367
|
}
|
|
338
368
|
|
|
339
369
|
if (killedBy !== undefined) {
|
|
340
|
-
|
|
370
|
+
// The PR and head are facts the session already established, so they
|
|
371
|
+
// survive the kill. Without them `shouldContinueAfterTurnsCap` sees no
|
|
372
|
+
// artifacts and charges an implementation attempt for a cap kill that had
|
|
373
|
+
// real work to continue from.
|
|
374
|
+
return withSessionFacts({
|
|
375
|
+
state: "killed",
|
|
376
|
+
turns,
|
|
377
|
+
spendUsd,
|
|
378
|
+
report,
|
|
379
|
+
killedBy,
|
|
380
|
+
...(claim === undefined ? {} : { prUrl: claim.prUrl, headSha: claim.headSha }),
|
|
381
|
+
});
|
|
341
382
|
}
|
|
342
383
|
|
|
384
|
+
// An explicit later verdict always wins: a worker that pushed green and then
|
|
385
|
+
// stopped to ask a question means the question. The earlier claim is only
|
|
386
|
+
// restored when the last thing said was not a verdict at all.
|
|
387
|
+
if (claim !== undefined && !hasVerdictLine(report)) {
|
|
388
|
+
return withSessionFacts({ state: "pushed-green", ...claim, turns, spendUsd, report });
|
|
389
|
+
}
|
|
343
390
|
return withSessionFacts({
|
|
344
391
|
...deriveResult(report),
|
|
345
392
|
turns,
|