@ai-dossier/sched 0.23.2 → 0.24.0

package/README.md

@@ -146,7 +146,7 @@ where every mechanical supervision decision is code, not remembered prose:
    itself — see `slot-released` below (#525).
 6. **Journal (AC6)** — every event (assigned, spawned, exit-detected, external-advance,
    progress, stalled, redispatched, fence-written, fence-failed, unit-failed,
-   dependents-blocked, slot-released, suspect-dispatch, dispatch-unhealthy,
+   dependents-blocked, slot-released, suspect-dispatch, dispatch-unhealthy, dispatch-failure,
    run-log-recorded, run-log-no-usage, run-log-skipped, run-log-failed, engine-stale,
    engine-auto-upgrade-attempted, engine-auto-upgrade-failed, stale-milestone-ignored, …) is
    appended to `events.jsonl`; `sched status` shows the live phase per unit, plus each
@@ -182,6 +182,33 @@ where every mechanical supervision decision is code, not remembered prose:
    a wall that hasn't actually cleared), which also clears the streak so `sched status`'s
    warning doesn't linger against a wall the operator already acted on.
 
+   **Confirmed dispatch failures (#629)** are a SEPARATE, deterministic signal alongside
+   the timing heuristic above: a dispatch result carrying `api_error_status` or
+   `terminal_reason: "api_error"` (`parseDispatchApiError`, `@ai-dossier/core`) is a
+   confirmed provider wall — a 429 spend/rate limit, an auth failure the provider itself
+   rejected — never an agent that ran, journaled **`dispatch-failure`** with the
+   provider's own status/reason/message and reset time (when supplied). Unlike
+   `suspect-dispatch`, a repeat from the SAME unit COUNTS toward its own
+   `consecutive_dispatch_api_errors` counter — no cross-unit correlation is needed when
+   the classification is a parsed field rather than a timing inference, and the incident
+   that motivated this (a batch tail respawning nine times in 33 minutes) was the SAME
+   unit throughout. At the same `DISPATCH_UNHEALTHY_THRESHOLD` it reuses `setPaused`/
+   `dispatch-unhealthy`. The redispatch itself does NOT escalate — same tier,
+   `recoveries` unchanged, `ESCALATION_CAP` never consumed, since a spend wall is not the
+   issue's fault — so once paused, BOTH the per-issue rail (`enterRecovery`/
+   `reconcileRecovering`, held in `recovering` until `sched resume`) and every batch
+   respawn wedge in `runBatchTick` (tail, member continuation, fix, report, and the
+   `ready` → `claimAndSetup` claim — `runValidate`'s local suite run is unaffected) stop
+   respawning into the wall, which previously ignored `paused` entirely. `dispatch-
+   health.ts` is its own module (not `engine.ts`) specifically so it is shared by BOTH
+   dispatch paths without `engine.ts` and `batch-dispatch.ts` importing each other.
+   Batch dispatch logs (per-role, append-mode) are fenced to `log_offset_at_spawn`,
+   stamped by each batch spawn function — a later dispatch that dies with NO result event
+   at all is never misclassified against a stale `api_error` result left in the same log
+   file by an earlier attempt. A verified completion/park resets the streak on the
+   per-issue path; a successful member/tail/report resets it on the batch path; `sched
+   resume` resets it on both.
+
 Config schema moves to 1.4.0 (#527): `dispatch` gains `tiers` — a per-tier
 `{ command?, model?, prompt? }` spawn spec. `command`/`tier_models`/`prompt` remain valid
 as the shorthand and are the fallback for any field a `tiers` entry leaves unset, so a
@@ -232,7 +259,12 @@ Two engine-safety policies were explicit product decisions on #464:
   network down), stall and verify-fail decisions pause for that unit — an outage can
   never kill a healthy agent or fail a unit as "unverified". An agent that exits during
   an outage holds in `verifying` until truth returns. Each pause is journaled as
-  `ground-truth-unreachable`.
+  `ground-truth-unreachable` — since #632 once per unbroken streak (carrying `since` and
+  `ticks_persisted`) rather than once per tick, re-announced every 20 ticks and re-armed
+  when truth recovers and fails anew, so an outage across `max_slots` units no longer
+  emits one line per unit per reconcile interval. Two sites are deliberately undeduped:
+  `failOrAdoptOpenPr`, which is terminal and so fires at most once per dispatch, and
+  `runTeardownFor`, which is audited but not yet fixed (#636).
 - **A completion milestone is fenced to its own dispatch (#575).** `isVerifiedComplete`
   (issues) and `isMemberComplete` (batch members) both accept the current dispatch's
   `SlotEntry.spawned_at` and reject a `report done` / `review done mode=slot` milestone
@@ -587,6 +619,39 @@ states migrate on load, backfilling `null` — nothing was ever recorded per-dis
 under the old once-per-tick behavior, so null is exact, not a guess. Resets with the
 slot on release (`CLEARED_SLOT_FIELDS`).
 
+Schema 1.13.0 (#630): `BatchEntry` gains `pr_watch_failed_reason` (the reason already
+journalled for the batch's current `pr-watch-failed` streak, null when the watch is
+healthy), `pr_watch_failed_since` (ISO, when the streak began) and
+`pr_watch_failed_ticks` (ticks the streak has persisted, silent ones included) — so
+`pr-watch-failed` fires once per distinct condition rather than once per reconcile tick,
+re-announcing every `JOURNAL_DEDUP_REANNOUNCE_TICKS` (20) ticks. 1.12.0 states migrate on
+load, backfilling `null`/`null`/`0` — no dedup marker was ever recorded under the old
+once-per-tick behavior, so it is exact, not a guess. Cleared whenever the batch leaves
+`awaiting-merge` (`CLEARED_PR_WATCH_FIELDS`), since the marker is scoped to one
+awaiting-merge stretch and a batch can return to that status on the same id via
+rebase-and-reship.
+
+Schema 1.14.0 (#632): `QueueEntry` gains `ground_truth_unreachable_since`/`_ticks` and
+`pr_watch_waiting_since`/`_ticks` — the same dedup scoped to the issue rather than the
+batch, because these sites span slot-held, parked and stale-failed units alike and
+`QueueEntry` is the one record every unit has either way. 1.13.0 and earlier states
+migrate on load, backfilling `null`/`0` — exact, not a guess. Reset on every requeue
+(`CLEARED_ENTRY_DEDUP_MARKERS`, applied by `requeueMember`,
+`requeueOrphanedDispatches` and the label-cleared path): a requeue is a fresh attempt,
+and a streak from the previous run must not silence this one's first occurrence. Written
+with `touchUpdatedAt: false` — `QueueEntry.updated_at` is load-bearing for
+`isStaleFailedPark`'s 7-day window, `status.ts`'s "parked since" and `readiness.ts`'s
+tiebreak, and a silent dedup tick must not reset them. `patchBatch` takes the same flag
+for the same reason on the batch rail.
+Schema 1.13.0 (#629): `SchedState` gains `consecutive_dispatch_api_errors` (number) and
+`dispatch_pause_reset_at` (string or null) — the confirmed-dispatch-failure streak and
+the provider's own reset time. Unlike the #505 suspect-dispatch pair these are NOT a
+single fact: a provider that reports no reset time leaves `dispatch_pause_reset_at`
+null while the counter is nonzero (`validateState` only enforces the reverse — a reset
+time can never outlive a streak that has already cleared to zero). 1.12.0 states
+migrate on load, backfilling `0`/`null` — no confirmed dispatch failures were ever
+tracked under them, so those values are exact, not a guess.
+
 New journal events: `batch-setup-done`, `batch-setup-failed`, `member-advanced`,
 `batch-warmup-done`, `batch-warmup-failed` (#561 — the cold-path warm step only; a pool
 claim emits neither). `gate-inconclusive` (#583 — the incremental gate came back
@@ -707,7 +772,7 @@ import {
                          //   so batch-dispatch.ts can share it.
   TRANSITIONS,           // the transition tables themselves (for previews)
   buildStatusReport,     // machine-readable status incl. blocked/failed sets
-  validateState,         // strict persisted-state validation (1.0.0-1.11.0 files migrate)
+  validateState,         // strict persisted-state validation (1.0.0-1.13.0 files migrate)
   DEFAULT_ISSUE_PRIORITY, DEFAULT_BATCH_PRIORITY, // priority defaults (0 / 10, #565)
   IllegalTransitionError, EnqueueError, CorruptStateError, LockTimeoutError,
   SchedNotFoundError,
@@ -795,9 +860,10 @@ ladder is exhausted, the terminal `unit-failed` (`agent-exited-unverified` /
 the dead-pid detection rail and the stall kill RECORD a fresh log slice (a `runs.jsonl`
 entry) in the same tick — `recordDispatchRunLog` is once-per-dispatch and refuses to
 append a second entry over the same slice. Since #620 a slot already `exited`/`verifying`
-when reconciled again still ATTRIBUTES one: `readLastToolForSlot` re-parses the same
-static slice for the tool name alone, writing nothing, so it carries no exactly-once
-constraint. That matters because the verify decision lands on a LATER tick than the
+when reconciled again still ATTRIBUTES one: `readDispatchSignalsForSlot` re-parses the
+same static slice (since #629, for the last-tool-call AND API-error-classification
+signals together — see the dispatch-health section above), writing nothing, so it
+carries no exactly-once constraint. That matters because the verify decision lands on a LATER tick than the
 dead-pid detection whenever ground truth was unreachable in between — previously the
 tool name was simply lost on exactly the runs hardest to diagnose. The read is deferred
 behind a thunk so only the tick that reaches the unverified-exit decision pays for it,
@@ -871,7 +937,8 @@ after-the-fact recovery, not a missing-data bug.
   queue data.
 - **Schema**: state/config files from #460 (schema 1.0.0), #464 (1.1.0), #468 (1.2.0),
   #472 (1.3.0), #500 (1.4.0), #505 (1.5.0), #504 (1.6.0), #523 (1.7.0) and #524 (1.8.0)
-  load and migrate to 1.12.0 automatically (slot `branch`/`last_head`/`pid_start`, slot `role` (inferred from the
+  load and migrate to 1.14.0 automatically (slot `branch`/`last_head`/`pid_start`, slot `role` (inferred from the
+  load and migrate to 1.13.0 automatically (slot `branch`/`last_head`/`pid_start`, slot `role` (inferred from the
   unit's queue entry, with the persisted `phase` as a fallback — #500), entry
   `pr`/`cleanup`/`failure_evidence`, batch `anchor`/`branch`/`run_id`/`eviction_groups`/
   `evictions`/`fix_attempts`/`rebase_attempts`, state-level `last_pr_poll_at` backfill to
@@ -879,16 +946,22 @@ after-the-fact recovery, not a missing-data bug.
   backfill to `0`/`null` — #505, slot `gen`/`fenced_at` backfill to `0`/`null` — #504, and
   slot `spawned_at`/`log_offset_at_spawn` backfill to `null`/`null` — #524,
   state-level `last_label_poll_at` backfill to `null` — #544, entry `priority` and batch
-  `member_gates`/`blocked_reason` — #565/#583, and slot
-  `stale_milestone_ignored_for` backfill to `null` — #610).
+  `member_gates`/`blocked_reason` — #565/#583, slot
+  `stale_milestone_ignored_for` backfill to `null` — #610, and state-level
+  `consecutive_dispatch_api_errors`/`dispatch_pause_reset_at` backfill to `0`/`null` —
+  #629).
 - **`max_slots`** bounds live units (`assigned | running | recovering`); dependency
   edges gate readiness — an issue with an unmerged dependency, and a batch behind an
   unmerged batch, are never runnable.
-- **Pause** stops new assignments only (including report-agent dispatch — #505); abandon
-  routes through the typed failure rails (`evicted → requeued{full}` for batch members —
-  nothing green is discarded). A pause can be manual (`sched pause`) or automatic
-  (dispatch-health, #505 above); `sched resume` clears both the flag and the
-  dispatch-health streak.
+- **Pause** stops new assignments only (including report-agent dispatch — #505) and,
+  since #629, every batch respawn wedge in `runBatchTick` — tail, member continuation,
+  fix, report, and the `ready` → `claimAndSetup` claim, which previously ignored `paused`
+  entirely (`runValidate` is excepted: a local suite run, not a dispatch) — and an
+  unescalated per-issue redispatch (`enterRecovery`/`reconcileRecovering`, held in
+  `recovering` until resumed). Abandon routes through the typed failure rails
+  (`evicted → requeued{full}` for batch members — nothing green is discarded). A pause
+  can be manual (`sched pause`) or automatic (dispatch-health, #505/#629 above); `sched
+  resume` clears the flag and both dispatch-health streaks.
 
 ## Hard-block labels are re-read every tick (#544)
 
@@ -1018,6 +1091,16 @@ in `events.jsonl` (`pr-parked` — two paths, see 1 and 1b above,
 `pr-watch-waiting`, `teardown-done`/`teardown-failed`, `report-dispatched`,
 `report-failed`, `ground-truth-unreachable`, `stale-failure-reconciled`).
 
+Since #610/#630/#632, `pr-watch-failed`, `pr-watch-waiting` and
+`ground-truth-unreachable` are emitted **once per unbroken streak** of the condition, not
+once per tick, re-announcing every `JOURNAL_DEDUP_REANNOUNCE_TICKS` (20) ticks. Each
+entry carries `at` (the engine's decision clock), `since` (the streak's onset) and
+`ticks_persisted` (ticks the streak has run, silent ones included). Read `since`, not the
+tick count, for duration: 20 ticks is ~20 min on the per-reconcile sites
+(`reconcile_interval_ms`, default 60 s) and ~50 min on the sites gated on the parked-PR
+poll (`pr_poll_interval_ms`, default 150 s), and both are operator-tunable. A count of
+these events is therefore a count of streaks and re-announcements, not of ticks.
+
 ## Development
 
 ```bash

package/dist/batch-dispatch.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;AAmBH,OAAO,EAUL,KAAK,gBAAgB,EAErB,KAAK,SAAS,EACf,MAAM,YAAY,CAAC;AACpB,OAAO,EACL,KAAK,WAAW,EAQjB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,KAAK,OAAO,EAAa,MAAM,WAAW,CAAC;AACpD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,EAUL,KAAK,WAAW,EACjB,MAAM,YAAY,CAAC;AAiBpB,OAAO,EAAE,KAAK,QAAQ,EAA2D,MAAM,YAAY,CAAC;AACpG,OAAO,KAAK,EAEV,UAAU,EAEV,oBAAoB,EAKpB,WAAW,EAGZ,MAAM,SAAS,CAAC;AAWjB,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAE1C,8DAA8D;AAC9D,MAAM,WAAW,iBAAiB;IAChC,KAAK,EAAE,UAAU,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,SAAS,EAAE,SAAS,CAAC;IACrB,GAAG,EAAE,MAAM,IAAI,CAAC;IAChB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,0FAA0F;IAC1F,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sGAAsG;IACtG,QAAQ,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,WAAW,CAAC;IAC5C;;;;;;;;;;;;;;;;;;;OAmBG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,KAAK,oBAAoB,CAAC;IACjF,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,8FAA8F;AAC9F,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,8FAA8F;IAC9F,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AA6oCD;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,KAAK,EAAE,UAAU,EACjB,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,aAAa,EACtB,GAAG,EAAE,IAAI,EACT,MAAM,EAAE,eAAe,GACtB,IAAI,CAyBN;AAuMD;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,wBAAgB,iBAAiB,CAC/B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,IAAI,GACR;IACD,OAAO,EAAE,eAAe,GAAG,SAAS,GAAG,WAAW,CAAC;IACnD,UAAU,EAAE,MAAM,CAAC;IACnB;;;;;OAKG;IACH,SAAS,CAAC,EAAE,cAAc,GAAG,qBAAqB,CAAC;IACnD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAyGA;AAwoBD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,YAAY,CAC1B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,GACzB,eAAe,CA2GjB"}
+{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;AAmBH,OAAO,EAWL,KAAK,gBAAgB,EAErB,KAAK,SAAS,EACf,MAAM,YAAY,CAAC;AAEpB,OAAO,EACL,KAAK,WAAW,EAQjB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,KAAK,OAAO,EAAa,MAAM,WAAW,CAAC;AACpD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,EAUL,KAAK,WAAW,EACjB,MAAM,YAAY,CAAC;AAkBpB,OAAO,EAAE,KAAK,QAAQ,EAA2D,MAAM,YAAY,CAAC;AACpG,OAAO,KAAK,EAEV,UAAU,EAEV,oBAAoB,EAKpB,WAAW,EAGZ,MAAM,SAAS,CAAC;AAYjB,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAE1C,8DAA8D;AAC9D,MAAM,WAAW,iBAAiB;IAChC,KAAK,EAAE,UAAU,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,SAAS,EAAE,SAAS,CAAC;IACrB,GAAG,EAAE,MAAM,IAAI,CAAC;IAChB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,0FAA0F;IAC1F,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sGAAsG;IACtG,QAAQ,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,WAAW,CAAC;IAC5C;;;;;;;;;;;;;;;;;;;OAmBG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,KAAK,oBAAoB,CAAC;IACjF,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,8FAA8F;AAC9F,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,8FAA8F;IAC9F,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AA4tCD;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,KAAK,EAAE,UAAU,EACjB,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,aAAa,EACtB,GAAG,EAAE,IAAI,EACT,MAAM,EAAE,eAAe,GACtB,IAAI,CAyBN;AAqND;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,wBAAgB,iBAAiB,CAC/B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,IAAI,GACR;IACD,OAAO,EAAE,eAAe,GAAG,SAAS,GAAG,WAAW,CAAC;IACnD,UAAU,EAAE,MAAM,CAAC;IACnB;;;;;OAKG;IACH,SAAS,CAAC,EAAE,cAAc,GAAG,qBAAqB,CAAC;IACnD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAyGA;AA8xBD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,YAAY,CAC1B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,GACzB,eAAe,CA+HjB"}

package/dist/batch-dispatch.js

@@ -104,6 +104,7 @@ const core_1 = require("@ai-dossier/core");
 const worktree_pool_1 = require("@ai-dossier/worktree-pool");
 const attribution_1 = require("./attribution");
 const dispatch_1 = require("./dispatch");
+const dispatch_health_1 = require("./dispatch-health");
 const groundtruth_1 = require("./groundtruth");
 const journal_1 = require("./journal");
 const readiness_1 = require("./readiness");
@@ -143,6 +144,55 @@ function sanitizeUntrustedText(value) {
 function journalEvent(deps, event, unitId, extra = {}) {
     deps.journal.append((0, journal_1.unitEvent)(event, unitId, extra), deps.now());
 }
+/**
+ * Read a dead batch dispatch's own log and classify it (#629 AC3) — tail,
+ * member, fix and report all reach a `dead` agent the same way (`pid` no
+ * longer alive), and every one of them previously treated that exit as a
+ * real failure (`tail-agent-exited-unverified`, member eviction,
+ * `report-failed`, a red fix-attempt resolution) with no way to tell a
+ * confirmed provider API error apart from an agent that actually ran.
+ *
+ * `offset` fences this read to THIS dispatch's slice, mirroring the
+ * per-issue path's `dispatchLogSlice`/`log_offset_at_spawn` (#524): these
+ * batch log paths are per-ROLE, not per-dispatch, and opened in append mode
+ * (`dispatch.ts`'s `openLogForAppend`), so a byte-0 read would keep
+ * classifying a LATER dead exit — one that itself wrote no `result` event at
+ * all (killed, OOM, an operator killing the tick — the #629 incident itself)
+ * — against a STALE `result` line from an earlier dispatch, permanently
+ * suppressing genuine failures and re-arming the pause on every reconcile
+ * (#629 review). Each batch spawn function stamps `log_offset_at_spawn` on
+ * its slot patch, exactly like the per-issue spawn path already does.
+ */
+function readDispatchApiError(logFile, offset) {
+    return (0, core_1.parseDispatchApiError)((0, run_log_1.readDispatchLog)(logFile, offset));
+}
+/**
+ * Classify a dead batch dispatch and, if it is a confirmed provider API
+ * error, record it against the shared #505/#629 pause (the same mechanism
+ * `engine.ts`'s `completeUnitOrRecover` uses — see `dispatch-health.ts`) and
+ * release the slot, all inside ONE lock (a crash between recording and
+ * releasing would otherwise leave the counter incremented with the slot
+ * still held by a dead pid — #629 review). Returns whether it classified as
+ * an API error — the caller's cue to skip its normal failure handling
+ * (eviction, `tail-agent-exited-unverified`, `report-failed`, a red
+ * fix-attempt resolution) entirely.
+ *
+ * `release: false` (the fix-agent caller) records without releasing — that
+ * caller has its own release a few lines later, after also deciding whether
+ * to resolve the fix attempt.
+ */
+function handleDeadDispatchApiError(deps, batchId, logFile, offset, now, options = {}) {
+    const apiError = readDispatchApiError(logFile, offset);
+    if (!apiError)
+        return false;
+    deps.store.withLock((s) => {
+        let n = (0, dispatch_health_1.recordDispatchApiError)((event, unitId, extra) => journalEvent(deps, event, unitId, extra), s, unit(batchId), apiError);
+        if (options.release ?? true)
+            n = releaseSlot(n, batchId, now);
+        return { state: n, result: undefined };
+    });
+    return true;
+}
 /**
  * Last ~500 bytes of a gate's output tail, falling back to the envelope's
  * `reason` when no subprocess ran (#583 AC1/AC3 review: `capability-unavailable`
@@ -527,6 +577,11 @@ function spawnMember(deps, dispatch, state, slot, batchId, now, result) {
     const cmd = spawnSpec.cmd;
     const prompt = (0, dispatch_1.buildMemberPrompt)(dispatch.memberPrompt, memberIssue, batchId, batch.worktree);
     const logFile = (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue);
+    // #629: captured BEFORE spawning, mirroring `engine.ts`'s own
+    // `spawnAndRecord` — the log is per-role and append-mode, so the size at
+    // this instant is exactly where THIS dispatch's own output starts. Fences
+    // `handleDeadDispatchApiError`'s classification to this dispatch's slice.
+    const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
     let pid;
     try {
         pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -549,6 +604,7 @@ function spawnMember(deps, dispatch, state, slot, batchId, now, result) {
         // (line ~745) — `spawnMember` never did, so `recordMemberRunLog`'s
         // `slot.spawned_at === null` guard silently skipped every member.
         spawned_at: now.toISOString(),
+        log_offset_at_spawn: logOffset,
     };
     const next = slot.status === 'assigned' || slot.status === 'recovering'
         ? (0, state_1.transitionSlot)(withStatus, slot.id, 'running', patch, now)
@@ -709,6 +765,9 @@ function spawnTailAgent(deps, config, dispatch, batchId, now, result) {
         const cmd = spawnSpec.cmd;
         const prompt = (0, dispatch_1.buildBatchTailPrompt)(dispatch.batchTailPrompt, batchId, batch.anchor, batch.members, batch.worktree);
         const logFile = (0, dispatch_1.batchTailLogPath)(deps.store.runsDir, batchId);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -726,6 +785,7 @@ function spawnTailAgent(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'reviewing',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned' ? (0, state_1.transitionSlot)(state, slot.id, 'running', patch, now) : state;
         deps.journal.append((0, journal_1.unitEvent)('spawned', unit(batchId), {
@@ -759,6 +819,9 @@ function spawnReportAgent(deps, config, dispatch, batchId, now, result) {
         const cmd = spawnSpec.cmd;
         const prompt = (0, dispatch_1.buildBatchReportPrompt)(dispatch.batchReportPrompt, batchId, batch.anchor, prNumber);
         const logFile = (0, dispatch_1.batchReportLogPath)(deps.store.runsDir, batchId);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -772,6 +835,7 @@ function spawnReportAgent(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'report',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned'
             ? (0, state_1.transitionSlot)(state, slot.id, 'running', patchState, now)
@@ -891,6 +955,9 @@ function runValidate(deps, config, dispatch, batchId, now, result) {
     }
     claimAndSpawn(deps, config, batchId, 'fixing', now, (s, slot) => {
         const logFile = (0, dispatch_1.batchFixLogPath)(deps.store.runsDir, batchId, offender);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(fixDispatch.command, fixDispatch.prompt, logFile);
@@ -913,6 +980,7 @@ function runValidate(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'fixing',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned' ? (0, state_1.transitionSlot)(s, slot.id, 'running', patch, now) : s;
         result.spawned.push(unit(batchId));
@@ -1103,6 +1171,14 @@ function runIncrementalGate(deps, config, dispatch, batchId, batch, memberIssue,
         // Journalled per member: a gate that silently does not run is its own
         // trap (#594's shape — absence reading as a verdict). Silence must never
         // be mistaken for a pass.
+        //
+        // #632: confirmed this fires once per member, not once per tick.
+        // `runIncrementalGate` is called only from the `isMemberComplete` branch
+        // of `reconcileMemberSlot`, and both of that branch's callees
+        // (`completeMemberGate`, `evictMemberAndContinue`) advance
+        // `batch.executing_member` before returning — so the very next tick
+        // reads a DIFFERENT `memberIssue` from `batch.members[executing_member -
+        // 1]`, and this gate never runs twice against the same member.
         journalEvent(deps, 'gate-skipped', unit(batchId), {
             issue: memberIssue,
             reason: `gate-skipped:${skipped.id}`,
@@ -1180,7 +1256,13 @@ function completeMemberGate(deps, config, dispatch, batchId, batch, memberIssue,
     // invariant exists to prevent).
     const ranges = (0, attribution_1.memberRanges)(boundaryCommits(deps, batch));
     deps.store.withLock((s) => {
-        let n = releaseSlot(s, batchId, now);
+        // #629: a verified member completion is proof dispatch is healthy —
+        // reset the confirmed-failure streak, mirroring the per-issue path's
+        // `resetDispatchApiErrorStreak` on its own verified-complete branch, or a
+        // healthy member sandwiched between two unrelated api-error hits (this
+        // one and a later one on the tail/report agent) would be invisible to
+        // the streak and could still tip it into a false-positive pause.
+        let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
         n = (0, state_1.patchBatch)(n, batchId, { ranges }, now);
         n = advanceMemberToValidated(n, memberIssue, now);
         return { state: n, result: undefined };
@@ -1316,7 +1398,15 @@ function recordMemberRunLog(deps, dispatch, state, batchId, batch, memberIssue,
     const tier = (0, state_1.findEntry)(state, memberIssue)?.tier ?? 'mid';
     const { cmd, model } = (0, dispatch_1.resolveTierSpawn)(dispatch, tier, memberIssue);
     const logFile = (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue);
-    const logContent = (0, run_log_1.readDispatchLog)(logFile, 0);
+    // #629: fenced to THIS dispatch's slice (`log_offset_at_spawn`, stamped by
+    // `spawnMember`/`spawnMemberContinuation` at spawn time) rather than a
+    // fixed byte-0 read. Before #629 a member log never outlived one attempt
+    // (eviction was the only exit besides completion), so byte-0 was safe; a
+    // confirmed-API-error hold now retries the SAME member in place, appending
+    // a SECOND attempt to the same file — an unfenced read would double-count
+    // the first attempt's tokens into the second's `runs.jsonl` entry, exactly
+    // the #524 divergence this telemetry system exists to prevent.
+    const logContent = (0, run_log_1.readDispatchLog)(logFile, slot.log_offset_at_spawn ?? 0);
     const runEntry = (0, run_log_1.buildSchedRunLogEntry)({
         unit: `issue:${memberIssue}`,
         role: 'batch-member',
@@ -1461,6 +1551,16 @@ function reconcileMemberSlot(deps, config, dispatch, batchId, slot, now, result)
         return;
     }
     const blockedNow = (0, groundtruth_1.isMemberBlocked)(milestone, slot.spawned_at);
+    if (dead &&
+        !blockedNow &&
+        handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue), slot.log_offset_at_spawn ?? 0, now)) {
+        // #629: a confirmed provider API error is not a real member failure —
+        // evicting the member for an account-wide spend wall would silently
+        // throw away real, correct work. `runBatchTick`'s "same wedge" retry
+        // (`spawnMemberContinuation`, now pause-gated) retries the SAME member in
+        // place, keeping its position.
+        return;
+    }
     if (blockedNow || dead) {
         // #605: only a milestone THIS dispatch posted may name the reason. A dead
         // agent whose issue carries only a stale `blocked` milestone is an
@@ -1630,6 +1730,19 @@ function reconcileFixSlot(deps, config, batchId, slot, now, result) {
     const offenderRecord = [...batch.fix_attempts].reverse().find((a) => a.outcome === 'dispatched');
     if (!offenderRecord)
         return;
+    // #629: a confirmed provider API error means the fix agent never got a
+    // real chance to fix anything — record it against the shared pause (AC1/
+    // AC2/AC3) so an operator reads "spend limit", not a fix that failed on
+    // its merits. `release: false`: this function releases the slot itself a
+    // few lines below, after also deciding whether to resolve the fix attempt
+    // — one release, not two. The one-shot fix-attempt model has no existing
+    // "retry the same attempt" primitive (unlike tail/member/report, which the
+    // tick loop itself retries), so the suite still runs and resolves the
+    // attempt below exactly as today: the member's one fix attempt is consumed
+    // and resolves per the suite's real result. An operator who wants it back
+    // must re-trigger the fix after `sched resume` — see
+    // packages/sched/README.md's dispatch-health section.
+    handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchFixLogPath)(deps.store.runsDir, batchId, offenderRecord.issue), slot.log_offset_at_spawn ?? 0, now, { release: false });
     deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
     const suite = safeSuite(deps, batchId, batch.worktree);
     const rDeps = recoveryDeps(deps, config, batch, now);
@@ -1672,7 +1785,10 @@ function reconcileTailSlot(deps, batchId, slot, now, result) {
         const pr = (0, groundtruth_1.prOfMilestone)(milestone);
         journalEvent(deps, 'pr-parked', unit(batchId), { pr: pr ?? undefined });
         deps.store.withLock((s) => {
-            let n = releaseSlot(s, batchId, now);
+            // #629: a verified park is proof dispatch is healthy — reset the
+            // confirmed-failure streak, mirroring the per-issue path's own parked
+            // branch.
+            let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
             const b = (0, state_1.findBatch)(n, batchId);
             if (!b)
                 return { state: n, result: undefined };
@@ -1684,6 +1800,13 @@ function reconcileTailSlot(deps, batchId, slot, now, result) {
         return;
     }
     if (dead) {
+        // #629: a confirmed provider API error is not a real tail-agent failure —
+        // record it against the shared pause and let `runBatchTick`'s "same
+        // wedge" retry (`spawnTailAgent`, now pause-gated) respawn it, instead of
+        // journaling a failure that will never stop respawning on its own.
+        if (handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchTailLogPath)(deps.store.runsDir, batchId), slot.log_offset_at_spawn ?? 0, now)) {
+            return;
+        }
         deps.journal.append((0, journal_1.unitEvent)('unit-failed', unit(batchId), { reason: 'tail-agent-exited-unverified' }), now);
         deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
         result.failed.push(unit(batchId));
@@ -1701,7 +1824,10 @@ function reconcileReportSlot(deps, batchId, slot, now, result) {
     if ((0, groundtruth_1.isBatchPhaseDone)(milestone, 'batch-report', slot.spawned_at)) {
         deps.journal.append((0, journal_1.unitEvent)('external-advance', unit(batchId), { detail: 'batch report done' }), now);
         deps.store.withLock((s) => {
-            let n = releaseSlot(s, batchId, now);
+            // #629: a verified report completion is proof dispatch is healthy —
+            // reset the confirmed-failure streak, mirroring the per-issue path's
+            // own verified-complete branch.
+            let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
             const b = (0, state_1.findBatch)(n, batchId);
             if (!b || b.status !== 'deployed')
                 return { state: n, result: undefined };
@@ -1714,16 +1840,46 @@ function reconcileReportSlot(deps, batchId, slot, now, result) {
         return;
     }
     if (dead) {
+        // #629: same reasoning as `reconcileTailSlot` — a confirmed API error is
+        // not a real report-agent failure; record it and let the `deployed`
+        // branch's `spawnReportAgent` retry (now pause-gated) instead of
+        // journaling `report-failed` for a wall that will keep respawning anyway.
+        if (handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchReportLogPath)(deps.store.runsDir, batchId), slot.log_offset_at_spawn ?? 0, now)) {
+            return;
+        }
         deps.journal.append((0, journal_1.unitEvent)('report-failed', unit(batchId), { detail: 'unverified exit' }), now);
         deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
     }
 }
 // --- PR watch for `awaiting-merge` batches (mirrors `pollParkedPrs`/`reconcileParked`) ---
+/**
+ * Reset a batch's `pr-watch-failed` dedup marker (#630/#633). A no-op when no
+ * streak is recorded, so it is safe to call on every non-`awaiting-merge`
+ * batch every tick.
+ */
+function clearPrWatchFailed(deps, batch, now) {
+    if (batch.pr_watch_failed_reason === null)
+        return;
+    deps.store.withLock((s) => ({
+        state: (0, state_1.patchBatch)(s, batch.id, state_1.CLEARED_PR_WATCH_FIELDS, now, false),
+        result: undefined,
+    }));
+}
 function reconcilePrWatch(deps, now, result) {
     const state = deps.store.load();
     for (const batch of state.batches) {
-        if (batch.status !== 'awaiting-merge')
+        if (batch.status !== 'awaiting-merge') {
+            // #633: the marker is scoped to ONE awaiting-merge stretch. Merged,
+            // blocked and dissolved batches all leave through this guard, and
+            // `BATCH_TRANSITIONS` lets a batch come back to `awaiting-merge` on the
+            // same id (`awaiting-merge → rebasing → re-validating → shipping →
+            // awaiting-merge`, `recovery.ts`'s `handlePrConflict`). Leaving a live
+            // reason behind would make the SAME reason recurring after a real
+            // rebase-and-reship read as an unchanged streak and stay silent —
+            // contradicting `BatchEntry.pr_watch_failed_reason`'s own contract.
+            clearPrWatchFailed(deps, batch, now);
             continue;
+        }
         const pr = batch.pr;
         if (pr === null)
             continue;
@@ -1756,14 +1912,52 @@ function reconcilePrWatch(deps, now, result) {
             continue;
         }
         if (truth.blocked || truth.mergeable === 'CONFLICTING') {
-            journalEvent(deps, 'pr-watch-failed', unit(batch.id), {
-                reason: truth.blocked ? 'auto-merge-blocked' : 'pr-conflicting',
-                pr,
-            });
+            const reason = truth.blocked ? 'auto-merge-blocked' : 'pr-conflicting';
+            const isNewStreak = batch.pr_watch_failed_reason !== reason;
+            const ticks = isNewStreak ? 1 : batch.pr_watch_failed_ticks + 1;
+            const since = isNewStreak ? now.toISOString() : batch.pr_watch_failed_since;
+            // #630: journal on the streak's first tick, then again only every
+            // `JOURNAL_DEDUP_REANNOUNCE_TICKS` — never every tick — mirrors #610's
+            // `stale_milestone_ignored_for` dedup for the "distinct condition"
+            // half, while still giving a still-blocked streak a later line an
+            // operator can read "still blocked, Nth check" off (a single onset
+            // entry never updates, so "after 40 minutes" would otherwise never be
+            // legible from any journal line at all). `at` is the engine's OWN
+            // decision time, never copied from `truth` (which carries no timestamp
+            // of its own); `since` is the streak's onset, and is what makes the
+            // duration legible — `ticks_persisted` is a tick count against an
+            // operator-tunable `reconcile_interval_ms`, not a fixed wall-clock.
+            if (isNewStreak || ticks % types_1.JOURNAL_DEDUP_REANNOUNCE_TICKS === 0) {
+                journalEvent(deps, 'pr-watch-failed', unit(batch.id), {
+                    reason,
+                    pr,
+                    at: now.toISOString(),
+                    since,
+                    ticks_persisted: ticks,
+                });
+            }
+            deps.store.withLock((s) => ({
+                // `touchUpdatedAt: false` — a silent tick that only advances the
+                // streak counter is bookkeeping, not activity (same rule as
+                // `patchEntry`'s dedup writes).
+                state: (0, state_1.patchBatch)(s, batch.id, {
+                    pr_watch_failed_reason: reason,
+                    pr_watch_failed_since: since,
+                    pr_watch_failed_ticks: ticks,
+                }, now, false),
+                result: undefined,
+            }));
             // #472's own rebase-and-reship path (RFC F.9) is a documented follow-up
             // for the batch PR-conflict rail; for now the batch stays parked and
             // the block is visible via the journal + `sched status`.
         }
+        else {
+            // #630: the condition cleared — reset the marker so a future
+            // re-occurrence (even the SAME reason) journals its own fresh entry.
+            // This is de-duplication, not suppression: only an UNCHANGED streak
+            // stays silent.
+            clearPrWatchFailed(deps, batch, now);
+        }
     }
 }
 /**
@@ -1825,10 +2019,23 @@ function runBatchTick(deps, config, dispatch) {
     // only the per-batch `slotFor`/`status` re-check inside the loop reads
     // fresh state (one `store.load()` per iteration, in case an earlier claim
     // in this same pass changed things).
-    const readyOrder = [...deps.store.load().batches]
-        .filter((b) => b.status === 'ready')
-        .sort((a, b) => (0, readiness_1.compareByPriority)((0, readiness_1.batchRank)(a), (0, readiness_1.batchRank)(b)))
-        .map((b) => b.id);
+    // #629: dispatch-health pause (#505, and now the confirmed-api-error
+    // streak) stops NEW per-issue assignments via `computeAssignments` — but
+    // nothing in this function ever read `paused` before #629, so claiming a
+    // `ready` batch here (a provider dispatch, `batch-setup`, the most literal
+    // "new assignment" there is) or a respawn wedge in the second loop below
+    // would keep happening every tick regardless, making the pause cosmetic
+    // for the exact incident (a batch tail respawning against a spend wall)
+    // that motivated it. Read once — every use below in this same tick must
+    // agree, and re-reading per iteration only risks a pause landing mid-loop
+    // and gating some batches but not others in one pass.
+    const paused = deps.store.load().paused;
+    const readyOrder = paused
+        ? []
+        : [...deps.store.load().batches]
+            .filter((b) => b.status === 'ready')
+            .sort((a, b) => (0, readiness_1.compareByPriority)((0, readiness_1.batchRank)(a), (0, readiness_1.batchRank)(b)))
+            .map((b) => b.id);
     for (const batchId of readyOrder) {
         const state = deps.store.load();
         const batch = (0, state_1.findBatch)(state, batchId);
@@ -1877,9 +2084,16 @@ function runBatchTick(deps, config, dispatch) {
         if (slot)
             continue; // live but neither running/assigned (e.g. mid-verify) — next tick
         if (batch.status === 'validating') {
+            // A local suite run, not a provider dispatch — unaffected by `paused`.
             runValidate(deps, config, dispatch, batch.id, now, result);
+            continue;
         }
-        else if (batch.status === 'deployed') {
+        // #629: every branch below spawns a provider agent — hold the wedge
+        // until `sched resume` (see the top of this function for why `paused`
+        // is read once, up front, rather than per iteration).
+        if (paused)
+            continue;
+        if (batch.status === 'deployed') {
             spawnReportAgent(deps, config, dispatch, batch.id, now, result);
         }
         else if (batch.status === 'executing') {