@ai-dossier/sched 0.23.5 → 0.25.0

package/README.md

@@ -146,7 +146,7 @@ where every mechanical supervision decision is code, not remembered prose:
    itself — see `slot-released` below (#525).
 6. **Journal (AC6)** — every event (assigned, spawned, exit-detected, external-advance,
    progress, stalled, redispatched, fence-written, fence-failed, unit-failed,
-   dependents-blocked, slot-released, suspect-dispatch, dispatch-unhealthy,
+   dependents-blocked, slot-released, suspect-dispatch, dispatch-unhealthy, dispatch-failure,
    run-log-recorded, run-log-no-usage, run-log-skipped, run-log-failed, engine-stale,
    engine-auto-upgrade-attempted, engine-auto-upgrade-failed, stale-milestone-ignored, …) is
    appended to `events.jsonl`; `sched status` shows the live phase per unit, plus each
@@ -182,6 +182,33 @@ where every mechanical supervision decision is code, not remembered prose:
    a wall that hasn't actually cleared), which also clears the streak so `sched status`'s
    warning doesn't linger against a wall the operator already acted on.
 
+   **Confirmed dispatch failures (#629)** are a SEPARATE, deterministic signal alongside
+   the timing heuristic above: a dispatch result carrying `api_error_status` or
+   `terminal_reason: "api_error"` (`parseDispatchApiError`, `@ai-dossier/core`) is a
+   confirmed provider wall — a 429 spend/rate limit, an auth failure the provider itself
+   rejected — never an agent that ran, journaled **`dispatch-failure`** with the
+   provider's own status/reason/message and reset time (when supplied). Unlike
+   `suspect-dispatch`, a repeat from the SAME unit COUNTS toward its own
+   `consecutive_dispatch_api_errors` counter — no cross-unit correlation is needed when
+   the classification is a parsed field rather than a timing inference, and the incident
+   that motivated this (a batch tail respawning nine times in 33 minutes) was the SAME
+   unit throughout. At the same `DISPATCH_UNHEALTHY_THRESHOLD` it reuses `setPaused`/
+   `dispatch-unhealthy`. The redispatch itself does NOT escalate — same tier,
+   `recoveries` unchanged, `ESCALATION_CAP` never consumed, since a spend wall is not the
+   issue's fault — so once paused, BOTH the per-issue rail (`enterRecovery`/
+   `reconcileRecovering`, held in `recovering` until `sched resume`) and every batch
+   respawn wedge in `runBatchTick` (tail, member continuation, fix, report, and the
+   `ready` → `claimAndSetup` claim — `runValidate`'s local suite run is unaffected) stop
+   respawning into the wall, which previously ignored `paused` entirely. `dispatch-
+   health.ts` is its own module (not `engine.ts`) specifically so it is shared by BOTH
+   dispatch paths without `engine.ts` and `batch-dispatch.ts` importing each other.
+   Batch dispatch logs (per-role, append-mode) are fenced to `log_offset_at_spawn`,
+   stamped by each batch spawn function — a later dispatch that dies with NO result event
+   at all is never misclassified against a stale `api_error` result left in the same log
+   file by an earlier attempt. A verified completion/park resets the streak on the
+   per-issue path; a successful member/tail/report resets it on the batch path; `sched
+   resume` resets it on both.
+
 Config schema moves to 1.4.0 (#527): `dispatch` gains `tiers` — a per-tier
 `{ command?, model?, prompt? }` spawn spec. `command`/`tier_models`/`prompt` remain valid
 as the shorthand and are the fallback for any field a `tiers` entry leaves unset, so a
@@ -261,6 +288,55 @@ This applies to `issue:<n>` unit dispatch (`dispatchAssignments`). `batch:<id>`
 through a separate pass with its own claim/reconcile logic — see
 [Batch dispatch (#523)](#batch-dispatch-523) below.
 
+### Supervised deployment (#679)
+
+Two supported shapes, one rule: **dispatched agents must outlive the tick or engine that
+spawned them.** Agents are spawned detached and unref'd precisely so they survive a sched
+crash or restart (restart reconciles by pid) — any supervisor that tears down the process
+tree on exit defeats that by construction. The symptom when it happens is cruelly
+misleading: agents die with zero usage events, get classified `unverified-exit`, and walk
+up the escalation ladder to `unverified-exit-at-strongest-tier` — a failure reason that
+reads as a model-capability verdict when the strongest tier was never given a chance to
+run.
+
+**Long-running engine** — `Type=simple` user service:
+
+```ini
+# ~/.config/systemd/user/dossier-sched-<project>.service
+[Unit]
+Description=dossier sched engine (<project>)
+
+[Service]
+Type=simple
+ExecStart=%h/.local/bin/ai-dossier sched start --project <project>
+Restart=on-failure
+# Kill ONLY the engine process on stop/restart. The default (control-group)
+# kills every dispatched agent with it — the engine's own restart would then
+# read those deaths as failed runs and escalate the units (see above).
+KillMode=process
+
+[Install]
+WantedBy=default.target
+```
+
+**One-shot tick** — cron or a `Type=oneshot` unit running `sched start --once` needs the
+same rule: the tick exits immediately after dispatching, so under the default
+`KillMode=control-group` the control group is torn down while every freshly spawned agent
+is still starting — EVERY tick kills EVERY agent it dispatched. `KillMode=process` on the
+unit (or spawning each agent into its own transient scope,
+`systemd-run --user --scope --collect …`) is required, not cosmetic.
+
+Verify a deployment survives its own ticks: dispatch one unit, let the tick or engine
+process that spawned it exit, and confirm the agent process is still alive 60 seconds
+later (`ps -p <agent-pid>`), with `run-log-no-usage` absent from the project's
+`events.jsonl` for that unit.
+
+Note the engine's own log (redirected stdout) carries one `✓ [ts] …` line per tick even
+when a tick does nothing (`nothing to do`) — a log that stops growing while
+`systemctl status` still says `active (running)` means the loop died silently, which is
+the failure mode #679 fixed: the engine exited cleanly after its first tick because its
+inter-tick sleep handles were unref'd.
+
 ### Zombie-run fencing (#504)
 
 The ladder redispatches the SAME run, so a takeover inherits the run id and its milestone
@@ -616,6 +692,14 @@ with `touchUpdatedAt: false` — `QueueEntry.updated_at` is load-bearing for
 `isStaleFailedPark`'s 7-day window, `status.ts`'s "parked since" and `readiness.ts`'s
 tiebreak, and a silent dedup tick must not reset them. `patchBatch` takes the same flag
 for the same reason on the batch rail.
+Schema 1.13.0 (#629): `SchedState` gains `consecutive_dispatch_api_errors` (number) and
+`dispatch_pause_reset_at` (string or null) — the confirmed-dispatch-failure streak and
+the provider's own reset time. Unlike the #505 suspect-dispatch pair these are NOT a
+single fact: a provider that reports no reset time leaves `dispatch_pause_reset_at`
+null while the counter is nonzero (`validateState` only enforces the reverse — a reset
+time can never outlive a streak that has already cleared to zero). 1.12.0 states
+migrate on load, backfilling `0`/`null` — no confirmed dispatch failures were ever
+tracked under them, so those values are exact, not a guess.
 
 New journal events: `batch-setup-done`, `batch-setup-failed`, `member-advanced`,
 `batch-warmup-done`, `batch-warmup-failed` (#561 — the cold-path warm step only; a pool
@@ -825,9 +909,10 @@ ladder is exhausted, the terminal `unit-failed` (`agent-exited-unverified` /
 the dead-pid detection rail and the stall kill RECORD a fresh log slice (a `runs.jsonl`
 entry) in the same tick — `recordDispatchRunLog` is once-per-dispatch and refuses to
 append a second entry over the same slice. Since #620 a slot already `exited`/`verifying`
-when reconciled again still ATTRIBUTES one: `readLastToolForSlot` re-parses the same
-static slice for the tool name alone, writing nothing, so it carries no exactly-once
-constraint. That matters because the verify decision lands on a LATER tick than the
+when reconciled again still ATTRIBUTES one: `readDispatchSignalsForSlot` re-parses the
+same static slice (since #629, for the last-tool-call AND API-error-classification
+signals together — see the dispatch-health section above), writing nothing, so it
+carries no exactly-once constraint. That matters because the verify decision lands on a LATER tick than the
 dead-pid detection whenever ground truth was unreachable in between — previously the
 tool name was simply lost on exactly the runs hardest to diagnose. The read is deferred
 behind a thunk so only the tick that reaches the unverified-exit decision pays for it,
@@ -902,6 +987,7 @@ after-the-fact recovery, not a missing-data bug.
 - **Schema**: state/config files from #460 (schema 1.0.0), #464 (1.1.0), #468 (1.2.0),
   #472 (1.3.0), #500 (1.4.0), #505 (1.5.0), #504 (1.6.0), #523 (1.7.0) and #524 (1.8.0)
   load and migrate to 1.14.0 automatically (slot `branch`/`last_head`/`pid_start`, slot `role` (inferred from the
+  load and migrate to 1.13.0 automatically (slot `branch`/`last_head`/`pid_start`, slot `role` (inferred from the
   unit's queue entry, with the persisted `phase` as a fallback — #500), entry
   `pr`/`cleanup`/`failure_evidence`, batch `anchor`/`branch`/`run_id`/`eviction_groups`/
   `evictions`/`fix_attempts`/`rebase_attempts`, state-level `last_pr_poll_at` backfill to
@@ -909,16 +995,22 @@ after-the-fact recovery, not a missing-data bug.
   backfill to `0`/`null` — #505, slot `gen`/`fenced_at` backfill to `0`/`null` — #504, and
   slot `spawned_at`/`log_offset_at_spawn` backfill to `null`/`null` — #524,
   state-level `last_label_poll_at` backfill to `null` — #544, entry `priority` and batch
-  `member_gates`/`blocked_reason` — #565/#583, and slot
-  `stale_milestone_ignored_for` backfill to `null` — #610).
+  `member_gates`/`blocked_reason` — #565/#583, slot
+  `stale_milestone_ignored_for` backfill to `null` — #610, and state-level
+  `consecutive_dispatch_api_errors`/`dispatch_pause_reset_at` backfill to `0`/`null` —
+  #629).
 - **`max_slots`** bounds live units (`assigned | running | recovering`); dependency
   edges gate readiness — an issue with an unmerged dependency, and a batch behind an
   unmerged batch, are never runnable.
-- **Pause** stops new assignments only (including report-agent dispatch — #505); abandon
-  routes through the typed failure rails (`evicted → requeued{full}` for batch members —
-  nothing green is discarded). A pause can be manual (`sched pause`) or automatic
-  (dispatch-health, #505 above); `sched resume` clears both the flag and the
-  dispatch-health streak.
+- **Pause** stops new assignments only (including report-agent dispatch — #505) and,
+  since #629, every batch respawn wedge in `runBatchTick` — tail, member continuation,
+  fix, report, and the `ready` → `claimAndSetup` claim, which previously ignored `paused`
+  entirely (`runValidate` is excepted: a local suite run, not a dispatch) — and an
+  unescalated per-issue redispatch (`enterRecovery`/`reconcileRecovering`, held in
+  `recovering` until resumed). Abandon routes through the typed failure rails
+  (`evicted → requeued{full}` for batch members — nothing green is discarded). A pause
+  can be manual (`sched pause`) or automatic (dispatch-health, #505/#629 above); `sched
+  resume` clears the flag and both dispatch-health streaks.
 
 ## Hard-block labels are re-read every tick (#544)
 

package/dist/batch-dispatch.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;AAmBH,OAAO,EAUL,KAAK,gBAAgB,EAErB,KAAK,SAAS,EACf,MAAM,YAAY,CAAC;AACpB,OAAO,EACL,KAAK,WAAW,EAQjB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,KAAK,OAAO,EAAa,MAAM,WAAW,CAAC;AACpD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,EAUL,KAAK,WAAW,EACjB,MAAM,YAAY,CAAC;AAkBpB,OAAO,EAAE,KAAK,QAAQ,EAA2D,MAAM,YAAY,CAAC;AACpG,OAAO,KAAK,EAEV,UAAU,EAEV,oBAAoB,EAKpB,WAAW,EAGZ,MAAM,SAAS,CAAC;AAYjB,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAE1C,8DAA8D;AAC9D,MAAM,WAAW,iBAAiB;IAChC,KAAK,EAAE,UAAU,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,SAAS,EAAE,SAAS,CAAC;IACrB,GAAG,EAAE,MAAM,IAAI,CAAC;IAChB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,0FAA0F;IAC1F,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sGAAsG;IACtG,QAAQ,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,WAAW,CAAC;IAC5C;;;;;;;;;;;;;;;;;;;OAmBG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,KAAK,oBAAoB,CAAC;IACjF,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,8FAA8F;AAC9F,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,8FAA8F;IAC9F,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AA6oCD;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,KAAK,EAAE,UAAU,EACjB,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,aAAa,EACtB,GAAG,EAAE,IAAI,EACT,MAAM,EAAE,eAAe,GACtB,IAAI,CAyBN;AA+MD;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,wBAAgB,iBAAiB,CAC/B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,IAAI,GACR;IACD,OAAO,EAAE,eAAe,GAAG,SAAS,GAAG,WAAW,CAAC;IACnD,UAAU,EAAE,MAAM,CAAC;IACnB;;;;;OAKG;IACH,SAAS,CAAC,EAAE,cAAc,GAAG,qBAAqB,CAAC;IACnD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAyGA;AA2sBD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,YAAY,CAC1B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,GACzB,eAAe,CA2GjB"}
+{"version":3,"file":"batch-dispatch.d.ts","sourceRoot":"","sources":["../src/batch-dispatch.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6DG;AAmBH,OAAO,EAWL,KAAK,gBAAgB,EAErB,KAAK,SAAS,EACf,MAAM,YAAY,CAAC;AAEpB,OAAO,EACL,KAAK,WAAW,EAQjB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,KAAK,OAAO,EAAa,MAAM,WAAW,CAAC;AACpD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,EAUL,KAAK,WAAW,EACjB,MAAM,YAAY,CAAC;AAkBpB,OAAO,EAAE,KAAK,QAAQ,EAA2D,MAAM,YAAY,CAAC;AACpG,OAAO,KAAK,EAEV,UAAU,EAEV,oBAAoB,EAKpB,WAAW,EAGZ,MAAM,SAAS,CAAC;AAYjB,YAAY,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AAE1C,8DAA8D;AAC9D,MAAM,WAAW,iBAAiB;IAChC,KAAK,EAAE,UAAU,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,SAAS,EAAE,SAAS,CAAC;IACrB,GAAG,EAAE,MAAM,IAAI,CAAC;IAChB,uFAAuF;IACvF,OAAO,EAAE,MAAM,CAAC;IAChB,0FAA0F;IAC1F,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sGAAsG;IACtG,QAAQ,EAAE,CAAC,QAAQ,EAAE,MAAM,KAAK,WAAW,CAAC;IAC5C;;;;;;;;;;;;;;;;;;;OAmBG;IACH,aAAa,CAAC,EAAE,CAAC,QAAQ,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,KAAK,oBAAoB,CAAC;IACjF,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,8FAA8F;AAC9F,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,aAAa,EAAE,MAAM,EAAE,CAAC;IACxB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,8FAA8F;IAC9F,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AA4tCD;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,KAAK,EAAE,UAAU,EACjB,WAAW,EAAE,MAAM,EACnB,OAAO,EAAE,aAAa,EACtB,GAAG,EAAE,IAAI,EACT,MAAM,EAAE,eAAe,GACtB,IAAI,CAyBN;AAqND;;;;;;;;;;;;;;;;;;;;;;;GAuBG;AACH,wBAAgB,iBAAiB,CAC/B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,EAC1B,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,IAAI,GACR;IACD,OAAO,EAAE,eAAe,GAAG,SAAS,GAAG,WAAW,CAAC;IACnD,UAAU,EAAE,MAAM,CAAC;IACnB;;;;;OAKG;IACH,SAAS,CAAC,EAAE,cAAc,GAAG,qBAAqB,CAAC;IACnD,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAyGA;AA8xBD;;;;;;;;;;;;;GAaG;AACH,wBAAgB,YAAY,CAC1B,IAAI,EAAE,iBAAiB,EACvB,MAAM,EAAE,WAAW,EACnB,QAAQ,EAAE,gBAAgB,GACzB,eAAe,CA+HjB"}

package/dist/batch-dispatch.js

@@ -104,6 +104,7 @@ const core_1 = require("@ai-dossier/core");
 const worktree_pool_1 = require("@ai-dossier/worktree-pool");
 const attribution_1 = require("./attribution");
 const dispatch_1 = require("./dispatch");
+const dispatch_health_1 = require("./dispatch-health");
 const groundtruth_1 = require("./groundtruth");
 const journal_1 = require("./journal");
 const readiness_1 = require("./readiness");
@@ -143,6 +144,55 @@ function sanitizeUntrustedText(value) {
 function journalEvent(deps, event, unitId, extra = {}) {
     deps.journal.append((0, journal_1.unitEvent)(event, unitId, extra), deps.now());
 }
+/**
+ * Read a dead batch dispatch's own log and classify it (#629 AC3) — tail,
+ * member, fix and report all reach a `dead` agent the same way (`pid` no
+ * longer alive), and every one of them previously treated that exit as a
+ * real failure (`tail-agent-exited-unverified`, member eviction,
+ * `report-failed`, a red fix-attempt resolution) with no way to tell a
+ * confirmed provider API error apart from an agent that actually ran.
+ *
+ * `offset` fences this read to THIS dispatch's slice, mirroring the
+ * per-issue path's `dispatchLogSlice`/`log_offset_at_spawn` (#524): these
+ * batch log paths are per-ROLE, not per-dispatch, and opened in append mode
+ * (`dispatch.ts`'s `openLogForAppend`), so a byte-0 read would keep
+ * classifying a LATER dead exit — one that itself wrote no `result` event at
+ * all (killed, OOM, an operator killing the tick — the #629 incident itself)
+ * — against a STALE `result` line from an earlier dispatch, permanently
+ * suppressing genuine failures and re-arming the pause on every reconcile
+ * (#629 review). Each batch spawn function stamps `log_offset_at_spawn` on
+ * its slot patch, exactly like the per-issue spawn path already does.
+ */
+function readDispatchApiError(logFile, offset) {
+    return (0, core_1.parseDispatchApiError)((0, run_log_1.readDispatchLog)(logFile, offset));
+}
+/**
+ * Classify a dead batch dispatch and, if it is a confirmed provider API
+ * error, record it against the shared #505/#629 pause (the same mechanism
+ * `engine.ts`'s `completeUnitOrRecover` uses — see `dispatch-health.ts`) and
+ * release the slot, all inside ONE lock (a crash between recording and
+ * releasing would otherwise leave the counter incremented with the slot
+ * still held by a dead pid — #629 review). Returns whether it classified as
+ * an API error — the caller's cue to skip its normal failure handling
+ * (eviction, `tail-agent-exited-unverified`, `report-failed`, a red
+ * fix-attempt resolution) entirely.
+ *
+ * `release: false` (the fix-agent caller) records without releasing — that
+ * caller has its own release a few lines later, after also deciding whether
+ * to resolve the fix attempt.
+ */
+function handleDeadDispatchApiError(deps, batchId, logFile, offset, now, options = {}) {
+    const apiError = readDispatchApiError(logFile, offset);
+    if (!apiError)
+        return false;
+    deps.store.withLock((s) => {
+        let n = (0, dispatch_health_1.recordDispatchApiError)((event, unitId, extra) => journalEvent(deps, event, unitId, extra), s, unit(batchId), apiError);
+        if (options.release ?? true)
+            n = releaseSlot(n, batchId, now);
+        return { state: n, result: undefined };
+    });
+    return true;
+}
 /**
  * Last ~500 bytes of a gate's output tail, falling back to the envelope's
  * `reason` when no subprocess ran (#583 AC1/AC3 review: `capability-unavailable`
@@ -527,6 +577,11 @@ function spawnMember(deps, dispatch, state, slot, batchId, now, result) {
     const cmd = spawnSpec.cmd;
     const prompt = (0, dispatch_1.buildMemberPrompt)(dispatch.memberPrompt, memberIssue, batchId, batch.worktree);
     const logFile = (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue);
+    // #629: captured BEFORE spawning, mirroring `engine.ts`'s own
+    // `spawnAndRecord` — the log is per-role and append-mode, so the size at
+    // this instant is exactly where THIS dispatch's own output starts. Fences
+    // `handleDeadDispatchApiError`'s classification to this dispatch's slice.
+    const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
     let pid;
     try {
         pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -549,6 +604,7 @@ function spawnMember(deps, dispatch, state, slot, batchId, now, result) {
         // (line ~745) — `spawnMember` never did, so `recordMemberRunLog`'s
         // `slot.spawned_at === null` guard silently skipped every member.
         spawned_at: now.toISOString(),
+        log_offset_at_spawn: logOffset,
     };
     const next = slot.status === 'assigned' || slot.status === 'recovering'
         ? (0, state_1.transitionSlot)(withStatus, slot.id, 'running', patch, now)
@@ -709,6 +765,9 @@ function spawnTailAgent(deps, config, dispatch, batchId, now, result) {
         const cmd = spawnSpec.cmd;
         const prompt = (0, dispatch_1.buildBatchTailPrompt)(dispatch.batchTailPrompt, batchId, batch.anchor, batch.members, batch.worktree);
         const logFile = (0, dispatch_1.batchTailLogPath)(deps.store.runsDir, batchId);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -726,6 +785,7 @@ function spawnTailAgent(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'reviewing',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned' ? (0, state_1.transitionSlot)(state, slot.id, 'running', patch, now) : state;
         deps.journal.append((0, journal_1.unitEvent)('spawned', unit(batchId), {
@@ -759,6 +819,9 @@ function spawnReportAgent(deps, config, dispatch, batchId, now, result) {
         const cmd = spawnSpec.cmd;
         const prompt = (0, dispatch_1.buildBatchReportPrompt)(dispatch.batchReportPrompt, batchId, batch.anchor, prNumber);
         const logFile = (0, dispatch_1.batchReportLogPath)(deps.store.runsDir, batchId);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(cmd, prompt, logFile);
@@ -772,6 +835,7 @@ function spawnReportAgent(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'report',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned'
             ? (0, state_1.transitionSlot)(state, slot.id, 'running', patchState, now)
@@ -891,6 +955,9 @@ function runValidate(deps, config, dispatch, batchId, now, result) {
     }
     claimAndSpawn(deps, config, batchId, 'fixing', now, (s, slot) => {
         const logFile = (0, dispatch_1.batchFixLogPath)(deps.store.runsDir, batchId, offender);
+        // #629: fences `handleDeadDispatchApiError`'s classification to this
+        // dispatch's slice — see `spawnMember`'s identical comment.
+        const logOffset = (0, dispatch_1.fileSizeOrZero)(logFile);
         let pid;
         try {
             pid = deps.spawnDeps.spawn(fixDispatch.command, fixDispatch.prompt, logFile);
@@ -913,6 +980,7 @@ function runValidate(deps, config, dispatch, batchId, now, result) {
             pid_start: deps.spawnDeps.processStart(pid),
             phase: 'fixing',
             last_progress_at: now.toISOString(),
+            log_offset_at_spawn: logOffset,
         };
         const next = slot.status === 'assigned' ? (0, state_1.transitionSlot)(s, slot.id, 'running', patch, now) : s;
         result.spawned.push(unit(batchId));
@@ -1188,7 +1256,13 @@ function completeMemberGate(deps, config, dispatch, batchId, batch, memberIssue,
     // invariant exists to prevent).
     const ranges = (0, attribution_1.memberRanges)(boundaryCommits(deps, batch));
     deps.store.withLock((s) => {
-        let n = releaseSlot(s, batchId, now);
+        // #629: a verified member completion is proof dispatch is healthy —
+        // reset the confirmed-failure streak, mirroring the per-issue path's
+        // `resetDispatchApiErrorStreak` on its own verified-complete branch, or a
+        // healthy member sandwiched between two unrelated api-error hits (this
+        // one and a later one on the tail/report agent) would be invisible to
+        // the streak and could still tip it into a false-positive pause.
+        let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
         n = (0, state_1.patchBatch)(n, batchId, { ranges }, now);
         n = advanceMemberToValidated(n, memberIssue, now);
         return { state: n, result: undefined };
@@ -1324,7 +1398,15 @@ function recordMemberRunLog(deps, dispatch, state, batchId, batch, memberIssue,
     const tier = (0, state_1.findEntry)(state, memberIssue)?.tier ?? 'mid';
     const { cmd, model } = (0, dispatch_1.resolveTierSpawn)(dispatch, tier, memberIssue);
     const logFile = (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue);
-    const logContent = (0, run_log_1.readDispatchLog)(logFile, 0);
+    // #629: fenced to THIS dispatch's slice (`log_offset_at_spawn`, stamped by
+    // `spawnMember`/`spawnMemberContinuation` at spawn time) rather than a
+    // fixed byte-0 read. Before #629 a member log never outlived one attempt
+    // (eviction was the only exit besides completion), so byte-0 was safe; a
+    // confirmed-API-error hold now retries the SAME member in place, appending
+    // a SECOND attempt to the same file — an unfenced read would double-count
+    // the first attempt's tokens into the second's `runs.jsonl` entry, exactly
+    // the #524 divergence this telemetry system exists to prevent.
+    const logContent = (0, run_log_1.readDispatchLog)(logFile, slot.log_offset_at_spawn ?? 0);
     const runEntry = (0, run_log_1.buildSchedRunLogEntry)({
         unit: `issue:${memberIssue}`,
         role: 'batch-member',
@@ -1469,6 +1551,16 @@ function reconcileMemberSlot(deps, config, dispatch, batchId, slot, now, result)
         return;
     }
     const blockedNow = (0, groundtruth_1.isMemberBlocked)(milestone, slot.spawned_at);
+    if (dead &&
+        !blockedNow &&
+        handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchMemberLogPath)(deps.store.runsDir, batchId, batch.executing_member, memberIssue), slot.log_offset_at_spawn ?? 0, now)) {
+        // #629: a confirmed provider API error is not a real member failure —
+        // evicting the member for an account-wide spend wall would silently
+        // throw away real, correct work. `runBatchTick`'s "same wedge" retry
+        // (`spawnMemberContinuation`, now pause-gated) retries the SAME member in
+        // place, keeping its position.
+        return;
+    }
     if (blockedNow || dead) {
         // #605: only a milestone THIS dispatch posted may name the reason. A dead
         // agent whose issue carries only a stale `blocked` milestone is an
@@ -1638,6 +1730,19 @@ function reconcileFixSlot(deps, config, batchId, slot, now, result) {
     const offenderRecord = [...batch.fix_attempts].reverse().find((a) => a.outcome === 'dispatched');
     if (!offenderRecord)
         return;
+    // #629: a confirmed provider API error means the fix agent never got a
+    // real chance to fix anything — record it against the shared pause (AC1/
+    // AC2/AC3) so an operator reads "spend limit", not a fix that failed on
+    // its merits. `release: false`: this function releases the slot itself a
+    // few lines below, after also deciding whether to resolve the fix attempt
+    // — one release, not two. The one-shot fix-attempt model has no existing
+    // "retry the same attempt" primitive (unlike tail/member/report, which the
+    // tick loop itself retries), so the suite still runs and resolves the
+    // attempt below exactly as today: the member's one fix attempt is consumed
+    // and resolves per the suite's real result. An operator who wants it back
+    // must re-trigger the fix after `sched resume` — see
+    // packages/sched/README.md's dispatch-health section.
+    handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchFixLogPath)(deps.store.runsDir, batchId, offenderRecord.issue), slot.log_offset_at_spawn ?? 0, now, { release: false });
     deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
     const suite = safeSuite(deps, batchId, batch.worktree);
     const rDeps = recoveryDeps(deps, config, batch, now);
@@ -1680,7 +1785,10 @@ function reconcileTailSlot(deps, batchId, slot, now, result) {
         const pr = (0, groundtruth_1.prOfMilestone)(milestone);
         journalEvent(deps, 'pr-parked', unit(batchId), { pr: pr ?? undefined });
         deps.store.withLock((s) => {
-            let n = releaseSlot(s, batchId, now);
+            // #629: a verified park is proof dispatch is healthy — reset the
+            // confirmed-failure streak, mirroring the per-issue path's own parked
+            // branch.
+            let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
             const b = (0, state_1.findBatch)(n, batchId);
             if (!b)
                 return { state: n, result: undefined };
@@ -1692,6 +1800,13 @@ function reconcileTailSlot(deps, batchId, slot, now, result) {
         return;
     }
     if (dead) {
+        // #629: a confirmed provider API error is not a real tail-agent failure —
+        // record it against the shared pause and let `runBatchTick`'s "same
+        // wedge" retry (`spawnTailAgent`, now pause-gated) respawn it, instead of
+        // journaling a failure that will never stop respawning on its own.
+        if (handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchTailLogPath)(deps.store.runsDir, batchId), slot.log_offset_at_spawn ?? 0, now)) {
+            return;
+        }
         deps.journal.append((0, journal_1.unitEvent)('unit-failed', unit(batchId), { reason: 'tail-agent-exited-unverified' }), now);
         deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
         result.failed.push(unit(batchId));
@@ -1709,7 +1824,10 @@ function reconcileReportSlot(deps, batchId, slot, now, result) {
     if ((0, groundtruth_1.isBatchPhaseDone)(milestone, 'batch-report', slot.spawned_at)) {
         deps.journal.append((0, journal_1.unitEvent)('external-advance', unit(batchId), { detail: 'batch report done' }), now);
         deps.store.withLock((s) => {
-            let n = releaseSlot(s, batchId, now);
+            // #629: a verified report completion is proof dispatch is healthy —
+            // reset the confirmed-failure streak, mirroring the per-issue path's
+            // own verified-complete branch.
+            let n = (0, dispatch_health_1.resetDispatchApiErrorStreak)(releaseSlot(s, batchId, now));
             const b = (0, state_1.findBatch)(n, batchId);
             if (!b || b.status !== 'deployed')
                 return { state: n, result: undefined };
@@ -1722,6 +1840,13 @@ function reconcileReportSlot(deps, batchId, slot, now, result) {
         return;
     }
     if (dead) {
+        // #629: same reasoning as `reconcileTailSlot` — a confirmed API error is
+        // not a real report-agent failure; record it and let the `deployed`
+        // branch's `spawnReportAgent` retry (now pause-gated) instead of
+        // journaling `report-failed` for a wall that will keep respawning anyway.
+        if (handleDeadDispatchApiError(deps, batchId, (0, dispatch_1.batchReportLogPath)(deps.store.runsDir, batchId), slot.log_offset_at_spawn ?? 0, now)) {
+            return;
+        }
         deps.journal.append((0, journal_1.unitEvent)('report-failed', unit(batchId), { detail: 'unverified exit' }), now);
         deps.store.withLock((s) => ({ state: releaseSlot(s, batchId, now), result: undefined }));
     }
@@ -1894,10 +2019,23 @@ function runBatchTick(deps, config, dispatch) {
     // only the per-batch `slotFor`/`status` re-check inside the loop reads
     // fresh state (one `store.load()` per iteration, in case an earlier claim
     // in this same pass changed things).
-    const readyOrder = [...deps.store.load().batches]
-        .filter((b) => b.status === 'ready')
-        .sort((a, b) => (0, readiness_1.compareByPriority)((0, readiness_1.batchRank)(a), (0, readiness_1.batchRank)(b)))
-        .map((b) => b.id);
+    // #629: dispatch-health pause (#505, and now the confirmed-api-error
+    // streak) stops NEW per-issue assignments via `computeAssignments` — but
+    // nothing in this function ever read `paused` before #629, so claiming a
+    // `ready` batch here (a provider dispatch, `batch-setup`, the most literal
+    // "new assignment" there is) or a respawn wedge in the second loop below
+    // would keep happening every tick regardless, making the pause cosmetic
+    // for the exact incident (a batch tail respawning against a spend wall)
+    // that motivated it. Read once — every use below in this same tick must
+    // agree, and re-reading per iteration only risks a pause landing mid-loop
+    // and gating some batches but not others in one pass.
+    const paused = deps.store.load().paused;
+    const readyOrder = paused
+        ? []
+        : [...deps.store.load().batches]
+            .filter((b) => b.status === 'ready')
+            .sort((a, b) => (0, readiness_1.compareByPriority)((0, readiness_1.batchRank)(a), (0, readiness_1.batchRank)(b)))
+            .map((b) => b.id);
     for (const batchId of readyOrder) {
         const state = deps.store.load();
         const batch = (0, state_1.findBatch)(state, batchId);
@@ -1946,9 +2084,16 @@ function runBatchTick(deps, config, dispatch) {
         if (slot)
             continue; // live but neither running/assigned (e.g. mid-verify) — next tick
         if (batch.status === 'validating') {
+            // A local suite run, not a provider dispatch — unaffected by `paused`.
             runValidate(deps, config, dispatch, batch.id, now, result);
+            continue;
         }
-        else if (batch.status === 'deployed') {
+        // #629: every branch below spawns a provider agent — hold the wedge
+        // until `sched resume` (see the top of this function for why `paused`
+        // is read once, up front, rather than per iteration).
+        if (paused)
+            continue;
+        if (batch.status === 'deployed') {
             spawnReportAgent(deps, config, dispatch, batch.id, now, result);
         }
         else if (batch.status === 'executing') {