code-coordinator 0.5.46__tar.gz → 0.5.48__tar.gz

{code_coordinator-0.5.46 → code_coordinator-0.5.48}/CLAUDE.md

@@ -82,7 +82,7 @@ coord merge [--dry-run] [--repo NAME] [--method rebase|squash|merge] [--order ID
 coord reconcile-merges     # Backfill missing branches + record out-of-band merges (#609/#611)
 coord retry|stop|resume <id>                    # Recovery; `coord done` ends the session
 ```
-Setup / diagnostics (discoverable via `--help`): `coord init`, `coord config`, `coord agent`, `coord serve`, `coord web`, `coord diagnose`, `coord sessions [--remote]`, `coord split`.
+Setup / diagnostics (discoverable via `--help`): `coord init`, `coord config`, `coord agent`, `coord serve`, `coord web`, `coord diagnose`, `coord sessions [--remote]`, `coord split`, `coord notifier` ([`docs/NOTIFIER.md`](docs/NOTIFIER.md)).
 
 ## Development
 
@@ -118,6 +118,7 @@ coord assign --dry-run precision claude-coordinator 42
 - **Test precedes Review — the pipeline order is `Work → Test → Review → Merge`.** The smoke test runs *before* the PR/review (the natural order: smoke before PR), reversing the #520 "get the review over with first" workaround now that the agent-assisted Testing stage is smooth. This is enforced in two places that must stay in sync: (1) the **displayed** stage order comes from `pipeline.default_gates = ["test","review","merge"]` (`coord/config.py`; an old DB carrying the #520-era `["review","test","merge"]` is migrated in `coord/db.py`); (2) the **headless** auto-loop holds review dispatch until the work has a `passed`/`skipped` test verdict whenever `default_gates` orders test before review — `PipelineConfig.test_precedes_review()` drives `dispatch_pending_reviews` (`coord/review.py`). The explicit `coord review`/`coord pr` paths stay ungated so a human can always force a review. (The merge gate already required a test verdict — `requires_smoke` — so the human test touchpoint just moved earlier, and review cycles are no longer burned on untested code.)
 - **Interactive testing + merge agents drive the Test → Review → Merge handoff (leg 3c / A3, #350/#581/#306/#606).** From a Pipeline row's right-click menu, the TUI routes **board-driven** verdicts (never TTY-scraped, ToS §3.7) along `Work → Test → Review → Merge`: **Start testing** = `coord assign --interactive --smoke-of <work_aid>` (read-only testing agent in the live checkout; records the verdict via `coord test --passed|--fail`); a `passed`/`skipped` test → **pass→review** (launches the interactive review); an approved review → **Start merge** = `--merge-of <work_aid>` (worktrees + proactively rebases onto the default branch #306, resolves mechanical conflicts (semantic with the operator), runs tests, `git push --force-with-lease`, then `coord verify-merge` and — if clean — `coord merge` itself to complete it #606); a `failed` test **or** request-changes review → one-key **`--fix-of`** on the same branch (a test-fail takes the identical action as a request-changes; `--fix-of` accepts a review id **or** a test-failed work id, the #581 front door). The merge agent is still gated on CI/review/smoke (a gate failure is reported, not forced) and `verify-merge` (#604) runs first. Full walkthrough: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
 - **Progress streaming from workers.** Workers emit `STATUS:` and `STUCK:` lines in their logs. The coordinator parses these for real-time progress reporting in `coord status` and the dashboard.
+- **The notifier tells you when NOBODY IS COMING — and nothing else (#1632).** `coord/notifier/` pushes to a self-hosted ntfy on the daemon host (over Tailscale, so no event text leaves the tailnet) when the pipeline **has stopped or is stalled and will not advance without a human**: a halted drive, a gate parked `HUMAN_REQUIRED`, a worker's `STUCK:`, a stall that survived `drive`'s nudge (#1593), a leg running far past comparable work, a fleet CRIT that invalidates in-flight work. It is **not** an error channel and **not** a progress feed — a failed test, a request-changes review and a mechanical merge conflict are all things the auto-loop handles, and pushing them is what trains an operator to mute the channel; in normal operation this fires approximately never. "Far too long" is **learned, never a fixed timeout**: p90 of a `(repo, type, tier)`-stratified population built from the durations milestone #37 already records, with a cold-start state (<5 samples → a generous absolute ceiling, said out loud in the message) so it never fires off a population of one. Quiet hours (22:00–08:00, daemon-local, `notifications.quiet_hours`) are a **deferral window, not a filter** — events are held and delivered as one 08:00 digest, nothing is discarded, and **no severity pierces them**; the only exception is `coord drive --urgent`, which is a deadline rather than a severity and expires with the drive. The whole subsystem is **advisory and isolated** (#1485's lesson): `tick()` never raises, every collector source fails open, state lives in its own `~/.coord/notifier.json`, and an unreachable ntfy server cannot affect dispatch, routing, the board or any verdict. It rides #1616's daemon clock rather than shipping a second one, and reads `drive`'s stall decision rather than defining "stalled" again. Off by default. Full rationale: [`docs/NOTIFIER.md`](docs/NOTIFIER.md).
 - **Failure reassignment.** Failed assignments can be retried on a different machine via `coord retry`. With `concurrency.auto_reassign: true`, reconciliation auto-retries on a different machine.
 - **Dependency freshness checks.** Before dispatching, `coord approve` checks whether upstream repos are up-to-date on the target machine. Stale dependencies trigger warnings or auto-pull with `--auto-pull`.
 

{code_coordinator-0.5.46/code_coordinator.egg-info → code_coordinator-0.5.48}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: code-coordinator
-Version: 0.5.46
+Version: 0.5.48
 Summary: Coordinate Claude Code workers across multiple machines and repos.
 Author-email: John Donaghy <johnfdonaghy@gmail.com>
 License-Expression: LicenseRef-FSL-1.1-MIT

{code_coordinator-0.5.46 → code_coordinator-0.5.48/code_coordinator.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: code-coordinator
-Version: 0.5.46
+Version: 0.5.48
 Summary: Coordinate Claude Code workers across multiple machines and repos.
 Author-email: John Donaghy <johnfdonaghy@gmail.com>
 License-Expression: LicenseRef-FSL-1.1-MIT

{code_coordinator-0.5.46 → code_coordinator-0.5.48}/code_coordinator.egg-info/SOURCES.txt

@@ -113,6 +113,7 @@ coord/review.py
 coord/scorecard.py
 coord/serve_app.py
 coord/smoke.py
+coord/spend_ceiling.py
 coord/split_work.py
 coord/stage_projection.py
 coord/state.py
@@ -310,6 +311,16 @@ coord/health/checks/toolchain.py
 coord/health/checks/unit_drift.py
 coord/health/checks/unit_enablement.py
 coord/health/checks/worktrees.py
+coord/notifier/__init__.py
+coord/notifier/baseline.py
+coord/notifier/cli.py
+coord/notifier/collect.py
+coord/notifier/digest.py
+coord/notifier/models.py
+coord/notifier/predicate.py
+coord/notifier/service.py
+coord/notifier/store.py
+coord/notifier/transport.py
 coord/providers/__init__.py
 coord/providers/base.py
 coord/providers/claude.py
@@ -349,6 +360,7 @@ docs/GRAPHIFY_SETUP.md
 docs/ISSUE_GUIDANCE.md
 docs/MAC_MINI.md
 docs/MERGE_AUTO_DRAIN_TRUST_BAR.md
+docs/NOTIFIER.md
 docs/OPENCODE_VERIFICATION.md
 docs/OPERATING_GOTCHAS.md
 docs/ORACLE_LOOP.md
@@ -401,6 +413,7 @@ tests/test_agent_deploy_units_endpoint.py
 tests/test_agent_dirty_worktree.py
 tests/test_agent_graph_self_heal.py
 tests/test_agent_health_h3.py
+tests/test_agent_idle_restart.py
 tests/test_agent_push_failure.py
 tests/test_agent_reap.py
 tests/test_agent_repos_and_pull.py
@@ -568,6 +581,7 @@ tests/test_interactive_briefing_echo.py
 tests/test_interactive_prefill_state.py
 tests/test_interactive_session_relaunch.py
 tests/test_interactive_test_verdict_backstop.py
+tests/test_interactive_token_attribution.py
 tests/test_interactive_worktree.py
 tests/test_issue_store_seam.py
 tests/test_liveness_auditor.py
@@ -590,6 +604,15 @@ tests/test_new_issue_chat.py
 tests/test_next_release_tag.py
 tests/test_no_direct_gh_calls.py
 tests/test_node_shim.py
+tests/test_notifier_baseline.py
+tests/test_notifier_cli.py
+tests/test_notifier_collect.py
+tests/test_notifier_config.py
+tests/test_notifier_drive_seam.py
+tests/test_notifier_isolation.py
+tests/test_notifier_predicate.py
+tests/test_notifier_quiet_hours.py
+tests/test_notifier_service.py
 tests/test_notify.py
 tests/test_notify_drain.py
 tests/test_openapi.py
@@ -652,6 +675,8 @@ tests/test_slice_issue_attribution.py
 tests/test_smoke.py
 tests/test_smoke_live_checkout_restore.py
 tests/test_smoke_workflow.py
+tests/test_spend_ceiling.py
+tests/test_spend_ceiling_coordinator.py
 tests/test_split.py
 tests/test_split_work.py
 tests/test_stage_projection.py
@@ -677,6 +702,7 @@ tests/test_transcript_floor.py
 tests/test_usage.py
 tests/test_usage_limits.py
 tests/test_usage_rollup.py
+tests/test_venv_live_install_2121.py
 tests/test_version_single_source.py
 tests/test_watch.py
 tests/test_worker_events.py

{code_coordinator-0.5.46 → code_coordinator-0.5.48}/code_coordinator.egg-info/scm_file_list.json

@@ -251,6 +251,16 @@
     "coord/models.py",
     "coord/network.py",
     "coord/new_issue_chat.py",
+    "coord/notifier/__init__.py",
+    "coord/notifier/baseline.py",
+    "coord/notifier/cli.py",
+    "coord/notifier/collect.py",
+    "coord/notifier/digest.py",
+    "coord/notifier/models.py",
+    "coord/notifier/predicate.py",
+    "coord/notifier/service.py",
+    "coord/notifier/store.py",
+    "coord/notifier/transport.py",
     "coord/notify.py",
     "coord/openapi.py",
     "coord/overlap_fence.py",
@@ -281,6 +291,7 @@
     "coord/serve_app.py",
     "coord/skills/update-issue/SKILL.md",
     "coord/smoke.py",
+    "coord/spend_ceiling.py",
     "coord/split_work.py",
     "coord/stage_projection.py",
     "coord/state.py",
@@ -327,6 +338,7 @@
     "docs/ISSUE_GUIDANCE.md",
     "docs/MAC_MINI.md",
     "docs/MERGE_AUTO_DRAIN_TRUST_BAR.md",
+    "docs/NOTIFIER.md",
     "docs/OPENCODE_VERIFICATION.md",
     "docs/OPERATING_GOTCHAS.md",
     "docs/ORACLE_LOOP.md",
@@ -431,6 +443,7 @@
     "tests/test_agent_dirty_worktree.py",
     "tests/test_agent_graph_self_heal.py",
     "tests/test_agent_health_h3.py",
+    "tests/test_agent_idle_restart.py",
     "tests/test_agent_push_failure.py",
     "tests/test_agent_reap.py",
     "tests/test_agent_repos_and_pull.py",
@@ -598,6 +611,7 @@
     "tests/test_interactive_prefill_state.py",
     "tests/test_interactive_session_relaunch.py",
     "tests/test_interactive_test_verdict_backstop.py",
+    "tests/test_interactive_token_attribution.py",
     "tests/test_interactive_worktree.py",
     "tests/test_issue_store_seam.py",
     "tests/test_liveness_auditor.py",
@@ -620,6 +634,15 @@
     "tests/test_next_release_tag.py",
     "tests/test_no_direct_gh_calls.py",
     "tests/test_node_shim.py",
+    "tests/test_notifier_baseline.py",
+    "tests/test_notifier_cli.py",
+    "tests/test_notifier_collect.py",
+    "tests/test_notifier_config.py",
+    "tests/test_notifier_drive_seam.py",
+    "tests/test_notifier_isolation.py",
+    "tests/test_notifier_predicate.py",
+    "tests/test_notifier_quiet_hours.py",
+    "tests/test_notifier_service.py",
     "tests/test_notify.py",
     "tests/test_notify_drain.py",
     "tests/test_openapi.py",
@@ -682,6 +705,8 @@
     "tests/test_smoke.py",
     "tests/test_smoke_live_checkout_restore.py",
     "tests/test_smoke_workflow.py",
+    "tests/test_spend_ceiling.py",
+    "tests/test_spend_ceiling_coordinator.py",
     "tests/test_split.py",
     "tests/test_split_work.py",
     "tests/test_stage_projection.py",
@@ -707,6 +732,7 @@
     "tests/test_usage.py",
     "tests/test_usage_limits.py",
     "tests/test_usage_rollup.py",
+    "tests/test_venv_live_install_2121.py",
     "tests/test_version_single_source.py",
     "tests/test_watch.py",
     "tests/test_worker_events.py",

{code_coordinator-0.5.46 → code_coordinator-0.5.48}/code_coordinator.egg-info/scm_version.json

@@ -1,7 +1,7 @@
 {
-  "tag": "0.5.46",
+  "tag": "0.5.48",
   "distance": 0,
-  "node": "g6b550e02d314120327e243cc96c1f09580e49af1",
+  "node": "gac33bdb3df0d6dce42857fa4ae59fa5ca29d9952",
   "dirty": false,
   "branch": "HEAD",
   "node_date": "2026-08-12"

{code_coordinator-0.5.46 → code_coordinator-0.5.48}/coord/agent.py

@@ -235,6 +235,15 @@ _PTY_INJECT_RETRY_BACKOFF_S = 0.4
 _FIRST_OUTPUT_TIMEOUT = 600.0    # seconds of zero output before the watchdog kills
 NO_FIRST_OUTPUT_EXIT = 124       # exit code reported when the TTFT watchdog fires
 
+# #2131: distinct exit code for a leg killed by the per-leg spend ceiling. It
+# must NOT collide with NO_FIRST_OUTPUT_EXIT (or any plausible worker exit
+# code) — `_reap` keys the `spend_ceiling_reason` stamp off it, and that stamp
+# is the ONLY thing that makes a ceiling kill distinguishable from a crash for
+# `coord retry` and the auto-reassign skip. Any non-zero value still lands the
+# assignment on FAILED via the existing reap branch; the value only selects
+# which diagnostic gets attached.
+SPEND_CEILING_EXIT = 125
+
 
 def _append_log_line(log_path: str, line: str) -> None:
     """Best-effort append of a single line to the assignment log. Never raises."""
@@ -344,6 +353,8 @@ def _wait_for_proc_or_result(
     grace_after_result: float = _REAP_GRACE_AFTER_RESULT,
     max_wait: float = _REAP_MAX_WAIT,
     first_output_timeout: float = _FIRST_OUTPUT_TIMEOUT,
+    cost_ceiling_usd: float | None = None,
+    read_cost_usd: "Callable[[], float | None] | None" = None,
     killpg: Callable[[int, int], None] = _killpg_safe,
     log_has_result: Callable[[str], bool] = _log_has_result,
     log_has_output: Callable[[str], bool] = _log_has_output,
@@ -364,12 +375,44 @@ def _wait_for_proc_or_result(
     satisfied permanently — it never re-arms — so a slow-but-emitting (e.g.
     rate-limited) worker is never killed by it. See #299.
 
+    Per-leg spend ceiling (#2131): when ``cost_ceiling_usd`` is a positive
+    number and ``read_cost_usd`` is supplied, the worker's live spend is
+    sampled on every poll.  At :data:`coord.spend_ceiling.WARN_FRACTION` of
+    the ceiling a one-shot ``STATUS:`` warning is written to the worker's own
+    log — so a watching operator gets a chance to intervene, and the kill is
+    legible after the fact.  At or above the ceiling the process group is
+    killed and :data:`SPEND_CEILING_EXIT` is returned so ``_reap`` records a
+    ceiling kill rather than a generic failure.
+
+    Three properties of that check matter and are deliberate:
+
+    * **Fail open.** ``read_cost_usd`` returning ``None`` (unreadable log,
+      not stream-json, nothing priceable yet) never kills anything — killing
+      real work over a parse failure is worse than the overspend.
+    * **Never after the result event.** Once the worker has emitted its final
+      ``result`` the leg is logically finished and the money is already
+      spent; killing then would only mislabel a completed leg as a ceiling
+      kill. So the check is skipped once ``result_seen_at`` is set.
+    * **SIGKILL, not SIGTERM.** The whole point is to stop spending now; a
+      graceful shutdown that keeps talking to the API defeats it. Whatever
+      the worker committed is still on disk, and ``_reap``'s existing
+      safety-net push still runs afterwards.
+
     The keyword-only parameters exist for tests to inject short timeouts and
-    mock kill/clock behavior.
+    mock kill/clock/cost behavior.
     """
     start = clock()
     result_seen_at: float | None = None
     output_seen = False
+    cost_warned = False
+    ceiling_armed = bool(
+        cost_ceiling_usd and cost_ceiling_usd > 0 and read_cost_usd is not None
+    )
+    if ceiling_armed:
+        # Hoisted out of the poll loop — this runs every `poll_interval`.
+        from coord.spend_ceiling import WARN_FRACTION  # noqa: PLC0415
+
+        warn_at = cost_ceiling_usd * WARN_FRACTION
 
     while True:
         try:
@@ -406,6 +449,37 @@ def _wait_for_proc_or_result(
                 "# reap: worker emitted result; awaiting clean exit\n",
             )
 
+        # #2131: per-leg spend ceiling. Only while the leg is still running
+        # (see the docstring for why a post-`result` kill is never useful),
+        # and only ever acting on a value the meter could actually read.
+        if ceiling_armed and result_seen_at is None:
+            try:
+                spent = read_cost_usd()
+            except Exception:  # noqa: BLE001 — fail open, never break the reap
+                spent = None
+            if spent is not None:
+                if not cost_warned and spent >= warn_at:
+                    cost_warned = True
+                    _append_log_line(
+                        log_path,
+                        f"STATUS: spend ${spent:.2f} has passed "
+                        f"{WARN_FRACTION:.0%} of the ${cost_ceiling_usd:.2f} "
+                        "per-leg ceiling → the leg will be killed if it "
+                        "reaches the ceiling → confidence: high (#2131)\n",
+                    )
+                if spent >= cost_ceiling_usd:
+                    _append_log_line(
+                        log_path,
+                        f"# reap: SIGKILL — spend ceiling breached "
+                        f"(${spent:.2f} of ${cost_ceiling_usd:.2f}) (#2131)\n",
+                    )
+                    killpg(proc.pid, signal.SIGKILL)
+                    try:
+                        proc.wait(timeout=5)
+                    except subprocess.TimeoutExpired:
+                        pass
+                    return SPEND_CEILING_EXIT
+
         if result_seen_at is not None and clock() - result_seen_at >= grace_after_result:
             # Worker logically done but process group still alive — force-kill.
             _append_log_line(
@@ -517,6 +591,19 @@ class AssignmentSpec:
     # :meth:`AgentServer._resolve_provider` for the full resolution chain —
     # this field existing does not, on its own, get honoured anywhere else.
     provider_def: "dict[str, Any] | None" = None
+    # #2131: per-leg spend ceiling in USD, resolved coordinator-side from
+    # `budget.ceiling_for(type)` (coord/config.py) and carried here rather
+    # than read from the agent's own config — a config-free agent
+    # (docs/EPHEMERAL_WORKERS.md) has none to read. `None` (the default, and
+    # what every agent sees when the coordinator has no `budget:` block)
+    # means NO CEILING: the reap's watchdog never samples cost and behaves
+    # exactly as it did pre-#2131.
+    #
+    # `coord/dispatch.py` omits the key entirely when there is no ceiling, so
+    # an agent predating this field never sees an unrecognized kwarg (
+    # `AssignmentSpec(**body)` in agent_app.py 400s on those) unless the
+    # operator has actually configured one.
+    cost_ceiling_usd: float | None = None
 
 
 class _GitError(RuntimeError):
@@ -2685,6 +2772,21 @@ class AgentAssignment:
     # instead of leaving it to expand at push time) surfaced only as a line
     # in the worker log that nothing downstream ever read.
     push_failure_reason: str | None = None
+    # #2131: set when the reap's watchdog killed this leg because its live
+    # spend crossed `spec.cost_ceiling_usd`. Formatted with
+    # `coord.spend_ceiling.format_spend_ceiling_reason` (stable "spend
+    # ceiling — " prefix, recognised by `is_spend_ceiling_reason`). Only ever
+    # set alongside `status == FAILED` (the kill returns the non-zero
+    # `SPEND_CEILING_EXIT`), and `None` on every other outcome.
+    #
+    # This field is the WHOLE POINT of #2131's "not a generic failure"
+    # requirement: the coordinator stamps it onto the board's persisted
+    # `failure_reason` (coord/reconcile.py) so `coord retry` can refuse to
+    # silently re-spend, `auto_reassign` can decline to re-dispatch it, and
+    # the escalation record can name what actually happened. Without it a
+    # ceiling kill is indistinguishable from a crash and the money is spent
+    # again on the next pass.
+    spend_ceiling_reason: str | None = None
 
     def to_dict(self) -> dict:
         d = asdict(self)
@@ -4934,6 +5036,18 @@ class AgentServer:
                     prog = None
                 if prog:
                     d["progress"] = prog
+                # #1632: when this worker last SAID anything, as Unix
+                # seconds.  The output-silence probe is the one that catches
+                # failures with no symptom except duration, and only this
+                # machine can see its own log file — the coordinator cannot
+                # stat a remote path.  A single `stat` on a file that is
+                # already open for append, wrapped so a vanished log can
+                # never break `/status`.
+                if a.log_path:
+                    try:
+                        d["last_output_at"] = os.path.getmtime(a.log_path)
+                    except OSError:
+                        pass
                 # Tail-read stream-json log for live summary fields.
                 if a.log_path and is_stream_json(a.log_path):
                     try:
@@ -6823,6 +6937,25 @@ class AgentServer:
         else:
             _log_has_result_fn = _log_has_result
 
+        # #2131: arm the per-leg spend ceiling when the coordinator sent one.
+        # The meter is built here (not inside the wait loop) so the same
+        # instance survives the whole run and can be re-read afterwards for
+        # the reason string — it reads the log incrementally, so a full
+        # re-parse never happens on any poll.
+        _cost_ceiling: float | None = None
+        _cost_meter = None
+        if _reap_start is not None:
+            _raw_ceiling = getattr(_reap_start.spec, "cost_ceiling_usd", None)
+            if isinstance(_raw_ceiling, (int, float)) and _raw_ceiling > 0:
+                try:
+                    from coord.spend_ceiling import LiveCostMeter  # noqa: PLC0415
+
+                    _cost_ceiling = float(_raw_ceiling)
+                    _cost_meter = LiveCostMeter(log_path)
+                except Exception:  # noqa: BLE001 — fail open, never break reap
+                    _cost_ceiling = None
+                    _cost_meter = None
+
         # Use a polling wait that handles claude-cli's well-known habit of
         # not exiting after emitting its final result event (a child of the
         # process group keeps the session alive). See #228.
@@ -6830,9 +6963,31 @@ class AgentServer:
             proc, log_path,
             first_output_timeout=self.first_output_timeout,
             log_has_result=_log_has_result_fn,
+            cost_ceiling_usd=_cost_ceiling,
+            read_cost_usd=_cost_meter.read if _cost_meter is not None else None,
         )
         log_fh.close()
 
+        # #2131: turn the ceiling kill into the stable, greppable diagnostic
+        # every downstream consumer keys off. Uses the meter's LAST observed
+        # value rather than re-reading, so the number in the reason is
+        # exactly the one the kill decision was made on.
+        _spend_ceiling_reason: str | None = None
+        if exit_code == SPEND_CEILING_EXIT and _cost_ceiling is not None:
+            try:
+                from coord.spend_ceiling import (  # noqa: PLC0415
+                    format_spend_ceiling_reason,
+                )
+
+                _observed = (_cost_meter.last if _cost_meter is not None else None)
+                _spend_ceiling_reason = format_spend_ceiling_reason(
+                    _observed if _observed is not None else _cost_ceiling,
+                    _cost_ceiling,
+                    getattr(_reap_start.spec, "type", None) if _reap_start else None,
+                )
+            except Exception:  # noqa: BLE001 — best-effort, never break reap
+                _spend_ceiling_reason = None
+
         # #1461: detect a usage-limit kill from the tail of the transcript.
         # Done HERE — immediately after the worker's own process has exited
         # and BEFORE any coordinator bookkeeping (push attempts, advisory
@@ -7103,6 +7258,13 @@ class AgentServer:
                 # checked independently of which branch set FAILED.
                 if _result_is_error and assignment.status == FAILED:
                     assignment.api_error_reason = _api_error_reason
+                # #2131: only ever attaches to a FAILED transition —
+                # `SPEND_CEILING_EXIT` is non-zero, so the `else` above has
+                # already set FAILED whenever `_spend_ceiling_reason` is set.
+                # Guarded anyway so a `POST /cancel` that raced the kill (and
+                # set CANCELLED before this block ran) is never relabelled.
+                if _spend_ceiling_reason is not None and assignment.status == FAILED:
+                    assignment.spend_ceiling_reason = _spend_ceiling_reason
             self._processes.pop(assignment_id, None)
 
         # #315/#324: parse the log for the worker's claude session_id (from the