agent-dispatch 0.17.0__tar.gz → 0.18.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/AGENTS.md +4 -2
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/CHANGELOG.md +60 -1
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/PKG-INFO +23 -9
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/README.md +22 -8
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/pyproject.toml +1 -1
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/__init__.py +1 -1
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/jobs.py +97 -16
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/models.py +2 -3
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/runner.py +118 -63
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/server.py +136 -45
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_jobs.py +98 -1
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_runner.py +281 -64
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_server.py +408 -15
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/.github/dependabot.yml +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/.github/workflows/ci.yml +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/.github/workflows/publish.yml +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/.gitignore +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/LICENSE +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/SECURITY.md +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/agents.example.yaml +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/assets/mascot.png +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/scripts/check_install.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/cache.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/cli.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/concurrency.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/config.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/progress.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/servers.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/src/agent_dispatch/usage.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/__init__.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/conftest.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_cache.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_cli.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_concurrency.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_config.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_models.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_progress.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_servers.py +0 -0
- {agent_dispatch-0.17.0 → agent_dispatch-0.18.0}/tests/test_usage.py +0 -0
|
@@ -48,6 +48,8 @@ The autouse guard in `tests/conftest.py` rejects real Claude process launches wh
|
|
|
48
48
|
- `--session-id` and `--resume` conflict — never pass both to `claude`.
|
|
49
49
|
- Valid permission modes: `default`, `plan`, `bypassPermissions` (`models.py: KNOWN_PERMISSION_MODES`).
|
|
50
50
|
- `JobStore.finish`/`fail` refuse already-terminal jobs (returns `None`) — this closes the race with force-cancel; never "fix" it by overwriting. `mark_running` likewise refuses any job that isn't `pending`, so a stale or duplicate worker can't resurrect a finished one.
|
|
51
|
+
- **A running job's liveness is its file's mtime, kept fresh by a heartbeat, not by progress.** `_run_owned_job` starts a thread that calls `JobStore.touch` every `_JOB_HEARTBEAT_INTERVAL` (60 s), stops it in `finally`, and bounds it at the dispatch timeout + `_JOB_HEARTBEAT_GRACE` so a worker stuck past its own deadline still comes to look abandoned. Progress writes alone left a job silent inside one long tool call (a nested dispatch runs up to two hours) looking abandoned after an hour: `recover_stale` on any starting server failed it and `finish()` then refused the paid result. Tests that fake an abandoned running job must age the file as well as `started_at`.
|
|
52
|
+
- `JobStore.create` checks for an active job with the requested resume session ID while holding the lock used to publish the new record. It skips running jobs that meet the startup recovery check; a `pending` job holds the session until it runs, is cancelled, or passes the sweep bound (24x the running threshold), at which point `create` **fails** it rather than skipping it — a skipped job stays runnable and would resume the session beside its replacement. The conflict error names `dispatch_wait` / `dispatch_cancel` — never `dispatch_session`, which does not check the reservation. `dispatch_status`, `dispatch_wait`, and `fetch_result` without a result share the polling serializer. The stored job retains the full task, context, and result.
|
|
51
53
|
- "Is this group member missing?" has exactly one implementation: `DispatchConfig.unknown_group_members()`. Any new surface that lists or validates membership calls it instead of re-deriving the check.
|
|
52
54
|
- Cancelling a *running* job requires the in-memory registry (server.py): `_owned_jobs` records ownership **before** `mark_running`, `_running_procs` the Popen once spawned — a cancel landing between the two is still honored, and the late process is killed on registration. The job is marked `cancelled` **before** the subprocess is killed. A job record that is merely *unreadable* at spawn is not treated as terminal. Don't persist PIDs to disk (PID reuse after restart could kill an unrelated process).
|
|
53
55
|
- `max_budget_usd` is enforced **by the claude CLI** (`_build_command` passes `--max-budget-usd`): a run stopped at the cap comes back `is_error` with no `result` text, and `_build_error_result` turns it into `error_type="budget"` + `budget_exceeded=True` + a resumable `session_id`. `_apply_budget` is the *secondary*, post-hoc signal for an overshoot that didn't stop the run; it never fails a dispatch.
|
|
@@ -71,10 +73,10 @@ The autouse guard in `tests/conftest.py` rejects real Claude process launches wh
|
|
|
71
73
|
- **`--json` output must be valid JSON on the empty path too.** `stats --json` printed prose when the journal had no records — i.e. on a fresh install, which is exactly when a script first pipes it somewhere.
|
|
72
74
|
- **The usage journal is written by a decorator, not at each return.** `runner._journaled` wraps `dispatch`/`dispatch_stream` because each has a dozen early returns; instrumenting them one by one guarantees the next new return path goes unrecorded. It records the outermost call only (`_use_session_flag` is False on the stream's old-CLI retry, which would otherwise double-count one dispatch). It also swallows exceptions itself even though `usage.record` already does: the result in hand is billed, and the guarantee has to hold at the boundary that owns the damage rather than on another module's promise (`test_a_broken_journal_never_breaks_a_paid_dispatch` removes record()'s guard to prove it).
|
|
73
75
|
- **A journal record is one `O_APPEND` write under `PIPE_BUF`**, never a locked read-modify-write: the CLI and every running server share one file, and this user has 14+ servers alive. Every field is capped so the line cannot approach 4096 bytes. Rotation is an atomic rename at ~2 MB, two generations kept.
|
|
74
|
-
- **`total_cost_usd` is per SESSION on `--resume`** (claude ≥ 2.1.277; verified live on 2.1.322: $0.0798, then $0.0953 after a resume that cost $0.0155). `runner._call_cost` turns it into the call's own cost by subtracting `usage.session_total(session_id)` — the newest journal record for that `sid` (`cum`, or `cost` for a session's first call) — read **before** the spawn, and carries the reported cumulative figure as `cum` in the journal telemetry. Amounts cannot tell cumulative from per-call reporting, so the CLI **version** decides (`_cli_reports_session_totals`: `claude --version` ≥ 2.1.277, memoized per binary+mtime, probe failure = do not adjust; `tests/conftest.py` stubs it to True). It subtracts only when the reported total is a finite number ≥ the prior total within `_COST_EPSILON` (the journal rounds to 6 places — without the tolerance an unchanged total was charged whole again), and not at all when the session is unknown to the journal or `usage_log` is off: those keep the reported figure. `session_total` reads the live generation before `.1` (the other order loses the newest records to a concurrent rotation) and never takes `cost` from a record that has a `cum` key. Known limits, accepted: two processes resuming one session at once can both subtract the same baseline; a costless record in between (timeout) folds its spend into the next call — the sum stays right, the split does not. `_apply_budget` therefore judges the call, not the session.
|
|
76
|
+
- **`total_cost_usd` is per SESSION on `--resume`** (claude ≥ 2.1.277; verified live on 2.1.322: $0.0798, then $0.0953 after a resume that cost $0.0155). `runner._call_cost` turns it into the call's own cost by subtracting `usage.session_total(session_id)` — the newest journal record for that `sid` (`cum`, or `cost` for a session's first call) — read **before** the spawn, and carries the reported cumulative figure as `cum` in the journal telemetry. Amounts cannot tell cumulative from per-call reporting, so the CLI **version** decides (`_cli_reports_session_totals`: `claude --version` ≥ 2.1.277, memoized per binary+mtime, probe failure = do not adjust; `tests/conftest.py` stubs it to True). It subtracts only when the reported total is a finite number ≥ the prior total within `_COST_EPSILON` (the journal rounds to 6 places — without the tolerance an unchanged total was charged whole again), and not at all when the session is unknown to the journal or `usage_log` is off: those keep the reported figure. `session_total` reads the live generation before `.1` (the other order loses the newest records to a concurrent rotation) and never takes `cost` from a record that has a `cum` key. Known limits, accepted: two processes resuming one session at once can both subtract the same baseline; a costless record in between (timeout) folds its spend into the next call — the sum stays right, the split does not. `_apply_budget` therefore judges the call, not the session. The streaming runner also resumes for `dispatch_async`, using the same pre-spawn baseline.
|
|
75
77
|
- **Token usage is journal-only.** `DispatchResult.token_usage` has `exclude=True`: never expose it in tool payloads or job files. `runner._token_usage` extracts `in`/`out`/`cr`/`cw` and the highest-cost `model` from CLI results, including failures; `_journaled` passes it to `usage.record`, which revalidates counts in `[0, 10**12)` and caps model names at 60 characters. Cache hits have no tokens. `stats` reports separate token sums, measured `records`, cache-read share, and model counts; text shows combined input/cache percentages plus output totals and each agent's most frequent model (`+N more` when needed). Old records remain valid, disk counts go through `as_float` plus plausibility bounds, and `_profile` must not grow.
|
|
76
78
|
- **Profiles are memoized on the journal's (path, size, mtime) and read a NARROWER tail than `stats`.** `list_agents`/`inspect_agent` run on the event-loop thread: a full 2 MB journal cost **24.8 ms** per discovery call at the 512 KB report window — 33x the config parse that 0.13.0 exists to have fixed. Now 3.3 ms cold, ~0 warm. Every profile consumer goes through `usage.profiles()`; never add a caller that re-parses per agent.
|
|
77
|
-
- **`usage_limit` is its own error type** (observed live: `You've hit your session limit · resets 6pm`). It is checked
|
|
79
|
+
- **`usage_limit` is its own error type** (observed live: `You've hit your session limit · resets 6pm`). It is checked before the permission patterns. Its hint names the model or period and reset time when present. A period phrase ("daily limit") counts only in Claude's own shapes — followed by "reached", "resets" or "· resets" (checked against code.claude.com/docs/en/errors; "exceeded" is Google's wording) — because the text is whatever the subprocess printed, and a third-party API's limit or a permission error mentioning one must not be answered with "wait for the reset". Text-classified errors attach their advisory through `_classification_hint`.
|
|
78
80
|
- **Server liveness is an advisory lock, not a PID.** `servers.register` holds an exclusive `flock` on `<config dir>/servers/<pid>.json` for the process lifetime; a reader that *takes* that lock owns a dead entry and deletes it. `os.kill(pid, 0)` would believe a recycled PID and would never notice a SIGKILLed server. Never close that fd (re-registering closes the old one first).
|
|
79
81
|
- **Startup trims are opt-in flags, never defaults.** `AgentConfig.skills` (default true; false → `--disable-slash-commands`) and `strict_mcp` (default false; true → `--strict-mcp-config`, plus `--mcp-config <dir>/.mcp.json` when that file exists) come from `runner._startup_trim_flags`, are sent on `--resume` too, and are pruned from YAML at their defaults. `--mcp-config` is variadic in the CLI: keep a flag right after its path. Mutation is CLI-only (`update --no-skills/--skills`, `--strict-mcp/--no-strict-mcp`); `doctor` probes the flags only when some agent uses them. Measured basis (claude 2.1.322): a new task text re-creates ~32k preamble tokens per dispatch; a stable `--append-system-prompt` does not help. A resumed session, by contrast, re-reads its preamble from cache: live, a follow-up via `dispatch_session` cost $0.005–0.017 against $0.089 for a fresh dispatch of the same agent.
|
|
80
82
|
- **The dispatch protocol goes in the system prompt, not the task.** `_build_system_prompt` (runner.py) renders the protocol (non-interactive, time/spend budget, lead with the outcome, trailing `STATUS:` line) plus the agent's `instructions`, and `_build_command` passes it as `--append-system-prompt` — on `--resume` too, since the CLI re-applies an appended prompt on every launch. `_build_prompt` (the `-p` text) is unchanged. Standing instructions and protocol settings are captured separately in `_cache_config_key`. The text always starts with a `##` header line so it can never be read as a flag. `settings.dispatch_protocol: false` turns the protocol off (for CLIs that predate the flag; `doctor` probes `claude --help` for it); per-agent `instructions` still go through when set.
|
|
@@ -7,6 +7,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.18.0] - 2026-11-20
|
|
11
|
+
|
|
12
|
+
The background-continuation round: an async job can now continue a session,
|
|
13
|
+
join a group and hand back a reference instead of the full text, and polling a
|
|
14
|
+
job no longer returns the whole task and context on every call.
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
- `dispatch_async(session_id=...)` continues an existing agent session in the
|
|
18
|
+
background (the streaming runner now passes `--resume`, with the same
|
|
19
|
+
per-call cost accounting as `dispatch_session`). A session holds at most one
|
|
20
|
+
pending or running job: a second `dispatch_async` for it returns `error` plus
|
|
21
|
+
`active_job_id`, and the error names the `dispatch_wait` / `dispatch_cancel`
|
|
22
|
+
calls for that job. The check and the new record are published under the
|
|
23
|
+
cross-process job lock. A running job whose start time and last file update
|
|
24
|
+
are both over an hour old no longer holds the session; a pending job older
|
|
25
|
+
than a day is failed and replaced (the bounds startup recovery already uses).
|
|
26
|
+
Limit: `dispatch_session` does not check this reservation — do not use it on
|
|
27
|
+
a session whose async job is still active.
|
|
28
|
+
- `group=` on `dispatch_async` and `dispatch_session`: membership is validated
|
|
29
|
+
before anything runs, and the group's `shared_context` goes before the
|
|
30
|
+
caller's context — the same string in the job record and in the prompt.
|
|
31
|
+
- `dispatch_async(return_ref=True)`: a successful job's `result` in
|
|
32
|
+
`dispatch_status` / `dispatch_wait` is `{ref, summary, session_id, ...}`
|
|
33
|
+
instead of the full text; `fetch_result(ref)` reads it. Failed jobs keep their
|
|
34
|
+
error and result inline.
|
|
35
|
+
- A `usage_limit` failure also carries its advice in `hint` (it stays in
|
|
36
|
+
`error` too), so a caller can act on it without parsing the error text.
|
|
37
|
+
|
|
38
|
+
### Changed
|
|
39
|
+
- **Polling payload shape.** `dispatch_status`, `dispatch_wait` and
|
|
40
|
+
`fetch_result` on a job without a result return `task_preview` and
|
|
41
|
+
`context_preview` (first 120 characters, one line) instead of the full `task`
|
|
42
|
+
and `context`. The job file and `agent-dispatch job <id> --json` keep the
|
|
43
|
+
complete text. A caller that read `task` / `context` from these tools must
|
|
44
|
+
switch to the previews or to the CLI.
|
|
45
|
+
- The usage-limit hint names the limit Claude reported — a model
|
|
46
|
+
("Fable limit") or a period ("weekly limit") — and its reset time. It no
|
|
47
|
+
longer claims the limit is account-wide: a model limit is not.
|
|
48
|
+
|
|
49
|
+
### Fixed
|
|
50
|
+
- Model and period limits (`You've hit your Fable limit`, `Weekly limit
|
|
51
|
+
reached`, `5-hour limit reached ∙ resets 3pm`) were reported as `cli_error`;
|
|
52
|
+
they are `usage_limit` now. A period phrase counts only in the shapes Claude
|
|
53
|
+
prints (`… limit reached`, `… limit · resets …`, `… session limit resets …`),
|
|
54
|
+
so another service's `daily limit of 5000 requests reached` or `Daily Limit
|
|
55
|
+
Exceeded`, or `Permission denied while changing the daily limit`, is not
|
|
56
|
+
mistaken for it.
|
|
57
|
+
- A worker thread that cannot start fails its job instead of leaving it
|
|
58
|
+
`pending`.
|
|
59
|
+
- A running job whose agent stayed silent for over an hour — one long tool
|
|
60
|
+
call, such as a nested dispatch — looked abandoned: the next server to start
|
|
61
|
+
marked it failed, and the worker's result, already paid for, was then
|
|
62
|
+
refused. The worker now touches its job file once a minute, so liveness no
|
|
63
|
+
longer depends on the agent emitting progress.
|
|
64
|
+
- A task, context or group `shared_context` containing a NUL character raised
|
|
65
|
+
out of `dispatch` / `dispatch_session` / `dispatch_stream` instead of
|
|
66
|
+
returning a `cli_error` result.
|
|
67
|
+
|
|
10
68
|
## [0.17.0] - 2026-11-10
|
|
11
69
|
|
|
12
70
|
The cost round, driven by measurements on the live CLI rather than by reading
|
|
@@ -938,7 +996,8 @@ cache bounding, and stale-job recovery.
|
|
|
938
996
|
- Dependabot for `pip` + `github-actions`, GitHub Actions pinned to
|
|
939
997
|
commit SHAs for supply-chain integrity.
|
|
940
998
|
|
|
941
|
-
[Unreleased]: https://github.com/ginkida/agent-dispatch/compare/v0.
|
|
999
|
+
[Unreleased]: https://github.com/ginkida/agent-dispatch/compare/v0.18.0...HEAD
|
|
1000
|
+
[0.18.0]: https://github.com/ginkida/agent-dispatch/compare/v0.17.0...v0.18.0
|
|
942
1001
|
[0.17.0]: https://github.com/ginkida/agent-dispatch/compare/v0.16.0...v0.17.0
|
|
943
1002
|
[0.16.0]: https://github.com/ginkida/agent-dispatch/compare/v0.15.1...v0.16.0
|
|
944
1003
|
[0.15.1]: https://github.com/ginkida/agent-dispatch/compare/v0.15.0...v0.15.1
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: agent-dispatch
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.18.0
|
|
4
4
|
Summary: MCP server that lets Claude Code agents delegate tasks to agents in other project directories
|
|
5
5
|
Project-URL: Homepage, https://github.com/ginkida/agent-dispatch
|
|
6
6
|
Project-URL: Repository, https://github.com/ginkida/agent-dispatch
|
|
@@ -179,7 +179,7 @@ Members reference agents by name; membership is many-to-many (a shared gateway c
|
|
|
179
179
|
|
|
180
180
|
**`inspect_group(name)`** — one group's full brief: `description`, the complete `shared_context`, and the member list. For a deep dive on a specific member, call `inspect_agent(member)` — `inspect_group` deliberately stays a cheap membership readout.
|
|
181
181
|
|
|
182
|
-
|
|
182
|
+
Pass `group=` to `dispatch`, `dispatch_session`, `dispatch_async`, or an item in `dispatch_parallel`. The agent must be a member of the group. The group’s `shared_context` goes before the caller’s context.
|
|
183
183
|
|
|
184
184
|
```python
|
|
185
185
|
# From the shop-web codebase, hand the deploy to the infra gateway.
|
|
@@ -246,7 +246,7 @@ dispatch(
|
|
|
246
246
|
}
|
|
247
247
|
```
|
|
248
248
|
|
|
249
|
-
**`error_type` values:** `permission` (tool/action denied), `timeout`, `recursion` (dispatch depth exceeded), `not_found` (missing directory or CLI), `budget` (the `claude` CLI stopped the session at `max_budget_usd`), `usage_limit` (
|
|
249
|
+
**`error_type` values:** `permission` (tool/action denied), `timeout`, `recursion` (dispatch depth exceeded), `not_found` (missing directory or CLI), `budget` (the `claude` CLI stopped the session at `max_budget_usd`), `usage_limit` (Claude reached a usage limit), `cli_error` (other failures). Permission, budget, usage-limit and timeout errors include an actionable hint.
|
|
250
250
|
|
|
251
251
|
**Resumable timeouts:** every fresh dispatch pre-assigns a session UUID (`--session-id`), so a timed-out dispatch still returns a `session_id` — the partial transcript survives the kill. The timeout error spells out the recovery: resume with `dispatch_session(agent, "Continue where you left off", session_id=...)`, retry with a bigger `timeout_seconds`, or use `dispatch_async`.
|
|
252
252
|
|
|
@@ -330,6 +330,7 @@ Multi-turn: continue a conversation with an agent. First call starts a session,
|
|
|
330
330
|
| `caller` | string | no | Who is dispatching |
|
|
331
331
|
| `goal` | string | no | Broader objective |
|
|
332
332
|
| `timeout_seconds` | int | no | One-off timeout override (0 = agent default; clamped to 10–7200) |
|
|
333
|
+
| `group` | string | no | Require group membership and prepend the group's `shared_context` to `context` |
|
|
333
334
|
|
|
334
335
|
`dispatch_session` is also the **timeout recovery path**: a timed-out `dispatch` returns a `session_id` — pass it here with `task="Continue where you left off"` to salvage the partial work instead of restarting.
|
|
335
336
|
|
|
@@ -394,7 +395,7 @@ Disconnecting or cancelling the tool call discards pending notifications and
|
|
|
394
395
|
stops buffering new ones. The worker still holds its concurrency slot until it
|
|
395
396
|
finishes.
|
|
396
397
|
|
|
397
|
-
|
|
398
|
+
`dispatch_stream` accepts `agent`, `task`, `context`, `caller`, `goal`, `response_format`, and `timeout_seconds`.
|
|
398
399
|
|
|
399
400
|
### `dispatch_dialogue`
|
|
400
401
|
|
|
@@ -492,10 +493,17 @@ fetch_result(ref="8f3a...e1", max_chars=2000) -> truncated, plus {"truncated":
|
|
|
492
493
|
```
|
|
493
494
|
|
|
494
495
|
Refs reuse the same storage as `dispatch_async` jobs (under `~/.config/agent-dispatch/jobs/`), so any `job_id` returned by `dispatch_async` is also a valid `ref` for `fetch_result`. `parsed_result` (when `response_format="json"` is set) is small and is always inlined directly in the ref response — no second fetch needed.
|
|
496
|
+
If the job has yet to produce a result, `fetch_result` returns the same snapshot as `dispatch_status`.
|
|
495
497
|
|
|
496
498
|
### Async dispatch — `dispatch_async`, `dispatch_status`, `dispatch_wait`, `dispatch_cancel`, `dispatch_jobs`, `dispatch_gc`
|
|
497
499
|
|
|
498
|
-
|
|
500
|
+
Async dispatch returns a `job_id` immediately. It accepts `agent`, `task`, `context`, `caller`, `goal`, `response_format`, and `timeout_seconds`, plus the following optional parameters. Async results bypass the result cache.
|
|
501
|
+
|
|
502
|
+
| Parameter | Type | Default | Description |
|
|
503
|
+
|-----------|------|---------|-------------|
|
|
504
|
+
| `group` | string | `""` | Require group membership and prepend the group's `shared_context` to `context`. An unknown group or non-member is rejected before creating a job. |
|
|
505
|
+
| `session_id` | string | `""` | Continue an existing session for the same agent. A conflict returns `error` and `active_job_id`. The error gives the job's status and age and names the `dispatch_wait` / `dispatch_cancel` calls for that job. `dispatch_session` does not check this reservation, so do not use it to get around a busy session. |
|
|
506
|
+
| `return_ref` | bool | `False` | On success, `dispatch_status` and `dispatch_wait` return `result.ref` and `result.summary` in place of `result.result`. Call `fetch_result(ref)` for the full text. Failed jobs retain their error and result. |
|
|
499
507
|
|
|
500
508
|
```
|
|
501
509
|
// 1. fire and forget (timeout_seconds= works here too for known-long tasks)
|
|
@@ -505,17 +513,23 @@ dispatch_async(agent="infra", task="audit every container log for OOM kills toda
|
|
|
505
513
|
// 2. do other work, then check progress (non-blocking)
|
|
506
514
|
// `progress` is a rolling tail of what the agent is doing right now
|
|
507
515
|
dispatch_status(job_id="8f3a...e1")
|
|
508
|
-
-> {"id": "8f3a...e1", "status": "running", "
|
|
516
|
+
-> {"id": "8f3a...e1", "status": "running", "task_preview": "audit every container log...",
|
|
517
|
+
"started_at": 1730000123.4,
|
|
509
518
|
"progress": ["Using tool: Bash", "Scanning container logs for OOM events..."], ...}
|
|
510
519
|
|
|
511
520
|
// 3. or block until done (timeout_seconds default: 60, capped at 3600)
|
|
512
521
|
dispatch_wait(job_id="8f3a...e1", timeout_seconds=120)
|
|
513
|
-
-> {"id": "8f3a...e1", "status": "done",
|
|
522
|
+
-> {"id": "8f3a...e1", "status": "done",
|
|
523
|
+
"result": {"agent": "infra", "success": true, "session_id": "sess-abc", ...}}
|
|
514
524
|
|
|
515
|
-
// If the timeout fires, the job keeps running
|
|
525
|
+
// If the timeout fires, the job keeps running. Call dispatch_wait again with the same job_id.
|
|
516
526
|
-> {"id": "...", "status": "running", "timed_out_waiting": true}
|
|
517
527
|
```
|
|
518
528
|
|
|
529
|
+
Both polling tools show one-line previews of the first 120 characters as `task_preview` and, when a context was supplied or added by a group, `context_preview`. The job file and `agent-dispatch job <id> --json` retain the complete task and context. Use the completed job's `result.session_id` for a follow-up with `dispatch_async` or `dispatch_session`.
|
|
530
|
+
|
|
531
|
+
Async continuation skips a stale running job when its start time and last file update are both at least an hour old; a pending job older than a day is marked failed and replaced. A live worker touches its job file once a minute, so a job whose agent is silent inside one long tool call is not mistaken for an abandoned one.
|
|
532
|
+
|
|
519
533
|
`dispatch_cancel(job_id)` cancels a **pending** job, and also kills a **running** job's `claude` process group when the job was started by the same server instance (the job is marked `cancelled` first, so the worker's trailing write can't undo it; partial work is lost but the progress tail is preserved). On POSIX, descendants in that process group are killed too, releasing inherited pipes and the concurrency slot. A cancel that lands after the job was claimed but before its process spawned is honored too: the process is killed as soon as it starts. A compatibility retry that races cancellation is killed when it registers. A running job started by a *previous* server run can't be killed safely and is left to finish. The response carries an `outcome` of `cancelled`, `cancelled_running`, `running` (not owned by this server), `already_terminal`, or `not_found`.
|
|
520
534
|
|
|
521
535
|
Async workers run with streaming under the hood: the job file keeps a rolling tail (last 20 lines, ~1 write/sec) of assistant text and tool-use events. `dispatch_status` shows it as `progress` while the job runs and keeps it afterwards as a post-mortem trace; `dispatch_jobs` shows `last_progress` for running jobs.
|
|
@@ -558,7 +572,7 @@ Failures are deterministic: check `success`, then branch on `error_type`.
|
|
|
558
572
|
| `not_found` | Agent directory or `claude` CLI missing | `list_agents()` → check `healthy`. Re-add the agent with an existing path, or run `agent-dispatch doctor` to find what's missing. |
|
|
559
573
|
| `recursion` | Dispatch nesting exceeded `max_dispatch_depth` (default 3) | Don't dispatch from dispatched agents; if the nesting is intentional, raise `max_dispatch_depth` in settings. |
|
|
560
574
|
| `budget` | The `claude` CLI ended the session at the `max_budget_usd` spend cap — the answer is incomplete | Raise the cap (`update_agent(name, max_budget_usd=2.0)`), switch to a cheaper `model`, or split the task. The partial session is resumable: `dispatch_session(agent, "Continue where you left off", session_id=<from the result>)`. |
|
|
561
|
-
| `usage_limit` |
|
|
575
|
+
| `usage_limit` | Claude reached a usage limit, including model-specific limits and time windows | Wait for the reset before retrying. The `hint` and `error` include the model or period and reset time when Claude reports them. If this recurs under load, lower `settings.max_concurrency`. |
|
|
562
576
|
| `cli_error` | Anything else from the `claude` subprocess | Read the `error` text; run `agent-dispatch doctor` for environment issues; retry once if transient. |
|
|
563
577
|
|
|
564
578
|
Three soft signals that arrive with `success: true`:
|
|
@@ -149,7 +149,7 @@ Members reference agents by name; membership is many-to-many (a shared gateway c
|
|
|
149
149
|
|
|
150
150
|
**`inspect_group(name)`** — one group's full brief: `description`, the complete `shared_context`, and the member list. For a deep dive on a specific member, call `inspect_agent(member)` — `inspect_group` deliberately stays a cheap membership readout.
|
|
151
151
|
|
|
152
|
-
|
|
152
|
+
Pass `group=` to `dispatch`, `dispatch_session`, `dispatch_async`, or an item in `dispatch_parallel`. The agent must be a member of the group. The group’s `shared_context` goes before the caller’s context.
|
|
153
153
|
|
|
154
154
|
```python
|
|
155
155
|
# From the shop-web codebase, hand the deploy to the infra gateway.
|
|
@@ -216,7 +216,7 @@ dispatch(
|
|
|
216
216
|
}
|
|
217
217
|
```
|
|
218
218
|
|
|
219
|
-
**`error_type` values:** `permission` (tool/action denied), `timeout`, `recursion` (dispatch depth exceeded), `not_found` (missing directory or CLI), `budget` (the `claude` CLI stopped the session at `max_budget_usd`), `usage_limit` (
|
|
219
|
+
**`error_type` values:** `permission` (tool/action denied), `timeout`, `recursion` (dispatch depth exceeded), `not_found` (missing directory or CLI), `budget` (the `claude` CLI stopped the session at `max_budget_usd`), `usage_limit` (Claude reached a usage limit), `cli_error` (other failures). Permission, budget, usage-limit and timeout errors include an actionable hint.
|
|
220
220
|
|
|
221
221
|
**Resumable timeouts:** every fresh dispatch pre-assigns a session UUID (`--session-id`), so a timed-out dispatch still returns a `session_id` — the partial transcript survives the kill. The timeout error spells out the recovery: resume with `dispatch_session(agent, "Continue where you left off", session_id=...)`, retry with a bigger `timeout_seconds`, or use `dispatch_async`.
|
|
222
222
|
|
|
@@ -300,6 +300,7 @@ Multi-turn: continue a conversation with an agent. First call starts a session,
|
|
|
300
300
|
| `caller` | string | no | Who is dispatching |
|
|
301
301
|
| `goal` | string | no | Broader objective |
|
|
302
302
|
| `timeout_seconds` | int | no | One-off timeout override (0 = agent default; clamped to 10–7200) |
|
|
303
|
+
| `group` | string | no | Require group membership and prepend the group's `shared_context` to `context` |
|
|
303
304
|
|
|
304
305
|
`dispatch_session` is also the **timeout recovery path**: a timed-out `dispatch` returns a `session_id` — pass it here with `task="Continue where you left off"` to salvage the partial work instead of restarting.
|
|
305
306
|
|
|
@@ -364,7 +365,7 @@ Disconnecting or cancelling the tool call discards pending notifications and
|
|
|
364
365
|
stops buffering new ones. The worker still holds its concurrency slot until it
|
|
365
366
|
finishes.
|
|
366
367
|
|
|
367
|
-
|
|
368
|
+
`dispatch_stream` accepts `agent`, `task`, `context`, `caller`, `goal`, `response_format`, and `timeout_seconds`.
|
|
368
369
|
|
|
369
370
|
### `dispatch_dialogue`
|
|
370
371
|
|
|
@@ -462,10 +463,17 @@ fetch_result(ref="8f3a...e1", max_chars=2000) -> truncated, plus {"truncated":
|
|
|
462
463
|
```
|
|
463
464
|
|
|
464
465
|
Refs reuse the same storage as `dispatch_async` jobs (under `~/.config/agent-dispatch/jobs/`), so any `job_id` returned by `dispatch_async` is also a valid `ref` for `fetch_result`. `parsed_result` (when `response_format="json"` is set) is small and is always inlined directly in the ref response — no second fetch needed.
|
|
466
|
+
If the job has yet to produce a result, `fetch_result` returns the same snapshot as `dispatch_status`.
|
|
465
467
|
|
|
466
468
|
### Async dispatch — `dispatch_async`, `dispatch_status`, `dispatch_wait`, `dispatch_cancel`, `dispatch_jobs`, `dispatch_gc`
|
|
467
469
|
|
|
468
|
-
|
|
470
|
+
Async dispatch returns a `job_id` immediately. It accepts `agent`, `task`, `context`, `caller`, `goal`, `response_format`, and `timeout_seconds`, plus the following optional parameters. Async results bypass the result cache.
|
|
471
|
+
|
|
472
|
+
| Parameter | Type | Default | Description |
|
|
473
|
+
|-----------|------|---------|-------------|
|
|
474
|
+
| `group` | string | `""` | Require group membership and prepend the group's `shared_context` to `context`. An unknown group or non-member is rejected before creating a job. |
|
|
475
|
+
| `session_id` | string | `""` | Continue an existing session for the same agent. A conflict returns `error` and `active_job_id`. The error gives the job's status and age and names the `dispatch_wait` / `dispatch_cancel` calls for that job. `dispatch_session` does not check this reservation, so do not use it to get around a busy session. |
|
|
476
|
+
| `return_ref` | bool | `False` | On success, `dispatch_status` and `dispatch_wait` return `result.ref` and `result.summary` in place of `result.result`. Call `fetch_result(ref)` for the full text. Failed jobs retain their error and result. |
|
|
469
477
|
|
|
470
478
|
```
|
|
471
479
|
// 1. fire and forget (timeout_seconds= works here too for known-long tasks)
|
|
@@ -475,17 +483,23 @@ dispatch_async(agent="infra", task="audit every container log for OOM kills toda
|
|
|
475
483
|
// 2. do other work, then check progress (non-blocking)
|
|
476
484
|
// `progress` is a rolling tail of what the agent is doing right now
|
|
477
485
|
dispatch_status(job_id="8f3a...e1")
|
|
478
|
-
-> {"id": "8f3a...e1", "status": "running", "
|
|
486
|
+
-> {"id": "8f3a...e1", "status": "running", "task_preview": "audit every container log...",
|
|
487
|
+
"started_at": 1730000123.4,
|
|
479
488
|
"progress": ["Using tool: Bash", "Scanning container logs for OOM events..."], ...}
|
|
480
489
|
|
|
481
490
|
// 3. or block until done (timeout_seconds default: 60, capped at 3600)
|
|
482
491
|
dispatch_wait(job_id="8f3a...e1", timeout_seconds=120)
|
|
483
|
-
-> {"id": "8f3a...e1", "status": "done",
|
|
492
|
+
-> {"id": "8f3a...e1", "status": "done",
|
|
493
|
+
"result": {"agent": "infra", "success": true, "session_id": "sess-abc", ...}}
|
|
484
494
|
|
|
485
|
-
// If the timeout fires, the job keeps running
|
|
495
|
+
// If the timeout fires, the job keeps running. Call dispatch_wait again with the same job_id.
|
|
486
496
|
-> {"id": "...", "status": "running", "timed_out_waiting": true}
|
|
487
497
|
```
|
|
488
498
|
|
|
499
|
+
Both polling tools show one-line previews of the first 120 characters as `task_preview` and, when a context was supplied or added by a group, `context_preview`. The job file and `agent-dispatch job <id> --json` retain the complete task and context. Use the completed job's `result.session_id` for a follow-up with `dispatch_async` or `dispatch_session`.
|
|
500
|
+
|
|
501
|
+
Async continuation skips a stale running job when its start time and last file update are both at least an hour old; a pending job older than a day is marked failed and replaced. A live worker touches its job file once a minute, so a job whose agent is silent inside one long tool call is not mistaken for an abandoned one.
|
|
502
|
+
|
|
489
503
|
`dispatch_cancel(job_id)` cancels a **pending** job, and also kills a **running** job's `claude` process group when the job was started by the same server instance (the job is marked `cancelled` first, so the worker's trailing write can't undo it; partial work is lost but the progress tail is preserved). On POSIX, descendants in that process group are killed too, releasing inherited pipes and the concurrency slot. A cancel that lands after the job was claimed but before its process spawned is honored too: the process is killed as soon as it starts. A compatibility retry that races cancellation is killed when it registers. A running job started by a *previous* server run can't be killed safely and is left to finish. The response carries an `outcome` of `cancelled`, `cancelled_running`, `running` (not owned by this server), `already_terminal`, or `not_found`.
|
|
490
504
|
|
|
491
505
|
Async workers run with streaming under the hood: the job file keeps a rolling tail (last 20 lines, ~1 write/sec) of assistant text and tool-use events. `dispatch_status` shows it as `progress` while the job runs and keeps it afterwards as a post-mortem trace; `dispatch_jobs` shows `last_progress` for running jobs.
|
|
@@ -528,7 +542,7 @@ Failures are deterministic: check `success`, then branch on `error_type`.
|
|
|
528
542
|
| `not_found` | Agent directory or `claude` CLI missing | `list_agents()` → check `healthy`. Re-add the agent with an existing path, or run `agent-dispatch doctor` to find what's missing. |
|
|
529
543
|
| `recursion` | Dispatch nesting exceeded `max_dispatch_depth` (default 3) | Don't dispatch from dispatched agents; if the nesting is intentional, raise `max_dispatch_depth` in settings. |
|
|
530
544
|
| `budget` | The `claude` CLI ended the session at the `max_budget_usd` spend cap — the answer is incomplete | Raise the cap (`update_agent(name, max_budget_usd=2.0)`), switch to a cheaper `model`, or split the task. The partial session is resumable: `dispatch_session(agent, "Continue where you left off", session_id=<from the result>)`. |
|
|
531
|
-
| `usage_limit` |
|
|
545
|
+
| `usage_limit` | Claude reached a usage limit, including model-specific limits and time windows | Wait for the reset before retrying. The `hint` and `error` include the model or period and reset time when Claude reports them. If this recurs under load, lower `settings.max_concurrency`. |
|
|
532
546
|
| `cli_error` | Anything else from the `claude` subprocess | Read the `error` text; run `agent-dispatch doctor` for environment issues; retry once if transient. |
|
|
533
547
|
|
|
534
548
|
Three soft signals that arrive with `success: true`:
|
|
@@ -24,6 +24,7 @@ logger = logging.getLogger(__name__)
|
|
|
24
24
|
|
|
25
25
|
JobStatus = Literal["pending", "running", "done", "failed", "cancelled"]
|
|
26
26
|
_TERMINAL_STATUSES: frozenset[str] = frozenset({"done", "failed", "cancelled"})
|
|
27
|
+
_STALE_RUNNING_SECONDS = 3600
|
|
27
28
|
|
|
28
29
|
# Job IDs are uuid4().hex — 32 lowercase hex chars. Anything else (notably
|
|
29
30
|
# values containing "/" or "..") is rejected so a caller-supplied ref/job_id
|
|
@@ -74,6 +75,8 @@ class Job(BaseModel):
|
|
|
74
75
|
# Kept after completion as a post-mortem trace of what the agent did.
|
|
75
76
|
progress: list[str] | None = None
|
|
76
77
|
progress_updated_at: float | None = Field(default=None, allow_inf_nan=False)
|
|
78
|
+
resume_session_id: str | None = None
|
|
79
|
+
return_ref: bool | None = None
|
|
77
80
|
|
|
78
81
|
def is_terminal(self) -> bool:
|
|
79
82
|
return self.status in _TERMINAL_STATUSES
|
|
@@ -106,6 +109,25 @@ class Job(BaseModel):
|
|
|
106
109
|
return entry
|
|
107
110
|
|
|
108
111
|
|
|
112
|
+
class ActiveSessionJobError(Exception):
|
|
113
|
+
"""A pending or running job already reserved the requested session."""
|
|
114
|
+
|
|
115
|
+
def __init__(self, job: Job, age: float):
|
|
116
|
+
self.job_id = job.id
|
|
117
|
+
# Name only calls that respect the reservation. dispatch_session does
|
|
118
|
+
# not check it: suggesting it here sent callers into two concurrent
|
|
119
|
+
# --resume runs of one session. dispatch_cancel is the way out when the
|
|
120
|
+
# job's worker is gone (a pending job is not swept for a day). The
|
|
121
|
+
# text does not say "retry": a caller re-sending a request whose first
|
|
122
|
+
# response was lost should collect that job's result, not run it twice.
|
|
123
|
+
super().__init__(
|
|
124
|
+
f"Session already has an active job {job.id} ({job.status}, age {age:.0f}s). "
|
|
125
|
+
f'Before sending this session another turn, wait for it with dispatch_wait(job_id="'
|
|
126
|
+
f'{job.id}") or cancel it with dispatch_cancel(job_id="{job.id}") — a running job '
|
|
127
|
+
"can only be cancelled by the server that started it."
|
|
128
|
+
)
|
|
129
|
+
|
|
130
|
+
|
|
109
131
|
class JobStore:
|
|
110
132
|
"""Thread-safe persistent store of dispatch jobs."""
|
|
111
133
|
|
|
@@ -155,6 +177,8 @@ class JobStore:
|
|
|
155
177
|
context: str | None = None,
|
|
156
178
|
caller: str | None = None,
|
|
157
179
|
goal: str | None = None,
|
|
180
|
+
resume_session_id: str | None = None,
|
|
181
|
+
return_ref: bool = False,
|
|
158
182
|
) -> Job:
|
|
159
183
|
"""Create a new pending job, persist it, return it."""
|
|
160
184
|
job = Job(
|
|
@@ -164,8 +188,55 @@ class JobStore:
|
|
|
164
188
|
context=context,
|
|
165
189
|
caller=caller,
|
|
166
190
|
goal=goal,
|
|
191
|
+
resume_session_id=resume_session_id or None,
|
|
192
|
+
return_ref=return_ref or None,
|
|
167
193
|
)
|
|
168
194
|
with self._lock:
|
|
195
|
+
# Both the check and publication must hold the same lock across servers.
|
|
196
|
+
if job.resume_session_id:
|
|
197
|
+
now = time.time()
|
|
198
|
+
expired: list[Job] = []
|
|
199
|
+
for path in self.directory.glob("*.json"):
|
|
200
|
+
existing = self._read(path)
|
|
201
|
+
if (
|
|
202
|
+
existing is not None
|
|
203
|
+
and not existing.is_terminal()
|
|
204
|
+
and existing.resume_session_id == job.resume_session_id
|
|
205
|
+
):
|
|
206
|
+
if existing.status == "running" and self._running_is_stale(
|
|
207
|
+
existing, now, _STALE_RUNNING_SECONDS
|
|
208
|
+
):
|
|
209
|
+
continue
|
|
210
|
+
# Same bound recover_stale uses, applied here because
|
|
211
|
+
# recovery only runs when some server starts: without
|
|
212
|
+
# it a job whose worker died in the queue held the
|
|
213
|
+
# session for as long as no server restarted. The job
|
|
214
|
+
# is failed, not merely skipped — a skipped one stays
|
|
215
|
+
# runnable, and a worker that does reach it later
|
|
216
|
+
# would resume the session alongside its replacement.
|
|
217
|
+
if existing.status == "pending" and (
|
|
218
|
+
now - existing.created_at
|
|
219
|
+
> _STALE_RUNNING_SECONDS * self._PENDING_STALE_MULTIPLIER
|
|
220
|
+
):
|
|
221
|
+
expired.append(existing)
|
|
222
|
+
continue
|
|
223
|
+
age = now - (existing.started_at or existing.created_at)
|
|
224
|
+
raise ActiveSessionJobError(existing, age)
|
|
225
|
+
for old in expired:
|
|
226
|
+
age = now - old.created_at
|
|
227
|
+
retired = self.fail(
|
|
228
|
+
old.id,
|
|
229
|
+
f"Abandoned in 'pending' for {age:.0f}s — "
|
|
230
|
+
f"job {job.id} took over its session.",
|
|
231
|
+
)
|
|
232
|
+
# fail() also returns None when it could not READ the
|
|
233
|
+
# record. Publishing the replacement then would leave the
|
|
234
|
+
# old job runnable beside it, so only a job positively
|
|
235
|
+
# seen terminal gives up the session.
|
|
236
|
+
if retired is None:
|
|
237
|
+
current = self.get(old.id)
|
|
238
|
+
if current is None or not current.is_terminal():
|
|
239
|
+
raise ActiveSessionJobError(old, age)
|
|
169
240
|
self._write(job)
|
|
170
241
|
return job
|
|
171
242
|
|
|
@@ -336,7 +407,30 @@ class JobStore:
|
|
|
336
407
|
except (OSError, ValueError): # pragma: no cover - racing unlink
|
|
337
408
|
return False
|
|
338
409
|
|
|
339
|
-
def
|
|
410
|
+
def touch(self, job_id: str) -> None:
|
|
411
|
+
"""Heartbeat: bump the job file's mtime without rewriting the record.
|
|
412
|
+
|
|
413
|
+
The mtime is the liveness signal `_running_is_stale` reads. Progress
|
|
414
|
+
writes alone are not one: an agent inside a single long tool call (a
|
|
415
|
+
nested dispatch can take two hours) emits nothing, and after an hour
|
|
416
|
+
of that silence the job looked abandoned — a second job could resume
|
|
417
|
+
the same session, and another server's startup recovery failed the
|
|
418
|
+
job, after which finish() refused the paid result. No lock: utime is
|
|
419
|
+
one syscall, and landing on a file that a concurrent write is about to
|
|
420
|
+
replace is harmless (the replacement carries a fresh mtime itself).
|
|
421
|
+
"""
|
|
422
|
+
try:
|
|
423
|
+
os.utime(self._path(job_id))
|
|
424
|
+
except (OSError, ValueError):
|
|
425
|
+
logger.debug("Could not touch job %s", job_id)
|
|
426
|
+
|
|
427
|
+
def _running_is_stale(self, job: Job, now: float, threshold: float) -> bool:
|
|
428
|
+
# A recent write or heartbeat (see touch) protects long-running jobs
|
|
429
|
+
# whose start time is already beyond the recovery threshold.
|
|
430
|
+
age = now - (job.started_at or job.created_at)
|
|
431
|
+
return age > threshold and not self._touched_after(job.id, now - threshold)
|
|
432
|
+
|
|
433
|
+
def recover_stale(self, stale_threshold_seconds: float = _STALE_RUNNING_SECONDS) -> int:
|
|
340
434
|
"""Mark jobs abandoned in 'running' or 'pending' beyond the threshold as failed.
|
|
341
435
|
|
|
342
436
|
Async workers are daemon threads — if the server dies mid-dispatch the
|
|
@@ -363,22 +457,9 @@ class JobStore:
|
|
|
363
457
|
# Claude Code session, routinely 14+.
|
|
364
458
|
for job in self.list():
|
|
365
459
|
if job.status == "running":
|
|
366
|
-
|
|
367
|
-
if age <= stale_threshold_seconds:
|
|
368
|
-
continue
|
|
369
|
-
# `started_at` alone does not prove abandonment: a dispatch
|
|
370
|
-
# may legitimately run up to the 7200s timeout ceiling, so
|
|
371
|
-
# any other server starting up would flip a live 2-hour job
|
|
372
|
-
# to 'failed' — and finish() then refuses it, losing an
|
|
373
|
-
# already-paid-for result. A live worker rewrites the file
|
|
374
|
-
# at least once per progress interval, so a recently
|
|
375
|
-
# *modified* file means someone is still on it. This can
|
|
376
|
-
# only ever skip a recovery (a genuinely abandoned job is
|
|
377
|
-
# picked up on a later start), never add one. Only 'running'
|
|
378
|
-
# gets this check: nothing rewrites a pending job's file, so
|
|
379
|
-
# there its mtime is just created_at by another name.
|
|
380
|
-
if self._touched_after(job.id, now - stale_threshold_seconds):
|
|
460
|
+
if not self._running_is_stale(job, now, stale_threshold_seconds):
|
|
381
461
|
continue
|
|
462
|
+
age = now - (job.started_at or job.created_at)
|
|
382
463
|
elif job.status == "pending":
|
|
383
464
|
age = now - job.created_at
|
|
384
465
|
if age <= pending_threshold:
|
|
@@ -234,7 +234,7 @@ class DispatchResult(BaseModel):
|
|
|
234
234
|
# Journal-only telemetry: never inflate tool responses or persisted jobs.
|
|
235
235
|
token_usage: dict[str, Any] | None = Field(default=None, exclude=True)
|
|
236
236
|
error: str | None = None
|
|
237
|
-
error_type: str | None = None # permission, timeout,
|
|
237
|
+
error_type: str | None = None # e.g. permission, timeout, usage_limit, cli_error
|
|
238
238
|
# Set when response_format="json" was requested AND the agent's result
|
|
239
239
|
# parsed cleanly. None means: not requested, or requested but unparseable.
|
|
240
240
|
parsed_result: Any | None = None
|
|
@@ -242,8 +242,7 @@ class DispatchResult(BaseModel):
|
|
|
242
242
|
# JSON output). Non-empty even on success=True — the agent may have
|
|
243
243
|
# completed with an incomplete answer because a tool was blocked.
|
|
244
244
|
denied_tools: list[str] | None = None
|
|
245
|
-
#
|
|
246
|
-
# Errors stay in `error`; hint is for successful-but-degraded results.
|
|
245
|
+
# Actionable guidance for a degraded result or account usage limit.
|
|
247
246
|
hint: str | None = None
|
|
248
247
|
# True when cost_usd exceeded the agent's max_budget_usd (or the settings
|
|
249
248
|
# default). Post-hoc only — the money is already spent, the dispatch is
|