agent-dispatch 0.15.1__tar.gz → 0.16.0__tar.gz

agent_dispatch-0.16.0/.github/workflows/ci.yml

@@ -0,0 +1,58 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+# Least privilege: CI only needs to read the repo.
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: ruff check src/ tests/ scripts/
+
+      - name: Test
+        run: pytest tests/ -v
+
+  installed-package:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Build wheel and source distribution
+        run: |
+          python -m pip install build twine
+          python -m build
+          python -m twine check dist/*
+      - name: Install wheel into a clean environment
+        run: |
+          python -m venv "$RUNNER_TEMP/package-check"
+          "$RUNNER_TEMP/package-check/bin/python" -m pip install dist/*.whl
+          "$RUNNER_TEMP/package-check/bin/python" -m pip check
+      - name: Verify CLI and MCP transport from installed wheel
+        run: '"$RUNNER_TEMP/package-check/bin/python" -I scripts/check_install.py'

{agent_dispatch-0.15.1 → agent_dispatch-0.16.0}/.gitignore

@@ -18,3 +18,6 @@ CLAUDE.md
 .claude/
 .mcp.json
 *.db
+.coverage
+.coverage.*
+htmlcov/

{agent_dispatch-0.15.1 → agent_dispatch-0.16.0}/AGENTS.md

@@ -11,13 +11,15 @@ MCP server + CLI that lets Claude Code agents delegate tasks to agents in other
 | File | Role |
 |------|------|
 | `src/agent_dispatch/runner.py` | Sync subprocess wrapper around `claude -p` — the actual work |
-| `src/agent_dispatch/server.py` | Async FastMCP interface (21 MCP tools), wraps runner in `asyncio.to_thread` + semaphore |
+| `src/agent_dispatch/server.py` | Async FastMCP interface (21 MCP tools), wraps runner in `asyncio.to_thread` with a shared process limit |
 | `src/agent_dispatch/usage.py` | Append-only dispatch journal (cost/duration/outcome) + aggregation for `stats` and the `typical` block |
 | `src/agent_dispatch/servers.py` | Registry of live `serve` processes and the version each one runs (`doctor` reads it) |
 | `src/agent_dispatch/cli.py` | Click CLI: `init`, `add`, `update`, `remove`, `list`, `describe`, `test`, `doctor`, `jobs`, `job`, `cancel`, `gc`, `group` (add/list/inspect/update/remove), `serve` |
 | `src/agent_dispatch/models.py` | Pydantic v2 models (`AgentConfig`, `DispatchGroup`/`GroupMember`, `Settings`, `DispatchResult`) |
 | `src/agent_dispatch/config.py` | YAML config load/save + project auto-description |
 | `src/agent_dispatch/cache.py` | Thread-safe in-memory TTL cache |
+| `src/agent_dispatch/concurrency.py` | Resizable process limit shared by async callers and background workers |
+| `src/agent_dispatch/progress.py` | Bounded progress buffer, discarded when a streaming client disconnects |
 | `src/agent_dispatch/jobs.py` | Persistent per-job JSON files for async dispatch |
 
 ## Dev setup
@@ -29,11 +31,13 @@ pip install -e ".[dev]"
 ## Gates — both must pass before a change is done (CI rejects otherwise)
 
 ```bash
-ruff check src/ tests/
-python3 -m pytest tests/ -v   # 714 tests, ~17s
+ruff check src/ tests/ scripts/
+python3 -m pytest tests/ -v
 ```
 
-Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.which` + `subprocess.run`/`Popen`; server tests mock `_get_config` + `runner.dispatch`. The one exception is `TestStreamPipeHandling`, which spawns a short-lived *python* subprocess: a pipe deadlock lives in the OS pipe buffer, so a mocked `Popen` structurally cannot reproduce it.
+Tests must **never** invoke the real `claude` CLI. Runner unit tests mock `shutil.which` + `_run_captured`/`Popen`; server tests mock `_get_config` + `runner.dispatch`. Pipe and decoding integration tests spawn short-lived *python* subprocesses: a pipe deadlock lives in the OS pipe buffer, so a mocked `Popen` structurally cannot reproduce it.
+
+The autouse guard in `tests/conftest.py` rejects real Claude process launches when a test forgets its mock. Diagnostic tests must mock discovery or CLI probes too.
 
 ## Non-obvious invariants (violating these breaks real behavior)
 
@@ -45,23 +49,23 @@ Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.whi
 - Valid permission modes: `default`, `plan`, `bypassPermissions` (`models.py: KNOWN_PERMISSION_MODES`).
 - `JobStore.finish`/`fail` refuse already-terminal jobs (returns `None`) — this closes the race with force-cancel; never "fix" it by overwriting. `mark_running` likewise refuses any job that isn't `pending`, so a stale or duplicate worker can't resurrect a finished one.
 - "Is this group member missing?" has exactly one implementation: `DispatchConfig.unknown_group_members()`. Any new surface that lists or validates membership calls it instead of re-deriving the check.
-- Cancelling a *running* job requires the in-memory `_running_procs` registry (server.py) — the job is marked `cancelled` **before** the subprocess is killed. Don't persist PIDs to disk (PID reuse after restart could kill an unrelated process).
+- Cancelling a *running* job requires the in-memory registry (server.py): `_owned_jobs` records ownership **before** `mark_running`, `_running_procs` the Popen once spawned — a cancel landing between the two is still honored, and the late process is killed on registration. The job is marked `cancelled` **before** the subprocess is killed. A job record that is merely *unreadable* at spawn is not treated as terminal. Don't persist PIDs to disk (PID reuse after restart could kill an unrelated process).
 - `max_budget_usd` is enforced **by the claude CLI** (`_build_command` passes `--max-budget-usd`): a run stopped at the cap comes back `is_error` with no `result` text, and `_build_error_result` turns it into `error_type="budget"` + `budget_exceeded=True` + a resumable `session_id`. `_apply_budget` is the *secondary*, post-hoc signal for an overshoot that didn't stop the run; it never fails a dispatch.
 - A CLI error payload can have no `result` field at all — the reason lives in `errors` / `subtype`. Read it via `_cli_error_details`, never assume `result` is populated on failure.
-- **Both subprocess pipes must be drained concurrently.** `dispatch_stream` reads stdout in a loop while a daemon thread drains stderr; reading stderr only after the loop deadlocks any child that writes more than ~64 KiB to it (the child blocks in `write(2)`, never emits its result, and the dispatch dies at the timeout). `dispatch()` is immune only because `subprocess.run(capture_output=True)` uses `communicate()`.
+- **Both subprocess pipes must be drained concurrently.** `dispatch_stream` reads stdout in a loop while a daemon thread drains stderr; reading stderr only after the loop deadlocks any child that writes more than ~64 KiB to it (the child blocks in `write(2)`, never emits its result, and the dispatch dies at the timeout). Ordinary `dispatch` uses `_run_captured` with two readers; binary `read1` retains flushed output even without a newline. Readers own their pipes, and stop retaining output when the caller returns. **Completion is stdout EOF + leader exit, never stderr EOF**: a stdio MCP server started by `claude` inherits fd 2 and outlives it, so waiting for stderr EOF turned every such dispatch into a full-timeout wait and a real CLI error into `error_type="timeout"`. stderr gets a short bounded join afterwards, then the leftover group is killed. `dispatch_stream` reads stdout through a reader thread too, so a descendant that *detached* from the group (unreachable by `killpg`) cannot park it forever once the run was killed.
 - **A received `result` event outranks the timeout flag.** The agent finished and was billed; a lingering process is a cleanup problem, reported as a `hint`, not a failure.
-- **The timeout must kill the process *tree*, not the child.** `dispatch_stream` spawns with `start_new_session=True` and `_kill_process_tree` sends SIGKILL to the group: a grandchild that inherited stdout keeps the read loop parked long past the deadline otherwise. `killpg` is guarded on a positive pid — `killpg(0)` would signal the dispatcher's own group.
+- **The timeout must kill the process *tree*, not the child.** Both runners spawn with `start_new_session=True` and `_kill_process_tree` sends SIGKILL to the group: a grandchild that inherited stdout keeps the read loop parked long past the deadline otherwise. `killpg` is guarded on a positive pid — `killpg(0)` would signal the dispatcher's own group — and on `proc.returncode is None`: a reaped leader's PID may already belong to an unrelated group. So nothing may reap the leader (`poll()`/`wait()`) before the kill decision; `_wait_leader_exit` observes exit without reaping (`waitid(WNOWAIT)` on Linux, a kqueue exit watch on macOS). `_run_captured` drains pipes before reaping the leader, so its PID cannot be recycled before a timeout kill. Cleanup waits are bounded even when a descendant deliberately leaves the process group. A full CLI `type=result` JSON payload outranks the ordinary timeout too; incomplete output does not.
 - **Never `close()` a pipe another thread may still be reading.** `close()` waits on the reader's buffer lock with *no timeout*, so it would hang the dispatch forever — the bounded `join()` before it buys nothing. `dispatch_stream` skips the stderr close while the drain thread is alive and lets the daemon reader + Popen finalizer release the fd. This is not hypothetical: a stdio MCP server inherits the child's `stderr`, so the pipe often has no EOF even after `claude` exits cleanly.
 - **No `await` inside `config_lock()`.** `ProcessLock`'s in-process guard is a `threading.RLock` — re-entrant per *thread* — and every MCP tool coroutine runs on the one event-loop thread. Suspending in the critical section lets a second coroutine re-enter the "held" lock and interleave its own load/mutate/save. Collect warnings as data, emit them after the `with` block (`test_no_await_inside_the_config_lock` enforces this by AST).
 - Cross-process locks are acquired with a **bounded** wait, never a blocking `flock`: the server takes them on its event-loop thread, so a wedged holder would freeze every tool. After the deadline it proceeds unlocked and logs — a possible lost update beats a permanent freeze.
 - `recover_stale` sweeps `pending` on a **much longer** threshold than `running`: the jobs directory is shared by every `agent-dispatch serve`, so an hours-old pending job may still be queued behind another live server's semaphore. For a *running* job, `started_at` alone does **not** prove abandonment — a dispatch may legitimately run to the 7200s timeout ceiling, so the file's own mtime is checked too (a live worker rewrites it on every progress flush). That check can only ever *skip* a recovery, never add one.
 - Deleting job records is **opt-in** (`settings.job_retention_days`, default `0` = off) and only ever happens at server start, never inside a tool. They are the user's dispatch history and the deletion is irreversible, so an unreadable config is treated as "do nothing" rather than falling back to a default retention.
-- `max_concurrency` must bound *subprocesses*, not coroutines. Cancelling a coroutine does not stop the thread behind `asyncio.to_thread`, so `async with sem:` gave the slot away while `claude` kept running and billing. Dispatches go through `_dispatch_guarded`, which releases from the future's done-callback and `shield`s the await. Never "simplify" it back to `async with`.
+- `max_concurrency` must bound *subprocesses*, not coroutines. All dispatch paths share one `DispatchLimiter` per server, resized in place so changing the limit cannot erase occupied slots. Ordinary and streaming dispatches go through `_start_guarded`, which releases from the future's done-callback; awaits on that future are shielded. Background jobs hold the same limiter in their worker thread. The limiter is **FIFO across threads and coroutines** — a freed slot is handed to the head waiter in `release()`; otherwise parked job threads, woken in microseconds, starved a queued synchronous `dispatch` indefinitely. A waiter cancelled after its grant passes the slot on exactly once. Cancelling a coroutine does not stop its thread: never release its slot from an `async with` or caller-side `finally`.
 - Tool responses are serialized through `server._dumps`, never bare `json.dumps`: the stdlib default (`ensure_ascii=True`) turns every non-ASCII character into a `\uXXXX` escape, tripling the bytes and tokenizing badly, for no gain — the stdio transport emits raw UTF-8 via pydantic anyway. A real `list_groups()` carried 8520 escapes and weighed 59 KB instead of 25 KB.
 - `agents.yaml` is parsed with libyaml's `CSafeLoader` when available (`config._YamlLoader`), falling back to the pure-Python **safe** loader — never `yaml.Loader`. The config is re-read on every tool call, so this parse is on the hot path of all 21 tools *and* blocks the event-loop thread: 9.80 ms → 0.76 ms on a real 38 KB config.
 - Pydantic does **not** validate on assignment. `Field(ge=...)` guards only the *load* path; every mutation surface (CLI `add`/`update`, MCP `add_agent`/`update_agent`) needs its own boundary check, or the bound escapes as a raw `ValidationError`.
 - Every state file (`agents.yaml`, job files) is written **temp file + `os.replace`**, never in place, and every load/mutate/save is wrapped in `config.ProcessLock` — the CLI and the MCP server are separate processes writing the same files, so a thread lock alone loses updates.
-- Anything that changes an agent's config must call `_invalidate_agent_cache` — the cache key holds the agent *name*, not its directory or permissions.
+- Anything that changes an agent's config must call `_invalidate_agent_cache` to reclaim old entries. Cache lookups and writes also receive the same `_cache_config_key` snapshot, captured before awaiting work: this covers CLI/YAML/other-server edits and prevents an old worker from repopulating the current configuration's cache. Include new dispatch-affecting settings in this snapshot.
 - Only *clean* successes are cached: `cache.put` refuses failures, `denied_tools` results, `budget_exceeded` results, and results whose `outcome` is `partial`/`blocked`, so the documented "grant access / fix the cause, then re-dispatch" recovery is never short-circuited.
 - **Coerce every number read off disk through `usage.as_float`.** `float("abc")` raises **ValueError** — not `JSONDecodeError`, not `OSError` — so it escaped the per-line handler in `usage.load` and the sort key in `servers.live_servers`, turning `agent-dispatch stats --days 7` and `doctor` into tracebacks over one hand-edited character. Same escape route as the `UnicodeDecodeError`-is-a-`ValueError` round. A journal or registry field is untrusted input: skip the record or sort it last, never raise from the command someone ran *because* something is already wrong.
 - **`--json` output must be valid JSON on the empty path too.** `stats --json` printed prose when the journal had no records — i.e. on a fresh install, which is exactly when a script first pipes it somewhere.
@@ -70,7 +74,7 @@ Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.whi
 - **Profiles are memoized on the journal's (path, size, mtime) and read a NARROWER tail than `stats`.** `list_agents`/`inspect_agent` run on the event-loop thread: a full 2 MB journal cost **24.8 ms** per discovery call at the 512 KB report window — 33x the config parse that 0.13.0 exists to have fixed. Now 3.3 ms cold, ~0 warm. Every profile consumer goes through `usage.profiles()`; never add a caller that re-parses per agent.
 - **`usage_limit` is its own error type** (observed live: `You've hit your session limit · resets 6pm`). It is checked *before* the permission patterns and its hint says the limit is account-wide, so "try another agent" is not a workaround. Text-classified errors attach their advisory through the single `_classification_hint`, not per site.
 - **Server liveness is an advisory lock, not a PID.** `servers.register` holds an exclusive `flock` on `<config dir>/servers/<pid>.json` for the process lifetime; a reader that *takes* that lock owns a dead entry and deletes it. `os.kill(pid, 0)` would believe a recycled PID and would never notice a SIGKILLed server. Never close that fd (re-registering closes the old one first).
-- **The dispatch protocol goes in the system prompt, not the task.** `_build_system_prompt` (runner.py) renders the protocol (non-interactive, time/spend budget, lead with the outcome, trailing `STATUS:` line) plus the agent's `instructions`, and `_build_command` passes it as `--append-system-prompt` — on `--resume` too, since the CLI re-applies an appended prompt on every launch. `_build_prompt` (the `-p` text) is unchanged, so the cache key is unchanged. The text always starts with a `##` header line so it can never be read as a flag. `settings.dispatch_protocol: false` turns the protocol off (for CLIs that predate the flag; `doctor` probes `claude --help` for it); per-agent `instructions` still go through when set.
+- **The dispatch protocol goes in the system prompt, not the task.** `_build_system_prompt` (runner.py) renders the protocol (non-interactive, time/spend budget, lead with the outcome, trailing `STATUS:` line) plus the agent's `instructions`, and `_build_command` passes it as `--append-system-prompt` — on `--resume` too, since the CLI re-applies an appended prompt on every launch. `_build_prompt` (the `-p` text) is unchanged. Standing instructions and protocol settings are captured separately in `_cache_config_key`. The text always starts with a `##` header line so it can never be read as a flag. `settings.dispatch_protocol: false` turns the protocol off (for CLIs that predate the flag; `doctor` probes `claude --help` for it); per-agent `instructions` still go through when set.
 - **`outcome` is lifted from the LAST line of the result** (`_split_outcome`), before JSON parsing, by the shared `_build_success_result` (the success-side twin of `_build_error_result` — both `dispatch` and `dispatch_stream` go through it) and by the plain-text fallback tier. A bare marker line is removed from `result`; a marker with a trailing reason stays. `outcome` never flips `success`; `partial`/`blocked` add a `hint` with the `dispatch_session(...)` continuation, after the denial hint. In JSON mode the protocol omits the STATUS instruction (the JSON footer governs), but a STATUS line that arrives anyway is still stripped so `parsed_result` survives.
 - Remediation text is a contract: a hint that names a flag must name one that exists (`test_printed_budget_hint_is_a_runnable_command` feeds the printed flags back into the CLI). Run the command you print.
 - The config error sets are declared **once** and in two halves: `config.CONFIG_LOAD_ERRORS` (read) and `config.CONFIG_SAVE_ERRORS` (write — `yaml.dump`'s `RepresenterError` is a `yaml.YAMLError`, therefore neither `OSError` nor `ConfigLoadError`, and used to escape both the MCP guard and the CLI's `_save_or_exit`). Two halves, not one set, because the remediations differ: a failed write is atomic so the old config survives, while a failed read needs the YAML fixed.
@@ -80,6 +84,7 @@ Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.whi
 
 - `mcp` is pinned **`>=1.2.0,<2`** deliberately: 2.0 removed `mcp.server.fastmcp`, which `server.py` imports, so an unbounded range gives every fresh install a dead `agent-dispatch serve`. Lifting the cap means porting to `mcp.server.mcpserver.MCPServer` — it is not a dependency bump.
 - Verify packaging in a **clean venv**, never the dev machine: build the wheel, install it fresh, import the server. A stale pin in local site-packages hides exactly the failure a new user hits first.
+- CI's `installed-package` job builds wheel and sdist, checks metadata with twine, then installs only the wheel and runtime dependencies into a fresh venv on Python 3.10 and 3.13. Run `python -I scripts/check_install.py` with that venv's Python to verify the installed CLI and a real MCP stdio connection. The check isolates all state in a temporary directory and cannot discover Claude on PATH.
 - Run **`twine check dist/*`** before cutting the tag, with a *current* twine. The clean-venv check does not cover this: it proves the wheel installs, not that PyPI's uploader will accept its metadata. `python -m build` pulls the newest hatchling at build time, so the emitted `Metadata-Version` climbs on its own, and `pypa/gh-action-pypi-publish` is pinned to a SHA that bundles a fixed twine — v0.13.0's publish failed on exactly that mismatch (hatchling 1.32 → metadata 2.5; pinned twine 6.1.0 → "not a valid metadata version"). When it happens, bump the action pin rather than pinning the backend down.
 
 ## Deliberately not built
@@ -96,4 +101,4 @@ Python ≥ 3.10 · `from __future__ import annotations` everywhere · Pydantic v
 
 ## More detail
 
-[README.md](README.md) documents every MCP tool with parameter tables, response shapes, and the error-recovery map — it doubles as the behavioral spec. The test suite (`tests/`, 714 tests) encodes the exact expected behavior of every layer: when in doubt, read the tests for the module you're touching (`test_runner.py`, `test_server.py`, `test_cli.py`, ...).
+[README.md](README.md) documents every MCP tool with parameter tables, response shapes, and the error-recovery map — it doubles as the behavioral spec. The test suite (`tests/`) encodes the exact expected behavior of every layer: when in doubt, read the tests for the module you're touching (`test_runner.py`, `test_server.py`, `test_cli.py`, ...).

{agent_dispatch-0.15.1 → agent_dispatch-0.16.0}/CHANGELOG.md

@@ -7,6 +7,110 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.16.0] - 2026-09-30
+
+The process-control round. One concurrency limit now covers every dispatch
+path and serves callers in arrival order; ordinary dispatch runs on the same
+bounded two-reader design as streaming; and a round of adversarial review
+aimed at pipes, process groups and cancellation closed the cases where a
+finished run waited out its whole timeout, a detached descendant hung a
+dispatch forever, or a paid result was thrown away over optional metadata.
+Scripts get machine-readable `doctor`, `jobs` and `job` output.
+
+### Added
+- CI verifies clean wheel installation on Python 3.10 and 3.13, distribution
+  metadata, and a real MCP stdio connection. `scripts/check_install.py` checks
+  CLI/MCP interoperability and full-result export with isolated temporary data.
+- `doctor --json` emits a versioned diagnostic report with checks, severity,
+  counts, and remediation details. `--strict` makes warnings fail the command,
+  for installation checks in scripts.
+- `agent-dispatch jobs --json` exports compact summaries; `job <id> --json`
+  exports the full record, including untruncated results and session metadata.
+- Filter job history by exact agent name with `jobs --agent NAME` or
+  `dispatch_jobs(agent="NAME")`, including history for removed agents.
+- Preview history cleanup with `agent-dispatch gc --dry-run` and
+  `dispatch_gc(dry_run=True)`. Both use the same eligibility rules as deletion.
+
+### Fixed
+- Non-finite budgets are rejected before CLI/MCP mutations and during YAML
+  loading. Invalid numeric fields in parallel dispatch return an error before
+  any task starts, including values that overflow integer conversion.
+- Ordinary dispatch now kills the process group on timeout, stopping descendants
+  that could continue work after the call failed. Pipe reads are concurrent and
+  cleanup is bounded even when a descendant detaches from the group.
+- A complete CLI result from ordinary dispatch survives a cleanup timeout,
+  preserving the paid answer, cost, session and error classification. Partial
+  output still returns a resumable timeout.
+- Running-job cancellation kills the process group, including descendants
+  holding output pipes open. A compatibility retry racing cancellation is
+  stopped too; the cancelled state is persisted before any kill.
+- Streaming ignores malformed progress events without losing the final paid
+  result. Failed process registration cleans up the child and its pipes.
+- Cache keys include the agent configuration and dispatch defaults. Edits from
+  the CLI, another server, or YAML cannot reuse an old answer, and workers
+  finishing under an older configuration cannot overwrite current answers.
+- The process-tree timeout regression test synchronizes subprocess setup before
+  its one-second deadline, avoiding startup-related failures on busy machines.
+- Non-finite and overflowing numeric metadata no longer poison usage reports
+  or crash server diagnostics. Invalid optional result costs are omitted while
+  preserving the dispatch output; journal records omit non-finite costs.
+- Doctor's server counts and version drift now use one registry snapshot,
+  avoiding inconsistent totals when servers start or exit during a check.
+- Two diagnostic tests now mock Claude discovery; a shared test guard rejects
+  accidental launches of the real Claude CLI.
+- Streaming progress uses a bounded tail instead of an unlimited queue.
+  Slow clients receive an explicit omitted-message count; disconnected clients
+  stop retaining progress. Full dispatch results remain intact.
+- All dispatch paths now share `max_concurrency` within a server process.
+  Mixing ordinary and background jobs no longer allows twice the configured
+  number of subprocesses. Changing the limit preserves occupied slots.
+- Cancelling a streaming tool call keeps its concurrency reservation until the
+  worker finishes, just like an ordinary dispatch.
+- Damaged job records cannot redirect status updates or stale recovery to a
+  different job: all reads verify the internal ID against the filename.
+- Invalid and non-finite job timestamps are rejected at the read boundary.
+  Unreadable records are preserved, including during history cleanup.
+- Job IDs with trailing newlines are rejected before file access.
+- Agent and group name validation rejects trailing newlines instead of accepting
+  a partial regular-expression match.
+- Ordinary dispatch completes when the CLI exits and its stdout closes, instead
+  of waiting for stderr. A stdio MCP server that inherited stderr no longer
+  turns a finished run into a full-timeout wait, and a real CLI error (for
+  example a usage limit) is no longer reported as a timeout.
+- Streaming dispatch and running-job cancellation are bounded even when a
+  descendant leaves the process group while holding stdout; the concurrency
+  slot is released.
+- A process group is never signalled after its leader was reaped, so a recycled
+  PID cannot direct the kill at an unrelated process group.
+- The shared concurrency limit serves waiters in arrival order. Background jobs
+  queued later can no longer starve an ordinary, streaming or parallel call.
+- Cancelling an owned job between its start and its process spawn now cancels
+  it instead of reporting it as belonging to another server. A job record that
+  cannot be read at spawn no longer makes the worker kill its own process.
+- Malformed `duration_ms`, `num_turns` or `session_id` in CLI output no longer
+  discard the paid result; a bad session id falls back to the generated one.
+- Usage reports ignore implausible journal values, so totals cannot overflow;
+  `stats --json` never prints `Infinity`.
+- `doctor` survives non-UTF-8 `claude mcp list` output and an unreadable config
+  directory (JSON is still emitted). Its old-CLI warning accounts for
+  `dispatch_protocol` and per-agent instructions, so applying its remedy clears
+  it under `--strict`.
+- `gc`, `gc --dry-run` and `cancel` report job-storage errors instead of
+  printing a traceback.
+- The test guard against launching the real Claude CLI now fails the test even
+  when the launch happens in a background worker thread.
+- A failing progress callback (for example a full disk while an async job
+  writes its progress) no longer kills the run or discards its paid result and
+  session. If the reader threads cannot be started, the spawned process is
+  killed instead of running unsupervised.
+- `dispatch_cancel` reports `already_terminal` when this server's own job
+  finishes during the cancel, instead of claiming another server owns it.
+- A lone surrogate escape in an agent's answer (typically a hand-escaped emoji
+  in a `response_format="json"` reply) no longer makes `dispatch`,
+  `dispatch_session`, `dispatch_stream` or `return_ref` raise instead of
+  answering, and no longer costs an async job its paid result. It is replaced
+  with U+FFFD where the CLI output is parsed.
+
 ## [0.15.1] - 2026-09-09
 
 Three defects in 0.15.0's own new code, found by re-auditing it the way this
@@ -802,7 +906,12 @@ cache bounding, and stale-job recovery.
 - Dependabot for `pip` + `github-actions`, GitHub Actions pinned to
   commit SHAs for supply-chain integrity.
 
-[Unreleased]: https://github.com/ginkida/agent-dispatch/compare/v0.12.1...HEAD
+[Unreleased]: https://github.com/ginkida/agent-dispatch/compare/v0.16.0...HEAD
+[0.16.0]: https://github.com/ginkida/agent-dispatch/compare/v0.15.1...v0.16.0
+[0.15.1]: https://github.com/ginkida/agent-dispatch/compare/v0.15.0...v0.15.1
+[0.15.0]: https://github.com/ginkida/agent-dispatch/compare/v0.14.0...v0.15.0
+[0.14.0]: https://github.com/ginkida/agent-dispatch/compare/v0.13.0...v0.14.0
+[0.13.0]: https://github.com/ginkida/agent-dispatch/compare/v0.12.1...v0.13.0
 [0.12.1]: https://github.com/ginkida/agent-dispatch/compare/v0.12.0...v0.12.1
 [0.12.0]: https://github.com/ginkida/agent-dispatch/compare/v0.11.0...v0.12.0
 [0.11.0]: https://github.com/ginkida/agent-dispatch/compare/v0.10.0...v0.11.0

{agent_dispatch-0.15.1 → agent_dispatch-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agent-dispatch
-Version: 0.15.1
+Version: 0.16.0
 Summary: MCP server that lets Claude Code agents delegate tasks to agents in other project directories
 Project-URL: Homepage, https://github.com/ginkida/agent-dispatch
 Project-URL: Repository, https://github.com/ginkida/agent-dispatch
@@ -385,6 +385,13 @@ Run multiple tasks concurrently. Much faster than sequential `dispatch` calls.
 
 Same as `dispatch` but shows live progress while the agent works. Use for long-running tasks. Not cached.
 
+Progress notifications keep the latest 100 pending messages, capped at 300
+characters each. If output arrives faster than the client can receive it,
+older notifications are omitted with a count; the final result is unaffected.
+Disconnecting or cancelling the tool call discards pending notifications and
+stops buffering new ones. The worker still holds its concurrency slot until it
+finishes.
+
 Parameters are the same as `dispatch` except `return_ref`/`summary_chars` (streaming is incompatible with ref-mode) and `group` (group context injection is supported only on `dispatch` and per-item in `dispatch_parallel`).
 
 ### `dispatch_dialogue`
@@ -426,7 +433,7 @@ Register a new project directory as an agent. Description is auto-generated from
 | `directory` | string | yes | Path to an existing project directory (`~` is expanded, relative paths resolved) |
 | `description` | string | no | What this agent can do — auto-generated if empty |
 | `timeout` | int | no | Timeout in seconds (0 = 300; this is a literal default, not `settings.default_timeout`) |
-| `max_budget_usd` | float | no | Max cost in USD per dispatch (0 = inherit `settings.default_max_budget_usd`; no cap only when that is unset too) |
+| `max_budget_usd` | float | no | Finite max cost in USD per dispatch (0 = inherit `settings.default_max_budget_usd`; no cap only when that is unset too) |
 | `permission_mode` | string | no | Permission mode (e.g. `default`, `plan`, `bypassPermissions`) |
 | `allowed_tools` | string | no | Comma-separated allowed tools (e.g. `"Bash,Read,Edit"`) |
 | `disallowed_tools` | string | no | Comma-separated disallowed tools |
@@ -452,7 +459,7 @@ Update an existing agent's configuration. Only non-empty fields are changed. Pas
 | `risky_capabilities` | string | no | Comma-separated. `"none"` to clear |
 | `instructions` | string | no | Standing orders for every dispatch (replaces the text). `"none"` to clear |
 
-Changing an agent's config drops that agent's cached results — the cache key holds the agent *name*, so a re-pointed or re-permissioned agent would otherwise keep answering from the previous config for the rest of the TTL. The same applies to `add_agent` and `remove_agent`.
+Changing an agent's config through an MCP mutation tool drops its cached results immediately. Cache keys also include the agent configuration and dispatch defaults, so edits through the CLI, another server, or YAML take effect on the next request. A worker finishing with an older configuration cannot overwrite answers for the new configuration. Unchanged agents keep their cache.
 
 ### `remove_agent`
 
@@ -507,11 +514,11 @@ dispatch_wait(job_id="8f3a...e1", timeout_seconds=120)
   -> {"id": "...", "status": "running", "timed_out_waiting": true}
 ```
 
-`dispatch_cancel(job_id)` cancels a **pending** job, and also kills a **running** job's `claude` subprocess when the job was started by the same server instance (the job is marked `cancelled` first, so the worker's trailing write can't undo it; partial work is lost but the progress tail is preserved). A running job started by a *previous* server run can't be killed safely and is left to finish. The response carries an `outcome` of `cancelled`, `cancelled_running`, `running` (not owned by this server), `already_terminal`, or `not_found`.
+`dispatch_cancel(job_id)` cancels a **pending** job, and also kills a **running** job's `claude` process group when the job was started by the same server instance (the job is marked `cancelled` first, so the worker's trailing write can't undo it; partial work is lost but the progress tail is preserved). On POSIX, descendants in that process group are killed too, releasing inherited pipes and the concurrency slot. A cancel that lands after the job was claimed but before its process spawned is honored too: the process is killed as soon as it starts. A compatibility retry that races cancellation is killed when it registers. A running job started by a *previous* server run can't be killed safely and is left to finish. The response carries an `outcome` of `cancelled`, `cancelled_running`, `running` (not owned by this server), `already_terminal`, or `not_found`.
 
 Async workers run with streaming under the hood: the job file keeps a rolling tail (last 20 lines, ~1 write/sec) of assistant text and tool-use events. `dispatch_status` shows it as `progress` while the job runs and keeps it afterwards as a post-mortem trace; `dispatch_jobs` shows `last_progress` for running jobs.
 
-`dispatch_jobs(status?)` lists recent jobs as summaries (filter by `pending` / `running` / `done` / `failed` / `cancelled`). `dispatch_gc(max_age_days=7)` purges terminal jobs older than the threshold — pending and running jobs are never deleted.
+`dispatch_jobs(status?, limit=50, agent?)` lists recent jobs as summaries (filter by `pending` / `running` / `done` / `failed` / `cancelled`). `agent` matches an exact name, including agents since removed from configuration; both filters apply before the limit. `dispatch_gc(max_age_days=7)` purges terminal jobs older than the threshold — pending and running jobs are never deleted. Use `dry_run=True` to preview eligible job IDs without deleting them.
 
 Job state persists to disk at `~/.config/agent-dispatch/jobs/` (override with `AGENT_DISPATCH_JOBS_DIR`). One JSON file per job, written owner-only (`0o600`) with atomic writes — safe to read or `ls` while jobs are in flight. Caller-supplied `job_id`s are validated as 32-char hex before any file access (no path traversal). On startup the server recovers jobs a crashed instance abandoned: `running` ones stuck over an hour, and `pending` ones over 24 hours, are marked `failed` so they stop being polled forever and become collectable by `dispatch_gc`. (The `pending` threshold is deliberately long — the jobs directory is shared by every running server, so a job queued behind another server's concurrency limit must not be swept.)
 
@@ -545,7 +552,7 @@ Failures are deterministic: check `success`, then branch on `error_type`.
 | `error_type` | Meaning | Recovery |
 |--------------|---------|----------|
 | `permission` | A tool call was denied | `update_agent(name, allowed_tools="Bash,Read")` (least privilege) or `update_agent(name, permission_mode="bypassPermissions")`, then re-dispatch. The `error` text includes a hint with the exact fix. |
-| `timeout` | Process killed at the timeout | Resume the partial work: `dispatch_session(agent, "Continue where you left off", session_id=<from the error text>)`. Or retry with a bigger `timeout_seconds=` — once the journal has history the error names the value that would have covered this agent's p90 — or use `dispatch_async`. A *streaming* dispatch that produced its answer before the deadline returns that answer with a `hint` instead of failing. |
+| `timeout` | Process group killed at the timeout | Resume the partial work: `dispatch_session(agent, "Continue where you left off", session_id=<from the error text>)`. Or retry with a bigger `timeout_seconds=` — once the journal has history the error names the value that would have covered this agent's p90 — or use `dispatch_async`. Both ordinary and streaming dispatches preserve a complete CLI result received before termination; a successful result includes a cleanup `hint`. Partial output remains a timeout. |
 | `not_found` | Agent directory or `claude` CLI missing | `list_agents()` → check `healthy`. Re-add the agent with an existing path, or run `agent-dispatch doctor` to find what's missing. |
 | `recursion` | Dispatch nesting exceeded `max_dispatch_depth` (default 3) | Don't dispatch from dispatched agents; if the nesting is intentional, raise `max_dispatch_depth` in settings. |
 | `budget` | The `claude` CLI ended the session at the `max_budget_usd` spend cap — the answer is incomplete | Raise the cap (`update_agent(name, max_budget_usd=2.0)`), switch to a cheaper `model`, or split the task. The partial session is resumable: `dispatch_session(agent, "Continue where you left off", session_id=<from the result>)`. |
@@ -617,7 +624,7 @@ settings:
   #   - Read
   #   - Edit
   max_dispatch_depth: 3     # recursion protection
-  max_concurrency: 5        # max parallel claude -p processes (per dispatch path)
+  max_concurrency: 5        # max parallel claude -p processes per server, across all tools
   # dispatch_protocol: true # send every agent the dispatch protocol (non-interactive,
   #                         # time budget, STATUS line → `outcome`). false = raw claude -p.
   # usage_log: true         # record every dispatch in usage.jsonl (cost, duration, outcome).
@@ -689,8 +696,19 @@ past dispatches and deleting them cannot be undone. Pending and running jobs are
 never touched.
 
 `agent-dispatch gc --days N` and the `dispatch_gc` tool apply the same rule as a
-one-off. Both *delete* immediately and report the count; neither previews, so
-check what is there first with `agent-dispatch jobs`.
+one-off. Preview eligible records before deleting them:
+
+```bash
+agent-dispatch gc --days 30 --dry-run
+agent-dispatch gc --days 30
+```
+
+`--dry-run` also works with `--all`. The MCP equivalent is
+`dispatch_gc(max_age_days=30, dry_run=True)`, returning `dry_run: true`,
+`purged: 0`, `would_purge`, `job_ids`, and `max_age_days`.
+A preview is a snapshot; deletion checks eligibility again, so its count can
+change as jobs finish. Unreadable records, invalid timestamps, and records whose
+internal ID differs from their filename are skipped and preserved for inspection.
 
 ### Auto-Description
 
@@ -725,8 +743,8 @@ Your Claude Code session
   ▼
 agent-dispatch MCP server
   ├─ cache check → hit? return cached result
-  ├─ semaphore → limit concurrent processes
-  └─ subprocess.run("claude -p ...", cwd=~/projects/infra/)
+  ├─ shared process limit → bound concurrency across all dispatch tools
+  └─ subprocess.Popen(["claude", "-p", ...], cwd=~/projects/infra/)
        │
        ▼
      New Claude Code session in ~/projects/infra/
@@ -745,9 +763,9 @@ agent-dispatch MCP server
 - **Path-traversal guard** — caller-supplied `job_id`/`ref` values are validated as 32-char hex before any filesystem access.
 - **Owner-only state** — job files, `agents.yaml`, the usage journal and the server registry are all written `0o600`; their directories are `0o700`.
 - **Cost control** — `max_budget_usd` per agent or globally is passed to the `claude` CLI as `--max-budget-usd`, so a runaway dispatch is stopped at the cap and comes back as `error_type: "budget"` with a resumable `session_id`. An overshoot that lands over budget without stopping is flagged post-hoc with `budget_exceeded: true` + a hint.
-- **Concurrency** — `max_concurrency` (default: 5) caps parallel `claude -p` processes. Note: the sync and async dispatch paths use separate semaphores, so the worst-case total is `2 × max_concurrency`.
-- **Timeout** — per-agent or global (default: 300s). A streaming dispatch runs the agent in its own process group, so the deadline kills the whole tree: a process the agent left running in the background can't hold the dispatch (and its concurrency slot) open past the timeout.
-- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, results flagged `budget_exceeded`, and results the agent itself reported as `partial` / `blocked` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
+- **Concurrency** — `max_concurrency` (default: 5) caps parallel `claude -p` processes across all dispatch tools within one server process. Ordinary, streaming, and background dispatches share the same limit, and waiting calls are served in arrival order, so a queue of background jobs cannot starve an interactive call. Cancelling a waiting call never starts its process; cancelling an already-started ordinary or streaming call keeps its slot occupied until the worker ends. Reloading a changed limit preserves occupied slots: lowering it lets existing work finish and holds new launches until capacity is available. Separate server processes each have their own limit.
+- **Timeout** — per-agent or global (default: 300s). Both ordinary and streaming dispatches run the agent in its own process group. On POSIX, the deadline kills that group, including descendants that inherited output pipes. Ordinary dispatch also bounds cleanup to one additional second if a descendant deliberately detached from the group; that detached process is outside the group's reach. A complete CLI result survives cleanup timeout, with a hint on successful results.
+- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests under the same agent configuration and dispatch defaults return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, results flagged `budget_exceeded`, and results the agent itself reported as `partial` / `blocked` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
 - **Durable config** — `agents.yaml` is written atomically (temp file + rename), so an interrupted write can never truncate it. Every mutation path (CLI and MCP server alike) also takes a cross-process advisory lock, so concurrent edits don't drop one another's agents. The lock is best-effort by design: after waiting 10 seconds it logs a warning and proceeds anyway, because a wedged lock holder must not freeze the MCP server — so on a heavily contended config a lost update is possible, while a truncated one is not.
 
 See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassPermissions` escalation risk and on-disk job files).
@@ -765,13 +783,50 @@ See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassP
 | `agent-dispatch describe <name>` | Show full configuration for one agent (tri-state tools, project files) |
 | `agent-dispatch test <name> [task] [--stream]` | Test an agent with a dispatch (`--stream` for live progress) |
 | `agent-dispatch stats [--days N --agent X --json]` | What dispatches cost: spend, durations, outcomes and failures per agent |
-| `agent-dispatch doctor` | Diagnose installation: Claude CLI (incl. `--append-system-prompt` support), running servers on stale code, MCP registration, agent health, and group membership |
-| `agent-dispatch jobs [--status --limit]` | List async dispatch jobs (most recent first) |
-| `agent-dispatch job <id>` | Show one job: status, progress tail, result preview |
+| `agent-dispatch doctor [--json --strict]` | Diagnose installation, stale servers, MCP registration, agent health, and group membership; `--strict` also fails on warnings |
+| `agent-dispatch jobs [--status STATUS --agent NAME --limit N --json]` | List recent job summaries, including results stored with `return_ref` |
+| `agent-dispatch job <id> [--json]` | Show one job; `--json` exports the full record without truncation |
 | `agent-dispatch cancel <id>` | Cancel a pending job (running jobs: use the `dispatch_cancel` MCP tool) |
-| `agent-dispatch gc [--days N \| --all]` | Purge terminal jobs older than N days (default 7; `--all` purges every age) |
+| `agent-dispatch gc [--days N \| --all] [--dry-run]` | Purge terminal jobs older than N days (default 7; `--all` purges every age); `--dry-run` previews without deleting |
 | `agent-dispatch serve` | Start MCP server (stdio, used by Claude Code) |
 
+To check an installation from a script:
+
+```bash
+agent-dispatch doctor --json --strict > diagnostics.json
+```
+
+The report contains `schema_version: 1`, overall `status` (`ok`, `warn`, or
+`fail`), `issues` and `warnings` counts, and a `checks` array. Each check has a
+`section`, `status`, `message`, and `details` array containing any remediation
+steps or supporting information. JSON is emitted even when checks fail.
+Failures exit with code 1; warnings exit with code 0 unless `--strict` is set.
+The same checks and exit policy apply to the normal human-readable output.
+This checks installation and configuration; it does not run a paid dispatch.
+
+To inspect history from scripts or save a full result:
+
+```bash
+agent-dispatch jobs --agent infra --status failed --limit 10 --json
+agent-dispatch job <job_id> --json > job.json
+```
+
+`jobs --json` returns an array of compact summaries in the same format as
+`dispatch_jobs`, or `[]` if no jobs match. Each summary includes the ID, agent,
+status, first 120 task characters, and timestamps; result cost, success, outcome,
+error type, and latest running progress are included when available.
+`job --json` returns the full stored job, including task/context, progress,
+result text, structured result, session ID, and recovery hints when present.
+Both formats preserve Unicode. Reading a failed job still exits successfully:
+check its `status` and `result.success` to determine the dispatch outcome.
+Missing/invalid job IDs and storage-open failures return `{"error": "..."}`
+with exit code 1. Invalid command-line options are reported by Click on stderr.
+
+Invalid or non-finite cost metadata is treated as unknown and omitted from
+result exports; it does not hide the stored result text. Usage summaries ignore
+unusable numeric measurements, and doctor omits uptime when a server's start
+timestamp is unreadable.
+
 ## Requirements
 
 - Python >= 3.10