agent-dispatch 0.12.1__tar.gz → 0.14.0__tar.gz

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/.github/workflows/publish.yml

@@ -26,5 +26,12 @@ jobs:
           pip install build
           python -m build
 
+      # Keep this current with the build backend. The pinned action bundles its
+      # own twine, and twine rejects a Metadata-Version it does not know: the
+      # v0.13.0 publish failed because hatchling 1.32 emits `Metadata-Version:
+      # 2.5` while the previous pin (Feb 2026) shipped twine 6.1.0 / packaging
+      # 25.0. `python -m build` resolves the newest hatchling at build time, so
+      # this drifts on its own with no repo change — see the release checklist's
+      # `twine check` step, which catches it before a tag is cut.
       - name: Publish to PyPI
-        uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
+        uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/AGENTS.md

@@ -28,7 +28,7 @@ pip install -e ".[dev]"
 
 ```bash
 ruff check src/ tests/
-python3 -m pytest tests/ -v   # 578 tests, ~5s
+python3 -m pytest tests/ -v   # 637 tests, ~15s
 ```
 
 Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.which` + `subprocess.run`/`Popen`; server tests mock `_get_config` + `runner.dispatch`. The one exception is `TestStreamPipeHandling`, which spawns a short-lived *python* subprocess: a pipe deadlock lives in the OS pipe buffer, so a mocked `Popen` structurally cannot reproduce it.
@@ -52,16 +52,27 @@ Tests must **never** invoke the real `claude` CLI. Runner tests mock `shutil.whi
 - **Never `close()` a pipe another thread may still be reading.** `close()` waits on the reader's buffer lock with *no timeout*, so it would hang the dispatch forever — the bounded `join()` before it buys nothing. `dispatch_stream` skips the stderr close while the drain thread is alive and lets the daemon reader + Popen finalizer release the fd. This is not hypothetical: a stdio MCP server inherits the child's `stderr`, so the pipe often has no EOF even after `claude` exits cleanly.
 - **No `await` inside `config_lock()`.** `ProcessLock`'s in-process guard is a `threading.RLock` — re-entrant per *thread* — and every MCP tool coroutine runs on the one event-loop thread. Suspending in the critical section lets a second coroutine re-enter the "held" lock and interleave its own load/mutate/save. Collect warnings as data, emit them after the `with` block (`test_no_await_inside_the_config_lock` enforces this by AST).
 - Cross-process locks are acquired with a **bounded** wait, never a blocking `flock`: the server takes them on its event-loop thread, so a wedged holder would freeze every tool. After the deadline it proceeds unlocked and logs — a possible lost update beats a permanent freeze.
-- `recover_stale` sweeps `pending` on a **much longer** threshold than `running`: the jobs directory is shared by every `agent-dispatch serve`, so an hours-old pending job may still be queued behind another live server's semaphore.
+- `recover_stale` sweeps `pending` on a **much longer** threshold than `running`: the jobs directory is shared by every `agent-dispatch serve`, so an hours-old pending job may still be queued behind another live server's semaphore. For a *running* job, `started_at` alone does **not** prove abandonment — a dispatch may legitimately run to the 7200s timeout ceiling, so the file's own mtime is checked too (a live worker rewrites it on every progress flush). That check can only ever *skip* a recovery, never add one.
+- Deleting job records is **opt-in** (`settings.job_retention_days`, default `0` = off) and only ever happens at server start, never inside a tool. They are the user's dispatch history and the deletion is irreversible, so an unreadable config is treated as "do nothing" rather than falling back to a default retention.
+- `max_concurrency` must bound *subprocesses*, not coroutines. Cancelling a coroutine does not stop the thread behind `asyncio.to_thread`, so `async with sem:` gave the slot away while `claude` kept running and billing. Dispatches go through `_dispatch_guarded`, which releases from the future's done-callback and `shield`s the await. Never "simplify" it back to `async with`.
+- Tool responses are serialized through `server._dumps`, never bare `json.dumps`: the stdlib default (`ensure_ascii=True`) turns every non-ASCII character into a `\uXXXX` escape, tripling the bytes and tokenizing badly, for no gain — the stdio transport emits raw UTF-8 via pydantic anyway. A real `list_groups()` carried 8520 escapes and weighed 59 KB instead of 25 KB.
+- `agents.yaml` is parsed with libyaml's `CSafeLoader` when available (`config._YamlLoader`), falling back to the pure-Python **safe** loader — never `yaml.Loader`. The config is re-read on every tool call, so this parse is on the hot path of all 21 tools *and* blocks the event-loop thread: 9.80 ms → 0.76 ms on a real 38 KB config.
 - Pydantic does **not** validate on assignment. `Field(ge=...)` guards only the *load* path; every mutation surface (CLI `add`/`update`, MCP `add_agent`/`update_agent`) needs its own boundary check, or the bound escapes as a raw `ValidationError`.
 - Every state file (`agents.yaml`, job files) is written **temp file + `os.replace`**, never in place, and every load/mutate/save is wrapped in `config.ProcessLock` — the CLI and the MCP server are separate processes writing the same files, so a thread lock alone loses updates.
 - Anything that changes an agent's config must call `_invalidate_agent_cache` — the cache key holds the agent *name*, not its directory or permissions.
-- Only *clean* successes are cached: `cache.put` refuses failures, `denied_tools` results, and `budget_exceeded` results, so the documented "grant access, then re-dispatch" recovery is never short-circuited.
+- Only *clean* successes are cached: `cache.put` refuses failures, `denied_tools` results, `budget_exceeded` results, and results whose `outcome` is `partial`/`blocked`, so the documented "grant access / fix the cause, then re-dispatch" recovery is never short-circuited.
+- **The dispatch protocol goes in the system prompt, not the task.** `_build_system_prompt` (runner.py) renders the protocol (non-interactive, time/spend budget, lead with the outcome, trailing `STATUS:` line) plus the agent's `instructions`, and `_build_command` passes it as `--append-system-prompt` — on `--resume` too, since the CLI re-applies an appended prompt on every launch. `_build_prompt` (the `-p` text) is unchanged, so the cache key is unchanged. The text always starts with a `##` header line so it can never be read as a flag. `settings.dispatch_protocol: false` turns the protocol off (for CLIs that predate the flag; `doctor` probes `claude --help` for it); per-agent `instructions` still go through when set.
+- **`outcome` is lifted from the LAST line of the result** (`_split_outcome`), before JSON parsing, by the shared `_build_success_result` (the success-side twin of `_build_error_result` — both `dispatch` and `dispatch_stream` go through it) and by the plain-text fallback tier. A bare marker line is removed from `result`; a marker with a trailing reason stays. `outcome` never flips `success`; `partial`/`blocked` add a `hint` with the `dispatch_session(...)` continuation, after the denial hint. In JSON mode the protocol omits the STATUS instruction (the JSON footer governs), but a STATUS line that arrives anyway is still stripped so `parsed_result` survives.
 - Remediation text is a contract: a hint that names a flag must name one that exists (`test_printed_budget_hint_is_a_runnable_command` feeds the printed flags back into the CLI). Run the command you print.
+- The config error sets are declared **once** and in two halves: `config.CONFIG_LOAD_ERRORS` (read) and `config.CONFIG_SAVE_ERRORS` (write — `yaml.dump`'s `RepresenterError` is a `yaml.YAMLError`, therefore neither `OSError` nor `ConfigLoadError`, and used to escape both the MCP guard and the CLI's `_save_or_exit`). Two halves, not one set, because the remediations differ: a failed write is atomic so the old config survives, while a failed read needs the YAML fixed.
 - MCP tools that load config carry `@_config_guard` under `@mcp.tool()` so a broken `agents.yaml` — or a failed *write* — returns the `{"error": ...}` envelope instead of a raw traceback. The set of load errors lives in one place (`config.CONFIG_LOAD_ERRORS`) because three surfaces handle it: **`UnicodeDecodeError` is a `ValueError`, not an `OSError`**, and listing types per-site is exactly how a cp1251 config slipped past all three.
 
 - Tests must not touch anything outside `tmp_path`. `test_server.py`'s autouse `_reset_globals` and `test_cli.py`'s `_isolated_config` redirect **both** `AGENT_DISPATCH_CONFIG` and `AGENT_DISPATCH_JOBS_DIR`: a mutation tool that bails out early (unknown agent) still takes `config_lock()` first, which would otherwise create a lock file beside the developer's real config.
 
+- `mcp` is pinned **`>=1.2.0,<2`** deliberately: 2.0 removed `mcp.server.fastmcp`, which `server.py` imports, so an unbounded range gives every fresh install a dead `agent-dispatch serve`. Lifting the cap means porting to `mcp.server.mcpserver.MCPServer` — it is not a dependency bump.
+- Verify packaging in a **clean venv**, never the dev machine: build the wheel, install it fresh, import the server. A stale pin in local site-packages hides exactly the failure a new user hits first.
+- Run **`twine check dist/*`** before cutting the tag, with a *current* twine. The clean-venv check does not cover this: it proves the wheel installs, not that PyPI's uploader will accept its metadata. `python -m build` pulls the newest hatchling at build time, so the emitted `Metadata-Version` climbs on its own, and `pypa/gh-action-pypi-publish` is pinned to a SHA that bundles a fixed twine — v0.13.0's publish failed on exactly that mismatch (hatchling 1.32 → metadata 2.5; pinned twine 6.1.0 → "not a valid metadata version"). When it happens, bump the action pin rather than pinning the backend down.
+
 ## Deliberately not built
 
 These were considered — some fully implemented — and cut on purpose: an agent router / auto-dispatch (`recommend_agent` / `dispatch_auto`, removed before 0.8.0 — a keyword scorer adds little over the calling LLM at a handful of agents, and auto-dispatch can spend money or mutate a repo on a guess); groups as an execution engine (they are a descriptive layer — no routing, no per-group settings); an agent-dispatch-side budget ledger across dispatches (the CLI's own `--max-budget-usd` covers a single run; anything cumulative would need state we deliberately don't keep). Please open an issue with the use case before adding any of them.
@@ -76,4 +87,4 @@ Python ≥ 3.10 · `from __future__ import annotations` everywhere · Pydantic v
 
 ## More detail
 
-[README.md](README.md) documents every MCP tool with parameter tables, response shapes, and the error-recovery map — it doubles as the behavioral spec. The test suite (`tests/`, 578 tests) encodes the exact expected behavior of every layer: when in doubt, read the tests for the module you're touching (`test_runner.py`, `test_server.py`, `test_cli.py`, ...).
+[README.md](README.md) documents every MCP tool with parameter tables, response shapes, and the error-recovery map — it doubles as the behavioral spec. The test suite (`tests/`, 637 tests) encodes the exact expected behavior of every layer: when in doubt, read the tests for the module you're touching (`test_runner.py`, `test_server.py`, `test_cli.py`, ...).

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/CHANGELOG.md

@@ -7,6 +7,138 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.14.0] - 2026-09-08
+
+The delegation round: what a dispatched agent is *told*, and what it tells
+back. Until now `claude -p` was launched with the task and nothing else — it
+did not know it was being driven by another agent, that nobody would answer a
+question, how long it had, or how to report an unfinished job. The failure
+mode was concrete and billed: an ambiguous task ended in *"Could you clarify
+which service you mean?"*, `success: true`, cached for the whole TTL.
+
+### Added
+- **The dispatch protocol.** Every dispatch now appends a short system prompt
+  (`--append-system-prompt`, never mixed into the task text) telling the agent
+  that it is non-interactive and dispatched by `caller`, that it must state
+  assumptions instead of asking, that a denied tool is something to report and
+  work around rather than stop on, what its time budget (and spend cap) is, to
+  lead with the outcome — which is what makes a `return_ref` summary, the
+  *head* of the text, worth reading — and to end with one line:
+  `STATUS: done | partial | blocked`. Passed on `--resume` too. Verified live
+  against claude 2.1.263 (`agent-dispatch test <agent> --stream`).
+- **`DispatchResult.outcome`** — that trailing line, lifted out of `result`
+  into a field: `done`, `partial` or `blocked`, or absent when the agent did
+  not report one. It never flips `success`. `partial`/`blocked` come with a
+  `hint` carrying the exact `dispatch_session(..., session_id=...)` call to
+  continue, ride the `return_ref` payload and `dispatch_jobs` summaries, are
+  shown by `agent-dispatch test` / `job <id>`, are labelled for the
+  `dispatch_parallel` aggregator so a blocked member is not synthesized as a
+  finished one — and are **not cached**: whatever the agent was missing is not
+  in the cache key, so a retry after fixing it must run fresh.
+- **Per-agent `instructions`** — standing orders appended after the protocol
+  on every dispatch ("read-only SQL only", "never restart a stack unless the
+  task says so"). `add_agent`/`update_agent` (`"none"` clears), CLI `add` /
+  `update --instructions` (`none` clears), shown by `inspect_agent` and
+  `describe`, pruned from YAML when empty. Changing them invalidates the
+  agent's cache entries like any other config change.
+- **`settings.dispatch_protocol`** (default `true`) — set to `false` for a raw
+  `claude -p` (a CLI that predates `--append-system-prompt`, or an A/B). The
+  protocol and the instructions share one flag, so instructions still go
+  through when set. `doctor` now probes `claude --help` for the flag and warns
+  with the exact remediation when it is missing.
+
+### Changed
+- In `response_format="json"` mode the protocol omits the "lead with the
+  outcome" and STATUS bullets — the JSON footer governs the reply shape. A
+  STATUS line that arrives anyway is still stripped before parsing, so
+  `parsed_result` survives it.
+- `dispatch` and `dispatch_stream` build their success result through one
+  `_build_success_result`, the twin of `_build_error_result`.
+
+
+## [0.13.0] - 2026-08-13
+
+An efficiency round, measured against a real 38 KB config (4 agents, 6 groups),
+a 294-record jobs directory and 14 concurrently running servers — not against
+synthetic fixtures. Findings came from a five-axis audit (caller context,
+latency/IO, spend, CPU, concurrency) whose every claim was re-measured
+adversarially before implementation; most were confirmed as real but too small
+to matter next to a 10–120 s subprocess and were deliberately left alone.
+
+### Changed
+- **Tool responses no longer escape non-ASCII, cutting discovery payloads
+  roughly in half.** `json.dumps` defaults to `ensure_ascii=True`, so every
+  Cyrillic character left as a `\uXXXX` escape — 6 bytes for what UTF-8 stores
+  in 2, and a far worse token sequence. One `list_groups()` carried **8520**
+  such escapes: 59 KB where 25 KB was enough. Measured over a full discovery
+  pass (`list_agents` + `list_groups` + `inspect_group` + `inspect_agent`):
+  104 KB → 51 KB, about **13 000 tokens saved per pass**, on every call, in
+  every session. Nothing is lost: the stdio transport serializes the JSON-RPC
+  envelope with pydantic's `model_dump_json`, which already emits raw UTF-8 —
+  which is also why the `dispatch` family never had this problem while every
+  `json.dumps` tool did. All 57 call sites now go through one `_dumps` helper
+  so the two agree. `_dumps` probes encodability and falls back to the escaped
+  form for the one payload that needs it — a **lone surrogate**, which an agent
+  produces by printing a literal `\udXXX` escape in its JSON output (`json.loads`
+  manufactures it from perfectly ASCII input, so the subprocess-level
+  `errors="replace"` cannot help). Unescaped it would raise `UnicodeEncodeError`
+  inside the stdio transport *after* the tool returned — past every guard.
+  The saving is proportional to how much non-ASCII your config and results
+  carry: on an all-ASCII setup it is exactly zero.
+- **`agents.yaml` parses ~13x faster.** The config is re-read on every single
+  tool call (deliberately — that is how a new agent is picked up without a
+  restart), so the parse sits on the hot path of all 21 tools *and* runs on the
+  event-loop thread, where it blocks every other tool. Switching to libyaml's
+  `CSafeLoader` takes the real config from **9.80 ms to 0.76 ms** per call.
+  Falls back to the pure-Python safe loader when PyYAML was built without
+  libyaml; both are the *safe* loader, so a config still cannot construct
+  arbitrary objects.
+
+### Added
+- **`settings.job_retention_days`** — when > 0, terminal job records older than
+  N days are deleted at server start. Every async dispatch and every
+  `return_ref` dispatch leaves a job file that nothing removed on its own, so
+  the directory grew forever while `dispatch_jobs` and stale-job recovery
+  parsed every file in it. **Defaults to `0` (off)**: those records are the
+  user's own dispatch history and deleting them is irreversible, so it is an
+  explicit opt-in rather than something a version bump starts doing to an
+  existing install. An unreadable config is treated as `0`, never guessed.
+
+### Fixed
+- **`max_concurrency` bounds live subprocesses again, not live coroutines.**
+  `async with sem:` released the slot when the *coroutine* unwound, but
+  cancelling a coroutine does not stop the thread behind `asyncio.to_thread` —
+  so an interrupted turn or a closed session handed the slot to the next
+  dispatch while the `claude` subprocess kept running and kept being billed. N
+  cancellations meant up to N extra concurrent subprocesses, and each abandoned
+  worker also held a thread out of the default executor (`min(32, cpu_count+4)`,
+  which `to_thread` needs for *every* dispatch) for the rest of its timeout —
+  up to 7200s. The slot is now tied to the worker's real lifetime via
+  `_dispatch_guarded`; cancellation still reaches the caller immediately.
+- **A legitimately long dispatch is no longer flipped to `failed` by an
+  unrelated server starting up.** `recover_stale` judged abandonment from
+  `started_at` alone, but a dispatch may legitimately run up to the 7200s
+  timeout ceiling — so any of the (routinely 14+) other servers booting would
+  mark a live 2-hour job `failed`, after which `finish()` refuses the real,
+  already-paid-for result. A *running* job whose file was modified within the
+  threshold is now left alone: a live worker rewrites it on every progress
+  flush, so only the file's own age proves abandonment. This can only ever skip
+  a recovery — a genuinely abandoned job is picked up on a later start.
+- **A config that cannot be *rendered* now returns an error envelope instead of
+  a traceback.** `_get_config` wraps read-side YAML failures into
+  `ConfigLoadError`, but `save_config` → `yaml.dump` raises `RepresenterError` —
+  a `yaml.YAMLError`, so neither an `OSError` nor a `ConfigLoadError`. It
+  therefore slipped past *both* arms of the MCP guard on
+  `add_agent`/`update_agent`/`remove_agent` **and** past the CLI's
+  `_save_or_exit`. Unreachable today (only JSON-native types are ever dumped)
+  and closed now because it is the same "an exception type escapes the handler
+  meant to catch it" class that produced the 0.12.0 and 0.12.1 rounds. The set
+  is declared once as `config.CONFIG_SAVE_ERRORS`, next to its read-side twin,
+  so the two surfaces cannot drift.
+- **Stale-job recovery scans the jobs directory once instead of twice.** It
+  called `list()` per status, and `list()` reads and parses every file in the
+  directory (~85 ms at 294 records) — at the start of every server process.
+
 ## [0.12.1] - 2026-07-30
 
 Two holes in 0.12.0's own "tools always return a clean error" fix.

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: agent-dispatch
-Version: 0.12.1
+Version: 0.14.0
 Summary: MCP server that lets Claude Code agents delegate tasks to agents in other project directories
 Project-URL: Homepage, https://github.com/ginkida/agent-dispatch
 Project-URL: Repository, https://github.com/ginkida/agent-dispatch
@@ -220,7 +220,8 @@ dispatch(
   "session_id": "sess-abc-123",
   "cost_usd": 0.02,
   "duration_ms": 5000,
-  "num_turns": 2
+  "num_turns": 2,
+  "outcome": "done"
 }
 
 // Response (failure — error_type helps you handle programmatically)
@@ -278,6 +279,30 @@ Error: TypeError at scheduler.py:42
 Check container logs for recent errors related to the scheduler service
 ```
 
+**The dispatch protocol and `outcome`.** `claude -p` on its own does not know it is being driven by another agent: given an ambiguous task it will happily end with *"Could you clarify which service you mean?"* — a billed run that answered nothing, and one that a plain cache would then serve again for the whole TTL. So every dispatch also appends a short **dispatch protocol** to the agent's system prompt (`--append-system-prompt`, never mixed into your task text):
+
+- it is running non-interactively, dispatched by `caller`, and nobody will answer a question or approve an action — state the assumption and proceed, take the safer reading when two differ;
+- a denied tool or a missing thing is something to report, not something to stop on — finish everything else that is possible;
+- its time budget is about the agent's timeout (and its spend cap, if one is set) — scope the work to fit; a complete partial answer beats an unfinished perfect one;
+- lead with the outcome, then the evidence; list what could not be done and why (this is what makes a `return_ref` summary — the **head** of the text — worth reading);
+- end with one line, `STATUS: done`, `STATUS: partial` or `STATUS: blocked`.
+
+That last line is lifted out of `result` into **`outcome`** — the agent's own verdict, deterministic to check: `"done"` means complete; `"partial"` / `"blocked"` mean the agent says the work is unfinished, the result names what is missing, and a `hint` spells out the continuation (usually `dispatch_session(..., session_id=...)`). Read `outcome` before reading the text. It never flips `success`, `partial` / `blocked` results are **not cached**, and a `dispatch_parallel(..., aggregate=...)` labels such members for the aggregator so a blocked report is not synthesized as a finished one. Absent when the agent did not report one — the protocol is off (`settings.dispatch_protocol: false`), `response_format="json"` was requested (the JSON footer governs the reply shape there), or the agent simply skipped it.
+
+**Standing orders.** Per-agent `instructions` (set via `add_agent` / `update_agent` or `agent-dispatch update <name> --instructions "..."`) follow the protocol in the same system prompt on **every** dispatch — "read-only SQL only", "never restart a stack unless the task says so", "answer with exact log lines". Unlike `context`, which is per call, and unlike the project's own `CLAUDE.md`, which is written for an interactive session, these are the rules for *being dispatched*.
+
+```json
+// Response (success, but the agent says it did not finish)
+{
+  "agent": "infra",
+  "success": true,
+  "result": "Restarted horizon. Could not verify the queue drained: the redis container is not reachable from here.",
+  "session_id": "sess-abc-123",
+  "outcome": "partial",
+  "hint": "The agent reports its work is PARTIAL — read the result for what is missing, then continue in the same session via dispatch_session(agent='infra', task='Continue where you left off', session_id='sess-abc-123') or re-dispatch with what it needed."
+}
+```
+
 ### `dispatch_session`
 
 Multi-turn: continue a conversation with an agent. First call starts a session, pass `session_id` back to continue. Never cached.
@@ -395,6 +420,7 @@ Register a new project directory as an agent. Description is auto-generated from
 | `disallowed_tools` | string | no | Comma-separated disallowed tools |
 | `capabilities` | string | no | Comma-separated capability labels (e.g. `"docker_logs,deploy_debug"`) |
 | `risky_capabilities` | string | no | Comma-separated high-risk labels (e.g. `"restart_services"`) |
+| `instructions` | string | no | Standing orders appended to the agent's system prompt on every dispatch (see [the dispatch protocol](#dispatch)) |
 
 ### `update_agent`
 
@@ -412,6 +438,7 @@ Update an existing agent's configuration. Only non-empty fields are changed. Pas
 | `disallowed_tools` | string | no | Comma-separated. `"none"` to clear |
 | `capabilities` | string | no | Comma-separated. `"none"` to clear |
 | `risky_capabilities` | string | no | Comma-separated. `"none"` to clear |
+| `instructions` | string | no | Standing orders for every dispatch (replaces the text). `"none"` to clear |
 
 Changing an agent's config drops that agent's cached results — the cache key holds the agent *name*, so a re-pointed or re-permissioned agent would otherwise keep answering from the previous config for the rest of the TTL. The same applies to `add_agent` and `remove_agent`.
 
@@ -515,6 +542,7 @@ Failures are deterministic: check `success`, then branch on `error_type`.
 Three soft signals that arrive with `success: true`:
 
 - **`denied_tools` + `hint`** — the agent finished but some tool calls were blocked; the result may be incomplete. Grant access (see the `permission` row) and re-dispatch.
+- **`outcome: "partial"` / `"blocked"`** — the agent's own verdict that it did not finish; the result names what is missing and the `hint` carries the `dispatch_session(...)` call to continue in the same session. Not cached, so a re-dispatch after fixing the cause runs fresh.
 - **`parsed_result: null` with `response_format="json"`** — the reply wasn't valid JSON; the raw text is still in `result`. Caveat: an agent that *can't* comply returns `{"error": "<reason>"}` — which parses successfully — so also check `parsed_result` for an `"error"` key.
 - **`budget_exceeded: true`** — `cost_usd` came in over the agent's `max_budget_usd` (or the settings default) without the CLI stopping the run (the final turn can overshoot the cap). The dispatch is not failed — the money is already spent — but a runaway agent is now visible. Tighten the task, pick a cheaper model, or raise the budget. A run the CLI *did* stop fails with `error_type: "budget"` instead.
 
@@ -539,6 +567,9 @@ agents:
       - deploy_debug
     risky_capabilities:     # high-risk labels, surfaced for visibility
       - restart_services
+    # instructions: |       # standing orders, appended to the system prompt on every dispatch
+    #   Read-only: never restart or redeploy unless the task says so.
+    #   Quote exact log lines with timestamps.
     # model: sonnet         # optional model override
     # max_budget_usd: 1.0   # cost limit per dispatch
     # permission_mode: bypassPermissions  # one of: default | plan | bypassPermissions
@@ -574,6 +605,9 @@ settings:
   #   - Edit
   max_dispatch_depth: 3     # recursion protection
   max_concurrency: 5        # max parallel claude -p processes (per dispatch path)
+  # dispatch_protocol: true # send every agent the dispatch protocol (non-interactive,
+  #                         # time budget, STATUS line → `outcome`). false = raw claude -p.
+  # job_retention_days: 30  # 0 (default) = never prune. See "Job retention" below.
   cache:
     enabled: true
     ttl: 300                # seconds
@@ -582,6 +616,30 @@ settings:
 
 Config is reloaded on every tool call — add agents without restarting.
 
+### Job retention
+
+Every `dispatch_async` **and** every `dispatch(..., return_ref=True)` writes a
+record to `~/.config/agent-dispatch/jobs/`, and nothing deletes it on its own —
+`dispatch_gc` has to be run by hand. The directory therefore grows without
+bound, and `dispatch_jobs` plus the stale-job recovery that runs at every server
+start read and parse *every* file in it.
+
+Set `job_retention_days` to prune terminal (done/failed/cancelled) records older
+than N days when a server starts:
+
+```yaml
+settings:
+  job_retention_days: 30
+```
+
+It defaults to `0` — **off** — because those records are your own history of
+past dispatches and deleting them cannot be undone. Pending and running jobs are
+never touched.
+
+`agent-dispatch gc --days N` and the `dispatch_gc` tool apply the same rule as a
+one-off. Both *delete* immediately and report the count; neither previews, so
+check what is there first with `agent-dispatch jobs`.
+
 ### Auto-Description
 
 `agent-dispatch add` without `--description` generates one from:
@@ -621,8 +679,9 @@ agent-dispatch MCP server
        ▼
      New Claude Code session in ~/projects/infra/
        ├─ Inherits: CLAUDE.md, .mcp.json, project tools
+       ├─ System prompt += dispatch protocol + the agent's standing `instructions`
        ├─ Receives structured prompt with goal/caller/context/task
-       └─ Returns result → cached for future identical requests
+       └─ Returns result (+ its own STATUS → `outcome`) → cached when complete
 ```
 
 ## Safety
@@ -634,7 +693,7 @@ agent-dispatch MCP server
 - **Cost control** — `max_budget_usd` per agent or globally is passed to the `claude` CLI as `--max-budget-usd`, so a runaway dispatch is stopped at the cap and comes back as `error_type: "budget"` with a resumable `session_id`. An overshoot that lands over budget without stopping is flagged post-hoc with `budget_exceeded: true` + a hint.
 - **Concurrency** — `max_concurrency` (default: 5) caps parallel `claude -p` processes. Note: the sync and async dispatch paths use separate semaphores, so the worst-case total is `2 × max_concurrency`.
 - **Timeout** — per-agent or global (default: 300s). A streaming dispatch runs the agent in its own process group, so the deadline kills the whole tree: a process the agent left running in the background can't hold the dispatch (and its concurrency slot) open past the timeout.
-- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, and results flagged `budget_exceeded` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
+- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, results flagged `budget_exceeded`, and results the agent itself reported as `partial` / `blocked` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
 - **Durable config** — `agents.yaml` is written atomically (temp file + rename), so an interrupted write can never truncate it. Every mutation path (CLI and MCP server alike) also takes a cross-process advisory lock, so concurrent edits don't drop one another's agents. The lock is best-effort by design: after waiting 10 seconds it logs a warning and proceeds anyway, because a wedged lock holder must not freeze the MCP server — so on a heavily contended config a lost update is possible, while a truncated one is not.
 
 See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassPermissions` escalation risk and on-disk job files).
@@ -645,13 +704,13 @@ See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassP
 |---------|-------------|
 | `agent-dispatch init` | Create config + register MCP server with Claude Code |
 | `agent-dispatch add <name> <dir>` | Add an agent (auto-generates description) |
-| `agent-dispatch update <name>` | Update agent config (permissions, timeout, model, etc.) |
+| `agent-dispatch update <name>` | Update agent config (permissions, timeout, model, `--instructions`, etc.) |
 | `agent-dispatch remove <name>` | Remove an agent |
 | `agent-dispatch list` | List agents with health status and permissions |
 | `agent-dispatch group <add\|list\|inspect\|update\|remove>` | Manage [groups](#groups) — cross-project working sets of agents |
 | `agent-dispatch describe <name>` | Show full configuration for one agent (tri-state tools, project files) |
 | `agent-dispatch test <name> [task] [--stream]` | Test an agent with a dispatch (`--stream` for live progress) |
-| `agent-dispatch doctor` | Diagnose installation: Claude CLI, MCP registration, agent health, and group membership |
+| `agent-dispatch doctor` | Diagnose installation: Claude CLI (incl. `--append-system-prompt` support), MCP registration, agent health, and group membership |
 | `agent-dispatch jobs [--status --limit]` | List async dispatch jobs (most recent first) |
 | `agent-dispatch job <id>` | Show one job: status, progress tail, result preview |
 | `agent-dispatch cancel <id>` | Cancel a pending job (running jobs: use the `dispatch_cancel` MCP tool) |

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/README.md

@@ -190,7 +190,8 @@ dispatch(
   "session_id": "sess-abc-123",
   "cost_usd": 0.02,
   "duration_ms": 5000,
-  "num_turns": 2
+  "num_turns": 2,
+  "outcome": "done"
 }
 
 // Response (failure — error_type helps you handle programmatically)
@@ -248,6 +249,30 @@ Error: TypeError at scheduler.py:42
 Check container logs for recent errors related to the scheduler service
 ```
 
+**The dispatch protocol and `outcome`.** `claude -p` on its own does not know it is being driven by another agent: given an ambiguous task it will happily end with *"Could you clarify which service you mean?"* — a billed run that answered nothing, and one that a plain cache would then serve again for the whole TTL. So every dispatch also appends a short **dispatch protocol** to the agent's system prompt (`--append-system-prompt`, never mixed into your task text):
+
+- it is running non-interactively, dispatched by `caller`, and nobody will answer a question or approve an action — state the assumption and proceed, take the safer reading when two differ;
+- a denied tool or a missing thing is something to report, not something to stop on — finish everything else that is possible;
+- its time budget is about the agent's timeout (and its spend cap, if one is set) — scope the work to fit; a complete partial answer beats an unfinished perfect one;
+- lead with the outcome, then the evidence; list what could not be done and why (this is what makes a `return_ref` summary — the **head** of the text — worth reading);
+- end with one line, `STATUS: done`, `STATUS: partial` or `STATUS: blocked`.
+
+That last line is lifted out of `result` into **`outcome`** — the agent's own verdict, deterministic to check: `"done"` means complete; `"partial"` / `"blocked"` mean the agent says the work is unfinished, the result names what is missing, and a `hint` spells out the continuation (usually `dispatch_session(..., session_id=...)`). Read `outcome` before reading the text. It never flips `success`, `partial` / `blocked` results are **not cached**, and a `dispatch_parallel(..., aggregate=...)` labels such members for the aggregator so a blocked report is not synthesized as a finished one. Absent when the agent did not report one — the protocol is off (`settings.dispatch_protocol: false`), `response_format="json"` was requested (the JSON footer governs the reply shape there), or the agent simply skipped it.
+
+**Standing orders.** Per-agent `instructions` (set via `add_agent` / `update_agent` or `agent-dispatch update <name> --instructions "..."`) follow the protocol in the same system prompt on **every** dispatch — "read-only SQL only", "never restart a stack unless the task says so", "answer with exact log lines". Unlike `context`, which is per call, and unlike the project's own `CLAUDE.md`, which is written for an interactive session, these are the rules for *being dispatched*.
+
+```json
+// Response (success, but the agent says it did not finish)
+{
+  "agent": "infra",
+  "success": true,
+  "result": "Restarted horizon. Could not verify the queue drained: the redis container is not reachable from here.",
+  "session_id": "sess-abc-123",
+  "outcome": "partial",
+  "hint": "The agent reports its work is PARTIAL — read the result for what is missing, then continue in the same session via dispatch_session(agent='infra', task='Continue where you left off', session_id='sess-abc-123') or re-dispatch with what it needed."
+}
+```
+
 ### `dispatch_session`
 
 Multi-turn: continue a conversation with an agent. First call starts a session, pass `session_id` back to continue. Never cached.
@@ -365,6 +390,7 @@ Register a new project directory as an agent. Description is auto-generated from
 | `disallowed_tools` | string | no | Comma-separated disallowed tools |
 | `capabilities` | string | no | Comma-separated capability labels (e.g. `"docker_logs,deploy_debug"`) |
 | `risky_capabilities` | string | no | Comma-separated high-risk labels (e.g. `"restart_services"`) |
+| `instructions` | string | no | Standing orders appended to the agent's system prompt on every dispatch (see [the dispatch protocol](#dispatch)) |
 
 ### `update_agent`
 
@@ -382,6 +408,7 @@ Update an existing agent's configuration. Only non-empty fields are changed. Pas
 | `disallowed_tools` | string | no | Comma-separated. `"none"` to clear |
 | `capabilities` | string | no | Comma-separated. `"none"` to clear |
 | `risky_capabilities` | string | no | Comma-separated. `"none"` to clear |
+| `instructions` | string | no | Standing orders for every dispatch (replaces the text). `"none"` to clear |
 
 Changing an agent's config drops that agent's cached results — the cache key holds the agent *name*, so a re-pointed or re-permissioned agent would otherwise keep answering from the previous config for the rest of the TTL. The same applies to `add_agent` and `remove_agent`.
 
@@ -485,6 +512,7 @@ Failures are deterministic: check `success`, then branch on `error_type`.
 Three soft signals that arrive with `success: true`:
 
 - **`denied_tools` + `hint`** — the agent finished but some tool calls were blocked; the result may be incomplete. Grant access (see the `permission` row) and re-dispatch.
+- **`outcome: "partial"` / `"blocked"`** — the agent's own verdict that it did not finish; the result names what is missing and the `hint` carries the `dispatch_session(...)` call to continue in the same session. Not cached, so a re-dispatch after fixing the cause runs fresh.
 - **`parsed_result: null` with `response_format="json"`** — the reply wasn't valid JSON; the raw text is still in `result`. Caveat: an agent that *can't* comply returns `{"error": "<reason>"}` — which parses successfully — so also check `parsed_result` for an `"error"` key.
 - **`budget_exceeded: true`** — `cost_usd` came in over the agent's `max_budget_usd` (or the settings default) without the CLI stopping the run (the final turn can overshoot the cap). The dispatch is not failed — the money is already spent — but a runaway agent is now visible. Tighten the task, pick a cheaper model, or raise the budget. A run the CLI *did* stop fails with `error_type: "budget"` instead.
 
@@ -509,6 +537,9 @@ agents:
       - deploy_debug
     risky_capabilities:     # high-risk labels, surfaced for visibility
       - restart_services
+    # instructions: |       # standing orders, appended to the system prompt on every dispatch
+    #   Read-only: never restart or redeploy unless the task says so.
+    #   Quote exact log lines with timestamps.
     # model: sonnet         # optional model override
     # max_budget_usd: 1.0   # cost limit per dispatch
     # permission_mode: bypassPermissions  # one of: default | plan | bypassPermissions
@@ -544,6 +575,9 @@ settings:
   #   - Edit
   max_dispatch_depth: 3     # recursion protection
   max_concurrency: 5        # max parallel claude -p processes (per dispatch path)
+  # dispatch_protocol: true # send every agent the dispatch protocol (non-interactive,
+  #                         # time budget, STATUS line → `outcome`). false = raw claude -p.
+  # job_retention_days: 30  # 0 (default) = never prune. See "Job retention" below.
   cache:
     enabled: true
     ttl: 300                # seconds
@@ -552,6 +586,30 @@ settings:
 
 Config is reloaded on every tool call — add agents without restarting.
 
+### Job retention
+
+Every `dispatch_async` **and** every `dispatch(..., return_ref=True)` writes a
+record to `~/.config/agent-dispatch/jobs/`, and nothing deletes it on its own —
+`dispatch_gc` has to be run by hand. The directory therefore grows without
+bound, and `dispatch_jobs` plus the stale-job recovery that runs at every server
+start read and parse *every* file in it.
+
+Set `job_retention_days` to prune terminal (done/failed/cancelled) records older
+than N days when a server starts:
+
+```yaml
+settings:
+  job_retention_days: 30
+```
+
+It defaults to `0` — **off** — because those records are your own history of
+past dispatches and deleting them cannot be undone. Pending and running jobs are
+never touched.
+
+`agent-dispatch gc --days N` and the `dispatch_gc` tool apply the same rule as a
+one-off. Both *delete* immediately and report the count; neither previews, so
+check what is there first with `agent-dispatch jobs`.
+
 ### Auto-Description
 
 `agent-dispatch add` without `--description` generates one from:
@@ -591,8 +649,9 @@ agent-dispatch MCP server
        ▼
      New Claude Code session in ~/projects/infra/
        ├─ Inherits: CLAUDE.md, .mcp.json, project tools
+       ├─ System prompt += dispatch protocol + the agent's standing `instructions`
        ├─ Receives structured prompt with goal/caller/context/task
-       └─ Returns result → cached for future identical requests
+       └─ Returns result (+ its own STATUS → `outcome`) → cached when complete
 ```
 
 ## Safety
@@ -604,7 +663,7 @@ agent-dispatch MCP server
 - **Cost control** — `max_budget_usd` per agent or globally is passed to the `claude` CLI as `--max-budget-usd`, so a runaway dispatch is stopped at the cap and comes back as `error_type: "budget"` with a resumable `session_id`. An overshoot that lands over budget without stopping is flagged post-hoc with `budget_exceeded: true` + a hint.
 - **Concurrency** — `max_concurrency` (default: 5) caps parallel `claude -p` processes. Note: the sync and async dispatch paths use separate semaphores, so the worst-case total is `2 × max_concurrency`.
 - **Timeout** — per-agent or global (default: 300s). A streaming dispatch runs the agent in its own process group, so the deadline kills the whole tree: a process the agent left running in the background can't hold the dispatch (and its concurrency slot) open past the timeout.
-- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, and results flagged `budget_exceeded` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
+- **Caching** — identical `(agent, task, context, caller, goal, response_format)` requests return cached results, bounded by `cache.max_size` (oldest entry evicted first). Only clean successes are cached: failures, results with `denied_tools`, results flagged `budget_exceeded`, and results the agent itself reported as `partial` / `blocked` are not, so the documented "grant access / raise the cap, then re-dispatch" recovery is never served a stale crippled answer. Changing an agent's config invalidates its entries. Sessions and dialogues are never cached. A `group=` dispatch folds the group's `shared_context` into `context`, so different groups cache separately and a plain dispatch is unaffected.
 - **Durable config** — `agents.yaml` is written atomically (temp file + rename), so an interrupted write can never truncate it. Every mutation path (CLI and MCP server alike) also takes a cross-process advisory lock, so concurrent edits don't drop one another's agents. The lock is best-effort by design: after waiting 10 seconds it logs a warning and proceeds anyway, because a wedged lock holder must not freeze the MCP server — so on a heavily contended config a lost update is possible, while a truncated one is not.
 
 See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassPermissions` escalation risk and on-disk job files).
@@ -615,13 +674,13 @@ See [SECURITY.md](SECURITY.md) for the full threat model (including the `bypassP
 |---------|-------------|
 | `agent-dispatch init` | Create config + register MCP server with Claude Code |
 | `agent-dispatch add <name> <dir>` | Add an agent (auto-generates description) |
-| `agent-dispatch update <name>` | Update agent config (permissions, timeout, model, etc.) |
+| `agent-dispatch update <name>` | Update agent config (permissions, timeout, model, `--instructions`, etc.) |
 | `agent-dispatch remove <name>` | Remove an agent |
 | `agent-dispatch list` | List agents with health status and permissions |
 | `agent-dispatch group <add\|list\|inspect\|update\|remove>` | Manage [groups](#groups) — cross-project working sets of agents |
 | `agent-dispatch describe <name>` | Show full configuration for one agent (tri-state tools, project files) |
 | `agent-dispatch test <name> [task] [--stream]` | Test an agent with a dispatch (`--stream` for live progress) |
-| `agent-dispatch doctor` | Diagnose installation: Claude CLI, MCP registration, agent health, and group membership |
+| `agent-dispatch doctor` | Diagnose installation: Claude CLI (incl. `--append-system-prompt` support), MCP registration, agent health, and group membership |
 | `agent-dispatch jobs [--status --limit]` | List async dispatch jobs (most recent first) |
 | `agent-dispatch job <id>` | Show one job: status, progress tail, result preview |
 | `agent-dispatch cancel <id>` | Cancel a pending job (running jobs: use the `dispatch_cancel` MCP tool) |

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/agents.example.yaml

@@ -9,6 +9,13 @@ agents:
     risky_capabilities:
       - restart_services
     timeout: 300
+    # Standing orders, appended to the agent's system prompt on EVERY dispatch
+    # (after the dispatch protocol). Unlike `context`, which is per call, and
+    # unlike the project's CLAUDE.md, which is written for an interactive
+    # session, these are the rules for being dispatched.
+    instructions: |
+      Read-only unless the task explicitly asks to restart or redeploy.
+      Quote exact log lines with timestamps; never paraphrase an error.
 
   # Analytics gateway — a browser + Yandex Metrica agent, no codebase of its own.
   # Its value is its MCP servers + access, not its source. Shared across groups.
@@ -93,6 +100,12 @@ settings:
   default_timeout: 300
   max_dispatch_depth: 3           # recursion protection: A -> B -> A
   max_concurrency: 5              # max parallel claude -p processes
+  # dispatch_protocol: true        # Append the dispatch protocol to every agent's system
+                                   # prompt: it is non-interactive (never end with a
+                                   # question), its time/spend budget, lead with the
+                                   # outcome, end with `STATUS: done|partial|blocked`
+                                   # (→ DispatchResult.outcome). false = raw claude -p;
+                                   # needed only for CLIs without --append-system-prompt.
   # default_max_budget_usd: 1.0   # spend cap per dispatch, passed to claude as --max-budget-usd
                                   # (a run stopped at the cap fails with error_type: budget)
   # default_permission_mode: bypassPermissions  # inherited by agents without override
@@ -100,6 +113,12 @@ settings:
   #   - Bash
   #   - Read
   #   - Edit
+  # job_retention_days: 30         # OFF by default (0). When > 0, terminal job records
+                                   # older than this are deleted at server start. Every
+                                   # async and every return_ref dispatch leaves a job
+                                   # file behind and nothing removes it otherwise, so
+                                   # the directory grows forever — but the records are
+                                   # your dispatch history, so pruning is opt-in.
   cache:
     enabled: true
     ttl: 300                       # seconds; identical (agent, task, context) requests are cached

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agent-dispatch"
-version = "0.12.1"
+version = "0.14.0"
 description = "MCP server that lets Claude Code agents delegate tasks to agents in other project directories"
 readme = "README.md"
 license = "MIT"

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/src/agent_dispatch/__init__.py

@@ -1,3 +1,3 @@
 """agent-dispatch: Delegate tasks between Claude Code agents across projects."""
 
-__version__ = "0.12.1"
+__version__ = "0.14.0"

{agent_dispatch-0.12.1 → agent_dispatch-0.14.0}/src/agent_dispatch/cache.py

@@ -102,6 +102,12 @@ class DispatchCache:
             # serve the same crippled answer back for the whole TTL and make that
             # recovery a no-op (the permission config is not part of the key).
             return
+        if result.outcome in ("partial", "blocked"):
+            # The agent itself says the work is unfinished. Whatever it was
+            # missing (a service that was down, a file that did not exist yet)
+            # is not in the key either, so a retry after fixing it would be
+            # served the same unfinished answer for the whole TTL.
+            return
         key = self._make_key(agent, task, context, caller, goal, response_format)
         with self._lock:
             # Bound memory: when at capacity and inserting a new key, evict the