@kontextmind/kxm 0.7.94 → 0.7.96

package/docs/guides/peer-messaging.md

@@ -0,0 +1,304 @@
+# Message peer agents
+
+Agents that share a KXM [hub](../glossary.md#hub) project can hand work to each other: send one focused request, keep working, and collect the reply when you need it. This guide shows each step in two forms, the MCP tool an agent calls in Claude Code or Pi and the `kxm peer` command an operator or script runs. It also covers delivery modes, safe retries, expiry, and the errors you are most likely to meet.
+
+## Before you begin
+
+- A running hub, for example one started with `kxm hub start`, with its URL in `KXM_SERVER_URL`.
+- Two or more agents connected to the same hub project with that project's token: the Claude Code plugin ([quick start](../start/quickstart-claude-code.md)), Pi with the KXM package ([quick start](../start/quickstart-pi.md)), or a [supervised Pi worker](pi-workers.md).
+- For the `kxm peer` examples, the `kxm` CLI with `KXM_AUTH_TOKEN` set to the project token (never the admin token), `KXM_PROJECT` set to the project, and a stable `KXM_AGENT_NAME`. [Use the CLI as an agent](#use-the-cli-as-an-agent) explains why the name matters.
+
+## How a request moves through the hub
+
+A request is a durable message record in the hub's SQLite store that moves from `queued` to exactly one terminal state.
+
+```mermaid
+stateDiagram-v2
+  state "error (reserved)" as err
+  [*] --> queued: sender posts
+  queued --> delivered: recipient acknowledges
+  queued --> replied: recipient replies
+  delivered --> replied: recipient replies
+  queued --> cancelled: sender cancels
+  delivered --> cancelled: sender cancels
+  queued --> expired: TTL elapses
+  delivered --> expired: TTL elapses
+  replied --> [*]
+  cancelled --> [*]
+  expired --> [*]
+  note right of queued
+    Stored in SQLite. Pushed again on every
+    reconnect until the recipient acknowledges it.
+  end note
+  classDef reserved stroke-dasharray:5 5
+  class err reserved
+```
+
+| State | Meaning |
+|---|---|
+| `queued` | Accepted and stored. Survives hub and agent restarts. The hub pushes it again each time the recipient reconnects, until the recipient acknowledges it. |
+| `delivered` | Acknowledged: by Pi when the model turn starts, by the Claude Code plugin on arrival. It is not pushed again after a reconnect. |
+| `replied`, `cancelled`, `expired` | Terminal. A later reply or cancel is refused. |
+| `error` | Declared in the protocol but never set by the hub. Treat it as reserved. |
+
+A recipient can reply while a request is still `queued`. Delivery is at least once, not exactly once, so make any side effect a peer performs safe to repeat.
+
+## Discover peers
+
+Call `kxm_list` to see the online agents in your project with their purpose, model, host label, and hub-clocked presence (`online`, `stale`, or `offline`). Set `includeOffline` to also list registered agents whose heartbeat lease has lapsed.
+
+Tool call (`kxm_list`):
+
+```json
+{ "includeOffline": true }
+```
+
+CLI:
+
+```bash
+kxm peer list --include-offline
+```
+
+Agent names are unique among live agents in a project. A second live registration with the same name is refused with `duplicate_agent_name`; the Claude Code plugin then registers as `<name>-<pid>` and says so.
+
+## Send a request
+
+Call `kxm_send` with a target (an agent name, case-insensitive, or an agent ID) and one focused task that states the response you expect. The call returns at once with a durable message ID.
+
+Tool call (`kxm_send`):
+
+```json
+{
+  "target": "reviewer",
+  "content": "Review docs/plan.md for correctness and security risks. List blocking issues first.",
+  "idempotencyKey": "plan-review-1"
+}
+```
+
+CLI:
+
+```bash
+kxm peer send reviewer "Review docs/plan.md for correctness and security risks. List blocking issues first." \
+  --idempotency-key plan-review-1
+```
+
+Expected output:
+
+```text
+{
+  "messageId": "msg_779f5e0f22e04ac1af6078589a874971",
+  "status": "queued",
+  "target": "reviewer"
+}
+```
+
+| Field | CLI flag | Default | Purpose |
+|---|---|---|---|
+| `delivery` | `--delivery` | `followUp` | How the recipient schedules the work. See [Choose a delivery mode](#choose-a-delivery-mode). |
+| `idempotencyKey` | `--idempotency-key` | none | Makes an exact retry return the original message. |
+| `correlationId` | `--correlation-id` | none | Groups related requests. It carries no authority. |
+| `ttlMs` | `--ttl-ms` | 24 hours | How long the request stays valid, from 1 second to 7 days. |
+| `allowOffline` | `--allow-offline` | `false` | Queues the request for a registered agent that is offline. |
+| `workflowContext` | `--workflow-context` | none | Binds the request to a workflow stage so its reply can count as evidence. See [Peer provenance and quorum gates](provenance-gates.md). |
+
+> [!WARNING]
+> The hub stores message bodies as sent, without application-level encryption. Never put credentials or unneeded private data in a request or reply.
+
+## Check and wait for the reply
+
+Call `kxm_get` to read the current status and any reply without blocking. Call `kxm_await` only when the reply blocks your next step: it waits up to 60 seconds, which is both the default and the maximum. A timed-out wait reports `timed out waiting for <message-id>` and leaves the request untouched, so check it again later with `kxm_get`.
+
+Tool calls (`kxm_get`, then `kxm_await`):
+
+```json
+{ "messageId": "msg_779f5e0f22e04ac1af6078589a874971" }
+```
+
+```json
+{ "messageId": "msg_779f5e0f22e04ac1af6078589a874971", "timeoutMs": 30000 }
+```
+
+CLI:
+
+```bash
+kxm peer get msg_779f5e0f22e04ac1af6078589a874971
+kxm peer await msg_779f5e0f22e04ac1af6078589a874971 --timeout-ms 30000
+```
+
+Only the sender and the recipient can read a message. For work that runs for minutes or hours, do not loop on `kxm_await`: put the work in a [webhook workflow](webhook-workflows.md) and pause it with `kxm_workflow_wait`.
+
+## Ask up to three peers at once
+
+Call `kxm_fanout` to [fan out](../glossary.md#fanout) the same request independently to one, two, or three peers and collect the replies for comparison. Duplicate names are merged, every request uses `followUp`, and the call waits locally for up to `timeoutMs` (default and maximum 30 minutes).
+
+Tool call (`kxm_fanout`):
+
+```json
+{
+  "targets": ["planner", "reviewer"],
+  "content": "Propose a bounded plan for PROD-123 with file ownership and risks.",
+  "idempotencyKeyPrefix": "prod-123-plan",
+  "timeoutMs": 600000
+}
+```
+
+CLI (`--targets` takes space-separated names):
+
+```bash
+kxm peer fanout --targets planner reviewer \
+  --content "Propose a bounded plan for PROD-123 with file ownership and risks." \
+  --idempotency-key-prefix prod-123-plan --timeout-ms 600000
+```
+
+Each entry in `responses` is either terminal (`replied`, `cancelled`, `expired`, or `error`) or `pending`. A pending entry means only that your local wait ended:
+
+```text
+{
+  "responses": [
+    {
+      "target": "reviewer",
+      "messageId": "msg_7aa3182bcadb476fbcaf2858878b172d",
+      "status": "pending",
+      "messageStatus": "queued",
+      "expiresAt": "2026-09-24T18:30:24.688Z",
+      "waitStatus": "timed_out"
+    }
+  ]
+}
+```
+
+Keep the `messageId` handles and follow up with `kxm_get`, or repeat the exact call: the same prefix, correlation ID, targets, and workflow context resolve to the same messages instead of duplicates. Do not switch to a new prefix while earlier work is pending. Fanout reaches online peers only; an offline target comes back as `error` with `online target not found`.
+
+Compare the replies yourself. A reply is technical input from another model, not a verified fact.
+
+## Cancel a request
+
+Call `kxm_cancel` (or run `kxm peer cancel <message-id>`) to withdraw a `queued` or `delivered` request you sent. The recipient receives a `cancelled` event: Pi drops the request from its queue, and the Claude Code inbox removes it. Cancelling an already-cancelled request returns it unchanged; a replied or expired request cannot be cancelled.
+
+Cancellation is not rollback. If the peer already edited files or called an external system, check and undo that work yourself.
+
+## Receive and reply
+
+How an agent receives requests depends on its harness.
+
+**Pi** handles inbound work for you. The KXM extension takes one request at a time (`steer` first, then `followUp`, then `nextTurn`), acknowledges it as the model turn starts, and returns the settled final answer as the reply. A reply longer than 32,000 characters is truncated with a note. You do not call `kxm_reply` in Pi.
+
+**Claude Code** has two modes, described fully in the [plugin guide](../../plugins/kxm/README.md#pushed-channel-mode-and-pull-mode):
+
+- **Pushed channel mode.** Start Claude Code with the KXM channel. Each request arrives as a `<channel source="kxm" message_id="...">` event; Claude handles it and calls `kxm_reply`.
+- **Pull mode**, the default. Claude calls `kxm_inbox` to list requests that still need a reply, handles one, and calls `kxm_reply`. Nothing arrives on its own, so ask Claude to check the inbox or to poll it with a backoff.
+
+```bash
+# Organization-approved channel
+claude --channels plugin:kxm@kxm
+# During the channels research preview
+claude --dangerously-load-development-channels plugin:kxm@kxm
+```
+
+Tool call (`kxm_reply`):
+
+```json
+{
+  "messageId": "msg_779f5e0f22e04ac1af6078589a874971",
+  "content": "The plan is sound. Add a test for a zero discount. No blocking issues."
+}
+```
+
+CLI (run with the recipient's `KXM_AGENT_NAME`):
+
+```bash
+kxm peer reply msg_779f5e0f22e04ac1af6078589a874971 "The plan is sound. Add a test for a zero discount."
+```
+
+Only the recipient can reply, and only once. From the CLI, `kxm peer inbox` always returns an empty list, because a one-shot command has no long-running inbox. Use `kxm dash --screen inbox` to see pending requests.
+
+## Choose a delivery mode
+
+A [delivery mode](../glossary.md#delivery-mode) tells the recipient how to schedule the work.
+
+| Mode | Use it for | Effect in Pi |
+|---|---|---|
+| `followUp` | Normal delegation. The default. | Handled after the current work settles. |
+| `steer` | An active blocker that must change course. | Moves ahead of queued work and starts at the next safe turn boundary. It does not abort a running tool call or write. |
+| `nextTurn` | Low-priority context. | Queued after other work, then run as a follow-up turn, because an autonomous worker has no later human prompt to wait for. |
+
+In Claude Code the mode arrives as `delivery` in the channel metadata, and the session decides what to do with it. Webhook workflow definitions accept only `followUp` and `steer`.
+
+## Make retries safe
+
+An **idempotency key** (up to 128 characters) is scoped to the sender. Repeating a send with the same key and identical fields returns the original message instead of a duplicate. Reusing the key with any different field (target, content, delivery, correlation ID, TTL, or workflow context) fails with `idempotency_conflict`. The key is remembered for as long as the hub retains the message.
+
+A **correlation ID** (up to 128 characters) groups related requests for your own bookkeeping. It does not deduplicate or authorize anything. When a request carries `workflowContext`, the hub sets the correlation ID to the workflow run ID and refuses any other value.
+
+Neither field proves where a reply came from. Workflow evidence uses `workflowContext`, which the hub authorizes and stores; see [Peer provenance and quorum gates](provenance-gates.md).
+
+## Limits, expiry, and retention
+
+| Limit | Value | Notes |
+|---|---|---|
+| Request or reply content | 32,000 characters | The whole HTTP body is capped at 256 KiB. |
+| Time to live (TTL) | 24 hours by default; 1 second to 7 days per request | Counted from the send, including time spent queued. The hub default is `KXM_MESSAGE_TTL_MS`. |
+| `kxm_await` | 60 seconds | Default and maximum. |
+| `kxm_fanout` local wait | 30 minutes | Default and maximum. |
+| Hop limit | 5 by default; `maxHops` 1 to 20 | See the note below. |
+| Retention | 7 days after a terminal state | Set with `KXM_MESSAGE_RETENTION_MS`. A purged message returns `message not found`. |
+| Rate limit | 600 requests per 60 seconds | Counted per agent, or per client address. Set with `KXM_RATE_LIMIT_MAX` and `KXM_RATE_LIMIT_WINDOW_MS`. |
+
+When a request expires, both the sender and the recipient receive an `expired` event.
+
+The hop limit refuses a request whose `hops` count has reached `maxHops` (`hop_limit_reached`). Only a client that forwards a request through the [Hub HTTP API reference](../reference/http-api.md) and carries its hop count is affected. `kxm_send`, `kxm_fanout`, and `kxm peer send` always start at hop 0, so the limit does not stop a chain of agents that each send a new request.
+
+## Queue work for an offline agent
+
+By default a send fails unless the target is online. Set `allowOffline` (`--allow-offline`) to queue the request for an agent that has registered in this project before but is offline now. The request stays `queued` until the agent reconnects or its TTL passes. An unknown name still fails, and `kxm_fanout` has no offline option.
+
+## Use the CLI as an agent
+
+Each `kxm peer` command registers with the hub as `KXM_AGENT_NAME` (default `cli-<pid>`), runs one call, and unregisters. This has four consequences:
+
+1. Set a stable `KXM_AGENT_NAME`. Later `get`, `await`, and `cancel` commands must run as the sender; any other name gets `message is not visible to this agent`.
+2. Between commands the CLI identity is offline. Peers can still reply to it; read the reply with `kxm peer get`. To send work to a CLI identity, use `--allow-offline`.
+3. The name must not be live elsewhere, or registration fails with `duplicate_agent_name`.
+4. `--dry-run` prints the parsed arguments without contacting the hub.
+
+On failure the CLI prints the hub's message, not its code, for example `{"ok":false,"error":"command_failed","detail":"cannot send a request to yourself"}`.
+
+## Errors worth knowing
+
+Tools and the CLI report the message text; the HTTP response also carries the code.
+
+| Message | Code | Cause and fix |
+|---|---|---|
+| `online target not found: <name>` | `target_not_found` | The target is offline or unknown. Check `kxm_list`; use `allowOffline` for a registered agent. |
+| `cannot send a request to yourself` | `self_target` | Send to a different agent. |
+| `idempotency key was already used for another request` | `idempotency_conflict` | A different request reused the key. Repeat the original exactly, or use a new key for new work. |
+| `hop limit reached (<hops>/<max>)` | `hop_limit_reached` | A forwarding client reached `maxHops`. Stop the chain. |
+| `ttlMs must be an integer between 1000 and 604800000` | `protocol_error` | Use a TTL from 1 second to 7 days. |
+| `message is not visible to this agent` | `message_forbidden` | Run as the sender or recipient. |
+| `message already has a reply` | `duplicate_reply` | The request is already answered. |
+| `cannot cancel a replied message` | `invalid_message_state` | The request is terminal. Read it with `kxm_get`. |
+| `message not found` | `message_not_found` | Wrong ID, or the message was purged after retention. |
+| `agent name already active in project: <name>` | `duplicate_agent_name` | Another live agent holds the name. Stop it or choose another name. |
+| `request rate limit exceeded` | `rate_limited` | Back off for the `retry-after` seconds. |
+
+Errors about `workflowContext` are covered in [Peer provenance and quorum gates](provenance-gates.md).
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| A request stays `queued` | The recipient is offline, busy with earlier work, or swapping its Pi session for a workflow run. | Check `kxm_list` and the recipient's log. Do not send a duplicate. |
+| A request stays `delivered` | The recipient's turn, tool, or provider call is still running, or a Claude Code session restarted after acknowledging it. | Wait, or cancel and send it again with a new idempotency key. |
+| `kxm_fanout` returns `pending` | The local wait ended before a reply. | Use the returned message IDs with `kxm_get`, or repeat the exact call. |
+| Claude Code never sees requests | Channel mode is off or blocked by policy. | Use `kxm_inbox` and `kxm_reply`. |
+| `kxm peer inbox` is always empty | The CLI has no long-running inbox. | Use `kxm dash --screen inbox`. |
+
+For hub-level problems, see [Troubleshoot KXM](../operations/troubleshooting.md).
+
+## Next steps
+
+- Run long-lived Pi agents that answer requests unattended: [Run supervised Pi workers](pi-workers.md)
+- Require verified replies from named peers before a stage passes: [Peer provenance and quorum gates](provenance-gates.md)
+- Drive multi-stage work from a Jira or GitHub webhook: [Run webhook workflows](webhook-workflows.md)
+- Every tool parameter: [Agent tools reference](../reference/tools.md); every flag: [CLI reference](../reference/cli-reference.md#kxm-peer)
+- How the hub, agents, and stores fit together: [Architecture](../concepts/architecture.md)

package/docs/guides/pi-workers.md

@@ -0,0 +1,219 @@
+# Run supervised Pi workers
+
+A [worker](../glossary.md#worker) is a long-lived Pi process that stays connected to the KXM hub and answers peer requests and workflow prompts without anyone at the keyboard. `kxm agent worker` starts Pi in headless RPC mode under a small supervisor that restarts it, rotates to fallback models, and can keep each workflow run in its own Pi session. This guide covers starting a worker, choosing models and tools, session isolation, recovery, settings, and stale-worker problems.
+
+## Before you begin
+
+- The `kxm` CLI, and Pi with the KXM package installed: see [Install KXM](../start/install.md) and the [Pi quick start](../start/quickstart-pi.md). Put `pi` on `PATH` or set `KXM_PI_COMMAND`.
+- A running hub and its URL in `KXM_SERVER_URL`.
+- The hub project's token, which the worker uses as `KXM_AUTH_TOKEN`. Never give a worker the admin token.
+- A checkout of the repository the worker should work in. The worker runs Pi in `KXM_WORKDIR`, or the current directory.
+- A model route that Pi may run. See [Choose models and fallbacks](#choose-models-and-fallbacks).
+
+## What the worker does
+
+The supervisor launches `pi --mode rpc` with the KXM extension, keeps its input open, and captures its output. The hub stays the only durable queue:
+
+- The extension takes one request at a time and acknowledges it only when the model turn starts, so waiting work stays `queued` on the hub and survives a restart.
+- When the turn settles, the extension returns the final answer as the reply. See [Message peer agents](peer-messaging.md) for the message lifecycle.
+- If Pi exits, the supervisor restarts it with exponential backoff from 1 second to 30 seconds. The backoff resets after a child has run for a minute.
+- A PID claim in `.kxm/state` stops a second supervisor for the same project and agent name.
+
+The worker does not read models, tools, roles, or ownership from any project file. Pass them as flags or environment variables. Run the worker itself under your service manager (systemd, launchd, or similar) for start at boot, resource limits, and log collection.
+
+## Start a worker
+
+Set the hub connection and project token, then start one worker per agent name.
+
+```bash
+export KXM_SERVER_URL=http://127.0.0.1:7331
+export KXM_AUTH_TOKEN="replace-with-the-project-token"
+export KXM_WORKDIR=~/work/product
+kxm agent worker --name reviewer --project product \
+  --model <pi-model> --tools read,grep,find,ls \
+  --session-isolation workflow
+```
+
+<details><summary>PowerShell</summary>
+
+```powershell
+$env:KXM_SERVER_URL = "http://127.0.0.1:7331"
+$env:KXM_AUTH_TOKEN = "replace-with-the-project-token"
+$env:KXM_WORKDIR = "$HOME\work\product"
+kxm agent worker --name reviewer --project product `
+  --model <pi-model> --tools read,grep,find,ls `
+  --session-isolation workflow
+```
+
+</details>
+
+The command runs in the foreground until it is stopped. Add `--dry-run` to check the settings without starting anything:
+
+```bash
+kxm agent worker --name reviewer --project product --tools read,grep,find,ls --session-isolation workflow --dry-run
+```
+
+Expected output:
+
+```text
+would start worker
+```
+
+> [!IMPORTANT]
+> Set `KXM_AUTH_TOKEN` to the project token in the worker's environment. When it is unset, the Pi extension falls back to the hub credential persisted on this machine, which is the admin token for a hub that `kxm hub start` or Pi auto-start created.
+
+| Flag | Environment variable | Effect |
+|---|---|---|
+| `--name` | `KXM_AGENT_NAME` | Agent name on the hub. Required. |
+| `--project` | `KXM_PROJECT` | Hub project. Required. |
+| `--model` | `KXM_WORKER_MODEL` | Primary model. Pi's default when unset. |
+| `--fallback-models` | `KXM_WORKER_FALLBACK_MODELS` | Up to eight comma-separated fallbacks. Requires a primary model. |
+| `--tools` | `KXM_WORKER_TOOLS` | Comma-separated Pi tool allowlist. |
+| `--session-isolation` | `KXM_WORKER_SESSION_ISOLATION` | `workflow` or `off` (the default). |
+| `--fresh-start` | `KXM_WORKER_INITIAL_CONTINUE=false` | Skip resuming the previous Pi session on first start only. |
+| `--no-continue` | `KXM_WORKER_CONTINUE=false` | Never resume a previous Pi session. |
+
+## Choose models and fallbacks
+
+`--model` selects the primary Pi model. `--fallback-models` lists up to eight more, tried in order when a provider fails for good. Pi finishes its own transient retries first; then the supervisor closes the session cleanly and restarts on the next model, resuming the same Pi session so completed tool and peer results are kept.
+
+When no unused fallback remains, the worker waits `KXM_WORKER_PROVIDER_RETRY_MS` (60 seconds by default) and tries the last model again. It does not return to the primary model until you restart the worker.
+
+> [!IMPORTANT]
+> A worker refuses to start on a model whose vendor has its own native harness. Before Pi starts, the worker runs `--model` and every `--fallback-models` entry through the Pi native-vendor brake, and exits 1 with `pi_native_impersonation_blocked` on the first one it refuses: a native provider (`xai/…`), that vendor's own Pi provider (`openai-codex/…`, `kimi-coding/…`), or an aggregator path to it (`openrouter/x-ai/…`). `--dry-run` does not run this check.
+
+Pick every model with [Harness routing](../reference/harness-routing.md), for example `openrouter/qwen/qwen3-coder-plus` with the fallback `openrouter/z-ai/glm-5.3-flash`. The brake cannot check a worker started without `--model`, which runs Pi's default model, or a bare model id that lets Pi choose the provider, so name the provider.
+
+An agent name is only a label. To use a native-harness model as a peer, connect that harness to the hub under the agent name instead, for example the Claude Code plugin with the name `reviewer-claude`.
+
+## Restrict tools
+
+`--tools` passes an allowlist to Pi. It limits which tools the model can call, not which files those tools can reach: a worker that keeps `write`, `edit`, or a shell tool can change anything its operating-system user can.
+
+- **Read-only reviewer:** `--tools read,grep,find,ls`. A reviewer does not need any `kxm_*` tool to answer; the extension returns its final answer as the reply.
+- **Coordinator of a [webhook workflow](webhook-workflows.md):** add the hub tools the workflow needs, for example `kxm_list`, `kxm_send`, `kxm_fanout`, `kxm_get`, `kxm_await`, `kxm_workflow_get`, `kxm_workflow_checkpoint`, `kxm_workflow_wait`, `kxm_workflow_record`, and `kxm_improvement_report`, plus only the edit and shell tools its stages require.
+
+```bash
+kxm agent worker --name coordinator --project product \
+  --model <pi-model> --fallback-models <fallback-model> \
+  --tools read,grep,find,ls,edit,write,bash,kxm_send,kxm_fanout,kxm_get,kxm_await,kxm_workflow_get,kxm_workflow_checkpoint,kxm_workflow_wait,kxm_workflow_record \
+  --session-isolation workflow
+```
+
+For stronger boundaries, give each worker its own operating-system user, a read-only worktree, or a container. `KXM_WORKER_EXTENSION_PATHS` and `KXM_WORKER_SKILL_PATHS` load exact extension and skill files instead of Pi's discovery; treat those paths as executable code with the worker's credentials, and never derive them from a webhook or workflow payload.
+
+## Isolate sessions per workflow run
+
+With `--session-isolation workflow`, one worker keeps a stable default Pi session for ordinary work and a separate Pi session for each hub workflow run, so unrelated histories never mix in one context window.
+
+| Message | Pi session |
+|---|---|
+| Ordinary peer or operator request | The default session |
+| Root workflow prompt, signal resume, or wait timeout notice | That run's session |
+| Peer request with an authorized `workflowContext` | That run's session |
+| Request with only a `run_…` correlation ID | The default session. A correlation ID is not authorization. |
+
+A switch never interrupts a turn and never runs two Pi processes at once. The next diagram shows a queued message for another run moving the worker to that run's session.
+
+```mermaid
+sequenceDiagram
+  participant Hub
+  participant A as Pi child (default session)
+  participant Sup as Worker supervisor
+  participant B as Pi child (run_1 session)
+  Hub->>A: push msg_1 for workflow run_1
+  A->>A: binding differs, leave msg_1 queued
+  A->>Sup: write route request (metadata only)
+  A-->>Sup: child closes
+  Sup->>Sup: update the binding manifest
+  Sup->>B: start one child with --session-dir runs/run_1
+  B->>Hub: reconnect
+  Hub->>B: replay msg_1 (same message ID)
+  B->>Hub: acknowledge and run the turn
+```
+
+The route request holds identity, supervisor generation, child incarnation, source and destination bindings, the Pi session ID, the message ID, and a timestamp, never a message body. The supervisor rejects a malformed, stale, or wrong-owner request without changing scope, and refuses session directories that are links.
+
+Session files live under `.kxm/state/pi-sessions/<worker-key>/default` and `.../runs/<run-id>`, and the active binding in `.kxm/state/worker-session-binding-<worker-key>.json`. After a restart the worker resumes the bound session only if it has Pi history. It keeps up to `KXM_WORKER_MAX_RUN_SESSIONS` run sessions (128 by default) and deletes the least recently used inactive ones beyond that. A corrupt manifest is quarantined with a `.corrupt-<timestamp>` suffix and replaced by the default binding.
+
+Isolation is off by default for upgrade compatibility. The first start with `workflow` begins fresh scoped sessions; KXM does not copy the old shared history, because it cannot be attributed to one run safely.
+
+> [!NOTE]
+> Pi session files are a convenience, not the record. The hub's workflow run, journal, messages, workspace assets, and Git are the recovery authority. Automatic per-run sessions apply to supervised Pi workers only; for Claude Code, use a separate session or agent name per run.
+
+Isolation keeps model context apart. It is not a sandbox: a shell-capable agent can still read files and environment values that its operating-system user can reach.
+
+## How the worker recovers
+
+| Failure | What the worker does |
+|---|---|
+| Pi exits or crashes | Restarts with backoff. Stops after `KXM_WORKER_MAX_RESTARTS` restarts when set. |
+| Final provider failure | Leaves the request `delivered`, records bounded metadata, and restarts on the next fallback model. |
+| A tool runs past `KXM_WORKER_TOOL_TIMEOUT_MS` | Stops the Pi process tree and resumes the request without changing model. |
+| A delivered request does not start a turn within `KXM_WORKER_ACTIVATION_TIMEOUT_MS` | Requests a restart and keeps the hub claim. |
+| Pi cannot resume a saved session | Retries once with a fresh session and writes a recovery envelope in `.kxm/state`. |
+| A workflow run needs a fresh session | Sends Pi a recovery prompt to call `kxm_workflow_get` and continue the current stage without repeating completed work. |
+
+The supervisor writes a structured lifecycle log to `.kxm/logs/kxm-worker-<worker-key>.jsonl` with bounded metadata only. Raw Pi output, which can contain model and tool output, goes to `.kxm/logs/pi-agent-<worker-key>.log`; protect it accordingly.
+
+| Log event | Meaning | Action |
+|---|---|---|
+| `worker_session_routed` | Expected swap to another run's session | None, unless it repeats for one message |
+| `worker_session_evicted` | An inactive run session was deleted at the limit | Keep workflow facts in the journal, assets, and Git |
+| `worker_session_state_recovered` | A bad manifest was quarantined | Inspect the `.corrupt-*` file, the hub run, and disk health |
+| `worker_session_request_rejected` | A route request failed identity or schema checks | Check for a version mismatch or a duplicate supervisor |
+| `worker_continue_fallback` | Pi history could not resume; started fresh | Read the recovery journal and the message state |
+| `worker_provider_failure` | A provider failed after Pi's own retries | Check the provider and fallback list |
+| `worker_restart_limit_reached` | `KXM_WORKER_MAX_RESTARTS` was hit | Fix the cause, then start the worker again |
+
+## Worker settings
+
+These variables configure a worker; flags override them. The complete list, with limits, is in the [environment variable reference](../reference/configuration.md).
+
+| Variable | Default | Effect |
+|---|---|---|
+| `KXM_WORKDIR` | Current directory | Repository Pi works in; `.kxm` paths resolve inside it. |
+| `KXM_PI_COMMAND` | `pi` (`pi.cmd` on Windows) | Pi executable. |
+| `KXM_WORKER_MAX_RUN_SESSIONS` | `128` | Run sessions kept per worker, 1 to 1024. |
+| `KXM_WORKER_TOOL_TIMEOUT_MS` | `1860000` | Longest single tool call, 1 second to 24 hours; `0` disables. The default sits one minute above the longest hub wait. |
+| `KXM_WORKER_ACTIVATION_TIMEOUT_MS` | `60000` | Time for a delivered request to start a turn, 1 second to 10 minutes. |
+| `KXM_WORKER_PROVIDER_RETRY_MS` | `60000` | Wait after the last fallback fails, 1 second to 1 hour. |
+| `KXM_WORKER_DRAIN_MS` | `15000` | Graceful stop window before the child is killed. |
+| `KXM_WORKER_MAX_RESTARTS` | Unlimited | Restart ceiling. |
+| `KXM_WORKER_EXTENSION_PATHS` | Pi discovery | Exact extension files, separated by `:` (`;` on Windows). |
+| `KXM_WORKER_SKILL_PATHS` | Pi discovery | Exact skill files or directories, same separator. |
+| `KXM_WORKER_LOG_PATH`, `KXM_AGENT_LOG_PATH` | `.kxm/logs/…` | Lifecycle log and raw Pi log locations. |
+
+Variables such as `KXM_WORKER_IDENTITY_KEY` and `KXM_WORKER_GENERATION` are set by the supervisor for its child; do not set them yourself. Restart the worker after changing any setting.
+
+## Stop a worker
+
+Press Ctrl+C in the worker's terminal, or send it `SIGTERM`. The supervisor gives Pi `KXM_WORKER_DRAIN_MS` to finish, then stops it. `kxm hub stop` and `kxm session stop` stop every managed hub and worker process in the workspace, not one worker.
+
+```bash
+# Stops the hub and every worker managed from this workspace
+kxm hub stop
+```
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `KXM worker PID claim is stale at <path>` | A previous supervisor was killed without cleanup. | Run `kxm session status --json`; a claim with `"live": false` is stale. Confirm no matching worker is running, delete that file, and start again. |
+| `KXM worker <project>/<name> is already managed by PID <pid>` | A supervisor for this agent is already running. | Use the running worker, or stop it first. |
+| The worker exits with `worker requires --name and --project` | Name or project missing. | Pass `--name` and `--project`, or set `KXM_AGENT_NAME` and `KXM_PROJECT`. |
+| `KXM_WORKER_FALLBACK_MODELS requires KXM_WORKER_MODEL` | Fallbacks without a primary model. | Add `--model`. |
+| The worker exits 1 with `pi_native_impersonation_blocked: …` | `--model` or a fallback is a model whose vendor has its own harness. | Run that model in its native harness, or use an admitted Pi route such as `openrouter/qwen/qwen3-coder-plus`. |
+| `kxm connection failed` in the Pi log, or Pi shows `hub:off` | Wrong URL, token, or project, or the name is live elsewhere (`duplicate_agent_name`). | Compare `KXM_SERVER_URL`, `KXM_AUTH_TOKEN`, and `KXM_PROJECT` with the hub; check `kxm peer list`. |
+| Workflow runs share one conversation | Isolation is `off`, the default. | Restart with `--session-isolation workflow`. |
+| A request stays `delivered` | A turn, tool, or provider call is still running, or the watchdogs are recovering it. | Read the lifecycle log. Do not send a duplicate. |
+
+Never repair a live route by editing files in `.kxm/state`. Stop the worker first, keep the evidence, and recover from the hub's workflow state. More symptoms are in [Troubleshoot KXM](../operations/troubleshooting.md).
+
+## Next steps
+
+- Send work to your worker and read its replies: [Message peer agents](peer-messaging.md)
+- Make a worker the coordinator of a signed webhook workflow: [Run webhook workflows](webhook-workflows.md)
+- Require replies from named reviewer workers before a stage passes: [Peer provenance and quorum gates](provenance-gates.md)
+- Pick allowed Pi routes: [Harness routing](../reference/harness-routing.md)
+- Every flag: [CLI reference](../reference/cli-reference.md#kxm-agent-worker)