@kontextmind/kxm 0.6.0

package/docs/kxm-handbook.md

@@ -0,0 +1,1090 @@
+# KXM Handbook
+
+> Installation, configuration, CLI, Pi, Claude Code, durable workflows, gates, session isolation, observability, and recovery.
+
+## Contents
+
+1. [What KXM is](#what-kxm-is)
+2. [Requirements and installation](#requirements-and-installation)
+3. [Five-minute setup](#five-minute-setup)
+4. [Workspace layout](#workspace-layout)
+5. [Authentication and configuration](#authentication-and-configuration)
+6. [Complete CLI guide](#complete-cli-guide)
+7. [Pi integration](#pi-integration)
+8. [Workflow-specific Pi sessions](#workflow-specific-pi-sessions)
+9. [Claude Code integration](#claude-code-integration)
+10. [Hub tools](#hub-tools)
+11. [Durable workflows](#durable-workflows)
+12. [Evidence gates and external callbacks](#evidence-gates-and-external-callbacks)
+13. [Live TUI and observability](#live-tui-and-observability)
+14. [Context operating system (v0.5)](#context-operating-system-v05)
+15. [Reliability, privacy, and security](#reliability-privacy-and-security)
+16. [Backup, upgrade, and recovery](#backup-upgrade-and-recovery)
+17. [Troubleshooting checklist](#troubleshooting-checklist)
+18. [Feature availability matrix](#feature-availability-matrix)
+
+---
+
+## What KXM is
+
+KXM connects Pi and Claude Code agents through one durable, authenticated hub.
+It provides:
+
+- peer discovery by name, model, and declared purpose;
+- bounded request/reply messaging over HTTP and server-sent events (SSE);
+- durable `queued → delivered → replied` message state in SQLite;
+- cancellation, expiry, idempotent retries, fanout, and non-blocking polling;
+- signed webhook workflows with ordered stages and attempt limits;
+- local evidence, hub-verified peer provenance, external waits, and callbacks;
+- long-lived Pi supervision with model fallback and tool watchdogs;
+- optional workflow-scoped Pi sessions: one stable ordinary context and one per durable run;
+- Claude Code MCP tools with optional pushed channel delivery;
+- a real-time, read-only, metadata-only terminal dashboard; and
+- structured workflow journals, retrospectives, and proposed improvement reports.
+
+KXM is not a filesystem sandbox, distributed scheduler, shared model context, or
+exactly-once execution engine. Use one writer per checkout or separate Git
+worktrees. Treat peer output as untrusted until independently verified.
+
+### Important terms
+
+| Term | Meaning |
+|---|---|
+| **Hub** | The Node.js service that authenticates clients, persists state, pushes events, and runs workflow transitions |
+| **Project** | Authentication and discovery namespace; agents see only peers in the same project |
+| **Agent** | A registered Pi or Claude Code identity with a unique name in one project |
+| **Message** | A durable request with one recipient and one reply lifecycle |
+| **KXM session manifest** | A human-reviewable roster and asset plan; it does not launch processes |
+| **Workflow run** | A durable ordered stage machine stored by the hub |
+| **Pi session** | A Pi model-conversation JSONL; supervised workers can isolate it by workflow run |
+| **Gate** | A deterministic CLI check, a workflow evidence requirement, or a signed external result, depending on context |
+
+---
+
+## Requirements and installation
+
+### Requirements
+
+- Node.js **22.19 or newer on the 22.x line**, or Node.js 24 or newer;
+- Git;
+- GitHub CLI (`gh`) for private release assets;
+- Pi for Pi agents;
+- Claude Code for Claude agents; and
+- access to `kontextmind/kxm` while the repository is private.
+
+### Install the `kxm` operator CLI
+
+The supported global installation is the versioned release tarball. Pi's Git
+package install does **not** place `kxm` on `PATH`.
+
+PowerShell:
+
+```powershell
+$version = "<release-version>"
+$asset = "kxm-$version.tgz"
+$releaseDir = Join-Path $PWD ".kxm-release"
+New-Item -ItemType Directory -Force -Path $releaseDir | Out-Null
+gh auth login
+gh release download "v$version" --repo kontextmind/kxm `
+  --pattern $asset --dir $releaseDir --clobber
+npm install --global --omit=peer (Join-Path $releaseDir $asset)
+kxm --help
+```
+
+Bash:
+
+```bash
+version='<release-version>'
+asset="kxm-${version}.tgz"
+mkdir -p .kxm-release
+gh auth login
+gh release download "v${version}" --repo kontextmind/kxm \
+  --pattern "$asset" --dir .kxm-release --clobber
+npm install --global --omit=peer ".kxm-release/$asset"
+kxm --help
+```
+
+Do not use `npx kxm` or a global `git+https` npm install.
+
+### Run from a source checkout
+
+```bash
+git clone <authorized-kxm-url>
+cd kxm
+npm ci
+npm run check
+node scripts/kxm.mjs --help
+```
+
+Use `node scripts/kxm.mjs` wherever this handbook shows `kxm`. The source
+wrapper runs the committed generated CLI, so maintainers must run `npm run build`
+after changing CLI source.
+
+### Install in Pi
+
+From Pi:
+
+```text
+pi install git:github.com/kontextmind/kxm@main
+```
+
+This installs the Pi extension and the `kxm` Agent Skill. Restart Pi after
+installation or package updates.
+
+### Install in Claude Code
+
+From Claude Code:
+
+```text
+/plugin marketplace add kontextmind/kxm
+/plugin install kxm
+/reload-plugins
+```
+
+The Claude plugin includes the bundled MCP runtime and shared skill.
+
+---
+
+## Five-minute setup
+
+### 1. Initialize the project
+
+From the project directory:
+
+```text
+kxm init
+```
+
+### 2. Start the hub in another terminal
+
+`kxm hub start` is foreground. Keep that terminal running. Use a high-entropy
+administrative token for hub operations and a different project token for
+agents. Never pass either token on a command line.
+
+PowerShell:
+
+```powershell
+$env:KXM_HOST = "127.0.0.1"
+$env:KXM_PORT = "7331"
+$env:KXM_AUTH_TOKEN = "<admin-token>"
+$env:KXM_PROJECT_TOKENS = '{"product":"<project-token>"}'
+kxm hub start
+```
+
+Bash:
+
+```bash
+export KXM_HOST=127.0.0.1
+export KXM_PORT=7331
+export KXM_AUTH_TOKEN='<admin-token>'
+export KXM_PROJECT_TOKENS='{"product":"<project-token>"}'
+kxm hub start
+```
+
+Store service values in an ACL-protected, gitignored environment file or secret
+manager. If using Node's `--env-file`, keep the file path—not its contents—on
+the command line.
+
+### 3. Bind, then confirm the session
+
+```text
+kxm hub bind http://127.0.0.1:7331
+kxm session brief
+```
+
+PowerShell health check:
+
+```powershell
+kxm hub view
+Invoke-RestMethod http://127.0.0.1:7331/ready
+```
+
+Bash:
+
+```bash
+kxm hub view
+curl --fail http://127.0.0.1:7331/ready
+```
+
+### 4. Start a supervised Pi worker
+
+Use only the project token in the worker environment:
+
+```powershell
+$env:KXM_SERVER_URL = "http://127.0.0.1:7331"
+$env:KXM_AUTH_TOKEN = "<project-token>"
+$env:KXM_PROJECT = "product"
+$env:KXM_WORKDIR = "C:\work\product"
+kxm agent worker --name coordinator --project product `
+  --model provider/model --session-isolation workflow
+```
+
+```bash
+export KXM_SERVER_URL=http://127.0.0.1:7331
+export KXM_AUTH_TOKEN='<project-token>'
+export KXM_PROJECT=product
+export KXM_WORKDIR=/work/product
+kxm agent worker --name coordinator --project product \
+  --model provider/model --session-isolation workflow
+```
+
+### 5. Connect another Pi or Claude peer
+
+Give it the same hub URL, project token, and project, but a different agent name.
+Call `kxm_list` to verify discovery, then send one focused request with
+`kxm_send`.
+
+---
+
+## Workspace layout
+
+```text
+.kxm/
+├── config/                         # reviewable configuration; no secret values
+│   ├── agents.json                 # optional roster used by session manifests
+│   ├── gates.json                  # optional documented gate roster
+│   ├── env.example                 # variable names and safe placeholders
+│   └── workflows/*.json            # active workflow definition candidates
+├── logs/                           # ignored runtime logs and telemetry
+│   ├── kxm-hub.jsonl
+│   ├── kxm-worker-*.jsonl
+│   ├── pi-agent-*.log              # raw Pi output; may be sensitive
+│   └── telemetry.jsonl
+├── assets/                         # intentional human-reviewable inputs/outputs
+│   ├── sessions/<id>/session.json
+│   ├── workflows/<definition>/...
+│   └── retrospectives/<runId>.{json,md}
+└── state/                          # ignored runtime state; protect with OS ACLs
+    ├── kxm.db
+    ├── hub.pid / worker-*.pid
+    ├── worker-recovery-*.json
+    ├── worker-session-binding-*.json
+    └── pi-sessions/<workerKey>/
+        ├── default/
+        └── runs/<runId>/
+```
+
+Commit reviewed configuration and intentional reusable assets. Do not commit
+runtime state, credentials, raw model logs, generated secrets, or SQLite files.
+Workflow stages and evidence live in `kxm.db`; important implementation results
+should also live in Git or another system of record.
+
+---
+
+## Authentication and configuration
+
+### Credential roles
+
+| Credential | Give it to | Capabilities |
+|---|---|---|
+| `KXM_AUTH_TOKEN` administrative token | Hub and trusted operator terminal | Operations snapshot/SSE, metrics where protected, quorum degradation, and fallback project access |
+| Entry in `KXM_PROJECT_TOKENS` | Hub only | Maps one project to its worker credential |
+| Project token | Pi and Claude agents in that project | Registration, discovery, messaging, and assigned workflow operations only |
+| Workflow start secret | Hub and webhook sender/operator | HMAC-signs a new workflow delivery |
+| Workflow signal secret | Hub and callback sender/operator | HMAC-signs an external result; may fall back to the start secret if the definition omits it |
+| Agent key | Returned and rotated internally | Authorizes one resumed agent identity; never configure manually |
+
+A project token cannot call administrative operations endpoints or approve quorum
+degradation. All holders of one project credential are inside the same
+provenance trust domain.
+
+### Core hub variables
+
+| Variable | Default | Purpose |
+|---|---:|---|
+| `KXM_HOST` | `127.0.0.1` | Bind interface |
+| `KXM_PORT` | `7331` | Hub port; `0` chooses a free port |
+| `KXM_AUTH_TOKEN` | none | Administrative bearer token |
+| `KXM_PROJECT_TOKENS` | none | JSON object of project-to-token mappings |
+| `KXM_WORKSPACE_DIR` | `.kxm` | Workspace root |
+| `KXM_DATA_PATH` | `.kxm/state/kxm.db` | SQLite path |
+| `KXM_MESSAGE_TTL_MS` | `86400000` | Default request lifetime |
+| `KXM_MESSAGE_RETENTION_MS` | `604800000` | Terminal-message retention |
+| `KXM_RATE_LIMIT_MAX` | `600` | Requests per bucket/window |
+| `KXM_RATE_LIMIT_WINDOW_MS` | `60000` | Rate-limit window |
+| `KXM_WEBHOOK_WORKFLOWS_FILE` | none | One active JSON workflow-definition file |
+| `KXM_WEBHOOK_WORKFLOWS` | none | Inline alternative; never set with the file variable |
+
+A non-loopback bind requires an administrative token. Use TLS termination and
+network controls before allowing remote access.
+
+### Agent variables
+
+| Variable | Default | Purpose |
+|---|---:|---|
+| `KXM_SERVER_URL` | `http://127.0.0.1:7331` | Hub URL |
+| `KXM_AUTH_TOKEN` | none | Project token for agents |
+| `KXM_PROJECT` | current directory name | Discovery/authentication namespace |
+| `KXM_AGENT_NAME` | harness-derived | Unique live name in the project |
+| `KXM_AGENT_PURPOSE` | general-purpose | Capability shown during peer discovery |
+
+### Supervised Pi variables
+
+| Variable | Default | Purpose |
+|---|---:|---|
+| `KXM_WORKDIR` | current directory | Repository used by Pi |
+| `KXM_PI_COMMAND` | `pi` / `pi.cmd` | Explicit Pi executable |
+| `KXM_WORKER_MODEL` | Pi default | Primary model selector |
+| `KXM_WORKER_FALLBACK_MODELS` | none | Up to eight ordered fallback models |
+| `KXM_WORKER_TOOLS` | Pi defaults | Comma-separated Pi tool allowlist |
+| `KXM_WORKER_CONTINUE` | `true` | Allow active-session resume |
+| `KXM_WORKER_INITIAL_CONTINUE` | same | Set `false` for a fresh first child only |
+| `KXM_WORKER_SESSION_ISOLATION` | `off` for upgrade compatibility | Set `workflow` to enable stable-default plus per-run contexts |
+| `KXM_WORKER_MAX_RUN_SESSIONS` | `128` | Retained workflow-specific sessions, range `1`–`1024` |
+| `KXM_WORKER_TOOL_TIMEOUT_MS` | `1860000` | Tool watchdog; `0` disables it |
+| `KXM_WORKER_ACTIVATION_TIMEOUT_MS` | `60000` | Delivered-message turn-start watchdog |
+| `KXM_WORKER_PROVIDER_RETRY_MS` | `60000` | Delay when provider fallbacks are exhausted |
+| `KXM_WORKER_DRAIN_MS` | `15000` | Graceful child shutdown window |
+| `KXM_WORKER_MAX_RESTARTS` | unlimited | Optional supervisor retry ceiling |
+| `KXM_WORKER_EXTENSION_PATHS` | discovery | Exact extension files, separated by the platform path delimiter |
+| `KXM_WORKER_SKILL_PATHS` | discovery | Exact skill files/directories, same delimiter |
+
+When exact extension or skill paths are supplied, automatic discovery is
+disabled only for that category. Review those paths as executable dependencies.
+
+The complete variable reference, limits, and examples are in
+[Configuration](configuration.md).
+
+---
+
+## Complete CLI guide
+
+Global options can appear on the root or a command group:
+
+```text
+--workspace <dir>   Select the .kxm workspace
+--json              Machine-readable output
+--dry-run           Plan without applying changes
+-h, --help          Contextual help
+```
+
+### Agent commands
+
+| Command | What it does |
+|---|---|
+| `kxm agent worker --name <n> --project <p>` | Starts one always-on supervised Pi RPC worker |
+| `--model <provider/model>` | Selects the primary Pi model |
+| `--fallback-models <a,b>` | Supplies ordered provider fallback models |
+| `--tools <a,b>` | Restricts available Pi tool names |
+| `--fresh-start` | Skips only the first resume; later recoveries may continue |
+| `--no-continue` | Disables all Pi session resume |
+| `--session-isolation <mode>` | Uses `workflow` for per-run contexts or `off` for the upgrade-compatible shared context (default) |
+
+The worker does not read model, tool, role, or ownership values from
+`agents.json`; pass operational settings explicitly.
+
+### Session commands
+
+| Command | What it does | Important limit |
+|---|---|---|
+| `kxm session start --id <id> --mix <names>` | Resolves roster names and writes a `kxm.session.v1` manifest | Does not start a process |
+| `kxm session start --id <id> --workflow <definition>` | Creates manifest and workflow asset directories | Records the whole roster; does not dispatch a workflow |
+| `kxm session brief [--status]` | Lists recent hub tasks and plans; `--status` is the one-line footer | Does not start a hub or a run |
+| `kxm session status` | Lists PID claims and recovery envelopes in local state | Does not read session manifests |
+| `kxm session stop [--wait-ms <n>]` | Requests managed process shutdown | Global workspace stop, identical in scope to `kxm hub stop` |
+
+### Workflow commands
+
+| Command | What it does |
+|---|---|
+| `kxm workflow list` | Lists runs from the local workspace SQLite database |
+| `kxm workflow get <runId>` | Shows one local run, stages, evidence, waits, and journal |
+| `kxm workflow start [definitionId] --payload <value> [--delivery-id <id>] [--event <name>]` | Posts one signed, deduplicated workflow-start webhook; payload is a JSON object or `@file` |
+| `kxm workflow export <runId> [--input <snapshot>] [--out-dir <assets-dir>]` | Writes proposed Markdown and JSON retrospectives |
+
+`list`, `get`, and the default `export` are local hub-host operations; they do
+not query a remote hub database.
+
+### Gate commands
+
+| Command | What it does |
+|---|---|
+| `kxm gate validate [--file <path>]` | Parses the same single file/inline source contract as the hub without printing secrets |
+| `kxm gate artifacts-exist --path <asset>` | Verifies a non-empty regular file remains inside the workspace asset root |
+| `kxm gate degrade <runId> <stageId> --requirement <key> --reason <text>` | Admin-only approval of a policy-declared lower peer minimum for the current attempt |
+| `kxm gate signal <runId> <signalKey> <status> <summary> [key=value...]` | Posts a signed, deduplicated `passed`, `warning`, or `failed` callback |
+| `kxm gate github watch --run-id ... --stage-id ... --signal-key ... --repo owner/name --pr n --required a,b` | Polls required GitHub checks and posts the signed result |
+
+`gate signal --delivery-id <id>` makes retries of one unchanged callback explicit. `gate github watch` also accepts `--timeout-ms`, `--interval-ms`, and `--delivery-id` in addition to its run, stage, signal, repository, pull request, and required-check options.
+
+There is no generic `kxm gate run <name>`. Names in `gates.json` are descriptive
+unless one of the implemented commands above executes them. `github watch`
+posts an exact failed signal and exits `4` on timeout.
+
+### Hub and session commands
+
+| Command | What it does |
+|---|---|
+| `kxm init` | Creates or validates a project; never copies package dogfood configuration |
+| `kxm hub start` | Starts the hub in the foreground and opens the local SQLite store |
+| `kxm hub bind <url>` | Binds this machine to a running hub |
+| `kxm hub unbind` | Removes this machine's hub binding |
+| `kxm hub view` | Checks `/health` and `/ready` |
+| `kxm hub stop [--wait-ms <n>]` | Requests generation-matched hub and worker shutdown |
+| `kxm dash` | Opens the real-time read-only metadata dashboard; prints one plain snapshot without a TTY |
+| `kxm session brief` | Prints the hub-local session snapshot |
+
+### Improvement command
+
+```text
+kxm improve [--target cli|project]
+```
+
+This groups redacted `.kxm/logs/telemetry.jsonl` records into a proposed-only
+report. Named project/workflow activity is classified as `project`; unscoped
+operator activity is `cli`. `KXM_IMPROVE_TARGET=cli|project` explicitly
+overrides that classification. The command does not automatically modify code,
+configuration, gates, or the workflow journal.
+
+### Context commands
+
+| Command | What it does |
+|---|---|
+| `kxm context get <project> --role <role> --task <task>` | Assembles a role-aware context packet |
+| `kxm context recall <project>` | Searches durable context records (metadata only) |
+| `kxm context state <project> <key>` | Current or historical value for one state key (`--as-of`) |
+| `kxm context episode <project>` | Episodic learning records from workflow journals |
+| `kxm context promote <project> <proposalId> --evidence <refs>` | Promotes an approved state proposal (control plane) |
+| `kxm context explain <project> <itemId>` | Explains which evidence and lineage back a context item |
+| `kxm context wiki-compile <project>` | Compiles the knowledge wiki for review |
+| `kxm context wiki-lint <project>` | Lints a compiled wiki for broken refs, orphans, and stale state |
+| `kxm skills` | Governed skill candidate lifecycle |
+
+Agents use the same surfaces through Pi/MCP tools `kxm_context`, `kxm_recall`,
+`kxm_state`, `kxm_episode`, and `kxm_promote`. There is no `kxm_explain` tool;
+lineage is an operator CLI query.
+
+---
+
+## Pi integration
+
+### Interactive Pi
+
+Set the agent variables before starting Pi:
+
+```powershell
+$env:KXM_SERVER_URL = "http://127.0.0.1:7331"
+$env:KXM_AUTH_TOKEN = "<project-token>"
+$env:KXM_PROJECT = "product"
+$env:KXM_AGENT_NAME = "reviewer"
+$env:KXM_AGENT_PURPOSE = "Independent correctness reviewer"
+pi
+```
+
+Use `/kxm hub` to display the current identity and hub connection. Ask Pi to use
+the `kxm` skill before delegating complex work.
+
+### Always-on Pi workers
+
+`kxm agent worker` launches `pi --mode rpc`, captures its output, supervises
+restarts, and stays online after every individual inference. The hub is the only
+durable queue. The extension activates one message at a time and immediately
+selects the next item after settlement.
+
+Delivery priority is:
+
+1. `steer` at the next safe turn boundary;
+2. `followUp` for ordinary work; and
+3. `nextTurn`, normalized to a triggered follow-up for autonomous workers.
+
+A steer changes course; it does not abort an in-flight atomic write. Waiting
+work remains `queued`. Only work entering a model turn becomes `delivered`.
+
+### Tool and write boundaries
+
+`--tools` restricts tool names, not filesystem paths. A review-only worker can
+use:
+
+```text
+--tools read,grep,find,ls,kxm_list,kxm_send,kxm_get,kxm_await
+```
+
+A writer needs only the mutation and shell tools required by its assignment.
+Roster `roles` and `ownership` fields are documentation, not runtime policy.
+Use separate OS users, read-only worktrees, or containers for stronger
+boundaries.
+
+### Provider and tool recovery
+
+- Pi's own transient retries finish first.
+- A final quota/provider failure leaves the hub message recoverable, records
+  bounded metadata, closes Pi cleanly, and selects the next unused fallback.
+- A long-running tool beyond the watchdog follows the same recovery path without
+  rotating models.
+- An invalid `--continue` history retries fresh in the same binding and records a
+  recovery envelope.
+- Raw provider/model output remains in the protected `pi-agent-*.log`; structured
+  lifecycle logs contain only bounded metadata.
+
+---
+
+## Workflow-specific Pi sessions
+
+This feature prevents one long-lived Pi worker from mixing unrelated workflow
+histories.
+
+### Routing model
+
+| Message kind | Pi session binding |
+|---|---|
+| Ordinary peer/operator work | Stable `{agent, default}` session |
+| Root workflow prompt | `{agent, workflowRunId}` |
+| Signed callback resume or timeout notification | Same workflow binding |
+| Peer request with authorized `workflowContext` | Same workflow binding |
+| Message with only `correlationId: run_*` | No workflow affinity; correlation is not authorization |
+
+The hub adds a canonical, hub-owned `workflowRunId` to every workflow-origin
+message. Run IDs must match `run_` plus 32 lowercase hexadecimal characters.
+
+### Safe switch lifecycle
+
+1. The extension examines the next queued message before acknowledgement.
+2. If its binding matches, the extension acknowledges it and starts one turn.
+3. If it differs, the message remains queued.
+4. The extension atomically writes a metadata-only, generation-bound route
+   request and asks the current Pi child to shut down.
+5. The supervisor waits for the child's `close` event, updates the binding
+   manifest, and starts exactly one replacement with the target `--session-dir`.
+6. The destination extension reconnects and receives the same queued message ID.
+
+No request/reply body is written to a routing file. A route request includes only
+identity, supervisor generation, child incarnation, source and destination
+bindings, Pi session ID, message ID, and timestamp. The supervisor also rejects symbolic-link/junction aliases
+for its scoped session roots and verifies canonical containment. A malformed, stale, wrong-owner, or wrong-source request is
+rejected without changing scope.
+
+### Durability and retention
+
+Bindings live in `.kxm/state/worker-session-binding-<workerKey>.json`. Restarting
+the supervisor reloads the active binding and continues only when that session
+directory has Pi JSONL history. Inactive workflow sessions are retained up to
+`KXM_WORKER_MAX_RUN_SESSIONS`; least-recently-used inactive entries and
+directories are deleted when the bound is reached.
+
+A corrupt binding manifest is quarantined with a `.corrupt-<timestamp>` suffix.
+The supervisor does not guess a run; it creates a safe default binding. The hub
+workflow run, journal, messages, assets, and Git remain the recovery authority.
+
+### Enablement and compatibility
+
+The upgrade-compatible default is one shared Pi history:
+
+```text
+kxm agent worker ... --session-isolation off
+```
+
+Enable scoped histories explicitly:
+
+```text
+kxm agent worker ... --session-isolation workflow
+```
+
+The first isolated start uses fresh scoped storage. KXM does not copy the old
+shared `--continue` history because one shared directory may contain sessions
+from several workers and cannot be attributed safely. Both the CLI and low-level
+supervisor default to `off` during this compatibility release.
+
+---
+
+## Claude Code integration
+
+### Configure the plugin
+
+Provide these plugin settings:
+
+| Setting | Example |
+|---|---|
+| KXM server URL | `http://127.0.0.1:7331` |
+| Authentication token | Project token, never the admin token |
+| Agent name | `claude-reviewer` or `fable` |
+| Agent purpose | `Independent review and UI/UX criticism` |
+| Project | `product` |
+
+Restart Claude Code after changing settings. Use `/mcp` to confirm the bundled
+`kxm` MCP server connected, then call `kxm_list`.
+
+KXM does not choose the Claude model. Select the required Claude CLI/model
+profile separately; the hub identity and model session remain different
+concepts.
+
+### Pushed channel mode
+
+During the Claude channel research preview, explicitly trust the community
+channel:
+
+```text
+claude --dangerously-load-development-channels plugin:kxm
+```
+
+If the organization has approved it through `allowedChannelPlugins`:
+
+```text
+claude --channels plugin:kxm
+```
+
+Inbound work arrives as `<channel source="kxm" ...>` events. Process one
+request, call `kxm_reply`, and keep the Claude session open for the next event.
+
+### MCP pull mode
+
+Without channels, Claude retains full outbound and workflow capability. For
+inbound work:
+
+1. call `kxm_inbox`;
+2. process one durable request;
+3. call `kxm_reply` with its message ID; and
+4. repeat with bounded backoff while the inbox is empty.
+
+Do not describe pull mode as push-driven liveness.
+
+### Claude and Pi context differences
+
+Per-workflow automatic Pi session routing applies to the supervised Pi worker,
+not to Claude Code. A Claude operator who needs strict run isolation should use
+a separate Claude session/agent identity per run or an external Claude
+supervisor. Workflow authority still comes from `kxm_workflow_get`, the hub
+journal, evidence, and assets—not from either harness's context window.
+
+---
+
+## Hub tools
+
+### Shared outbound and workflow tools
+
+These tools are available in Pi and Claude MCP:
+
+| Tool | Purpose |
+|---|---|
+| `kxm_list` | List online peers, purposes, and models |
+| `kxm_send` | Send one focused request; returns a durable message ID |
+| `kxm_fanout` | Send the same independent request to one through three peers |
+| `kxm_get` | Inspect a request without blocking |
+| `kxm_await` | Wait only when the response blocks progress |
+| `kxm_cancel` | Cancel queued/delivered work owned by the sender |
+| `kxm_workflow_list` | List durable workflow runs assigned to this coordinator |
+| `kxm_workflow_get` | Read stages, evidence policies, waits, and journal |
+| `kxm_workflow_checkpoint` | Submit a stage result with keyed evidence and verified message references |
+| `kxm_workflow_wait` | Save evidence and pause until an authenticated callback |
+| `kxm_workflow_record` | Record a plan, decision, contradiction, error, or lesson |
+| `kxm_improvement_report` | Summarize learning by improvement area |
+
+Claude MCP also exposes:
+
+| Tool | Purpose |
+|---|---|
+| `kxm_inbox` | Reconcile and list durable inbound work in pull mode |
+| `kxm_reply` | Reply to one inbound request |
+
+### Messaging rules
+
+- Use `followUp` by default; reserve `steer` for an active blocker.
+- Supply an idempotency key when a send may be retried.
+- A local `kxm_await` or fanout timeout does not cancel the durable message.
+- Use `kxm_get` or repeat the exact idempotent operation; do not invent a new
+  request while the first remains pending.
+- Cancellation cannot undo filesystem or external side effects.
+- Never put credentials or unnecessary private data in a hub message.
+- Keep one task and one owner per request.
+
+---
+
+## Durable workflows
+
+### Configure the active definition source
+
+Set exactly one of:
+
+```text
+KXM_WEBHOOK_WORKFLOWS_FILE=.kxm/config/workflows/product.json
+```
+
+or:
+
+```text
+KXM_WEBHOOK_WORKFLOWS=[...inline JSON...]
+```
+
+The hub loads only that source for its current boot. A workflow definition
+contains an ID, source/provider, project, target coordinator, secret environment
+variable names, filters, delivery mode, prompt template, ordered stages,
+required evidence, attempt limits, and optional peer policies.
+
+Validate before restart:
+
+```powershell
+kxm gate validate --file .kxm/config/workflows/product.json
+```
+
+Secrets belong in environment variables named by `secretEnv` and
+`signalSecretEnv`, never in definition JSON.
+
+### Start and inspect
+
+```powershell
+kxm workflow start product-workflow `
+  --payload '@.kxm/assets/workflows/product/inputs/request.json' `
+  --delivery-id 'ticket-123-attempt-1'
+kxm workflow list
+kxm workflow get run_<32-hex-characters>
+```
+
+The stable delivery ID deduplicates identical provider retries. Reusing it with
+a conflicting body fails.
+
+### Coordinator procedure
+
+For every active stage:
+
+1. call `kxm_workflow_get`;
+2. follow only `currentStage`;
+3. record material plans, decisions, contradictions, errors, and lessons;
+4. gather exact required evidence;
+5. send peer-policy work with exact `workflowContext` when required;
+6. checkpoint or enter an external wait; and
+7. correct warnings/failures until passed or attempts are exhausted.
+
+Settling while a run is still `running` without a valid checkpoint fails the
+run. Settling after a successful `kxm_workflow_wait` is expected and releases
+compute until the callback creates a fresh message.
+
+### Workflow journal categories
+
+| Category | Use |
+|---|---|
+| `plan` | Intended execution and ownership |
+| `decision` | Selected option and rationale |
+| `contradiction` | Incompatible evidence, claims, requirements, or tests |
+| `error` | Failed tools, assumptions, integrations, or gates |
+| `lesson` | Evidence-supported reusable improvement |
+
+Areas are `harness`, `gates`, `implementation`, `workflow`, `documentation`,
+`security`, or `other`.
+
+---
+
+## Evidence gates and external callbacks
+
+### Ordinary evidence
+
+Every `requiredEvidence` key needs a non-empty string under the exact normalized
+key. Extra evidence cannot substitute for a missing requirement.
+
+### Peer-reply evidence
+
+A `peer-reply` policy requires durable replies from eligible stable agent IDs.
+The coordinator must send or fan out with:
+
+```json
+{
+  "workflowContext": {
+    "runId": "run_<32-hex>",
+    "stageId": "review",
+    "requirementKey": "independent review",
+    "attempt": 1
+  }
+}
+```
+
+Then cite the replied message IDs under the exact requirement in
+`evidenceRefs`. The hub verifies sender, recipient, project, run, stage,
+requirement, attempt, lifecycle, and eligible producer. Quorum counts unique
+producers. Caller-authored text, correlation IDs, idempotency keys, and copied
+result envelopes never satisfy peer provenance.
+
+The coordinator is never an eligible producer for its own run. Peer provenance
+proves durable routing inside the shared project-token trust boundary—not truth,
+model identity, non-collusion, independence, or human approval.
+
+### Explicit degradation
+
+Only a human/operator with the administrative token can approve a lower minimum,
+and only when the policy declared one:
+
+```powershell
+kxm gate --dry-run --json degrade <runId> <stageId> `
+  --requirement "independent review" --reason "documented incident"
+kxm gate --json degrade <runId> <stageId> `
+  --requirement "independent review" --reason "documented incident"
+```
+
+Approval is journaled and bound to the current attempt. It does not pass the
+stage; the coordinator must still provide the approved minimum references.
+
+### External waits and signals
+
+The coordinator calls `kxm_workflow_wait` with a stable signal key, expected
+result, timeout, and any already verified evidence. A separate operator or
+integration posts:
+
+```powershell
+kxm gate signal <runId> github-pr-42-checks passed "CI passed" `
+  "github.check:ci=https://ci.example/pr/42"
+```
+
+Callbacks are HMAC-signed, context-checked, deduplicated, and accumulated with
+saved evidence. A failed callback consumes the attempt; re-enter the wait before
+sending a new result.
+
+---
+
+## Live TUI and observability
+
+Start the dashboard with the administrative credential:
+
+```powershell
+kxm --workspace C:\work\product\.kxm dash
+```
+
+The TUI uses `@earendil-works/pi-tui`. It subscribes to authenticated
+metadata-only `/v1/ops/events` and refreshes `/v1/ops/snapshot`. It never loads
+or renders request/reply bodies. If admin operations access is unavailable, it
+honestly labels and uses a hub-enforced presence-only SSE stream plus local
+metadata. It refuses an older/unmarked stream that cannot guarantee this mode. Synthetic TUI observers
+are excluded from the dashboard's own agent table and displayed counts, while
+remaining ordinary authenticated hub identities during fallback.
+
+| Key | Action |
+|---|---|
+| `1`–`6` | Agents, Tasks, Workflows, Plans, Inbox, or Procs |
+| `Tab` / `]` | Next tab |
+| `[` | Previous tab |
+| `←` / `→` | List pane / detail pane |
+| `↑` / `↓` | Select |
+| `PgUp` / `PgDn` | Scroll |
+| `h` | Toggle help |
+| `q` | Quit |
+
+Without a TTY, the command prints one ANSI-free snapshot. Use
+`kxm hub view --json` for a health result intended for automation.
+
+### Health and operations endpoints
+
+| Endpoint | Authentication | Content |
+|---|---|---|
+| `GET /health` | none | Liveness and online-agent count |
+| `GET /ready` | none | Storage readiness |
+| `GET /metrics` | admin outside loopback | Prometheus metrics |
+| `GET /v1/ops/snapshot?project=<p>` | admin | Project-scoped metadata, no bodies |
+| `GET /v1/ops/events?project=<p>` | admin | Metadata-only SSE wakeups |
+
+Structured hub and worker logs omit message bodies. Raw `pi-agent-*.log` files
+may contain model output and must be protected accordingly.
+
+---
+
+## Context operating system (v0.5)
+
+The `kxm context` command group exposes KXM's context operating system:
+role-aware packets, temporal project state, episodic learning, a compiled
+knowledge wiki, and a governed skill lifecycle. Agents use the same features
+through `kxm_context`, `kxm_recall`, `kxm_state`, `kxm_episode`, and
+`kxm_promote` (Pi and MCP); provider-specific memory APIs are never exposed.
+
+### Role-aware packets, recall, episodes, and explain
+
+`kxm context get <project> --role <role> --task <task>` assembles a
+token-budgeted packet. Roles shape selection: repro agents get prior
+reproductions and incidents; planners get state and decisions; critics get
+contradictions and failed approaches; implementers get the approved plan and
+skills; verifiers get acceptance evidence. Superseded and rejected records
+are excluded by default, and every packet is project-isolated.
+
+`kxm context recall <project> --query <text>` searches durable context records
+and returns metadata only. `kxm context episode <project>` lists episodic
+learning records from workflow journals (`--run` limits to one run).
+`kxm context explain <project> <itemId>` explains which evidence and lineage
+back a context item.
+
+### Temporal state and promotion
+
+State keys follow a `proposed → current → superseded` lifecycle with
+deterministic historical queries (`--as-of`). Agents may propose changes;
+promotion requires durable evidence and an authorized control-plane decision:
+
+```bash
+kxm context promote <project> <proposalId> --evidence "receipt:run_9/verify"
+```
+
+### Compiled knowledge wiki
+
+`kxm context wiki-compile` renders `.kxm/knowledge/wiki/` from reviewed
+records; every claim links its evidence, superseded state stays visible, and
+open contradictions are never silently resolved. `kxm context wiki-lint`
+checks broken refs, orphan pages, stale state links, and unsurfaced
+contradictions.
+
+### Governed skills
+
+`kxm skills` turns verified episodes into candidates, gates them behind
+static, sandbox, functional, and safety evaluations, and pins promoted skills
+with content hashes. See `docs/skills.md` for the full lifecycle.
+
+### Reference /fix workflow
+
+`.kxm/config/workflows/fix.json` implements the reference bug-fix flow:
+read-only exploration, a tests-only reproduction draft, independent two-critic
+`repro-review` that captures the immutable oracle (a sibling API is invalid),
+plan review by independent critics, a human approval gate, bounded rework
+through typed transitions, and a ready-for-human-acceptance end state.
+Delivery never auto-merges. After `repro_invalidated`, rewrite against the
+named seam; `new DbContext()` is not grounds for `blocked`.
+
+## Reliability, privacy, and security
+
+### Delivery guarantees
+
+- SQLite is the durable source for agents, messages, runs, and journals.
+- Queued and delivered messages replay after hub or agent restart.
+- Delivery is at-least-once, not exactly-once.
+- Make external side effects idempotent and use stable delivery/message keys.
+- Terminal messages expire after the configured retention window.
+
+### Privacy contract
+
+- The TUI and operations SSE are metadata-only.
+- Structured logs include actors, project, delivery, status, model, and state,
+  but not prompt/reply bodies.
+- SQLite stores message bodies as sent and is not application-encrypted.
+- Raw Pi logs may contain model/tool output.
+- Routing manifests and requests contain no message or reply bodies.
+
+Protect `.kxm/state` and `.kxm/logs` with OS permissions and encrypted storage
+where required.
+
+### Security boundaries
+
+- Bind to loopback by default.
+- Use distinct high-entropy admin and project credentials.
+- Give workers only project tokens.
+- Terminate TLS at a trusted proxy for remote access and disable SSE buffering.
+- Protect workflow HMAC secrets separately from hub tokens.
+- Treat plugin extension paths and skills as executable privileged inputs.
+- Do not rely on prompts, roster roles, or ownership fields as a sandbox.
+- Do not expose the hub directly to the public internet.
+
+Session isolation prevents accidental model-context mixing; it is not a sandbox
+against a hostile process running under the same OS account. A shell-capable
+agent can reach files and routing environment values available to that account.
+Use separate accounts, containers, read-only worktrees, or stricter tool sets
+when the agent itself is outside the trust boundary.
+
+---
+
+## Backup, upgrade, and recovery
+
+### Backup
+
+SQLite uses WAL mode. The simplest safe backup is:
+
+1. stop the hub gracefully;
+2. copy `.kxm/state/kxm.db` to protected storage;
+3. record the package version and reviewed configuration; and
+4. restart and verify `/ready`.
+
+For online backup, use a SQLite-aware backup tool or capture the database,
+`-wal`, and `-shm` consistently.
+
+Pi session directories are supplementary model history, not authoritative
+workflow state. Include them only if your recovery policy needs local model
+context.
+
+### Upgrade
+
+1. back up SQLite;
+2. install the target release;
+3. stop the old hub and workers cleanly;
+4. start the new hub against the same database;
+5. verify health, readiness, operations SSE, and one request/reply; and
+6. restart workers so they use the matching extension and supervisor release.
+
+### Session-state recovery
+
+| Event | Meaning | Action |
+|---|---|---|
+| `worker_session_routed` | Expected clean child swap to another binding | No action unless repeated for one message |
+| `worker_session_evicted` | Inactive LRU run history removed at the configured bound | Preserve workflow facts in journal/assets/Git |
+| `worker_session_state_recovered` | Invalid manifest quarantined; safe default created | Inspect `.corrupt-*`, hub run state, and disk/concurrency health |
+| `worker_session_request_rejected` | Route request failed identity/schema/source checks | Check release match, generation, permissions, and duplicate supervisors |
+| `worker_continue_fallback` | Pi history could not continue; same binding starts fresh | Read recovery journal and durable message state |
+
+Never repair a live route by editing state files. Stop the exact worker first,
+preserve evidence, and recover from hub-owned workflow state.
+
+---
+
+## Troubleshooting checklist
+
+1. Confirm `kxm hub start` is running.
+2. Check `/health`, then `/ready`.
+3. Compare URL, project, and project token on both agents.
+4. Confirm unique live names.
+5. Run `/kxm hub` in Pi or `kxm_list` in Pi/Claude.
+6. Inspect structured logs without copying secrets or raw model content.
+7. For queued workflow work, look for the one expected session route swap.
+8. For a delivered message, inspect the recipient and activation/tool watchdogs;
+   do not send duplicates.
+9. For a workflow, call `kxm_workflow_get` and use the exact active stage,
+   requirement keys, attempt, wait key, and coordinator.
+10. For Claude, inspect `/mcp`; if channel push is unavailable, use
+    `kxm_inbox`/`kxm_reply`.
+
+Common causes:
+
+| Symptom | Likely cause |
+|---|---|
+| `kxm` not found | Only the Pi package was installed; install the release CLI or use the source wrapper |
+| Pi shows `hub:off` | URL/token/project mismatch, duplicate live name, missing extension, or unreachable hub |
+| Claude tools missing | Plugin not reloaded, MCP bundle unavailable, or unsupported Node version |
+| Request stays queued | Recipient offline/SSE unavailable, or one safe Pi session swap is in progress |
+| Request stays delivered | Agent turn, approval, tool, provider, or settlement is still active |
+| Fanout returns pending | Local wait expired; use returned message IDs, not replacement sends |
+| Workflow cannot advance | Missing exact evidence, wrong attempt/context, insufficient verified producers, or exhausted attempts |
+| Admin route returns 401/403 | Project token used where the distinct admin token is required |
+| Degradation returns 503 | Hub has no configured administrative credential |
+| Shared workflow memory appears | Worker is using the upgrade-compatible isolation `off` default; restart explicitly with `--session-isolation workflow` |
+
+See [Troubleshooting](troubleshooting.md) for error-specific recovery.
+
+---
+
+## Feature availability matrix
+
+| Feature | Operator CLI | Pi | Claude Code |
+|---|---:|---:|---:|
+| Start/stop hub and workers | Yes | No | No |
+| Initialize workspace | Yes | No | No |
+| Peer discovery | Status/TUI only | `kxm_list` | `kxm_list` |
+| Send, poll, wait, cancel | No | Yes | Yes |
+| Fanout to 1–3 peers | No | Yes | Yes |
+| Pushed inbound turns | N/A | Native extension | Optional channel |
+| Pull inbox and explicit reply | N/A | Extension owns queue | `kxm_inbox`, `kxm_reply` |
+| Always-on worker supervision | Starts Pi worker | Yes | External Claude supervision required |
+| Workflow-specific model sessions | Configures mode | Automatic for supervised Pi | Use separate Claude sessions externally |
+| Workflow list/get | Local SQLite CLI | Hub tools | Hub tools |
+| Start signed workflow | Yes | No | No |
+| Checkpoint/wait/journal | No | Yes | Yes |
+| Peer provenance context/references | No | Yes | Yes |
+| Admin quorum degradation | Yes | Forbidden | Forbidden |
+| Signed external signal | Yes | No | No |
+| GitHub check watcher | Yes | No | No |
+| Artifact existence gate | Yes | Can invoke CLI if shell is allowed | Can invoke CLI if shell is allowed |
+| Retrospective export | Yes | Can record source evidence | Can record source evidence |
+| Proposed telemetry improvement report | Yes | `kxm_improvement_report` covers workflow journal separately | Same |
+| Context packets, recall, state, episodes, explain | Yes | `kxm_context`, `kxm_recall`, `kxm_state`, `kxm_episode` | Same MCP tools |
+| Promote state proposals | Yes | `kxm_promote` | `kxm_promote` |
+| Governed skills | Yes | No | No |
+| Live metadata-only TUI | Yes | No | No |
+
+---
+
+## Related pages
+
+- [Getting started](getting-started.md)
+- [Configuration reference](configuration.md)
+- [Architecture](architecture.md)
+- [Operations guide](operations.md)
+- [Webhook workflows](webhook-workflows.md)
+- [Peer provenance and quorum gates](provenance-gates.md)
+- [Troubleshooting](troubleshooting.md)
+- [Test matrix](test-matrix.md)
+- [Changelog](../CHANGELOG.md)