@kontextmind/kxm 0.7.95 → 0.7.96

package/docs/operations/runtime-sync.md

@@ -0,0 +1,192 @@
+# Operate Runtime sync and leases
+
+The [Runtime](../glossary.md#runtime) supervisor pushes a redacted summary of every run event to the [hub](../glossary.md#hub), so hub-side views such as `kxm tenant status` see runs from every machine. This page is for operators: it explains the sync loop, how to read its state, how to clear stalled or refused rows, and what the hub's fenced leases do and do not cover today.
+
+## Before you begin
+
+- A KXM project (`kxm init`) with at least one Runtime run, for example from `kxm run`.
+- A hub for this machine: `kxm hub bind <url>`, or `KXM_SERVER_URL` in the environment that starts the supervisor.
+- A credential the hub accepts for the project's `prj_*` id (see [Which hub and credential sync uses](#which-hub-and-credential-sync-uses)).
+
+## How sync works
+
+Sync is outbound only: the Runtime pushes, and the hub never calls the Runtime. The following diagram shows one event from commit to acknowledgement, and the operator step for a refused row.
+
+```mermaid
+sequenceDiagram
+  participant E as Run event store
+  participant S as Supervisor sync tick
+  participant H as Hub
+  actor O as Operator
+  E->>E: Commit event and outbox row in one transaction
+  loop Every KXM_RUNTIME_SYNC_INTERVAL_MS (10 s)
+    S->>H: POST /v1/runtime/presence
+    S->>H: POST /v1/sync/events (pending rows in outbox order)
+    H-->>S: accepted, duplicate, conflict or rejected per event
+    S->>E: Ack accepted and duplicate rows
+    S->>E: Park conflict and rejected rows with the hub's code
+  end
+  Note over S,H: Hub unreachable or credential refused: rows stay pending and the tick backs off
+  O->>S: kxm runtime sync-retry, after fixing the hub side
+  S->>E: Re-queue parked rows
+```
+
+### What an outbox row contains
+
+Every committed Runtime event writes one outbox row in the same SQLite transaction, so an event is never committed without its row. The row holds a new `kxm.sync-event.v1` object built from the event, never the event itself:
+
+- Only allowlisted fields are copied. Every other field is dropped and named in `redaction.fieldsOmitted`.
+- Registered secret values (the hub token and the values of `KXM_*_TOKEN`, `KXM_*_KEY`, `*_API_KEY` and `*_SECRET` environment variables) and credential shapes are replaced with `[redacted]`.
+- Absolute paths become `[path]`, control characters are removed, and text is bounded.
+- The default sync policy allows prompt titles only, bounded result summaries, evidence references, artifact metadata and changed-file paths. It allows no raw logs, diffs or environment values.
+
+The prompt text itself stays in the local `run-prompts.json` sidecar next to the run store.
+
+### The sync tick
+
+Every `KXM_RUNTIME_SYNC_INTERVAL_MS` (default 10 seconds, clamped to 250 ms to 60 seconds, read when the supervisor starts), the supervisor visits each project it owns:
+
+1. It posts `POST /v1/runtime/presence` with its Runtime id and host label. The hub stamps the heartbeat with its own clock.
+2. It posts pending rows to `POST /v1/sync/events` in outbox order, up to 32 rows or about 200 KB per request. The hub accepts at most 100 events and 256 KiB per request.
+3. It acknowledges rows the hub accepted or already held, and parks rows the hub refused for good.
+
+Local execution never waits on sync. With no hub bound, rows simply stay pending.
+
+### How the hub decides
+
+The hub accepts each event once, keyed by `{projectId, runId, sequence}`:
+
+- The same bytes again are an idempotent `duplicate`.
+- Out-of-order events are held. The per-run cursor is the gapless prefix, so a gap stays open until the missing event arrives.
+- Different bytes under a used sequence, a project id already claimed by another hub project, a run homed on another Runtime, and an event pushed by a Runtime other than its home are refused and raise a `security_alert` in the hub log.
+- An event that fails the sync schema is refused without an alert.
+
+### Which hub and credential sync uses
+
+- **Hub:** `KXM_SERVER_URL` from the supervisor's environment, else the `kxm hub bind` binding, which is re-read on every tick.
+- **Project:** the `prj_*` id in `.kxm/project.yaml`, not the package or directory name. The hub pins a project id to the first hub project that claims it.
+- **Credential:** `KXM_AUTH_TOKEN` from the supervisor's environment, else the project token saved under that `prj_*` id in `hub-env.json`, else the saved admin token. To use a project token, key its `KXM_PROJECT_TOKENS` entry by the `prj_*` id.
+
+> [!IMPORTANT]
+> The supervisor keeps the environment of the command that started it. After you change `KXM_SERVER_URL` or `KXM_AUTH_TOKEN`, run `kxm runtime stop` and `kxm runtime start`.
+
+## Check sync status
+
+`kxm runtime status` reads what the last tick saw from the supervisor's `GET /v1/sync/status`, a token-protected loopback endpoint:
+
+```bash
+kxm runtime status
+```
+
+Expected output:
+
+```text
+runtime supervisor running: rtm_74573b9df67705d8f6596518 pid 41437 on 127.0.0.1:50724
+sync prj_a17d607765e144cd95b93a8715d360b6: ok (pending 0, acked 1, refused 0)
+```
+
+A hub that went away looks like this:
+
+```text
+sync prj_a17d607765e144cd95b93a8715d360b6: blocked (pending 0, acked 1, refused 0) — last error: fetch failed; next attempt 2026-09-23T18:41:31.675Z
+```
+
+| State | Meaning | Action |
+|---|---|---|
+| `ok` | Rows are being acknowledged | None |
+| `no_hub` | No hub is bound or set; rows are kept locally by design | Bind a hub if you want hub-side views |
+| `blocked` | Transport or credential failure; rows stay pending and the tick backs off, up to 5 minutes | Fix the cause; the next attempt resumes on its own |
+| `refusing` | The hub durably refused rows, or answered without a result for some | Fix the hub side, then run `kxm runtime sync-retry` |
+
+A project whose run store this release cannot open shows `blocked (its store is not readable by this build)` with the reason; see [Upgrade KXM](upgrade.md#understand-schema-changes). Right after the supervisor starts, the status can read `no project registered with this Runtime yet` until the first tick.
+
+Add `--json` for the full record: `state`, the outbox counts with refusal codes and counts, `consecutiveFailures`, `storeReadable`, `hubUrl`, `lastCompletedAt`, `lastPushed`, `lastAcked`, `lastRefused`, `lastUnconfirmed`, `lastError` and `nextAttemptAt`.
+
+The same changes appear once each in the Runtime log, `runtime/logs/kxm-runtime.jsonl` under the user state root:
+
+```text
+{"level":"warn","component":"runtime","event":"runtime_sync_stalled","projectId":"prj_a17d607765e144cd95b93a8715d360b6","state":"blocked","pending":0,"acked":1,"refused":0,"reason":"fetch failed","nextAttemptAt":"2026-09-23T18:41:31.675Z"}
+```
+
+## Restart or replace the supervisor
+
+The Runtime ID survives restarts. The registry holds one supervisor record. A new supervisor that finds it stopped, or finds its heartbeat older than 15 seconds or its process gone, takes the record over and keeps its `rtm_` ID, so every run's home Runtime stays valid. A supervisor whose record was taken over exits.
+
+A live supervisor (running, with a fresh heartbeat and a live process) keeps the record: a second supervisor refuses to start with `runtime_supervisor_conflict`, and `kxm runtime start` reuses the running one and prints `already running`.
+
+On start, and again on every tick, the supervisor reopens every project registered to its Runtime ID, so pending rows resume syncing without a new `kxm run`. A project that cannot reopen, such as a moved checkout or a store this build refuses, shows `blocked` with `storeReadable: false`, and the Runtime log records `runtime_sync_context_unavailable` once per change.
+
+## Clear a refused sync
+
+A refused row leaves the pending queue with the hub's code, so it can neither block the rows behind it nor re-alert the hub on every tick. Refused rows are never deleted, and the Runtime never decides on its own that a refusal has become retryable.
+
+| Code | Meaning | What to do |
+|---|---|---|
+| `sync_project_mismatch` | The hub already recorded this `prj_*` id under another hub project, usually from an older release that sent the package name | No command reassigns the claim; resolve it on the hub, then retry |
+| `sync_home_runtime_mismatch` | The run is homed on a different Runtime, for example a run store copied to a second machine | Sync the run only from its home Runtime |
+| `sync_runtime_mismatch` | The event names a different home Runtime than the one pushing it | Investigate the `security_alert`; do not retry blindly |
+| `sync_sequence_reused` | The hub holds different bytes for the same run and sequence | Investigate the `security_alert`; a diverged copy of a run store is the usual cause |
+| `sync_event_invalid` | The event failed the hub's sync schema, usually a release mismatch | Run the same release on the Runtime and the hub, then retry |
+| `sync_row_too_large` | One row exceeds the hub's request size on its own | It will be refused again; keep it for diagnosis |
+| `sync_row_unreadable` | The local outbox row is corrupt | Keep the store for diagnosis |
+
+After the hub side is fixed, re-queue the parked rows from the project checkout:
+
+```bash
+kxm runtime sync-retry --dry-run
+kxm runtime sync-retry
+```
+
+Expected output:
+
+```text
+re-queued 3 refused outbox rows for prj_a17d607765e144cd95b93a8715d360b6
+```
+
+The command needs a running supervisor and never starts one. It clears the backoff, so the rows go out on the next tick.
+
+## See synced runs on the hub
+
+The hub's admin `GET /v1/ops/snapshot?project=<prj-id>` adds `homeRuntimes`: each Runtime with its host label, heartbeat and presence lease (`heartbeatAt` plus the hub's stale window, on the hub clock), and its runs with bounded title and status, `lastSequence` and `pendingGap`. A Runtime whose lease lapses is marked `orphaned`. That is a view state only: nothing moves the runs elsewhere.
+
+`kxm tenant status` shows hub metadata next to the authoritative local run state, labelled by source. Hub workflow runs and Runtime runs use separate id spaces, so a comparison without a shared id is reported as unverified, never as agreement.
+
+The hub counts sync traffic in `kxm_sync_events_accepted_total`, `kxm_sync_events_duplicate_total`, `kxm_sync_events_refused_total`, `kxm_sync_conflicts_total` and `kxm_runtime_heartbeats_total`; see [Monitor KXM](monitoring.md#scrape-metrics).
+
+## Use fenced leases
+
+A [lease](../glossary.md#lease) lets one agent at a time act on a shared resource, such as a branch, and lets that resource refuse a holder whose time has run out.
+
+### What the hub provides
+
+- `POST /v1/leases/<resource>/acquire`, `/renew` and `/release`, authenticated as a registered agent with its project token and agent key. The hub prefixes the caller's project onto the resource name, so two projects that name the same branch never contend.
+- A time-to-live of 5 seconds to 10 minutes (default 5 minutes), decided on the hub clock inside one store transaction, so a client with a skewed clock cannot extend its own lease.
+- A fencing token that starts at 1, stays the same across renewals, and increments whenever an expired lease is taken again. Present it with every write to the shared resource so a late holder is refused.
+- Clear refusals: `lease_held` (another live holder), `lease_expired` (re-acquire to continue), `lease_superseded` (your token is stale), all HTTP 409, and `lease_not_found` (HTTP 404).
+- Expired rows kept for the 7-day run-retention window, so the next takeover increments past their token, then purged.
+- Metrics `kxm_leases_granted_total`, `kxm_leases_refused_total` and `kxm_leases_released_total`, and log events `lease_acquired`, `lease_renewed`, `lease_released`, `lease_denied` and `lease_purged`.
+
+The `HubClient` class in `@kontextmind/kxm/client` wraps the three routes as `acquireLease`, `renewLease` and `releaseLease`.
+
+### What is not wired
+
+> [!IMPORTANT]
+> No `kxm` command or agent tool takes a lease, and the Runtime does not take one before an external effect such as a push, a pull request, a tracker issue or an outgoing webhook. Leases protect only the integrations that call the API themselves. The intended effect-and-recovery design is described in [Effects and recovery](../contracts/effects-and-recovery.md).
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| State stays `no_hub` although a hub runs | The supervisor has no `KXM_SERVER_URL` and the machine is not bound | Run `kxm hub bind <url>` |
+| `blocked` with `401 invalid_auth` | The hub does not accept the credential for the `prj_*` id | Add a project token keyed by the `prj_*` id, or export the right token and restart the supervisor |
+| `blocked` with `fetch failed` | The hub is down or the URL is wrong | Start the hub, or fix the binding |
+| `refusing` right after an upgrade | The Runtime and the hub run different releases | Upgrade both, then run `kxm runtime sync-retry` |
+| `runtime_not_running` from `kxm runtime sync-retry` | The supervisor is stopped | Run `kxm runtime start` first |
+| Runs appear as `orphaned` on the hub | That Runtime stopped sending presence | Start it, or accept that its runs are no longer live |
+
+## Next steps
+
+- Watch sync alongside the rest of the service: [Monitor KXM](monitoring.md)
+- Protect the outbox and run stores: [Back up and restore KXM](backup-and-restore.md)
+- The designed sync contract: [Hub synchronization contract](../contracts/synchronization.md)
+- Hub routes for sync and leases: [Hub HTTP API reference](../reference/http-api.md)

package/docs/operations/troubleshooting.md

@@ -0,0 +1,265 @@
+# Troubleshoot KXM
+
+This page is a reference of symptoms, causes and fixes, grouped by area. Start with the quick check, then go to the area that matches: install, hub and authentication, Claude Code, peer messaging, Pi workers, workflows, Runtime runs, context and memory, or operations.
+
+## Start with a quick check
+
+Work from the smallest boundary outward:
+
+1. Run `kxm hub view`. Both `health` and `ready` must be `true`.
+2. Compare the hub URL, project and project token on every agent involved (`KXM_SERVER_URL`, `KXM_PROJECT`, and the token or plugin `auth_token`).
+3. Confirm every agent has a unique name within its project.
+4. List peers: `/kxm hub` in Pi, `kxm_list` in Claude Code, or `kxm peer list`.
+5. Read the hub log for `agent_registered`, `agent_stale` and `request_error` events; see [Monitor KXM](monitoring.md#read-the-logs).
+6. For a workflow, call `kxm_workflow_get` and work only on its `currentStage`, attempt and required evidence.
+7. For Runtime runs, run `kxm runtime status`.
+
+## Install
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `kxm: command not found` | Only the Pi package or the Claude Code plugin was installed; neither installs the CLI | `npm install --global --omit=peer @kontextmind/kxm`; see [Install KXM](../start/install.md) |
+| `kxm` works in one shell but not in another | npm's global bin directory is not on that shell's `PATH` | Run `kxm completion install` (bash and zsh also get a `PATH` entry), then open a new shell |
+| Tab completion does nothing | Completion is not installed for this shell | `kxm completion install --shell bash`, `zsh` or `fish`; it is idempotent, and `--dry-run` previews it |
+| Plugin tools fail on start | Node.js is older than 22.19 on the 22.x line, or older than 24 | Install a supported Node.js on the `PATH` the harness uses |
+
+Install the CLI from the npm registry as shown above; [Install KXM](../start/install.md) covers every supported path. Set `KXM_SKIP_COMPLETION_PROMPT=1` to skip the completion offer after `kxm init`.
+
+### `pi update` fails with `couldn't find remote ref refs/heads/master`
+
+The KXM default branch is `main`, and an older Pi checkout still tracks `master`. Remove the package and install it again with an explicit branch:
+
+```bash
+pi remove git:github.com/kontextmind/kxm
+pi install git:github.com/kontextmind/kxm@main
+```
+
+### `kxm harness list` says Claude Code is `not_detected` on Windows
+
+npm installs Claude Code as `claude.cmd` and an extensionless shim, not `claude.exe` on `PATH`. `kxm harness list` retries `claude.exe`, then the package's inner `claude.exe`, then `claude.cmd`, and reports `issues: ["windows_shim"]` when only the shim answered.
+
+If it still reports `not_detected`, confirm that `%AppData%\Roaming\npm` is on the `PATH` of the process that runs `kxm`, then check `claude --version` and `claude auth status`. Assignment dispatch (`just assign`) runs the inner `claude.exe`, and Pi's `node.exe` with `cli.js`, without a shell; it refuses unverified `.cmd` launchers. The same shim miss can affect `pi`.
+
+## Hub and authentication
+
+### The hub refuses to start
+
+| Message | Fix |
+|---|---|
+| `KXM_PORT must be an integer between 0 and 65535` | Set a valid port, or unset `KXM_PORT` to use `7331` |
+| `KXM_AUTH_TOKEN is required when binding beyond localhost` | Set `KXM_HOST=127.0.0.1`, or provide the admin token; see [Deploy KXM](deploy.md#choose-loopback-or-a-network-bind) |
+| `KXM hub env file is malformed at <file>` or `does not use schema kxm.hub-env.v1` | Fix the JSON. Removing the file drops every saved token, so start with the full `KXM_PROJECT_TOKENS` map and update all clients |
+| `KXM hub is already managed by PID <pid>` | A hub already owns this state directory; run `kxm hub stop` first |
+| `EADDRINUSE` (address already in use) | Stop the other process, or choose another `KXM_PORT` and update every client's hub URL |
+| `local_state_root_not_absolute` | Make `KXM_STATE_HOME` an absolute path |
+| `runtime_schema_newer` | The database came from a newer release; do not delete it; upgrade KXM |
+| `runtime_schema_outdated` | The database predates this release; see [Upgrade KXM](upgrade.md#understand-schema-changes) |
+
+### Agents fail with `invalid_auth` after a hub restart
+
+`KXM_PROJECT_TOKENS` replaces the hub's saved project map, and the hub saves the replacement. A start with a one-project value removes every other project, so their agents are refused. Restart the hub with the full map; the merge command in [Start the hub](../start/quickstart-claude-code.md#3-start-the-hub) builds it from the saved file.
+
+### A hub PID claim is stale
+
+KXM refuses to replace a live hub or worker claim, and `kxm hub stop` ignores a claim that is invalid, not running, or owned by someone else rather than guessing. A claim whose wrapper died is reclaimed on the next `kxm hub start`, which first stops an orphaned server child; `kxm hub stop` can stop such an orphan directly.
+
+If a malformed claim remains, read the exact `.pid` JSON in the state directory and confirm its PID is not running and, for a hub, that nothing listens on the port. Then remove only that `.pid` file and its recorded `.stop` control file, and start once. Worker claim names include an identity digest, so do not substitute a similar file name. Never delete the state directory or the database to clear a claim.
+
+### Pi shows `hub:off`
+
+- Confirm the hub is reachable from the Pi terminal (`kxm hub view`).
+- Check the project name, and give the agent its project token rather than the admin token.
+- Check whether a live agent already uses the same name in the project.
+- Restart Pi after you change environment variables.
+- For long-lived workers, check `KXM_WORKER_EXTENSION_PATHS` and `KXM_WORKER_SKILL_PATHS`; invalid paths fail before supervision starts. See [Run supervised Pi workers](../guides/pi-workers.md).
+
+### Admin routes return 401 or 503
+
+`/metrics`, `/v1/ops/*`, state promotion and quorum degradation need the admin token. A project token never substitutes for it, even when you hold every project's token.
+
+- **401 `invalid_auth`:** the request carried a project token or a wrong admin token. Once the hub has an admin token, this applies on loopback too.
+- **503 `admin_auth_not_configured`:** the hub was started without an admin token, which `kxm hub start` never does. Stop the hub, set `KXM_AUTH_TOKEN`, keep the full `KXM_PROJECT_TOKENS` map, and restart against the same database. Give the admin token only to the operator terminal.
+
+### `kxm hub bind` refuses the URL
+
+| Error | Fix |
+|---|---|
+| `hub_url_invalid` | Use an `http` or `https` URL without user info, query or fragment |
+| `hub_bind_unauthenticated` | The URL is remote and this machine has no token for the project; export it and bind again |
+| `hub_credential_unreadable` | Repair or remove `hub-env.json` under the user state root |
+
+## Claude Code plugin and MCP
+
+The [plugin troubleshooting guide](../../plugins/kxm/README.md#troubleshooting) has the full list. Run fixes in your own terminal, and never paste a token into Claude.
+
+| Symptom | Fix |
+|---|---|
+| `kxm_*` tools do not appear | Check `/mcp` for the `kxm` server, check `node --version`, run `/reload-plugins`, and confirm `claude plugin list` shows `kxm@kxm` enabled |
+| `KXM hub unreachable at <url>` | Start the hub, or correct `server_url` with `/plugin configure kxm@kxm` |
+| `no project token for project <p>` | Enter the project token at `/plugin configure kxm@kxm`, or add `<p>` to the hub's full `KXM_PROJECT_TOKENS` map and restart the hub |
+| `KXM hub rejected the project token for project <p>` | Enter the token the hub holds for `<p>` |
+| `tool_policy_denied: Session token on disk is malformed or expired` | Run `kxm session token --clear` |
+| Pushed requests never arrive | Start Claude Code with `claude --dangerously-load-development-channels plugin:kxm@kxm` and accept the trust prompt, or use `kxm_inbox` and `kxm_reply` |
+| `kxm is already at the latest version` but the plugin is old | The release job sets the version only inside its build and never commits it, so updates never refresh the plugin. [Reinstall it](../start/quickstart-claude-code.md#update-the-plugin) |
+| No KXM brief at session start | Start Claude Code from the directory that contains `.kxm/` |
+
+## Peer messaging
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| An expected peer is missing | Different `KXM_PROJECT` values, or the peer stopped sending heartbeats | Compare settings and look for `agent_stale`; names display case-sensitively but live-name uniqueness ignores case |
+| Registration returns HTTP 409 `duplicate_agent_name` | A live agent already uses the name in this project | Stop the old session, or choose another name |
+| A request stays `queued` | The recipient has no active event stream | Confirm its process is connected; proxies must not buffer `/v1/events` |
+| A request stays `delivered` | The recipient acknowledged it and is still working, waiting for approval, or blocked | Check the recipient directly; do not resend. `kxm_cancel` changes hub state only; it cannot undo work already done |
+| A message disappears after completion | Terminal messages are purged after 7 days | Raise `KXM_MESSAGE_RETENTION_MS`, and keep durable results in Git |
+
+### `kxm_await` times out
+
+`kxm_await` waits at most 60 seconds, which is both its default and its cap. A timeout does not end the request. Check it with `kxm_get`, and use `kxm_workflow_wait` for long external work. `cancelled`, `expired` and `error` are terminal. Resend only work that is safe to repeat, with an idempotency key after an uncertain network result.
+
+### Fanout returns pending before a model replies
+
+`kxm_fanout.timeoutMs` is a local wait, not the message lifetime. A pending result includes the durable `messageId`, its status and expiry, and whether the wait timed out or was aborted. Inspect it with `kxm_get`, or repeat the exact fanout with the same correlation ID, idempotency prefix, targets and content. Do not send a replacement with a new prefix. Usually omit `ttlMs` for model work, and never count a pending peer toward a checkpoint. See [Message peer agents](../guides/peer-messaging.md).
+
+## Pi workers
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| A continued Pi session rejects every turn | The worker stopped during a tool call, leaving a `tool_use` without its `tool_result` | Nothing: the worker retries once without `--continue` and writes a recovery envelope; keep the same project and agent name |
+| The agent settles on a quota or provider error | Pi's own retries are exhausted | Set `--fallback-models` (or `KXM_WORKER_FALLBACK_MODELS`) to rotate at once; otherwise the worker retries after `KXM_WORKER_PROVIDER_RETRY_MS` |
+| The heartbeat is healthy but one tool never finishes | A tool exceeded `KXM_WORKER_TOOL_TIMEOUT_MS` (31 minutes by default, above the 30-minute fanout wait) | The worker logs `worker_tool_timeout` and restarts the child; raise the limit only above the longest legitimate call |
+| A long-lived worker keeps restarting | Pi is missing from the service `PATH`, the working directory is gone, or model credentials are missing | Read `worker_process_error` and `worker_exited`; set `KXM_PI_COMMAND` to an explicit path |
+| A read-only reviewer edits files | Prompt wording does not remove tools | Set `KXM_WORKER_TOOLS=read,grep,find,ls` |
+| The worker exits at once with `pi_native_impersonation_blocked` | `--model` or a `--fallback-models` entry is a model whose vendor has its own harness, such as `xai/…`, `openai-codex/…` or `openrouter/x-ai/…` | Run that model in its native harness, or pick an admitted Pi route such as `openrouter/qwen/qwen3-coder-plus`; see [Harness routing](../reference/harness-routing.md#what-the-brake-refuses) |
+
+Use `--fresh-start`, not `--no-continue`, when only the first launch must avoid old session state.
+
+### A workflow message stays queued while the worker restarts once
+
+With `--session-isolation workflow`, a message for a different run is deliberately left queued in the current Pi context. Look for `worker_session_routed`: the old child closes, one child starts with the run's `--session-dir`, and the same message ID replays and becomes `delivered`.
+
+If it repeats, read `worker_session_request_rejected` and check that the worker was started with `kxm agent worker`, that `KXM_WORKER_SESSION_SCOPE` was not set by hand, that only the service account can write the state directory, that the hub and worker run the same release, and that the message has a hub-owned `workflowRunId`. Never acknowledge the message by hand, edit the route request, copy a run history into `default`, or start a second worker with the same identity.
+
+### Session events
+
+| Event | Meaning | Action |
+|---|---|---|
+| `worker_session_routed` | Expected swap to another run's session | None unless it repeats for one message |
+| `worker_session_evicted` | An inactive run history was removed at the retention bound | Keep workflow facts in the journal, assets or Git |
+| `worker_session_state_recovered` | A malformed binding manifest was quarantined as `.corrupt-<timestamp>` and routing restarted at `default` | Read `kxm_workflow_get`, re-drive unfinished work from durable message IDs, keep the quarantined file, and check disk and permissions |
+| `worker_continue_fallback` | Pi history could not continue, so the same binding started fresh | Read the recovery envelope and the durable message state |
+
+If a workflow seems to remember another run, confirm the worker log says `"sessionIsolation":"workflow"`. Isolation is off by default; restart with `--session-isolation workflow`. The first isolated start uses fresh storage, and history from a formerly shared session cannot be separated afterward. See [Run supervised Pi workers](../guides/pi-workers.md).
+
+## Workflows and gates
+
+### A workflow tool returns `workflow_forbidden`
+
+Read `operation`, `assignedCoordinatorName` and `nextAction` in the error. Only the assigned coordinator can read, wait on, checkpoint or journal a run; do not retry as a peer.
+
+### A workflow cannot advance
+
+A passing checkpoint needs a keyed, non-empty value for every declared `requiredEvidence` key; extra keys do not count. Warnings and failures stay active until corrected. When attempts run out, or the coordinator settles early, the run fails and its journal records why. Decide whether repeating external effects is safe before you start a new delivery.
+
+For a `peer-reply` requirement, read `resolvedEvidencePolicies`, `verifiedEvidence` and the current attempt. An evidence string cannot satisfy it; the checkpoint must cite replied message IDs in `evidenceRefs` before those messages are purged, and every eligible agent must have registered before the run started.
+
+| Code | Meaning |
+|---|---|
+| `workflow_context_forbidden` | The sender is not the run's assigned coordinator |
+| `workflow_context_inactive` | The run or stage is not running |
+| `workflow_context_attempt_mismatch` | Use `stage.attempts + 1`, and send fresh work after a retry |
+| `workflow_evidence_producer_forbidden` | The target is not in the run's eligible producer set |
+| `workflow_evidence_policy_missing`, `workflow_evidence_policy_unresolved` | The requirement has no usable peer policy |
+| `workflow_provenance_invalid` | A cited message is missing, pending, ineligible, or bound to another project, run, stage, requirement or attempt |
+| `workflow_evidence_incomplete` | Fewer unique verified producers than the minimum; several replies from one peer count once |
+
+If the policy declares a lower `degradation.minProducers`, an operator with the admin token can approve it for the current stage and attempt. Preview with `kxm gate degrade <run-id> <stage-id> --requirement <key> --reason <text> --dry-run --json`, then run it without `--dry-run`. Approval does not advance the run; the coordinator still checkpoints. See [Peer provenance and quorum gates](../guides/provenance-gates.md).
+
+### GitHub checks passed but the run is still waiting
+
+The hub does not poll GitHub. Run `kxm gate github watch` with the run's `--run-id`, `--stage-id` and `--signal-key`. On timeout it posts the signed `failed` signal and exits `4`; it never invents `passed`.
+
+### A webhook or callback is rejected
+
+| Response | Meaning |
+|---|---|
+| 401 | The HMAC-SHA256 signature is missing or does not match the raw body; a callback must use `signalSecretEnv` when the definition sets it |
+| 400 | The delivery ID or JSON body is missing; `workflow_evidence_incomplete` names `missingRequirements`; `invalid_workflow_evidence` means non-string or duplicate keys |
+| 404 | The definition or run ID does not match this hub |
+| 409 `workflow_target_unavailable` | The configured coordinator has never registered; start it once with the matching project and name |
+| 409 `workflow_not_waiting` | No wait is active: `kxm_workflow_wait` was not called, the deadline failed the run, or a prior signal advanced it |
+| 409 `workflow_signal_mismatch`, `workflow_signal_context_mismatch` | Use the run's exact `waiting.signalKey`; fix or omit `workflow.run`, `workflow.stage` and `workflow.signal` evidence |
+| 204 | The event or filter did not match, so no run was intended |
+| 200 with `duplicate: true` | A retry of the same delivery ID was deduplicated |
+
+A failed or timed-out callback consumes that wait. Re-enter the wait and start a new `github watch` or `kxm gate signal`; keep an explicit `--delivery-id` only for retries of one unchanged body. See [Run webhook workflows](../guides/webhook-workflows.md).
+
+### A workflow file fails to load with `gate_outcome_impossible`
+
+A gate step declares an outcome its `expect` value never produces. The message names the outcomes to declare instead.
+
+## Runtime runs
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `runtime supervisor is not running` or `runtime_not_running` | The supervisor stopped | `kxm runtime start`; commands such as `kxm run` also start it on demand |
+| `runtime_supervisor_unreachable` | A live supervisor process does not answer its token probe | Check the PID from `kxm runtime status`, stop a hung process with your OS tools, then `kxm runtime start` |
+| `project_required` | The command ran outside a KXM project | Run it from the checkout, or run `kxm init` |
+| `producer_route_not_admitted` | A live drive uses a model route that is not admitted | `kxm routes admit --model <provider/model>`, or drive with `--simulated` |
+| `pi_not_authenticated: pi harness not detected (pi_native_impersonation_blocked)` | A Pi agent's model belongs to a vendor with its own harness | Set the agent's native `harness:`, or choose an admitted Pi route whose vendor has none |
+| `run_busy` (HTTP 409) | The run is already admitted or queued for a drive | Wait, and check `kxm runs status <run-id>` |
+| A run store is refused with `runtime_schema_outdated` | The store predates this release | See [Upgrade KXM](upgrade.md#understand-schema-changes) |
+| Runs do not reach the hub | Sync is `no_hub`, `blocked` or `refusing` | See [Operate Runtime sync and leases](runtime-sync.md#check-sync-status) |
+
+## Context and memory
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| 403 `context_isolation_violation` | An agent asked for another project's context | Use the agent's own project; cross-project reads need the admin token |
+| State promotion returns 401 or 503 | Promotion needs the configured admin token; agents can only propose | Promote with `kxm context promote <project> <proposal-id>` from the operator terminal |
+| `kxm context recall` returns nothing | No live record matches; superseded and rejected records are excluded | Broaden the query, or check `unresolvedGaps` |
+| A new memory fact does not appear in `kxm memory brief` | `kxm memory note` records a candidate, which becomes active only when promoted through a pull request | Review and merge the candidate, then run `kxm memory sync` |
+| `memory sync failed: none of AGENTS.md, CLAUDE.md, GEMINI.md exists` | Sync updates only the instruction files a project already has | Create the file your harness reads, then sync again |
+| `memory sync failed: <file> has …; wrote no file` | An orphan memory marker, an end marker before its start, or a second block | Keep exactly one `kxm:memory` marker pair in the file, or delete both, then sync again |
+
+See [Context and memory](../guides/context-and-memory.md).
+
+## Operations
+
+| Failure | Behavior | Recovery |
+|---|---|---|
+| An agent exits | It is marked offline after the stale window | Restart it with the same name to resume its ID |
+| The event stream drops | The client reconnects while heartbeats continue | If it repeats, check the network and proxy buffering |
+| The hub or a worker exits | SQLite keeps agents and messages, and binding manifests keep the Pi scope | Restart. Queued messages are pushed again, by the same ID, until acknowledged; delivered messages are not replayed |
+| The disk fails or fills | Readiness or writes fail | Restore storage, then check database integrity and `/ready` |
+| An external callback is lost | The run stays `waiting` until its deadline, then fails and notifies the coordinator | Retry with the same delivery ID, or review side effects before a new delivery |
+| Requests return 429 `rate_limited` | The per-agent or per-address window is full | Honor `Retry-After`; raise `KXM_RATE_LIMIT_MAX` if the load is expected |
+| `kxm restore` refuses with `runtime_schema_mismatch` | A backup file's schema version differs from the one its manifest records | Use another backup set; never mix files between sets |
+
+For backup and restore errors such as `backup_no_stores`, see [Back up and restore KXM](backup-and-restore.md#troubleshooting). The dashboard's action keys do not act on runs; see [Monitor KXM](monitoring.md#keys).
+
+## Collect a useful bug report
+
+Include:
+
+- the operating system and Node.js version;
+- the Pi or Claude Code version, and the KXM version (`kxm --version`) or Git commit;
+- whether the hub is local or behind a proxy;
+- redacted environment values, never tokens;
+- the relevant structured hub events;
+- exact reproduction steps and the expected behavior.
+
+Never attach tokens, private prompts, credentials, raw `pi-agent-*.log` files, or unrelated repository content.
+
+## For maintainers
+
+- **`kxm --help` prints an old flat command list.** The committed `plugins/kxm/dist/cli.js` is stale. Run `npm run build` and commit the generated `dist`.
+- **Every CI job stays queued while a runner is online.** See [CI and release](../contributing/ci-and-release.md#ci-jobs-stay-queued-while-a-runner-is-online).
+- **Loading an exact extension in Pi during development.** Use `pi --no-extensions -e ./plugins/kxm/src/extension.ts`, adding every required provider extension with another `-e`; see [Develop KXM](../contributing/development.md).
+
+## Related
+
+- [Monitor KXM](monitoring.md)
+- [Deploy KXM](deploy.md)
+- [CLI reference](../reference/cli-reference.md)
+- [Claude Code plugin](../../plugins/kxm/README.md)

package/docs/operations/upgrade.md

@@ -0,0 +1,124 @@
+# Upgrade KXM
+
+Move the `kxm` CLI, the hub, the Runtime, and the Claude Code or Pi integrations to a new release, and roll back if the new release misbehaves. This page is for operators. KXM stores have no migrations, so the order of steps matters more than usual.
+
+## Before you begin
+
+- Protected storage for a full backup; see [Back up and restore KXM](backup-and-restore.md).
+- A window in which the hub, the Runtime supervisor and long-lived workers may stop.
+- The GitHub CLI (`gh`, signed in) if you apply updates with `kxm update --kxm` from the default `github` source, which downloads the release tarball with it.
+
+## Check for an update
+
+```bash
+kxm update --check
+```
+
+Expected output when a newer release exists:
+
+```text
+kxm 0.7.1 → 0.7.2 available · kxm update --kxm
+```
+
+From a source checkout, the check makes no network call and reports the checkout instead:
+
+```text
+kxm 0.7.1 (running from source at /work/kxm)
+```
+
+The check reads `update.yaml` under the user state root. Its `source` is `github` (release tarballs) by default; `npm` is also accepted. See [Updater settings](../reference/config-reference.md#updater-settings-kxmupdatev1). Add `--json` to see the install kind KXM detected.
+
+## Stop the services and back up
+
+Stop everything that holds a KXM database open, in this order, and confirm each one is down:
+
+```bash
+kxm runtime stop
+kxm hub stop            # also stops long-lived workers that recorded PID claims
+kxm runtime status      # expect "runtime supervisor is not running"
+```
+
+Pause service-manager restarts and background hub starts (`hub.autoStart`) until the upgrade finishes. An old process that keeps running against a store the new release rewrites, or a new process that opens an old store, fails closed at best.
+
+Then take the stopped-state backup in [Back up everything else](backup-and-restore.md#back-up-everything-else), and record the current version with `kxm --version`.
+
+## Upgrade the CLI
+
+Use the path that matches how KXM was installed. `kxm update --kxm` applies an update only for a global npm install; for every other install kind it exits `2` and prints the command to use instead.
+
+| Install kind | Upgrade with |
+|---|---|
+| Global npm install | `kxm update --kxm` (release tarball, or npm with `source: npm`), or `npm install --global --omit=peer @kontextmind/kxm@latest` |
+| Pi package | `pi update` |
+| Claude Code marketplace plugin | Reinstall the plugin; see the plugin note below |
+| Project dependency | `npm install @kontextmind/kxm@latest` in that project |
+| Source checkout | `git pull`, then `npm ci` |
+
+`kxm update --kxm` refuses to install a GitHub release that does not publish a SHA-256 digest for its tarball (`release_digest_missing`) or whose download does not match it (`release_digest_mismatch`). Preview the steps first:
+
+```bash
+kxm update --kxm --dry-run
+```
+
+### Update harnesses and plugins
+
+Without `--check` or a lone `--kxm`, `kxm update` also runs the native updaters of detected harnesses. Narrow it with a harness id and one scope:
+
+```bash
+kxm update --dry-run               # plan every detected harness
+kxm update pi --extensions         # pi update --extensions
+kxm update claude --extensions     # claude plugin update kxm -y (user scope); see the note below
+kxm update pi --models             # refresh Pi model catalogs
+```
+
+> [!IMPORTANT]
+> The release job sets the plugin's version only inside its build and never commits it, so the plugin's version never changes. Claude Code installs new plugin code only when that version changes, so `claude plugin update` (and `kxm update claude --extensions`, which runs it) reports "already at the latest version" and keeps the old copy. Reinstalling is the upgrade path: follow [Update the plugin](../start/quickstart-claude-code.md#update-the-plugin).
+
+## Start and verify
+
+1. Start the hub (or its service), then the Runtime with `kxm runtime start`.
+2. Check `kxm hub view`, `/ready` and `/metrics`.
+3. Check `kxm runtime status` until every project reports `ok` or `no_hub`.
+4. Send one request between two agents and read one Runtime run with `kxm runs status <run-id>`.
+5. Restart long-lived workers and reload Claude Code (`/reload-plugins`) so every agent runs the matching release.
+
+## Understand schema changes
+
+Each store records a schema version, and KXM never upgrades a store in place.
+
+| Store | Location | Created by |
+|---|---|---|
+| Hub store | `.kxm/state/kxm.db` (`KXM_DATA_PATH`) | `kxm hub start` |
+| Runtime registry | `runtime/registry.db` under the user state root | The Runtime supervisor |
+| Run event store | `runtime/projects/<key>/run-events.db` under the user state root | The Runtime supervisor |
+
+- A store **newer** than the running release refuses to open (`runtime_schema_newer`). Do not delete it; run the release that created it, or upgrade.
+- A store **older** than the running release also refuses to open (`runtime_schema_outdated`), and the error names the store. There is no migration.
+
+When a release changes a schema, either stay on the old release, or accept a fresh store: stop the owner, move the old file aside (keep it with your backup), and start the owner, which re-creates it. `kxm init` rebuilds no database. A fresh hub store loses message and workflow history; a fresh Runtime store loses local run history. Check the changelog for schema changes before you upgrade.
+
+> [!CAUTION]
+> Deleting a store is irreversible for the history it holds. Move it aside instead, and keep the pre-upgrade backup until the new release has run cleanly.
+
+## Roll back
+
+1. Stop the Runtime, the hub and the workers.
+2. Reinstall the earlier release the same way you upgraded (for npm, `npm install --global --omit=peer @kontextmind/kxm@<version>`).
+3. Restore the pre-upgrade backup of every store the new release touched. Never open a newer-schema store with an older release.
+4. Start and verify as above.
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `install_kind_source` or `install_kind_<kind>` from `kxm update --kxm` | KXM was not installed with a global npm install | Use the command from the table above |
+| `release_digest_missing` or `release_digest_mismatch` | The GitHub release has no or a different digest | Wait for a published release, or install from npm |
+| `kxm_update_config_invalid` | `update.yaml` has an unknown field or a wrong type | Fix it; only `schema`, `auto` and `source` are allowed |
+| The hub refuses to start with `runtime_schema_outdated`, or a Runtime project reports that its store is not readable | The store predates this release | Roll back, or move the store aside and start fresh |
+| Claude still runs the old plugin | Releases never change the plugin's version, so an update keeps the cached copy | Reinstall as [Update the plugin](../start/quickstart-claude-code.md#update-the-plugin) describes |
+
+## Next steps
+
+- Confirm the service is healthy: [Monitor KXM](monitoring.md)
+- Recover from a failed upgrade: [Back up and restore KXM](backup-and-restore.md)
+- Every `kxm update` flag: [CLI reference](../reference/cli-reference.md#kxm-update)

package/docs/prompts/browser-annotate-feedback.md

@@ -2,32 +2,32 @@
 schema: "kxm.doc.v1"
 id: "PROMPT-BROWSER-006"
 type: "prompt"
-title: "Capturing UI Section Annotations and Sending Changes to Agent"
+title: "Capture UI section annotations and send changes to an agent"
 project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-15"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
 summary: "Turn annotated Steel UI-section feedback into source changes and recapture verified proof."
 tags: ["browser", "annotation", "feedback", "prompt"]
-related: ["docs/browser-automation.md", "docs/kb/how-to-capture-and-annotate-section.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/how-to-capture-and-annotate-section.md"]
 ---
 
-# Task Template: Capturing UI Section Annotations and Sending Changes to Agent
+# Task template: capture UI section annotations and send changes to an agent
 
 ## Purpose
 
 Use this prompt when a human operator or design critic has reviewed a UI section in a Steel browser session and wants to send annotated visual change requests back to the agent for remediation.
 
-## Canonical Skill References
+## Canonical skill references
 
 - `kxm-browser-annotate`
 - `kxm-browser-verify`
 - `kxm-browser-session`
 
-## Parameters & Placeholders
+## Parameters and placeholders
 
 - **PROJECT_ID**: `{{PROJECT_ID}}`
 - **TASK_ID**: `{{TASK_ID}}`
@@ -40,7 +40,7 @@ Use this prompt when a human operator or design critic has reviewed a UI section
 
 ---
 
-## Instructions for Agent
+## Instructions for the agent
 
 1. **Review Visual Feedback**:
    - Inspect the section screenshot at `{{SCREENSHOT_ARTIFACT}}`.