@kontextmind/kxm 0.7.95 → 0.7.97

package/docs/kb/how-to-recover-expired-session-or-orphan.md

@@ -7,48 +7,58 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Procedures for detecting and releasing stale or orphaned browser sessions on Steel."
+summary: "Find and release stale or orphaned browser sessions on Steel."
 tags: ["browser", "cleanup", "orphans", "troubleshooting"]
-related: ["docs/browser-automation.md", "docs/kb/why-authentication-disappeared.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/why-authentication-disappeared.md"]
 ---
 
 # How do I recover an expired session or remove an orphaned browser?
 
-If an automation run crashed or disconnected without calling `/release`, a browser container may remain idling on DOKS.
+If an automation run crashed or disconnected without releasing its session, the
+browser can keep running on your Steel deployment until its timeout.
 
-## 1. List Active Remote Sessions
+## 1. List active sessions
+
+Load `STEEL_API_URL` and `STEEL_API_KEY` from your secret manager first. With
+`pass-cli`, for example:
 
 ```bash
-STEEL_KEY=$(pass-cli item view --vault-name "AI Provider Keys" --item-title "Steel Browser (KontextMind DOKS)" --field STEEL_API_KEY)
+export STEEL_API_URL="https://<steel-host>"
+STEEL_API_KEY=$(pass-cli item view --vault-name "<vault>" --item-title "<item>" --field STEEL_API_KEY)
+export STEEL_API_KEY
 
-curl -s https://steel.kontextmind.com/v1/sessions \
-  -H "x-steel-api-key: $STEEL_KEY" | jq .
+curl -s "$STEEL_API_URL/v1/sessions" \
+  -H "x-steel-api-key: $STEEL_API_KEY" | jq .
 ```
 
-## 2. Release Orphaned Sessions
-
-To terminate a specific stale session:
+## 2. Release an orphaned session
 
 ```bash
-curl -s -X POST https://steel.kontextmind.com/v1/sessions/<SESSION_ID>/release \
-  -H "x-steel-api-key: $STEEL_KEY"
+curl -s -X POST "$STEEL_API_URL/v1/sessions/<session-id>/release" \
+  -H "x-steel-api-key: $STEEL_API_KEY"
 ```
 
-## 3. Automatic Orphan Sweeping via KXM Client
+## 3. Sweep orphans with the KXM client
 
-The KXM client provides `checkOrphanedSessions(maxIdleMs)` to automate this:
+`checkOrphanedSessions(maxIdleMs)` returns two kinds of session: live or idle
+sessions this client does not track that have run longer than the limit, and
+tracked sessions idle longer than the limit, unless a person has taken over.
+`releaseSession()` releases one:
 
 ```typescript
 import { SteelClient } from "@kontextmind/kxm/runtime";
 
 const client = new SteelClient();
-const orphans = await client.checkOrphanedSessions(600000); // > 10 min idle
+const orphans = await client.checkOrphanedSessions(600000); // idle over 10 minutes
 
 for (const sessionId of orphans) {
   console.log(`Releasing orphaned session: ${sessionId}`);
   await client.releaseSession(sessionId);
 }
 ```
+
+It returns an empty list when the Steel API request fails, so an empty result
+does not prove there are no orphans. Check with the `curl` call above.

package/docs/kb/how-to-resume-after-mfa.md

@@ -7,22 +7,30 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Details the verification and observation refresh sequence when resuming automation after MFA."
+summary: "The verification and observation refresh an agent performs before resuming after MFA."
 tags: ["browser", "mfa", "resume", "verification"]
-related: ["docs/kb/how-to-take-over-session.md", "docs/browser-automation.md"]
+related: ["docs/kb/how-to-take-over-session.md", "docs/guides/browser-automation.md"]
 ---
 
 # How does an agent resume after MFA?
 
-Once the human completes the MFA challenge in the browser viewer, the agent must not immediately execute blind clicks. It follows this sequence:
+After you complete an MFA challenge in the session viewer, the agent does not
+click blindly. It follows this sequence:
 
-1. **State Transition**: Moves from `HUMAN_CONTROL` to `VERIFY_AUTHENTICATION`.
-2. **CDP Re-attachment**: Re-queries the active tab target from the remote Steel CDP endpoint.
-3. **App State Verification**:
-   - Inspects `page.url()` to confirm the browser navigated away from the MFA prompt to the intended destination (e.g. `/dashboard` or `/overview`).
-   - Checks for authenticated elements (e.g. account menu, logout button, user profile avatar).
-4. **Observation Refresh**: Runs a fresh `agent-browser snapshot` or queries fresh DOM locators before executing the next action.
-5. **Restore Control**: Returns to `AGENT_CONTROL` and proceeds with the task.
+1. **State transition.** The session moves from `HUMAN_CONTROL` to
+   `VERIFY_AUTHENTICATION`. The `SteelClient` keeps this state in memory for
+   its own lifetime; a new client does not inherit it.
+2. **CDP re-attachment.** The agent re-queries the active tab from the Steel
+   CDP endpoint.
+3. **Application state check.**
+   - It confirms that `page.url()` left the MFA prompt for the intended page,
+     for example `/dashboard`.
+   - It looks for signed-in elements such as an account menu or a sign-out
+     button.
+4. **Observation refresh.** It takes a fresh `agent-browser snapshot`, or
+   queries fresh DOM locators, before the next action.
+5. **Control returns.** The session returns to `AGENT_CONTROL` and the task
+   continues.

package/docs/kb/how-to-take-over-session.md

@@ -7,26 +7,30 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Instructions for taking over an active Steel browser session during an authentication gate."
+summary: "Take over an active Steel browser session at an authentication gate."
 tags: ["browser", "takeover", "auth", "mfa"]
-related: ["docs/browser-automation.md", "docs/kb/how-to-resume-after-mfa.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/how-to-resume-after-mfa.md"]
 ---
 
 # How do I take over a browser session to log in?
 
-When an agent encounters a login screen, OAuth prompt, or security challenge, it triggers the `kxm-browser-takeover` protocol.
+When an agent reaches a login screen, an OAuth prompt or a security challenge,
+it stops and asks you to take over, following the `kxm-browser-takeover` skill.
 
 ## Steps
 
-1. **Copy the Takeover URL**:
-   The agent will emit a message in the terminal with a link like:
-   `https://steel.kontextmind.com/ui?sessionId=<SESSION_ID>`
-2. **Open the Session Viewer**:
-   Open that URL in your desktop browser. You will see the live screencast of the exact remote Chrome container the agent was operating.
-3. **Interact and Authenticate**:
-   Enter the username, password, or security key into the session, or use the devtools inspector (`https://steel.kontextmind.com/v1/devtools/inspector.html`) to trigger the submission.
-4. **Signal Completion**:
-   Return to your Herdr or Pi terminal session and notify the agent: `auth complete` or `proceed`.
+1. **Copy the takeover URL.** The agent prints a session viewer link such as
+   `<steel-ui-url>?sessionId=<session-id>`. The viewer URL is `STEEL_UI_URL`,
+   or `<steel-api-url>/ui` when that is unset.
+2. **Open the session viewer** in your desktop browser. It shows a live
+   screencast of the remote browser the agent was driving.
+3. **Sign in.** Enter the username, password or security key in the session.
+   If clicks on the screencast do not register, use the DevTools inspector at
+   `<steel-api-url>/v1/devtools/inspector.html`; see
+   [Why can I view a session but not control it?](why-session-viewer-cannot-control.md).
+4. **Signal completion.** Return to the agent's terminal session and tell it
+   `auth complete` or `proceed`. It then verifies the sign-in before it resumes;
+   see [How does an agent resume after MFA?](how-to-resume-after-mfa.md).

package/docs/kb/why-authentication-disappeared.md

@@ -7,26 +7,34 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Diagnosing lost authentication state, session expiration, and ephemeral container recreation."
+summary: "Diagnose lost sign-in state: session expiry, a new session instead of an attached one, and cookie scoping."
 tags: ["browser", "authentication", "cookies", "troubleshooting"]
-related: ["docs/browser-automation.md", "docs/kb/why-automation-opened-different-browser.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/why-automation-opened-different-browser.md"]
 ---
 
 # Why did authentication disappear?
 
-If an agent was authenticated on a previous step or run and suddenly encounters a login screen again, the root causes are typically:
+If an agent was signed in on an earlier step or run and now sees a login screen
+again, one of these is usually the cause.
 
-## Root Causes & Fixes
+## Causes and fixes
 
-1. **Session Released or Expired**:
-   - Steel sessions are ephemeral by default. Once a session reaches its timeout (e.g. 5–30 minutes) or is released via `POST /v1/sessions/:id/release`, all memory cookies and local storage are cleared.
-   - **Fix**: To reuse state across tasks, save `storageState` via Playwright and reload it on the next session initialization.
-2. **New Session Launched Instead of Attaching**:
-   - If the agent created a brand-new session instead of passing the existing `sessionId`, it opened a clean Chrome profile.
-   - **Fix**: Verify that the task passes `sessionId` to `getSession()` or uses the existing CDP endpoint.
-3. **Domain / Subdomain Cookie Scoping**:
-   - OAuth flows often set cookies on subdomains (e.g., `auth.example.com`) that do not automatically share with `app.example.com`.
-   - **Fix**: Ensure cookies were issued for the primary domain or that SSO redirect completed fully before saving state.
+1. **The session was released or expired.**
+   - Steel sessions are ephemeral. When a session reaches its timeout (the KXM
+     client defaults to five minutes) or is released through
+     `POST /v1/sessions/<session-id>/release`, its cookies and local storage
+     are gone.
+   - **Fix:** to reuse sign-in state across tasks, save Playwright
+     `storageState` and load it when you start the next session.
+2. **A new session was launched instead of attaching to the existing one.**
+   - A new session starts with a clean browser profile.
+   - **Fix:** make sure the task passes the existing `sessionId` to
+     `getSession()`, or connects to the existing CDP endpoint.
+3. **Cookies were scoped to another domain.**
+   - Sign-in flows often set cookies on a subdomain, such as
+     `auth.example.com`, that `app.example.com` does not receive.
+   - **Fix:** make sure the cookies were issued for the primary domain, or that
+     the single sign-on redirect finished, before you save state.

package/docs/kb/why-automation-opened-different-browser.md

@@ -7,26 +7,35 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Preventing accidental local browser launches and ensuring Playwright and agent-browser connect to remote Steel."
+summary: "Prevent accidental local browser launches and make Playwright and agent-browser attach to remote Steel."
 tags: ["browser", "cdp", "playwright", "agent-browser", "troubleshooting"]
-related: ["docs/browser-automation.md", "docs/kb/how-to-connect-playwright-to-steel.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/how-to-connect-playwright-to-steel.md"]
 ---
 
 # Why did automation open a different browser?
 
-If you expected automation to run on DOKS Steel but saw a local Chrome window pop up or failed to see the agent's actions in the Steel session viewer:
+You expected automation to run on your Steel deployment, but a local Chrome
+window opened, or the agent's actions never appeared in the Steel session
+viewer.
 
-## Root Causes
+## Causes
 
-1. **Called `chromium.launch()` Instead of `chromium.connectOverCDP()`**:
-   - `chromium.launch()` spawns a local browser process on the workstation.
-   - **Fix**: In Playwright, always use `chromium.connectOverCDP(cdpUrl)`.
-2. **Missing `--cdp` Flag in `agent-browser`**:
-   - Running `agent-browser open <url>` without `--cdp` spawns a local headless browser.
-   - **Fix**: Always pass `--cdp "wss://steel.kontextmind.com/v1/devtools?sessionId=<id>&apiKey=<key>"`.
-3. **Missing Environment Variables**:
-   - If `STEEL_CDP_URL` is undefined, scripts that fall back to local execution will launch a local browser.
-   - **Fix**: Ensure `STEEL_CDP_URL` or `STEEL_API_KEY` is loaded from `pass-cli`.
+1. **The script called `chromium.launch()` instead of
+   `chromium.connectOverCDP()`.**
+   - `chromium.launch()` starts a browser on the local machine.
+   - **Fix:** in Playwright, connect with `chromium.connectOverCDP(cdpUrl)`.
+2. **`agent-browser` ran without `--cdp`.**
+   - `agent-browser open <url>` without `--cdp` starts a local headless
+     browser.
+   - **Fix:** always pass the session's CDP URL:
+     `--cdp "wss://<steel-host>/v1/devtools?sessionId=<session-id>&apiKey=<steel-api-key>"`.
+     Build it with `formatCDPEndpoint()`; see
+     [How do I connect Playwright to the existing Steel session?](how-to-connect-playwright-to-steel.md).
+3. **Environment variables were missing.**
+   - A script that falls back to local execution when `STEEL_CDP_URL` is
+     unset launches a local browser.
+   - **Fix:** load `STEEL_CDP_URL`, or `STEEL_API_URL` and `STEEL_API_KEY`,
+     from your secret manager before the run.

package/docs/kb/why-session-viewer-cannot-control.md

@@ -7,25 +7,26 @@ project: "kxm"
 status: "accepted"
 owner: "@operator"
 created: "2026-09-14"
-updated: "2026-09-14"
+updated: "2026-09-23"
 authority: "instruction"
 confidence: "verified"
-summary: "Understanding self-hosted Steel OSS screencast viewer capabilities vs devtools inspector input modes."
+summary: "The self-hosted Steel screencast viewer versus the DevTools inspector for interactive control."
 tags: ["browser", "takeover", "steel", "ui"]
-related: ["docs/browser-automation.md", "docs/kb/how-to-take-over-session.md"]
+related: ["docs/guides/browser-automation.md", "docs/kb/how-to-take-over-session.md"]
 ---
 
 # Why can I view a session but not control it?
 
-In self-hosted open-source Steel (`steel-dev/steel-browser`), the web UI at `/ui` provides a real-time screencast stream, event logs, and network tracking. Depending on browser canvas capture modes, direct clicks on the video canvas may not send synthetic mouse events.
+In self-hosted open-source Steel (`steel-dev/steel-browser`), the web UI at
+`/ui` shows a live screencast, event logs and network activity. Depending on how
+the canvas is captured, clicks on the video may not reach the browser.
 
 ## Resolution
 
-1. **Use the Chrome DevTools Inspector**:
-   Navigate to the devtools inspector page for the session:
-   `https://steel.kontextmind.com/v1/devtools/inspector.html`
-   or open the remote debugger on port 9223.
-2. **Interact via the DOM Console**:
-   The devtools inspector gives full interactive control over the DOM, console, network, and storage.
-3. **Agent Automation Coexistence**:
-   Ensure the agent is in `HUMAN_CONTROL` state so automation does not race or override your clicks.
+1. **Use the Chrome DevTools inspector.** Open the inspector page of your Steel
+   deployment, `<steel-api-url>/v1/devtools/inspector.html`, or the remote
+   debugger port your deployment exposes.
+2. **Interact through DevTools.** The inspector gives full control of the DOM,
+   console, network and storage.
+3. **Keep the agent out of the way.** Make sure the session is in
+   `HUMAN_CONTROL`, so automation does not race your clicks.

package/docs/operations/backup-and-restore.md

@@ -0,0 +1,248 @@
+# Back up and restore KXM
+
+Protect every piece of KXM state, not only the hub database, and bring it back after a disk loss, a bad change, or a move to a new host. This page is for operators. You learn which six roots hold state, what `kxm backup` and `kxm restore` do and do not cover, and a stopped-state procedure for the rest.
+
+## Before you begin
+
+- The `kxm` CLI and the project checkout, on the machine that runs the hub and the [Runtime](../glossary.md#runtime).
+- Permission to stop the hub, the Runtime supervisor, and anything that restarts them.
+- Protected, preferably encrypted, backup storage. Backups contain message bodies, prompts and, if you include them, credentials.
+
+## Know where state lives
+
+KXM spreads state over six roots. Some of them move with environment variables, and a backup that assumes one location silently misses another. The following diagram shows each root, what it holds, and the variable that moves it.
+
+```mermaid
+flowchart TB
+  subgraph R["$R checkout root: the Git checkout, never moves"]
+    R1["Project definition: .kxm/project.yaml, config.yaml, agents/, models/, workflows/, gates.yaml, roles/, role-hosts.yaml, routes.yaml, roster.yaml, prices.yaml, repo/, project/env.yaml"]
+    R2["Durable records: .kxm/memory/, skills/, goals/, tasks/, candidates/"]
+  end
+  subgraph D["$D workspace: KXM_WORKSPACE_DIR or --workspace, default $R/.kxm"]
+    D1["logs/ (KXM_LOGS_DIR): kxm-hub.jsonl (KXM_LOG_PATH), worker logs, telemetry.jsonl"]
+    D2["assets/ (KXM_ASSETS_DIR): retrospectives, improvements, evidence"]
+  end
+  subgraph W["$W workspace state: KXM_STATE_DIR, default $D/state"]
+    W1["kxm.db hub store (KXM_DATA_PATH)"]
+    W2["Pi sessions, worker manifests, PID claims"]
+  end
+  subgraph S["$S user state root: KXM_STATE_HOME"]
+    S1["runtime/registry.db, runtime/projects/KEY/run-events.db and prompt sidecars"]
+    S2["hub-env.json, hub-binding.json, update.yaml, projects/HASH/repository-bindings.json"]
+  end
+  subgraph C["$C user config: KXM_USER_CONFIG_DIR, default ~/.config/kxm"]
+    C1["config.yaml, roles/, workflows/, role-hosts.yaml, session.token"]
+  end
+  subgraph T["$T federated telemetry: XDG_CONFIG_HOME/kxm/telemetry"]
+    T1["model-metrics.jsonl, not written by any command today"]
+  end
+  R -->|".kxm/ by default"| D
+  D -->|"state/ by default"| W
+```
+
+| Root | Default | Moved by |
+|---|---|---|
+| `$R` checkout | The Git checkout | Nothing. `.kxm/` definition files always stay here |
+| `$D` workspace | `$R/.kxm` | `KXM_WORKSPACE_DIR` or `--workspace`, relative to `KXM_WORKDIR` or the current directory |
+| `$W` workspace state | `$D/state` | `KXM_STATE_DIR`; `KXM_DATA_PATH` moves only `kxm.db` |
+| `$S` user state root | `~/.local/state/kxm` (Linux, honoring `XDG_STATE_HOME`), `~/Library/Application Support/KXM` (macOS), `%LOCALAPPDATA%\KXM` (Windows) | `KXM_STATE_HOME`, which must be absolute |
+| `$C` user config | `~/.config/kxm` | `KXM_USER_CONFIG_DIR` |
+| `$T` federated telemetry | `~/.config/kxm/telemetry` | `XDG_CONFIG_HOME` |
+
+Three override rules catch people out:
+
+- `--workspace` derives `config/`, `logs/`, `assets/` and `state/` from one directory and ignores the per-directory variables. Without it, `KXM_CONFIG_DIR`, `KXM_LOGS_DIR`, `KXM_ASSETS_DIR` and `KXM_STATE_DIR` each move only their own target. `KXM_STATE_DIR=/srv/state` alone leaves `$D` at `$R/.kxm`.
+- A relative `KXM_STATE_HOME` fails with `local_state_root_not_absolute`. A relative `XDG_STATE_HOME` or `LOCALAPPDATA` base is ignored without an error, and the platform default is used.
+- `KXM_WORKER_LOG_PATH` and `KXM_AGENT_LOG_PATH` move worker logs out of `$D/logs`.
+
+Record every override with the backup. A restore that lands where the running service does not look is not a restore.
+
+## Know what each backup covers
+
+`kxm backup` protects SQLite stores it can find from the current directory. Everything else needs the stopped-state copy described below.
+
+> [!WARNING]
+> `kxm backup` does not back up the Runtime. It looks for Runtime stores under `.kxm/runtime/` in the checkout, but the Runtime writes them under the user state root (`$S/runtime/registry.db` and `$S/runtime/projects/<key>/run-events.db`). In practice a backup holds only the hub store. Back up the Runtime stores and their prompt sidecars by hand, stopped, as shown in [Back up everything else](#back-up-everything-else).
+
+| Path | Holds | In `kxm backup` |
+|---|---|---|
+| `$W/kxm.db` | Hub store: agents, messages, workflow runs, journals, context items, leases, synced run facts | Yes, at `<current directory>/.kxm/state/kxm.db` only |
+| `$S/runtime/registry.db` | Runtime registry: projects, their roots, the supervisor identity and claim | No |
+| `$S/runtime/projects/<key>/run-events.db` | Event-sourced runs, drive receipts, gate evidence, the sync outbox | No |
+| `$S/runtime/projects/<key>/run-events.db.run-prompts.json` | Run prompt text; restoring a store without it loses every prompt | No |
+| `$S/projects/<hash>/repository-bindings.json`, `$S/update.yaml` | Member repository paths; updater settings | No |
+| `$S/hub-env.json`, `$S/hub-binding.json`, `$C/session.token` | Credentials and the machine's hub binding | No; prefer regenerating secrets to copying them |
+| `$R/.kxm/` definition files and durable records | Project, roles, routes, prices, roster, memory, skills, goals, tasks, candidates | No; commit them to Git or copy the checkout |
+| Each member repository's `.kxm/repo/*.yaml` | Member definition and environment | No; they live in the member's own checkout |
+| `$W/worker-*.json`, `$W/pi-sessions/` | Worker routing and recovery manifests; Pi model history | No; manifests are required for resumable workers, Pi history is optional |
+| `$D/assets/`, `$D/logs/` | Retrospectives and evidence; logs and local usage accounting (`telemetry.jsonl`) | No |
+| `$C` | User-level roles, workflows and settings | No |
+
+A restore without `roster.yaml`, `routes.yaml` or `prices.yaml` comes back healthy but with different admission and cost behavior, so treat them as part of the backup even though they are plain files. `kxm improve report --out-dir` can write candidates outside `$R/.kxm/candidates/`; include that directory if you use it.
+
+These files are disposable and need no backup: `hub.pid`, `hub.stop`, `worker-*.pid`, `session-brief.json`, `update-check.json`, `runtime/supervisor.token`, `runtime/supervisor.error`.
+
+## Back up the hub store with `kxm backup`
+
+Run it from the checkout root. It resolves stores from the current directory and ignores `--workspace`, `KXM_WORKDIR`, `KXM_STATE_DIR` and `KXM_DATA_PATH`, so a relocated hub database is not found.
+
+Preview first. A dry run lists what it would write, opens no store and writes nothing:
+
+```bash
+cd /srv/kxm/product
+kxm backup --dry-run
+```
+
+Expected output:
+
+```text
+dry run: back up 1 store(s) to /srv/kxm/product/.kxm/backups/backup-2026-09-23T18-29-09-107Z (sources are not opened, so their WAL is not checkpointed)
+  would write /srv/kxm/product/.kxm/backups/backup-2026-09-23T18-29-09-107Z/kxm.db
+  would write /srv/kxm/product/.kxm/backups/backup-2026-09-23T18-29-09-107Z/manifest.json
+```
+
+Then write the backup outside the checkout:
+
+```bash
+kxm backup --out /backups/kxm/2026-09-23/hub
+```
+
+Expected output:
+
+```text
+Created SQLite backup with 1 store(s):
+  - hub-store: /srv/kxm/product/.kxm/state/kxm.db -> kxm.db (schema v5, 110592 bytes, sha256 sha256:86549...)
+Manifest: /backups/kxm/2026-09-23/hub/manifest.json
+```
+
+For each store, `kxm backup` checkpoints the write-ahead log, runs `PRAGMA integrity_check`, copies the database with `VACUUM INTO`, checks the copy's integrity, sets mode `0600`, and records its schema version and SHA-256 in `manifest.json` (`kxm.backup-manifest.v1`). `VACUUM INTO` reads one consistent snapshot, so the hub can keep running during this step.
+
+> [!NOTE]
+> Without `--out`, backups go to `.kxm/backups/` inside the checkout. Keep that directory out of Git, or always pass `--out`.
+
+It fails with `backup_no_stores` when `.kxm/state/kxm.db` does not exist under the current directory, and with `database_corrupted` when an integrity check fails.
+
+## Back up everything else
+
+Copy the remaining state with both services stopped. SQLite runs in write-ahead-log mode, so a plain copy of a live database can miss committed data.
+
+1. Stop the Runtime, then the hub, and confirm both are down:
+
+   ```bash
+   kxm runtime stop
+   kxm hub stop
+   kxm runtime status   # expect "runtime supervisor is not running" and exit status 1
+   kxm hub view         # expect "hub health=false ready=false" and exit status 1
+   ```
+
+2. Keep them down until the copy finishes. Pause your service manager's restart policy, Pi sessions with `hub.autoStart: background`, and anything that runs Runtime commands, because `kxm run`, `kxm runs list` and similar commands start the supervisor on demand.
+3. Copy the hub store and the user state root:
+
+   ```bash
+   S="${KXM_STATE_HOME:-$HOME/.local/state/kxm}"   # macOS: "$HOME/Library/Application Support/KXM"
+   B=/backups/kxm/2026-09-23
+   mkdir -p "$B"
+   kxm backup --out "$B/hub"
+   tar -C "$S" --exclude 'supervisor.token' --exclude 'hub-env.json' -czf "$B/user-state.tgz" .
+   ```
+
+   The archive holds `registry.db`, every project's `run-events.db` with any `-wal` and `-shm` files and its `.run-prompts.json` sidecar, the repository bindings, the hub binding and `update.yaml`. It leaves out the credential file; keep tokens in your secret store, or include `hub-env.json` and protect the archive as a secret.
+
+   If `KXM_STATE_DIR` or `KXM_DATA_PATH` moved the hub database, `kxm backup` fails with `backup_no_stores`. With both services stopped, copy that database file and any `-wal` and `-shm` files instead.
+4. Copy the other roots your recovery needs: the checkout's untracked `.kxm/` records, `$D/assets/`, `$W/worker-*.json` (and `$W/pi-sessions/` only if your policy keeps model history), and `$C`.
+5. Record the KXM version (`kxm --version`), the configuration commit, the schema versions from `manifest.json`, and every override variable, next to the copy.
+6. Start the hub, then the Runtime with `kxm runtime start`, and resume the paused restart policies.
+
+Keep at least one previous backup, and bound retention: run events and prompt sidecars grow with every run.
+
+## Restore with `kxm restore`
+
+`kxm restore` overwrites the live database and deletes its `-wal` and `-shm` files, and it does not check whether the hub is running. Stop the hub and the Runtime first, and move the current state aside rather than deleting it.
+
+Preview the restore. The dry run performs every check below and lists what it would overwrite:
+
+```bash
+cd /srv/kxm/product
+kxm restore /backups/kxm/2026-09-23/hub/manifest.json --dry-run
+```
+
+Expected output:
+
+```text
+dry run: restore 1 store(s) from /backups/kxm/2026-09-23/hub/manifest.json; digests verified against the manifest
+  would write /srv/kxm/product/.kxm/state/kxm.db
+  would delete /srv/kxm/product/.kxm/state/kxm.db-wal
+  would delete /srv/kxm/product/.kxm/state/kxm.db-shm
+```
+
+Then run it without `--dry-run`:
+
+```bash
+kxm restore /backups/kxm/2026-09-23/hub/manifest.json
+```
+
+Expected output:
+
+```text
+Restored 1 SQLite store(s) from /backups/kxm/2026-09-23/hub/manifest.json:
+  - hub-store: -> /srv/kxm/product/.kxm/state/kxm.db (schema v5, integrity ok)
+```
+
+Before it overwrites anything, `kxm restore` checks that the manifest is a `kxm.backup-manifest.v1` document, that every listed file exists, that each file's SHA-256 matches the manifest, and that no store is newer than this build supports. The manifest's own `manifestSha256` is not checked.
+
+It then checks each backup's integrity and schema version again, copies it into place with mode `0600`, and checks the result. Each store returns to its recorded path, rebased onto the current directory when the manifest came from another checkout.
+
+### Restore ceilings
+
+A backup newer than this build is refused with `runtime_schema_newer` before any file changes. KXM has no migrations: an older backup restores, but its store then refuses to open with `runtime_schema_outdated`. Restore an older backup with the release that wrote it; see [Upgrade KXM](upgrade.md#understand-schema-changes).
+
+The ceilings come from `KXM_BACKUP_CEILINGS` in `plugins/kxm/src/database.ts` and match each store's own schema version:
+
+| Store id | Highest schema version restored |
+|---|---|
+| `hub-store` | 5 |
+| `registry` | 1 |
+| `events:<key>` | 7 |
+| `binding-store` | 1 (no current store uses it) |
+
+### Restore the Runtime stores by hand
+
+1. Stop the Runtime and the hub, as in the backup procedure.
+2. Move `$S/runtime/registry.db` and `$S/runtime/projects/` aside.
+3. Extract the archive into the user state root:
+
+   ```bash
+   S="${KXM_STATE_HOME:-$HOME/.local/state/kxm}"
+   tar -C "$S" -xzf /backups/kxm/2026-09-23/user-state.tgz
+   ```
+
+   The archive also brings back the hub binding, `update.yaml` and repository bindings.
+4. Keep the checkout at the same absolute path. The Runtime derives each project's store key from the canonical checkout path, so a checkout restored elsewhere does not find its runs.
+
+## Verify the restore
+
+1. Start the hub and check `/ready` and `kxm hub view`, including that the reported `loopback` or `remote` scope matches the environment.
+2. Run `kxm runtime start`, then `kxm runtime status`, and wait for the sync state to return to `ok`.
+3. Run `kxm runs list`, open one run with `kxm runs status <run-id>`, and confirm its prompt sidecar came back. A run store without its sidecar is a partial restore.
+4. Send one test request between two agents.
+
+Test a full restore on a spare machine before you rely on it, and repeat the test periodically.
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `backup_no_stores` | No `.kxm/state/kxm.db` under the current directory, or it was moved with `KXM_STATE_DIR` or `KXM_DATA_PATH` | Run from the checkout root; copy a relocated database with the stopped-state procedure |
+| Runs are missing after a restore | `kxm backup` never contained the Runtime stores | Restore `$S/runtime/` from the stopped-state archive |
+| `restore_manifest_digest_mismatch` | A backup file changed after the manifest was written | Use another backup; do not edit files in a backup set |
+| `restore_file_missing` | A file listed in the manifest is not beside it | Copy the whole backup directory, not only `manifest.json` |
+| `runtime_schema_mismatch` | A backup file's schema version differs from the one its manifest records | Use another backup set; never mix files between sets |
+| `runtime_schema_newer` | The backup came from a newer release | Upgrade KXM first, then restore |
+| The hub refuses to start with `runtime_schema_outdated` | The restored store is older than this build | Run the release that wrote it, or start fresh |
+
+## Next steps
+
+- Move to a new release safely: [Upgrade KXM](upgrade.md)
+- What each store contains and how sensitive it is: [Data and storage](../concepts/data-and-storage.md)
+- Watch the restored service: [Monitor KXM](monitoring.md)
+- Exact flags and JSON fields: [CLI reference](../reference/cli-reference.md#kxm-backup)