@quantumclaw/quantumclaw 2026.3.22

package/docs/concepts/architecture.md

@@ -0,0 +1,137 @@
+---
+summary: "WebSocket gateway architecture, components, and client flows"
+read_when:
+  - Working on gateway protocol, clients, or transports
+title: "Gateway Architecture"
+---
+
+# Gateway architecture
+
+## Overview
+
+- A single long‑lived **Gateway** owns all messaging surfaces (WhatsApp via
+  Baileys, Telegram via grammY, Slack, Discord, Signal, iMessage, WebChat).
+- Control-plane clients (macOS app, CLI, web UI, automations) connect to the
+  Gateway over **WebSocket** on the configured bind host (default
+  `127.0.0.1:18789`).
+- **Nodes** (macOS/iOS/Android/headless) also connect over **WebSocket**, but
+  declare `role: node` with explicit caps/commands.
+- One Gateway per host; it is the only place that opens a WhatsApp session.
+- The **canvas host** is served by the Gateway HTTP server under:
+  - `/__quantumclaw__/canvas/` (agent-editable HTML/CSS/JS)
+  - `/__quantumclaw__/a2ui/` (A2UI host)
+    It uses the same port as the Gateway (default `18789`).
+
+## Components and flows
+
+### Gateway (daemon)
+
+- Maintains provider connections.
+- Exposes a typed WS API (requests, responses, server‑push events).
+- Validates inbound frames against JSON Schema.
+- Emits events like `agent`, `chat`, `presence`, `health`, `heartbeat`, `cron`.
+
+### Clients (mac app / CLI / web admin)
+
+- One WS connection per client.
+- Send requests (`health`, `status`, `send`, `agent`, `system-presence`).
+- Subscribe to events (`tick`, `agent`, `presence`, `shutdown`).
+
+### Nodes (macOS / iOS / Android / headless)
+
+- Connect to the **same WS server** with `role: node`.
+- Provide a device identity in `connect`; pairing is **device‑based** (role `node`) and
+  approval lives in the device pairing store.
+- Expose commands like `canvas.*`, `camera.*`, `screen.record`, `location.get`.
+
+Protocol details:
+
+- [Gateway protocol](/gateway/protocol)
+
+### WebChat
+
+- Static UI that uses the Gateway WS API for chat history and sends.
+- In remote setups, connects through the same SSH/Tailscale tunnel as other
+  clients.
+
+## Connection lifecycle (single client)
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Gateway
+
+    Client->>Gateway: req:connect
+    Gateway-->>Client: res (ok)
+    Note right of Gateway: or res error + close
+    Note left of Client: payload=hello-ok<br>snapshot: presence + health
+
+    Gateway-->>Client: event:presence
+    Gateway-->>Client: event:tick
+
+    Client->>Gateway: req:agent
+    Gateway-->>Client: res:agent<br>ack {runId, status:"accepted"}
+    Gateway-->>Client: event:agent<br>(streaming)
+    Gateway-->>Client: res:agent<br>final {runId, status, summary}
+```
+
+## Wire protocol (summary)
+
+- Transport: WebSocket, text frames with JSON payloads.
+- First frame **must** be `connect`.
+- After handshake:
+  - Requests: `{type:"req", id, method, params}` → `{type:"res", id, ok, payload|error}`
+  - Events: `{type:"event", event, payload, seq?, stateVersion?}`
+- If `QUANTUMCLAW_GATEWAY_TOKEN` (or `--token`) is set, `connect.params.auth.token`
+  must match or the socket closes.
+- Idempotency keys are required for side‑effecting methods (`send`, `agent`) to
+  safely retry; the server keeps a short‑lived dedupe cache.
+- Nodes must include `role: "node"` plus caps/commands/permissions in `connect`.
+
+## Pairing + local trust
+
+- All WS clients (operators + nodes) include a **device identity** on `connect`.
+- New device IDs require pairing approval; the Gateway issues a **device token**
+  for subsequent connects.
+- **Local** connects (loopback or the gateway host’s own tailnet address) can be
+  auto‑approved to keep same‑host UX smooth.
+- All connects must sign the `connect.challenge` nonce.
+- Signature payload `v3` also binds `platform` + `deviceFamily`; the gateway
+  pins paired metadata on reconnect and requires repair pairing for metadata
+  changes.
+- **Non‑local** connects still require explicit approval.
+- Gateway auth (`gateway.auth.*`) still applies to **all** connections, local or
+  remote.
+
+Details: [Gateway protocol](/gateway/protocol), [Pairing](/channels/pairing),
+[Security](/gateway/security).
+
+## Protocol typing and codegen
+
+- TypeBox schemas define the protocol.
+- JSON Schema is generated from those schemas.
+- Swift models are generated from the JSON Schema.
+
+## Remote access
+
+- Preferred: Tailscale or VPN.
+- Alternative: SSH tunnel
+
+  ```bash
+  ssh -N -L 18789:127.0.0.1:18789 user@host
+  ```
+
+- The same handshake + auth token apply over the tunnel.
+- TLS + optional pinning can be enabled for WS in remote setups.
+
+## Operations snapshot
+
+- Start: `quantumclaw gateway` (foreground, logs to stdout).
+- Health: `health` over WS (also included in `hello-ok`).
+- Supervision: launchd/systemd for auto‑restart.
+
+## Invariants
+
+- Exactly one Gateway controls a single Baileys session per host.
+- Handshake is mandatory; any non‑JSON or non‑connect first frame is a hard close.
+- Events are not replayed; clients must refresh on gaps.

package/docs/concepts/compaction.md

@@ -0,0 +1,123 @@
+---
+summary: "Context window + compaction: how QuantumClaw keeps sessions under model limits"
+read_when:
+  - You want to understand auto-compaction and /compact
+  - You are debugging long sessions hitting context limits
+title: "Compaction"
+---
+
+# Context Window & Compaction
+
+Every model has a **context window** (max tokens it can see). Long-running chats accumulate messages and tool results; once the window is tight, QuantumClaw **compacts** older history to stay within limits.
+
+## What compaction is
+
+Compaction **summarizes older conversation** into a compact summary entry and keeps recent messages intact. The summary is stored in the session history, so future requests use:
+
+- The compaction summary
+- Recent messages after the compaction point
+
+Compaction **persists** in the session’s JSONL history.
+
+## Configuration
+
+Use the `agents.defaults.compaction` setting in your `quantumclaw.json` to configure compaction behavior (mode, target tokens, etc.).
+Compaction summarization preserves opaque identifiers by default (`identifierPolicy: "strict"`). You can override this with `identifierPolicy: "off"` or provide custom text with `identifierPolicy: "custom"` and `identifierInstructions`.
+
+You can optionally specify a different model for compaction summarization via `agents.defaults.compaction.model`. This is useful when your primary model is a local or small model and you want compaction summaries produced by a more capable model. The override accepts any `provider/model-id` string:
+
+```json
+{
+  "agents": {
+    "defaults": {
+      "compaction": {
+        "model": "openrouter/anthropic/claude-sonnet-4-6"
+      }
+    }
+  }
+}
+```
+
+This also works with local models, for example a second Ollama model dedicated to summarization or a fine-tuned compaction specialist:
+
+```json
+{
+  "agents": {
+    "defaults": {
+      "compaction": {
+        "model": "ollama/llama3.1:8b"
+      }
+    }
+  }
+}
+```
+
+When unset, compaction uses the agent's primary model.
+
+## Auto-compaction (default on)
+
+When a session nears or exceeds the model’s context window, QuantumClaw triggers auto-compaction and may retry the original request using the compacted context.
+
+You’ll see:
+
+- `🧹 Auto-compaction complete` in verbose mode
+- `/status` showing `🧹 Compactions: <count>`
+
+Before compaction, QuantumClaw can run a **silent memory flush** turn to store
+durable notes to disk. See [Memory](/concepts/memory) for details and config.
+
+## Manual compaction
+
+Use `/compact` (optionally with instructions) to force a compaction pass:
+
+```
+/compact Focus on decisions and open questions
+```
+
+## Context window source
+
+Context window is model-specific. QuantumClaw uses the model definition from the configured provider catalog to determine limits.
+
+## Compaction vs pruning
+
+- **Compaction**: summarises and **persists** in JSONL.
+- **Session pruning**: trims old **tool results** only, **in-memory**, per request.
+
+See [/concepts/session-pruning](/concepts/session-pruning) for pruning details.
+
+## OpenAI server-side compaction
+
+QuantumClaw also supports OpenAI Responses server-side compaction hints for
+compatible direct OpenAI models. This is separate from local QuantumClaw
+compaction and can run alongside it.
+
+- Local compaction: QuantumClaw summarizes and persists into session JSONL.
+- Server-side compaction: OpenAI compacts context on the provider side when
+  `store` + `context_management` are enabled.
+
+See [OpenAI provider](/providers/openai) for model params and overrides.
+
+## Custom context engines
+
+Compaction behavior is owned by the active
+[context engine](/concepts/context-engine). The legacy engine uses the built-in
+summarization described above. Plugin engines (selected via
+`plugins.slots.contextEngine`) can implement any compaction strategy — DAG
+summaries, vector retrieval, incremental condensation, etc.
+
+When a plugin engine sets `ownsCompaction: true`, QuantumClaw delegates all
+compaction decisions to the engine and does not run built-in auto-compaction.
+
+When `ownsCompaction` is `false` or unset, QuantumClaw may still use Pi's
+built-in in-attempt auto-compaction, but the active engine's `compact()` method
+still handles `/compact` and overflow recovery. There is no automatic fallback
+to the legacy engine's compaction path.
+
+If you are building a non-owning context engine, implement `compact()` by
+calling `delegateCompactionToRuntime(...)` from `quantumclaw/plugin-sdk/core`.
+
+## Tips
+
+- Use `/compact` when sessions feel stale or context is bloated.
+- Large tool outputs are already truncated; pruning can further reduce tool-result buildup.
+- If you need a fresh slate, `/new` or `/reset` starts a new session id.

package/docs/concepts/context-engine.md

@@ -0,0 +1,268 @@
+---
+summary: "Context engine: pluggable context assembly, compaction, and subagent lifecycle"
+read_when:
+  - You want to understand how QuantumClaw assembles model context
+  - You are switching between the legacy engine and a plugin engine
+  - You are building a context engine plugin
+title: "Context Engine"
+---
+
+# Context Engine
+
+A **context engine** controls how QuantumClaw builds model context for each run.
+It decides which messages to include, how to summarize older history, and how
+to manage context across subagent boundaries.
+
+QuantumClaw ships with a built-in `legacy` engine. Plugins can register
+alternative engines that replace the active context-engine lifecycle.
+
+## Quick start
+
+Check which engine is active:
+
+```bash
+quantumclaw doctor
+# or inspect config directly:
+cat ~/.quantumclaw/quantumclaw.json | jq '.plugins.slots.contextEngine'
+```
+
+### Installing a context engine plugin
+
+Context engine plugins are installed like any other QuantumClaw plugin. Install
+first, then select the engine in the slot:
+
+```bash
+# Install from npm
+quantumclaw plugins install @martian-engineering/lossless-claw
+
+# Or install from a local path (for development)
+quantumclaw plugins install -l ./my-context-engine
+```
+
+Then enable the plugin and select it as the active engine in your config:
+
+```json5
+// quantumclaw.json
+{
+  plugins: {
+    slots: {
+      contextEngine: "lossless-claw", // must match the plugin's registered engine id
+    },
+    entries: {
+      "lossless-claw": {
+        enabled: true,
+        // Plugin-specific config goes here (see the plugin's docs)
+      },
+    },
+  },
+}
+```
+
+Restart the gateway after installing and configuring.
+
+To switch back to the built-in engine, set `contextEngine` to `"legacy"` (or
+remove the key entirely — `"legacy"` is the default).
+
+## How it works
+
+Every time QuantumClaw runs a model prompt, the context engine participates at
+four lifecycle points:
+
+1. **Ingest** — called when a new message is added to the session. The engine
+   can store or index the message in its own data store.
+2. **Assemble** — called before each model run. The engine returns an ordered
+   set of messages (and an optional `systemPromptAddition`) that fit within
+   the token budget.
+3. **Compact** — called when the context window is full, or when the user runs
+   `/compact`. The engine summarizes older history to free space.
+4. **After turn** — called after a run completes. The engine can persist state,
+   trigger background compaction, or update indexes.
+
+### Subagent lifecycle (optional)
+
+QuantumClaw currently calls one subagent lifecycle hook:
+
+- **onSubagentEnded** — clean up when a subagent session completes or is swept.
+
+The `prepareSubagentSpawn` hook is part of the interface for future use, but
+the runtime does not invoke it yet.
+
+### System prompt addition
+
+The `assemble` method can return a `systemPromptAddition` string. QuantumClaw
+prepends this to the system prompt for the run. This lets engines inject
+dynamic recall guidance, retrieval instructions, or context-aware hints
+without requiring static workspace files.
+
+## The legacy engine
+
+The built-in `legacy` engine preserves QuantumClaw's original behavior:
+
+- **Ingest**: no-op (the session manager handles message persistence directly).
+- **Assemble**: pass-through (the existing sanitize → validate → limit pipeline
+  in the runtime handles context assembly).
+- **Compact**: delegates to the built-in summarization compaction, which creates
+  a single summary of older messages and keeps recent messages intact.
+- **After turn**: no-op.
+
+The legacy engine does not register tools or provide a `systemPromptAddition`.
+
+When no `plugins.slots.contextEngine` is set (or it's set to `"legacy"`), this
+engine is used automatically.
+
+## Plugin engines
+
+A plugin can register a context engine using the plugin API:
+
+```ts
+export default function register(api) {
+  api.registerContextEngine("my-engine", () => ({
+    info: {
+      id: "my-engine",
+      name: "My Context Engine",
+      ownsCompaction: true,
+    },
+
+    async ingest({ sessionId, message, isHeartbeat }) {
+      // Store the message in your data store
+      return { ingested: true };
+    },
+
+    async assemble({ sessionId, messages, tokenBudget }) {
+      // Return messages that fit the budget
+      return {
+        messages: buildContext(messages, tokenBudget),
+        estimatedTokens: countTokens(messages),
+        systemPromptAddition: "Use lcm_grep to search history...",
+      };
+    },
+
+    async compact({ sessionId, force }) {
+      // Summarize older context
+      return { ok: true, compacted: true };
+    },
+  }));
+}
+```
+
+Then enable it in config:
+
+```json5
+{
+  plugins: {
+    slots: {
+      contextEngine: "my-engine",
+    },
+    entries: {
+      "my-engine": {
+        enabled: true,
+      },
+    },
+  },
+}
+```
+
+### The ContextEngine interface
+
+Required members:
+
+| Member             | Kind     | Purpose                                                  |
+| ------------------ | -------- | -------------------------------------------------------- |
+| `info`             | Property | Engine id, name, version, and whether it owns compaction |
+| `ingest(params)`   | Method   | Store a single message                                   |
+| `assemble(params)` | Method   | Build context for a model run (returns `AssembleResult`) |
+| `compact(params)`  | Method   | Summarize/reduce context                                 |
+
+`assemble` returns an `AssembleResult` with:
+
+- `messages` — the ordered messages to send to the model.
+- `estimatedTokens` (required, `number`) — the engine's estimate of total
+  tokens in the assembled context. QuantumClaw uses this for compaction threshold
+  decisions and diagnostic reporting.
+- `systemPromptAddition` (optional, `string`) — prepended to the system prompt.
+
+Optional members:
+
+| Member                         | Kind   | Purpose                                                                                                         |
+| ------------------------------ | ------ | --------------------------------------------------------------------------------------------------------------- |
+| `bootstrap(params)`            | Method | Initialize engine state for a session. Called once when the engine first sees a session (e.g., import history). |
+| `ingestBatch(params)`          | Method | Ingest a completed turn as a batch. Called after a run completes, with all messages from that turn at once.     |
+| `afterTurn(params)`            | Method | Post-run lifecycle work (persist state, trigger background compaction).                                         |
+| `prepareSubagentSpawn(params)` | Method | Set up shared state for a child session.                                                                        |
+| `onSubagentEnded(params)`      | Method | Clean up after a subagent ends.                                                                                 |
+| `dispose()`                    | Method | Release resources. Called during gateway shutdown or plugin reload — not per-session.                           |
+
+### ownsCompaction
+
+`ownsCompaction` controls whether Pi's built-in in-attempt auto-compaction stays
+enabled for the run:
+
+- `true` — the engine owns compaction behavior. QuantumClaw disables Pi's built-in
+  auto-compaction for that run, and the engine's `compact()` implementation is
+  responsible for `/compact`, overflow recovery compaction, and any proactive
+  compaction it wants to do in `afterTurn()`.
+- `false` or unset — Pi's built-in auto-compaction may still run during prompt
+  execution, but the active engine's `compact()` method is still called for
+  `/compact` and overflow recovery.
+
+`ownsCompaction: false` does **not** mean QuantumClaw automatically falls back to
+the legacy engine's compaction path.
+
+That means there are two valid plugin patterns:
+
+- **Owning mode** — implement your own compaction algorithm and set
+  `ownsCompaction: true`.
+- **Delegating mode** — set `ownsCompaction: false` and have `compact()` call
+  `delegateCompactionToRuntime(...)` from `quantumclaw/plugin-sdk/core` to use
+  QuantumClaw's built-in compaction behavior.
+
+A no-op `compact()` is unsafe for an active non-owning engine because it
+disables the normal `/compact` and overflow-recovery compaction path for that
+engine slot.
+
+## Configuration reference
+
+```json5
+{
+  plugins: {
+    slots: {
+      // Select the active context engine. Default: "legacy".
+      // Set to a plugin id to use a plugin engine.
+      contextEngine: "legacy",
+    },
+  },
+}
+```
+
+The slot is exclusive at run time — only one registered context engine is
+resolved for a given run or compaction operation. Other enabled
+`kind: "context-engine"` plugins can still load and run their registration
+code; `plugins.slots.contextEngine` only selects which registered engine id
+QuantumClaw resolves when it needs a context engine.
+
+## Relationship to compaction and memory
+
+- **Compaction** is one responsibility of the context engine. The legacy engine
+  delegates to QuantumClaw's built-in summarization. Plugin engines can implement
+  any compaction strategy (DAG summaries, vector retrieval, etc.).
+- **Memory plugins** (`plugins.slots.memory`) are separate from context engines.
+  Memory plugins provide search/retrieval; context engines control what the
+  model sees. They can work together — a context engine might use memory
+  plugin data during assembly.
+- **Session pruning** (trimming old tool results in-memory) still runs
+  regardless of which context engine is active.
+
+## Tips
+
+- Use `quantumclaw doctor` to verify your engine is loading correctly.
+- If switching engines, existing sessions continue with their current history.
+  The new engine takes over for future runs.
+- Engine errors are logged and surfaced in diagnostics. If a plugin engine
+  fails to register or the selected engine id cannot be resolved, QuantumClaw
+  does not fall back automatically; runs fail until you fix the plugin or
+  switch `plugins.slots.contextEngine` back to `"legacy"`.
+- For development, use `quantumclaw plugins install -l ./my-engine` to link a
+  local plugin directory without copying.
+
+See also: [Compaction](/concepts/compaction), [Context](/concepts/context),
+[Plugins](/tools/plugin), [Plugin manifest](/plugins/manifest).