@quantumclaw/quantumclaw 2026.3.22

package/docs/network.md

@@ -0,0 +1,54 @@
+---
+summary: "Network hub: gateway surfaces, pairing, discovery, and security"
+read_when:
+  - You need the network architecture + security overview
+  - You are debugging local vs tailnet access or pairing
+  - You want the canonical list of networking docs
+title: "Network"
+---
+
+# Network hub
+
+This hub links the core docs for how QuantumClaw connects, pairs, and secures
+devices across localhost, LAN, and tailnet.
+
+## Core model
+
+- [Gateway architecture](/concepts/architecture)
+- [Gateway protocol](/gateway/protocol)
+- [Gateway runbook](/gateway)
+- [Web surfaces + bind modes](/web)
+
+## Pairing + identity
+
+- [Pairing overview (DM + nodes)](/channels/pairing)
+- [Gateway-owned node pairing](/gateway/pairing)
+- [Devices CLI (pairing + token rotation)](/cli/devices)
+- [Pairing CLI (DM approvals)](/cli/pairing)
+
+Local trust:
+
+- Local connections (loopback or the gateway host’s own tailnet address) can be
+  auto‑approved for pairing to keep same‑host UX smooth.
+- Non‑local tailnet/LAN clients still require explicit pairing approval.
+
+## Discovery + transports
+
+- [Discovery & transports](/gateway/discovery)
+- [Bonjour / mDNS](/gateway/bonjour)
+- [Remote access (SSH)](/gateway/remote)
+- [Tailscale](/gateway/tailscale)
+
+## Nodes + transports
+
+- [Nodes overview](/nodes)
+- [Bridge protocol (legacy nodes)](/gateway/bridge-protocol)
+- [Node runbook: iOS](/platforms/ios)
+- [Node runbook: Android](/platforms/android)
+
+## Security
+
+- [Security overview](/gateway/security)
+- [Gateway config reference](/gateway/configuration)
+- [Troubleshooting](/gateway/troubleshooting)
+- [Doctor](/gateway/doctor)

package/docs/nodes/audio.md

@@ -0,0 +1,187 @@
+---
+summary: "How inbound audio/voice notes are downloaded, transcribed, and injected into replies"
+read_when:
+  - Changing audio transcription or media handling
+title: "Audio and Voice Notes"
+---
+
+# Audio / Voice Notes (2026-01-17)
+
+## What works
+
+- **Media understanding (audio)**: If audio understanding is enabled (or auto‑detected), QuantumClaw:
+  1. Locates the first audio attachment (local path or URL) and downloads it if needed.
+  2. Enforces `maxBytes` before sending to each model entry.
+  3. Runs the first eligible model entry in order (provider or CLI).
+  4. If it fails or skips (size/timeout), it tries the next entry.
+  5. On success, it replaces `Body` with an `[Audio]` block and sets `{{Transcript}}`.
+- **Command parsing**: When transcription succeeds, `CommandBody`/`RawBody` are set to the transcript so slash commands still work.
+- **Verbose logging**: In `--verbose`, we log when transcription runs and when it replaces the body.
+
+## Auto-detection (default)
+
+If you **don’t configure models** and `tools.media.audio.enabled` is **not** set to `false`,
+QuantumClaw auto-detects in this order and stops at the first working option:
+
+1. **Local CLIs** (if installed)
+   - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
+   - `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
+   - `whisper` (Python CLI; downloads models automatically)
+2. **Gemini CLI** (`gemini`) using `read_many_files`
+3. **Provider keys** (OpenAI → Groq → Deepgram → Google)
+
+To disable auto-detection, set `tools.media.audio.enabled: false`.
+To customize, set `tools.media.audio.models`.
+Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
+
+## Config examples
+
+### Provider + CLI fallback (OpenAI + Whisper CLI)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        maxBytes: 20971520,
+        models: [
+          { provider: "openai", model: "gpt-4o-mini-transcribe" },
+          {
+            type: "cli",
+            command: "whisper",
+            args: ["--model", "base", "{{MediaPath}}"],
+            timeoutSeconds: 45,
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+### Provider-only with scope gating
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        scope: {
+          default: "allow",
+          rules: [{ action: "deny", match: { chatType: "group" } }],
+        },
+        models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
+      },
+    },
+  },
+}
+```
+
+### Provider-only (Deepgram)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        models: [{ provider: "deepgram", model: "nova-3" }],
+      },
+    },
+  },
+}
+```
+
+### Provider-only (Mistral Voxtral)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        models: [{ provider: "mistral", model: "voxtral-mini-latest" }],
+      },
+    },
+  },
+}
+```
+
+### Echo transcript to chat (opt-in)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        echoTranscript: true, // default is false
+        echoFormat: '📝 "{transcript}"', // optional, supports {transcript}
+        models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
+      },
+    },
+  },
+}
+```
+
+## Notes & limits
+
+- Provider auth follows the standard model auth order (auth profiles, env vars, `models.providers.*.apiKey`).
+- Deepgram picks up `DEEPGRAM_API_KEY` when `provider: "deepgram"` is used.
+- Deepgram setup details: [Deepgram (audio transcription)](/providers/deepgram).
+- Mistral setup details: [Mistral](/providers/mistral).
+- Audio providers can override `baseUrl`, `headers`, and `providerOptions` via `tools.media.audio`.
+- Default size cap is 20MB (`tools.media.audio.maxBytes`). Oversize audio is skipped for that model and the next entry is tried.
+- Tiny/empty audio files below 1024 bytes are skipped before provider/CLI transcription.
+- Default `maxChars` for audio is **unset** (full transcript). Set `tools.media.audio.maxChars` or per-entry `maxChars` to trim output.
+- OpenAI auto default is `gpt-4o-mini-transcribe`; set `model: "gpt-4o-transcribe"` for higher accuracy.
+- Use `tools.media.audio.attachments` to process multiple voice notes (`mode: "all"` + `maxAttachments`).
+- Transcript is available to templates as `{{Transcript}}`.
+- `tools.media.audio.echoTranscript` is off by default; enable it to send transcript confirmation back to the originating chat before agent processing.
+- `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`).
+- CLI stdout is capped (5MB); keep CLI output concise.
+
+### Proxy environment support
+
+Provider-based audio transcription honors standard outbound proxy env vars:
+
+- `HTTPS_PROXY`
+- `HTTP_PROXY`
+- `https_proxy`
+- `http_proxy`
+
+If no proxy env vars are set, direct egress is used. If proxy config is malformed, QuantumClaw logs a warning and falls back to direct fetch.
+
+## Mention Detection in Groups
+
+When `requireMention: true` is set for a group chat, QuantumClaw now transcribes audio **before** checking for mentions. This allows voice notes to be processed even when they contain mentions.
+
+**How it works:**
+
+1. If a voice message has no text body and the group requires mentions, QuantumClaw performs a "preflight" transcription.
+2. The transcript is checked for mention patterns (e.g., `@BotName`, emoji triggers).
+3. If a mention is found, the message proceeds through the full reply pipeline.
+4. The transcript is used for mention detection so voice notes can pass the mention gate.
+
+**Fallback behavior:**
+
+- If transcription fails during preflight (timeout, API error, etc.), the message is processed based on text-only mention detection.
+- This ensures that mixed messages (text + audio) are never incorrectly dropped.
+
+**Opt-out per Telegram group/topic:**
+
+- Set `channels.telegram.groups.<chatId>.disableAudioPreflight: true` to skip preflight transcript mention checks for that group.
+- Set `channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight` to override per-topic (`true` to skip, `false` to force-enable).
+- Default is `false` (preflight enabled when mention-gated conditions match).
+
+**Example:** A user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with `requireMention: true`. The voice note is transcribed, the mention is detected, and the agent replies.
+
+## Gotchas
+
+- Scope rules use first-match wins. `chatType` is normalized to `direct`, `group`, or `room`.
+- Ensure your CLI exits 0 and prints plain text; JSON needs to be massaged via `jq -r .text`.
+- For `parakeet-mlx`, if you pass `--output-dir`, QuantumClaw reads `<output-dir>/<media-basename>.txt` when `--output-format` is `txt` (or omitted); non-`txt` output formats fall back to stdout parsing.
+- Keep timeouts reasonable (`timeoutSeconds`, default 60s) to avoid blocking the reply queue.
+- Preflight transcription only processes the **first** audio attachment for mention detection. Additional audio is processed during the main media understanding phase.

package/docs/nodes/camera.md

@@ -0,0 +1,162 @@
+---
+summary: "Camera capture (iOS/Android nodes + macOS app) for agent use: photos (jpg) and short video clips (mp4)"
+read_when:
+  - Adding or modifying camera capture on iOS/Android nodes or macOS
+  - Extending agent-accessible MEDIA temp-file workflows
+title: "Camera Capture"
+---
+
+# Camera capture (agent)
+
+QuantumClaw supports **camera capture** for agent workflows:
+
+- **iOS node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+- **Android node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+- **macOS app** (node via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+
+All camera access is gated behind **user-controlled settings**.
+
+## iOS node
+
+### User setting (default on)
+
+- iOS Settings tab → **Camera** → **Allow Camera** (`camera.enabled`)
+  - Default: **on** (missing key is treated as enabled).
+  - When off: `camera.*` commands return `CAMERA_DISABLED`.
+
+### Commands (via Gateway `node.invoke`)
+
+- `camera.list`
+  - Response payload:
+    - `devices`: array of `{ id, name, position, deviceType }`
+
+- `camera.snap`
+  - Params:
+    - `facing`: `front|back` (default: `front`)
+    - `maxWidth`: number (optional; default `1600` on the iOS node)
+    - `quality`: `0..1` (optional; default `0.9`)
+    - `format`: currently `jpg`
+    - `delayMs`: number (optional; default `0`)
+    - `deviceId`: string (optional; from `camera.list`)
+  - Response payload:
+    - `format: "jpg"`
+    - `base64: "<...>"`
+    - `width`, `height`
+  - Payload guard: photos are recompressed to keep the base64 payload under 5 MB.
+
+- `camera.clip`
+  - Params:
+    - `facing`: `front|back` (default: `front`)
+    - `durationMs`: number (default `3000`, clamped to a max of `60000`)
+    - `includeAudio`: boolean (default `true`)
+    - `format`: currently `mp4`
+    - `deviceId`: string (optional; from `camera.list`)
+  - Response payload:
+    - `format: "mp4"`
+    - `base64: "<...>"`
+    - `durationMs`
+    - `hasAudio`
+
+### Foreground requirement
+
+Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
+
+### CLI helper (temp files + MEDIA)
+
+The easiest way to get attachments is via the CLI helper, which writes decoded media to a temp file and prints `MEDIA:<path>`.
+
+Examples:
+
+```bash
+quantumclaw nodes camera snap --node <id>               # default: both front + back (2 MEDIA lines)
+quantumclaw nodes camera snap --node <id> --facing front
+quantumclaw nodes camera clip --node <id> --duration 3000
+quantumclaw nodes camera clip --node <id> --no-audio
+```
+
+Notes:
+
+- `nodes camera snap` defaults to **both** facings to give the agent both views.
+- Output files are temporary (in the OS temp directory) unless you build your own wrapper.
+
+## Android node
+
+### Android user setting (default on)
+
+- Android Settings sheet → **Camera** → **Allow Camera** (`camera.enabled`)
+  - Default: **on** (missing key is treated as enabled).
+  - When off: `camera.*` commands return `CAMERA_DISABLED`.
+
+### Permissions
+
+- Android requires runtime permissions:
+  - `CAMERA` for both `camera.snap` and `camera.clip`.
+  - `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`.
+
+If permissions are missing, the app will prompt when possible; if denied, `camera.*` requests fail with a
+`*_PERMISSION_REQUIRED` error.
+
+### Android foreground requirement
+
+Like `canvas.*`, the Android node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
+
+### Android commands (via Gateway `node.invoke`)
+
+- `camera.list`
+  - Response payload:
+    - `devices`: array of `{ id, name, position, deviceType }`
+
+### Payload guard
+
+Photos are recompressed to keep the base64 payload under 5 MB.
+
+## macOS app
+
+### User setting (default off)
+
+The macOS companion app exposes a checkbox:
+
+- **Settings → General → Allow Camera** (`quantumclaw.cameraEnabled`)
+  - Default: **off**
+  - When off: camera requests return “Camera disabled by user”.
+
+### CLI helper (node invoke)
+
+Use the main `quantumclaw` CLI to invoke camera commands on the macOS node.
+
+Examples:
+
+```bash
+quantumclaw nodes camera list --node <id>            # list camera ids
+quantumclaw nodes camera snap --node <id>            # prints MEDIA:<path>
+quantumclaw nodes camera snap --node <id> --max-width 1280
+quantumclaw nodes camera snap --node <id> --delay-ms 2000
+quantumclaw nodes camera snap --node <id> --device-id <id>
+quantumclaw nodes camera clip --node <id> --duration 10s          # prints MEDIA:<path>
+quantumclaw nodes camera clip --node <id> --duration-ms 3000      # prints MEDIA:<path> (legacy flag)
+quantumclaw nodes camera clip --node <id> --device-id <id>
+quantumclaw nodes camera clip --node <id> --no-audio
+```
+
+Notes:
+
+- `quantumclaw nodes camera snap` defaults to `maxWidth=1600` unless overridden.
+- On macOS, `camera.snap` waits `delayMs` (default 2000ms) after warm-up/exposure settle before capturing.
+- Photo payloads are recompressed to keep base64 under 5 MB.
+
+## Safety + practical limits
+
+- Camera and microphone access trigger the usual OS permission prompts (and require usage strings in Info.plist).
+- Video clips are capped (currently `<= 60s`) to avoid oversized node payloads (base64 overhead + message limits).
+
+## macOS screen video (OS-level)
+
+For _screen_ video (not camera), use the macOS companion:
+
+```bash
+quantumclaw nodes screen record --node <id> --duration 10s --fps 15   # prints MEDIA:<path>
+```
+
+Notes:
+
+- Requires macOS **Screen Recording** permission (TCC).

package/docs/nodes/images.md

@@ -0,0 +1,72 @@
+---
+summary: "Image and media handling rules for send, gateway, and agent replies"
+read_when:
+  - Modifying media pipeline or attachments
+title: "Image and Media Support"
+---
+
+# Image & Media Support (2025-12-05)
+
+The WhatsApp channel runs via **Baileys Web**. This document captures the current media handling rules for send, gateway, and agent replies.
+
+## Goals
+
+- Send media with optional captions via `quantumclaw message send --media`.
+- Allow auto-replies from the web inbox to include media alongside text.
+- Keep per-type limits sane and predictable.
+
+## CLI Surface
+
+- `quantumclaw message send --media <path-or-url> [--message <caption>]`
+  - `--media` optional; caption can be empty for media-only sends.
+  - `--dry-run` prints the resolved payload; `--json` emits `{ channel, to, messageId, mediaUrl, caption }`.
+
+## WhatsApp Web channel behavior
+
+- Input: local file path **or** HTTP(S) URL.
+- Flow: load into a Buffer, detect media kind, and build the correct payload:
+  - **Images:** resize & recompress to JPEG (max side 2048px) targeting `agents.defaults.mediaMaxMb` (default 5 MB), capped at 6 MB.
+  - **Audio/Voice/Video:** pass-through up to 16 MB; audio is sent as a voice note (`ptt: true`).
+  - **Documents:** anything else, up to 100 MB, with filename preserved when available.
+- WhatsApp GIF-style playback: send an MP4 with `gifPlayback: true` (CLI: `--gif-playback`) so mobile clients loop inline.
+- MIME detection prefers magic bytes, then headers, then file extension.
+- Caption comes from `--message` or `reply.text`; empty caption is allowed.
+- Logging: non-verbose shows `↩️`/`✅`; verbose includes size and source path/URL.
+
+## Auto-Reply Pipeline
+
+- `getReplyFromConfig` returns `{ text?, mediaUrl?, mediaUrls? }`.
+- When media is present, the web sender resolves local paths or URLs using the same pipeline as `quantumclaw message send`.
+- Multiple media entries are sent sequentially if provided.
+
+## Inbound Media to Commands (Pi)
+
+- When inbound web messages include media, QuantumClaw downloads to a temp file and exposes templating variables:
+  - `{{MediaUrl}}` pseudo-URL for the inbound media.
+  - `{{MediaPath}}` local temp path written before running the command.
+- When a per-session Docker sandbox is enabled, inbound media is copied into the sandbox workspace and `MediaPath`/`MediaUrl` are rewritten to a relative path like `media/inbound/<filename>`.
+- Media understanding (if configured via `tools.media.*` or shared `tools.media.models`) runs before templating and can insert `[Image]`, `[Audio]`, and `[Video]` blocks into `Body`.
+  - Audio sets `{{Transcript}}` and uses the transcript for command parsing so slash commands still work.
+  - Video and image descriptions preserve any caption text for command parsing.
+- By default only the first matching image/audio/video attachment is processed; set `tools.media.<cap>.attachments` to process multiple attachments.
+
+## Limits & Errors
+
+**Outbound send caps (WhatsApp web send)**
+
+- Images: ~6 MB cap after recompression.
+- Audio/voice/video: 16 MB cap; documents: 100 MB cap.
+- Oversize or unreadable media → clear error in logs and the reply is skipped.
+
+**Media understanding caps (transcription/description)**
+
+- Image default: 10 MB (`tools.media.image.maxBytes`).
+- Audio default: 20 MB (`tools.media.audio.maxBytes`).
+- Video default: 50 MB (`tools.media.video.maxBytes`).
+- Oversize media skips understanding, but replies still go through with the original body.
+
+## Notes for Tests
+
+- Cover send + reply flows for image/audio/document cases.
+- Validate recompression for images (size bound) and voice-note flag for audio.
+- Ensure multi-media replies fan out as sequential sends.