@termix-it/cryptoclaw 1.0.0

package/docs/nodes/media-understanding.md

@@ -0,0 +1,379 @@
+---
+summary: "Inbound image/audio/video understanding (optional) with provider + CLI fallbacks"
+read_when:
+  - Designing or refactoring media understanding
+  - Tuning inbound audio/video/image preprocessing
+title: "Media Understanding"
+---
+
+# Media Understanding (Inbound) — 2026-01-17
+
+OpenClaw can **summarize inbound media** (image/audio/video) before the reply pipeline runs. It auto‑detects when local tools or provider keys are available, and can be disabled or customized. If understanding is off, models still receive the original files/URLs as usual.
+
+## Goals
+
+- Optional: pre‑digest inbound media into short text for faster routing + better command parsing.
+- Preserve original media delivery to the model (always).
+- Support **provider APIs** and **CLI fallbacks**.
+- Allow multiple models with ordered fallback (error/size/timeout).
+
+## High‑level behavior
+
+1. Collect inbound attachments (`MediaPaths`, `MediaUrls`, `MediaTypes`).
+2. For each enabled capability (image/audio/video), select attachments per policy (default: **first**).
+3. Choose the first eligible model entry (size + capability + auth).
+4. If a model fails or the media is too large, **fall back to the next entry**.
+5. On success:
+   - `Body` becomes `[Image]`, `[Audio]`, or `[Video]` block.
+   - Audio sets `{{Transcript}}`; command parsing uses caption text when present,
+     otherwise the transcript.
+   - Captions are preserved as `User text:` inside the block.
+
+If understanding fails or is disabled, **the reply flow continues** with the original body + attachments.
+
+## Config overview
+
+`tools.media` supports **shared models** plus per‑capability overrides:
+
+- `tools.media.models`: shared model list (use `capabilities` to gate).
+- `tools.media.image` / `tools.media.audio` / `tools.media.video`:
+  - defaults (`prompt`, `maxChars`, `maxBytes`, `timeoutSeconds`, `language`)
+  - provider overrides (`baseUrl`, `headers`, `providerOptions`)
+  - Deepgram audio options via `tools.media.audio.providerOptions.deepgram`
+  - optional **per‑capability `models` list** (preferred before shared models)
+  - `attachments` policy (`mode`, `maxAttachments`, `prefer`)
+  - `scope` (optional gating by channel/chatType/session key)
+- `tools.media.concurrency`: max concurrent capability runs (default **2**).
+
+```json5
+{
+  tools: {
+    media: {
+      models: [
+        /* shared list */
+      ],
+      image: {
+        /* optional overrides */
+      },
+      audio: {
+        /* optional overrides */
+      },
+      video: {
+        /* optional overrides */
+      },
+    },
+  },
+}
+```
+
+### Model entries
+
+Each `models[]` entry can be **provider** or **CLI**:
+
+```json5
+{
+  type: "provider", // default if omitted
+  provider: "openai",
+  model: "gpt-5.2",
+  prompt: "Describe the image in <= 500 chars.",
+  maxChars: 500,
+  maxBytes: 10485760,
+  timeoutSeconds: 60,
+  capabilities: ["image"], // optional, used for multi‑modal entries
+  profile: "vision-profile",
+  preferredProfile: "vision-fallback",
+}
+```
+
+```json5
+{
+  type: "cli",
+  command: "gemini",
+  args: [
+    "-m",
+    "gemini-3-flash",
+    "--allowed-tools",
+    "read_file",
+    "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+  ],
+  maxChars: 500,
+  maxBytes: 52428800,
+  timeoutSeconds: 120,
+  capabilities: ["video", "image"],
+}
+```
+
+CLI templates can also use:
+
+- `{{MediaDir}}` (directory containing the media file)
+- `{{OutputDir}}` (scratch dir created for this run)
+- `{{OutputBase}}` (scratch file base path, no extension)
+
+## Defaults and limits
+
+Recommended defaults:
+
+- `maxChars`: **500** for image/video (short, command‑friendly)
+- `maxChars`: **unset** for audio (full transcript unless you set a limit)
+- `maxBytes`:
+  - image: **10MB**
+  - audio: **20MB**
+  - video: **50MB**
+
+Rules:
+
+- If media exceeds `maxBytes`, that model is skipped and the **next model is tried**.
+- If the model returns more than `maxChars`, output is trimmed.
+- `prompt` defaults to simple “Describe the {media}.” plus the `maxChars` guidance (image/video only).
+- If `<capability>.enabled: true` but no models are configured, OpenClaw tries the
+  **active reply model** when its provider supports the capability.
+
+### Auto-detect media understanding (default)
+
+If `tools.media.<capability>.enabled` is **not** set to `false` and you haven’t
+configured models, OpenClaw auto-detects in this order and **stops at the first
+working option**:
+
+1. **Local CLIs** (audio only; if installed)
+   - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
+   - `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
+   - `whisper` (Python CLI; downloads models automatically)
+2. **Gemini CLI** (`gemini`) using `read_many_files`
+3. **Provider keys**
+   - Audio: OpenAI → Groq → Deepgram → Google
+   - Image: OpenAI → Anthropic → Google → MiniMax
+   - Video: Google
+
+To disable auto-detection, set:
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: false,
+      },
+    },
+  },
+}
+```
+
+Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
+
+## Capabilities (optional)
+
+If you set `capabilities`, the entry only runs for those media types. For shared
+lists, OpenClaw can infer defaults:
+
+- `openai`, `anthropic`, `minimax`: **image**
+- `google` (Gemini API): **image + audio + video**
+- `groq`: **audio**
+- `deepgram`: **audio**
+
+For CLI entries, **set `capabilities` explicitly** to avoid surprising matches.
+If you omit `capabilities`, the entry is eligible for the list it appears in.
+
+## Provider support matrix (OpenClaw integrations)
+
+| Capability | Provider integration                             | Notes                                             |
+| ---------- | ------------------------------------------------ | ------------------------------------------------- |
+| Image      | OpenAI / Anthropic / Google / others via `pi-ai` | Any image-capable model in the registry works.    |
+| Audio      | OpenAI, Groq, Deepgram, Google                   | Provider transcription (Whisper/Deepgram/Gemini). |
+| Video      | Google (Gemini API)                              | Provider video understanding.                     |
+
+## Recommended providers
+
+**Image**
+
+- Prefer your active model if it supports images.
+- Good defaults: `openai/gpt-5.2`, `anthropic/claude-opus-4-6`, `google/gemini-3-pro-preview`.
+
+**Audio**
+
+- `openai/gpt-4o-mini-transcribe`, `groq/whisper-large-v3-turbo`, or `deepgram/nova-3`.
+- CLI fallback: `whisper-cli` (whisper-cpp) or `whisper`.
+- Deepgram setup: [Deepgram (audio transcription)](/providers/deepgram).
+
+**Video**
+
+- `google/gemini-3-flash-preview` (fast), `google/gemini-3-pro-preview` (richer).
+- CLI fallback: `gemini` CLI (supports `read_file` on video/audio).
+
+## Attachment policy
+
+Per‑capability `attachments` controls which attachments are processed:
+
+- `mode`: `first` (default) or `all`
+- `maxAttachments`: cap the number processed (default **1**)
+- `prefer`: `first`, `last`, `path`, `url`
+
+When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
+
+## Config examples
+
+### 1) Shared models list + overrides
+
+```json5
+{
+  tools: {
+    media: {
+      models: [
+        { provider: "openai", model: "gpt-5.2", capabilities: ["image"] },
+        {
+          provider: "google",
+          model: "gemini-3-flash-preview",
+          capabilities: ["image", "audio", "video"],
+        },
+        {
+          type: "cli",
+          command: "gemini",
+          args: [
+            "-m",
+            "gemini-3-flash",
+            "--allowed-tools",
+            "read_file",
+            "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+          ],
+          capabilities: ["image", "video"],
+        },
+      ],
+      audio: {
+        attachments: { mode: "all", maxAttachments: 2 },
+      },
+      video: {
+        maxChars: 500,
+      },
+    },
+  },
+}
+```
+
+### 2) Audio + Video only (image off)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        models: [
+          { provider: "openai", model: "gpt-4o-mini-transcribe" },
+          {
+            type: "cli",
+            command: "whisper",
+            args: ["--model", "base", "{{MediaPath}}"],
+          },
+        ],
+      },
+      video: {
+        enabled: true,
+        maxChars: 500,
+        models: [
+          { provider: "google", model: "gemini-3-flash-preview" },
+          {
+            type: "cli",
+            command: "gemini",
+            args: [
+              "-m",
+              "gemini-3-flash",
+              "--allowed-tools",
+              "read_file",
+              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+            ],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+### 3) Optional image understanding
+
+```json5
+{
+  tools: {
+    media: {
+      image: {
+        enabled: true,
+        maxBytes: 10485760,
+        maxChars: 500,
+        models: [
+          { provider: "openai", model: "gpt-5.2" },
+          { provider: "anthropic", model: "claude-opus-4-6" },
+          {
+            type: "cli",
+            command: "gemini",
+            args: [
+              "-m",
+              "gemini-3-flash",
+              "--allowed-tools",
+              "read_file",
+              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+            ],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+### 4) Multi‑modal single entry (explicit capabilities)
+
+```json5
+{
+  tools: {
+    media: {
+      image: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+      audio: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+      video: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+## Status output
+
+When media understanding runs, `/status` includes a short summary line:
+
+```
+📎 Media: image ok (openai/gpt-5.2) · audio skipped (maxBytes)
+```
+
+This shows per‑capability outcomes and the chosen provider/model when applicable.
+
+## Notes
+
+- Understanding is **best‑effort**. Errors do not block replies.
+- Attachments are still passed to models even when understanding is disabled.
+- Use `scope` to limit where understanding runs (e.g. only DMs).
+
+## Related docs
+
+- [Configuration](/gateway/configuration)
+- [Image & Media Support](/nodes/images)

package/docs/nodes/talk.md

@@ -0,0 +1,90 @@
+---
+summary: "Talk mode: continuous speech conversations with ElevenLabs TTS"
+read_when:
+  - Implementing Talk mode on macOS/iOS/Android
+  - Changing voice/TTS/interrupt behavior
+title: "Talk Mode"
+---
+
+# Talk Mode
+
+Talk mode is a continuous voice conversation loop:
+
+1. Listen for speech
+2. Send transcript to the model (main session, chat.send)
+3. Wait for the response
+4. Speak it via ElevenLabs (streaming playback)
+
+## Behavior (macOS)
+
+- **Always-on overlay** while Talk mode is enabled.
+- **Listening → Thinking → Speaking** phase transitions.
+- On a **short pause** (silence window), the current transcript is sent.
+- Replies are **written to WebChat** (same as typing).
+- **Interrupt on speech** (default on): if the user starts talking while the assistant is speaking, we stop playback and note the interruption timestamp for the next prompt.
+
+## Voice directives in replies
+
+The assistant may prefix its reply with a **single JSON line** to control voice:
+
+```json
+{ "voice": "<voice-id>", "once": true }
+```
+
+Rules:
+
+- First non-empty line only.
+- Unknown keys are ignored.
+- `once: true` applies to the current reply only.
+- Without `once`, the voice becomes the new default for Talk mode.
+- The JSON line is stripped before TTS playback.
+
+Supported keys:
+
+- `voice` / `voice_id` / `voiceId`
+- `model` / `model_id` / `modelId`
+- `speed`, `rate` (WPM), `stability`, `similarity`, `style`, `speakerBoost`
+- `seed`, `normalize`, `lang`, `output_format`, `latency_tier`
+- `once`
+
+## Config (`~/.openclaw/openclaw.json`)
+
+```json5
+{
+  talk: {
+    voiceId: "elevenlabs_voice_id",
+    modelId: "eleven_v3",
+    outputFormat: "mp3_44100_128",
+    apiKey: "elevenlabs_api_key",
+    interruptOnSpeech: true,
+  },
+}
+```
+
+Defaults:
+
+- `interruptOnSpeech`: true
+- `voiceId`: falls back to `ELEVENLABS_VOICE_ID` / `SAG_VOICE_ID` (or first ElevenLabs voice when API key is available)
+- `modelId`: defaults to `eleven_v3` when unset
+- `apiKey`: falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available)
+- `outputFormat`: defaults to `pcm_44100` on macOS/iOS and `pcm_24000` on Android (set `mp3_*` to force MP3 streaming)
+
+## macOS UI
+
+- Menu bar toggle: **Talk**
+- Config tab: **Talk Mode** group (voice id + interrupt toggle)
+- Overlay:
+  - **Listening**: cloud pulses with mic level
+  - **Thinking**: sinking animation
+  - **Speaking**: radiating rings
+  - Click cloud: stop speaking
+  - Click X: exit Talk mode
+
+## Notes
+
+- Requires Speech + Microphone permissions.
+- Uses `chat.send` against session key `main`.
+- TTS uses ElevenLabs streaming API with `ELEVENLABS_API_KEY` and incremental playback on macOS/iOS/Android for lower latency.
+- `stability` for `eleven_v3` is validated to `0.0`, `0.5`, or `1.0`; other models accept `0..1`.
+- `latency_tier` is validated to `0..4` when set.
+- Android supports `pcm_16000`, `pcm_22050`, `pcm_24000`, and `pcm_44100` output formats for low-latency AudioTrack streaming.

package/docs/nodes/voicewake.md

@@ -0,0 +1,65 @@
+---
+summary: "Global voice wake words (Gateway-owned) and how they sync across nodes"
+read_when:
+  - Changing voice wake words behavior or defaults
+  - Adding new node platforms that need wake word sync
+title: "Voice Wake"
+---
+
+# Voice Wake (Global Wake Words)
+
+OpenClaw treats **wake words as a single global list** owned by the **Gateway**.
+
+- There are **no per-node custom wake words**.
+- **Any node/app UI may edit** the list; changes are persisted by the Gateway and broadcast to everyone.
+- Each device still keeps its own **Voice Wake enabled/disabled** toggle (local UX + permissions differ).
+
+## Storage (Gateway host)
+
+Wake words are stored on the gateway machine at:
+
+- `~/.openclaw/settings/voicewake.json`
+
+Shape:
+
+```json
+{ "triggers": ["openclaw", "claude", "computer"], "updatedAtMs": 1730000000000 }
+```
+
+## Protocol
+
+### Methods
+
+- `voicewake.get` → `{ triggers: string[] }`
+- `voicewake.set` with params `{ triggers: string[] }` → `{ triggers: string[] }`
+
+Notes:
+
+- Triggers are normalized (trimmed, empties dropped). Empty lists fall back to defaults.
+- Limits are enforced for safety (count/length caps).
+
+### Events
+
+- `voicewake.changed` payload `{ triggers: string[] }`
+
+Who receives it:
+
+- All WebSocket clients (macOS app, WebChat, etc.)
+- All connected nodes (iOS/Android), and also on node connect as an initial “current state” push.
+
+## Client behavior
+
+### macOS app
+
+- Uses the global list to gate `VoiceWakeRuntime` triggers.
+- Editing “Trigger words” in Voice Wake settings calls `voicewake.set` and then relies on the broadcast to keep other clients in sync.
+
+### iOS node
+
+- Uses the global list for `VoiceWakeManager` trigger detection.
+- Editing Wake Words in Settings calls `voicewake.set` (over the Gateway WS) and also keeps local wake-word detection responsive.
+
+### Android node
+
+- Exposes a Wake Words editor in Settings.
+- Calls `voicewake.set` over the Gateway WS so edits sync everywhere.

package/docs/perplexity.md

@@ -0,0 +1,80 @@
+---
+summary: "Perplexity Sonar setup for web_search"
+read_when:
+  - You want to use Perplexity Sonar for web search
+  - You need PERPLEXITY_API_KEY or OpenRouter setup
+title: "Perplexity Sonar"
+---
+
+# Perplexity Sonar
+
+OpenClaw can use Perplexity Sonar for the `web_search` tool. You can connect
+through Perplexity’s direct API or via OpenRouter.
+
+## API options
+
+### Perplexity (direct)
+
+- Base URL: https://api.perplexity.ai
+- Environment variable: `PERPLEXITY_API_KEY`
+
+### OpenRouter (alternative)
+
+- Base URL: https://openrouter.ai/api/v1
+- Environment variable: `OPENROUTER_API_KEY`
+- Supports prepaid/crypto credits.
+
+## Config example
+
+```json5
+{
+  tools: {
+    web: {
+      search: {
+        provider: "perplexity",
+        perplexity: {
+          apiKey: "pplx-...",
+          baseUrl: "https://api.perplexity.ai",
+          model: "perplexity/sonar-pro",
+        },
+      },
+    },
+  },
+}
+```
+
+## Switching from Brave
+
+```json5
+{
+  tools: {
+    web: {
+      search: {
+        provider: "perplexity",
+        perplexity: {
+          apiKey: "pplx-...",
+          baseUrl: "https://api.perplexity.ai",
+        },
+      },
+    },
+  },
+}
+```
+
+If both `PERPLEXITY_API_KEY` and `OPENROUTER_API_KEY` are set, set
+`tools.web.search.perplexity.baseUrl` (or `tools.web.search.perplexity.apiKey`)
+to disambiguate.
+
+If no base URL is set, OpenClaw chooses a default based on the API key source:
+
+- `PERPLEXITY_API_KEY` or `pplx-...` → direct Perplexity (`https://api.perplexity.ai`)
+- `OPENROUTER_API_KEY` or `sk-or-...` → OpenRouter (`https://openrouter.ai/api/v1`)
+- Unknown key formats → OpenRouter (safe fallback)
+
+## Models
+
+- `perplexity/sonar` — fast Q&A with web search
+- `perplexity/sonar-pro` (default) — multi-step reasoning + web search
+- `perplexity/sonar-reasoning-pro` — deep research
+
+See [Web tools](/tools/web) for the full web_search configuration.

package/docs/pi-dev.md

@@ -0,0 +1,70 @@
+---
+title: "Pi Development Workflow"
+---
+
+# Pi Development Workflow
+
+This guide summarizes a sane workflow for working on the pi integration in OpenClaw.
+
+## Type Checking and Linting
+
+- Type check and build: `pnpm build`
+- Lint: `pnpm lint`
+- Format check: `pnpm format`
+- Full gate before pushing: `pnpm lint && pnpm build && pnpm test`
+
+## Running Pi Tests
+
+Use the dedicated script for the pi integration test set:
+
+```bash
+scripts/pi/run-tests.sh
+```
+
+To include the live test that exercises real provider behavior:
+
+```bash
+scripts/pi/run-tests.sh --live
+```
+
+The script runs all pi related unit tests via these globs:
+
+- `src/agents/pi-*.test.ts`
+- `src/agents/pi-embedded-*.test.ts`
+- `src/agents/pi-tools*.test.ts`
+- `src/agents/pi-settings.test.ts`
+- `src/agents/pi-tool-definition-adapter.test.ts`
+- `src/agents/pi-extensions/*.test.ts`
+
+## Manual Testing
+
+Recommended flow:
+
+- Run the gateway in dev mode:
+  - `pnpm gateway:dev`
+- Trigger the agent directly:
+  - `pnpm openclaw agent --message "Hello" --thinking low`
+- Use the TUI for interactive debugging:
+  - `pnpm tui`
+
+For tool call behavior, prompt for a `read` or `exec` action so you can see tool streaming and payload handling.
+
+## Clean Slate Reset
+
+State lives under the OpenClaw state directory. Default is `~/.openclaw`. If `OPENCLAW_STATE_DIR` is set, use that directory instead.
+
+To reset everything:
+
+- `openclaw.json` for config
+- `credentials/` for auth profiles and tokens
+- `agents/<agentId>/sessions/` for agent session history
+- `agents/<agentId>/sessions.json` for the session index
+- `sessions/` if legacy paths exist
+- `workspace/` if you want a blank workspace
+
+If you only want to reset sessions, delete `agents/<agentId>/sessions/` and `agents/<agentId>/sessions.json` for that agent. Keep `credentials/` if you do not want to reauthenticate.
+
+## References
+
+- https://docs.openclaw.ai/testing
+- https://docs.openclaw.ai/start/getting-started