vora-ai 0.1.0

package/docs/nodes/location-command.md

@@ -0,0 +1,98 @@
+---
+summary: "Location command for nodes (location.get), permission modes, and Android foreground behavior"
+read_when:
+  - Adding location node support or permissions UI
+  - Designing Android location permissions or foreground behavior
+title: "Location Command"
+---
+
+# Location command (nodes)
+
+## TL;DR
+
+- `location.get` is a node command (via `node.invoke`).
+- Off by default.
+- Android app settings use a selector: Off / While Using.
+- Separate toggle: Precise Location.
+
+## Why a selector (not just a switch)
+
+OS permissions are multi-level. We can expose a selector in-app, but the OS still decides the actual grant.
+
+- iOS/macOS may expose **While Using** or **Always** in system prompts/Settings.
+- Android app currently supports foreground location only.
+- Precise location is a separate grant (iOS 14+ “Precise”, Android “fine” vs “coarse”).
+
+Selector in UI drives our requested mode; actual grant lives in OS settings.
+
+## Settings model
+
+Per node device:
+
+- `location.enabledMode`: `off | whileUsing`
+- `location.preciseEnabled`: bool
+
+UI behavior:
+
+- Selecting `whileUsing` requests foreground permission.
+- If OS denies requested level, revert to the highest granted level and show status.
+
+## Permissions mapping (node.permissions)
+
+Optional. macOS node reports `location` via the permissions map; iOS/Android may omit it.
+
+## Command: `location.get`
+
+Called via `node.invoke`.
+
+Params (suggested):
+
+```json
+{
+  "timeoutMs": 10000,
+  "maxAgeMs": 15000,
+  "desiredAccuracy": "coarse|balanced|precise"
+}
+```
+
+Response payload:
+
+```json
+{
+  "lat": 48.20849,
+  "lon": 16.37208,
+  "accuracyMeters": 12.5,
+  "altitudeMeters": 182.0,
+  "speedMps": 0.0,
+  "headingDeg": 270.0,
+  "timestamp": "2026-01-03T12:34:56.000Z",
+  "isPrecise": true,
+  "source": "gps|wifi|cell|unknown"
+}
+```
+
+Errors (stable codes):
+
+- `LOCATION_DISABLED`: selector is off.
+- `LOCATION_PERMISSION_REQUIRED`: permission missing for requested mode.
+- `LOCATION_BACKGROUND_UNAVAILABLE`: app is backgrounded but only While Using allowed.
+- `LOCATION_TIMEOUT`: no fix in time.
+- `LOCATION_UNAVAILABLE`: system failure / no providers.
+
+## Background behavior
+
+- Android app denies `location.get` while backgrounded.
+- Keep Vora open when requesting location on Android.
+- Other node platforms may differ.
+
+## Model/tooling integration
+
+- Tool surface: `nodes` tool adds `location_get` action (node required).
+- CLI: `vora nodes location get --node <id>`.
+- Agent guidelines: only call when user enabled location and understands the scope.
+
+## UX copy (suggested)
+
+- Off: “Location sharing is disabled.”
+- While Using: “Only when Vora is open.”
+- Precise: “Use precise GPS location. Toggle off to share approximate location.”

package/docs/nodes/media-understanding.md

@@ -0,0 +1,394 @@
+---
+summary: "Inbound image/audio/video understanding (optional) with provider + CLI fallbacks"
+read_when:
+  - Designing or refactoring media understanding
+  - Tuning inbound audio/video/image preprocessing
+title: "Media Understanding"
+---
+
+# Media Understanding - Inbound (2026-01-17)
+
+Vora can **summarize inbound media** (image/audio/video) before the reply pipeline runs. It auto‑detects when local tools or provider keys are available, and can be disabled or customized. If understanding is off, models still receive the original files/URLs as usual.
+
+Vendor-specific media behavior is registered by vendor plugins, while Vora
+core owns the shared `tools.media` config, fallback order, and reply-pipeline
+integration.
+
+## Goals
+
+- Optional: pre‑digest inbound media into short text for faster routing + better command parsing.
+- Preserve original media delivery to the model (always).
+- Support **provider APIs** and **CLI fallbacks**.
+- Allow multiple models with ordered fallback (error/size/timeout).
+
+## High-level behavior
+
+1. Collect inbound attachments (`MediaPaths`, `MediaUrls`, `MediaTypes`).
+2. For each enabled capability (image/audio/video), select attachments per policy (default: **first**).
+3. Choose the first eligible model entry (size + capability + auth).
+4. If a model fails or the media is too large, **fall back to the next entry**.
+5. On success:
+   - `Body` becomes `[Image]`, `[Audio]`, or `[Video]` block.
+   - Audio sets `{{Transcript}}`; command parsing uses caption text when present,
+     otherwise the transcript.
+   - Captions are preserved as `User text:` inside the block.
+
+If understanding fails or is disabled, **the reply flow continues** with the original body + attachments.
+
+## Config overview
+
+`tools.media` supports **shared models** plus per‑capability overrides:
+
+- `tools.media.models`: shared model list (use `capabilities` to gate).
+- `tools.media.image` / `tools.media.audio` / `tools.media.video`:
+  - defaults (`prompt`, `maxChars`, `maxBytes`, `timeoutSeconds`, `language`)
+  - provider overrides (`baseUrl`, `headers`, `providerOptions`)
+  - Deepgram audio options via `tools.media.audio.providerOptions.deepgram`
+  - audio transcript echo controls (`echoTranscript`, default `false`; `echoFormat`)
+  - optional **per‑capability `models` list** (preferred before shared models)
+  - `attachments` policy (`mode`, `maxAttachments`, `prefer`)
+  - `scope` (optional gating by channel/chatType/session key)
+- `tools.media.concurrency`: max concurrent capability runs (default **2**).
+
+```json5
+{
+  tools: {
+    media: {
+      models: [
+        /* shared list */
+      ],
+      image: {
+        /* optional overrides */
+      },
+      audio: {
+        /* optional overrides */
+        echoTranscript: true,
+        echoFormat: '📝 "{transcript}"',
+      },
+      video: {
+        /* optional overrides */
+      },
+    },
+  },
+}
+```
+
+### Model entries
+
+Each `models[]` entry can be **provider** or **CLI**:
+
+```json5
+{
+  type: "provider", // default if omitted
+  provider: "openai",
+  model: "gpt-5.2",
+  prompt: "Describe the image in <= 500 chars.",
+  maxChars: 500,
+  maxBytes: 10485760,
+  timeoutSeconds: 60,
+  capabilities: ["image"], // optional, used for multi‑modal entries
+  profile: "vision-profile",
+  preferredProfile: "vision-fallback",
+}
+```
+
+```json5
+{
+  type: "cli",
+  command: "gemini",
+  args: [
+    "-m",
+    "gemini-3-flash",
+    "--allowed-tools",
+    "read_file",
+    "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+  ],
+  maxChars: 500,
+  maxBytes: 52428800,
+  timeoutSeconds: 120,
+  capabilities: ["video", "image"],
+}
+```
+
+CLI templates can also use:
+
+- `{{MediaDir}}` (directory containing the media file)
+- `{{OutputDir}}` (scratch dir created for this run)
+- `{{OutputBase}}` (scratch file base path, no extension)
+
+## Defaults and limits
+
+Recommended defaults:
+
+- `maxChars`: **500** for image/video (short, command‑friendly)
+- `maxChars`: **unset** for audio (full transcript unless you set a limit)
+- `maxBytes`:
+  - image: **10MB**
+  - audio: **20MB**
+  - video: **50MB**
+
+Rules:
+
+- If media exceeds `maxBytes`, that model is skipped and the **next model is tried**.
+- Audio files smaller than **1024 bytes** are treated as empty/corrupt and skipped before provider/CLI transcription.
+- If the model returns more than `maxChars`, output is trimmed.
+- `prompt` defaults to simple “Describe the {media}.” plus the `maxChars` guidance (image/video only).
+- If `<capability>.enabled: true` but no models are configured, Vora tries the
+  **active reply model** when its provider supports the capability.
+
+### Auto-detect media understanding (default)
+
+If `tools.media.<capability>.enabled` is **not** set to `false` and you haven’t
+configured models, Vora auto-detects in this order and **stops at the first
+working option**:
+
+1. **Local CLIs** (audio only; if installed)
+   - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
+   - `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
+   - `whisper` (Python CLI; downloads models automatically)
+2. **Gemini CLI** (`gemini`) using `read_many_files`
+3. **Provider keys**
+   - Audio: OpenAI → Groq → Deepgram → Google
+   - Image: OpenAI → Anthropic → Google → MiniMax
+   - Video: Google
+
+To disable auto-detection, set:
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: false,
+      },
+    },
+  },
+}
+```
+
+Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
+
+### Proxy environment support (provider models)
+
+When provider-based **audio** and **video** media understanding is enabled, Vora
+honors standard outbound proxy environment variables for provider HTTP calls:
+
+- `HTTPS_PROXY`
+- `HTTP_PROXY`
+- `https_proxy`
+- `http_proxy`
+
+If no proxy env vars are set, media understanding uses direct egress.
+If the proxy value is malformed, Vora logs a warning and falls back to direct
+fetch.
+
+## Capabilities (optional)
+
+If you set `capabilities`, the entry only runs for those media types. For shared
+lists, Vora can infer defaults:
+
+- `openai`, `anthropic`, `minimax`: **image**
+- `moonshot`: **image + video**
+- `google` (Gemini API): **image + audio + video**
+- `mistral`: **audio**
+- `zai`: **image**
+- `groq`: **audio**
+- `deepgram`: **audio**
+
+For CLI entries, **set `capabilities` explicitly** to avoid surprising matches.
+If you omit `capabilities`, the entry is eligible for the list it appears in.
+
+## Provider support matrix (Vora integrations)
+
+| Capability | Provider integration                               | Notes                                                                   |
+| ---------- | -------------------------------------------------- | ----------------------------------------------------------------------- |
+| Image      | OpenAI, Anthropic, Google, MiniMax, Moonshot, Z.AI | Vendor plugins register image support against core media understanding. |
+| Audio      | OpenAI, Groq, Deepgram, Google, Mistral            | Provider transcription (Whisper/Deepgram/Gemini/Voxtral).               |
+| Video      | Google, Moonshot                                   | Provider video understanding via vendor plugins.                        |
+
+## Model selection guidance
+
+- Prefer the strongest latest-generation model available for each media capability when quality and safety matter.
+- For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
+- Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
+- CLI fallbacks (`whisper-cli`, `whisper`, `gemini`) are useful when provider APIs are unavailable.
+- `parakeet-mlx` note: with `--output-dir`, Vora reads `<output-dir>/<media-basename>.txt` when output format is `txt` (or unspecified); non-`txt` formats fall back to stdout.
+
+## Attachment policy
+
+Per‑capability `attachments` controls which attachments are processed:
+
+- `mode`: `first` (default) or `all`
+- `maxAttachments`: cap the number processed (default **1**)
+- `prefer`: `first`, `last`, `path`, `url`
+
+When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
+
+## Config examples
+
+### 1) Shared models list + overrides
+
+```json5
+{
+  tools: {
+    media: {
+      models: [
+        { provider: "openai", model: "gpt-5.2", capabilities: ["image"] },
+        {
+          provider: "google",
+          model: "gemini-3-flash-preview",
+          capabilities: ["image", "audio", "video"],
+        },
+        {
+          type: "cli",
+          command: "gemini",
+          args: [
+            "-m",
+            "gemini-3-flash",
+            "--allowed-tools",
+            "read_file",
+            "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+          ],
+          capabilities: ["image", "video"],
+        },
+      ],
+      audio: {
+        attachments: { mode: "all", maxAttachments: 2 },
+      },
+      video: {
+        maxChars: 500,
+      },
+    },
+  },
+}
+```
+
+### 2) Audio + Video only (image off)
+
+```json5
+{
+  tools: {
+    media: {
+      audio: {
+        enabled: true,
+        models: [
+          { provider: "openai", model: "gpt-4o-mini-transcribe" },
+          {
+            type: "cli",
+            command: "whisper",
+            args: ["--model", "base", "{{MediaPath}}"],
+          },
+        ],
+      },
+      video: {
+        enabled: true,
+        maxChars: 500,
+        models: [
+          { provider: "google", model: "gemini-3-flash-preview" },
+          {
+            type: "cli",
+            command: "gemini",
+            args: [
+              "-m",
+              "gemini-3-flash",
+              "--allowed-tools",
+              "read_file",
+              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+            ],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+### 3) Optional image understanding
+
+```json5
+{
+  tools: {
+    media: {
+      image: {
+        enabled: true,
+        maxBytes: 10485760,
+        maxChars: 500,
+        models: [
+          { provider: "openai", model: "gpt-5.2" },
+          { provider: "anthropic", model: "claude-opus-4-6" },
+          {
+            type: "cli",
+            command: "gemini",
+            args: [
+              "-m",
+              "gemini-3-flash",
+              "--allowed-tools",
+              "read_file",
+              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
+            ],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+### 4) Multi-modal single entry (explicit capabilities)
+
+```json5
+{
+  tools: {
+    media: {
+      image: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3.1-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+      audio: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3.1-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+      video: {
+        models: [
+          {
+            provider: "google",
+            model: "gemini-3.1-pro-preview",
+            capabilities: ["image", "video", "audio"],
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+## Status output
+
+When media understanding runs, `/status` includes a short summary line:
+
+```
+📎 Media: image ok (openai/gpt-5.2) · audio skipped (maxBytes)
+```
+
+This shows per‑capability outcomes and the chosen provider/model when applicable.
+
+## Notes
+
+- Understanding is **best‑effort**. Errors do not block replies.
+- Attachments are still passed to models even when understanding is disabled.
+- Use `scope` to limit where understanding runs (e.g. only DMs).
+
+## Related docs
+
+- [Configuration](/gateway/configuration)
+- [Image & Media Support](/nodes/images)

package/docs/nodes/talk.md

@@ -0,0 +1,92 @@
+---
+summary: "Talk mode: continuous speech conversations with ElevenLabs TTS"
+read_when:
+  - Implementing Talk mode on macOS/iOS/Android
+  - Changing voice/TTS/interrupt behavior
+title: "Talk Mode"
+---
+
+# Talk Mode
+
+Talk mode is a continuous voice conversation loop:
+
+1. Listen for speech
+2. Send transcript to the model (main session, chat.send)
+3. Wait for the response
+4. Speak it via ElevenLabs (streaming playback)
+
+## Behavior (macOS)
+
+- **Always-on overlay** while Talk mode is enabled.
+- **Listening → Thinking → Speaking** phase transitions.
+- On a **short pause** (silence window), the current transcript is sent.
+- Replies are **written to WebChat** (same as typing).
+- **Interrupt on speech** (default on): if the user starts talking while the assistant is speaking, we stop playback and note the interruption timestamp for the next prompt.
+
+## Voice directives in replies
+
+The assistant may prefix its reply with a **single JSON line** to control voice:
+
+```json
+{ "voice": "<voice-id>", "once": true }
+```
+
+Rules:
+
+- First non-empty line only.
+- Unknown keys are ignored.
+- `once: true` applies to the current reply only.
+- Without `once`, the voice becomes the new default for Talk mode.
+- The JSON line is stripped before TTS playback.
+
+Supported keys:
+
+- `voice` / `voice_id` / `voiceId`
+- `model` / `model_id` / `modelId`
+- `speed`, `rate` (WPM), `stability`, `similarity`, `style`, `speakerBoost`
+- `seed`, `normalize`, `lang`, `output_format`, `latency_tier`
+- `once`
+
+## Config (`~/.vora/vora.json`)
+
+```json5
+{
+  talk: {
+    voiceId: "elevenlabs_voice_id",
+    modelId: "eleven_v3",
+    outputFormat: "mp3_44100_128",
+    apiKey: "elevenlabs_api_key",
+    silenceTimeoutMs: 1500,
+    interruptOnSpeech: true,
+  },
+}
+```
+
+Defaults:
+
+- `interruptOnSpeech`: true
+- `silenceTimeoutMs`: when unset, Talk keeps the platform default pause window before sending the transcript (`700 ms on macOS and Android, 900 ms on iOS`)
+- `voiceId`: falls back to `ELEVENLABS_VOICE_ID` / `SAG_VOICE_ID` (or first ElevenLabs voice when API key is available)
+- `modelId`: defaults to `eleven_v3` when unset
+- `apiKey`: falls back to `ELEVENLABS_API_KEY` (or gateway shell profile if available)
+- `outputFormat`: defaults to `pcm_44100` on macOS/iOS and `pcm_24000` on Android (set `mp3_*` to force MP3 streaming)
+
+## macOS UI
+
+- Menu bar toggle: **Talk**
+- Config tab: **Talk Mode** group (voice id + interrupt toggle)
+- Overlay:
+  - **Listening**: cloud pulses with mic level
+  - **Thinking**: sinking animation
+  - **Speaking**: radiating rings
+  - Click cloud: stop speaking
+  - Click X: exit Talk mode
+
+## Notes
+
+- Requires Speech + Microphone permissions.
+- Uses `chat.send` against session key `main`.
+- TTS uses ElevenLabs streaming API with `ELEVENLABS_API_KEY` and incremental playback on macOS/iOS/Android for lower latency.
+- `stability` for `eleven_v3` is validated to `0.0`, `0.5`, or `1.0`; other models accept `0..1`.
+- `latency_tier` is validated to `0..4` when set.
+- Android supports `pcm_16000`, `pcm_22050`, `pcm_24000`, and `pcm_44100` output formats for low-latency AudioTrack streaming.

package/docs/nodes/troubleshooting.md

@@ -0,0 +1,114 @@
+---
+summary: "Troubleshoot node pairing, foreground requirements, permissions, and tool failures"
+read_when:
+  - Node is connected but camera/canvas/screen/exec tools fail
+  - You need the node pairing versus approvals mental model
+title: "Node Troubleshooting"
+---
+
+# Node troubleshooting
+
+Use this page when a node is visible in status but node tools fail.
+
+## Command ladder
+
+```bash
+vora status
+vora gateway status
+vora logs --follow
+vora doctor
+vora channels status --probe
+```
+
+Then run node specific checks:
+
+```bash
+vora nodes status
+vora nodes describe --node <idOrNameOrIp>
+vora approvals get --node <idOrNameOrIp>
+```
+
+Healthy signals:
+
+- Node is connected and paired for role `node`.
+- `nodes describe` includes the capability you are calling.
+- Exec approvals show expected mode/allowlist.
+
+## Foreground requirements
+
+`canvas.*`, `camera.*`, and `screen.*` are foreground only on iOS/Android nodes.
+
+Quick check and fix:
+
+```bash
+vora nodes describe --node <idOrNameOrIp>
+vora nodes canvas snapshot --node <idOrNameOrIp>
+vora logs --follow
+```
+
+If you see `NODE_BACKGROUND_UNAVAILABLE`, bring the node app to the foreground and retry.
+
+## Permissions matrix
+
+| Capability                   | iOS                                     | Android                                      | macOS node app                | Typical failure code           |
+| ---------------------------- | --------------------------------------- | -------------------------------------------- | ----------------------------- | ------------------------------ |
+| `camera.snap`, `camera.clip` | Camera (+ mic for clip audio)           | Camera (+ mic for clip audio)                | Camera (+ mic for clip audio) | `*_PERMISSION_REQUIRED`        |
+| `screen.record`              | Screen Recording (+ mic optional)       | Screen capture prompt (+ mic optional)       | Screen Recording              | `*_PERMISSION_REQUIRED`        |
+| `location.get`               | While Using or Always (depends on mode) | Foreground/Background location based on mode | Location permission           | `LOCATION_PERMISSION_REQUIRED` |
+| `system.run`                 | n/a (node host path)                    | n/a (node host path)                         | Exec approvals required       | `SYSTEM_RUN_DENIED`            |
+
+## Pairing versus approvals
+
+These are different gates:
+
+1. **Device pairing**: can this node connect to the gateway?
+2. **Exec approvals**: can this node run a specific shell command?
+
+Quick checks:
+
+```bash
+vora devices list
+vora nodes status
+vora approvals get --node <idOrNameOrIp>
+vora approvals allowlist add --node <idOrNameOrIp> "/usr/bin/uname"
+```
+
+If pairing is missing, approve the node device first.
+If pairing is fine but `system.run` fails, fix exec approvals/allowlist.
+
+## Common node error codes
+
+- `NODE_BACKGROUND_UNAVAILABLE` → app is backgrounded; bring it foreground.
+- `CAMERA_DISABLED` → camera toggle disabled in node settings.
+- `*_PERMISSION_REQUIRED` → OS permission missing/denied.
+- `LOCATION_DISABLED` → location mode is off.
+- `LOCATION_PERMISSION_REQUIRED` → requested location mode not granted.
+- `LOCATION_BACKGROUND_UNAVAILABLE` → app is backgrounded but only While Using permission exists.
+- `SYSTEM_RUN_DENIED: approval required` → exec request needs explicit approval.
+- `SYSTEM_RUN_DENIED: allowlist miss` → command blocked by allowlist mode.
+  On Windows node hosts, shell-wrapper forms like `cmd.exe /c ...` are treated as allowlist misses in
+  allowlist mode unless approved via ask flow.
+
+## Fast recovery loop
+
+```bash
+vora nodes status
+vora nodes describe --node <idOrNameOrIp>
+vora approvals get --node <idOrNameOrIp>
+vora logs --follow
+```
+
+If still stuck:
+
+- Re-approve device pairing.
+- Re-open node app (foreground).
+- Re-grant OS permissions.
+- Recreate/adjust exec approval policy.
+
+Related:
+
+- [/nodes/index](/nodes/index)
+- [/nodes/camera](/nodes/camera)
+- [/nodes/location-command](/nodes/location-command)
+- [/tools/exec-approvals](/tools/exec-approvals)
+- [/gateway/pairing](/gateway/pairing)