fengming 0.3.9 → 0.3.11

package/docs/nodes/audio.md

@@ -1,216 +0,0 @@
----
-summary: "How inbound audio/voice notes are downloaded, transcribed, and injected into replies"
-read_when:
-  - Changing audio transcription or media handling
-title: "Audio and voice notes"
----
-
-## What works
-
-- **Media understanding (audio)**: If audio understanding is enabled (or auto-detected), FengMing:
-  1. Locates the first audio attachment (local path or URL) and downloads it if needed.
-  2. Enforces `maxBytes` before sending to each model entry.
-  3. Runs the first eligible model entry in order (provider or CLI).
-  4. If it fails or skips (size/timeout), it tries the next entry.
-  5. On success, it replaces `Body` with an `[Audio]` block and sets `{{Transcript}}`.
-- **Command parsing**: When transcription succeeds, `CommandBody`/`RawBody` are set to the transcript so slash commands still work.
-- **Verbose logging**: In `--verbose`, we log when transcription runs and when it replaces the body.
-
-## Auto-detection (default)
-
-If you **don't configure models** and `tools.media.audio.enabled` is **not** set to `false`,
-FengMing auto-detects in this order and stops at the first working option:
-
-1. **Active reply model** when its provider supports audio understanding.
-2. **Local CLIs** (if installed)
-   - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
-   - `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
-   - `whisper` (Python CLI; downloads models automatically)
-3. **Provider auth**
-   - Configured `models.providers.*` entries that support audio are tried first
-   - Bundled fallback order: OpenAI → Groq → xAI → Deepgram → Google → SenseAudio → ElevenLabs → Mistral
-
-As of 2026-05-22, Gemini CLI auto-detect is no longer supported for media understanding. Google is transitioning Gemini CLI users to Antigravity CLI; audio should use local or provider transcription, while image/video CLI fallback should move to Antigravity CLI (`agy`).
-
-To disable auto-detection, set `tools.media.audio.enabled: false`.
-To customize, set `tools.media.audio.models`.
-Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
-
-## Config examples
-
-### Provider + CLI fallback (OpenAI + Whisper CLI)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        maxBytes: 20971520,
-        models: [
-          { provider: "openai", model: "gpt-4o-mini-transcribe" },
-          {
-            type: "cli",
-            command: "whisper",
-            args: ["--model", "base", "{{MediaPath}}"],
-            timeoutSeconds: 45,
-          },
-        ],
-      },
-    },
-  },
-}
-```
-
-### Provider-only with scope gating
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        scope: {
-          default: "allow",
-          rules: [{ action: "deny", match: { chatType: "group" } }],
-        },
-        models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
-      },
-    },
-  },
-}
-```
-
-### Provider-only (Deepgram)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        models: [{ provider: "deepgram", model: "nova-3" }],
-      },
-    },
-  },
-}
-```
-
-### Provider-only (Mistral Voxtral)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        models: [{ provider: "mistral", model: "voxtral-mini-latest" }],
-      },
-    },
-  },
-}
-```
-
-### Provider-only (SenseAudio)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        models: [{ provider: "senseaudio", model: "senseaudio-asr-pro-1.5-260319" }],
-      },
-    },
-  },
-}
-```
-
-### Echo transcript to chat (opt-in)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        echoTranscript: true, // default is false
-        echoFormat: '📝 "{transcript}"', // optional, supports {transcript}
-        models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
-      },
-    },
-  },
-}
-```
-
-## Notes and limits
-
-- Provider auth follows the standard model auth order (auth profiles, env vars, `models.providers.*.apiKey`).
-- Groq setup details: [Groq](/providers/groq).
-- Deepgram picks up `DEEPGRAM_API_KEY` when `provider: "deepgram"` is used.
-- Deepgram setup details: [Deepgram (audio transcription)](/providers/deepgram).
-- Mistral setup details: [Mistral](/providers/mistral).
-- SenseAudio picks up `SENSEAUDIO_API_KEY` when `provider: "senseaudio"` is used.
-- SenseAudio setup details: [SenseAudio](/providers/senseaudio).
-- Audio providers can override `baseUrl`, `headers`, and `providerOptions` via `tools.media.audio`.
-- Default size cap is 20MB (`tools.media.audio.maxBytes`). Oversize audio is skipped for that model and the next entry is tried.
-- Tiny/empty audio files below 1024 bytes are skipped before provider/CLI transcription.
-- Default `maxChars` for audio is **unset** (full transcript). Set `tools.media.audio.maxChars` or per-entry `maxChars` to trim output.
-- OpenAI auto default is `gpt-4o-mini-transcribe`; set `model: "gpt-4o-transcribe"` for higher accuracy.
-- Use `tools.media.audio.attachments` to process multiple voice notes (`mode: "all"` + `maxAttachments`).
-- Transcript is available to templates as `{{Transcript}}`.
-- `tools.media.audio.echoTranscript` is off by default; enable it to send transcript confirmation back to the originating chat before agent processing.
-- `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`).
-- CLI stdout is capped (5MB); keep CLI output concise.
-- CLI `args` should use `{{MediaPath}}` for the local audio file path. Run `fengming doctor --fix` to migrate deprecated `{input}` placeholders from older `audio.transcription.command` configs.
-
-### Proxy environment support
-
-Provider-based audio transcription honors standard outbound proxy env vars:
-
-- `HTTPS_PROXY`
-- `HTTP_PROXY`
-- `ALL_PROXY`
-- `https_proxy`
-- `http_proxy`
-- `all_proxy`
-
-If no proxy env vars are set, direct egress is used. If proxy config is malformed, FengMing logs a warning and falls back to direct fetch.
-
-## Mention detection in groups
-
-When `requireMention: true` is set for a group chat, FengMing now transcribes audio **before** checking for mentions. This allows voice notes to be processed even when they contain mentions.
-
-**How it works:**
-
-1. If a voice message has no text body and the group requires mentions, FengMing performs a "preflight" transcription.
-2. The transcript is checked for mention patterns (e.g., `@BotName`, emoji triggers).
-3. If a mention is found, the message proceeds through the full reply pipeline.
-4. The transcript is used for mention detection so voice notes can pass the mention gate.
-
-**Fallback behavior:**
-
-- If transcription fails during preflight (timeout, API error, etc.), the message is processed based on text-only mention detection.
-- This ensures that mixed messages (text + audio) are never incorrectly dropped.
-
-**Opt-out per Telegram group/topic:**
-
-- Set `channels.telegram.groups.<chatId>.disableAudioPreflight: true` to skip preflight transcript mention checks for that group.
-- Set `channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight` to override per-topic (`true` to skip, `false` to force-enable).
-- Default is `false` (preflight enabled when mention-gated conditions match).
-
-**Example:** A user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with `requireMention: true`. The voice note is transcribed, the mention is detected, and the agent replies.
-
-## Gotchas
-
-- Scope rules use first-match wins. `chatType` is normalized to `direct`, `group`, or `room`.
-- Ensure your CLI exits 0 and prints plain text; JSON needs to be massaged via `jq -r .text`.
-- For `parakeet-mlx`, if you pass `--output-dir`, FengMing reads `<output-dir>/<media-basename>.txt` when `--output-format` is `txt` (or omitted); non-`txt` output formats fall back to stdout parsing.
-- Keep timeouts reasonable (`timeoutSeconds`, default 60s) to avoid blocking the reply queue.
-- Preflight transcription only processes the **first** audio attachment for mention detection. Additional audio is processed during the main media understanding phase.
-
-## Related
-
-- [Media understanding](/nodes/media-understanding)
-- [Talk mode](/nodes/talk)
-- [Voice wake](/nodes/voicewake)

package/docs/nodes/camera.md

@@ -1,166 +0,0 @@
----
-summary: "Camera capture (iOS/Android nodes + macOS app) for agent use: photos (jpg) and short video clips (mp4)"
-read_when:
-  - Adding or modifying camera capture on iOS/Android nodes or macOS
-  - Extending agent-accessible MEDIA temp-file workflows
-title: "Camera capture"
----
-
-FengMing supports **camera capture** for agent workflows:
-
-- **iOS node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
-- **Android node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
-- **macOS app** (node via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
-
-All camera access is gated behind **user-controlled settings**.
-
-## iOS node
-
-### User setting (default on)
-
-- iOS Settings tab → **Camera** → **Allow Camera** (`camera.enabled`)
-  - Default: **on** (missing key is treated as enabled).
-  - When off: `camera.*` commands return `CAMERA_DISABLED`.
-
-### Commands (via Gateway `node.invoke`)
-
-- `camera.list`
-  - Response payload:
-    - `devices`: array of `{ id, name, position, deviceType }`
-
-- `camera.snap`
-  - Params:
-    - `facing`: `front|back` (default: `front`)
-    - `maxWidth`: number (optional; default `1600` on the iOS node)
-    - `quality`: `0..1` (optional; default `0.9`)
-    - `format`: currently `jpg`
-    - `delayMs`: number (optional; default `0`)
-    - `deviceId`: string (optional; from `camera.list`)
-  - Response payload:
-    - `format: "jpg"`
-    - `base64: "<...>"`
-    - `width`, `height`
-  - Payload guard: photos are recompressed to keep the base64 payload under 5 MB.
-
-- `camera.clip`
-  - Params:
-    - `facing`: `front|back` (default: `front`)
-    - `durationMs`: number (default `3000`, clamped to a max of `60000`)
-    - `includeAudio`: boolean (default `true`)
-    - `format`: currently `mp4`
-    - `deviceId`: string (optional; from `camera.list`)
-  - Response payload:
-    - `format: "mp4"`
-    - `base64: "<...>"`
-    - `durationMs`
-    - `hasAudio`
-
-### Foreground requirement
-
-Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
-
-### CLI helper (temp files + MEDIA)
-
-The easiest way to get attachments is via the CLI helper, which writes decoded media to a temp file and prints `MEDIA:<path>`.
-
-Examples:
-
-```bash
-fengming nodes camera snap --node <id>               # default: both front + back (2 MEDIA lines)
-fengming nodes camera snap --node <id> --facing front
-fengming nodes camera clip --node <id> --duration 3000
-fengming nodes camera clip --node <id> --no-audio
-```
-
-Notes:
-
-- `nodes camera snap` defaults to **both** facings to give the agent both views.
-- Output files are temporary (in the OS temp directory) unless you build your own wrapper.
-
-## Android node
-
-### Android user setting (default on)
-
-- Android Settings sheet → **Camera** → **Allow Camera** (`camera.enabled`)
-  - Default: **on** (missing key is treated as enabled).
-  - When off: `camera.*` commands return `CAMERA_DISABLED`.
-
-### Permissions
-
-- Android requires runtime permissions:
-  - `CAMERA` for both `camera.snap` and `camera.clip`.
-  - `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`.
-
-If permissions are missing, the app will prompt when possible; if denied, `camera.*` requests fail with a
-`*_PERMISSION_REQUIRED` error.
-
-### Android foreground requirement
-
-Like `canvas.*`, the Android node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
-
-### Android commands (via Gateway `node.invoke`)
-
-- `camera.list`
-  - Response payload:
-    - `devices`: array of `{ id, name, position, deviceType }`
-
-### Payload guard
-
-Photos are recompressed to keep the base64 payload under 5 MB.
-
-## macOS app
-
-### User setting (default off)
-
-The macOS companion app exposes a checkbox:
-
-- **Settings → General → Allow Camera** (`fengming.cameraEnabled`)
-  - Default: **off**
-  - When off: camera requests return "Camera disabled by user".
-
-### CLI helper (node invoke)
-
-Use the main `fengming` CLI to invoke camera commands on the macOS node.
-
-Examples:
-
-```bash
-fengming nodes camera list --node <id>            # list camera ids
-fengming nodes camera snap --node <id>            # prints MEDIA:<path>
-fengming nodes camera snap --node <id> --max-width 1280
-fengming nodes camera snap --node <id> --delay-ms 2000
-fengming nodes camera snap --node <id> --device-id <id>
-fengming nodes camera clip --node <id> --duration 10s          # prints MEDIA:<path>
-fengming nodes camera clip --node <id> --duration-ms 3000      # prints MEDIA:<path> (legacy flag)
-fengming nodes camera clip --node <id> --device-id <id>
-fengming nodes camera clip --node <id> --no-audio
-```
-
-Notes:
-
-- `fengming nodes camera snap` defaults to `maxWidth=1600` unless overridden.
-- On macOS, `camera.snap` waits `delayMs` (default 2000ms) after warm-up/exposure settle before capturing.
-- Photo payloads are recompressed to keep base64 under 5 MB.
-
-## Safety + practical limits
-
-- Camera and microphone access trigger the usual OS permission prompts (and require usage strings in Info.plist).
-- Video clips are capped (currently `<= 60s`) to avoid oversized node payloads (base64 overhead + message limits).
-
-## macOS screen video (OS-level)
-
-For _screen_ video (not camera), use the macOS companion:
-
-```bash
-fengming nodes screen record --node <id> --duration 10s --fps 15   # prints MEDIA:<path>
-```
-
-Notes:
-
-- Requires macOS **Screen Recording** permission (TCC).
-
-## Related
-
-- [Image and media support](/nodes/images)
-- [Media understanding](/nodes/media-understanding)
-- [Location command](/nodes/location-command)

package/docs/nodes/images.md

@@ -1,77 +0,0 @@
----
-summary: "Image and media handling rules for send, gateway, and agent replies"
-read_when:
-  - Modifying media pipeline or attachments
-title: "Image and media support"
----
-
-The WhatsApp channel runs via **Baileys Web**. This document captures the current media handling rules for send, gateway, and agent replies.
-
-## Goals
-
-- Send media with optional captions via `fengming message send --media`.
-- Allow auto-replies from the web inbox to include media alongside text.
-- Keep per-type limits sane and predictable.
-
-## CLI Surface
-
-- `fengming message send --media <path-or-url> [--message <caption>]`
-  - `--media` optional; caption can be empty for media-only sends.
-  - `--dry-run` prints the resolved payload; `--json` emits `{ channel, to, messageId, mediaUrl, caption }`.
-
-## WhatsApp Web channel behavior
-
-- Input: local file path **or** HTTP(S) URL.
-- Flow: load into a Buffer, detect media kind, and build the correct payload:
-  - **Images:** resize & recompress to JPEG (max side 2048px) targeting `channels.whatsapp.mediaMaxMb` (default: 50 MB).
-  - **Audio/Voice/Video:** pass-through up to 16 MB; audio is sent as a voice note (`ptt: true`).
-  - **Documents:** anything else, up to 100 MB, with filename preserved when available.
-- WhatsApp GIF-style playback: send an MP4 with `gifPlayback: true` (CLI: `--gif-playback`) so mobile clients loop inline.
-- MIME detection prefers magic bytes, then headers, then file extension.
-- Caption comes from `--message` or `reply.text`; empty caption is allowed.
-- Logging: non-verbose shows `↩️`/`✅`; verbose includes size and source path/URL.
-
-## Auto-Reply Pipeline
-
-- `getReplyFromConfig` returns `{ text?, mediaUrl?, mediaUrls? }`.
-- When media is present, the web sender resolves local paths or URLs using the same pipeline as `fengming message send`.
-- Multiple media entries are sent sequentially if provided.
-
-## Inbound Media To Commands
-
-- When inbound web messages include media, FengMing downloads to a temp file and exposes templating variables:
-  - `{{MediaUrl}}` pseudo-URL for the inbound media.
-  - `{{MediaPath}}` local temp path written before running the command.
-- When a per-session Docker sandbox is enabled, inbound media is copied into the sandbox workspace and `MediaPath`/`MediaUrl` are rewritten to a relative path like `media/inbound/<filename>`.
-- Media understanding (if configured via `tools.media.*` or shared `tools.media.models`) runs before templating and can insert `[Image]`, `[Audio]`, and `[Video]` blocks into `Body`.
-  - Audio sets `{{Transcript}}` and uses the transcript for command parsing so slash commands still work.
-  - Video and image descriptions preserve any caption text for command parsing.
-  - If the active primary image model already supports vision natively, FengMing skips the `[Image]` summary block and passes the original image to the model instead.
-- By default only the first matching image/audio/video attachment is processed; set `tools.media.<cap>.attachments` to process multiple attachments.
-
-## Limits and errors
-
-**Outbound send caps (WhatsApp web send)**
-
-- Images: up to `channels.whatsapp.mediaMaxMb` (default: 50 MB) after recompression.
-- Audio/voice/video: 16 MB cap; documents: 100 MB cap.
-- Oversize or unreadable media → clear error in logs and the reply is skipped.
-
-**Media understanding caps (transcription/description)**
-
-- Image default: 10 MB (`tools.media.image.maxBytes`).
-- Audio default: 20 MB (`tools.media.audio.maxBytes`).
-- Video default: 50 MB (`tools.media.video.maxBytes`).
-- Oversize media skips understanding, but replies still go through with the original body.
-
-## Notes for Tests
-
-- Cover send + reply flows for image/audio/document cases.
-- Validate recompression for images (size bound) and voice-note flag for audio.
-- Ensure multi-media replies fan out as sequential sends.
-
-## Related
-
-- [Camera capture](/nodes/camera)
-- [Media understanding](/nodes/media-understanding)
-- [Audio and voice notes](/nodes/audio)