@symerian/symi 3.0.21 → 3.0.22

package/docs/logging.md

@@ -1,350 +0,0 @@
----
-summary: "Logging overview: file logs, console output, CLI tailing, and the Control UI"
-read_when:
-  - You need a beginner-friendly overview of logging
-  - You want to configure log levels or formats
-  - You are troubleshooting and need to find logs quickly
-title: "Logging"
----
-
-# Logging
-
-Symi logs in two places:
-
-- **File logs** (JSON lines) written by the Gateway.
-- **Console output** shown in terminals and the Control UI.
-
-This page explains where logs live, how to read them, and how to configure log
-levels and formats.
-
-## Where logs live
-
-By default, the Gateway writes a rolling log file under:
-
-`/tmp/symi/symi-YYYY-MM-DD.log`
-
-The date uses the gateway host's local timezone.
-
-You can override this in `~/.symi/symi.json`:
-
-```json
-{
-  "logging": {
-    "file": "/path/to/symi.log"
-  }
-}
-```
-
-## How to read logs
-
-### CLI: live tail (recommended)
-
-Use the CLI to tail the gateway log file via RPC:
-
-```bash
-symi logs --follow
-```
-
-Output modes:
-
-- **TTY sessions**: pretty, colorized, structured log lines.
-- **Non-TTY sessions**: plain text.
-- `--json`: line-delimited JSON (one log event per line).
-- `--plain`: force plain text in TTY sessions.
-- `--no-color`: disable ANSI colors.
-
-In JSON mode, the CLI emits `type`-tagged objects:
-
-- `meta`: stream metadata (file, cursor, size)
-- `log`: parsed log entry
-- `notice`: truncation / rotation hints
-- `raw`: unparsed log line
-
-If the Gateway is unreachable, the CLI prints a short hint to run:
-
-```bash
-symi doctor
-```
-
-### Control UI (web)
-
-The Control UI’s **Logs** tab tails the same file using `logs.tail`.
-See [/web/control-ui](/web/control-ui) for how to open it.
-
-### Channel-only logs
-
-To filter channel activity (WhatsApp/Telegram/etc), use:
-
-```bash
-symi channels logs --channel whatsapp
-```
-
-## Log formats
-
-### File logs (JSONL)
-
-Each line in the log file is a JSON object. The CLI and Control UI parse these
-entries to render structured output (time, level, subsystem, message).
-
-### Console output
-
-Console logs are **TTY-aware** and formatted for readability:
-
-- Subsystem prefixes (e.g. `gateway/channels/whatsapp`)
-- Level coloring (info/warn/error)
-- Optional compact or JSON mode
-
-Console formatting is controlled by `logging.consoleStyle`.
-
-## Configuring logging
-
-All logging configuration lives under `logging` in `~/.symi/symi.json`.
-
-```json
-{
-  "logging": {
-    "level": "info",
-    "file": "/tmp/symi/symi-YYYY-MM-DD.log",
-    "consoleLevel": "info",
-    "consoleStyle": "pretty",
-    "redactSensitive": "tools",
-    "redactPatterns": ["sk-.*"]
-  }
-}
-```
-
-### Log levels
-
-- `logging.level`: **file logs** (JSONL) level.
-- `logging.consoleLevel`: **console** verbosity level.
-
-`--verbose` only affects console output; it does not change file log levels.
-
-### Console styles
-
-`logging.consoleStyle`:
-
-- `pretty`: human-friendly, colored, with timestamps.
-- `compact`: tighter output (best for long sessions).
-- `json`: JSON per line (for log processors).
-
-### Redaction
-
-Tool summaries can redact sensitive tokens before they hit the console:
-
-- `logging.redactSensitive`: `off` | `tools` (default: `tools`)
-- `logging.redactPatterns`: list of regex strings to override the default set
-
-Redaction affects **console output only** and does not alter file logs.
-
-## Diagnostics + OpenTelemetry
-
-Diagnostics are structured, machine-readable events for model runs **and**
-message-flow telemetry (webhooks, queueing, session state). They do **not**
-replace logs; they exist to feed metrics, traces, and other exporters.
-
-Diagnostics events are emitted in-process, but exporters only attach when
-diagnostics + the exporter plugin are enabled.
-
-### OpenTelemetry vs OTLP
-
-- **OpenTelemetry (OTel)**: the data model + SDKs for traces, metrics, and logs.
-- **OTLP**: the wire protocol used to export OTel data to a collector/backend.
-- Symi exports via **OTLP/HTTP (protobuf)** today.
-
-### Signals exported
-
-- **Metrics**: counters + histograms (token usage, message flow, queueing).
-- **Traces**: spans for model usage + webhook/message processing.
-- **Logs**: exported over OTLP when `diagnostics.otel.logs` is enabled. Log
-  volume can be high; keep `logging.level` and exporter filters in mind.
-
-### Diagnostic event catalog
-
-Model usage:
-
-- `model.usage`: tokens, cost, duration, context, provider/model/channel, session ids.
-
-Message flow:
-
-- `webhook.received`: webhook ingress per channel.
-- `webhook.processed`: webhook handled + duration.
-- `webhook.error`: webhook handler errors.
-- `message.queued`: message enqueued for processing.
-- `message.processed`: outcome + duration + optional error.
-
-Queue + session:
-
-- `queue.lane.enqueue`: command queue lane enqueue + depth.
-- `queue.lane.dequeue`: command queue lane dequeue + wait time.
-- `session.state`: session state transition + reason.
-- `session.stuck`: session stuck warning + age.
-- `run.attempt`: run retry/attempt metadata.
-- `diagnostic.heartbeat`: aggregate counters (webhooks/queue/session).
-
-### Enable diagnostics (no exporter)
-
-Use this if you want diagnostics events available to plugins or custom sinks:
-
-```json
-{
-  "diagnostics": {
-    "enabled": true
-  }
-}
-```
-
-### Diagnostics flags (targeted logs)
-
-Use flags to turn on extra, targeted debug logs without raising `logging.level`.
-Flags are case-insensitive and support wildcards (e.g. `telegram.*` or `*`).
-
-```json
-{
-  "diagnostics": {
-    "flags": ["telegram.http"]
-  }
-}
-```
-
-Env override (one-off):
-
-```
-SYMI_DIAGNOSTICS=telegram.http,telegram.payload
-```
-
-Notes:
-
-- Flag logs go to the standard log file (same as `logging.file`).
-- Output is still redacted according to `logging.redactSensitive`.
-- Full guide: [/diagnostics/flags](/diagnostics/flags).
-
-### Export to OpenTelemetry
-
-Diagnostics can be exported via the `diagnostics-otel` plugin (OTLP/HTTP). This
-works with any OpenTelemetry collector/backend that accepts OTLP/HTTP.
-
-```json
-{
-  "plugins": {
-    "allow": ["diagnostics-otel"],
-    "entries": {
-      "diagnostics-otel": {
-        "enabled": true
-      }
-    }
-  },
-  "diagnostics": {
-    "enabled": true,
-    "otel": {
-      "enabled": true,
-      "endpoint": "http://otel-collector:4318",
-      "protocol": "http/protobuf",
-      "serviceName": "symi-gateway",
-      "traces": true,
-      "metrics": true,
-      "logs": true,
-      "sampleRate": 0.2,
-      "flushIntervalMs": 60000
-    }
-  }
-}
-```
-
-Notes:
-
-- You can also enable the plugin with `symi plugins enable diagnostics-otel`.
-- `protocol` currently supports `http/protobuf` only. `grpc` is ignored.
-- Metrics include token usage, cost, context size, run duration, and message-flow
-  counters/histograms (webhooks, queueing, session state, queue depth/wait).
-- Traces/metrics can be toggled with `traces` / `metrics` (default: on). Traces
-  include model usage spans plus webhook/message processing spans when enabled.
-- Set `headers` when your collector requires auth.
-- Environment variables supported: `OTEL_EXPORTER_OTLP_ENDPOINT`,
-  `OTEL_SERVICE_NAME`, `OTEL_EXPORTER_OTLP_PROTOCOL`.
-
-### Exported metrics (names + types)
-
-Model usage:
-
-- `symi.tokens` (counter, attrs: `symi.token`, `symi.channel`,
-  `symi.provider`, `symi.model`)
-- `symi.cost.usd` (counter, attrs: `symi.channel`, `symi.provider`,
-  `symi.model`)
-- `symi.run.duration_ms` (histogram, attrs: `symi.channel`,
-  `symi.provider`, `symi.model`)
-- `symi.context.tokens` (histogram, attrs: `symi.context`,
-  `symi.channel`, `symi.provider`, `symi.model`)
-
-Message flow:
-
-- `symi.webhook.received` (counter, attrs: `symi.channel`,
-  `symi.webhook`)
-- `symi.webhook.error` (counter, attrs: `symi.channel`,
-  `symi.webhook`)
-- `symi.webhook.duration_ms` (histogram, attrs: `symi.channel`,
-  `symi.webhook`)
-- `symi.message.queued` (counter, attrs: `symi.channel`,
-  `symi.source`)
-- `symi.message.processed` (counter, attrs: `symi.channel`,
-  `symi.outcome`)
-- `symi.message.duration_ms` (histogram, attrs: `symi.channel`,
-  `symi.outcome`)
-
-Queues + sessions:
-
-- `symi.queue.lane.enqueue` (counter, attrs: `symi.lane`)
-- `symi.queue.lane.dequeue` (counter, attrs: `symi.lane`)
-- `symi.queue.depth` (histogram, attrs: `symi.lane` or
-  `symi.channel=heartbeat`)
-- `symi.queue.wait_ms` (histogram, attrs: `symi.lane`)
-- `symi.session.state` (counter, attrs: `symi.state`, `symi.reason`)
-- `symi.session.stuck` (counter, attrs: `symi.state`)
-- `symi.session.stuck_age_ms` (histogram, attrs: `symi.state`)
-- `symi.run.attempt` (counter, attrs: `symi.attempt`)
-
-### Exported spans (names + key attributes)
-
-- `symi.model.usage`
-  - `symi.channel`, `symi.provider`, `symi.model`
-  - `symi.sessionKey`, `symi.sessionId`
-  - `symi.tokens.*` (input/output/cache_read/cache_write/total)
-- `symi.webhook.processed`
-  - `symi.channel`, `symi.webhook`, `symi.chatId`
-- `symi.webhook.error`
-  - `symi.channel`, `symi.webhook`, `symi.chatId`,
-    `symi.error`
-- `symi.message.processed`
-  - `symi.channel`, `symi.outcome`, `symi.chatId`,
-    `symi.messageId`, `symi.sessionKey`, `symi.sessionId`,
-    `symi.reason`
-- `symi.session.stuck`
-  - `symi.state`, `symi.ageMs`, `symi.queueDepth`,
-    `symi.sessionKey`, `symi.sessionId`
-
-### Sampling + flushing
-
-- Trace sampling: `diagnostics.otel.sampleRate` (0.0–1.0, root spans only).
-- Metric export interval: `diagnostics.otel.flushIntervalMs` (min 1000ms).
-
-### Protocol notes
-
-- OTLP/HTTP endpoints can be set via `diagnostics.otel.endpoint` or
-  `OTEL_EXPORTER_OTLP_ENDPOINT`.
-- If the endpoint already contains `/v1/traces` or `/v1/metrics`, it is used as-is.
-- If the endpoint already contains `/v1/logs`, it is used as-is for logs.
-- `diagnostics.otel.logs` enables OTLP log export for the main logger output.
-
-### Log export behavior
-
-- OTLP logs use the same structured records written to `logging.file`.
-- Respect `logging.level` (file log level). Console redaction does **not** apply
-  to OTLP logs.
-- High-volume installs should prefer OTLP collector sampling/filtering.
-
-## Troubleshooting tips
-
-- **Gateway not reachable?** Run `symi doctor` first.
-- **Logs empty?** Check that the Gateway is running and writing to the file path
-  in `logging.file`.
-- **Need more detail?** Set `logging.level` to `debug` or `trace` and retry.

package/docs/nav-tabs-underline.js

@@ -1,100 +0,0 @@
-(() => {
-  const NAV_TABS_SELECTOR = ".nav-tabs";
-  const ACTIVE_UNDERLINE_SELECTOR = ".nav-tabs-item > div.bg-primary";
-  const UNDERLINE_CLASS = "nav-tabs-underline";
-  const READY_CLASS = "nav-tabs-underline-ready";
-
-  let navTabs = null;
-  let navTabsObserver = null;
-  let lastX = null;
-  let lastWidth = null;
-
-  const ensureUnderline = (tabs) => {
-    let underline = tabs.querySelector(`.${UNDERLINE_CLASS}`);
-    if (!underline) {
-      underline = document.createElement("div");
-      underline.className = UNDERLINE_CLASS;
-      tabs.appendChild(underline);
-    }
-    return underline;
-  };
-
-  const getActiveTab = (tabs) => {
-    const activeUnderline = tabs.querySelector(ACTIVE_UNDERLINE_SELECTOR);
-    return activeUnderline?.closest(".nav-tabs-item") ?? null;
-  };
-
-  const updateUnderline = () => {
-    if (!navTabs) {
-      return;
-    }
-
-    ensureUnderline(navTabs);
-
-    const activeTab = getActiveTab(navTabs);
-    if (!activeTab) {
-      navTabs.classList.remove(READY_CLASS);
-      return;
-    }
-
-    const navRect = navTabs.getBoundingClientRect();
-    const tabRect = activeTab.getBoundingClientRect();
-    const left = tabRect.left - navRect.left;
-
-    navTabs.style.setProperty("--nav-tab-underline-x", `${left}px`);
-    navTabs.style.setProperty("--nav-tab-underline-width", `${tabRect.width}px`);
-    navTabs.classList.add(READY_CLASS);
-
-    lastX = left;
-    lastWidth = tabRect.width;
-  };
-
-  const scheduleUpdate = () => {
-    requestAnimationFrame(updateUnderline);
-  };
-
-  const setupNavTabsObserver = (tabs) => {
-    if (!tabs || tabs === navTabs) {
-      return;
-    }
-
-    navTabs = tabs;
-    ensureUnderline(navTabs);
-    if (lastX !== null && lastWidth !== null) {
-      navTabs.style.setProperty("--nav-tab-underline-x", `${lastX}px`);
-      navTabs.style.setProperty("--nav-tab-underline-width", `${lastWidth}px`);
-      navTabs.classList.add(READY_CLASS);
-    }
-    navTabsObserver?.disconnect();
-    navTabsObserver = new MutationObserver(scheduleUpdate);
-    navTabsObserver.observe(navTabs, {
-      subtree: true,
-      attributes: true,
-      attributeFilter: ["class"],
-    });
-
-    scheduleUpdate();
-  };
-
-  const setupObservers = () => {
-    const tabs = document.querySelector(NAV_TABS_SELECTOR);
-    if (tabs) {
-      setupNavTabsObserver(tabs);
-    }
-  };
-
-  const rootObserver = new MutationObserver(setupObservers);
-
-  if (document.readyState === "loading") {
-    document.addEventListener("DOMContentLoaded", () => {
-      setupObservers();
-      rootObserver.observe(document.body, { childList: true, subtree: true });
-    });
-  } else {
-    setupObservers();
-    rootObserver.observe(document.body, { childList: true, subtree: true });
-  }
-
-  window.addEventListener("resize", scheduleUpdate);
-  void document.fonts?.ready?.then(scheduleUpdate, () => {});
-})();

package/docs/network.md

@@ -1,54 +0,0 @@
----
-summary: "Network hub: gateway surfaces, pairing, discovery, and security"
-read_when:
-  - You need the network architecture + security overview
-  - You are debugging local vs tailnet access or pairing
-  - You want the canonical list of networking docs
-title: "Network"
----
-
-# Network hub
-
-This hub links the core docs for how Symi connects, pairs, and secures
-devices across localhost, LAN, and tailnet.
-
-## Core model
-
-- [Gateway architecture](/concepts/architecture)
-- [Gateway protocol](/gateway/protocol)
-- [Gateway runbook](/gateway)
-- [Web surfaces + bind modes](/web)
-
-## Pairing + identity
-
-- [Pairing overview (DM + nodes)](/channels/pairing)
-- [Gateway-owned node pairing](/gateway/pairing)
-- [Devices CLI (pairing + token rotation)](/cli/devices)
-- [Pairing CLI (DM approvals)](/cli/pairing)
-
-Local trust:
-
-- Local connections (loopback or the gateway host’s own tailnet address) can be
-  auto‑approved for pairing to keep same‑host UX smooth.
-- Non‑local tailnet/LAN clients still require explicit pairing approval.
-
-## Discovery + transports
-
-- [Discovery & transports](/gateway/discovery)
-- [Bonjour / mDNS](/gateway/bonjour)
-- [Remote access (SSH)](/gateway/remote)
-- [Tailscale](/gateway/tailscale)
-
-## Nodes + transports
-
-- [Nodes overview](/nodes)
-- [Bridge protocol (legacy nodes)](/gateway/bridge-protocol)
-- [Node runbook: iOS](/platforms/ios)
-- [Node runbook: Android](/platforms/android)
-
-## Security
-
-- [Security overview](/gateway/security)
-- [Gateway config reference](/gateway/configuration)
-- [Troubleshooting](/gateway/troubleshooting)
-- [Doctor](/gateway/doctor)

package/docs/nodes/audio.md

@@ -1,133 +0,0 @@
----
-summary: "How inbound audio/voice notes are downloaded, transcribed, and injected into replies"
-read_when:
-  - Changing audio transcription or media handling
-title: "Audio and Voice Notes"
----
-
-# Audio / Voice Notes — 2026-01-17
-
-## What works
-
-- **Media understanding (audio)**: If audio understanding is enabled (or auto‑detected), Symi:
-  1. Locates the first audio attachment (local path or URL) and downloads it if needed.
-  2. Enforces `maxBytes` before sending to each model entry.
-  3. Runs the first eligible model entry in order (provider or CLI).
-  4. If it fails or skips (size/timeout), it tries the next entry.
-  5. On success, it replaces `Body` with an `[Audio]` block and sets `{{Transcript}}`.
-- **Command parsing**: When transcription succeeds, `CommandBody`/`RawBody` are set to the transcript so slash commands still work.
-- **Verbose logging**: In `--verbose`, we log when transcription runs and when it replaces the body.
-
-## Auto-detection (default)
-
-If you **don’t configure models** and `tools.media.audio.enabled` is **not** set to `false`,
-Symi auto-detects in this order and stops at the first working option:
-
-1. **Local CLIs** (if installed)
-   - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with encoder/decoder/joiner/tokens)
-   - `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or the bundled tiny model)
-   - `whisper` (Python CLI; downloads models automatically)
-2. **Gemini CLI** (`gemini`) using `read_many_files`
-3. **Provider keys** (OpenAI → Groq → Deepgram → Google)
-
-To disable auto-detection, set `tools.media.audio.enabled: false`.
-To customize, set `tools.media.audio.models`.
-Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (we expand `~`), or set an explicit CLI model with a full command path.
-
-## Config examples
-
-### Provider + CLI fallback (OpenAI + Whisper CLI)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        maxBytes: 20971520,
-        models: [
-          { provider: "openai", model: "gpt-4o-mini-transcribe" },
-          {
-            type: "cli",
-            command: "whisper",
-            args: ["--model", "base", "{{MediaPath}}"],
-            timeoutSeconds: 45,
-          },
-        ],
-      },
-    },
-  },
-}
-```
-
-### Provider-only with scope gating
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        scope: {
-          default: "allow",
-          rules: [{ action: "deny", match: { chatType: "group" } }],
-        },
-        models: [{ provider: "openai", model: "gpt-4o-mini-transcribe" }],
-      },
-    },
-  },
-}
-```
-
-### Provider-only (Deepgram)
-
-```json5
-{
-  tools: {
-    media: {
-      audio: {
-        enabled: true,
-        models: [{ provider: "deepgram", model: "nova-3" }],
-      },
-    },
-  },
-}
-```
-
-## Notes & limits
-
-- Provider auth follows the standard model auth order (auth profiles, env vars, `models.providers.*.apiKey`).
-- Deepgram picks up `DEEPGRAM_API_KEY` when `provider: "deepgram"` is used.
-- Deepgram setup details: [Deepgram (audio transcription)](/providers/deepgram).
-- Audio providers can override `baseUrl`, `headers`, and `providerOptions` via `tools.media.audio`.
-- Default size cap is 20MB (`tools.media.audio.maxBytes`). Oversize audio is skipped for that model and the next entry is tried.
-- Default `maxChars` for audio is **unset** (full transcript). Set `tools.media.audio.maxChars` or per-entry `maxChars` to trim output.
-- OpenAI auto default is `gpt-4o-mini-transcribe`; set `model: "gpt-4o-transcribe"` for higher accuracy.
-- Use `tools.media.audio.attachments` to process multiple voice notes (`mode: "all"` + `maxAttachments`).
-- Transcript is available to templates as `{{Transcript}}`.
-- CLI stdout is capped (5MB); keep CLI output concise.
-
-## Mention Detection in Groups
-
-When `requireMention: true` is set for a group chat, Symi now transcribes audio **before** checking for mentions. This allows voice notes to be processed even when they contain mentions.
-
-**How it works:**
-
-1. If a voice message has no text body and the group requires mentions, Symi performs a "preflight" transcription.
-2. The transcript is checked for mention patterns (e.g., `@BotName`, emoji triggers).
-3. If a mention is found, the message proceeds through the full reply pipeline.
-4. The transcript is used for mention detection so voice notes can pass the mention gate.
-
-**Fallback behavior:**
-
-- If transcription fails during preflight (timeout, API error, etc.), the message is processed based on text-only mention detection.
-- This ensures that mixed messages (text + audio) are never incorrectly dropped.
-
-**Example:** A user sends a voice note saying "Hey @Claude, what's the weather?" in a Telegram group with `requireMention: true`. The voice note is transcribed, the mention is detected, and the agent replies.
-
-## Gotchas
-
-- Scope rules use first-match wins. `chatType` is normalized to `direct`, `group`, or `room`.
-- Ensure your CLI exits 0 and prints plain text; JSON needs to be massaged via `jq -r .text`.
-- Keep timeouts reasonable (`timeoutSeconds`, default 60s) to avoid blocking the reply queue.
-- Preflight transcription only processes the **first** audio attachment for mention detection. Additional audio is processed during the main media understanding phase.