@symerian/symi 3.0.20 → 3.0.22

package/docs/reference/api-usage-costs.md

@@ -1,138 +0,0 @@
----
-summary: "Audit what can spend money, which keys are used, and how to view usage"
-read_when:
-  - You want to understand which features may call paid APIs
-  - You need to audit keys, costs, and usage visibility
-  - You’re explaining /status or /usage cost reporting
-title: "API Usage and Costs"
----
-
-# API usage & costs
-
-This doc lists **features that can invoke API keys** and where their costs show up. It focuses on
-Symi features that can generate provider usage or paid API calls.
-
-## Where costs show up (chat + CLI)
-
-**Per-session cost snapshot**
-
-- `/status` shows the current session model, context usage, and last response tokens.
-- If the model uses **API-key auth**, `/status` also shows **estimated cost** for the last reply.
-
-**Per-message cost footer**
-
-- `/usage full` appends a usage footer to every reply, including **estimated cost** (API-key only).
-- `/usage tokens` shows tokens only; OAuth flows hide dollar cost.
-
-**CLI usage windows (provider quotas)**
-
-- `symi status --usage` and `symi channels list` show provider **usage windows**
-  (quota snapshots, not per-message costs).
-
-See [Token use & costs](/reference/token-use) for details and examples.
-
-## How keys are discovered
-
-Symi can pick up credentials from:
-
-- **Auth profiles** (per-agent, stored in `auth-profiles.json`).
-- **Environment variables** (e.g. `OPENAI_API_KEY`, `BRAVE_API_KEY`, `FIRECRAWL_API_KEY`).
-- **Config** (`models.providers.*.apiKey`, `tools.web.search.*`, `tools.web.fetch.firecrawl.*`,
-  `memorySearch.*`, `talk.apiKey`).
-- **Skills** (`skills.entries.<name>.apiKey`) which may export keys to the skill process env.
-
-## Features that can spend keys
-
-### 1) Core model responses (chat + tools)
-
-Every reply or tool call uses the **current model provider** (OpenAI, Anthropic, etc). This is the
-primary source of usage and cost.
-
-See [Models](/providers/models) for pricing config and [Token use & costs](/reference/token-use) for display.
-
-### 2) Media understanding (audio/image/video)
-
-Inbound media can be summarized/transcribed before the reply runs. This uses model/provider APIs.
-
-- Audio: OpenAI / Groq / Deepgram (now **auto-enabled** when keys exist).
-- Image: OpenAI / Anthropic / Google.
-- Video: Google.
-
-See [Media understanding](/nodes/media-understanding).
-
-### 3) Memory embeddings + semantic search
-
-Semantic memory search uses **embedding APIs** when configured for remote providers:
-
-- `memorySearch.provider = "openai"` → OpenAI embeddings
-- `memorySearch.provider = "gemini"` → Gemini embeddings
-- `memorySearch.provider = "voyage"` → Voyage embeddings
-- Optional fallback to a remote provider if local embeddings fail
-
-You can keep it local with `memorySearch.provider = "local"` (no API usage).
-
-See [Memory](/concepts/memory).
-
-### 4) Web search tool (Brave / Perplexity via OpenRouter)
-
-`web_search` uses API keys and may incur usage charges:
-
-- **Brave Search API**: `BRAVE_API_KEY` or `tools.web.search.apiKey`
-- **Perplexity** (via OpenRouter): `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY`
-
-**Brave free tier (generous):**
-
-- **2,000 requests/month**
-- **1 request/second**
-- **Credit card required** for verification (no charge unless you upgrade)
-
-See [Web tools](/tools/web).
-
-### 5) Web fetch tool (Firecrawl)
-
-`web_fetch` can call **Firecrawl** when an API key is present:
-
-- `FIRECRAWL_API_KEY` or `tools.web.fetch.firecrawl.apiKey`
-
-If Firecrawl isn’t configured, the tool falls back to direct fetch + readability (no paid API).
-
-See [Web tools](/tools/web).
-
-### 6) Provider usage snapshots (status/health)
-
-Some status commands call **provider usage endpoints** to display quota windows or auth health.
-These are typically low-volume calls but still hit provider APIs:
-
-- `symi status --usage`
-- `symi models status --json`
-
-See [Models CLI](/cli/models).
-
-### 7) Compaction safeguard summarization
-
-The compaction safeguard can summarize session history using the **current model**, which
-invokes provider APIs when it runs.
-
-See [Session management + compaction](/reference/session-management-compaction).
-
-### 8) Model scan / probe
-
-`symi models scan` can probe OpenRouter models and uses `OPENROUTER_API_KEY` when
-probing is enabled.
-
-See [Models CLI](/cli/models).
-
-### 9) Talk (speech)
-
-Talk mode can invoke **ElevenLabs** when configured:
-
-- `ELEVENLABS_API_KEY` or `talk.apiKey`
-
-See [Talk mode](/nodes/talk).
-
-### 10) Skills (third-party APIs)
-
-Skills can store `apiKey` in `skills.entries.<name>.apiKey`. If a skill uses that key for external
-APIs, it can incur costs according to the skill’s provider.
-
-See [Skills](/tools/skills).

package/docs/reference/credits.md

@@ -1,28 +0,0 @@
----
-summary: "Project origin, contributors, and license."
-read_when:
-  - You want the project backstory or contributor credits
-title: "Credits"
----
-
-## The name
-
-Symi = SHELL + TARDIS, because every space pipeline needs a time and space machine.
-
-## Credits
-
-- **Peter Steinberger** ([@steipete](https://x.com/steipete)) - Creator, pipeline whisperer
-- **Mario Zechner** ([@badlogicc](https://x.com/badlogicgames)) - Pi creator, security pen tester
-- **Symi** - The space pipeline who demanded a better name
-
-## Core contributors
-
-- **Maxim Vovshin** (@Hyaxia, [36747317+Hyaxia@users.noreply.github.com](mailto:36747317+Hyaxia@users.noreply.github.com)) - Blogwatcher skill
-- **Nacho Iacovino** (@nachoiacovino, [nacho.iacovino@gmail.com](mailto:nacho.iacovino@gmail.com)) - Location parsing (Telegram and WhatsApp)
-- **Vincent Koc** ([@vincentkoc](https://github.com/vincentkoc), [@vincent_koc](https://x.com/vincent_koc)) - Agents, Telemetry, Hooks, Security
-
-## License
-
-MIT - Free as a pipeline in the ocean.
-
-> "We are all just playing with our own prompts." (An AI, probably high on tokens)

package/docs/reference/device-models.md

@@ -1,47 +0,0 @@
----
-summary: "How Symi vendors Apple device model identifiers for friendly names in the macOS app."
-read_when:
-  - Updating device model identifier mappings or NOTICE/license files
-  - Changing how Instances UI displays device names
-title: "Device Model Database"
----
-
-# Device model database (friendly names)
-
-The macOS companion app shows friendly Apple device model names in the **Instances** UI by mapping Apple model identifiers (e.g. `iPad16,6`, `Mac16,6`) to human-readable names.
-
-The mapping is vendored as JSON under:
-
-- `apps/macos/Sources/Symi/Resources/DeviceModels/`
-
-## Data source
-
-We currently vendor the mapping from the MIT-licensed repository:
-
-- `kyle-seongwoo-jun/apple-device-identifiers`
-
-To keep builds deterministic, the JSON files are pinned to specific upstream commits (recorded in `apps/macos/Sources/Symi/Resources/DeviceModels/NOTICE.md`).
-
-## Updating the database
-
-1. Pick the upstream commits you want to pin to (one for iOS, one for macOS).
-2. Update the commit hashes in `apps/macos/Sources/Symi/Resources/DeviceModels/NOTICE.md`.
-3. Re-download the JSON files, pinned to those commits:
-
-```bash
-IOS_COMMIT="<commit sha for ios-device-identifiers.json>"
-MAC_COMMIT="<commit sha for mac-device-identifiers.json>"
-
-curl -fsSL "https://raw.githubusercontent.com/kyle-seongwoo-jun/apple-device-identifiers/${IOS_COMMIT}/ios-device-identifiers.json" \
-  -o apps/macos/Sources/Symi/Resources/DeviceModels/ios-device-identifiers.json
-
-curl -fsSL "https://raw.githubusercontent.com/kyle-seongwoo-jun/apple-device-identifiers/${MAC_COMMIT}/mac-device-identifiers.json" \
-  -o apps/macos/Sources/Symi/Resources/DeviceModels/mac-device-identifiers.json
-```
-
-4. Ensure `apps/macos/Sources/Symi/Resources/DeviceModels/LICENSE.apple-device-identifiers.txt` still matches upstream (replace it if the upstream license changes).
-5. Verify the macOS app builds cleanly (no warnings):
-
-```bash
-swift build --package-path apps/macos
-```

package/docs/reference/rpc.md

@@ -1,43 +0,0 @@
----
-summary: "RPC adapters for external CLIs (signal-cli, legacy imsg) and gateway patterns"
-read_when:
-  - Adding or changing external CLI integrations
-  - Debugging RPC adapters (signal-cli, imsg)
-title: "RPC Adapters"
----
-
-# RPC adapters
-
-Symi integrates external CLIs via JSON-RPC. Two patterns are used today.
-
-## Pattern A: HTTP daemon (signal-cli)
-
-- `signal-cli` runs as a daemon with JSON-RPC over HTTP.
-- Event stream is SSE (`/api/v1/events`).
-- Health probe: `/api/v1/check`.
-- Symi owns lifecycle when `channels.signal.autoStart=true`.
-
-See [Signal](/channels/signal) for setup and endpoints.
-
-## Pattern B: stdio child process (legacy: imsg)
-
-> **Note:** For new iMessage setups, use [BlueBubbles](/channels/bluebubbles) instead.
-
-- Symi spawns `imsg rpc` as a child process (legacy iMessage integration).
-- JSON-RPC is line-delimited over stdin/stdout (one JSON object per line).
-- No TCP port, no daemon required.
-
-Core methods used:
-
-- `watch.subscribe` → notifications (`method: "message"`)
-- `watch.unsubscribe`
-- `send`
-- `chats.list` (probe/diagnostics)
-
-See [iMessage](/channels/imessage) for legacy setup and addressing (`chat_id` preferred).
-
-## Adapter guidelines
-
-- Gateway owns the process (start/stop tied to provider lifecycle).
-- Keep RPC clients resilient: timeouts, restart on exit.
-- Prefer stable IDs (e.g., `chat_id`) over display strings.

package/docs/reference/session-management-compaction.md

@@ -1,284 +0,0 @@
----
-summary: "Deep dive: session store + transcripts, lifecycle, and (auto)compaction internals"
-read_when:
-  - You need to debug session ids, transcript JSONL, or sessions.json fields
-  - You are changing auto-compaction behavior or adding “pre-compaction” housekeeping
-  - You want to implement memory flushes or silent system turns
-title: "Session Management Deep Dive"
----
-
-# Session Management & Compaction (Deep Dive)
-
-This document explains how Symi manages sessions end-to-end:
-
-- **Session routing** (how inbound messages map to a `sessionKey`)
-- **Session store** (`sessions.json`) and what it tracks
-- **Transcript persistence** (`*.jsonl`) and its structure
-- **Transcript hygiene** (provider-specific fixups before runs)
-- **Context limits** (context window vs tracked tokens)
-- **Compaction** (manual + auto-compaction) and where to hook pre-compaction work
-- **Silent housekeeping** (e.g. memory writes that shouldn’t produce user-visible output)
-
-If you want a higher-level overview first, start with:
-
-- [/concepts/session](/concepts/session)
-- [/concepts/compaction](/concepts/compaction)
-- [/concepts/session-pruning](/concepts/session-pruning)
-- [/reference/transcript-hygiene](/reference/transcript-hygiene)
-
----
-
-## Source of truth: the Gateway
-
-Symi is designed around a single **Gateway process** that owns session state.
-
-- UIs (macOS app, web Control UI, TUI) should query the Gateway for session lists and token counts.
-- In remote mode, session files are on the remote host; “checking your local Mac files” won’t reflect what the Gateway is using.
-
----
-
-## Two persistence layers
-
-Symi persists sessions in two layers:
-
-1. **Session store (`sessions.json`)**
-   - Key/value map: `sessionKey -> SessionEntry`
-   - Small, mutable, safe to edit (or delete entries)
-   - Tracks session metadata (current session id, last activity, toggles, token counters, etc.)
-
-2. **Transcript (`<sessionId>.jsonl`)**
-   - Append-only transcript with tree structure (entries have `id` + `parentId`)
-   - Stores the actual conversation + tool calls + compaction summaries
-   - Used to rebuild the model context for future turns
-
----
-
-## On-disk locations
-
-Per agent, on the Gateway host:
-
-- Store: `~/.symi/agents/<agentId>/sessions/sessions.json`
-- Transcripts: `~/.symi/agents/<agentId>/sessions/<sessionId>.jsonl`
-  - Telegram topic sessions: `.../<sessionId>-topic-<threadId>.jsonl`
-
-Symi resolves these via `src/config/sessions.ts`.
-
----
-
-## Session keys (`sessionKey`)
-
-A `sessionKey` identifies _which conversation bucket_ you’re in (routing + isolation).
-
-Common patterns:
-
-- Main/direct chat (per agent): `agent:<agentId>:<mainKey>` (default `main`)
-- Group: `agent:<agentId>:<channel>:group:<id>`
-- Cron: `cron:<job.id>`
-- Webhook: `hook:<uuid>` (unless overridden)
-
-The canonical rules are documented at [/concepts/session](/concepts/session).
-
----
-
-## Session ids (`sessionId`)
-
-Each `sessionKey` points at a current `sessionId` (the transcript file that continues the conversation).
-
-Rules of thumb:
-
-- **Reset** (`/new`, `/reset`) creates a new `sessionId` for that `sessionKey`.
-- **Daily reset** (default 4:00 AM local time on the gateway host) creates a new `sessionId` on the next message after the reset boundary.
-- **Idle expiry** (`session.reset.idleMinutes` or legacy `session.idleMinutes`) creates a new `sessionId` when a message arrives after the idle window. When daily + idle are both configured, whichever expires first wins.
-
-Implementation detail: the decision happens in `initSessionState()` in `src/auto-reply/reply/session.ts`.
-
----
-
-## Session store schema (`sessions.json`)
-
-The store’s value type is `SessionEntry` in `src/config/sessions.ts`.
-
-Key fields (not exhaustive):
-
-- `sessionId`: current transcript id (filename is derived from this unless `sessionFile` is set)
-- `updatedAt`: last activity timestamp
-- `sessionFile`: optional explicit transcript path override
-- `chatType`: `direct | group | room` (helps UIs and send policy)
-- `provider`, `subject`, `room`, `space`, `displayName`: metadata for group/channel labeling
-- Toggles:
-  - `thinkingLevel`, `verboseLevel`, `reasoningLevel`, `elevatedLevel`
-  - `sendPolicy` (per-session override)
-- Model selection:
-  - `providerOverride`, `modelOverride`, `authProfileOverride`
-- Token counters (best-effort / provider-dependent):
-  - `inputTokens`, `outputTokens`, `totalTokens`, `contextTokens`
-- `compactionCount`: how often auto-compaction completed for this session key
-- `memoryFlushAt`: timestamp for the last pre-compaction memory flush
-- `memoryFlushCompactionCount`: compaction count when the last flush ran
-
-The store is safe to edit, but the Gateway is the authority: it may rewrite or rehydrate entries as sessions run.
-
----
-
-## Transcript structure (`*.jsonl`)
-
-Transcripts are managed by `@mariozechner/pi-coding-agent`’s `SessionManager`.
-
-The file is JSONL:
-
-- First line: session header (`type: "session"`, includes `id`, `cwd`, `timestamp`, optional `parentSession`)
-- Then: session entries with `id` + `parentId` (tree)
-
-Notable entry types:
-
-- `message`: user/assistant/toolResult messages
-- `custom_message`: extension-injected messages that _do_ enter model context (can be hidden from UI)
-- `custom`: extension state that does _not_ enter model context
-- `compaction`: persisted compaction summary with `firstKeptEntryId` and `tokensBefore`
-- `branch_summary`: persisted summary when navigating a tree branch
-
-Symi intentionally does **not** “fix up” transcripts; the Gateway uses `SessionManager` to read/write them.
-
----
-
-## Context windows vs tracked tokens
-
-Two different concepts matter:
-
-1. **Model context window**: hard cap per model (tokens visible to the model)
-2. **Session store counters**: rolling stats written into `sessions.json` (used for /status and dashboards)
-
-If you’re tuning limits:
-
-- The context window comes from the model catalog (and can be overridden via config).
-- `contextTokens` in the store is a runtime estimate/reporting value; don’t treat it as a strict guarantee.
-
-For more, see [/token-use](/reference/token-use).
-
----
-
-## Compaction: what it is
-
-Compaction summarizes older conversation into a persisted `compaction` entry in the transcript and keeps recent messages intact.
-
-After compaction, future turns see:
-
-- The compaction summary
-- Messages after `firstKeptEntryId`
-
-Compaction is **persistent** (unlike session pruning). See [/concepts/session-pruning](/concepts/session-pruning).
-
----
-
-## When auto-compaction happens (Pi runtime)
-
-In the embedded Pi agent, auto-compaction triggers in two cases:
-
-1. **Overflow recovery**: the model returns a context overflow error → compact → retry.
-2. **Threshold maintenance**: after a successful turn, when:
-
-`contextTokens > contextWindow - reserveTokens`
-
-Where:
-
-- `contextWindow` is the model’s context window
-- `reserveTokens` is headroom reserved for prompts + the next model output
-
-These are Pi runtime semantics (Symi consumes the events, but Pi decides when to compact).
-
----
-
-## Compaction settings (`reserveTokens`, `keepRecentTokens`)
-
-Pi’s compaction settings live in Pi settings:
-
-```json5
-{
-  compaction: {
-    enabled: true,
-    reserveTokens: 16384,
-    keepRecentTokens: 20000,
-  },
-}
-```
-
-Symi also enforces a safety floor for embedded runs:
-
-- If `compaction.reserveTokens < reserveTokensFloor`, Symi bumps it.
-- Default floor is `20000` tokens.
-- Set `agents.defaults.compaction.reserveTokensFloor: 0` to disable the floor.
-- If it’s already higher, Symi leaves it alone.
-
-Why: leave enough headroom for multi-turn “housekeeping” (like memory writes) before compaction becomes unavoidable.
-
-Implementation: `ensurePiCompactionReserveTokens()` in `src/agents/pi-settings.ts`
-(called from `src/agents/pi-embedded-runner.ts`).
-
----
-
-## User-visible surfaces
-
-You can observe compaction and session state via:
-
-- `/status` (in any chat session)
-- `symi status` (CLI)
-- `symi sessions` / `sessions --json`
-- Verbose mode: `🧹 Auto-compaction complete` + compaction count
-
----
-
-## Silent housekeeping (`NO_REPLY`)
-
-Symi supports “silent” turns for background tasks where the user should not see intermediate output.
-
-Convention:
-
-- The assistant starts its output with `NO_REPLY` to indicate “do not deliver a reply to the user”.
-- Symi strips/suppresses this in the delivery layer.
-
-As of `2026.1.10`, Symi also suppresses **draft/typing streaming** when a partial chunk begins with `NO_REPLY`, so silent operations don’t leak partial output mid-turn.
-
----
-
-## Pre-compaction “memory flush” (implemented)
-
-Goal: before auto-compaction happens, run a silent agentic turn that writes durable
-state to disk (e.g. `memory/YYYY-MM-DD.md` in the agent workspace) so compaction can’t
-erase critical context.
-
-Symi uses the **pre-threshold flush** approach:
-
-1. Monitor session context usage.
-2. When it crosses a “soft threshold” (below Pi’s compaction threshold), run a silent
-   “write memory now” directive to the agent.
-3. Use `NO_REPLY` so the user sees nothing.
-
-Config (`agents.defaults.compaction.memoryFlush`):
-
-- `enabled` (default: `true`)
-- `softThresholdTokens` (default: `4000`)
-- `prompt` (user message for the flush turn)
-- `systemPrompt` (extra system prompt appended for the flush turn)
-
-Notes:
-
-- The default prompt/system prompt include a `NO_REPLY` hint to suppress delivery.
-- The flush runs once per compaction cycle (tracked in `sessions.json`).
-- The flush runs only for embedded Pi sessions (CLI backends skip it).
-- The flush is skipped when the session workspace is read-only (`workspaceAccess: "ro"` or `"none"`).
-- See [Memory](/concepts/memory) for the workspace file layout and write patterns.
-
-Pi also exposes a `session_before_compact` hook in the extension API, but Symi’s
-flush logic lives on the Gateway side today.
-
----
-
-## Troubleshooting checklist
-
-- Session key wrong? Start with [/concepts/session](/concepts/session) and confirm the `sessionKey` in `/status`.
-- Store vs transcript mismatch? Confirm the Gateway host and the store path from `symi status`.
-- Compaction spam? Check:
-  - model context window (too small)
-  - compaction settings (`reserveTokens` too high for the model window can cause earlier compaction)
-  - tool-result bloat: enable/tune session pruning
-- Silent turns leaking? Confirm the reply starts with `NO_REPLY` (exact token) and you’re on a build that includes the streaming suppression fix.

package/docs/reference/test.md

@@ -1,51 +0,0 @@
----
-summary: "How to run tests locally (vitest) and when to use force/coverage modes"
-read_when:
-  - Running or fixing tests
-title: "Tests"
----
-
-# Tests
-
-- Full testing kit (suites, live, Docker): [Testing](/help/testing)
-
-- `pnpm test:force`: Kills any lingering gateway process holding the default control port, then runs the full Vitest suite with an isolated gateway port so server tests don’t collide with a running instance. Use this when a prior gateway run left port 18789 occupied.
-- `pnpm test:coverage`: Runs the unit suite with V8 coverage (via `vitest.unit.config.ts`). Global thresholds are 70% lines/branches/functions/statements. Coverage excludes integration-heavy entrypoints (CLI wiring, gateway/telegram bridges, webchat static server) to keep the target focused on unit-testable logic.
-- `pnpm test` on Node 24+: Symi auto-disables Vitest `vmForks` and uses `forks` to avoid `ERR_VM_MODULE_LINK_FAILURE` / `module is already linked`. You can force behavior with `SYMI_TEST_VM_FORKS=0|1`.
-- `pnpm test:e2e`: Runs gateway end-to-end smoke tests (multi-instance WS/HTTP/node pairing). Defaults to `vmForks` + adaptive workers in `vitest.e2e.config.ts`; tune with `SYMI_E2E_WORKERS=<n>` and set `SYMI_E2E_VERBOSE=1` for verbose logs.
-- `pnpm test:live`: Runs provider live tests (minimax/zai). Requires API keys and `LIVE=1` (or provider-specific `*_LIVE_TEST=1`) to unskip.
-
-## Model latency bench (local keys)
-
-Script: [`scripts/bench-model.ts`](https://github.com/symi/symi/blob/main/scripts/bench-model.ts)
-
-Usage:
-
-- `source ~/.profile && pnpm tsx scripts/bench-model.ts --runs 10`
-- Optional env: `MINIMAX_API_KEY`, `MINIMAX_BASE_URL`, `MINIMAX_MODEL`, `ANTHROPIC_API_KEY`
-- Default prompt: “Reply with a single word: ok. No punctuation or extra text.”
-
-Last run (2025-12-31, 20 runs):
-
-- minimax median 1279ms (min 1114, max 2431)
-- opus median 2454ms (min 1224, max 3170)
-
-## Onboarding E2E (Docker)
-
-Docker is optional; this is only needed for containerized onboarding smoke tests.
-
-Full cold-start flow in a clean Linux container:
-
-```bash
-scripts/e2e/onboard-docker.sh
-```
-
-This script drives the interactive wizard via a pseudo-tty, verifies config/workspace/session files, then starts the gateway and runs `symi health`.
-
-## QR import smoke (Docker)
-
-Ensures `qrcode-terminal` loads under Node 22+ in Docker:
-
-```bash
-pnpm test:docker:qr
-```