@symerian/symi 3.0.20 → 3.0.22

package/docs/glass-ui-rendering-invariants.md

@@ -1,451 +0,0 @@
-# Glass UI Rendering Invariants
-
-**Created:** 2026-04-29
-**Branch:** `symi-phase3`
-**Purpose:** Specification of the Glass UI rendering properties that the
-pipeline-collapse work (Phase 1-6) MUST preserve.
-
-This document is the contract every subsequent phase has to honor. Every
-property below maps to user-visible Glass UI behaviour and to specific
-file:line references in the current codebase. If a refactor breaks any of
-these, the rendering quality regresses and the user sees it.
-
-The corresponding parity test suite lives at
-`src/gateway/glass-ui-rendering-parity.test.ts` — every invariant has a
-test that asserts the contract.
-
----
-
-## Why this exists
-
-The pipeline-collapse plan (Phases 1-6) refactors how events flow from
-`runAgentTurn` to the user's surface. Phase 5 specifically touches the
-WebSocket subscription layer that Glass UI consumes. Without an explicit
-contract, a "cleaner" event-shape rewrite could silently drop fields the
-renderer depends on, regressing user-visible quality.
-
-The user's directive: _"the way symi renders in the glass ui is perfect
-now and i dont want to lose the ability."_
-
-This doc is the operationalization of that directive.
-
----
-
-## The 15 invariants
-
-### I-1. Incremental delta semantics
-
-**What:** When the gateway broadcasts a delta event, the `delta` (or
-`message.content[0].text` in the broadcast payload) carries ONLY the
-incremental tokens since the last delta — never accumulated text.
-
-**Why it matters:** Glass UI's `updateStream(text)` does
-`textContent += text`. If the gateway sent accumulated text, every delta
-would re-append the entire response so far, producing the "looping" bug
-documented in master doc §22 and §14.1. That bug took 18 releases and
-~6,000 lines of changes to find.
-
-**Source:**
-
-- Gateway: `src/gateway/server-chat.ts:557` — `evt.data.delta` is read,
-  not `evt.data.text`.
-- Browser: `glass-ui/js/app.js:177-184` — `updateStream(text)` appends.
-
-**Test:** `parity.test.ts > I-1: delta is incremental`
-
----
-
-### I-2. Streaming bubble lifecycle
-
-**What:** First delta opens a stream bubble. Subsequent deltas append.
-Final event closes the stream bubble and replaces plain text with rendered
-markdown HTML.
-
-**Why it matters:** Without the stream-bubble lifecycle, users see plain
-text first, then a flash to rendered HTML, then markdown. The current
-behaviour gives smooth token-by-token typing-style UX with formatted
-output at the end.
-
-**Source:**
-
-- `glass-ui/js/app.js:165-171` — `openStreamBubble()`
-- `glass-ui/js/app.js:177-184` — `updateStream(text)`
-- `glass-ui/js/app.js:185-220` — `closeStreamBubble(model)`
-- `glass-ui/js/app.js:725-731` — wired to delta events
-- `glass-ui/js/app.js:782-794` — wired to final events
-
-**Test:** `parity.test.ts > I-2: bubble open/close on delta/final`
-
----
-
-### I-3. Markdown rendering
-
-**What:** `marked.js` v15 with custom renderer for code blocks, links
-opening in new tabs, tables, lists.
-
-**Why it matters:** Code blocks have language-specific syntax highlighting
-via hljs. Tables render as actual tables. Links open in new tabs (not
-breaking the SPA). Lists indent correctly. Without this, every assistant
-reply is plain text with markdown syntax visible.
-
-**Source:**
-
-- `glass-ui/js/render.js:5-37` — `marked.setOptions` + `marked.use`
-- `glass-ui/js/render.js:38-87` — custom renderer for code, link, etc.
-- `glass-ui/js/marked.min.js` — vendored library
-
-**Test:** Manual visual regression — text with ` ```ts\nconst x = 1\n``` `
-must render as a syntax-highlighted code block, not literal triple-backtick
-text.
-
----
-
-### I-4. Syntax highlighting
-
-**What:** Code blocks invoke `hljs.highlight(raw, { language })` with
-language detection from the fence (` ```ts ` → typescript) or fallback to
-plaintext.
-
-**Why it matters:** Code reviews / paste-from-Symi workflows depend on
-readable, colored code. Without it, `<pre>` blocks are unstyled monospace.
-
-**Source:**
-
-- `glass-ui/js/render.js:57-65` — language resolution + hljs invocation
-- `glass-ui/js/highlight.min.js` — vendored library
-
-**Test:** Manual visual regression — fenced code blocks have hljs
-class names applied (`hljs-keyword`, `hljs-string`, etc.).
-
----
-
-### I-5. Thinking → Reasoning panel
-
-**What:** Agent events with `stream: "thinking"` accumulate into a
-per-`runId` buffer. On `thinking_end` (or final), the buffer flushes to a
-dedicated Reasoning panel (sidebar), not the main chat feed.
-
-**Why it matters:** Reasoning models (Nemotron, Qwen 3.5/3.6) emit
-thousands of tokens of internal monologue. Showing it inline in the chat
-bubble would drown the actual answer. The Reasoning panel keeps it
-visible-but-separate.
-
-**Source:**
-
-- `glass-ui/js/app.js:133` — `thinkingBuffers = new Map()`
-- `glass-ui/js/app.js:145` — `set(runId, "")` on thinking_start
-- `glass-ui/js/app.js:151-152` — accumulate delta or content
-- `glass-ui/js/app.js:156-157` — flush + delete on thinking_end
-
-**Test:** `parity.test.ts > I-5: thinking events route to thinking buffer`
-
----
-
-### I-6. Heartbeat → Symipulse panel
-
-**What:** Events with `isHeartbeat: true` route to the dedicated Symipulse
-panel via `window.appendToSymipulsePanel(text, timestamp)`. Heartbeat
-content does NOT appear in the main chat feed.
-
-**Why it matters:** Symipulse fires every 30 minutes producing ack/alert
-content. Without routing, the main feed accumulates routine "all clear"
-acks and obscures real conversation history.
-
-**Source:**
-
-- `glass-ui/js/app.js:406-422` — `if (m.isHeartbeat) appendToSymipulsePanel(...)`
-- Gateway-side `isHeartbeat` injection: `src/gateway/server-chat.ts` —
-  added to broadcast payload when `heartbeatCtx?.isHeartbeat`
-
-**Test:** `parity.test.ts > I-6: isHeartbeat flag preserved through broadcast`
-
----
-
-### I-7. Status orb states
-
-**What:** A visual orb in the UI header transitions through six states
-based on agent lifecycle events: `STANDBY` (cyan), `WAITING` (amber),
-`PROCESSING` (violet), `RESPONDING` (green), `COMPLETE` (green, then
-reverts to `STANDBY` after 3s), `FAILED` (red).
-
-**Why it matters:** Without the orb, users have no peripheral signal of
-what the agent is doing. They have to look at the chat feed to see
-activity. The orb gives a glance-able status.
-
-**Source:**
-
-- `glass-ui/js/app.js` — `setAgentStatus(state)` function
-- Driven by: `state: "thinking"`, `state: "delta"`, `state: "final"`,
-  `state: "error"` events from gateway
-
-**Test:** `parity.test.ts > I-7: orb-driving events emit correct state values`
-
----
-
-### I-8. Watchdog timer
-
-**What:** A 300-second watchdog timer re-arms on every event from the
-gateway. If no event arrives within 300s, the orb transitions to FAILED
-and a notice appears in the feed.
-
-**Why it matters:** Long Nemotron reasoning phases used to look like a
-hung agent (master doc §26.A.3). The watchdog catches genuine hangs while
-allowing legitimate long pauses (when at least heartbeat events flow).
-
-**Source:**
-
-- `glass-ui/js/app.js:69` — watchdog interval
-- `glass-ui/js/app.js:278` — re-arm on event arrival
-- Per-profile `watchdogMs` from `ModelProfile.ui.watchdogMs`
-
-**Test:** `parity.test.ts > I-8: watchdog re-arms on every gateway event`
-
----
-
-### I-9. Per-WebSocket dedup
-
-**What:** Each browser WebSocket connection tracks `lastBroadcastSeq`. The
-gateway's `broadcastToBrowsers` drops frames with `seq <= lastBroadcastSeq`
-for that client. New events monotonically increment `seq`.
-
-**Why it matters:** When a tab reconnects after a brief network hiccup, it
-sees the full event history from the buffer. Without dedup, it would
-re-render every event since gateway start.
-
-**Source:**
-
-- `src/gateway/glass-ui-ws.ts:159-175` — `broadcastToBrowsers` with seq filter
-- `src/gateway/server-chat.ts` — monotonic `agentRunSeq` counter
-
-**Test:** `parity.test.ts > I-9: every emitted event has monotonic seq`
-
----
-
-### I-10. localStorage feed cache
-
-**What:** Last 200 messages cached in browser localStorage under
-`symi:feed-cache`. Restored on:
-
-- Server returns empty history (gateway restart edge case)
-- WebSocket disconnect with no history available
-- Page reload while gateway unreachable
-
-**Why it matters:** Master doc §27 documented the original feed-disappears
-bug. Without the cache, every gateway restart wipes the user's view.
-
-**Source:**
-
-- `glass-ui/js/app.js:22-54` — `feedCache` module (save/load/append/clear)
-- Populated from server history (bulk save), user sends (append), assistant
-  finals (append)
-
-**Test:** Manual — restart gateway, reload Glass UI, feed must restore from
-localStorage.
-
----
-
-### I-11. Profile event handling
-
-**What:** When the active model changes, the gateway broadcasts a `profile`
-event with the resolved `ModelProfile`. Glass UI updates:
-
-- Model badge (text label like "Claude" / "Spider" / "CoreWeave")
-- `WATCHDOG_MS` (per-profile timeout)
-- `armWatchdogOnSend` flag (whether watchdog starts on user send vs first
-  delta)
-- `stripPatterns` for client-side rendering
-
-**Why it matters:** Different models have different streaming behaviour.
-Without profile updates, watchdog timing is wrong and stripping is
-incorrect.
-
-**Source:**
-
-- `src/gateway/server-reload-handlers.ts` — profile broadcast on model change
-- `glass-ui/js/gateway.js` — profile event listener
-- `glass-ui/js/app.js` — `window.activeModelProfile` consumer
-
-**Test:** `parity.test.ts > I-11: profile event payload shape preserved`
-
----
-
-### I-12. Tool-call view in transcript modal
-
-**What:** History drawer's transcript modal renders `tool_use` and
-`tool_result` content blocks inline with chat. `tool_use` shows the tool
-name + truncated input JSON; `tool_result` shows ok/error status +
-truncated output. Toggleable via "Show tool calls" button (Stage 2.8.17).
-
-**Why it matters:** Debugging "what did the agent actually do during this
-session?" — without tool-call view, the user only sees text replies.
-Sub-agent and cron review depend on this.
-
-**Source:**
-
-- `glass-ui/js/history.js:loadTranscript` + `renderTranscript` (split for
-  toggle re-render)
-- `glass-ui/js/history.js:extractToolBlocks` + `renderToolBlock`
-- Master doc §46.4
-
-**Test:** `parity.test.ts > I-12: tool_use and tool_result blocks preserved
-as structured objects in events (not flattened to strings)`
-
----
-
-### I-13. History drawer + cross-session search
-
-**What:** Persistent debounced search input above the history drawer.
-Stage 2.8.17 added cross-session content search via `sessions.search` RPC.
-Results show `<mark>`-highlighted snippets, click to open transcript modal,
-"Load Session" to adopt.
-
-**Why it matters:** Without it, users can't find prior conversations. The
-history drawer is one of Glass UI's signature features.
-
-**Source:**
-
-- `glass-ui/js/history.js` — search box, `runSessionSearch`,
-  `renderSearchMatch`, `highlightSnippet`
-- `src/gateway/server-methods/sessions.ts` — `sessions.search` RPC
-- Master doc §46.3
-
-**Test:** Server-side `sessions.search` RPC test exists; client-side is
-manual (visual regression).
-
----
-
-### I-14. Reply-tag stripping in renderer input
-
-**What:** `[[reply_to_current]]` and `[[reply_to:<id>]]` tags are stripped
-gateway-side before reaching the renderer. The renderer assumes its input
-is clean.
-
-**Why it matters:** Without stripping, literal `[[reply_to_current]]` text
-appears in every assistant message. Stage 1.2 verified the parser is
-already tolerant; Phase 1-6 must not inadvertently route un-stripped text
-to Glass UI.
-
-**Source:**
-
-- Gateway: `src/utils/directive-tags.ts:parseInlineDirectives` (tolerant
-  parser)
-- Gateway: applied in `src/auto-reply/reply/reply-payloads.ts`
-- Renderer: assumes clean input — no stripping in `glass-ui/js/render.js`
-
-**Test:** `parity.test.ts > I-14: events delivered to renderer never
-contain reply tags`
-
----
-
-### I-15. Active subagent count
-
-**What:** Final events carry `activeSubagentCount: number`. Glass UI shows
-"N subagents active" indicator when > 0.
-
-**Why it matters:** Sub-agent fan-out (master doc §19) means the user
-might have multiple agents working in parallel. Without the indicator,
-fan-out is invisible.
-
-**Source:**
-
-- Gateway: `src/gateway/server-chat.ts:346` — `countActiveRunsForSession`
-  injected into final payload
-- Glass UI: reads `payload.activeSubagentCount` in final-event handler
-
-**Test:** `parity.test.ts > I-15: activeSubagentCount preserved on final events`
-
----
-
-## How this doc is consumed
-
-### During Phase 1 (outbound filter lift)
-
-Pre-merge check: zero invariants affected. The filter moves up one layer;
-the renderer's input is unchanged.
-
-### During Phase 2 (universal event stream)
-
-Pre-merge check: every invariant must have a corresponding field in the
-universal `AgentEvent` shape. Specifically:
-
-- `delta` field must remain incremental (I-1)
-- `state` field must drive the orb (I-7)
-- `isHeartbeat` field must survive the universal-shape redesign (I-6)
-- `seq` field must stay monotonic (I-9)
-- `activeSubagentCount` field must be preserved (I-15)
-- `message.content[]` must preserve structured tool blocks (I-12)
-- `stream: "thinking"` must remain a distinct event type (I-5)
-
-If the universal shape design omits any of these, redesign before merging.
-
-### During Phase 3-4 (dispatch chain collapse + RPC merge)
-
-Pre-merge check: the renderer's input shape doesn't depend on the dispatch
-chain at all — events come from `emitChatDelta`/`emitChatFinal`, which sit
-outside the chain. Verify these functions still produce the same shapes
-post-collapse.
-
-### During Phase 5 (surface adapters)
-
-**This is the highest-risk phase for rendering regression.** Pre-merge
-checks:
-
-1. `glass-ui/js/render.js` is **untouched**.
-2. `glass-ui/js/app.js` rendering functions (`openStreamBubble`,
-   `updateStream`, `closeStreamBubble`, `renderMessage`) are **untouched**.
-3. `glass-ui/js/history.js`, `symipulse.js`, marked.js / hljs config are
-   **untouched**.
-4. The only file that may be refactored is `glass-ui/js/gateway.js` —
-   the WebSocket subscription layer. It must convert the universal
-   `AgentEvent` to the legacy `chat`/`agent` event shapes the rest of the
-   browser code consumes.
-5. Run the full parity test suite + manual visual smoke test before merge.
-
-### During Phase 6 (cleanup)
-
-Re-run parity tests on the final shape. Confirm no invariants regressed
-through the series.
-
----
-
-## Manual visual regression checklist
-
-In addition to automated parity tests, every Phase 1-6 PR must include a
-manual smoke test of the 15 properties before merge:
-
-- [ ] **I-1, I-2** — Send a multi-paragraph response. Watch tokens stream
-      in token-by-token. On final, confirm rendered markdown replaces plain
-      text.
-- [ ] **I-3, I-4** — Send a code-task message. Confirm code block has
-      syntax highlighting (colored keywords, strings, comments).
-- [ ] **I-5** — Switch to a reasoning model (Qwen 3.6 / Claude with
-      thinking). Send a math problem. Confirm thinking content appears in the
-      Reasoning panel, not the main chat feed.
-- [ ] **I-6** — Wait for a Symipulse heartbeat (or trigger one). Confirm
-      it appears in the Symipulse panel, not the main feed.
-- [ ] **I-7** — Send a message. Watch orb transition: STANDBY → WAITING
-      → PROCESSING → RESPONDING → COMPLETE → STANDBY.
-- [ ] **I-8** — Trigger a 300s+ stuck run (e.g., model with no response).
-      Confirm watchdog fires FAILED state.
-- [ ] **I-9** — Open Glass UI in two tabs. Send a message. Confirm both
-      tabs receive each event exactly once.
-- [ ] **I-10** — Restart gateway. Confirm feed restores from localStorage.
-- [ ] **I-11** — Switch model via UI dropdown. Confirm model badge
-      updates immediately.
-- [ ] **I-12** — Open history drawer → click a session → toggle "Show
-      tool calls". Confirm tool_use blocks render with name + input JSON.
-- [ ] **I-13** — Open history drawer → type in search box. Confirm
-      cross-session matches with `<mark>` highlighting.
-- [ ] **I-14** — Trigger a reply-quote scenario. Confirm no literal
-      `[[reply_to_current]]` text in any assistant message.
-- [ ] **I-15** — Spawn a sub-agent task. Confirm subagent indicator shows
-      N>0 during fan-out.
-
----
-
-## Revisions
-
-| Date                       | Phase   | Change                                          |
-| -------------------------- | ------- | ----------------------------------------------- |
-| 2026-04-29                 | Phase 0 | Initial authorship — 15 invariants, parity test |
-| skeleton, visual checklist |

package/docs/help/debugging.md

@@ -1,162 +0,0 @@
----
-summary: "Debugging tools: watch mode, raw model streams, and tracing reasoning leakage"
-read_when:
-  - You need to inspect raw model output for reasoning leakage
-  - You want to run the Gateway in watch mode while iterating
-  - You need a repeatable debugging workflow
-title: "Debugging"
----
-
-# Debugging
-
-This page covers debugging helpers for streaming output, especially when a
-provider mixes reasoning into normal text.
-
-## Runtime debug overrides
-
-Use `/debug` in chat to set **runtime-only** config overrides (memory, not disk).
-`/debug` is disabled by default; enable with `commands.debug: true`.
-This is handy when you need to toggle obscure settings without editing `symi.json`.
-
-Examples:
-
-```
-/debug show
-/debug set messages.responsePrefix="[symi]"
-/debug unset messages.responsePrefix
-/debug reset
-```
-
-`/debug reset` clears all overrides and returns to the on-disk config.
-
-## Gateway watch mode
-
-For fast iteration, run the gateway under the file watcher:
-
-```bash
-pnpm gateway:watch
-```
-
-This maps to:
-
-```bash
-node --watch-path src --watch-path tsconfig.json --watch-path package.json --watch-preserve-output scripts/run-node.mjs gateway --force
-```
-
-Add any gateway CLI flags after `gateway:watch` and they will be passed through
-on each restart.
-
-## Dev profile + dev gateway (--dev)
-
-Use the dev profile to isolate state and spin up a safe, disposable setup for
-debugging. There are **two** `--dev` flags:
-
-- **Global `--dev` (profile):** isolates state under `~/.symi-dev` and
-  defaults the gateway port to `19001` (derived ports shift with it).
-- **`gateway --dev`: tells the Gateway to auto-create a default config +
-  workspace** when missing (and skip BOOTSTRAP.md).
-
-Recommended flow (dev profile + dev bootstrap):
-
-```bash
-pnpm gateway:dev
-SYMI_PROFILE=dev symi tui
-```
-
-If you don’t have a global install yet, run the CLI via `pnpm symi ...`.
-
-What this does:
-
-1. **Profile isolation** (global `--dev`)
-   - `SYMI_PROFILE=dev`
-   - `SYMI_STATE_DIR=~/.symi-dev`
-   - `SYMI_CONFIG_PATH=~/.symi-dev/symi.json`
-   - `SYMI_GATEWAY_PORT=19001` (browser/canvas shift accordingly)
-
-2. **Dev bootstrap** (`gateway --dev`)
-   - Writes a minimal config if missing (`gateway.mode=local`, bind loopback).
-   - Sets `agent.workspace` to the dev workspace.
-   - Sets `agent.skipBootstrap=true` (no BOOTSTRAP.md).
-   - Seeds the workspace files if missing:
-     `AGENTS.md`, `SYMICORE.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `SYMIPULSE.md`.
-   - Default identity: **C3‑PO** (protocol droid).
-   - Skips channel providers in dev mode (`SYMI_SKIP_CHANNELS=1`).
-
-Reset flow (fresh start):
-
-```bash
-pnpm gateway:dev:reset
-```
-
-Note: `--dev` is a **global** profile flag and gets eaten by some runners.
-If you need to spell it out, use the env var form:
-
-```bash
-SYMI_PROFILE=dev symi gateway --dev --reset
-```
-
-`--reset` wipes config, credentials, sessions, and the dev workspace (using
-`trash`, not `rm`), then recreates the default dev setup.
-
-Tip: if a non‑dev gateway is already running (launchd/systemd), stop it first:
-
-```bash
-symi gateway stop
-```
-
-## Raw stream logging (Symi)
-
-Symi can log the **raw assistant stream** before any filtering/formatting.
-This is the best way to see whether reasoning is arriving as plain text deltas
-(or as separate thinking blocks).
-
-Enable it via CLI:
-
-```bash
-pnpm gateway:watch --raw-stream
-```
-
-Optional path override:
-
-```bash
-pnpm gateway:watch --raw-stream --raw-stream-path ~/.symi/logs/raw-stream.jsonl
-```
-
-Equivalent env vars:
-
-```bash
-SYMI_RAW_STREAM=1
-SYMI_RAW_STREAM_PATH=~/.symi/logs/raw-stream.jsonl
-```
-
-Default file:
-
-`~/.symi/logs/raw-stream.jsonl`
-
-## Raw chunk logging (pi-mono)
-
-To capture **raw OpenAI-compat chunks** before they are parsed into blocks,
-pi-mono exposes a separate logger:
-
-```bash
-PI_RAW_STREAM=1
-```
-
-Optional path:
-
-```bash
-PI_RAW_STREAM_PATH=~/.pi-mono/logs/raw-openai-completions.jsonl
-```
-
-Default file:
-
-`~/.pi-mono/logs/raw-openai-completions.jsonl`
-
-> Note: this is only emitted by processes using pi-mono’s
-> `openai-completions` provider.
-
-## Safety notes
-
-- Raw stream logs can include full prompts, tool output, and user data.
-- Keep logs local and delete them after debugging.
-- If you share logs, scrub secrets and PII first.