omnius 1.0.591 → 1.0.592

package/docs/trajectory-grounding.md

@@ -0,0 +1,160 @@
+# Trajectory Grounding
+
+## Purpose
+
+Trajectory Grounding gives a running agent a compact, evidence-bound account of
+where a project stands, what changed, what remains uncertain, and the safest
+next action. Before the first main-agent request, Omnius asks the selected model
+to form a specific, structured orientation from that evidence. This is a
+control-plane checkpoint, not free-form self-talk and not a request to reveal
+private reasoning.
+
+The feature exists to prevent loops where changing a payload appears to be
+progress even though the underlying prerequisite has not changed. The
+myactuator repeated full-file write incident is the reference failure mode:
+an existing target rejected unguarded `file_write` calls while generic recovery
+guidance kept steering toward another full write.
+
+## Invariants
+
+- Every completed, failed, file-state, or verifier claim must point to a
+  sanitized evidence reference: a tool result, file hash, verifier outcome, or
+  child-agent result.
+- Unknown or stale state is rendered as an open question, never as completed
+  work.
+- One current checkpoint is regenerated into the active context frame. It must
+  not accumulate as independent system messages.
+- The first active frame always attempts a tool-less model grounding pass. Its
+  concise situation assessment must cite supplied evidence labels and be visible
+  in the live trajectory box; if inference is unavailable, the UI labels the
+  deterministic safety orientation rather than pretending it was generative.
+- Focus-supervisor and explicit safety contracts override generic trajectory
+  suggestions.
+- The model must use the checkpoint to choose a next tool action, but must not
+  echo it or generate a reasoning transcript for the user.
+- Child agents receive only a scoped slice: parent goal, delegated scope,
+  relevant constraint, and exit evidence requirement.
+
+## Data contract
+
+```ts
+type TrajectoryAssessment =
+  | "on_trajectory"
+  | "recovery_required"
+  | "verification_due"
+  | "blocked";
+
+interface TrajectoryCheckpoint {
+  schemaVersion: 1;
+  id: string;
+  revision: number;
+  turn: number;
+  trigger: string;
+  assessment: TrajectoryAssessment;
+  goal: string;
+  currentStep?: string;
+  situationAssessment?: string;
+  groundingSource?: "model" | "deterministic";
+  groundingEvidenceRefs?: string[];
+  completedWork: string[];
+  groundedFacts: Array<{
+    statement: string;
+    evidence: string;
+    freshness: "fresh" | "stale" | "unknown";
+  }>;
+  openQuestions: string[];
+  nextAction: string;
+  successEvidence: string;
+  doNotRepeat: string[];
+}
+```
+
+The model-facing rendering is capped and follows this form:
+
+```text
+[TRAJECTORY CHECKPOINT]
+Goal: ...
+Assessment: recovery_required
+Reasoned situation: the full-file write was rejected and there is no fresh read
+of the target, so a changed payload would not resolve the actual prerequisite.
+Grounded facts:
+- [turn 18/tool_result] file_write on pal.cpp was blocked: existing target
+  lacks fresh overwrite/hash evidence.
+Open question: current pal.cpp bytes have not been freshly read.
+Required next action: file_read pal.cpp, then use file_edit/file_patch.
+Success evidence: a targeted mutation or explicit blocker evidence.
+Do not repeat: file_write pal.cpp with a changed full-file payload.
+```
+
+## Integration path
+
+1. `packages/orchestrator/src/trajectory-checkpoint.ts` owns pure types,
+   deterministic safety assessment, model-output validation/merging, rendering,
+   fingerprints, and scoped child slices.
+2. `AgenticRunner` records bounded sanitized observations for both executed
+   tools and synthetic/preflight blocks. It runs the bounded tool-less
+   grounding request before the first main frame, then rebuilds checkpoints in
+   `_buildTurnContextFrame()`, which serves normal and brute-force loops.
+3. `ContextFabric` receives a first-class `trajectory_checkpoint` signal with
+   high priority, one-turn TTL, and a single semantic conflict group.
+4. `context-compiler` recognizes the block for last-surface deduplication.
+5. Tier prompts tell the model to reconcile its next tool call against the
+   checkpoint without narrating it.
+6. The runner emits a typed trajectory event. TUI, API events, debug artifacts,
+   session handoffs, and child prompts consume bounded projections of it.
+7. Large whole-file reads hand their isolated extraction branch an agentic
+   request built from the current trajectory, active read action, and trigger
+   evidence. The branch receives the specific unresolved question and return
+   contract, never a raw copy of the original user prompt as its query.
+
+## Trigger policy
+
+The checkpoint is recomputed before each model request, but only emits a new
+revision when content or direction changes. The model grounding pass runs at
+task start and, in adaptive mode, after material failures, safety-direction
+changes, compaction, or user steering; ordinary successful reads do not create
+an extra planning call. Important triggers are task start, tool failure,
+synthetic safety block, mutation, verifier result, child result, compaction,
+user steering, focus-directive change, and adaptive cadence.
+
+Use total tool-call count for cadence rather than a loop-local turn number so
+brute-force re-engagement cannot reset the schedule.
+
+## Exposure
+
+- **Model context:** a first-class active-frame signal placed after goal/user
+  steering and before generic guidance.
+- **TUI:** a compact, collapsible main-view dynamic block and a direction detail
+  in the live stage/footer; show updates only when the revision changes.
+- **Sub-agents:** a scoped, non-authoritative parent slice in their prompt.
+- **API/debug:** typed event plus debug artifact on direction change.
+- **Handoff/session diary:** persist the latest meaningful checkpoint only at
+  compaction, direction changes, handoff, or completion. Restored checkpoints
+  are orientation, not fresh file evidence.
+
+## Safety and rollout
+
+`OMNIUS_TRAJECTORY_CHECKPOINT=off|shadow|adaptive|always` controls checkpoint
+visibility and cadence. `OMNIUS_TRAJECTORY_GROUNDING=off|initial|adaptive`
+controls model orientation: `initial` always grounds the first main-agent
+request; `adaptive` also regrounds material direction changes. The main TUI
+uses `adaptive`; child agents receive the parent slice rather than recursively
+running a second grounding pass.
+`shadow` computes and records checkpoints without model injection. `adaptive`
+is the default target: inject on meaningful state changes and bounded cadence.
+
+No mandatory `trajectory_update` tool is introduced. A tool would create
+ceremony and failure loops for smaller models. Existing todo/workboard state is
+updated only when the trajectory materially changes.
+
+## Acceptance criteria
+
+- Changed-payload full-write retries produce `recovery_required`, a fresh read
+  prerequisite, and a no-repeat constraint.
+- Cached/replayed reads do not satisfy a fresh-read prerequisite.
+- Child completion yields verification work, not an unsupported completion
+  claim.
+- Context contains at most one current checkpoint and compaction retains its
+  current evidence references.
+- TUI/API/debug views all agree on checkpoint revision and assessment.
+- The feature is bounded, sanitized, and disabled cleanly by configuration.

package/docs/voice-flow-architecture.md

@@ -0,0 +1,489 @@
+# Voice TTS Flow — Architecture & Implementation Guide
+
+This document describes the voice synthesis system in omnius: how it's built,
+how all the pieces connect, and how to add new voice features following the same patterns.
+
+---
+
+## System Overview
+
+```
+┌─────────────┐     ┌──────────────────┐     ┌──────────────┐
+│ interactive  │────▶│   VoiceEngine    │────▶│  System Audio │
+│    .ts       │     │    voice.ts      │     │  (afplay/     │
+│              │     │                  │     │   paplay)     │
+│  ┌────────┐  │     │  ┌────────────┐  │     └──────────────┘
+│  │ Agent  │──┤     │  │ ONNX Model │  │
+│  │ Events │  │     │  │  Session   │  │     ┌──────────────┐
+│  └────────┘  │     │  └────────────┘  │────▶│ WebSocket    │
+│              │     │  ┌────────────┐  │     │  Clients     │
+│  ┌────────┐  │     │  │ MLX Audio  │  │     │ (voice-      │
+│  │Emotion │──┤     │  │  Backend   │  │     │  session.ts) │
+│  │Context │  │     │  └────────────┘  │     └──────────────┘
+│  └────────┘  │     └──────────────────┘
+│              │              ▲
+│  ┌────────┐  │              │
+│  │Narrate │──┘     ┌────────┴────────┐
+│  │ Engine │        │  Model Registry │
+│  └────────┘        │  glados/over-   │
+└─────────────┘      │  watch/kokoro   │
+                     └─────────────────┘
+```
+
+## File Map
+
+| File | Purpose | Lines |
+|------|---------|-------|
+| `packages/cli/src/tui/voice.ts` | VoiceEngine class + narration engine | ~2260 |
+| `packages/cli/src/tui/voice-session.ts` | WebSocket streaming + cloudflared tunnel | ~885 |
+| `packages/cli/src/tui/interactive.ts` | Wiring: instantiation, events, narration calls | ~2300 |
+| `packages/cli/src/tui/commands.ts` | `/voice` command handler | ~1850 |
+| `packages/cli/src/tui/render.ts` | `renderInfo`/`renderWarning` + content-write hook | ~670 |
+| `packages/cli/tests/voice-narration.test.ts` | Narration unit tests | varies |
+| `packages/cli/tests/voice-session.test.ts` | Session unit tests | varies |
+
+---
+
+## 1. Model Registry
+
+All voice models are defined in `VOICE_MODELS` (voice.ts, top of file):
+
+```typescript
+const VOICE_MODELS: Record<string, VoiceModel> = {
+  glados:   { id: "glados",   backend: "onnx", onnxUrl: "...", configUrl: "..." },
+  overwatch:{ id: "overwatch", backend: "onnx", onnxUrl: "...", configUrl: "..." },
+  kokoro:   { id: "kokoro",   backend: "mlx",  mlxModelId: "mlx-community/Kokoro-82M-bf16", mlxVoice: "af_heart" },
+  // ... more kokoro voices
+};
+```
+
+**To add a new model:**
+
+1. Add an entry to `VOICE_MODELS` with the appropriate `backend` field
+2. For ONNX: provide `onnxUrl` + `configUrl` (Piper ONNX format)
+3. For MLX: provide `mlxModelId` (Hugging Face repo) + `mlxVoice` + `mlxLangCode`
+4. The model is immediately available via `/voice <id>`
+
+---
+
+## 2. VoiceEngine Lifecycle
+
+### Instantiation
+
+A single `VoiceEngine` instance is created in `interactive.ts:1359`:
+
+```typescript
+const voiceEngine = new VoiceEngine();
+```
+
+### Startup
+
+If the user previously enabled voice (persisted in settings):
+
+```typescript
+if (savedSettings.voice) {
+  voiceEngine.toggle();                          // Enable TTS
+  if (savedSettings.voiceModel)
+    voiceEngine.setModel(savedSettings.voiceModel); // Load specific model
+}
+```
+
+### toggle() Flow
+
+```
+toggle()
+  ├─ If disabling: set enabled=false, killPlayback(), done
+  └─ If enabling:
+       ├─ MLX model?
+       │    └─ ensureMlxAudio()          → pip install mlx-audio
+       └─ ONNX model?
+            ├─ ensureRuntime()           → npm install onnxruntime-node + phonemizer
+            ├─ ensureModel(id)           → download .onnx + .json from GitHub
+            └─ loadSession()             → create InferenceSession
+       set enabled=true, ready=true
+```
+
+### setModel(id) Flow
+
+```
+setModel(id)
+  ├─ Validate id exists in VOICE_MODELS
+  ├─ Reset: session=null, config=null, ready=false
+  └─ If currently enabled:
+       └─ Load the new model (same as toggle enable path)
+```
+
+### Shutdown
+
+```typescript
+voiceEngine.dispose();  // in interactive.ts cleanup
+```
+
+---
+
+## 3. Synthesis Pipeline
+
+### ONNX Path (glados, overwatch)
+
+```
+text
+ │
+ ├── chunkText()           Split >200 char text on newlines + sentence boundaries
+ │
+ ├── For each chunk:
+ │    ├── textToPhonemes()  espeak-ng WASM phonemization
+ │    ├── phonemesToIds()   Map phonemes → integer IDs via config.phoneme_id_map
+ │    │                     Format: BOS → PAD → (phoneme + PAD)* → EOS
+ │    ├── Build tensors:    input (int64), input_lengths (int64), scales (float32)
+ │    ├── session.run()     ONNX inference → Float32 audio samples
+ │    └── Concatenate with 180ms silence gaps between sentences
+ │
+ ├── Apply volume scaling   (0.0–1.0 multiplier per sample)
+ ├── Apply pitch shift      (linear-interpolation resampling)
+ │
+ ├── Stream PCM to WebSocket clients (if onPCMOutput wired)
+ ├── Write WAV to temp file
+ ├── Play via system command (afplay / paplay / pw-play / aplay)
+ └── Delete temp file
+```
+
+### MLX Path (kokoro, kokoro:af_heart, etc.)
+
+```
+text
+ │
+ ├── Clean markdown (* removal)
+ ├── Build Python command:
+ │     python3 -c "from mlx_audio.tts import generate; generate.main([...])"
+ │     --model mlx-community/Kokoro-82M-bf16
+ │     --text "..."
+ │     --voice af_heart
+ │     --lang_code a
+ │     --audio_path /tmp/omnius-mlx-{ts}.wav
+ │
+ ├── execSync() with 60s timeout
+ │    (fallback: python3 -m mlx_audio.tts.generate CLI)
+ │
+ ├── Apply volume scaling (rewrite WAV PCM samples)
+ ├── Stream PCM to WebSocket clients (parse WAV header for sample rate)
+ ├── Play via system command
+ └── Delete temp file
+```
+
+---
+
+## 4. Narration Engine
+
+The narration system generates context-aware spoken descriptions of agent activity.
+It lives in voice.ts below the VoiceEngine class (~line 1268+).
+
+### Personality Levels
+
+```
+1 = minimal   "Reading file.ts"
+2 = brief      "Reading file.ts"
+3 = conv       "Let me take a look at file.ts"
+4 = chatty     "Alright, let's crack open file.ts"
+5 = theatrical "Alright, let's crack open file.ts and see what we're working with"
+```
+
+Mapped from the agent's personality preset:
+```typescript
+{ concise: 1, balanced: 3, verbose: 4, pedagogical: 5 }
+```
+
+### describeToolCall(toolName, args, level, emotion)
+
+Called from `interactive.ts` on every `tool_call` event:
+
+```typescript
+if (voice?.enabled) {
+  const desc = describeToolCall(event.toolName, event.toolArgs, vLevel, emoCtx);
+  voice.speakSubordinate(desc, emoCtx);  // 55% volume, 0.92x pitch
+}
+```
+
+**Variant pools** — each tool has 3 tiers of phrasings:
+- `FILE_READ_VARIANTS.terse` / `.conv` / `.chatty`
+- `FILE_WRITE_VARIANTS`, `FILE_EDIT_VARIANTS`, `GREP_VARIANTS`, etc.
+- `SHELL_VARIANTS` — categorized by shell command type (git, npm, test, etc.)
+
+**Context modifiers** applied based on narration state:
+- After errors: prefix with "Okay, " or "Right, "
+- Same file again: "Back to " or "Still working on "
+- Revisiting a file: "Coming back to " or "Revisiting "
+- Progress beats every 8 tools: "Making good progress. "
+
+### describeToolResult(toolName, success, level, content, emotion)
+
+Called on every `tool_result` event. Generates success/failure descriptions:
+- Success: "Got it", "Done", "Found it"
+- Failure: "That didn't work", "Hit a snag"
+
+**Content-aware extraction** via `extractResultDigest()`:
+- ETH balances, test results, error messages, wallet addresses, file paths
+
+### describeTaskComplete(summary, complete, level)
+
+Announces when the agent finishes a task:
+- "Task complete", "Got it done", "All set"
+
+### Narration State
+
+```typescript
+interface NarrationState {
+  toolCount: number;              // Total tools this session
+  toolCounts: Record<string, number>; // Per-tool counts
+  consecutiveErrors: number;      // Reset on success
+  totalErrors: number;
+  lastTool: string;
+  lastFile: string;
+  filesSeen: Set<string>;         // All files visited
+  lastVariantIdx: Record<string, number>; // Avoid repeats
+  lastResultDigest: string;       // Last tool result summary
+}
+```
+
+`resetNarrationContext()` is called at the start of each task.
+
+### pick(key, variants)
+
+Selects a random variant from a pool, avoiding the last-used index for that key.
+Ensures you never hear the same phrasing twice in a row.
+
+---
+
+## 5. Emotion Modulation
+
+The emotion engine provides valence-arousal context to the voice:
+
+```typescript
+interface VoiceEmotionContext {
+  valence: number;  // -1 (sad) to +1 (happy)
+  arousal: number;  // 0 (calm) to 1 (activated)
+  label: string;    // "excited", "focused", etc.
+  emoji: string;
+}
+```
+
+### Pitch Bias
+
+`emotionToPitchBias(emotion)` converts emotion to a pitch adjustment:
+
+```
+pitch_bias = valence × 0.6 + (arousal - 0.5) × 0.4
+clamped to [-0.10, +0.10]
+```
+
+- **Excited** (high valence + high arousal) → voice pitch rises
+- **Dejected** (low valence + low arousal) → voice pitch drops
+
+Applied in `speak()` and `speakSubordinate()`:
+
+```typescript
+speak(text, emotion):    pitchFactor = 1.0 + pitchBias
+speakSubordinate(text):  pitchFactor = 0.92 + pitchBias  (lower base pitch)
+```
+
+### Emotion Coloring
+
+At personality >= 3, ~30% of narrations get emotion-colored prefixes:
+- Excited: "Feeling good about this"
+- Stressed: "Pushing through"
+- Calm: "Nice and steady"
+- Subdued: "Being careful here"
+
+---
+
+## 6. Queue & Playback
+
+### Speech Queue
+
+```typescript
+private speakQueue: SpeakItem[] = [];
+```
+
+Items are queued FIFO. `drainQueue()` processes them sequentially:
+1. Pop item from front
+2. Synthesize + play to completion
+3. 250ms silence gap
+4. Next item
+
+Queue overflow protection: if > 30 items backed up, clear the queue.
+
+### Volume Levels
+
+```
+speak()             → volume 1.0   (full)
+speakSubordinate()  → volume 0.55  (reduced for tool narration)
+```
+
+### Playback
+
+WAV temp file → system audio command:
+- macOS: `afplay`
+- Linux: `paplay` → `pw-play` → `aplay` (tries in order)
+- Windows: PowerShell `Media.SoundPlayer`
+
+15-second safety timeout per playback. `killPlayback()` sends SIGTERM.
+
+---
+
+## 7. WebSocket Voice Session
+
+`VoiceSession` (voice-session.ts) enables real-time voice interaction via browser:
+
+```
+Browser Client  ←──WebSocket──→  VoiceSession  ←──PCM──→  VoiceEngine
+    (mic)        binary PCM        (server)     onPCMOutput   (TTS)
+    (speaker)    binary PCM
+```
+
+### Setup
+
+```
+start()
+  ├── Start HTTP server (serves HTML single-page app)
+  ├── Start WebSocket server (ws)
+  ├── Launch cloudflared tunnel for public URL
+  └── Wire VoiceEngine.onPCMOutput → broadcast to all clients
+```
+
+### Audio Format
+
+- 16kHz, 16-bit, mono PCM (Int16Array)
+- Binary WebSocket frames
+- Echo cancellation: suppress mic input while TTS is playing
+
+### Frontend
+
+Embedded HTML with:
+- Braille waveform animator (color-coded: idle/listening/speaking)
+- WebAudio API for mic capture + speaker playback
+- Transcript view (user + agent messages)
+- Start/stop mic button
+
+---
+
+## 8. Settings Persistence
+
+Voice settings are saved per-project or globally via `resolveSettings()`:
+
+```typescript
+{
+  voice: boolean,       // Enabled/disabled
+  voiceModel: string    // Model ID ("glados", "kokoro:af_heart", etc.)
+}
+```
+
+The `/voice` command handler in commands.ts:
+```typescript
+case "voice":
+  if (arg) {
+    ctx.voiceSetModel(arg);     // /voice kokoro
+    save({ voice: true, voiceModel: arg });
+  } else {
+    ctx.voiceToggle();          // /voice (toggle on/off)
+    save({ voice: isOn });
+  }
+```
+
+---
+
+## 9. Content-Write Hook (TUI Safety)
+
+Voice operations can trigger `renderInfo()` / `renderWarning()` messages asynchronously.
+To prevent these from overwriting the TUI input area, a global content-write hook
+brackets all render calls with scroll-region management:
+
+```typescript
+// render.ts
+export function renderInfo(message: string): void {
+  _contentWriteHook?.begin();  // → statusBar.beginContentWrite()
+  process.stdout.write(`ℹ ${message}\n`);
+  _contentWriteHook?.end();    // → statusBar.endContentWrite()
+}
+```
+
+Registered once in interactive.ts after StatusBar activation:
+```typescript
+setContentWriteHook({
+  begin: () => statusBar.beginContentWrite(),
+  end: () => statusBar.endContentWrite(),
+});
+```
+
+**Rule:** Never use raw `process.stdout.write()` in voice.ts for user-facing messages.
+Always use `renderInfo()` / `renderWarning()` / `renderError()` so the content hook
+routes output to the scroll region above the status bar and input area.
+
+---
+
+## 10. How to Add a New Voice Feature
+
+### Adding a New ONNX Voice Model
+
+1. Host the `.onnx` + `.onnx.json` files (Piper format) on a public URL
+2. Add entry to `VOICE_MODELS`:
+   ```typescript
+   myvoice: {
+     id: "myvoice", label: "My Voice", backend: "onnx",
+     onnxUrl: "https://...", configUrl: "https://...",
+   },
+   ```
+3. Done. `/voice myvoice` works immediately.
+
+### Adding a New MLX Voice
+
+1. Find the Hugging Face model ID (must be MLX-compatible)
+2. Add entry to `VOICE_MODELS`:
+   ```typescript
+   "newmodel:voice_name": {
+     id: "newmodel:voice_name", label: "New Model (MLX)", backend: "mlx",
+     mlxModelId: "mlx-community/NewModel", mlxVoice: "voice_name", mlxLangCode: "a",
+     onnxUrl: "", configUrl: "",
+   },
+   ```
+3. Done. `/voice newmodel:voice_name` works on macOS Apple Silicon.
+
+### Adding a New TTS Backend
+
+1. Add a new `backend` value to the `VoiceModel` interface
+2. Add a method like `ensureNewBackend()` for installation
+3. Add a method like `synthesizeWithNewBackend()` for synthesis
+4. Gate in `toggle()`, `setModel()`, `synthesizeAndPlay()`, `synthesizeToBuffer()`, `synthesizeToPCM()`
+5. Follow the pattern: the backend outputs a WAV file, then existing `playWav()` handles playback
+
+### Adding New Narration Variants
+
+1. Find the relevant variant pool (e.g., `FILE_READ_VARIANTS`)
+2. Add new strings to the `terse`, `conv`, and `chatty` tiers
+3. The `pick()` function automatically rotates through variants
+
+### Adding Emotion-Aware Features
+
+1. Receive `VoiceEmotionContext` from the emotion engine
+2. Use `emotionToPitchBias()` for pitch modulation
+3. Use `emotionColor()` for spoken prefixes
+4. Valence drives tone (happy/sad), arousal drives energy (calm/activated)
+
+---
+
+## Quick Reference: Key Functions
+
+| Function | Location | Purpose |
+|----------|----------|---------|
+| `VoiceEngine.toggle()` | voice.ts | Enable/disable TTS |
+| `VoiceEngine.speak()` | voice.ts | Queue speech (full volume) |
+| `VoiceEngine.speakSubordinate()` | voice.ts | Queue speech (55% volume) |
+| `VoiceEngine.synthesizeAndPlay()` | voice.ts | Core synthesis + playback |
+| `VoiceEngine.synthesizeWithMlx()` | voice.ts | MLX backend synthesis |
+| `VoiceEngine.ensureRuntime()` | voice.ts | Install ONNX runtime |
+| `VoiceEngine.ensureMlxAudio()` | voice.ts | Install mlx-audio pip package |
+| `describeToolCall()` | voice.ts | Generate tool narration text |
+| `describeToolResult()` | voice.ts | Generate result narration text |
+| `describeTaskComplete()` | voice.ts | Generate completion narration |
+| `emotionToPitchBias()` | voice.ts | Emotion → pitch modulation |
+| `pick()` | voice.ts | Random variant selection (no repeats) |
+| `setContentWriteHook()` | render.ts | Register TUI scroll-region safety |
+| `VoiceSession.start()` | voice-session.ts | Start WebSocket voice server |