open-agents-ai 0.103.61 → 0.103.64

package/README.md

@@ -56,7 +56,7 @@ The agent uses tools autonomously in a loop — reading errors, fixing code, and
 - **Live Voice Session** — `/listen` with `/voice` enabled spawns a cloudflared tunnel with a real-time WebSocket audio endpoint. A floating presence UI shows live transcription, connected users, and audio visualization. Echo cancellation prevents TTS feedback loops
 - **Call Sub-Agent** — each WebSocket caller gets a dedicated AgenticRunner for low-latency voice-to-voice loops, with admin/public access tiers and bidirectional activity sharing with the main agent
 - **Telegram Voice** — `/voice` enabled via Telegram forwards TTS audio as voice messages alongside text responses. Incoming voice messages are auto-transcribed and handled as text
-- **Neural TTS** — hear what the agent is doing via GLaDOS or Overwatch ONNX voices, with personality-driven expressiveness
+- **Neural TTS** — hear what the agent is doing via GLaDOS, Overwatch, Kokoro, or LuxTTS voice clone, with literature-grounded narration engine (sNeuron-TST structure rotation, Moshi ring buffer dedup, UDDETTS emotion-driven prosody, SEST metadata, LuxTTS flow-matching voice cloning)
 - **Personality Core** — SAC framework-based style control (concise/balanced/verbose/pedagogical) that shapes agent response depth, voice expressiveness, and system prompt behavior
 - **Human expert speed ratio** — real-time `Exp: Nx` gauge comparing agent speed to a leading human expert, calibrated across 47 tool baselines
 - **Cost tracking** — real-time token cost estimation for 15+ cloud providers
@@ -526,10 +526,11 @@ The personality system draws on:
 
 ## Emotion Engine — Affective State Modulation
 
-The agent stack includes a real-time emotion system that modulates behavior based on an appraisal-based affective model. Built on Russell's circumplex model of affect, the engine maintains a continuous emotional state defined by two axes:
+The agent stack includes a real-time emotion system that modulates behavior based on an appraisal-based affective model. Built on Russell's circumplex model of affect extended with the dominance axis from UDDETTS ADV space ([arXiv:2505.10599](https://arxiv.org/abs/2505.10599)), the engine maintains a continuous emotional state defined by three axes:
 
 - **Valence** (-1 to +1): displeasure ↔ pleasure
 - **Arousal** (0 to 1): calm ↔ energized
+- **Dominance** (0 to 1): submissive/collaborative ↔ dominant/assertive
 
 Every agent event (tool success/failure, task completion, errors, context pressure) is appraised and shifts the emotional state, which decays back toward a baseline over ~5 minutes. The emotional state modulates agent behavior across all layers: system prompt behavioral hints, voice narration tone, and decision-making style:
 
@@ -589,12 +590,93 @@ The emotion system is informed by peer-reviewed and preprint research:
 
 ```bash
 /voice              # Toggle on/off (default: GLaDOS)
-/voice glados       # GLaDOS voice
-/voice overwatch    # Overwatch voice
+/voice glados       # GLaDOS voice (ONNX, ~50MB)
+/voice overwatch    # Overwatch voice (ONNX, ~50MB)
+/voice kokoro       # Kokoro voice (MLX, macOS Apple Silicon)
+/voice luxtts       # LuxTTS voice clone (flow-matching, any platform)
+/voice clone <file> # Set clone reference audio for LuxTTS (wav/mp3/ogg/flac)
+/voice clone glados # Generate clone ref from GLaDOS → LuxTTS
+/voice clone overwatch  # Generate clone ref from Overwatch → LuxTTS
 ```
 
 Auto-downloads the ONNX voice model (~50MB) on first use. Install `espeak-ng` for best quality (`apt install espeak-ng` / `brew install espeak-ng`).
 
+### LuxTTS Voice Cloning
+
+[LuxTTS](https://github.com/ysharma3501/LuxTTS) is a flow-matching voice cloning TTS engine that synthesizes speech in any voice from a short reference audio clip. It runs locally via a dedicated Python venv (`~/.open-agents/voice/luxtts-venv/`) and downloads the model (~1.2GB) from HuggingFace on first use.
+
+**Setup** (automatic on `/voice luxtts`):
+1. Creates isolated venv with PyTorch (CPU)
+2. Clones LuxTTS repo + installs deps (lhotse, LinaCodec, piper_phonemize)
+3. Downloads YatharthS/LuxTTS model via huggingface_hub
+4. Auto-detects CUDA/MPS/CPU device
+
+**Voice cloning workflow**:
+- Drop an audio file into the terminal while LuxTTS is active → auto-sets as clone reference
+- `/voice clone glados` or `/voice clone overwatch` → generates a synthetic reference from the ONNX voice
+- Custom voice: `/voice clone /path/to/voice-sample.wav` (min ~3 seconds of speech)
+
+**Emotion passthrough**: LuxTTS receives the same ADV-driven prosody as ONNX voices:
+- **Speed** → LuxTTS native `speed` parameter (arousal-driven)
+- **Pitch** → post-synthesis resampling via `resamplePitch()` (valence+arousal tanh curve)
+- **Volume** → WAV sample scaling (dominance-driven)
+
+Output: 48kHz WAV, compatible with Telegram voice messages and WebSocket streaming.
+
+### Narration Engine Architecture
+
+The voice narration system produces **zero static phrase pools** — every spoken sentence is dynamically composed from live tool state, session metrics, and emotion coordinates. The architecture is grounded in 2024-2026 TTS and emotion research:
+
+**Composable sentence anatomy**: `[emotion_interjection] [verb] [object] [flow_context]`
+
+- **verb**: extracted from tool type via `extractToolVerb()` — returns `[terse, expanded, past_tense]` triple (past tense defined at source, no regex reverse-engineering)
+- **object**: extracted from tool args via `extractToolObject()` — the file, command, pattern, or URL being acted on
+- **flow_context**: error recovery framing, same-file continuity, cross-tool content threading (carries result digests forward)
+
+**Sentence structure rotation** ([sNeuron-TST, EMNLP 2024](https://arxiv.org/abs/2410.00593)): Static sentence patterns always activate the same style-specific neurons in TTS models, producing monotone output. The engine cycles through 4 syntactic frames per call:
+
+| Pattern | Frame | Example |
+|---------|-------|---------|
+| 0 | SVO standard | "Looking at voice.ts" |
+| 1 | Object-first | "voice.ts, reading it" |
+| 2 | Contextual opener | "Moving to voice.ts" |
+| 3 | Gerund-led | "Taking a deeper look at voice.ts now" |
+
+**Ring buffer deduplication** ([Moshi inner monologue, arXiv:2410.00037](https://arxiv.org/abs/2410.00037)): A sliding window of the last 8 utterances catches near-duplicates via Jaccard word-level similarity (threshold 0.7). When a near-duplicate is detected, **DITTO adaptive rotation** ([arXiv:2206.02369](https://arxiv.org/abs/2206.02369), NeurIPS 2022) advances the structure pattern by 2 positions to break self-reinforcing repetition loops.
+
+**State-computed emotion interjections**: Instead of word pools, emotion interjections are computed from real session metrics. The emotion quadrant (from ADV coordinates) determines *which* metrics to surface:
+
+| Quadrant | Metrics Surfaced | Example |
+|----------|-----------------|---------|
+| Excited (Q1) | Success streaks, throughput | "12 clean operations." |
+| Stressed (Q2) | Error counts, attempt numbers | "3 consecutive errors now." |
+| Calm (Q3) | Stability, zero-error runs | "28 operations, zero errors." |
+| Subdued (Q4) | Complexity, file count | "6 files in play." |
+
+### Emotion-Driven Prosody (SEST)
+
+The voice engine modulates **three prosodic dimensions** from the emotion state — text vocabulary stays factual, emotion is expressed through *how* it sounds, not *what* it says ([EmoShift, arXiv:2601.22873](https://arxiv.org/abs/2601.22873)):
+
+| Dimension | Source | Effect | Range |
+|-----------|--------|--------|-------|
+| **Pitch** | Valence (50%) + Arousal (30%) + Dominance (20%) | Happy/energized = higher, sad/calm = lower | [-0.10, +0.10] normal, [-0.16, +0.16] stark |
+| **Speed** | Arousal (primary) + Dominance (secondary) | High arousal = faster, high dominance = more deliberate | [0.85x, 1.15x] |
+| **Volume** | Speaker role | Primary = 100%, subordinate (sub-agent) = 55% | [0.55, 1.0] |
+
+Pitch and speed use **nonlinear tanh squashing** ([UDDETTS, arXiv:2505.10599](https://arxiv.org/abs/2505.10599)) — moderate emotions get amplified for expressiveness, extreme emotions saturate gracefully instead of clipping.
+
+Each narration also emits a **ProsodyHint** metadata object following the RLAIF-SPA SEST schema ([arXiv:2510.14628](https://arxiv.org/abs/2510.14628)) — Structure/Emotion/Speed/Tone — which downstream consumers (WebSocket voice sessions, Telegram TTS) can use independently:
+
+```typescript
+interface ProsodyHint {
+  structure: number;    // Sentence pattern index (0-3)
+  emotion: { valence, arousal, dominance };
+  speed: number;        // Speech rate factor
+  tone: number;         // Pitch bias factor
+  quadrant: number;     // Emotion quadrant (1-4)
+}
+```
+
 ### Personality-Aware Voice
 
 Voice output adapts to the active personality style — the same tool call sounds different depending on the `/style` preset:
@@ -602,11 +684,27 @@ Voice output adapts to the active personality style — the same tool call sound
 | Style | Example (file_read) | Example (npm test) |
 |-------|--------------------|--------------------|
 | **concise** | "Reading app.ts" | "Running tests" |
-| **balanced** | "Let me take a look at app.ts" | "Let's run the tests and see how we're doing" |
-| **verbose** | "Alright, let's crack open app.ts and see what we're working with" | "Alright, moment of truth, let's see if the tests pass" |
+| **balanced** | "Looking at app.ts" | "Running tests, checking results" |
+| **verbose** | "Taking a deeper look at app.ts now" | "Running the test suite, 8 clean operations so far" |
 
 Task completion, tool failures, and all TTS announcements follow the same personality tier. Set the style with `/style verbose` and the voice output becomes conversational rather than robotic.
 
+### Voice Narration Research Foundations
+
+The narration engine is informed by peer-reviewed and preprint research:
+
+1. **sNeuron-TST** — Style-specific neurons in text style transfer ([arXiv:2410.00593](https://arxiv.org/abs/2410.00593), EMNLP 2024). Static sentence patterns activate the same neurons monotonically; structure rotation prevents this.
+
+2. **Moshi Inner Monologue** — Streaming LLM with self-tracking ring buffer ([arXiv:2410.00037](https://arxiv.org/abs/2410.00037), 2024). Prevents repetition loops in streaming speech via recent-output awareness.
+
+3. **DITTO** — Pseudo-repetition penalization ([arXiv:2206.02369](https://arxiv.org/abs/2206.02369), NeurIPS 2022). Repetition is self-reinforcing at the sentence level; active disruption of recurring patterns is necessary.
+
+4. **UDDETTS** — ADV emotion space with nonlinear quantification ([arXiv:2505.10599](https://arxiv.org/abs/2505.10599), 2025). Three-axis (arousal/dominance/valence) dimensional emotion conditioning for TTS, with tanh-based mapping to acoustic features.
+
+5. **EmoShift** — Lightweight activation steering for per-sentence emotion ([arXiv:2601.22873](https://arxiv.org/abs/2601.22873), ICASSP 2026). Emotion expressed through prosody modulation (pitch, rate, emphasis), not vocabulary changes.
+
+6. **RLAIF-SPA** — SEST schema for prosody annotation ([arXiv:2510.14628](https://arxiv.org/abs/2510.14628), 2025). Structure/Emotion/Speed/Tone 4-dimension metadata framework for emotional speech synthesis.
+
 ### Live Voice Session
 
 When both `/voice` and `/listen` are enabled, the system spawns a **live voice session** — a real-time bidirectional audio endpoint exposed through a cloudflared tunnel:
@@ -933,7 +1031,7 @@ The TUI features an animated multilingual phrase carousel, live metrics bar with
 | `/compact` | Force context compaction now (default strategy) |
 | `/compact <strategy>` | Compact with strategy: `aggressive`, `decisions`, `errors`, `summary`, `structured` |
 | **Audio & Vision** | |
-| `/voice [model]` | Toggle TTS voice (GLaDOS, Overwatch) |
+| `/voice [model]` | Toggle TTS voice (GLaDOS, Overwatch, Kokoro, LuxTTS) |
 | `/listen [mode]` | Toggle live microphone transcription |
 | `/dream [mode]` | Start dream mode (default, deep, lucid) |
 | **Display & Behavior** | |