open-agents-ai 0.47.0 → 0.49.0

package/README.md

@@ -39,6 +39,8 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - **Dream Mode** — creative idle exploration modeled after real sleep architecture (NREM→REM cycles)
 - **Autoresearch Swarm** — 5-agent GPU experiment loop during REM sleep: Researcher, Monitor, Evaluator, Critic, Flow Maintainer autonomously run ML training experiments, keep improvements, discard regressions
 - **Live Listen** — bidirectional voice communication with real-time Whisper transcription
+- **Live Voice Session** — `/listen` with `/voice` enabled spawns a cloudflared tunnel with a real-time WebSocket audio endpoint. A floating presence UI shows live transcription, connected users, and audio visualization. Echo cancellation prevents TTS feedback loops
+- **Telegram Voice** — `/voice` enabled via Telegram forwards TTS audio as voice messages alongside text responses. Incoming voice messages are auto-transcribed and handled as text
 - **Neural TTS** — hear what the agent is doing via GLaDOS or Overwatch ONNX voices, with personality-driven expressiveness
 - **Personality Core** — SAC framework-based style control (concise/balanced/verbose/pedagogical) that shapes agent response depth, voice expressiveness, and system prompt behavior
 - **Human expert speed ratio** — real-time `Exp: Nx` gauge comparing agent speed to a leading human expert, calibrated across 47 tool baselines
@@ -1005,6 +1007,50 @@ Voice output adapts to the active personality style — the same tool call sound
 
 Task completion, tool failures, and all TTS announcements follow the same personality tier. Set the style with `/style verbose` and the voice output becomes conversational rather than robotic.
 
+### Live Voice Session
+
+When both `/voice` and `/listen` are enabled, the system spawns a **live voice session** — a real-time bidirectional audio endpoint exposed through a cloudflared tunnel:
+
+```bash
+/voice              # Enable TTS
+/listen             # Starts mic + spawns voice session
+```
+
+What happens:
+1. A local HTTP + WebSocket server starts on a random port
+2. `cloudflared tunnel --url` exposes it publicly with a `*.trycloudflare.com` URL
+3. The terminal shows a `☁` cloud icon with live session runtime
+4. Visiting the URL shows a **floating presence** UI that:
+   - Undulates with the model's TTS audio output
+   - Captures your microphone (with echo cancellation)
+   - Shows live transcription for both sides
+   - Displays connected users
+
+**Echo cancellation**: The server mutes ASR input while TTS is playing, preventing the model from hearing its own voice.
+
+**Terminal waterfall**: The cloud session sits in the normal TUI waterfall alongside other activity, showing connected users and session runtime.
+
+```
+  ☁ Live Voice Session
+    ⎿ URL: https://abc-xyz.trycloudflare.com
+    ⎿ Bidirectional PCM audio + live transcription
+    ⎿ → web-user connected
+    ⎿ ☁ [user] hello, what are you working on?
+    ⎿ ☁ [agent] I'm analyzing the codebase structure...
+```
+
+Stop with `/listen stop` or `/listen off`.
+
+### Telegram Voice Messages
+
+When `/voice` is enabled and the Telegram bridge is active:
+- **Outgoing**: Agent responses are synthesized to audio via TTS and sent as Telegram voice messages (OGG/Opus) alongside the text response
+- **Incoming**: Voice messages sent to the bot are auto-transcribed via Whisper and handled as text — no need for the agent to explicitly call `transcribe_file`
+
+### Auto-Install Dependencies
+
+Cloudflared is automatically installed at startup alongside other dependencies (moondream, tesseract, transcribe-cli). The install is non-blocking and runs in the background.
+
 ## Personality Core — SAC Framework Style Control
 
 The personality system controls how the agent communicates — from silent operator to teacher mode. It's based on the **SAC framework** (arXiv:2506.20993) which models personality along five behavioral intensity dimensions rather than binary trait toggles.