agello 0.4.3 → 0.4.5

package/README.md

@@ -8,6 +8,7 @@
 - Interactive terminal view of the existing herdr pane: live cursor, keyboard input, and automatic resize
 - Optional live view of a `terminal-browser` screen, with an on/off switch to control it yourself
 - Annotate: point at an element on that screen and send a request about it, with its CSS selector
+- Tab sound: hear that screen's audio on the page instead of the agent's machine
 - Embeddable `<agent-bridge>` web component
 
 Chat currently supports **Claude Code** sessions. The terminal view connects directly to the pane, including ordinary shells and other terminal applications.
@@ -73,6 +74,10 @@ The `주석` button on the screen panel turns on annotate mode: hovering tints t
 
 Selector: nearest unique `id` / `data-testid` ancestor, then `tag:nth-of-type` steps (light DOM only). Coordinates are CSS px of the viewport. Off while presenting and while "내 조작" is on.
 
+### Tab sound
+
+The `소리` button on the screen panel plays the relayed tab's audio on that page. While at least one page has it on, the tab is muted in the agent's terminal-browser; when the last one turns it off or closes, local playback returns within 4 seconds. agello reroutes the page's Web Audio output and `<audio>`/`<video>` elements into PCM chunks (`GET /screen/audio`, SSE), since CDP has no audio stream. Not captured: Web Audio connected before it was turned on (reload the tab), cross-origin media without CORS (keeps playing locally), cross-site iframes. Audio trails the screen by about 0.2s.
+
 ### Commands
 
 | Command | Description |
@@ -136,6 +141,30 @@ Write the talk ahead so each line appears the moment its screen does:
 - `script load` again after editing keeps the position; `present goto 2.2` then `present resume` replays from a fixed line
 - at the end the agent gets `action=present-done`; while the script is paused and the agent is working, the page shows a small loader
 
+#### Optional TTS and recording
+
+The screen panel's **발표 설정** remains available during presentation. TTS and recording both start OFF and can be used independently:
+
+| TTS | Recording | Result |
+|---|---|---|
+| OFF | OFF | Existing timed script captions |
+| OFF | ON | Presentation images and captions in a silent video |
+| ON | OFF | Spoken script, with captions synchronized to audio playback |
+| ON | ON | Presentation images, captions, and TTS audio in the controlling viewer's video |
+
+1. Open **발표 설정**. Enter a Typecast API key and optionally change the voice ID, then click **키·목소리 적용**. The default voice is `tc_6a0e85a97f7750959b970d5d`; the provider uses Korean (`kor`) and model `ssfm-v30`.
+2. Alternatively, **로컬 키 불러오기** reads only `TYPECAST_API_KEY` from `../yt-outlier/.env`, relative to the server's working directory. It does not execute the file or import other settings. A server started with `TYPECAST_API_KEY` in its environment also has a key ready, with TTS still OFF.
+3. Click **TTS 켜기** in the viewer that should play sound. This click grants browser audio playback permission. Click **대본 재생** after loading a script; **일시정지** stops playback, and **다시 재생** starts a completed script again from its first step.
+4. Click **녹화 시작**, then **녹화 종료** and **영상 저장** to download the recording. Recording can start before presentation if a screen image is already available. Stopping recording leaves script and audio playback running.
+
+With TTS enabled, each script caption appears after that viewer starts the actual audio, and closes when the audio finishes. The audio's duration replaces the script's `hold`; ordinary agent replies still use their existing caption timer. Raising a hand immediately stops audio locally, and resumed playback repeats the interrupted line. Pausing, changing steps, updating a script, or disconnecting cancels outdated speech. A TTS failure pauses the script; fix the key or connection, or turn TTS OFF, then resume.
+
+Only the viewer that enabled TTS plays audio and acknowledges speech progress. Other viewers receive the same captions and can record a silent video. To include TTS audio, record in the viewer that enabled TTS. Audio settings are fixed when recording starts: stop recording before changing TTS. If another viewer changes the controlling audio role, a recording whose audio setting changes is finalized automatically.
+
+Recording uses a 1280×720 canvas that fits the relayed browser image with letterboxing and draws the current captions below it. It records these images and plain caption text, rather than the page's entire DOM: chat, settings, question input, and control buttons are excluded; caption Markdown styling is flattened. Long captions are wrapped and truncated to the safe area. The canvas targets 30fps, while source screen updates can be slower (presentation screenshots are approximately 10fps). A background or minimized tab may render more slowly. Use a current Chromium browser for the verified recording path; other browsers depend on their `canvas.captureStream`, Web Audio, and `MediaRecorder` support. MP4 is preferred when supported, with WebM as a fallback. Streaming WebM duration metadata is finalized when its header supports the correction; otherwise the original browser recording is retained.
+
+Recording stops at 15 minutes or 256MB, at presentation end, on server changes, or on connection loss. Keep the page open until **영상 저장** appears and download it before closing or reloading; recording data is held in browser memory. A later recording replaces the earlier download link.
+
 ## Embed
 
 ```html
@@ -167,9 +196,11 @@ See `web/embed-example.html`.
 
 The server listens on `127.0.0.1` only. Data and input endpoints accept requests from the same origin and `localhost` pages only; other origins get `403` unless added with `--allow-origin`. Allowed origins can send chat prompts and connect to `/terminal` to type directly into the pane, so keep the allow list short.
 
+TTS keys remain in server memory until replaced, cleared with **키 삭제**, or the server stops. Password inputs are cleared after applying; keys and speech-driver tokens are never put in browser storage, chat history, scripts, or SSE broadcasts. Keys are sent only to the local server and Typecast. Script text is sent to Typecast when TTS is ON, and generated audio is cached in server memory for reuse. Allowed origins can also configure TTS and read generated audio, so allow only trusted pages.
+
 ## Development
 
-Run `bun install` and `bun test`. Tests cover the pane tree and creation (against a mock herdr), local terminal assets, origin rejection, ANSI frames, keyboard bytes, resize, exclusive control, and reconnect.
+Run `bun install` and `bun test`. Tests cover the pane tree and creation (against a mock herdr), local terminal assets, origin rejection, ANSI frames, keyboard bytes, resize, exclusive control, reconnect, TTS coordination, stale speech cancellation, and recording lifecycle and metadata.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agello",
-  "version": "0.4.3",
+  "version": "0.4.5",
   "description": "Talk to a coding agent in a herdr pane from the browser: chat, live status, and a shared browser screen.",
   "keywords": [
     "herdr",

package/src/page-audio.ts

@@ -0,0 +1,159 @@
+// Page audio relay, page side. Injected into the relayed terminal-browser tab
+// through CDP while at least one viewer listens (GET /screen/audio).
+//
+// CDP has no audio stream and terminal-browser (Electron) denies tab capture,
+// so the page's own audio graph is rerouted:
+//   - AudioNode.connect(ctx.destination) goes to a per-context tap instead
+//   - <audio>/<video> playing while remote are moved into a WebAudio graph
+//     (createMediaElementSource), which also takes them off the speakers
+//   tap -> out (gain 1 local, 0 remote) -> real destination
+//   tap -> ScriptProcessor -> 16-bit stereo PCM -> binding -> agello server
+//
+// Remote mode lasts while the server keeps calling ping() (every ~1.5s); after
+// 4s without a ping (server gone, tab no longer relayed) local playback returns.
+//
+// Not captured: connections made before injection (reload the page),
+// cross-origin media without CORS (routing them would play silence, so they
+// keep playing locally), cross-site iframes (separate CDP targets).
+
+export const AUDIO_BINDING = "__agelloAudioPcm";
+
+export const PAGE_AUDIO_JS = `(() => {
+  if (window.__agelloAudio) return;
+  const AC = window.AudioContext || window.webkitAudioContext;
+  if (!AC || !window.AudioNode) return;
+  const BINDING = ${JSON.stringify(AUDIO_BINDING)};
+  const QUIET_CHUNKS = 24; // ~2s of silence before chunks stop being sent
+  const origConnect = AudioNode.prototype.connect;
+  const origDisconnect = AudioNode.prototype.disconnect;
+  const origCreateSource = AC.prototype.createMediaElementSource;
+  const origPlay = HTMLMediaElement.prototype.play;
+  const taps = new WeakMap(); // context -> tap record
+  const records = new Set();
+  const owned = new WeakSet(); // media elements moved into our graph
+  let remote = false, lastPing = 0, nextId = 1, mediaCtx = null;
+
+  function send(t, buf) {
+    const n = buf.length;
+    const L = buf.getChannelData(0), R = buf.numberOfChannels > 1 ? buf.getChannelData(1) : L;
+    const pcm = new Int16Array(n * 2);
+    let loud = false;
+    for (let i = 0; i < n; i++) {
+      const l = L[i], r = R[i];
+      if (l || r) loud = true;
+      pcm[2 * i] = (l < -1 ? -1 : l > 1 ? 1 : l) * 32767;
+      pcm[2 * i + 1] = (r < -1 ? -1 : r > 1 ? 1 : r) * 32767;
+    }
+    t.quiet = loud ? 0 : t.quiet + 1;
+    if (t.quiet > QUIET_CHUNKS) return;
+    const bytes = new Uint8Array(pcm.buffer);
+    let s = "";
+    for (let i = 0; i < bytes.length; i += 0x8000) s += String.fromCharCode.apply(null, bytes.subarray(i, i + 0x8000));
+    try { window[BINDING](JSON.stringify({ c: t.id, r: t.ctx.sampleRate, d: btoa(s) })); } catch {}
+  }
+
+  function tapOf(ctx) {
+    let t = taps.get(ctx);
+    if (t) return t;
+    const tap = ctx.createGain(), out = ctx.createGain();
+    out.gain.value = remote ? 0 : 1;
+    origConnect.call(tap, out);
+    origConnect.call(out, ctx.destination);
+    t = { id: nextId++, ctx, tap, out, quiet: QUIET_CHUNKS + 1 };
+    if (ctx.createScriptProcessor) {
+      const proc = ctx.createScriptProcessor(4096, 2, 2), sink = ctx.createGain();
+      sink.gain.value = 0;
+      origConnect.call(tap, proc);
+      origConnect.call(proc, sink);
+      origConnect.call(sink, ctx.destination);
+      proc.onaudioprocess = (e) => { if (remote) send(t, e.inputBuffer); };
+      t.proc = proc;
+    }
+    taps.set(ctx, t);
+    records.add(t);
+    return t;
+  }
+
+  AudioNode.prototype.connect = function (dest, output) {
+    const ctx = this.context;
+    if (dest && ctx instanceof AC && dest === ctx.destination) {
+      origConnect.call(this, tapOf(ctx).tap, output || 0);
+      return dest;
+    }
+    return origConnect.apply(this, arguments);
+  };
+  AudioNode.prototype.disconnect = function (dest) {
+    const ctx = this.context, t = ctx && taps.get(ctx);
+    if (t && dest && dest === ctx.destination) {
+      const args = [...arguments];
+      args[0] = t.tap;
+      if (args.length > 2) args.length = 2; // the tap has one input
+      return origDisconnect.apply(this, args);
+    }
+    return origDisconnect.apply(this, arguments);
+  };
+
+  function capturable(el) {
+    if (el.srcObject) return true;
+    const src = el.currentSrc || el.src;
+    if (!src) return false;
+    if (el.crossOrigin != null) return true;
+    try {
+      const u = new URL(src, location.href);
+      return u.protocol === "data:" || u.origin === location.origin;
+    } catch { return false; }
+  }
+  function own(el) {
+    if (!remote || !(el instanceof HTMLMediaElement) || owned.has(el) || !capturable(el)) return;
+    try {
+      mediaCtx = mediaCtx || new AC();
+      const src = origCreateSource.call(mediaCtx, el);
+      owned.add(el);
+      origConnect.call(src, tapOf(mediaCtx).tap);
+      if (mediaCtx.state !== "running") mediaCtx.resume().catch(() => {});
+    } catch {}
+  }
+  HTMLMediaElement.prototype.play = function () {
+    own(this);
+    return origPlay.apply(this, arguments);
+  };
+  for (const type of ["play", "playing"]) document.addEventListener(type, (e) => own(e.target), true);
+
+  function setRemote(on) {
+    if (on) lastPing = Date.now();
+    if (remote === on) return;
+    remote = on;
+    for (const t of records) {
+      t.out.gain.value = on ? 0 : 1;
+      t.quiet = QUIET_CHUNKS + 1;
+    }
+    if (on) {
+      document.querySelectorAll("audio,video").forEach((el) => { if (!el.paused) own(el); });
+      if (mediaCtx && mediaCtx.state !== "running") mediaCtx.resume().catch(() => {});
+    }
+  }
+  setInterval(() => { if (remote && Date.now() - lastPing > 4000) setRemote(false); }, 1000);
+
+  window.__agelloAudio = {
+    ping: () => setRemote(true),
+    off: () => setRemote(false),
+    state: () => ({ remote, contexts: records.size, media: !!mediaCtx }),
+  };
+})()`;
+
+// Viewer-bound chunk, validated (the binding is callable by the page itself).
+export type PcmChunk = { c: number; r: number; d: string };
+export function parsePcm(payload: string): PcmChunk | null {
+  if (payload.length > 512 * 1024) return null;
+  let m: any;
+  try {
+    m = JSON.parse(payload);
+  } catch {
+    return null;
+  }
+  const { c, r, d } = m ?? {};
+  if (!Number.isInteger(c) || c < 1 || c > 1e6) return null;
+  if (!Number.isFinite(r) || r < 8000 || r > 192000) return null;
+  if (typeof d !== "string" || !d || d.length % 4 || !/^[A-Za-z0-9+/]+={0,2}$/.test(d)) return null;
+  return { c, r, d };
+}

package/src/player.ts

@@ -23,6 +23,7 @@ export type PlayerInfo = {
   next?: string; // next line to show ("3.2")
   last?: string; // last line shown
   steps: number;
+  error?: string;
 };
 
 // Pacing: the audience looks at a new screen first, then reads and thinks.
@@ -80,7 +81,10 @@ export function parsePos(v: string): { step: number; line: number } | null {
 
 export function createPlayer(deps: {
   go: (expr: string) => Promise<void>;
-  say: (text: string, at: string, hold: number) => void;
+  say: (text: string, at: string, hold: number, narration?: string) => void;
+  // null selects text pacing; a promise resolves only after actual playback ends.
+  narrate?: (text: string, at: string, signal: AbortSignal,
+    started: (duration: number, id: string) => void) => Promise<void> | null;
   done: () => void;
   changed: () => void;
 }) {
@@ -91,6 +95,8 @@ export function createPlayer(deps: {
   let entered = false; // `go` of cur.step already run in this play run
   let run = 0; // bumps on every pause/goto/load, stale timers check it
   let timer: Timer | null = null;
+  let speech: AbortController | null = null;
+  let error: string | undefined;
 
   const fmt = (p: { step: number; line: number }) => `${p.step + 1}.${p.line + 1}`;
   const set = (s: PlayerInfo["state"]) => {
@@ -101,6 +107,8 @@ export function createPlayer(deps: {
     run++;
     if (timer) clearTimeout(timer);
     timer = null;
+    speech?.abort();
+    speech = null;
   };
   const wait = (ms: number, id: number, fn: () => void) => {
     timer = setTimeout(() => id === run && fn(), ms);
@@ -117,6 +125,7 @@ export function createPlayer(deps: {
       steps,
       next: script && p.step < steps ? fmt(p) : undefined,
       last: last ? fmt(last) : undefined,
+      ...(error ? { error } : {}),
     };
   }
 
@@ -155,13 +164,38 @@ export function createPlayer(deps: {
       return;
     }
     const text = step.say[cur.line];
-    const hold = lineHold(text, step.hold);
+    const position = { ...cur };
     const lastLine = cur.line === step.say.length - 1;
+    const gap = lastLine ? PACE.afterStep : PACE.betweenLines;
+    speech = new AbortController();
+    const narration = deps.narrate?.(text, fmt(cur), speech.signal, (duration, narrationId) => {
+      if (id !== run || state !== "playing") return;
+      last = position;
+      deps.say(text, fmt(position), duration, narrationId);
+      deps.changed();
+    });
+    if (narration) {
+      try {
+        await narration;
+        if (id !== run || state !== "playing") return;
+        speech = null;
+        cur = { step: position.step, line: position.line + 1 };
+        deps.changed();
+        wait(PACE.fade + gap, id, () => tick(id));
+      } catch {
+        if (id !== run) return;
+        cancel();
+        error = "narration_failed";
+        set("paused");
+      }
+      return;
+    }
+    speech = null;
+    const hold = lineHold(text, step.hold);
     last = { ...cur };
     deps.say(text, fmt(cur), hold);
     cur = { step: cur.step, line: cur.line + 1 };
     deps.changed();
-    const gap = lastLine ? PACE.afterStep : PACE.betweenLines;
     wait(hold * 1000 + PACE.fade + gap, id, () => tick(id));
   }
 
@@ -173,6 +207,7 @@ export function createPlayer(deps: {
     load(next: Script) {
       const playing = state === "playing";
       cancel();
+      error = undefined;
       script = next;
       if (cur.step > next.steps.length) cur = { step: next.steps.length, line: 0 };
       normalize();
@@ -190,6 +225,7 @@ export function createPlayer(deps: {
       if (state === "playing") return null;
       if (state === "done") return "done";
       cancel();
+      error = undefined;
       entered = false; // bring the screen back to the current step first
       set("playing");
       tick(run);
@@ -220,6 +256,7 @@ export function createPlayer(deps: {
       if (pos.line > 0 && pos.line >= step.say.length) return "no_such_line";
       const playing = state === "playing";
       cancel();
+      error = undefined;
       cur = { ...pos };
       entered = false;
       if (playing) {
@@ -235,8 +272,8 @@ export function createPlayer(deps: {
 
     // Presentation ended: stop where it is.
     stop() {
+      cancel();
       if (state === "playing") {
-        cancel();
         set("paused");
       }
     },

package/src/screencast.ts

@@ -1,4 +1,5 @@
 import { runJson } from "./run.ts";
+import { AUDIO_BINDING, PAGE_AUDIO_JS, parsePcm } from "./page-audio.ts";
 
 // terminal-browser screen relay: CDP Page.startScreencast -> SSE (/screen).
 //
@@ -6,6 +7,10 @@ import { runJson } from "./run.ts";
 // follows its active tab, and streams JPEG frames only while at least one
 // viewer is connected. input() forwards whitelisted CDP Input.* commands
 // from the viewer to the page (user control).
+//
+// Page audio (GET /screen/audio, SSE): while at least one viewer listens, the
+// tab's audio is rerouted into PCM chunks for those viewers and muted locally
+// (see page-audio.ts). The last listener leaving restores local playback.
 
 type Browser = {
   key: string;
@@ -24,10 +29,12 @@ export type ScreenMeta = {
   url?: string;
   title?: string;
   agentControlled?: boolean;
+  audioListeners?: number; // status only
 };
 
 export function createScreencast(opts: { browserKey?: string; herdrTab?: () => Promise<string | undefined> }) {
   const viewers = new Set<ReadableStreamDefaultController>();
+  const listeners = new Set<ReadableStreamDefaultController>(); // page audio
   const enc = new TextEncoder();
 
   let meta: ScreenMeta = { connected: false };
@@ -39,6 +46,7 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
   let wsTarget = "";
   let msgId = 0;
   let timer: Timer | null = null;
+  let audioOn = false; // audio hook installed on the current CDP session
   const pending = new Map<number, (result: any) => void>();
 
   const emit = (c: ReadableStreamDefaultController, event: string, data: unknown) => {
@@ -46,9 +54,12 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
       c.enqueue(enc.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`));
     } catch {
       viewers.delete(c);
+      listeners.delete(c);
     }
   };
   const broadcast = (event: string, data: unknown) => viewers.forEach((c) => emit(c, event, data));
+  const AUDIO_OFF = "window.__agelloAudio?.off()";
+  const AUDIO_PING = "window.__agelloAudio?.ping()";
 
   function setMeta(next: ScreenMeta) {
     if (JSON.stringify(next) === JSON.stringify(meta)) return;
@@ -78,6 +89,10 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
   }
 
   function disconnect() {
+    // the old tab plays locally again (its own 4s timeout covers a lost message)
+    if (audioOn && ws?.readyState === WebSocket.OPEN)
+      ws.send(JSON.stringify({ id: ++msgId, method: "Runtime.evaluate", params: { expression: AUDIO_OFF } }));
+    audioOn = false;
     ws?.close();
     ws = null;
     wsTarget = "";
@@ -153,6 +168,7 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
       call("Page.enable");
       call("Page.startScreencast", { format: "jpeg", quality: 70, maxWidth: 1600, maxHeight: 1600 });
       if (clip) capture();
+      if (listeners.size) audioStart();
     };
     sock.onmessage = (ev) => {
       const msg = JSON.parse(String(ev.data));
@@ -162,6 +178,11 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
         resolve(msg.result ?? null);
         return;
       }
+      if (msg.method === "Runtime.bindingCalled" && msg.params?.name === AUDIO_BINDING) {
+        const chunk = listeners.size ? parsePcm(String(msg.params.payload ?? "")) : null;
+        if (chunk) listeners.forEach((c) => emit(c, "pcm", chunk));
+        return;
+      }
       if (msg.method !== "Page.screencastFrame") return;
       const { data, metadata, sessionId } = msg.params;
       call("Page.screencastFrameAck", { sessionId });
@@ -177,6 +198,25 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
     };
   }
 
+  // Install the page hook (new documents too) and switch the tab to remote.
+  function audioStart() {
+    const sock = ws;
+    if (audioOn || !sock || sock.readyState !== WebSocket.OPEN) return;
+    audioOn = true;
+    const call = (method: string, params: object = {}) =>
+      sock.send(JSON.stringify({ id: ++msgId, method, params }));
+    call("Runtime.enable");
+    call("Runtime.addBinding", { name: AUDIO_BINDING });
+    call("Page.addScriptToEvaluateOnNewDocument", { source: PAGE_AUDIO_JS });
+    call("Runtime.evaluate", { expression: PAGE_AUDIO_JS });
+    call("Runtime.evaluate", { expression: AUDIO_PING });
+  }
+  function audioPing() {
+    if (!listeners.size || !ws || ws.readyState !== WebSocket.OPEN) return;
+    if (!audioOn) return audioStart();
+    ws.send(JSON.stringify({ id: ++msgId, method: "Runtime.evaluate", params: { expression: AUDIO_PING } }));
+  }
+
   let syncing = false; // skip a tick while the previous sync is still running
   async function sync() {
     if (syncing) return;
@@ -200,6 +240,7 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
     }
     const url = `ws://127.0.0.1:${b.cdpPort}/devtools/page/${tab.targetId}`;
     if (url !== wsTarget || !ws) connect(b.cdpPort, tab.targetId);
+    else audioPing();
     setMeta({
       connected: true,
       browser: b.key,
@@ -248,6 +289,31 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
     });
   }
 
+  // SSE of page audio for one listener. Only meaningful alongside /screen
+  // (the relay runs while screen viewers are connected).
+  function handleAudio(): Response {
+    let ctrl!: ReadableStreamDefaultController;
+    let ping: Timer;
+    const stream = new ReadableStream({
+      start(c) {
+        ctrl = c;
+        listeners.add(c);
+        emit(c, "ready", {});
+        ping = setInterval(() => emit(c, "ping", {}), 15000);
+        audioPing();
+      },
+      cancel() {
+        clearInterval(ping);
+        listeners.delete(ctrl);
+        if (listeners.size === 0 && audioOn && ws?.readyState === WebSocket.OPEN)
+          ws.send(JSON.stringify({ id: ++msgId, method: "Runtime.evaluate", params: { expression: AUDIO_OFF } }));
+      },
+    });
+    return new Response(stream, {
+      headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", Connection: "keep-alive" },
+    });
+  }
+
   const INPUT_METHODS = new Set(["Input.dispatchMouseEvent", "Input.dispatchKeyEvent", "Input.insertText"]);
   function input(raw: string) {
     let m: any;
@@ -263,7 +329,7 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
 
   // Which browser the relay uses (or would use), for status reporting.
   async function describe(): Promise<ScreenMeta> {
-    if (meta.connected) return meta;
+    if (meta.connected) return { ...meta, audioListeners: listeners.size };
     const { browser, reason } = await pickBrowser();
     return browser ? { connected: false, browser: browser.key } : { connected: false, reason };
   }
@@ -287,7 +353,7 @@ export function createScreencast(opts: { browserKey?: string; herdrTab?: () => P
     return r && !r.exceptionDetails ? (r.result?.value ?? null) : null;
   }
 
-  return { handle, input, describe, setClip, evaluate, inspect };
+  return { handle, handleAudio, input, describe, setClip, evaluate, inspect };
 }
 
 export type InspectQuery = { x?: number; y?: number; selector?: string; full?: boolean };

package/src/server.ts

@@ -8,6 +8,7 @@
 //                        (images are saved to $TMPDIR/agello-uploads and sent as [image: <path>])
 //                        (action "annotate": + target {selector, label, rect, text?, url?} from inspect)
 // GET  /screen           SSE: terminal-browser screen frames (CDP screencast)
+// GET  /screen/audio     SSE: tab audio as PCM chunks; muted in the browser while listened
 // POST /upload           {images: [{type, data(base64)}]} -> {paths}: terminal image paste
 // WS   /terminal         interactive herdr terminal frames, keyboard input, resize
 // GET  /panes            herdr workspace -> tab -> pane tree
@@ -31,6 +32,7 @@ import { join } from "node:path";
 import { runJson } from "./run.ts";
 import { createPlayer, parsePos, parseScript, type PlayerInfo } from "./player.ts";
 import { createScreencast, type Inspected, type Rect } from "./screencast.ts";
+import { createTts } from "./tts.ts";
 
 export type ServerOptions = {
   port: number;
@@ -189,12 +191,21 @@ export async function startServer(opts: ServerOptions) {
   // ----- SSE clients -----
 
   const clients = new Set<ReadableStreamDefaultController>();
+  const viewers = new Map<ReadableStreamDefaultController, string | null>();
+  const clientPings = new Map<ReadableStreamDefaultController, Timer>();
+  const dropClient = (ctrl: ReadableStreamDefaultController) => {
+    if (!clients.delete(ctrl)) return;
+    clearInterval(clientPings.get(ctrl));
+    clientPings.delete(ctrl);
+    tts.disconnect(viewers.get(ctrl) ?? null);
+    viewers.delete(ctrl);
+  };
   const enc = new TextEncoder();
   const send = (ctrl: ReadableStreamDefaultController, event: string, data: unknown) => {
     try {
       ctrl.enqueue(enc.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`));
     } catch {
-      clients.delete(ctrl);
+      dropClient(ctrl);
     }
   };
   const broadcast = (event: string, data: unknown) => clients.forEach((c) => send(c, event, data));
@@ -382,13 +393,16 @@ export async function startServer(opts: ServerOptions) {
 
   // Script player: agello owns order and timing, the agent owns the deck
   // (each step's `go`) and decides when to resume after a question.
+  const tts = createTts({ emit: broadcast, pause: () => player.pause() });
   const player = createPlayer({
     async go(expr) {
       const js = expr.startsWith("#") ? `location.hash = ${JSON.stringify(expr)}` : expr;
       if (!(await screen.evaluate(js))) console.error(`present: go failed: ${expr}`);
     },
-    say(text, at, hold) {
-      broadcast("message", { role: "assistant", text, ts: new Date().toISOString(), script: at, hold });
+    narrate: tts.narrate,
+    say(text, at, hold, narration) {
+      broadcast("message", { role: "assistant", text, ts: new Date().toISOString(), script: at, hold,
+        ...(narration ? { narration } : {}) });
     },
     done() {
       const n = player.info().steps;
@@ -482,13 +496,15 @@ export async function startServer(opts: ServerOptions) {
     return res;
   }
 
-  function events(): Response {
+  function events(viewer: string | null): Response {
     let ctrl!: ReadableStreamDefaultController;
     let ping: Timer;
     const stream = new ReadableStream({
       start(c) {
         ctrl = c;
         clients.add(c);
+        viewers.set(c, viewer);
+        tts.connect(viewer);
         send(c, "hello", {
           transcript: !!transcript,
           pane: PANE,
@@ -498,12 +514,16 @@ export async function startServer(opts: ServerOptions) {
         });
         if (lastStatus) send(c, "status", lastStatus);
         send(c, "present", presentView());
+        send(c, "tts", tts.status());
+        const narration = tts.currentNarration();
+        if (narration) send(c, "message", { role: "assistant", ts: new Date().toISOString(), ...narration });
         for (const t of pendingTools.values()) send(c, "tool_start", t);
         ping = setInterval(() => send(c, "ping", {}), 15000);
+        clientPings.set(c, ping);
       },
       cancel() {
         clearInterval(ping);
-        clients.delete(ctrl);
+        dropClient(ctrl);
       },
     });
     return new Response(stream, {
@@ -546,6 +566,8 @@ export async function startServer(opts: ServerOptions) {
   }
 
   async function route(req: Request, url: URL): Promise<Response> {
+    const ttsResponse = await tts.handle(req, url.pathname);
+    if (ttsResponse) return ttsResponse;
     if (url.pathname.startsWith("/panes")) {
       const res = await panesRoute(req, url.pathname, PANE);
       if (res) return res;
@@ -559,9 +581,11 @@ export async function startServer(opts: ServerOptions) {
           present: presentView(),
         });
       case "/events":
-        return events();
+        return events(url.searchParams.get("viewer"));
       case "/screen":
         return screen.handle();
+      case "/screen/audio":
+        return screen.handleAudio();
       case "/terminal.js":
         return new Response(terminalJS, { headers: { "Content-Type": "text/javascript; charset=utf-8" } });
       case "/terminal.css":
@@ -570,6 +594,10 @@ export async function startServer(opts: ServerOptions) {
         return new Response(COMPONENT, {
           headers: { "Content-Type": "text/javascript; charset=utf-8", "Cache-Control": "no-store" },
         });
+      case "/presentation-media.js":
+        return new Response(Bun.file(join(WEB_DIR, "presentation-media.js")), {
+          headers: { "Content-Type": "text/javascript; charset=utf-8", "Cache-Control": "no-store" },
+        });
       case "/send":
         if (req.method === "POST") return sendPrompt(req);
         break;
@@ -623,7 +651,7 @@ export async function startServer(opts: ServerOptions) {
     async fetch(req, server) {
       const url = new URL(req.url);
       const origin = req.headers.get("Origin");
-      const isScript = ["/agent-bridge.js", "/terminal.js", "/terminal.css"].includes(url.pathname);
+      const isScript = ["/agent-bridge.js", "/presentation-media.js", "/terminal.js", "/terminal.css"].includes(url.pathname);
 
       // The component script itself is public; data and input endpoints are origin-checked.
       if (!isScript && !originAllowed(origin, url.origin))
@@ -657,6 +685,15 @@ export async function startServer(opts: ServerOptions) {
     session: SESSION,
     transcript,
     stop() {
+      player.stop();
+      tts.stop();
+      for (const client of clients) {
+        clearInterval(clientPings.get(client));
+        try { client.close(); } catch {}
+      }
+      clients.clear();
+      viewers.clear();
+      clientPings.clear();
       terminal.stop();
       timers.forEach(clearInterval);
       server.stop(true);