@todoforai/voiceloop 0.1.8 → 0.1.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,18 +1,31 @@
1
1
  # voiceloop
2
2
 
3
- **The fastest way to put a real, interruptible voice agent in the browser.**
3
+ **The fastest voice agent loop in the browser**
4
+
5
+ ### ▶︎ [Try the live demo](https://todoforai.github.io/voiceloop/) — no keys, no install, runs in your tab
6
+
7
+ ## Why
8
+
9
+ I just couldn't find a solution that actually creates the fluid JARVIS feel in the browser (and it's crazy that it's 2026, not 2010, and there's still no really good one).
10
+
11
+ Everybody should have the best voice loop. That's why we created this, and we'll keep pushing the limits here — observations and contributions welcome, let's push the boundaries together!
12
+
13
+ ## Features
4
14
 
5
15
  A zero-dependency JavaScript library that runs the full voice loop — **VAD → STT → LLM → TTS** — with the hard parts already solved:
6
16
 
7
- - **Real barge-in** — interrupt the agent mid-sentence. Triggers on *transcribed novel words*, not raw mic energy, so the agent's own voice leaking into the mic never cuts it off.
8
- - **Self-echo filtering** — a word-match filter compares what the mic hears against what the agent is currently saying. Benchmarked under real speaker→mic coupling with echo cancellation fully off: 0 self-interruptions in 30 turns, at full speed — while Pipecat (default config) cut its own reply on 20/30 turns and OpenAI Realtime on 17/30 ([echo results](bench/results/RESULTS.md#scenario-echo-smalltalk--speakermic-coupling-15db--30ms)).
9
- - **First-sentence streaming TTS** — the LLM streams text, TTS synthesizes the first sentence *while the rest is still generating*, then stays one sentence ahead. First audio in well under a second.
10
- - **Speculative prefetch** — the LLM call starts while you're still finishing your sentence, overlapping model latency with the end-of-turn pause. Replies feel instant.
11
- - **Tap-to-seek** — playback keeps a tape of synthesized clips; tap anywhere in the transcript to jump the voice there, backward or forward.
12
- - **Turn serialization** — rapid-fire turns, tool results, holds and replays can never overlap or talk over each other. Locked in by a 100-test regression suite.
13
- - **Local-first defaults** — Silero VAD (WASM) and Piper TTS (WASM) run in the browser. Free, no cloud round-trip for voice output (assets load from CDN, then cache).
17
+ - **Real barge-in** — triggers on transcribed *novel words*, not mic energy, so the agent's own voice never cuts it off.
18
+ - **Self-echo filtering** — 0 self-interruptions with AEC off (Pipecat cut itself 20/30, OpenAI Realtime 17/30 [bench](bench/results/RESULTS.md#scenario-echo-smalltalk--speakermic-coupling-15db--30ms)).
19
+ - **First audio <1s** — TTS speaks sentence 1 while the LLM writes sentence 2, and the LLM call starts speculatively during your end-of-turn pause.
20
+ - **Serialized turns** — rapid-fire turns, tool results, holds and replays can never talk over each other. Locked in by 178 tests.
21
+ - **Local-first** — Silero VAD and Piper TTS run as WASM in the tab: free, no cloud round-trip (CDN, then cache).
22
+
23
+ Extra:
24
+ - **Tap-to-seek** — click anywhere in the transcript to jump the voice there, backward or forward.
14
25
 
15
- Everything is pluggable: bring your own LLM (any OpenAI-compatible endpoint or a custom async generator), pick an STT provider (Web Speech, ElevenLabs Scribe, Deepgram Flux, Speechmatics), swap the TTS.
26
+ **The point: an in-browser voice agent that just works properly.**
27
+
28
+ Everything is pluggable: any OpenAI-compatible LLM (or a custom async generator), STT (Web Speech, ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS.
16
29
 
17
30
  ## Quick start
18
31
 
@@ -29,7 +42,7 @@ const agent = new VoiceAgent({
29
42
  // Any OpenAI-compatible /chat/completions endpoint. In production point this at
30
43
  // YOUR proxy route — never ship a provider secret key to the page.
31
44
  llmUrl: '/api/chat/completions',
32
- model: 'gpt-4o-mini',
45
+ model: 'claude-haiku-4-5', // voice wants first-token speed — use the fastest model your endpoint serves
33
46
 
34
47
  persona: 'You are a friendly cooking assistant.',
35
48
  sysmsg: 'The user is on the recipes page.', // live context, update via setSysmsg()
@@ -52,35 +65,25 @@ button.onclick = async () => {
52
65
 
53
66
  That's it. Speak, get spoken answers, interrupt at will.
54
67
 
55
- ## Try it in 20 seconds
68
+ ## The demo
69
+
70
+ **[todoforai.github.io/voiceloop](https://todoforai.github.io/voiceloop/)** starts in echo mode —
71
+ browser Web Speech STT, local Piper TTS, a stand-in "LLM" that repeats you — so it costs nothing
72
+ and needs no backend. THIS is not the BEST configuration (deepgram STT + elevenlabs flash TTS). the best goes from 2100ms -> 900ms, that's where it start to be LIVE!
56
73
 
57
- **[todoforai.github.io/voiceloop](https://todoforai.github.io/voiceloop/)** — the live demo, no
58
- install, no keys and no backend of ours: the default path is your browser's own Web Speech STT,
59
- local Piper TTS, and an echo stand-in for the LLM. (Web Speech is the *browser's* recognizer —
60
- on most builds it still sends audio to the browser vendor. Pick a pipeline STT provider in
61
- settings to see voiceloop's own VAD→STT path.)
74
+ It shows the two numbers that decide whether an agent feels alive (**first token**, **first sound**), and click-to-interrupt strikes through the words you cut off.
75
+ Open *settings* to switch to a pipeline STT provider (Web Speech bypasses voiceloop's VAD→STT
76
+ path) and point it at any OpenAI-compatible endpoint (Ollama, your proxy).
62
77
 
63
- Or locally, to point it at your own LLM:
78
+ Run it locally against your own LLM:
64
79
 
65
80
  ```sh
66
- npx serve . # from a clone
67
- npx serve node_modules/@todoforai/voiceloop # from npm
81
+ npx serve . # from a clone
82
+ npx serve node_modules/@todoforai/voiceloop # from npm — then open index.html
68
83
  ```
69
84
 
70
- Open **`/examples/demo.html`** — the whole loop behind one orb that breathes with the
71
- mic level (and, while it answers, with how fast its spoken cursor advances), the live
72
- interim transcript, click-to-interrupt with the unspoken remainder struck through where
73
- you cut it off, and the two numbers that decide whether it feels alive: **first token**
74
- and **first sound**. It starts in **echo mode**
75
- — browser-native Web Speech STT, local Piper TTS and a stand-in "LLM" that
76
- repeats you — so it runs with no keys, no backend and no cost. Open *settings*
77
- to point it at any OpenAI-compatible endpoint (Ollama, your proxy) and a cloud
78
- STT provider.
79
-
80
- `/examples/simple-browser.html` is the same loop with no UI at all — 100 lines,
81
- the place to start reading.
82
-
83
- (`file://` won't work — mic and module imports need `http://localhost` or https.)
85
+ `file://` won't work — mic and module imports need `http://localhost` or https.
86
+ Start reading at [`examples/simple-browser.html`](examples/simple-browser.html): the same loop, no UI, ~100 lines.
84
87
 
85
88
  ## How the latency adds up
86
89
 
@@ -111,17 +114,13 @@ into the mic (no AEC):
111
114
  | Pipecat 1.8.1 (same providers) | 1050ms | 2/30 | **20/30** |
112
115
  | ElevenLabs ConvAI | 1450ms | **0/30** | 0/30 |
113
116
 
114
- Overlapping a hesitating user is fine — people do it, and every stack here except Pipecat backs
115
- off the moment the user keeps talking (voiceloop yields in 420ms, and only enters 2 of 30 turns
116
- at all). The failure that still separates them is echo: fed its own voice through the mic with
117
- no AEC, the other fast stacks hear themselves as the user and cut their own replies.
118
- voiceloop's word-level echo filter runs echo-coupled turns at full speed (930ms, parity with
119
- clean).
117
+ The separating failure is echo: with no AEC, the other fast stacks hear their own voice as the
118
+ user and cut their own replies. voiceloop's word-level echo filter runs echo-coupled turns at
119
+ full speed (930ms — parity with clean).
120
120
 
121
- \* Realtime is speech-to-speech — it can't use the bench's fixed mock LLM, so its row isn't
122
- fully apples-to-apples.
121
+ \* Realtime is speech-to-speech, so it can't use the bench's fixed mock LLM — not fully apples-to-apples.
123
122
 
124
- Full tables, method and reproduction steps: [`results/RESULTS.md`](https://github.com/todoforai/voice-agent-bench/blob/master/results/RESULTS.md).
123
+ Full tables, method, reproduction: [`results/RESULTS.md`](https://github.com/todoforai/voice-agent-bench/blob/master/results/RESULTS.md).
125
124
 
126
125
  ## STT providers
127
126
 
@@ -134,7 +133,7 @@ new VoiceAgent({ sttProvider: 'speechmatics', sttTokenUrl: '/api/stt/token' })
134
133
 
135
134
  Cloud providers authenticate with a **short-TTL token minted by your backend** so the raw API key never reaches the page. Either:
136
135
 
137
- - `sttTokenUrl` — a POST route on your server that returns `{ token }` (mint it against the provider's temp-token API with your secret key), or
136
+ - `sttTokenUrl` — a POST route on your server that returns `{ token }` (mint it against the provider's temp-token API with your secret key — a complete single-file Cloudflare Worker doing exactly this for Deepgram, plus ElevenLabs TTS and LLM proxies with rate limits, is in [`examples/token-worker/`](examples/token-worker/)), or
138
137
  - `getSttToken` — an async callback `(provider) => ({ token })` when your auth doesn't fit a bare POST, or when each provider mints a different credential. It is handed the provider actually running (after the fallback below), so you pick the route without re-deriving that rule:
139
138
 
140
139
  ```js
@@ -226,6 +225,8 @@ agent.setHeld(bool) // hold: queue utterances, reply over all of them
226
225
  agent.setMuted(bool) // mic mute
227
226
  agent.setTtsMuted(bool) // silent mode: transcript + tools still run
228
227
  agent.setSysmsg(text) // update live context mid-session
228
+ agent.interrupt() // programmatic barge-in: stop the current reply
229
+ agent.setVoice(id) / agent.setSpeed(x) / agent.listVoices() // voice controls (engine-dependent)
229
230
  agent.seek(nwIndex) // tap-to-seek within the current reply (nwIndex = count of
230
231
  // NON-WHITESPACE chars before the tap — whitespace doesn't count)
231
232
  agent.replay(text, fromNw?, onProgress?) // re-voice a past reply; onProgress(spokenText, done)
@@ -264,7 +265,7 @@ Two of them exist purely so a UI can look alive, and both cost you nothing to ig
264
265
  - **interruption** — there is no "cut short" event: the final `assistant` event carries both what
265
266
  was heard (`text`) and what was written (`full`), so a non-empty `full.slice(text.length)` means
266
267
  the reply was cut before that remainder was ever spoken. Render it struck through and the
267
- transcript shows exactly where playback stopped (see `examples/demo.html`). It does not say *why*:
268
+ transcript shows exactly where playback stopped (see `index.html`). It does not say *why*:
268
269
  a voice barge-in, `interrupt()`, a superseding turn and `setTtsMuted(true)` all end a reply the
269
270
  same way.
270
271
 
@@ -315,3 +316,5 @@ The suite locks in turn serialization, hold/release semantics, tool dedup, self-
315
316
  ## License
316
317
 
317
318
  MIT
319
+
320
+ "voiceloop" is a trademark of TODOforAI. You are free to use, modify and redistribute the code under the MIT license, but not to present a fork or derivative as the official voiceloop project or publish it under the voiceloop name.
@@ -46,12 +46,12 @@
46
46
  const agent = new VoiceAgent({
47
47
  // Point at any OpenAI-compatible /chat/completions endpoint.
48
48
  // Quick local test with Ollama (from the repo root):
49
- // ollama pull llama3.2
49
+ // ollama pull qwen3
50
50
  // OLLAMA_ORIGINS='*' ollama serve
51
51
  // npx serve . → open http://localhost:3000/examples/simple-browser.html
52
52
  // (OLLAMA_ORIGINS allows this page's origin; mic access needs http://localhost or https.)
53
53
  llmUrl: 'http://localhost:11434/v1/chat/completions',
54
- model: 'llama3.2',
54
+ model: 'qwen3',
55
55
 
56
56
  persona: 'You are a concise, friendly voice assistant.',
57
57
 
@@ -60,6 +60,10 @@
60
60
  $('state').textContent = e.state;
61
61
  } else if (e.type === 'stt') {
62
62
  userEl ??= bubble('user');
63
+ // A turn can die WITHOUT turnComplete (echo-dropped, retracted, reply error) — userEl
64
+ // then survives at its old position and ??= would paint the NEXT turn's speech above
65
+ // later messages. append() moves an existing node back to the bottom (no-op otherwise).
66
+ log.append(userEl);
63
67
  // partial: committed prefix + live tail; turnComplete: e.text is the whole turn
64
68
  userEl.textContent = e.turnComplete ? e.text : `${e.committed ?? ''} ${e.text}`.trim();
65
69
  if (e.turnComplete) userEl = null;
@@ -0,0 +1,65 @@
1
+ # voiceloop-token
2
+
3
+ The tiny backend the live demo needs. GitHub Pages is static, so it has no server to mint
4
+ credentials — and cloud STT/TTS keys must never reach the page. This Worker holds them and hands
5
+ out only what each hop needs.
6
+
7
+ | route | returns | why |
8
+ |---|---|---|
9
+ | `POST /stt/token` | `{ token, expires_in }` | short-TTL Deepgram key; the browser opens the WS itself |
10
+ | `POST /tts` | `audio/mpeg` | ElevenLabs proxy — the `xi-api-key` stays server-side |
11
+ | `POST /llm` | OpenAI-format SSE | Anthropic proxy — Claude has no short-TTL key to mint, so the key stays here |
12
+
13
+ (The voiceloop demo runs this exact Worker.)
14
+
15
+ ## Deploy
16
+
17
+ ```sh
18
+ wrangler deploy
19
+ wrangler secret put DEEPGRAM_API_KEY
20
+ wrangler secret put ELEVENLABS_API_KEY
21
+ wrangler secret put ANTHROPIC_API_KEY
22
+ ```
23
+
24
+ `ALLOWED_ORIGINS` (in `wrangler.toml`) is a comma-separated allowlist. These routes spend real
25
+ money, so an open CORS policy is an open wallet — a request from an unlisted origin gets no CORS
26
+ headers and a 403. But CORS is **not** authentication: it governs what browser JS may read, and a
27
+ request with no `Origin` (curl, a script) never engages it while a non-browser caller can forge one.
28
+ So every paid route also passes per-IP rate limits in two windows: a 60s `[[ratelimits]]` burst
29
+ limiter (best-effort, per-Cloudflare-location — `/stt/token` gets a tight one, the `/tts`+`/llm`
30
+ conversation hot path a roomier one) and a long-window quota (`QUOTA_LIMIT` calls per
31
+ `QUOTA_PERIOD_H` hours, default 300/6h) counted globally in a Durable Object — that one is the
32
+ actual wallet cap. Combine it with provider-side spend caps; anything costlier than a demo key
33
+ needs real auth in front.
34
+
35
+ `/tts` caps text at 600 chars and `/llm` caps the conversation at 8000.
36
+
37
+ `/llm` pins the model server-side (`ANTHROPIC_MODEL`, default `claude-haiku-4-5-20251001`) and caps
38
+ `max_tokens`, and forwards an ALLOWLIST rather than the caller's object (only `messages` and a
39
+ clamped `max_tokens` are read from the request; model and stream mode are server-owned):
40
+ the page picks neither model nor limits, and a large `tools` schema can't smuggle billable input
41
+ tokens past the size check. Anthropic serves an OpenAI-compatible `/v1/chat/completions`, so the SSE passes straight
42
+ through to voiceloop's built-in `makeOpenAILLM` — no client-side adapter.
43
+
44
+ The persona is pinned server-side too (`SYSTEM_PROMPT` var, default: Jarvis, answer extremely
45
+ briefly): caller-supplied `system` messages are stripped, so a client can't prompt the hosted
46
+ demo into long — i.e. expensive — answers. Brevity doubles as a latency control: the reply is
47
+ only as fast as the sentence the TTS is waiting to finish.
48
+
49
+ ## Use it from the demo
50
+
51
+ In the demo's settings: STT = `deepgram`, STT token url = `<worker>/stt/token`. Mode =
52
+ `full loop — hosted claude haiku` uses `/llm` and needs no key or LLM url at all.
53
+
54
+ | route | returns |
55
+ |---|---|
56
+ | `GET /tts` | `{ voices: [{ id, name }] }` — listing needs the key too, so the browser can't ask ElevenLabs itself |
57
+
58
+ ## The fastest config
59
+
60
+ Benchmarked at **984ms voice→voice vs 2126ms** for the zero-key path ([RESULTS](../../bench/results/RESULTS.md)) —
61
+ the whole gap is end-of-turn (366ms vs 1565ms). In the demo's settings:
62
+
63
+ - **STT** `deepgram` · **STT token url** `<worker>/stt/token`
64
+ - **TTS** `elevenlabs` · **TTS url** `<worker>/tts`
65
+ - **mode** `hosted claude haiku` — measured 568ms to first token, 893ms to first sound in-browser
@@ -0,0 +1,173 @@
1
+ // voiceloop-token — the tiny backend the live demo needs, and nothing more.
2
+ //
3
+ // GitHub Pages is static, so the demo has no server to mint credentials. Cloud STT/TTS keys must
4
+ // never reach the page, so this Worker holds them and hands out only what each hop needs:
5
+ //
6
+ // POST /stt/token → { token, expires_in } short-TTL Deepgram key (browser opens the WS itself)
7
+ // POST /tts → audio/mpeg bytes ElevenLabs proxy (the xi-api-key stays here)
8
+ // POST /llm → OpenAI-format SSE Anthropic proxy (there is no short-TTL key to mint)
9
+ //
10
+ // Secrets (wrangler secret put): DEEPGRAM_API_KEY, ELEVENLABS_API_KEY, ANTHROPIC_API_KEY.
11
+ // ALLOWED_ORIGINS (var) is a comma-separated allowlist — these routes spend real money, so an
12
+ // open CORS policy is an open wallet.
13
+ //
14
+ // Two layers, because they stop different things. The origin allowlist keeps OTHER PAGES from
15
+ // spending the keys — that is all CORS can do: it governs what browser JS may read, and a request
16
+ // with no Origin header (curl, a script) never engages it, while a non-browser caller can forge one.
17
+ // So every paid route also passes through a per-IP rate limit, which is what actually bounds the
18
+ // bill when the URL leaks. Neither is authentication; a demo backend holding capped, cheap keys is
19
+ // the threat model. Anything costlier needs real auth in front of it.
20
+
21
+ // Persona for the hosted demo (override with the SYSTEM_PROMPT var). The brevity rule is a cost
22
+ // AND a latency control: every output token is billed, and in a voice loop the reply is only as
23
+ // fast as the sentence the TTS is waiting to finish.
24
+ const HOSTED_PERSONA =
25
+ 'You are Jarvis, a voice assistant. Your reply is spoken aloud, so answer EXTREMELY briefly — ' +
26
+ 'one short sentence, two at most. Plain speakable prose: no markdown, no lists, no headings. ' +
27
+ 'Never explain that you are being brief.';
28
+
29
+ const json = (obj, status, origin) =>
30
+ new Response(JSON.stringify(obj), { status, headers: { 'content-type': 'application/json', ...cors(origin) } });
31
+
32
+ const cors = (origin) => origin ? {
33
+ 'access-control-allow-origin': origin,
34
+ 'access-control-allow-headers': 'content-type,x-api-key',
35
+ 'access-control-allow-methods': 'GET,POST,OPTIONS',
36
+ 'vary': 'Origin',
37
+ } : {};
38
+
39
+ const allow = (req, env) => {
40
+ const origin = req.headers.get('Origin');
41
+ if (!origin) return null;
42
+ const list = (env.ALLOWED_ORIGINS || '').split(',').map((s) => s.trim()).filter(Boolean);
43
+ return list.includes(origin) ? origin : null;
44
+ };
45
+
46
+ // Two windows per IP: the 60s limiter binding is burst protection, and the Quota Durable Object
47
+ // is the long-window wallet cap ([[ratelimits]] only supports 10s/60s periods, hence the DO).
48
+ const rateLimited = async (req, env, limiter = env.PAID_RATE_LIMITER) => {
49
+ const ip = req.headers.get('CF-Connecting-IP') || 'unknown';
50
+ if (limiter && !(await limiter.limit({ key: ip })).success) return true;
51
+ if (!env.QUOTA) return false;
52
+ const r = await env.QUOTA.get(env.QUOTA.idFromName(ip)).fetch('https://quota/', {
53
+ method: 'POST',
54
+ body: JSON.stringify({ limit: Number(env.QUOTA_LIMIT) || 300, periodMs: (Number(env.QUOTA_PERIOD_H) || 6) * 3600_000 }),
55
+ });
56
+ return !(await r.json()).ok;
57
+ };
58
+
59
+ // Per-IP fixed-window counter. A Durable Object because the count must be one number, globally —
60
+ // the Workers runtime itself has no cross-datacenter state, and the [[ratelimits]] binding is
61
+ // per-datacenter best-effort. One DO instance per IP (idFromName), state survives restarts.
62
+ export class Quota {
63
+ constructor(state) { this.state = state; }
64
+ async fetch(req) {
65
+ const { limit, periodMs } = await req.json();
66
+ const now = Date.now();
67
+ let { start = now, count = 0 } = await this.state.storage.get(['start', 'count']).then((m) => Object.fromEntries(m)) ?? {};
68
+ if (now - start >= periodMs) { start = now; count = 0; }
69
+ const ok = count < limit;
70
+ if (ok) await this.state.storage.put({ start, count: count + 1 });
71
+ return new Response(JSON.stringify({ ok }));
72
+ }
73
+ }
74
+
75
+ export default {
76
+ async fetch(req, env) {
77
+ const url = new URL(req.url);
78
+ const origin = allow(req, env);
79
+
80
+ if (req.method === 'OPTIONS') {
81
+ // A disallowed origin gets no CORS headers at all — the browser then blocks it, which is the point.
82
+ return new Response(null, { status: origin ? 204 : 403, headers: cors(origin) });
83
+ }
84
+ if (req.headers.get('Origin') && !origin) return json({ error: 'origin not allowed' }, 403, null);
85
+
86
+ // GET /tts → the voice list. Listing needs the key too, so the browser can't ask ElevenLabs
87
+ // itself (that endpoint is NOT public); the picker in a UI has to be fed from here.
88
+ if (req.method === 'GET' && url.pathname === '/tts') {
89
+ if (!env.ELEVENLABS_API_KEY) return json({ error: 'ELEVENLABS_API_KEY not configured' }, 500, origin);
90
+ if (await rateLimited(req, env, env.CONVO_RATE_LIMITER)) return json({ error: 'rate limited — try again in a minute' }, 429, origin); // listing hits the key too
91
+ const r = await fetch('https://api.elevenlabs.io/v2/voices?page_size=100', { headers: { 'xi-api-key': env.ELEVENLABS_API_KEY } });
92
+ const body = await r.json().catch(() => ({}));
93
+ if (!r.ok) return json({ error: body.detail?.message || 'voice list failed' }, 502, origin);
94
+ return json({ voices: (body.voices ?? []).map((v) => ({ id: v.voice_id, name: v.name })) }, 200, origin);
95
+ }
96
+
97
+ if (req.method !== 'POST') return json({ error: 'POST only' }, 405, origin);
98
+
99
+ // Everything below mints a credential or bills a provider. /stt/token mints at most one
100
+ // credential per 5 min so it takes the tight limiter; /tts fires once per spoken SENTENCE and
101
+ // /llm once per turn plus prefetch, so the conversation hot path gets its own, roomier window.
102
+ const limiter = url.pathname === '/stt/token' ? env.PAID_RATE_LIMITER : env.CONVO_RATE_LIMITER;
103
+ if (await rateLimited(req, env, limiter)) return json({ error: 'rate limited — try again in a minute' }, 429, origin);
104
+
105
+ if (url.pathname === '/stt/token') {
106
+ if (!env.DEEPGRAM_API_KEY) return json({ error: 'DEEPGRAM_API_KEY not configured' }, 500, origin);
107
+ // Deepgram mints the short-lived credential; the browser never sees the project key.
108
+ const r = await fetch('https://api.deepgram.com/v1/auth/grant', {
109
+ method: 'POST',
110
+ headers: { authorization: `Token ${env.DEEPGRAM_API_KEY}`, 'content-type': 'application/json' },
111
+ body: JSON.stringify({ ttl_seconds: 300 }),
112
+ });
113
+ const body = await r.json().catch(() => ({}));
114
+ if (!r.ok || !body.access_token) return json({ error: body.err_msg || body.error || 'grant failed' }, 502, origin);
115
+ return json({ token: body.access_token, expires_in: body.expires_in ?? 300 }, 200, origin);
116
+ }
117
+
118
+ if (url.pathname === '/tts') {
119
+ if (!env.ELEVENLABS_API_KEY) return json({ error: 'ELEVENLABS_API_KEY not configured' }, 500, origin);
120
+ const { text, voice_id, model_id, output_format } = await req.json().catch(() => ({}));
121
+ if (!text) return json({ error: 'text required' }, 400, origin);
122
+ // One sentence per call is the contract (that's how the library streams); a whole essay is
123
+ // not a sentence. Also the cost cap: a leaked demo URL should cost cents, not a plan.
124
+ if (text.length > 600) return json({ error: 'text too long' }, 413, origin);
125
+ const fmt = output_format || 'mp3_44100_64';
126
+ const r = await fetch(
127
+ `https://api.elevenlabs.io/v1/text-to-speech/${voice_id || 'JBFqnCBsd6RMkjVDRZzb'}?output_format=${fmt}`,
128
+ {
129
+ method: 'POST',
130
+ headers: { 'xi-api-key': env.ELEVENLABS_API_KEY, 'content-type': 'application/json' },
131
+ body: JSON.stringify({ text, model_id: model_id || 'eleven_flash_v2_5' }),
132
+ });
133
+ if (!r.ok) return json({ error: `elevenlabs ${r.status}: ${(await r.text().catch(() => '')).slice(0, 200)}` }, 502, origin);
134
+ // Stream the audio straight through — buffering it here would add latency to the first clip.
135
+ return new Response(r.body, { status: 200, headers: { 'content-type': 'audio/mpeg', ...cors(origin) } });
136
+ }
137
+
138
+ // Anthropic speaks /v1/chat/completions in OpenAI's format, so the SSE stream passes straight
139
+ // through to the built-in makeOpenAILLM — this route adds the key and an allowlist, nothing else.
140
+ // Streamed, never buffered: the first token is what the user waits for before any audio starts.
141
+ if (url.pathname === '/llm') {
142
+ if (!env.ANTHROPIC_API_KEY) return json({ error: 'ANTHROPIC_API_KEY not configured' }, 500, origin);
143
+ const body = await req.json().catch(() => ({}));
144
+ // Strip any caller-supplied system prompt: the persona is pinned HERE, like the model —
145
+ // otherwise a caller could prompt the hosted demo into long (= expensive) answers.
146
+ const messages = Array.isArray(body.messages) ? body.messages.filter((m) => m?.role !== 'system') : [];
147
+ if (!messages.length) return json({ error: 'messages required' }, 400, origin);
148
+ // Same reasoning as /tts's length cap: a leaked demo URL should cost cents, not a plan.
149
+ if (JSON.stringify(messages).length > 8000) return json({ error: 'conversation too long' }, 413, origin);
150
+ const r = await fetch('https://api.anthropic.com/v1/chat/completions', {
151
+ method: 'POST',
152
+ headers: { authorization: `Bearer ${env.ANTHROPIC_API_KEY}`, 'content-type': 'application/json' },
153
+ // Built from an ALLOWLIST, never `...body`: forwarding the caller's object would carry
154
+ // fields the size check above never measured — a huge `tools` schema bills as input tokens
155
+ // on a 200-byte `messages`, and `max_completion_tokens` is a second output cap whose
156
+ // precedence over max_tokens is undocumented. Only these four fields cross this boundary,
157
+ // and the model is pinned here so a caller can't ask for the priciest model on the account.
158
+ body: JSON.stringify({
159
+ model: env.ANTHROPIC_MODEL || 'claude-haiku-4-5-20251001',
160
+ // Clamped to [1,1024] as an integer — Math.min alone would forward a negative or
161
+ // fractional value, a guaranteed upstream 400 that still burned rate-limit quota.
162
+ max_tokens: Math.min(Math.max(Math.floor(Number(body.max_tokens)) || 512, 1), 1024),
163
+ messages: [{ role: 'system', content: env.SYSTEM_PROMPT || HOSTED_PERSONA }, ...messages],
164
+ stream: true,
165
+ }),
166
+ });
167
+ if (!r.ok) return json({ error: `anthropic ${r.status}: ${(await r.text().catch(() => '')).slice(0, 200)}` }, 502, origin);
168
+ return new Response(r.body, { status: 200, headers: { 'content-type': 'text/event-stream', ...cors(origin) } });
169
+ }
170
+
171
+ return json({ error: 'not found' }, 404, origin);
172
+ },
173
+ };
@@ -0,0 +1,39 @@
1
+ name = "voiceloop-token"
2
+ main = "src/index.js"
3
+ compatibility_date = "2026-01-01"
4
+
5
+ [vars]
6
+ ALLOWED_ORIGINS = "https://todoforai.github.io,http://localhost:8899,http://localhost:3000"
7
+ # Long-window quota (per IP, all paid routes): 300 calls / 6h — this is the actual wallet cap.
8
+ # The [[ratelimits]] bindings below only support 10s/60s periods, so this window lives in the
9
+ # Quota Durable Object instead; the 60s limiters remain as burst protection.
10
+ QUOTA_LIMIT = "300"
11
+ QUOTA_PERIOD_H = "6"
12
+
13
+ [durable_objects]
14
+ bindings = [{ name = "QUOTA", class_name = "Quota" }]
15
+
16
+ [[migrations]]
17
+ tag = "v1"
18
+ new_sqlite_classes = ["Quota"]
19
+
20
+ # /stt/token mints at most one credential per 5 min (300s TTL), so 30/min is already generous.
21
+ [[ratelimits]]
22
+ name = "PAID_RATE_LIMITER"
23
+ namespace_id = "1001"
24
+
25
+ [ratelimits.simple]
26
+ limit = 30
27
+ period = 60
28
+
29
+ # Conversation hot path: /tts fires once per spoken SENTENCE (5-10 per chatty reply) and /llm
30
+ # once per turn plus prefetch. Sized by cost, not call count — each /tts call is capped at
31
+ # 600 chars and each /llm call at 8000 chars in / 1024 tokens out on Haiku, so even 300/min
32
+ # from a leaked URL bounds the bill to cents.
33
+ [[ratelimits]]
34
+ name = "CONVO_RATE_LIMITER"
35
+ namespace_id = "1002"
36
+
37
+ [ratelimits.simple]
38
+ limit = 300
39
+ period = 60
@@ -2,7 +2,7 @@
2
2
  <!-- voiceloop demo: the whole loop behind one button, plus the numbers that say
3
3
  whether it feels seamless. No bundler, no framework, no dependencies.
4
4
 
5
- from a clone: npx serve . → /examples/demo.html
5
+ from a clone: npx serve . → http://localhost:3000/
6
6
  from npm: npx serve node_modules/@todoforai/voiceloop
7
7
 
8
8
  file:// won't work — mic + module imports need http://localhost or https.
@@ -230,15 +230,28 @@
230
230
  min-height pins the result at 44px instead of trusting 13px text to round up to it. */
231
231
  @media (pointer: coarse) { summary { padding:11px 0; min-height:44px; box-sizing:border-box } }
232
232
  summary::before { content:'▸ ' } details[open] summary::before { content:'▾ ' }
233
- .grid { display:flex; flex-wrap:wrap; gap:10px; padding:12px 0 4px }
233
+ /* One row per pipeline stage, in the order the audio flows (LLM group first only because `mode`
234
+ is the master switch). A flat 11-field soup made "which url goes with which provider" a puzzle;
235
+ the heading answers it before the placeholder has to. */
236
+ .grid { display:flex; flex-direction:column; gap:14px; padding:12px 0 4px }
237
+ .set { display:flex; flex-wrap:wrap; gap:10px; align-items:flex-end }
238
+ .set h4 { flex-basis:100%; margin:0 0 -4px; font-size:10px; font-weight:600;
239
+ letter-spacing:.1em; text-transform:uppercase; color:var(--dim); opacity:.65 }
234
240
  label { font-size:11px; color:var(--dim); display:flex; flex-direction:column; gap:4px }
241
+ /* A field the current choices ignore must LOOK ignored, not merely refuse focus. */
242
+ input:disabled, select:disabled { opacity:.4 }
235
243
  #log { font:11px/1.5 ui-monospace,monospace; color:var(--dim);
236
244
  max-height:220px; overflow:auto; white-space:pre-wrap; padding-top:8px }
237
245
  /* Which code is actually running — the demo must never let you mistake the browser's own
238
- recognizer for voiceloop's pipeline, or an echo stand-in for a real LLM. */
239
- #pipeline { font-size:11.5px; color:var(--dim); text-align:center; line-height:1.7; max-width:46ch }
246
+ recognizer for voiceloop's pipeline, or an echo stand-in for a real LLM. Sits right above the
247
+ collapsed settings (it describes what they configure), not in the hero area under the orb.
248
+ Arrow chains are .nb (no-break): "Silero VAD →<br>feed" reads as a broken message. */
249
+ #pipeline { width:100%; font-size:11.5px; color:var(--dim); line-height:1.7;
250
+ border-top:1px solid var(--line); padding-top:10px }
251
+ #pipeline:empty { display:none }
240
252
  #pipeline b { color:#c8d0da; font-weight:600 }
241
253
  #pipeline .warn { color:#e0a94a }
254
+ #pipeline .nb { white-space:nowrap }
242
255
  </style>
243
256
  </head>
244
257
  <body>
@@ -250,7 +263,6 @@
250
263
  </div>
251
264
  <div id="status">click to start · then just talk</div>
252
265
  <div id="lat"></div>
253
- <div id="pipeline"></div>
254
266
 
255
267
  <div id="chat"></div>
256
268
 
@@ -259,21 +271,41 @@
259
271
  <button class="btn" id="send">Send</button>
260
272
  </div>
261
273
 
274
+ <div id="pipeline"></div>
275
+
262
276
  <details>
263
277
  <summary>settings</summary>
264
278
  <div class="grid">
265
- <label>mode<select id="mode">
266
- <option value="echo">echo — no keys, no backend</option>
267
- <option value="llm">full loop — your LLM</option>
268
- </select></label>
269
- <label>LLM url<input id="llmUrl" size="30" value="http://localhost:11434/v1/chat/completions"></label>
270
- <label>model<input id="model" size="14" value="llama3.2"></label>
271
- <label>STT<select id="sttProvider"></select></label>
272
- <label>STT token url<input id="sttTokenUrl" size="30" placeholder="your backend route (cloud STT only)"></label>
273
- <label>api key<input id="apiKey" size="18" placeholder="Bearer for LLM · X-API-Key for token url"></label>
274
- <label>lang<select id="sttLang"></select></label>
275
- <label>voice<select id="voice"></select></label>
276
- <label>mic<select id="micDeviceId"><option value="">default</option></select></label>
279
+ <div class="set">
280
+ <h4>llm</h4>
281
+ <label>mode<select id="mode">
282
+ <option value="echo">echo — no keys, no backend</option>
283
+ <option value="hosted">full loop — hosted claude haiku</option>
284
+ <option value="llm">full loop — your LLM</option>
285
+ </select></label>
286
+ <label>LLM url<input id="llmUrl" size="30" value="http://localhost:11434/v1/chat/completions"></label>
287
+ <label>model<input id="model" size="14" value="qwen3"></label>
288
+ </div>
289
+ <div class="set">
290
+ <h4>speech to text</h4>
291
+ <label>provider<select id="sttProvider"></select></label>
292
+ <label>language<select id="sttLang"></select></label>
293
+ <label>token url<input id="sttTokenUrl" size="30" placeholder="your backend route (see examples/token-worker)"></label>
294
+ </div>
295
+ <div class="set">
296
+ <h4>text to speech</h4>
297
+ <label>provider<select id="ttsProvider">
298
+ <option value="piper">piper — free, in-browser</option>
299
+ <option value="elevenlabs">elevenlabs — needs TTS url</option>
300
+ </select></label>
301
+ <label>voice<select id="voice"></select></label>
302
+ <label>TTS url<input id="ttsUrl" size="30" placeholder="your backend route (see examples/token-worker)"></label>
303
+ </div>
304
+ <div class="set">
305
+ <h4>input &amp; auth</h4>
306
+ <label>mic<select id="micDeviceId"><option value="">default</option></select></label>
307
+ <label>api key<input id="apiKey" size="26" placeholder="Bearer for your LLM · X-API-Key for token url"></label>
308
+ </div>
277
309
  </div>
278
310
  <div class="bar">
279
311
  <button class="btn" id="clear">Clear</button>
@@ -286,7 +318,7 @@
286
318
  </main>
287
319
 
288
320
  <script type="module">
289
- import { VoiceAgent, PiperTTS, unlockAudio, prewarmVoice, STT_PROVIDERS, sttSelfCaptures, LANG_NAMES } from '../src/index.js';
321
+ import { VoiceAgent, PiperTTS, ElevenLabsTTS, makeOpenAILLM, unlockAudio, prewarmVoice, STT_PROVIDERS, sttSelfCaptures, LANG_NAMES } from './src/index.js';
290
322
 
291
323
  // Load VAD/WASM + build the Silero session while the page sits idle — but ONLY for a provider that
292
324
  // will actually use it. A selfCapture provider (webspeech) builds no pipeline, so prewarming there
@@ -302,7 +334,7 @@ const fill = (id, entries) => $(id).append(...entries.map(([value, textContent])
302
334
  // second source of truth that silently rots — and a demo that lies about what the library supports.
303
335
  fill('sttProvider', Object.keys(STT_PROVIDERS).map(p => [p, p === 'webspeech' ? 'webspeech — free, in-browser' : p]));
304
336
  fill('sttLang', Object.entries(LANG_NAMES).map(([code, name]) => [code, `${code} — ${name}`]));
305
- const SETTINGS = ['mode','llmUrl','model','sttProvider','sttTokenUrl','apiKey','sttLang','voice','micDeviceId'];
337
+ const SETTINGS = ['mode','llmUrl','model','sttProvider','sttTokenUrl','apiKey','sttLang','ttsProvider','ttsUrl','micDeviceId'];
306
338
  for (const id of SETTINGS) {
307
339
  const v = localStorage.getItem('vl.' + id); if (v) $(id).value = v;
308
340
  $(id).addEventListener('change', () => {
@@ -317,7 +349,25 @@ const log = (...a) => {
317
349
 
318
350
  // ── transcript ──────────────────────────────────────────────────────────────
319
351
  const chat = $('chat');
320
- let interimEl = null, aiEl = null, vadOff = null, spokenLen = 0;
352
+ let interimEl = null, vadOff = null, spokenLen = 0;
353
+ // The transcript's single source of truth for replies: turn id → its bubble. EVERY assistant event
354
+ // routes through it — there is no separate "current bubble" to fall out of sync with. A turn's
355
+ // entry lives from its first painted event to its settle (final / drop / reply-scoped error), so a
356
+ // straggler from an interrupted turn finds its own bubble by id, wherever it sits in the chat.
357
+ // Ids are per-agent and monotonic, so `liveTurn` (the highest id seen) tells present from past:
358
+ // smaller = a straggler correcting history — it paints its bubble but drives no cosmetics.
359
+ const turnEls = new Map();
360
+ let liveTurn = 0; // turns count from 1; reset with the agent that issued them
361
+
362
+ // The dimmed tail is "written but not yet heard": on a final, a non-empty tail means the reply was
363
+ // cut off mid-sentence. Shared by the live path and the late-straggler path so an interrupted reply
364
+ // renders identically no matter when its last event lands.
365
+ const paintTail = (el, e) => {
366
+ const tail = (e.full ?? e.text).slice(e.text.length);
367
+ if (tail) el.append(Object.assign(document.createElement('span'), { className:'tail', textContent:tail }));
368
+ el.classList.toggle('cut', !!(e.final && tail.trim())); // toggle: a late repaint must also CLEAR a stale cut mark
369
+ return tail;
370
+ };
321
371
  const toolEls = new Map();
322
372
  const bubble = (cls, text = '') => {
323
373
  const el = document.createElement('div');
@@ -333,11 +383,30 @@ const showLat = () => $('lat').textContent = [
333
383
  tAudio && `first sound ${tAudio - t0}ms`,
334
384
  ].filter(Boolean).join(' · ');
335
385
 
336
- // undefined (not '') when unset — so PiperTTS applies its OWN default voice instead of this page
337
- // re-declaring one. The picker is populated async below and adopts whatever it resolved to.
338
- const tts = new PiperTTS(localStorage.getItem('vl.voice') || undefined);
386
+ // `tts` is a LET built ONLY by applyTts() — including the first one, from the RESTORED settings.
387
+ // It used to be initialised here with a hardcoded PiperTTS, which ignored a saved elevenlabs
388
+ // choice: the page came back on Piper every reload while the form still read "elevenlabs".
389
+ let tts; // assigned by applyTts(), below buildTts()'s definition
339
390
  let agent = null;
340
391
 
392
+ // ElevenLabs needs a backend route (the key must not reach the page — see examples/token-worker).
393
+ // Without one this returns NULL rather than quietly substituting Piper: falling back would speak in
394
+ // a different voice than the form promises, and "elevenlabs selected, Piper's voice coming out" is
395
+ // indistinguishable from a broken integration. No synthesizer → the page says so and won't start.
396
+ const buildTts = () => {
397
+ const url = $('ttsUrl').value.trim();
398
+ if ($('ttsProvider').value === 'elevenlabs') {
399
+ if (!url) return null;
400
+ const voice = localStorage.getItem('vl.elVoice') || undefined; // Piper ids are not EL ids — keep them apart
401
+ return new ElevenLabsTTS({ ttsUrl: url, ...(voice ? { voiceId: voice } : {}) });
402
+ }
403
+ return new PiperTTS(localStorage.getItem('vl.voice') || undefined);
404
+ };
405
+ // The demo's own backend (examples/token-worker). Same origin-allowlisted Worker as /stt/token and
406
+ // /tts, so 'hosted' costs the visitor no key at all — the point of a demo you can just click.
407
+ const HOSTED_LLM_URL = 'https://voiceloop-token.havlikmarcell.workers.dev/llm';
408
+ const TTS_MISSING = 'elevenlabs needs a TTS url — paste your backend route (see examples/token-worker) or switch TTS back to piper';
409
+
341
410
  // Echo mode: a stand-in "LLM" that repeats the turn VERBATIM — exercises the entire
342
411
  // VAD→STT→…→TTS loop, barge-in included, with no key and no backend. Nothing is added to the
343
412
  // text: a reply that is exactly the transcript is the whole point, since any prefix would be
@@ -371,15 +440,29 @@ const describePipeline = (a) => {
371
440
  const self = a?.stt ? !!a.stt.selfCapture : sttSelfCaptures($('sttProvider').value);
372
441
  const stt = self
373
442
  ? `<b>${$('sttProvider').value}</b> — the browser's own recognizer owns the mic <span class="warn">(voiceloop's VAD + STT pipeline bypassed)</span>${IS_IOS ? IOS_WEBSPEECH_WARNING : ''}`
374
- : `<b>${$('sttProvider').value}</b> via voiceloop's pipeline — Silero VAD → feed → commit`;
443
+ : `<b>${$('sttProvider').value}</b> via voiceloop's pipeline — <span class="nb">Silero VAD → feed → commit</span>`;
444
+ // new URL() throws on a half-typed url — a banner must never take the page down with it.
445
+ const host = (u) => { try { return new URL(u).host; } catch { return u || 'unset'; } };
375
446
  const llm = echo
376
447
  ? '<span class="warn"><b>echo</b> — no model: replies repeat your words back</span>'
377
- : `<b>${$('model').value || 'your model'}</b> at ${new URL($('llmUrl').value).host}`;
448
+ : hosted()
449
+ ? `<b>claude haiku</b> — hosted by this demo, no key needed`
450
+ : `<b>${$('model').value || 'your model'}</b> at ${host($('llmUrl').value)}`;
378
451
  return `STT: ${stt}<br>LLM: ${llm}`;
379
452
  };
380
453
  const showPipeline = () => { $('pipeline').innerHTML = describePipeline(agent); };
381
454
  // Hoisted (function declaration) so the settings listeners above can call it before this point.
382
- function onSettingChanged() { showPipeline(); prewarmIfPipelined(); }
455
+ function onSettingChanged() {
456
+ // Say it with the UI, not just the banner: fields the current choices IGNORE are disabled, so the
457
+ // form can't imply that a typed url or key is doing something. (ttsUrl is applyTts()'s to
458
+ // disable — it lives on the provider-change path, not here.)
459
+ const custom = $('mode').value === 'llm';
460
+ const self = sttSelfCaptures($('sttProvider').value); // webspeech: no token route, no pipeline
461
+ for (const id of ['llmUrl', 'model']) $(id).disabled = !custom;
462
+ $('sttTokenUrl').disabled = self;
463
+ $('apiKey').disabled = !custom && self; // used as LLM Bearer OR token-route X-API-Key — dead only when neither hop exists
464
+ showPipeline(); prewarmIfPipelined();
465
+ }
383
466
 
384
467
  // ── the orb's level ─────────────────────────────────────────────────────────
385
468
  // One CSS var, --lvl (0..1), drives size/glow/rim/halo. Three sources feed it, in order of truth:
@@ -442,15 +525,20 @@ const setState = s => {
442
525
  document.body.dataset.state = micOn || s !== 'listening' ? s : 'idle';
443
526
  if (modelPct === null) $('status').textContent = STATUS[s]?.() ?? s;
444
527
  };
445
- tts.setOnModelProgress?.((loaded, total) => {
528
+ // Re-attached on every rebuild: the hook lives on the TTS instance, so a new synthesizer would
529
+ // otherwise download its 60MB model in silence again.
530
+ const wireModelProgress = () => tts?.setOnModelProgress?.((loaded, total) => {
446
531
  const done = total && loaded >= total;
447
532
  modelPct = done ? null : total ? Math.floor((loaded / total) * 100) : 0;
448
533
  $('status').textContent = done ? (STATUS[document.body.dataset.state]?.() ?? '')
449
534
  : `downloading the voice — ${modelPct}% (${(loaded / 1048576).toFixed(0)}MB${total ? ` of ${(total / 1048576).toFixed(0)}MB` : ''}, once)`;
450
- });
535
+ }); // applyTts() attaches this — including for the FIRST synthesizer, which it also builds
451
536
 
452
537
  function onEvent(e) {
453
- if (e.type !== 'stt' || e.turnComplete) log(e);
538
+ // High-rate streams stay out of the log — STT interims, the TTS spoken-cursor draft (final:false
539
+ // assistant events), and level ticks (4/s, the orb already shows them live). Logging them evicts
540
+ // everything that matters from the capped buffer; only settled events land here.
541
+ if ((e.type !== 'stt' || e.turnComplete) && (e.type !== 'assistant' || e.final) && e.type !== 'level') log(e);
454
542
  switch (e.type) {
455
543
  case 'state':
456
544
  // The agent can stop itself (mic denied, STT socket died) — the mic light
@@ -474,24 +562,45 @@ function onEvent(e) {
474
562
  ripple(); // your turn closed — the instant the latency clock starts
475
563
  interimEl?.remove(); interimEl = null;
476
564
  bubble('user', e.text);
477
- aiEl = null; spokenLen = 0; t0 = Date.now(); tText = tAudio = 0; $('lat').textContent = '';
565
+ t0 = Date.now(); tText = tAudio = 0; $('lat').textContent = '';
478
566
  } else {
479
567
  interimEl ??= bubble('user interim');
568
+ // A turn can die WITHOUT turnComplete (echo-dropped, retracted to '', reply error) — the
569
+ // interim bubble then survives at its old position and `??=` reuses it, painting the NEXT
570
+ // turn's live speech ABOVE messages appended since. append() moves an existing node.
571
+ chat.append(interimEl);
480
572
  interimEl.textContent = `${e.committed ?? ''} ${e.text}`.trim();
481
573
  interimEl.scrollIntoView({ block:'nearest' });
482
574
  }
483
575
  break;
484
576
  case 'assistant': {
485
- // Empty and non-final means "that draft never happened" (interrupted before
486
- // a word was spoken) — drop the bubble rather than blank it.
487
- if (!e.text && !e.final) { aiEl?.remove(); aiEl = null; spokenLen = 0; break; }
577
+ // ONE routing rule: this event paints the bubble of ITS turn — looked up in turnEls, created
578
+ // on the turn's first painted event. A barge-in ends turn N while its final is still in
579
+ // flight, so it can land after turn N+1 exists; keyed by id, it still finds its own bubble.
580
+ // `live` (id >= the highest seen) does NOT change where text goes — it only gates cosmetics:
581
+ // a straggler correcting history must not pulse the orb, stamp latency, ripple, or scroll.
582
+ const live = e.turn >= liveTurn;
583
+ if (live && e.turn > liveTurn) { liveTurn = e.turn; spokenLen = 0; } // a NEW reply starts its cursor at 0 (notify() has no stt event to reset it)
584
+ let el = turnEls.get(e.turn);
585
+ // "That draft never happened" (interrupted before a word was spoken) → drop the bubble
586
+ // rather than blank it. Told apart from the FIRST progress tick — also empty (cursor at 0)
587
+ // but carrying `full` — which must NOT tear the live bubble down.
588
+ if (!e.text && !e.final && e.full == null) { el?.remove(); turnEls.delete(e.turn); break; }
589
+ if (!el) {
590
+ if (!live) break; // a settled turn with no bubble has no correct place to paint — ignore
591
+ // Tag the bubble (never the text) when no model produced it — the reply is a verbatim
592
+ // echo, and only the UI should say so.
593
+ el = bubble($('mode').value === 'echo' ? 'ai echoed' : 'ai');
594
+ turnEls.set(e.turn, el);
595
+ }
596
+ el.textContent = e.text; // spoken so far
597
+ const tail = paintTail(el, e);
598
+ if (e.final) turnEls.delete(e.turn); // settled: stop tracking (the bubble stays)
599
+ if (!live) break; // history corrected; the present is untouched
488
600
  if (!tText && t0) { tText = Date.now(); showLat(); }
489
- // `full` only exists once the TTS progress cursor is live — i.e. a clip is
490
- // actually playing. The 'speaking' state is just the first token, not sound.
601
+ // `full` only exists once the TTS progress cursor is live — i.e. a clip is actually playing.
602
+ // The 'speaking' state is just the first token, not sound.
491
603
  if (e.full != null && t0 && !tAudio) { tAudio = Date.now(); showLat(); }
492
- // Tag the bubble (never the text) when no model produced it — the reply is a verbatim echo,
493
- // and only the UI should say so.
494
- aiEl ??= bubble($('mode').value === 'echo' ? 'ai echoed' : 'ai');
495
604
  // The agent's own voice has no level event (TTS plays through an internal AEC sink), so drive
496
605
  // the orb from how fast the spoken CURSOR advances — the orb rides its own speech. Only events
497
606
  // carrying `full` have a cursor; a pre-audio streaming draft would otherwise run spokenLen up
@@ -500,33 +609,37 @@ function onEvent(e) {
500
609
  setLevel(0.35 + Math.min(0.6, Math.max(0, e.text.length - spokenLen) * 0.09));
501
610
  spokenLen = e.text.length;
502
611
  }
503
- aiEl.textContent = e.text; // spoken so far
504
- const tail = (e.full ?? e.text).slice(e.text.length); // generated, not yet heard
505
- if (tail) aiEl.append(Object.assign(document.createElement('span'), { className:'tail', textContent:tail }));
506
- // Cut short: the final says what was HEARD (text) and what was written (full) — a gap between
507
- // them is a barge-in, the one thing the transcript would otherwise render as a short answer.
508
- // (trim: a natural finish can leave trailing whitespace in `full` — not an interruption)
509
- if (e.final) {
510
- if (tail.trim()) { aiEl.classList.add('cut'); ripple(); }
511
- spokenLen = 0;
512
- }
513
- aiEl.scrollIntoView({ block:'nearest' });
612
+ if (e.final && tail.trim()) ripple();
613
+ el.scrollIntoView({ block:'nearest' });
514
614
  break;
515
615
  }
516
616
  case 'tool': {
517
- // running and result arrive as two events with one id: update, don't stack.
617
+ // running and result arrive as two events with one id: update, don't stack. The id stays
618
+ // mapped after the outcome — a tool finalized as "interrupted" whose detached run settles
619
+ // later corrects that SAME chip in place (re-emits are deduped library-side, and a dead
620
+ // agent's late event can't get here at all: makeAgent's onEvent is identity-guarded).
518
621
  const el = toolEls.get(e.id) ?? bubble('tool');
519
622
  if (e.id) toolEls.set(e.id, el);
520
623
  el.textContent = `🔧 ${e.name}(${JSON.stringify(e.args)})${e.running ? ' …' : ` → ${JSON.stringify(e.result ?? '')}`.slice(0,200)}`;
521
624
  break;
522
625
  }
523
626
  case 'echo': bubble('note', `self-echo dropped: ${e.text}`); break;
524
- case 'error': bubble('err', e.error); break;
627
+ case 'error':
628
+ bubble('err', e.error);
629
+ // A reply-scoped error (it carries the turn) ends that reply without a final assistant event —
630
+ // release its bubble from tracking (the partial text stays visible) so the map can't grow.
631
+ if (e.turn != null) turnEls.delete(e.turn);
632
+ break;
525
633
  }
526
634
  }
527
635
 
528
636
  const muted = id => $(id).dataset.on === '1';
637
+ const hosted = () => $('mode').value === 'hosted';
529
638
  const makeAgent = () => {
639
+ // Hosted mode gets its OWN llm client rather than blanking the shared `apiKey`: that one field
640
+ // also authenticates the STT token route (X-API-Key), so clearing it would silently break a
641
+ // custom STT backend. This drops the Authorization header on the LLM hop ONLY — the hosted route
642
+ // holds its own key and pins its own model, so neither belongs in the request.
530
643
  const a = new VoiceAgent({
531
644
  llmUrl: $('llmUrl').value.trim(),
532
645
  model: $('model').value.trim(),
@@ -537,17 +650,38 @@ const makeAgent = () => {
537
650
  micDeviceId: $('micDeviceId').value,
538
651
  persona: 'You are a concise, friendly voice assistant. Keep answers to a sentence or two.',
539
652
  tts,
540
- llm: $('mode').value === 'echo' ? echoLLM : undefined,
541
- onEvent,
653
+ llm: $('mode').value === 'echo' ? echoLLM
654
+ : hosted() ? makeOpenAILLM({ llmUrl: HOSTED_LLM_URL }) // no apiKey, no model: the route owns both
655
+ : undefined, // 'llm' → built from llmUrl/model/apiKey above
656
+ // Identity-guarded: destroy() stops the loop but a DETACHED tool result still reports through
657
+ // this agent's onEvent after it. Turn/tool ids are per-agent, so such a late event could match
658
+ // the NEXT agent's same-numbered entry and repaint its bubble — dropping everything from a
659
+ // non-current agent closes that whole class, instead of chasing each map it could touch.
660
+ onEvent: (e) => { if (agent === a) onEvent(e); },
542
661
  });
543
662
  a.setMuted(muted('muteMic')); a.setTtsMuted(muted('muteTts')); // the toggles survive a rebuild
544
663
  log('agent created; stt =', a.sttProvider); showPipeline(); // now reflects the LIVE session, not just the form
545
664
  return a;
546
665
  };
547
666
 
667
+ // The ONLY way the page discards an agent. Turn ids are per-agent (each starts counting at 1), so
668
+ // tracking state keyed by them must die with the agent that issued them: a surviving turnEls entry
669
+ // would match the NEXT agent's same-numbered reply and hand it the dead agent's bubble to
670
+ // overwrite. (destroy() is terminal — the old agent can emit nothing further, so nothing is lost.)
671
+ const dropAgent = (why) => {
672
+ if (!agent) return;
673
+ agent.destroy(); agent = null; // null FIRST-equivalent: the identity guard now drops every late event from it
674
+ liveTurn = 0; spokenLen = 0; turnEls.clear(); // the next agent counts turns from 1 again
675
+ toolEls.clear(); // tool ids are provider-issued and can repeat — same hazard
676
+ interimEl?.remove(); interimEl = null; // a half-heard interim went nowhere; the dead agent will never settle it
677
+ clearTimeout(vadOff); document.body.dataset.vad = '0';
678
+ t0 = tText = tAudio = 0; $('lat').textContent = ''; // latency legend described the dead session
679
+ log(why, '→ agent rebuilt on next turn');
680
+ };
681
+
548
682
  // ── one button does everything ──────────────────────────────────────────────
549
683
  const stop = () => {
550
- agent?.destroy(); agent = null; micOn = false;
684
+ dropAgent('stop'); micOn = false;
551
685
  document.body.dataset.vad = '0'; setState('idle');
552
686
  onSettingChanged(); // describe the restored settings, and prewarm only if they'll use the VAD
553
687
  };
@@ -557,6 +691,7 @@ $('mic').onclick = async () => {
557
691
  // or not — it cancels the turn and keeps the session.
558
692
  if (agent && (agent.state === 'thinking' || agent.state === 'speaking')) return agent.interrupt();
559
693
  if (micOn) return stop();
694
+ if (!tts) return bubble('err', TTS_MISSING); // no voice → refuse to start, rather than open the mic on a loop that can't reply
560
695
  agent ??= makeAgent();
561
696
  micOn = true; setState('listening');
562
697
  try { await agent.start(); }
@@ -565,6 +700,7 @@ $('mic').onclick = async () => {
565
700
 
566
701
  $('send').onclick = () => {
567
702
  const text = $('typed').value.trim(); if (!text) return;
703
+ if (!tts) return bubble('err', TTS_MISSING);
568
704
  unlockAudio();
569
705
  agent ??= makeAgent();
570
706
  agent.sendUserText(text); // emits its own user bubble via the stt turnComplete event
@@ -573,7 +709,7 @@ $('send').onclick = () => {
573
709
  $('typed').onkeydown = e => { if (e.key === 'Enter') $('send').click(); };
574
710
 
575
711
  $('clear').onclick = () => { chat.textContent = ''; $('log').textContent = ''; $('lat').textContent = '';
576
- interimEl = aiEl = null; toolEls.clear(); };
712
+ interimEl = null; turnEls.clear(); toolEls.clear(); };
577
713
  const toggle = (id, fn) => $(id).onclick = () => {
578
714
  const on = $(id).dataset.on !== '1';
579
715
  $(id).dataset.on = on ? '1' : '0';
@@ -585,19 +721,58 @@ toggle('muteTts', on => agent?.setTtsMuted(on));
585
721
 
586
722
  // Voice switches live. The rest are constructor options, so they can only apply
587
723
  // to a NEW agent — drop the current one so the next turn builds it from these.
588
- $('voice').addEventListener('change', () => { tts.setVoice($('voice').value); log('voice →', $('voice').value); });
589
- for (const id of SETTINGS.filter(i => i !== 'voice'))
724
+ $('voice').addEventListener('change', () => {
725
+ // Piper ids and ElevenLabs voice ids share one picker but not one namespace — store them apart so
726
+ // switching provider and back doesn't hand Piper an EL id (which resolves to no voice at all).
727
+ if (!tts) return;
728
+ localStorage.setItem(tts instanceof ElevenLabsTTS ? 'vl.elVoice' : 'vl.voice', $('voice').value);
729
+ tts.setVoice($('voice').value); log('voice →', $('voice').value);
730
+ });
731
+ for (const id of SETTINGS.filter(i => !['ttsProvider','ttsUrl'].includes(i)))
590
732
  $(id).addEventListener('change', () => {
591
733
  if (micOn) return bubble('note', `stop the mic to apply ${id}`);
592
- if (agent) { agent.destroy(); agent = null; log(id, '→ agent rebuilt on next turn'); }
734
+ dropAgent(id);
593
735
  });
594
736
 
737
+ // The synthesizer is a CONSTRUCTOR choice, so switching provider replaces the object — and the
738
+ // agent holds a reference to it, so it has to be rebuilt too (same rule as the settings above).
739
+ const applyTts = () => {
740
+ $('ttsUrl').disabled = $('ttsProvider').value !== 'elevenlabs';
741
+ tts = buildTts();
742
+ if (!tts) bubble('note', TTS_MISSING);
743
+ wireModelProgress();
744
+ dropAgent('tts');
745
+ loadVoices();
746
+ log('tts →', $('ttsProvider').value);
747
+ };
748
+ let ttsUrlTimer;
749
+ for (const id of ['ttsProvider', 'ttsUrl'])
750
+ for (const ev of ['change', 'input'])
751
+ $(id).addEventListener(ev, () => {
752
+ if (micOn) return bubble('note', `stop the mic to apply ${id}`);
753
+ localStorage.setItem('vl.' + id, $(id).value);
754
+ clearTimeout(ttsUrlTimer);
755
+ ttsUrlTimer = setTimeout(applyTts, ev === 'input' ? 400 : 0); // don't rebuild on every keystroke
756
+ });
757
+
595
758
  // ── device + voice lists ────────────────────────────────────────────────────
596
- tts.voices().then(vs => {
597
- fill('voice', vs.map(v => [v.id, v.id]));
598
- $('voice').value = localStorage.getItem('vl.voice') || tts.voiceId; // the library's own default, not a copy of it
599
- tts.setVoice($('voice').value);
600
- }).catch(e => log('voice list failed', String(e)));
759
+ // Re-run on every rebuild: the voice list belongs to the synthesizer, and a stale Piper list would
760
+ // offer ids ElevenLabs can't speak.
761
+ function loadVoices() {
762
+ $('voice').replaceChildren();
763
+ if (!tts) return fill('voice', [['', TTS_MISSING]]); // nothing to ask — say why, where the user is looking
764
+ // Ask the synthesizer that is ACTUALLY live, never the form: buildTts() is the one place that
765
+ // decides. Re-reading the form here would be a second copy of that rule, free to disagree with
766
+ // it — and a picker offering ids the live voice can't speak is exactly the bug this avoids.
767
+ const key = tts instanceof ElevenLabsTTS ? 'vl.elVoice' : 'vl.voice';
768
+ tts.voices().then(vs => {
769
+ if (!vs.length) return fill('voice', [['', "couldn't reach the TTS url — check the route + its origin allowlist"]]);
770
+ fill('voice', vs.map(v => [v.id, v.name || v.id]));
771
+ $('voice').value = localStorage.getItem(key) || tts.voiceId; // the library's own default, not a copy of it
772
+ if ($('voice').value) tts.setVoice($('voice').value);
773
+ }).catch(e => log('voice list failed', String(e)));
774
+ }
775
+ applyTts();
601
776
  navigator.mediaDevices?.enumerateDevices().then(ds => {
602
777
  for (const d of ds.filter(d => d.kind === 'audioinput'))
603
778
  $('micDeviceId').append(Object.assign(document.createElement('option'),
@@ -607,6 +782,7 @@ navigator.mediaDevices?.enumerateDevices().then(ds => {
607
782
  setState('idle');
608
783
  onSettingChanged(); // the banner must describe the restored settings from the FIRST frame, not only after a click
609
784
  window.agent = () => agent; // console poking: agent().history, …
785
+ window.tts = () => tts; // …and the live synthesizer: tts().voiceId, tts().constructor.name
610
786
  </script>
611
787
  </body>
612
788
  </html>
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@todoforai/voiceloop",
3
- "version": "0.1.8",
3
+ "version": "0.1.9",
4
4
  "description": "Fast, interruptible browser voice agent: VAD → STT → LLM → TTS with real barge-in, echo filtering, streaming first-sentence TTS and tap-to-seek. Zero runtime dependencies.",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
@@ -17,7 +17,8 @@
17
17
  "files": [
18
18
  "src",
19
19
  "!src/*.test.js",
20
- "examples"
20
+ "examples",
21
+ "index.html"
21
22
  ],
22
23
  "scripts": {
23
24
  "test": "node --test src/*.test.js"
package/src/index.d.ts CHANGED
@@ -5,7 +5,11 @@
5
5
  export type VoiceAgentEvent =
6
6
  | { type: 'state'; state: 'idle' | 'listening' | 'thinking' | 'speaking' }
7
7
  | { type: 'stt'; turnComplete: boolean; text: string; ms: number; committed?: string } // text=live tail; committed=locked segments (Speechmatics AddTranscript); turnComplete=user turn ended → send to LLM
8
- | { type: 'assistant'; text: string; final: boolean; full?: string } // text=spoken-so-far (solid, advances live as TTS plays); full=complete answer → tail full.slice(text.length) not-yet-spoken (dim)
8
+ // text=spoken-so-far (solid, advances live as TTS plays); full=complete answer → tail
9
+ // full.slice(text.length) not-yet-spoken (dim). turn identifies the reply: a barge-in ends turn N
10
+ // while its final event is still in flight, so it can land AFTER turn N+1 started — route by turn,
11
+ // not by "the current bubble", or the late text overwrites the newer reply.
12
+ | { type: 'assistant'; text: string; final: boolean; full?: string; turn: number }
9
13
  | { type: 'vad'; active: boolean } // VAD hears speech?
10
14
  // Mic loudness, 0..1 on a perceptual curve, ~every 256ms while capturing. Cosmetic only (meters,
11
15
  // a reacting orb) — nothing in the pipeline reads it. Never emitted for a selfCapture STT
@@ -20,7 +24,7 @@ export type VoiceAgentEvent =
20
24
  // Did that speculation pay off? hit = the real reply opened with the presynthesized text (clip
21
25
  // reused, latency saved); miss = it didn't (clip discarded). `pre` is what we had guessed.
22
26
  | { type: 'diag'; diag: 'presynth-hit' | 'presynth-miss'; text: string; pre: string }
23
- | { type: 'error'; error: string };
27
+ | { type: 'error'; error: string; turn?: number }; // turn present = that reply's TERMINAL event (a reply-killing TTS failure emits no assistant final)
24
28
 
25
29
  export interface VoiceTool {
26
30
  description?: string;
@@ -326,9 +330,10 @@ export class StreamingTTS implements VoiceTTS {
326
330
  seek(nwIndex: number): boolean;
327
331
  }
328
332
  /** Human-grade cloud voices. Pass `ttsUrl` (your backend proxy, key stays server-side); `apiKey`
329
- * talks to ElevenLabs directly and is dev-only. */
333
+ * talks to ElevenLabs directly and is dev-only. `headers` authenticate proxy requests
334
+ * (e.g. { 'X-API-Key': ... }) and are never sent on a direct ElevenLabs call. */
330
335
  export class ElevenLabsTTS extends StreamingTTS {
331
- constructor(options?: { ttsUrl?: string; apiKey?: string; voiceId?: string; modelId?: string; format?: string });
336
+ constructor(options?: { ttsUrl?: string; apiKey?: string; voiceId?: string; modelId?: string; format?: string; headers?: Record<string, string> });
332
337
  warm(): Promise<void>;
333
338
  }
334
339
 
package/src/stt.js CHANGED
@@ -187,6 +187,7 @@ export function makeElevenLabsSTT(opts) {
187
187
  open() { if (!isClosed() && (!ws || ws.readyState >= 2)) open(); },
188
188
  feed(i16) {
189
189
  if (!isClosed() && (!ws || ws.readyState >= 2)) open();
190
+ if (!sentSinceCommit) sttStart = performance.now(); // first audio of a fresh turn anchors its ms clock (audio is VAD-gated ≈ speech start)
190
191
  sendAudio(i16); sentSinceCommit = true;
191
192
  },
192
193
  commit() {
@@ -412,6 +413,7 @@ export function makeSpeechmaticsSTT(opts) {
412
413
  open() { if (!isClosed() && (!ws || ws.readyState >= 2)) open(); },
413
414
  feed(i16) {
414
415
  if (!isClosed() && (!ws || ws.readyState >= 2)) open();
416
+ if (!sentSinceCommit) sttStart = performance.now(); // first audio of a fresh turn anchors its ms clock (audio is VAD-gated ≈ speech start)
415
417
  sentSamples += i16.length; // stream position (counts outbox-queued audio too — sent in order)
416
418
  sendAudio(i16); sentSinceCommit = true;
417
419
  },
@@ -545,6 +547,10 @@ export function makeDeepgramSTT(opts) {
545
547
  ws.onmessage = ev => {
546
548
  if (typeof ev.data !== 'string') return;
547
549
  const m = JSON.parse(ev.data);
550
+ // Continuous stream: the socket lives for the whole session, so the ms clock re-anchors at
551
+ // each turn's FIRST transcript activity (turnText empty → this is the turn's opening event),
552
+ // not at socket open — otherwise ms grows monotonically across the session.
553
+ if (m.type === 'TurnInfo' && !turnText && (m.transcript || '').trim()) sttStart = performance.now();
548
554
  const ms = Math.round(performance.now() - sttStart);
549
555
  if (m.type === 'TurnInfo') {
550
556
  const t = (m.transcript || '').trim();
@@ -5,7 +5,9 @@
5
5
  //
6
6
  // Auth, two modes (mirrors the STT providers' "key stays server-side" stance):
7
7
  // • ttsUrl — YOUR backend proxy: POST { text, voice_id, model_id } → audio bytes. The proxy adds
8
- // the xi-api-key and forwards to ElevenLabs. Recommended for production.
8
+ // the xi-api-key and forwards to ElevenLabs. Recommended for production. Pass
9
+ // `headers` to authenticate to YOUR proxy (e.g. { 'X-API-Key': ... }) — they ride on
10
+ // every proxy request (synth + voices), never on a direct ElevenLabs call.
9
11
  // • apiKey — direct browser→ElevenLabs (dev/bench only: the key is visible in the page).
10
12
  //
11
13
  // Latency: eleven_flash_v2_5 is EL's low-latency model (~75ms model time); the rest of a clip's
@@ -17,10 +19,27 @@ const EL_API = 'https://api.elevenlabs.io/v1/text-to-speech';
17
19
 
18
20
  export class ElevenLabsTTS extends StreamingTTS {
19
21
  constructor({ ttsUrl, apiKey, voiceId = 'JBFqnCBsd6RMkjVDRZzb' /* George */,
20
- modelId = 'eleven_flash_v2_5', format = 'mp3_44100_64' } = {}) {
22
+ modelId = 'eleven_flash_v2_5', format = 'mp3_44100_64', headers = {} } = {}) {
21
23
  super(voiceId);
22
24
  if (!ttsUrl && !apiKey) throw new Error('ElevenLabsTTS needs ttsUrl (backend proxy) or apiKey (direct, dev only)');
23
25
  this.ttsUrl = ttsUrl; this.apiKey = apiKey; this.modelId = modelId; this.format = format;
26
+ this.headers = headers; // proxy-path auth (X-API-Key etc.) — never sent to ElevenLabs directly
27
+ }
28
+
29
+ // Listing voices needs the key, so on the proxy path it has to come from YOUR backend: a GET on
30
+ // the same ttsUrl returns [{ id, name }] (see examples/token-worker). Direct mode asks ElevenLabs
31
+ // with the key it already has. Either way a failure is non-fatal — an empty list just means the
32
+ // host keeps whatever voiceId it was constructed with, and synthesis is unaffected.
33
+ async voices() {
34
+ const [url, headers] = this.ttsUrl
35
+ ? [this.ttsUrl, this.headers]
36
+ : ['https://api.elevenlabs.io/v2/voices?page_size=100', { 'xi-api-key': this.apiKey }];
37
+ try {
38
+ const r = await fetch(url, { headers, signal: AbortSignal.timeout(5000) });
39
+ if (!r.ok) return [];
40
+ const body = await r.json();
41
+ return (body.voices ?? []).map((v) => ({ id: v.voice_id ?? v.id, name: v.name }));
42
+ } catch { return []; }
24
43
  }
25
44
 
26
45
  // Pre-establish the connection (DNS + TLS + any proxy hop) so the first real clip only pays TTFB.
@@ -37,7 +56,7 @@ export class ElevenLabsTTS extends StreamingTTS {
37
56
  const url = direct ? `${EL_API}/${this.voiceId}?output_format=${this.format}` : this.ttsUrl;
38
57
  const res = await fetch(url, {
39
58
  method: 'POST',
40
- headers: { 'Content-Type': 'application/json', ...(direct ? { 'xi-api-key': this.apiKey } : {}) },
59
+ headers: { 'Content-Type': 'application/json', ...(direct ? { 'xi-api-key': this.apiKey } : this.headers) },
41
60
  // The proxy form carries voice/model/format in the body so one backend route serves any voice.
42
61
  body: JSON.stringify(direct ? { text, model_id: this.modelId }
43
62
  : { text, voice_id: this.voiceId, model_id: this.modelId, output_format: this.format }),
@@ -909,7 +909,7 @@ export class VoiceAgent {
909
909
  if (this.history[this.history.length - 1]?.role !== 'user') return;
910
910
  this._set('thinking'); // LLM stage
911
911
  this._abort = pf?.ctl ?? new AbortController(); // adopted prefetch keeps ITS controller so barge-in aborts the right fetch
912
- await this._speakTurn(this._abort.signal, pf?.gen); // _speakTurn never throws — errors surface via onEvent
912
+ await this._speakTurn(this._abort.signal, pf?.gen, seq); // _speakTurn never throws — errors surface via onEvent
913
913
  })());
914
914
  }
915
915
 
@@ -917,7 +917,9 @@ export class VoiceAgent {
917
917
  // prefix, not the full generated answer) plus a ledger of the tools it dispatched. Stored as
918
918
  // `this._turn` so the next user turn can await it. Never throws (errors surface via onEvent) so
919
919
  // the await is safe — and never awaits a tool, so a hung one can't hold the next turn hostage.
920
- async _speakTurn(signal, prefetched = null) {
920
+ // `turn` tags every assistant event with the reply it belongs to, so a host can tell a LATE
921
+ // event from a superseded turn apart from one for the current turn (see the demo's transcript).
922
+ async _speakTurn(signal, prefetched = null, turn = 0) {
921
923
  // Fresh turn → drop the previous reply's audible text so user speech heard before THIS turn's
922
924
  // first audio plays can't be misjudged as echo of the OLD reply — but PARK it in _prevReplyText:
923
925
  // its speakers→mic echo tail is still in flight for a moment (acoustic delay + STT latency), and
@@ -989,7 +991,7 @@ export class VoiceAgent {
989
991
  // read live each tick), so streaming the whole answer here would yank the cursor to the end.
990
992
  // Once the cursor is live we only grow `answer` (the hook reads it); until then (muted, or the
991
993
  // first sentence is still synthesizing) we stream the full answer so the bubble isn't empty.
992
- if (!self._cursorLive) self.onEvent({ type: 'assistant', text: answer, final: false });
994
+ if (!self._cursorLive) self.onEvent({ type: 'assistant', text: answer, final: false, turn });
993
995
  yield item.text;
994
996
  }
995
997
  })(this, stripToolCallMimicry(prefetched ?? this.llm(this.history, this.sysmsg, signal))); // prefetched stream already carries this turn's user text (see _startPrefetch)
@@ -1010,13 +1012,20 @@ export class VoiceAgent {
1010
1012
  return answer.slice(0, i);
1011
1013
  };
1012
1014
  this._cursorLive = false;
1015
+ let lastDraft = ''; // last emitted heard+answer — the rAF ticker repeats itself
1013
1016
  this.tts.setOnProgress?.((spokenSoFar, scope) => {
1014
1017
  if (signal.aborted) return;
1015
1018
  this._cursorLive = true; // hook now drives the draft → LLM stream stops emitting full text
1016
1019
  const heard = rawPrefix(spokenSoFar);
1017
1020
  this._replyText = heard; // AUDIBLE prefix — drives the spoken cursor
1018
1021
  this._replyEcho = scope ? rawPrefix(scope) : heard; // clip-bounded echo reference (cursor lags real audio)
1019
- this.onEvent({ type: 'assistant', text: heard, full: answer, final: false });
1022
+ // The player ticks every animation frame but the time→text estimate only advances a char every
1023
+ // few frames — drop the identical repeats so hosts see one event per visible change, not 60/s.
1024
+ // (`answer` is in the key too: while streaming, `full` still grows between identical cursors.)
1025
+ const draft = heard.length + '\u0000' + answer.length;
1026
+ if (draft === lastDraft) return;
1027
+ lastDraft = draft;
1028
+ this.onEvent({ type: 'assistant', text: heard, full: answer, final: false, turn });
1020
1029
  });
1021
1030
  try {
1022
1031
  // TTS muted: drain the same stream so the transcript + tool calls still run, but never voice
@@ -1030,7 +1039,9 @@ export class VoiceAgent {
1030
1039
  } catch (e) {
1031
1040
  if (e.name !== 'AbortError') { // real failure (not barge-in) → surface it
1032
1041
  this.tts.setOnProgress?.(null); this._cursorLive = false;
1033
- this.onEvent({ type: 'error', error: e.message });
1042
+ // Tagged with the turn: this error is this reply's TERMINAL event (no assistant final
1043
+ // follows), and a host tracking bubbles per turn needs to know which one just died.
1044
+ this.onEvent({ type: 'error', error: e.message, turn });
1034
1045
  // Tools fire DURING streaming, so a reply that dies here may already have dispatched one.
1035
1046
  // Record it anyway: the ledger is about what was DONE, not about what got spoken — dropping
1036
1047
  // it would let the next turn call the same tool again.
@@ -1063,11 +1074,11 @@ export class VoiceAgent {
1063
1074
  // full.slice(text.length) tail aligns. keepFull/normal-finish → whole answer (no tail); a barge-in
1064
1075
  // → the heard prefix solid + the unspoken remainder dimmed, showing exactly where TTS got.
1065
1076
  const shown = signal.keepFull ? answer : rawPrefix(spoken);
1066
- this.onEvent({ type: 'assistant', text: shown, full: answer, final: true });
1077
+ this.onEvent({ type: 'assistant', text: shown, full: answer, final: true, turn });
1067
1078
  } else if (answer) {
1068
1079
  // Interrupted before anything was heard (nothing voiced yet): drop the streamed draft so the
1069
1080
  // transcript doesn't keep a dangling unfinished bubble. Empty text → host clears the draft.
1070
- this.onEvent({ type: 'assistant', text: '', final: false });
1081
+ this.onEvent({ type: 'assistant', text: '', final: false, turn });
1071
1082
  }
1072
1083
  // Tools are NOT joined here. They were dispatched during streaming (in parallel with TTS) and
1073
1084
  // _runTool reports its own outcome via onEvent, so waiting adds nothing — while a tool that
@@ -1849,7 +1860,18 @@ export class StreamingTTS {
1849
1860
  if (idx >= tape.length) { await awaitEntry(idx); if (takePending()) continue; if (idx >= tape.length) break; } // wait for the producer to reach this sentence (stream end → done)
1850
1861
  const e = tape[idx];
1851
1862
  const buf = await e.blobP;
1852
- if (e.err) throw e.err; // this clip's synth failed → surface it (only when we actually reach it)
1863
+ // A clip that failed to RENDER is a voice problem, not an answer problem: the model already
1864
+ // produced this text and the user is already reading it. Throwing here abandoned every
1865
+ // REMAINING sentence — the reply went silent partway through and speak() resolved with only
1866
+ // the prefix, so history recorded less than what was on screen. Skip the mute sentence
1867
+ // instead, count it as delivered (heardMax), and keep speaking the rest.
1868
+ if (e.err) {
1869
+ // Say it out loud on the same channel as the other TTS diagnostics: a sentence that is
1870
+ // silently skipped is exactly the "it just got lost" symptom this branch exists to avoid.
1871
+ this.onEvent?.({ type: 'error', error: `TTS synth failed, sentence not voiced: ${e.err.message}` });
1872
+ heardMax = Math.max(heardMax, e.nwStart + nw(e.text));
1873
+ idx++; startNw = 0; continue;
1874
+ }
1853
1875
  if (takePending()) continue; // a tap during synth/stream gap → reposition
1854
1876
  this._curIdx = idx;
1855
1877
  const prefix = tape.slice(0, idx).map(x => x.text).join(' ') + (idx ? ' ' : '');