@todoforai/voiceloop 0.1.8 → 0.1.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -44
- package/examples/simple-browser.html +6 -2
- package/examples/token-worker/README.md +65 -0
- package/examples/token-worker/index.js +173 -0
- package/examples/token-worker/wrangler.toml +39 -0
- package/{examples/demo.html → index.html} +239 -63
- package/package.json +3 -2
- package/src/index.d.ts +9 -4
- package/src/stt.js +6 -0
- package/src/tts-elevenlabs.js +22 -3
- package/src/voice-agent.js +30 -8
package/README.md
CHANGED
|
@@ -1,18 +1,31 @@
|
|
|
1
1
|
# voiceloop
|
|
2
2
|
|
|
3
|
-
**The fastest
|
|
3
|
+
**The fastest voice agent loop in the browser**
|
|
4
|
+
|
|
5
|
+
### ▶︎ [Try the live demo](https://todoforai.github.io/voiceloop/) — no keys, no install, runs in your tab
|
|
6
|
+
|
|
7
|
+
## Why
|
|
8
|
+
|
|
9
|
+
I just couldn't find a solution that actually creates the fluid JARVIS feel in the browser (and it's crazy that it's 2026, not 2010, and there's still no really good one).
|
|
10
|
+
|
|
11
|
+
Everybody should have the best voice loop. That's why we created this, and we'll keep pushing the limits here — observations and contributions welcome, let's push the boundaries together!
|
|
12
|
+
|
|
13
|
+
## Features
|
|
4
14
|
|
|
5
15
|
A zero-dependency JavaScript library that runs the full voice loop — **VAD → STT → LLM → TTS** — with the hard parts already solved:
|
|
6
16
|
|
|
7
|
-
- **Real barge-in** —
|
|
8
|
-
- **Self-echo filtering** —
|
|
9
|
-
- **First
|
|
10
|
-
- **
|
|
11
|
-
- **
|
|
12
|
-
|
|
13
|
-
|
|
17
|
+
- **Real barge-in** — triggers on transcribed *novel words*, not mic energy, so the agent's own voice never cuts it off.
|
|
18
|
+
- **Self-echo filtering** — 0 self-interruptions with AEC off (Pipecat cut itself 20/30, OpenAI Realtime 17/30 [bench](bench/results/RESULTS.md#scenario-echo-smalltalk--speakermic-coupling-15db--30ms)).
|
|
19
|
+
- **First audio <1s** — TTS speaks sentence 1 while the LLM writes sentence 2, and the LLM call starts speculatively during your end-of-turn pause.
|
|
20
|
+
- **Serialized turns** — rapid-fire turns, tool results, holds and replays can never talk over each other. Locked in by 178 tests.
|
|
21
|
+
- **Local-first** — Silero VAD and Piper TTS run as WASM in the tab: free, no cloud round-trip (CDN, then cache).
|
|
22
|
+
|
|
23
|
+
Extra:
|
|
24
|
+
- **Tap-to-seek** — click anywhere in the transcript to jump the voice there, backward or forward.
|
|
14
25
|
|
|
15
|
-
|
|
26
|
+
**The point: an in-browser voice agent that just works properly.**
|
|
27
|
+
|
|
28
|
+
Everything is pluggable: any OpenAI-compatible LLM (or a custom async generator), STT (Web Speech, ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS.
|
|
16
29
|
|
|
17
30
|
## Quick start
|
|
18
31
|
|
|
@@ -29,7 +42,7 @@ const agent = new VoiceAgent({
|
|
|
29
42
|
// Any OpenAI-compatible /chat/completions endpoint. In production point this at
|
|
30
43
|
// YOUR proxy route — never ship a provider secret key to the page.
|
|
31
44
|
llmUrl: '/api/chat/completions',
|
|
32
|
-
model: '
|
|
45
|
+
model: 'claude-haiku-4-5', // voice wants first-token speed — use the fastest model your endpoint serves
|
|
33
46
|
|
|
34
47
|
persona: 'You are a friendly cooking assistant.',
|
|
35
48
|
sysmsg: 'The user is on the recipes page.', // live context, update via setSysmsg()
|
|
@@ -52,35 +65,25 @@ button.onclick = async () => {
|
|
|
52
65
|
|
|
53
66
|
That's it. Speak, get spoken answers, interrupt at will.
|
|
54
67
|
|
|
55
|
-
##
|
|
68
|
+
## The demo
|
|
69
|
+
|
|
70
|
+
**[todoforai.github.io/voiceloop](https://todoforai.github.io/voiceloop/)** starts in echo mode —
|
|
71
|
+
browser Web Speech STT, local Piper TTS, a stand-in "LLM" that repeats you — so it costs nothing
|
|
72
|
+
and needs no backend. THIS is not the BEST configuration (deepgram STT + elevenlabs flash TTS). the best goes from 2100ms -> 900ms, that's where it start to be LIVE!
|
|
56
73
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
on most builds it still sends audio to the browser vendor. Pick a pipeline STT provider in
|
|
61
|
-
settings to see voiceloop's own VAD→STT path.)
|
|
74
|
+
It shows the two numbers that decide whether an agent feels alive (**first token**, **first sound**), and click-to-interrupt strikes through the words you cut off.
|
|
75
|
+
Open *settings* to switch to a pipeline STT provider (Web Speech bypasses voiceloop's VAD→STT
|
|
76
|
+
path) and point it at any OpenAI-compatible endpoint (Ollama, your proxy).
|
|
62
77
|
|
|
63
|
-
|
|
78
|
+
Run it locally against your own LLM:
|
|
64
79
|
|
|
65
80
|
```sh
|
|
66
|
-
npx serve .
|
|
67
|
-
npx serve node_modules/@todoforai/voiceloop # from npm
|
|
81
|
+
npx serve . # from a clone
|
|
82
|
+
npx serve node_modules/@todoforai/voiceloop # from npm — then open index.html
|
|
68
83
|
```
|
|
69
84
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
interim transcript, click-to-interrupt with the unspoken remainder struck through where
|
|
73
|
-
you cut it off, and the two numbers that decide whether it feels alive: **first token**
|
|
74
|
-
and **first sound**. It starts in **echo mode**
|
|
75
|
-
— browser-native Web Speech STT, local Piper TTS and a stand-in "LLM" that
|
|
76
|
-
repeats you — so it runs with no keys, no backend and no cost. Open *settings*
|
|
77
|
-
to point it at any OpenAI-compatible endpoint (Ollama, your proxy) and a cloud
|
|
78
|
-
STT provider.
|
|
79
|
-
|
|
80
|
-
`/examples/simple-browser.html` is the same loop with no UI at all — 100 lines,
|
|
81
|
-
the place to start reading.
|
|
82
|
-
|
|
83
|
-
(`file://` won't work — mic and module imports need `http://localhost` or https.)
|
|
85
|
+
`file://` won't work — mic and module imports need `http://localhost` or https.
|
|
86
|
+
Start reading at [`examples/simple-browser.html`](examples/simple-browser.html): the same loop, no UI, ~100 lines.
|
|
84
87
|
|
|
85
88
|
## How the latency adds up
|
|
86
89
|
|
|
@@ -111,17 +114,13 @@ into the mic (no AEC):
|
|
|
111
114
|
| Pipecat 1.8.1 (same providers) | 1050ms | 2/30 | **20/30** |
|
|
112
115
|
| ElevenLabs ConvAI | 1450ms | **0/30** | 0/30 |
|
|
113
116
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
no AEC, the other fast stacks hear themselves as the user and cut their own replies.
|
|
118
|
-
voiceloop's word-level echo filter runs echo-coupled turns at full speed (930ms, parity with
|
|
119
|
-
clean).
|
|
117
|
+
The separating failure is echo: with no AEC, the other fast stacks hear their own voice as the
|
|
118
|
+
user and cut their own replies. voiceloop's word-level echo filter runs echo-coupled turns at
|
|
119
|
+
full speed (930ms — parity with clean).
|
|
120
120
|
|
|
121
|
-
\* Realtime is speech-to-speech
|
|
122
|
-
fully apples-to-apples.
|
|
121
|
+
\* Realtime is speech-to-speech, so it can't use the bench's fixed mock LLM — not fully apples-to-apples.
|
|
123
122
|
|
|
124
|
-
Full tables, method
|
|
123
|
+
Full tables, method, reproduction: [`results/RESULTS.md`](https://github.com/todoforai/voice-agent-bench/blob/master/results/RESULTS.md).
|
|
125
124
|
|
|
126
125
|
## STT providers
|
|
127
126
|
|
|
@@ -134,7 +133,7 @@ new VoiceAgent({ sttProvider: 'speechmatics', sttTokenUrl: '/api/stt/token' })
|
|
|
134
133
|
|
|
135
134
|
Cloud providers authenticate with a **short-TTL token minted by your backend** so the raw API key never reaches the page. Either:
|
|
136
135
|
|
|
137
|
-
- `sttTokenUrl` — a POST route on your server that returns `{ token }` (mint it against the provider's temp-token API with your secret key), or
|
|
136
|
+
- `sttTokenUrl` — a POST route on your server that returns `{ token }` (mint it against the provider's temp-token API with your secret key — a complete single-file Cloudflare Worker doing exactly this for Deepgram, plus ElevenLabs TTS and LLM proxies with rate limits, is in [`examples/token-worker/`](examples/token-worker/)), or
|
|
138
137
|
- `getSttToken` — an async callback `(provider) => ({ token })` when your auth doesn't fit a bare POST, or when each provider mints a different credential. It is handed the provider actually running (after the fallback below), so you pick the route without re-deriving that rule:
|
|
139
138
|
|
|
140
139
|
```js
|
|
@@ -226,6 +225,8 @@ agent.setHeld(bool) // hold: queue utterances, reply over all of them
|
|
|
226
225
|
agent.setMuted(bool) // mic mute
|
|
227
226
|
agent.setTtsMuted(bool) // silent mode: transcript + tools still run
|
|
228
227
|
agent.setSysmsg(text) // update live context mid-session
|
|
228
|
+
agent.interrupt() // programmatic barge-in: stop the current reply
|
|
229
|
+
agent.setVoice(id) / agent.setSpeed(x) / agent.listVoices() // voice controls (engine-dependent)
|
|
229
230
|
agent.seek(nwIndex) // tap-to-seek within the current reply (nwIndex = count of
|
|
230
231
|
// NON-WHITESPACE chars before the tap — whitespace doesn't count)
|
|
231
232
|
agent.replay(text, fromNw?, onProgress?) // re-voice a past reply; onProgress(spokenText, done)
|
|
@@ -264,7 +265,7 @@ Two of them exist purely so a UI can look alive, and both cost you nothing to ig
|
|
|
264
265
|
- **interruption** — there is no "cut short" event: the final `assistant` event carries both what
|
|
265
266
|
was heard (`text`) and what was written (`full`), so a non-empty `full.slice(text.length)` means
|
|
266
267
|
the reply was cut before that remainder was ever spoken. Render it struck through and the
|
|
267
|
-
transcript shows exactly where playback stopped (see `
|
|
268
|
+
transcript shows exactly where playback stopped (see `index.html`). It does not say *why*:
|
|
268
269
|
a voice barge-in, `interrupt()`, a superseding turn and `setTtsMuted(true)` all end a reply the
|
|
269
270
|
same way.
|
|
270
271
|
|
|
@@ -315,3 +316,5 @@ The suite locks in turn serialization, hold/release semantics, tool dedup, self-
|
|
|
315
316
|
## License
|
|
316
317
|
|
|
317
318
|
MIT
|
|
319
|
+
|
|
320
|
+
"voiceloop" is a trademark of TODOforAI. You are free to use, modify and redistribute the code under the MIT license, but not to present a fork or derivative as the official voiceloop project or publish it under the voiceloop name.
|
|
@@ -46,12 +46,12 @@
|
|
|
46
46
|
const agent = new VoiceAgent({
|
|
47
47
|
// Point at any OpenAI-compatible /chat/completions endpoint.
|
|
48
48
|
// Quick local test with Ollama (from the repo root):
|
|
49
|
-
// ollama pull
|
|
49
|
+
// ollama pull qwen3
|
|
50
50
|
// OLLAMA_ORIGINS='*' ollama serve
|
|
51
51
|
// npx serve . → open http://localhost:3000/examples/simple-browser.html
|
|
52
52
|
// (OLLAMA_ORIGINS allows this page's origin; mic access needs http://localhost or https.)
|
|
53
53
|
llmUrl: 'http://localhost:11434/v1/chat/completions',
|
|
54
|
-
model: '
|
|
54
|
+
model: 'qwen3',
|
|
55
55
|
|
|
56
56
|
persona: 'You are a concise, friendly voice assistant.',
|
|
57
57
|
|
|
@@ -60,6 +60,10 @@
|
|
|
60
60
|
$('state').textContent = e.state;
|
|
61
61
|
} else if (e.type === 'stt') {
|
|
62
62
|
userEl ??= bubble('user');
|
|
63
|
+
// A turn can die WITHOUT turnComplete (echo-dropped, retracted, reply error) — userEl
|
|
64
|
+
// then survives at its old position and ??= would paint the NEXT turn's speech above
|
|
65
|
+
// later messages. append() moves an existing node back to the bottom (no-op otherwise).
|
|
66
|
+
log.append(userEl);
|
|
63
67
|
// partial: committed prefix + live tail; turnComplete: e.text is the whole turn
|
|
64
68
|
userEl.textContent = e.turnComplete ? e.text : `${e.committed ?? ''} ${e.text}`.trim();
|
|
65
69
|
if (e.turnComplete) userEl = null;
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# voiceloop-token
|
|
2
|
+
|
|
3
|
+
The tiny backend the live demo needs. GitHub Pages is static, so it has no server to mint
|
|
4
|
+
credentials — and cloud STT/TTS keys must never reach the page. This Worker holds them and hands
|
|
5
|
+
out only what each hop needs.
|
|
6
|
+
|
|
7
|
+
| route | returns | why |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| `POST /stt/token` | `{ token, expires_in }` | short-TTL Deepgram key; the browser opens the WS itself |
|
|
10
|
+
| `POST /tts` | `audio/mpeg` | ElevenLabs proxy — the `xi-api-key` stays server-side |
|
|
11
|
+
| `POST /llm` | OpenAI-format SSE | Anthropic proxy — Claude has no short-TTL key to mint, so the key stays here |
|
|
12
|
+
|
|
13
|
+
(The voiceloop demo runs this exact Worker.)
|
|
14
|
+
|
|
15
|
+
## Deploy
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
wrangler deploy
|
|
19
|
+
wrangler secret put DEEPGRAM_API_KEY
|
|
20
|
+
wrangler secret put ELEVENLABS_API_KEY
|
|
21
|
+
wrangler secret put ANTHROPIC_API_KEY
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
`ALLOWED_ORIGINS` (in `wrangler.toml`) is a comma-separated allowlist. These routes spend real
|
|
25
|
+
money, so an open CORS policy is an open wallet — a request from an unlisted origin gets no CORS
|
|
26
|
+
headers and a 403. But CORS is **not** authentication: it governs what browser JS may read, and a
|
|
27
|
+
request with no `Origin` (curl, a script) never engages it while a non-browser caller can forge one.
|
|
28
|
+
So every paid route also passes per-IP rate limits in two windows: a 60s `[[ratelimits]]` burst
|
|
29
|
+
limiter (best-effort, per-Cloudflare-location — `/stt/token` gets a tight one, the `/tts`+`/llm`
|
|
30
|
+
conversation hot path a roomier one) and a long-window quota (`QUOTA_LIMIT` calls per
|
|
31
|
+
`QUOTA_PERIOD_H` hours, default 300/6h) counted globally in a Durable Object — that one is the
|
|
32
|
+
actual wallet cap. Combine it with provider-side spend caps; anything costlier than a demo key
|
|
33
|
+
needs real auth in front.
|
|
34
|
+
|
|
35
|
+
`/tts` caps text at 600 chars and `/llm` caps the conversation at 8000.
|
|
36
|
+
|
|
37
|
+
`/llm` pins the model server-side (`ANTHROPIC_MODEL`, default `claude-haiku-4-5-20251001`) and caps
|
|
38
|
+
`max_tokens`, and forwards an ALLOWLIST rather than the caller's object (only `messages` and a
|
|
39
|
+
clamped `max_tokens` are read from the request; model and stream mode are server-owned):
|
|
40
|
+
the page picks neither model nor limits, and a large `tools` schema can't smuggle billable input
|
|
41
|
+
tokens past the size check. Anthropic serves an OpenAI-compatible `/v1/chat/completions`, so the SSE passes straight
|
|
42
|
+
through to voiceloop's built-in `makeOpenAILLM` — no client-side adapter.
|
|
43
|
+
|
|
44
|
+
The persona is pinned server-side too (`SYSTEM_PROMPT` var, default: Jarvis, answer extremely
|
|
45
|
+
briefly): caller-supplied `system` messages are stripped, so a client can't prompt the hosted
|
|
46
|
+
demo into long — i.e. expensive — answers. Brevity doubles as a latency control: the reply is
|
|
47
|
+
only as fast as the sentence the TTS is waiting to finish.
|
|
48
|
+
|
|
49
|
+
## Use it from the demo
|
|
50
|
+
|
|
51
|
+
In the demo's settings: STT = `deepgram`, STT token url = `<worker>/stt/token`. Mode =
|
|
52
|
+
`full loop — hosted claude haiku` uses `/llm` and needs no key or LLM url at all.
|
|
53
|
+
|
|
54
|
+
| route | returns |
|
|
55
|
+
|---|---|
|
|
56
|
+
| `GET /tts` | `{ voices: [{ id, name }] }` — listing needs the key too, so the browser can't ask ElevenLabs itself |
|
|
57
|
+
|
|
58
|
+
## The fastest config
|
|
59
|
+
|
|
60
|
+
Benchmarked at **984ms voice→voice vs 2126ms** for the zero-key path ([RESULTS](../../bench/results/RESULTS.md)) —
|
|
61
|
+
the whole gap is end-of-turn (366ms vs 1565ms). In the demo's settings:
|
|
62
|
+
|
|
63
|
+
- **STT** `deepgram` · **STT token url** `<worker>/stt/token`
|
|
64
|
+
- **TTS** `elevenlabs` · **TTS url** `<worker>/tts`
|
|
65
|
+
- **mode** `hosted claude haiku` — measured 568ms to first token, 893ms to first sound in-browser
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
// voiceloop-token — the tiny backend the live demo needs, and nothing more.
|
|
2
|
+
//
|
|
3
|
+
// GitHub Pages is static, so the demo has no server to mint credentials. Cloud STT/TTS keys must
|
|
4
|
+
// never reach the page, so this Worker holds them and hands out only what each hop needs:
|
|
5
|
+
//
|
|
6
|
+
// POST /stt/token → { token, expires_in } short-TTL Deepgram key (browser opens the WS itself)
|
|
7
|
+
// POST /tts → audio/mpeg bytes ElevenLabs proxy (the xi-api-key stays here)
|
|
8
|
+
// POST /llm → OpenAI-format SSE Anthropic proxy (there is no short-TTL key to mint)
|
|
9
|
+
//
|
|
10
|
+
// Secrets (wrangler secret put): DEEPGRAM_API_KEY, ELEVENLABS_API_KEY, ANTHROPIC_API_KEY.
|
|
11
|
+
// ALLOWED_ORIGINS (var) is a comma-separated allowlist — these routes spend real money, so an
|
|
12
|
+
// open CORS policy is an open wallet.
|
|
13
|
+
//
|
|
14
|
+
// Two layers, because they stop different things. The origin allowlist keeps OTHER PAGES from
|
|
15
|
+
// spending the keys — that is all CORS can do: it governs what browser JS may read, and a request
|
|
16
|
+
// with no Origin header (curl, a script) never engages it, while a non-browser caller can forge one.
|
|
17
|
+
// So every paid route also passes through a per-IP rate limit, which is what actually bounds the
|
|
18
|
+
// bill when the URL leaks. Neither is authentication; a demo backend holding capped, cheap keys is
|
|
19
|
+
// the threat model. Anything costlier needs real auth in front of it.
|
|
20
|
+
|
|
21
|
+
// Persona for the hosted demo (override with the SYSTEM_PROMPT var). The brevity rule is a cost
|
|
22
|
+
// AND a latency control: every output token is billed, and in a voice loop the reply is only as
|
|
23
|
+
// fast as the sentence the TTS is waiting to finish.
|
|
24
|
+
const HOSTED_PERSONA =
|
|
25
|
+
'You are Jarvis, a voice assistant. Your reply is spoken aloud, so answer EXTREMELY briefly — ' +
|
|
26
|
+
'one short sentence, two at most. Plain speakable prose: no markdown, no lists, no headings. ' +
|
|
27
|
+
'Never explain that you are being brief.';
|
|
28
|
+
|
|
29
|
+
const json = (obj, status, origin) =>
|
|
30
|
+
new Response(JSON.stringify(obj), { status, headers: { 'content-type': 'application/json', ...cors(origin) } });
|
|
31
|
+
|
|
32
|
+
const cors = (origin) => origin ? {
|
|
33
|
+
'access-control-allow-origin': origin,
|
|
34
|
+
'access-control-allow-headers': 'content-type,x-api-key',
|
|
35
|
+
'access-control-allow-methods': 'GET,POST,OPTIONS',
|
|
36
|
+
'vary': 'Origin',
|
|
37
|
+
} : {};
|
|
38
|
+
|
|
39
|
+
const allow = (req, env) => {
|
|
40
|
+
const origin = req.headers.get('Origin');
|
|
41
|
+
if (!origin) return null;
|
|
42
|
+
const list = (env.ALLOWED_ORIGINS || '').split(',').map((s) => s.trim()).filter(Boolean);
|
|
43
|
+
return list.includes(origin) ? origin : null;
|
|
44
|
+
};
|
|
45
|
+
|
|
46
|
+
// Two windows per IP: the 60s limiter binding is burst protection, and the Quota Durable Object
|
|
47
|
+
// is the long-window wallet cap ([[ratelimits]] only supports 10s/60s periods, hence the DO).
|
|
48
|
+
const rateLimited = async (req, env, limiter = env.PAID_RATE_LIMITER) => {
|
|
49
|
+
const ip = req.headers.get('CF-Connecting-IP') || 'unknown';
|
|
50
|
+
if (limiter && !(await limiter.limit({ key: ip })).success) return true;
|
|
51
|
+
if (!env.QUOTA) return false;
|
|
52
|
+
const r = await env.QUOTA.get(env.QUOTA.idFromName(ip)).fetch('https://quota/', {
|
|
53
|
+
method: 'POST',
|
|
54
|
+
body: JSON.stringify({ limit: Number(env.QUOTA_LIMIT) || 300, periodMs: (Number(env.QUOTA_PERIOD_H) || 6) * 3600_000 }),
|
|
55
|
+
});
|
|
56
|
+
return !(await r.json()).ok;
|
|
57
|
+
};
|
|
58
|
+
|
|
59
|
+
// Per-IP fixed-window counter. A Durable Object because the count must be one number, globally —
|
|
60
|
+
// the Workers runtime itself has no cross-datacenter state, and the [[ratelimits]] binding is
|
|
61
|
+
// per-datacenter best-effort. One DO instance per IP (idFromName), state survives restarts.
|
|
62
|
+
export class Quota {
|
|
63
|
+
constructor(state) { this.state = state; }
|
|
64
|
+
async fetch(req) {
|
|
65
|
+
const { limit, periodMs } = await req.json();
|
|
66
|
+
const now = Date.now();
|
|
67
|
+
let { start = now, count = 0 } = await this.state.storage.get(['start', 'count']).then((m) => Object.fromEntries(m)) ?? {};
|
|
68
|
+
if (now - start >= periodMs) { start = now; count = 0; }
|
|
69
|
+
const ok = count < limit;
|
|
70
|
+
if (ok) await this.state.storage.put({ start, count: count + 1 });
|
|
71
|
+
return new Response(JSON.stringify({ ok }));
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
export default {
|
|
76
|
+
async fetch(req, env) {
|
|
77
|
+
const url = new URL(req.url);
|
|
78
|
+
const origin = allow(req, env);
|
|
79
|
+
|
|
80
|
+
if (req.method === 'OPTIONS') {
|
|
81
|
+
// A disallowed origin gets no CORS headers at all — the browser then blocks it, which is the point.
|
|
82
|
+
return new Response(null, { status: origin ? 204 : 403, headers: cors(origin) });
|
|
83
|
+
}
|
|
84
|
+
if (req.headers.get('Origin') && !origin) return json({ error: 'origin not allowed' }, 403, null);
|
|
85
|
+
|
|
86
|
+
// GET /tts → the voice list. Listing needs the key too, so the browser can't ask ElevenLabs
|
|
87
|
+
// itself (that endpoint is NOT public); the picker in a UI has to be fed from here.
|
|
88
|
+
if (req.method === 'GET' && url.pathname === '/tts') {
|
|
89
|
+
if (!env.ELEVENLABS_API_KEY) return json({ error: 'ELEVENLABS_API_KEY not configured' }, 500, origin);
|
|
90
|
+
if (await rateLimited(req, env, env.CONVO_RATE_LIMITER)) return json({ error: 'rate limited — try again in a minute' }, 429, origin); // listing hits the key too
|
|
91
|
+
const r = await fetch('https://api.elevenlabs.io/v2/voices?page_size=100', { headers: { 'xi-api-key': env.ELEVENLABS_API_KEY } });
|
|
92
|
+
const body = await r.json().catch(() => ({}));
|
|
93
|
+
if (!r.ok) return json({ error: body.detail?.message || 'voice list failed' }, 502, origin);
|
|
94
|
+
return json({ voices: (body.voices ?? []).map((v) => ({ id: v.voice_id, name: v.name })) }, 200, origin);
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
if (req.method !== 'POST') return json({ error: 'POST only' }, 405, origin);
|
|
98
|
+
|
|
99
|
+
// Everything below mints a credential or bills a provider. /stt/token mints at most one
|
|
100
|
+
// credential per 5 min so it takes the tight limiter; /tts fires once per spoken SENTENCE and
|
|
101
|
+
// /llm once per turn plus prefetch, so the conversation hot path gets its own, roomier window.
|
|
102
|
+
const limiter = url.pathname === '/stt/token' ? env.PAID_RATE_LIMITER : env.CONVO_RATE_LIMITER;
|
|
103
|
+
if (await rateLimited(req, env, limiter)) return json({ error: 'rate limited — try again in a minute' }, 429, origin);
|
|
104
|
+
|
|
105
|
+
if (url.pathname === '/stt/token') {
|
|
106
|
+
if (!env.DEEPGRAM_API_KEY) return json({ error: 'DEEPGRAM_API_KEY not configured' }, 500, origin);
|
|
107
|
+
// Deepgram mints the short-lived credential; the browser never sees the project key.
|
|
108
|
+
const r = await fetch('https://api.deepgram.com/v1/auth/grant', {
|
|
109
|
+
method: 'POST',
|
|
110
|
+
headers: { authorization: `Token ${env.DEEPGRAM_API_KEY}`, 'content-type': 'application/json' },
|
|
111
|
+
body: JSON.stringify({ ttl_seconds: 300 }),
|
|
112
|
+
});
|
|
113
|
+
const body = await r.json().catch(() => ({}));
|
|
114
|
+
if (!r.ok || !body.access_token) return json({ error: body.err_msg || body.error || 'grant failed' }, 502, origin);
|
|
115
|
+
return json({ token: body.access_token, expires_in: body.expires_in ?? 300 }, 200, origin);
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
if (url.pathname === '/tts') {
|
|
119
|
+
if (!env.ELEVENLABS_API_KEY) return json({ error: 'ELEVENLABS_API_KEY not configured' }, 500, origin);
|
|
120
|
+
const { text, voice_id, model_id, output_format } = await req.json().catch(() => ({}));
|
|
121
|
+
if (!text) return json({ error: 'text required' }, 400, origin);
|
|
122
|
+
// One sentence per call is the contract (that's how the library streams); a whole essay is
|
|
123
|
+
// not a sentence. Also the cost cap: a leaked demo URL should cost cents, not a plan.
|
|
124
|
+
if (text.length > 600) return json({ error: 'text too long' }, 413, origin);
|
|
125
|
+
const fmt = output_format || 'mp3_44100_64';
|
|
126
|
+
const r = await fetch(
|
|
127
|
+
`https://api.elevenlabs.io/v1/text-to-speech/${voice_id || 'JBFqnCBsd6RMkjVDRZzb'}?output_format=${fmt}`,
|
|
128
|
+
{
|
|
129
|
+
method: 'POST',
|
|
130
|
+
headers: { 'xi-api-key': env.ELEVENLABS_API_KEY, 'content-type': 'application/json' },
|
|
131
|
+
body: JSON.stringify({ text, model_id: model_id || 'eleven_flash_v2_5' }),
|
|
132
|
+
});
|
|
133
|
+
if (!r.ok) return json({ error: `elevenlabs ${r.status}: ${(await r.text().catch(() => '')).slice(0, 200)}` }, 502, origin);
|
|
134
|
+
// Stream the audio straight through — buffering it here would add latency to the first clip.
|
|
135
|
+
return new Response(r.body, { status: 200, headers: { 'content-type': 'audio/mpeg', ...cors(origin) } });
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
// Anthropic speaks /v1/chat/completions in OpenAI's format, so the SSE stream passes straight
|
|
139
|
+
// through to the built-in makeOpenAILLM — this route adds the key and an allowlist, nothing else.
|
|
140
|
+
// Streamed, never buffered: the first token is what the user waits for before any audio starts.
|
|
141
|
+
if (url.pathname === '/llm') {
|
|
142
|
+
if (!env.ANTHROPIC_API_KEY) return json({ error: 'ANTHROPIC_API_KEY not configured' }, 500, origin);
|
|
143
|
+
const body = await req.json().catch(() => ({}));
|
|
144
|
+
// Strip any caller-supplied system prompt: the persona is pinned HERE, like the model —
|
|
145
|
+
// otherwise a caller could prompt the hosted demo into long (= expensive) answers.
|
|
146
|
+
const messages = Array.isArray(body.messages) ? body.messages.filter((m) => m?.role !== 'system') : [];
|
|
147
|
+
if (!messages.length) return json({ error: 'messages required' }, 400, origin);
|
|
148
|
+
// Same reasoning as /tts's length cap: a leaked demo URL should cost cents, not a plan.
|
|
149
|
+
if (JSON.stringify(messages).length > 8000) return json({ error: 'conversation too long' }, 413, origin);
|
|
150
|
+
const r = await fetch('https://api.anthropic.com/v1/chat/completions', {
|
|
151
|
+
method: 'POST',
|
|
152
|
+
headers: { authorization: `Bearer ${env.ANTHROPIC_API_KEY}`, 'content-type': 'application/json' },
|
|
153
|
+
// Built from an ALLOWLIST, never `...body`: forwarding the caller's object would carry
|
|
154
|
+
// fields the size check above never measured — a huge `tools` schema bills as input tokens
|
|
155
|
+
// on a 200-byte `messages`, and `max_completion_tokens` is a second output cap whose
|
|
156
|
+
// precedence over max_tokens is undocumented. Only these four fields cross this boundary,
|
|
157
|
+
// and the model is pinned here so a caller can't ask for the priciest model on the account.
|
|
158
|
+
body: JSON.stringify({
|
|
159
|
+
model: env.ANTHROPIC_MODEL || 'claude-haiku-4-5-20251001',
|
|
160
|
+
// Clamped to [1,1024] as an integer — Math.min alone would forward a negative or
|
|
161
|
+
// fractional value, a guaranteed upstream 400 that still burned rate-limit quota.
|
|
162
|
+
max_tokens: Math.min(Math.max(Math.floor(Number(body.max_tokens)) || 512, 1), 1024),
|
|
163
|
+
messages: [{ role: 'system', content: env.SYSTEM_PROMPT || HOSTED_PERSONA }, ...messages],
|
|
164
|
+
stream: true,
|
|
165
|
+
}),
|
|
166
|
+
});
|
|
167
|
+
if (!r.ok) return json({ error: `anthropic ${r.status}: ${(await r.text().catch(() => '')).slice(0, 200)}` }, 502, origin);
|
|
168
|
+
return new Response(r.body, { status: 200, headers: { 'content-type': 'text/event-stream', ...cors(origin) } });
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
return json({ error: 'not found' }, 404, origin);
|
|
172
|
+
},
|
|
173
|
+
};
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
name = "voiceloop-token"
|
|
2
|
+
main = "src/index.js"
|
|
3
|
+
compatibility_date = "2026-01-01"
|
|
4
|
+
|
|
5
|
+
[vars]
|
|
6
|
+
ALLOWED_ORIGINS = "https://todoforai.github.io,http://localhost:8899,http://localhost:3000"
|
|
7
|
+
# Long-window quota (per IP, all paid routes): 300 calls / 6h — this is the actual wallet cap.
|
|
8
|
+
# The [[ratelimits]] bindings below only support 10s/60s periods, so this window lives in the
|
|
9
|
+
# Quota Durable Object instead; the 60s limiters remain as burst protection.
|
|
10
|
+
QUOTA_LIMIT = "300"
|
|
11
|
+
QUOTA_PERIOD_H = "6"
|
|
12
|
+
|
|
13
|
+
[durable_objects]
|
|
14
|
+
bindings = [{ name = "QUOTA", class_name = "Quota" }]
|
|
15
|
+
|
|
16
|
+
[[migrations]]
|
|
17
|
+
tag = "v1"
|
|
18
|
+
new_sqlite_classes = ["Quota"]
|
|
19
|
+
|
|
20
|
+
# /stt/token mints at most one credential per 5 min (300s TTL), so 30/min is already generous.
|
|
21
|
+
[[ratelimits]]
|
|
22
|
+
name = "PAID_RATE_LIMITER"
|
|
23
|
+
namespace_id = "1001"
|
|
24
|
+
|
|
25
|
+
[ratelimits.simple]
|
|
26
|
+
limit = 30
|
|
27
|
+
period = 60
|
|
28
|
+
|
|
29
|
+
# Conversation hot path: /tts fires once per spoken SENTENCE (5-10 per chatty reply) and /llm
|
|
30
|
+
# once per turn plus prefetch. Sized by cost, not call count — each /tts call is capped at
|
|
31
|
+
# 600 chars and each /llm call at 8000 chars in / 1024 tokens out on Haiku, so even 300/min
|
|
32
|
+
# from a leaked URL bounds the bill to cents.
|
|
33
|
+
[[ratelimits]]
|
|
34
|
+
name = "CONVO_RATE_LIMITER"
|
|
35
|
+
namespace_id = "1002"
|
|
36
|
+
|
|
37
|
+
[ratelimits.simple]
|
|
38
|
+
limit = 300
|
|
39
|
+
period = 60
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
<!-- voiceloop demo: the whole loop behind one button, plus the numbers that say
|
|
3
3
|
whether it feels seamless. No bundler, no framework, no dependencies.
|
|
4
4
|
|
|
5
|
-
from a clone: npx serve . → /
|
|
5
|
+
from a clone: npx serve . → http://localhost:3000/
|
|
6
6
|
from npm: npx serve node_modules/@todoforai/voiceloop
|
|
7
7
|
|
|
8
8
|
file:// won't work — mic + module imports need http://localhost or https.
|
|
@@ -230,15 +230,28 @@
|
|
|
230
230
|
min-height pins the result at 44px instead of trusting 13px text to round up to it. */
|
|
231
231
|
@media (pointer: coarse) { summary { padding:11px 0; min-height:44px; box-sizing:border-box } }
|
|
232
232
|
summary::before { content:'▸ ' } details[open] summary::before { content:'▾ ' }
|
|
233
|
-
|
|
233
|
+
/* One row per pipeline stage, in the order the audio flows (LLM group first only because `mode`
|
|
234
|
+
is the master switch). A flat 11-field soup made "which url goes with which provider" a puzzle;
|
|
235
|
+
the heading answers it before the placeholder has to. */
|
|
236
|
+
.grid { display:flex; flex-direction:column; gap:14px; padding:12px 0 4px }
|
|
237
|
+
.set { display:flex; flex-wrap:wrap; gap:10px; align-items:flex-end }
|
|
238
|
+
.set h4 { flex-basis:100%; margin:0 0 -4px; font-size:10px; font-weight:600;
|
|
239
|
+
letter-spacing:.1em; text-transform:uppercase; color:var(--dim); opacity:.65 }
|
|
234
240
|
label { font-size:11px; color:var(--dim); display:flex; flex-direction:column; gap:4px }
|
|
241
|
+
/* A field the current choices ignore must LOOK ignored, not merely refuse focus. */
|
|
242
|
+
input:disabled, select:disabled { opacity:.4 }
|
|
235
243
|
#log { font:11px/1.5 ui-monospace,monospace; color:var(--dim);
|
|
236
244
|
max-height:220px; overflow:auto; white-space:pre-wrap; padding-top:8px }
|
|
237
245
|
/* Which code is actually running — the demo must never let you mistake the browser's own
|
|
238
|
-
recognizer for voiceloop's pipeline, or an echo stand-in for a real LLM.
|
|
239
|
-
|
|
246
|
+
recognizer for voiceloop's pipeline, or an echo stand-in for a real LLM. Sits right above the
|
|
247
|
+
collapsed settings (it describes what they configure), not in the hero area under the orb.
|
|
248
|
+
Arrow chains are .nb (no-break): "Silero VAD →<br>feed" reads as a broken message. */
|
|
249
|
+
#pipeline { width:100%; font-size:11.5px; color:var(--dim); line-height:1.7;
|
|
250
|
+
border-top:1px solid var(--line); padding-top:10px }
|
|
251
|
+
#pipeline:empty { display:none }
|
|
240
252
|
#pipeline b { color:#c8d0da; font-weight:600 }
|
|
241
253
|
#pipeline .warn { color:#e0a94a }
|
|
254
|
+
#pipeline .nb { white-space:nowrap }
|
|
242
255
|
</style>
|
|
243
256
|
</head>
|
|
244
257
|
<body>
|
|
@@ -250,7 +263,6 @@
|
|
|
250
263
|
</div>
|
|
251
264
|
<div id="status">click to start · then just talk</div>
|
|
252
265
|
<div id="lat"></div>
|
|
253
|
-
<div id="pipeline"></div>
|
|
254
266
|
|
|
255
267
|
<div id="chat"></div>
|
|
256
268
|
|
|
@@ -259,21 +271,41 @@
|
|
|
259
271
|
<button class="btn" id="send">Send</button>
|
|
260
272
|
</div>
|
|
261
273
|
|
|
274
|
+
<div id="pipeline"></div>
|
|
275
|
+
|
|
262
276
|
<details>
|
|
263
277
|
<summary>settings</summary>
|
|
264
278
|
<div class="grid">
|
|
265
|
-
<
|
|
266
|
-
<
|
|
267
|
-
<
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
<
|
|
276
|
-
|
|
279
|
+
<div class="set">
|
|
280
|
+
<h4>llm</h4>
|
|
281
|
+
<label>mode<select id="mode">
|
|
282
|
+
<option value="echo">echo — no keys, no backend</option>
|
|
283
|
+
<option value="hosted">full loop — hosted claude haiku</option>
|
|
284
|
+
<option value="llm">full loop — your LLM</option>
|
|
285
|
+
</select></label>
|
|
286
|
+
<label>LLM url<input id="llmUrl" size="30" value="http://localhost:11434/v1/chat/completions"></label>
|
|
287
|
+
<label>model<input id="model" size="14" value="qwen3"></label>
|
|
288
|
+
</div>
|
|
289
|
+
<div class="set">
|
|
290
|
+
<h4>speech to text</h4>
|
|
291
|
+
<label>provider<select id="sttProvider"></select></label>
|
|
292
|
+
<label>language<select id="sttLang"></select></label>
|
|
293
|
+
<label>token url<input id="sttTokenUrl" size="30" placeholder="your backend route (see examples/token-worker)"></label>
|
|
294
|
+
</div>
|
|
295
|
+
<div class="set">
|
|
296
|
+
<h4>text to speech</h4>
|
|
297
|
+
<label>provider<select id="ttsProvider">
|
|
298
|
+
<option value="piper">piper — free, in-browser</option>
|
|
299
|
+
<option value="elevenlabs">elevenlabs — needs TTS url</option>
|
|
300
|
+
</select></label>
|
|
301
|
+
<label>voice<select id="voice"></select></label>
|
|
302
|
+
<label>TTS url<input id="ttsUrl" size="30" placeholder="your backend route (see examples/token-worker)"></label>
|
|
303
|
+
</div>
|
|
304
|
+
<div class="set">
|
|
305
|
+
<h4>input & auth</h4>
|
|
306
|
+
<label>mic<select id="micDeviceId"><option value="">default</option></select></label>
|
|
307
|
+
<label>api key<input id="apiKey" size="26" placeholder="Bearer for your LLM · X-API-Key for token url"></label>
|
|
308
|
+
</div>
|
|
277
309
|
</div>
|
|
278
310
|
<div class="bar">
|
|
279
311
|
<button class="btn" id="clear">Clear</button>
|
|
@@ -286,7 +318,7 @@
|
|
|
286
318
|
</main>
|
|
287
319
|
|
|
288
320
|
<script type="module">
|
|
289
|
-
import { VoiceAgent, PiperTTS, unlockAudio, prewarmVoice, STT_PROVIDERS, sttSelfCaptures, LANG_NAMES } from '
|
|
321
|
+
import { VoiceAgent, PiperTTS, ElevenLabsTTS, makeOpenAILLM, unlockAudio, prewarmVoice, STT_PROVIDERS, sttSelfCaptures, LANG_NAMES } from './src/index.js';
|
|
290
322
|
|
|
291
323
|
// Load VAD/WASM + build the Silero session while the page sits idle — but ONLY for a provider that
|
|
292
324
|
// will actually use it. A selfCapture provider (webspeech) builds no pipeline, so prewarming there
|
|
@@ -302,7 +334,7 @@ const fill = (id, entries) => $(id).append(...entries.map(([value, textContent])
|
|
|
302
334
|
// second source of truth that silently rots — and a demo that lies about what the library supports.
|
|
303
335
|
fill('sttProvider', Object.keys(STT_PROVIDERS).map(p => [p, p === 'webspeech' ? 'webspeech — free, in-browser' : p]));
|
|
304
336
|
fill('sttLang', Object.entries(LANG_NAMES).map(([code, name]) => [code, `${code} — ${name}`]));
|
|
305
|
-
const SETTINGS = ['mode','llmUrl','model','sttProvider','sttTokenUrl','apiKey','sttLang','
|
|
337
|
+
const SETTINGS = ['mode','llmUrl','model','sttProvider','sttTokenUrl','apiKey','sttLang','ttsProvider','ttsUrl','micDeviceId'];
|
|
306
338
|
for (const id of SETTINGS) {
|
|
307
339
|
const v = localStorage.getItem('vl.' + id); if (v) $(id).value = v;
|
|
308
340
|
$(id).addEventListener('change', () => {
|
|
@@ -317,7 +349,25 @@ const log = (...a) => {
|
|
|
317
349
|
|
|
318
350
|
// ── transcript ──────────────────────────────────────────────────────────────
|
|
319
351
|
const chat = $('chat');
|
|
320
|
-
let interimEl = null,
|
|
352
|
+
let interimEl = null, vadOff = null, spokenLen = 0;
|
|
353
|
+
// The transcript's single source of truth for replies: turn id → its bubble. EVERY assistant event
|
|
354
|
+
// routes through it — there is no separate "current bubble" to fall out of sync with. A turn's
|
|
355
|
+
// entry lives from its first painted event to its settle (final / drop / reply-scoped error), so a
|
|
356
|
+
// straggler from an interrupted turn finds its own bubble by id, wherever it sits in the chat.
|
|
357
|
+
// Ids are per-agent and monotonic, so `liveTurn` (the highest id seen) tells present from past:
|
|
358
|
+
// smaller = a straggler correcting history — it paints its bubble but drives no cosmetics.
|
|
359
|
+
const turnEls = new Map();
|
|
360
|
+
let liveTurn = 0; // turns count from 1; reset with the agent that issued them
|
|
361
|
+
|
|
362
|
+
// The dimmed tail is "written but not yet heard": on a final, a non-empty tail means the reply was
|
|
363
|
+
// cut off mid-sentence. Shared by the live path and the late-straggler path so an interrupted reply
|
|
364
|
+
// renders identically no matter when its last event lands.
|
|
365
|
+
const paintTail = (el, e) => {
|
|
366
|
+
const tail = (e.full ?? e.text).slice(e.text.length);
|
|
367
|
+
if (tail) el.append(Object.assign(document.createElement('span'), { className:'tail', textContent:tail }));
|
|
368
|
+
el.classList.toggle('cut', !!(e.final && tail.trim())); // toggle: a late repaint must also CLEAR a stale cut mark
|
|
369
|
+
return tail;
|
|
370
|
+
};
|
|
321
371
|
const toolEls = new Map();
|
|
322
372
|
const bubble = (cls, text = '') => {
|
|
323
373
|
const el = document.createElement('div');
|
|
@@ -333,11 +383,30 @@ const showLat = () => $('lat').textContent = [
|
|
|
333
383
|
tAudio && `first sound ${tAudio - t0}ms`,
|
|
334
384
|
].filter(Boolean).join(' · ');
|
|
335
385
|
|
|
336
|
-
//
|
|
337
|
-
//
|
|
338
|
-
|
|
386
|
+
// `tts` is a LET built ONLY by applyTts() — including the first one, from the RESTORED settings.
|
|
387
|
+
// It used to be initialised here with a hardcoded PiperTTS, which ignored a saved elevenlabs
|
|
388
|
+
// choice: the page came back on Piper every reload while the form still read "elevenlabs".
|
|
389
|
+
let tts; // assigned by applyTts(), below buildTts()'s definition
|
|
339
390
|
let agent = null;
|
|
340
391
|
|
|
392
|
+
// ElevenLabs needs a backend route (the key must not reach the page — see examples/token-worker).
|
|
393
|
+
// Without one this returns NULL rather than quietly substituting Piper: falling back would speak in
|
|
394
|
+
// a different voice than the form promises, and "elevenlabs selected, Piper's voice coming out" is
|
|
395
|
+
// indistinguishable from a broken integration. No synthesizer → the page says so and won't start.
|
|
396
|
+
const buildTts = () => {
|
|
397
|
+
const url = $('ttsUrl').value.trim();
|
|
398
|
+
if ($('ttsProvider').value === 'elevenlabs') {
|
|
399
|
+
if (!url) return null;
|
|
400
|
+
const voice = localStorage.getItem('vl.elVoice') || undefined; // Piper ids are not EL ids — keep them apart
|
|
401
|
+
return new ElevenLabsTTS({ ttsUrl: url, ...(voice ? { voiceId: voice } : {}) });
|
|
402
|
+
}
|
|
403
|
+
return new PiperTTS(localStorage.getItem('vl.voice') || undefined);
|
|
404
|
+
};
|
|
405
|
+
// The demo's own backend (examples/token-worker). Same origin-allowlisted Worker as /stt/token and
|
|
406
|
+
// /tts, so 'hosted' costs the visitor no key at all — the point of a demo you can just click.
|
|
407
|
+
const HOSTED_LLM_URL = 'https://voiceloop-token.havlikmarcell.workers.dev/llm';
|
|
408
|
+
const TTS_MISSING = 'elevenlabs needs a TTS url — paste your backend route (see examples/token-worker) or switch TTS back to piper';
|
|
409
|
+
|
|
341
410
|
// Echo mode: a stand-in "LLM" that repeats the turn VERBATIM — exercises the entire
|
|
342
411
|
// VAD→STT→…→TTS loop, barge-in included, with no key and no backend. Nothing is added to the
|
|
343
412
|
// text: a reply that is exactly the transcript is the whole point, since any prefix would be
|
|
@@ -371,15 +440,29 @@ const describePipeline = (a) => {
|
|
|
371
440
|
const self = a?.stt ? !!a.stt.selfCapture : sttSelfCaptures($('sttProvider').value);
|
|
372
441
|
const stt = self
|
|
373
442
|
? `<b>${$('sttProvider').value}</b> — the browser's own recognizer owns the mic <span class="warn">(voiceloop's VAD + STT pipeline bypassed)</span>${IS_IOS ? IOS_WEBSPEECH_WARNING : ''}`
|
|
374
|
-
: `<b>${$('sttProvider').value}</b> via voiceloop's pipeline — Silero VAD → feed → commit
|
|
443
|
+
: `<b>${$('sttProvider').value}</b> via voiceloop's pipeline — <span class="nb">Silero VAD → feed → commit</span>`;
|
|
444
|
+
// new URL() throws on a half-typed url — a banner must never take the page down with it.
|
|
445
|
+
const host = (u) => { try { return new URL(u).host; } catch { return u || 'unset'; } };
|
|
375
446
|
const llm = echo
|
|
376
447
|
? '<span class="warn"><b>echo</b> — no model: replies repeat your words back</span>'
|
|
377
|
-
:
|
|
448
|
+
: hosted()
|
|
449
|
+
? `<b>claude haiku</b> — hosted by this demo, no key needed`
|
|
450
|
+
: `<b>${$('model').value || 'your model'}</b> at ${host($('llmUrl').value)}`;
|
|
378
451
|
return `STT: ${stt}<br>LLM: ${llm}`;
|
|
379
452
|
};
|
|
380
453
|
const showPipeline = () => { $('pipeline').innerHTML = describePipeline(agent); };
|
|
381
454
|
// Hoisted (function declaration) so the settings listeners above can call it before this point.
|
|
382
|
-
function onSettingChanged() {
|
|
455
|
+
function onSettingChanged() {
|
|
456
|
+
// Say it with the UI, not just the banner: fields the current choices IGNORE are disabled, so the
|
|
457
|
+
// form can't imply that a typed url or key is doing something. (ttsUrl is applyTts()'s to
|
|
458
|
+
// disable — it lives on the provider-change path, not here.)
|
|
459
|
+
const custom = $('mode').value === 'llm';
|
|
460
|
+
const self = sttSelfCaptures($('sttProvider').value); // webspeech: no token route, no pipeline
|
|
461
|
+
for (const id of ['llmUrl', 'model']) $(id).disabled = !custom;
|
|
462
|
+
$('sttTokenUrl').disabled = self;
|
|
463
|
+
$('apiKey').disabled = !custom && self; // used as LLM Bearer OR token-route X-API-Key — dead only when neither hop exists
|
|
464
|
+
showPipeline(); prewarmIfPipelined();
|
|
465
|
+
}
|
|
383
466
|
|
|
384
467
|
// ── the orb's level ─────────────────────────────────────────────────────────
|
|
385
468
|
// One CSS var, --lvl (0..1), drives size/glow/rim/halo. Three sources feed it, in order of truth:
|
|
@@ -442,15 +525,20 @@ const setState = s => {
|
|
|
442
525
|
document.body.dataset.state = micOn || s !== 'listening' ? s : 'idle';
|
|
443
526
|
if (modelPct === null) $('status').textContent = STATUS[s]?.() ?? s;
|
|
444
527
|
};
|
|
445
|
-
|
|
528
|
+
// Re-attached on every rebuild: the hook lives on the TTS instance, so a new synthesizer would
|
|
529
|
+
// otherwise download its 60MB model in silence again.
|
|
530
|
+
const wireModelProgress = () => tts?.setOnModelProgress?.((loaded, total) => {
|
|
446
531
|
const done = total && loaded >= total;
|
|
447
532
|
modelPct = done ? null : total ? Math.floor((loaded / total) * 100) : 0;
|
|
448
533
|
$('status').textContent = done ? (STATUS[document.body.dataset.state]?.() ?? '')
|
|
449
534
|
: `downloading the voice — ${modelPct}% (${(loaded / 1048576).toFixed(0)}MB${total ? ` of ${(total / 1048576).toFixed(0)}MB` : ''}, once)`;
|
|
450
|
-
});
|
|
535
|
+
}); // applyTts() attaches this — including for the FIRST synthesizer, which it also builds
|
|
451
536
|
|
|
452
537
|
function onEvent(e) {
|
|
453
|
-
|
|
538
|
+
// High-rate streams stay out of the log — STT interims, the TTS spoken-cursor draft (final:false
|
|
539
|
+
// assistant events), and level ticks (4/s, the orb already shows them live). Logging them evicts
|
|
540
|
+
// everything that matters from the capped buffer; only settled events land here.
|
|
541
|
+
if ((e.type !== 'stt' || e.turnComplete) && (e.type !== 'assistant' || e.final) && e.type !== 'level') log(e);
|
|
454
542
|
switch (e.type) {
|
|
455
543
|
case 'state':
|
|
456
544
|
// The agent can stop itself (mic denied, STT socket died) — the mic light
|
|
@@ -474,24 +562,45 @@ function onEvent(e) {
|
|
|
474
562
|
ripple(); // your turn closed — the instant the latency clock starts
|
|
475
563
|
interimEl?.remove(); interimEl = null;
|
|
476
564
|
bubble('user', e.text);
|
|
477
|
-
|
|
565
|
+
t0 = Date.now(); tText = tAudio = 0; $('lat').textContent = '';
|
|
478
566
|
} else {
|
|
479
567
|
interimEl ??= bubble('user interim');
|
|
568
|
+
// A turn can die WITHOUT turnComplete (echo-dropped, retracted to '', reply error) — the
|
|
569
|
+
// interim bubble then survives at its old position and `??=` reuses it, painting the NEXT
|
|
570
|
+
// turn's live speech ABOVE messages appended since. append() moves an existing node.
|
|
571
|
+
chat.append(interimEl);
|
|
480
572
|
interimEl.textContent = `${e.committed ?? ''} ${e.text}`.trim();
|
|
481
573
|
interimEl.scrollIntoView({ block:'nearest' });
|
|
482
574
|
}
|
|
483
575
|
break;
|
|
484
576
|
case 'assistant': {
|
|
485
|
-
//
|
|
486
|
-
//
|
|
487
|
-
|
|
577
|
+
// ONE routing rule: this event paints the bubble of ITS turn — looked up in turnEls, created
|
|
578
|
+
// on the turn's first painted event. A barge-in ends turn N while its final is still in
|
|
579
|
+
// flight, so it can land after turn N+1 exists; keyed by id, it still finds its own bubble.
|
|
580
|
+
// `live` (id >= the highest seen) does NOT change where text goes — it only gates cosmetics:
|
|
581
|
+
// a straggler correcting history must not pulse the orb, stamp latency, ripple, or scroll.
|
|
582
|
+
const live = e.turn >= liveTurn;
|
|
583
|
+
if (live && e.turn > liveTurn) { liveTurn = e.turn; spokenLen = 0; } // a NEW reply starts its cursor at 0 (notify() has no stt event to reset it)
|
|
584
|
+
let el = turnEls.get(e.turn);
|
|
585
|
+
// "That draft never happened" (interrupted before a word was spoken) → drop the bubble
|
|
586
|
+
// rather than blank it. Told apart from the FIRST progress tick — also empty (cursor at 0)
|
|
587
|
+
// but carrying `full` — which must NOT tear the live bubble down.
|
|
588
|
+
if (!e.text && !e.final && e.full == null) { el?.remove(); turnEls.delete(e.turn); break; }
|
|
589
|
+
if (!el) {
|
|
590
|
+
if (!live) break; // a settled turn with no bubble has no correct place to paint — ignore
|
|
591
|
+
// Tag the bubble (never the text) when no model produced it — the reply is a verbatim
|
|
592
|
+
// echo, and only the UI should say so.
|
|
593
|
+
el = bubble($('mode').value === 'echo' ? 'ai echoed' : 'ai');
|
|
594
|
+
turnEls.set(e.turn, el);
|
|
595
|
+
}
|
|
596
|
+
el.textContent = e.text; // spoken so far
|
|
597
|
+
const tail = paintTail(el, e);
|
|
598
|
+
if (e.final) turnEls.delete(e.turn); // settled: stop tracking (the bubble stays)
|
|
599
|
+
if (!live) break; // history corrected; the present is untouched
|
|
488
600
|
if (!tText && t0) { tText = Date.now(); showLat(); }
|
|
489
|
-
// `full` only exists once the TTS progress cursor is live — i.e. a clip is
|
|
490
|
-
//
|
|
601
|
+
// `full` only exists once the TTS progress cursor is live — i.e. a clip is actually playing.
|
|
602
|
+
// The 'speaking' state is just the first token, not sound.
|
|
491
603
|
if (e.full != null && t0 && !tAudio) { tAudio = Date.now(); showLat(); }
|
|
492
|
-
// Tag the bubble (never the text) when no model produced it — the reply is a verbatim echo,
|
|
493
|
-
// and only the UI should say so.
|
|
494
|
-
aiEl ??= bubble($('mode').value === 'echo' ? 'ai echoed' : 'ai');
|
|
495
604
|
// The agent's own voice has no level event (TTS plays through an internal AEC sink), so drive
|
|
496
605
|
// the orb from how fast the spoken CURSOR advances — the orb rides its own speech. Only events
|
|
497
606
|
// carrying `full` have a cursor; a pre-audio streaming draft would otherwise run spokenLen up
|
|
@@ -500,33 +609,37 @@ function onEvent(e) {
|
|
|
500
609
|
setLevel(0.35 + Math.min(0.6, Math.max(0, e.text.length - spokenLen) * 0.09));
|
|
501
610
|
spokenLen = e.text.length;
|
|
502
611
|
}
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
if (tail) aiEl.append(Object.assign(document.createElement('span'), { className:'tail', textContent:tail }));
|
|
506
|
-
// Cut short: the final says what was HEARD (text) and what was written (full) — a gap between
|
|
507
|
-
// them is a barge-in, the one thing the transcript would otherwise render as a short answer.
|
|
508
|
-
// (trim: a natural finish can leave trailing whitespace in `full` — not an interruption)
|
|
509
|
-
if (e.final) {
|
|
510
|
-
if (tail.trim()) { aiEl.classList.add('cut'); ripple(); }
|
|
511
|
-
spokenLen = 0;
|
|
512
|
-
}
|
|
513
|
-
aiEl.scrollIntoView({ block:'nearest' });
|
|
612
|
+
if (e.final && tail.trim()) ripple();
|
|
613
|
+
el.scrollIntoView({ block:'nearest' });
|
|
514
614
|
break;
|
|
515
615
|
}
|
|
516
616
|
case 'tool': {
|
|
517
|
-
// running and result arrive as two events with one id: update, don't stack.
|
|
617
|
+
// running and result arrive as two events with one id: update, don't stack. The id stays
|
|
618
|
+
// mapped after the outcome — a tool finalized as "interrupted" whose detached run settles
|
|
619
|
+
// later corrects that SAME chip in place (re-emits are deduped library-side, and a dead
|
|
620
|
+
// agent's late event can't get here at all: makeAgent's onEvent is identity-guarded).
|
|
518
621
|
const el = toolEls.get(e.id) ?? bubble('tool');
|
|
519
622
|
if (e.id) toolEls.set(e.id, el);
|
|
520
623
|
el.textContent = `🔧 ${e.name}(${JSON.stringify(e.args)})${e.running ? ' …' : ` → ${JSON.stringify(e.result ?? '')}`.slice(0,200)}`;
|
|
521
624
|
break;
|
|
522
625
|
}
|
|
523
626
|
case 'echo': bubble('note', `self-echo dropped: ${e.text}`); break;
|
|
524
|
-
case 'error':
|
|
627
|
+
case 'error':
|
|
628
|
+
bubble('err', e.error);
|
|
629
|
+
// A reply-scoped error (it carries the turn) ends that reply without a final assistant event —
|
|
630
|
+
// release its bubble from tracking (the partial text stays visible) so the map can't grow.
|
|
631
|
+
if (e.turn != null) turnEls.delete(e.turn);
|
|
632
|
+
break;
|
|
525
633
|
}
|
|
526
634
|
}
|
|
527
635
|
|
|
528
636
|
const muted = id => $(id).dataset.on === '1';
|
|
637
|
+
const hosted = () => $('mode').value === 'hosted';
|
|
529
638
|
const makeAgent = () => {
|
|
639
|
+
// Hosted mode gets its OWN llm client rather than blanking the shared `apiKey`: that one field
|
|
640
|
+
// also authenticates the STT token route (X-API-Key), so clearing it would silently break a
|
|
641
|
+
// custom STT backend. This drops the Authorization header on the LLM hop ONLY — the hosted route
|
|
642
|
+
// holds its own key and pins its own model, so neither belongs in the request.
|
|
530
643
|
const a = new VoiceAgent({
|
|
531
644
|
llmUrl: $('llmUrl').value.trim(),
|
|
532
645
|
model: $('model').value.trim(),
|
|
@@ -537,17 +650,38 @@ const makeAgent = () => {
|
|
|
537
650
|
micDeviceId: $('micDeviceId').value,
|
|
538
651
|
persona: 'You are a concise, friendly voice assistant. Keep answers to a sentence or two.',
|
|
539
652
|
tts,
|
|
540
|
-
llm: $('mode').value === 'echo' ? echoLLM
|
|
541
|
-
|
|
653
|
+
llm: $('mode').value === 'echo' ? echoLLM
|
|
654
|
+
: hosted() ? makeOpenAILLM({ llmUrl: HOSTED_LLM_URL }) // no apiKey, no model: the route owns both
|
|
655
|
+
: undefined, // 'llm' → built from llmUrl/model/apiKey above
|
|
656
|
+
// Identity-guarded: destroy() stops the loop but a DETACHED tool result still reports through
|
|
657
|
+
// this agent's onEvent after it. Turn/tool ids are per-agent, so such a late event could match
|
|
658
|
+
// the NEXT agent's same-numbered entry and repaint its bubble — dropping everything from a
|
|
659
|
+
// non-current agent closes that whole class, instead of chasing each map it could touch.
|
|
660
|
+
onEvent: (e) => { if (agent === a) onEvent(e); },
|
|
542
661
|
});
|
|
543
662
|
a.setMuted(muted('muteMic')); a.setTtsMuted(muted('muteTts')); // the toggles survive a rebuild
|
|
544
663
|
log('agent created; stt =', a.sttProvider); showPipeline(); // now reflects the LIVE session, not just the form
|
|
545
664
|
return a;
|
|
546
665
|
};
|
|
547
666
|
|
|
667
|
+
// The ONLY way the page discards an agent. Turn ids are per-agent (each starts counting at 1), so
|
|
668
|
+
// tracking state keyed by them must die with the agent that issued them: a surviving turnEls entry
|
|
669
|
+
// would match the NEXT agent's same-numbered reply and hand it the dead agent's bubble to
|
|
670
|
+
// overwrite. (destroy() is terminal — the old agent can emit nothing further, so nothing is lost.)
|
|
671
|
+
const dropAgent = (why) => {
|
|
672
|
+
if (!agent) return;
|
|
673
|
+
agent.destroy(); agent = null; // null FIRST-equivalent: the identity guard now drops every late event from it
|
|
674
|
+
liveTurn = 0; spokenLen = 0; turnEls.clear(); // the next agent counts turns from 1 again
|
|
675
|
+
toolEls.clear(); // tool ids are provider-issued and can repeat — same hazard
|
|
676
|
+
interimEl?.remove(); interimEl = null; // a half-heard interim went nowhere; the dead agent will never settle it
|
|
677
|
+
clearTimeout(vadOff); document.body.dataset.vad = '0';
|
|
678
|
+
t0 = tText = tAudio = 0; $('lat').textContent = ''; // latency legend described the dead session
|
|
679
|
+
log(why, '→ agent rebuilt on next turn');
|
|
680
|
+
};
|
|
681
|
+
|
|
548
682
|
// ── one button does everything ──────────────────────────────────────────────
|
|
549
683
|
const stop = () => {
|
|
550
|
-
|
|
684
|
+
dropAgent('stop'); micOn = false;
|
|
551
685
|
document.body.dataset.vad = '0'; setState('idle');
|
|
552
686
|
onSettingChanged(); // describe the restored settings, and prewarm only if they'll use the VAD
|
|
553
687
|
};
|
|
@@ -557,6 +691,7 @@ $('mic').onclick = async () => {
|
|
|
557
691
|
// or not — it cancels the turn and keeps the session.
|
|
558
692
|
if (agent && (agent.state === 'thinking' || agent.state === 'speaking')) return agent.interrupt();
|
|
559
693
|
if (micOn) return stop();
|
|
694
|
+
if (!tts) return bubble('err', TTS_MISSING); // no voice → refuse to start, rather than open the mic on a loop that can't reply
|
|
560
695
|
agent ??= makeAgent();
|
|
561
696
|
micOn = true; setState('listening');
|
|
562
697
|
try { await agent.start(); }
|
|
@@ -565,6 +700,7 @@ $('mic').onclick = async () => {
|
|
|
565
700
|
|
|
566
701
|
$('send').onclick = () => {
|
|
567
702
|
const text = $('typed').value.trim(); if (!text) return;
|
|
703
|
+
if (!tts) return bubble('err', TTS_MISSING);
|
|
568
704
|
unlockAudio();
|
|
569
705
|
agent ??= makeAgent();
|
|
570
706
|
agent.sendUserText(text); // emits its own user bubble via the stt turnComplete event
|
|
@@ -573,7 +709,7 @@ $('send').onclick = () => {
|
|
|
573
709
|
$('typed').onkeydown = e => { if (e.key === 'Enter') $('send').click(); };
|
|
574
710
|
|
|
575
711
|
$('clear').onclick = () => { chat.textContent = ''; $('log').textContent = ''; $('lat').textContent = '';
|
|
576
|
-
interimEl =
|
|
712
|
+
interimEl = null; turnEls.clear(); toolEls.clear(); };
|
|
577
713
|
const toggle = (id, fn) => $(id).onclick = () => {
|
|
578
714
|
const on = $(id).dataset.on !== '1';
|
|
579
715
|
$(id).dataset.on = on ? '1' : '0';
|
|
@@ -585,19 +721,58 @@ toggle('muteTts', on => agent?.setTtsMuted(on));
|
|
|
585
721
|
|
|
586
722
|
// Voice switches live. The rest are constructor options, so they can only apply
|
|
587
723
|
// to a NEW agent — drop the current one so the next turn builds it from these.
|
|
588
|
-
$('voice').addEventListener('change', () => {
|
|
589
|
-
|
|
724
|
+
$('voice').addEventListener('change', () => {
|
|
725
|
+
// Piper ids and ElevenLabs voice ids share one picker but not one namespace — store them apart so
|
|
726
|
+
// switching provider and back doesn't hand Piper an EL id (which resolves to no voice at all).
|
|
727
|
+
if (!tts) return;
|
|
728
|
+
localStorage.setItem(tts instanceof ElevenLabsTTS ? 'vl.elVoice' : 'vl.voice', $('voice').value);
|
|
729
|
+
tts.setVoice($('voice').value); log('voice →', $('voice').value);
|
|
730
|
+
});
|
|
731
|
+
for (const id of SETTINGS.filter(i => !['ttsProvider','ttsUrl'].includes(i)))
|
|
590
732
|
$(id).addEventListener('change', () => {
|
|
591
733
|
if (micOn) return bubble('note', `stop the mic to apply ${id}`);
|
|
592
|
-
|
|
734
|
+
dropAgent(id);
|
|
593
735
|
});
|
|
594
736
|
|
|
737
|
+
// The synthesizer is a CONSTRUCTOR choice, so switching provider replaces the object — and the
|
|
738
|
+
// agent holds a reference to it, so it has to be rebuilt too (same rule as the settings above).
|
|
739
|
+
const applyTts = () => {
|
|
740
|
+
$('ttsUrl').disabled = $('ttsProvider').value !== 'elevenlabs';
|
|
741
|
+
tts = buildTts();
|
|
742
|
+
if (!tts) bubble('note', TTS_MISSING);
|
|
743
|
+
wireModelProgress();
|
|
744
|
+
dropAgent('tts');
|
|
745
|
+
loadVoices();
|
|
746
|
+
log('tts →', $('ttsProvider').value);
|
|
747
|
+
};
|
|
748
|
+
let ttsUrlTimer;
|
|
749
|
+
for (const id of ['ttsProvider', 'ttsUrl'])
|
|
750
|
+
for (const ev of ['change', 'input'])
|
|
751
|
+
$(id).addEventListener(ev, () => {
|
|
752
|
+
if (micOn) return bubble('note', `stop the mic to apply ${id}`);
|
|
753
|
+
localStorage.setItem('vl.' + id, $(id).value);
|
|
754
|
+
clearTimeout(ttsUrlTimer);
|
|
755
|
+
ttsUrlTimer = setTimeout(applyTts, ev === 'input' ? 400 : 0); // don't rebuild on every keystroke
|
|
756
|
+
});
|
|
757
|
+
|
|
595
758
|
// ── device + voice lists ────────────────────────────────────────────────────
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
759
|
+
// Re-run on every rebuild: the voice list belongs to the synthesizer, and a stale Piper list would
|
|
760
|
+
// offer ids ElevenLabs can't speak.
|
|
761
|
+
function loadVoices() {
|
|
762
|
+
$('voice').replaceChildren();
|
|
763
|
+
if (!tts) return fill('voice', [['', TTS_MISSING]]); // nothing to ask — say why, where the user is looking
|
|
764
|
+
// Ask the synthesizer that is ACTUALLY live, never the form: buildTts() is the one place that
|
|
765
|
+
// decides. Re-reading the form here would be a second copy of that rule, free to disagree with
|
|
766
|
+
// it — and a picker offering ids the live voice can't speak is exactly the bug this avoids.
|
|
767
|
+
const key = tts instanceof ElevenLabsTTS ? 'vl.elVoice' : 'vl.voice';
|
|
768
|
+
tts.voices().then(vs => {
|
|
769
|
+
if (!vs.length) return fill('voice', [['', "couldn't reach the TTS url — check the route + its origin allowlist"]]);
|
|
770
|
+
fill('voice', vs.map(v => [v.id, v.name || v.id]));
|
|
771
|
+
$('voice').value = localStorage.getItem(key) || tts.voiceId; // the library's own default, not a copy of it
|
|
772
|
+
if ($('voice').value) tts.setVoice($('voice').value);
|
|
773
|
+
}).catch(e => log('voice list failed', String(e)));
|
|
774
|
+
}
|
|
775
|
+
applyTts();
|
|
601
776
|
navigator.mediaDevices?.enumerateDevices().then(ds => {
|
|
602
777
|
for (const d of ds.filter(d => d.kind === 'audioinput'))
|
|
603
778
|
$('micDeviceId').append(Object.assign(document.createElement('option'),
|
|
@@ -607,6 +782,7 @@ navigator.mediaDevices?.enumerateDevices().then(ds => {
|
|
|
607
782
|
setState('idle');
|
|
608
783
|
onSettingChanged(); // the banner must describe the restored settings from the FIRST frame, not only after a click
|
|
609
784
|
window.agent = () => agent; // console poking: agent().history, …
|
|
785
|
+
window.tts = () => tts; // …and the live synthesizer: tts().voiceId, tts().constructor.name
|
|
610
786
|
</script>
|
|
611
787
|
</body>
|
|
612
788
|
</html>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@todoforai/voiceloop",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.9",
|
|
4
4
|
"description": "Fast, interruptible browser voice agent: VAD → STT → LLM → TTS with real barge-in, echo filtering, streaming first-sentence TTS and tap-to-seek. Zero runtime dependencies.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -17,7 +17,8 @@
|
|
|
17
17
|
"files": [
|
|
18
18
|
"src",
|
|
19
19
|
"!src/*.test.js",
|
|
20
|
-
"examples"
|
|
20
|
+
"examples",
|
|
21
|
+
"index.html"
|
|
21
22
|
],
|
|
22
23
|
"scripts": {
|
|
23
24
|
"test": "node --test src/*.test.js"
|
package/src/index.d.ts
CHANGED
|
@@ -5,7 +5,11 @@
|
|
|
5
5
|
export type VoiceAgentEvent =
|
|
6
6
|
| { type: 'state'; state: 'idle' | 'listening' | 'thinking' | 'speaking' }
|
|
7
7
|
| { type: 'stt'; turnComplete: boolean; text: string; ms: number; committed?: string } // text=live tail; committed=locked segments (Speechmatics AddTranscript); turnComplete=user turn ended → send to LLM
|
|
8
|
-
|
|
8
|
+
// text=spoken-so-far (solid, advances live as TTS plays); full=complete answer → tail
|
|
9
|
+
// full.slice(text.length) not-yet-spoken (dim). turn identifies the reply: a barge-in ends turn N
|
|
10
|
+
// while its final event is still in flight, so it can land AFTER turn N+1 started — route by turn,
|
|
11
|
+
// not by "the current bubble", or the late text overwrites the newer reply.
|
|
12
|
+
| { type: 'assistant'; text: string; final: boolean; full?: string; turn: number }
|
|
9
13
|
| { type: 'vad'; active: boolean } // VAD hears speech?
|
|
10
14
|
// Mic loudness, 0..1 on a perceptual curve, ~every 256ms while capturing. Cosmetic only (meters,
|
|
11
15
|
// a reacting orb) — nothing in the pipeline reads it. Never emitted for a selfCapture STT
|
|
@@ -20,7 +24,7 @@ export type VoiceAgentEvent =
|
|
|
20
24
|
// Did that speculation pay off? hit = the real reply opened with the presynthesized text (clip
|
|
21
25
|
// reused, latency saved); miss = it didn't (clip discarded). `pre` is what we had guessed.
|
|
22
26
|
| { type: 'diag'; diag: 'presynth-hit' | 'presynth-miss'; text: string; pre: string }
|
|
23
|
-
| { type: 'error'; error: string };
|
|
27
|
+
| { type: 'error'; error: string; turn?: number }; // turn present = that reply's TERMINAL event (a reply-killing TTS failure emits no assistant final)
|
|
24
28
|
|
|
25
29
|
export interface VoiceTool {
|
|
26
30
|
description?: string;
|
|
@@ -326,9 +330,10 @@ export class StreamingTTS implements VoiceTTS {
|
|
|
326
330
|
seek(nwIndex: number): boolean;
|
|
327
331
|
}
|
|
328
332
|
/** Human-grade cloud voices. Pass `ttsUrl` (your backend proxy, key stays server-side); `apiKey`
|
|
329
|
-
* talks to ElevenLabs directly and is dev-only.
|
|
333
|
+
* talks to ElevenLabs directly and is dev-only. `headers` authenticate proxy requests
|
|
334
|
+
* (e.g. { 'X-API-Key': ... }) and are never sent on a direct ElevenLabs call. */
|
|
330
335
|
export class ElevenLabsTTS extends StreamingTTS {
|
|
331
|
-
constructor(options?: { ttsUrl?: string; apiKey?: string; voiceId?: string; modelId?: string; format?: string });
|
|
336
|
+
constructor(options?: { ttsUrl?: string; apiKey?: string; voiceId?: string; modelId?: string; format?: string; headers?: Record<string, string> });
|
|
332
337
|
warm(): Promise<void>;
|
|
333
338
|
}
|
|
334
339
|
|
package/src/stt.js
CHANGED
|
@@ -187,6 +187,7 @@ export function makeElevenLabsSTT(opts) {
|
|
|
187
187
|
open() { if (!isClosed() && (!ws || ws.readyState >= 2)) open(); },
|
|
188
188
|
feed(i16) {
|
|
189
189
|
if (!isClosed() && (!ws || ws.readyState >= 2)) open();
|
|
190
|
+
if (!sentSinceCommit) sttStart = performance.now(); // first audio of a fresh turn anchors its ms clock (audio is VAD-gated ≈ speech start)
|
|
190
191
|
sendAudio(i16); sentSinceCommit = true;
|
|
191
192
|
},
|
|
192
193
|
commit() {
|
|
@@ -412,6 +413,7 @@ export function makeSpeechmaticsSTT(opts) {
|
|
|
412
413
|
open() { if (!isClosed() && (!ws || ws.readyState >= 2)) open(); },
|
|
413
414
|
feed(i16) {
|
|
414
415
|
if (!isClosed() && (!ws || ws.readyState >= 2)) open();
|
|
416
|
+
if (!sentSinceCommit) sttStart = performance.now(); // first audio of a fresh turn anchors its ms clock (audio is VAD-gated ≈ speech start)
|
|
415
417
|
sentSamples += i16.length; // stream position (counts outbox-queued audio too — sent in order)
|
|
416
418
|
sendAudio(i16); sentSinceCommit = true;
|
|
417
419
|
},
|
|
@@ -545,6 +547,10 @@ export function makeDeepgramSTT(opts) {
|
|
|
545
547
|
ws.onmessage = ev => {
|
|
546
548
|
if (typeof ev.data !== 'string') return;
|
|
547
549
|
const m = JSON.parse(ev.data);
|
|
550
|
+
// Continuous stream: the socket lives for the whole session, so the ms clock re-anchors at
|
|
551
|
+
// each turn's FIRST transcript activity (turnText empty → this is the turn's opening event),
|
|
552
|
+
// not at socket open — otherwise ms grows monotonically across the session.
|
|
553
|
+
if (m.type === 'TurnInfo' && !turnText && (m.transcript || '').trim()) sttStart = performance.now();
|
|
548
554
|
const ms = Math.round(performance.now() - sttStart);
|
|
549
555
|
if (m.type === 'TurnInfo') {
|
|
550
556
|
const t = (m.transcript || '').trim();
|
package/src/tts-elevenlabs.js
CHANGED
|
@@ -5,7 +5,9 @@
|
|
|
5
5
|
//
|
|
6
6
|
// Auth, two modes (mirrors the STT providers' "key stays server-side" stance):
|
|
7
7
|
// • ttsUrl — YOUR backend proxy: POST { text, voice_id, model_id } → audio bytes. The proxy adds
|
|
8
|
-
// the xi-api-key and forwards to ElevenLabs. Recommended for production.
|
|
8
|
+
// the xi-api-key and forwards to ElevenLabs. Recommended for production. Pass
|
|
9
|
+
// `headers` to authenticate to YOUR proxy (e.g. { 'X-API-Key': ... }) — they ride on
|
|
10
|
+
// every proxy request (synth + voices), never on a direct ElevenLabs call.
|
|
9
11
|
// • apiKey — direct browser→ElevenLabs (dev/bench only: the key is visible in the page).
|
|
10
12
|
//
|
|
11
13
|
// Latency: eleven_flash_v2_5 is EL's low-latency model (~75ms model time); the rest of a clip's
|
|
@@ -17,10 +19,27 @@ const EL_API = 'https://api.elevenlabs.io/v1/text-to-speech';
|
|
|
17
19
|
|
|
18
20
|
export class ElevenLabsTTS extends StreamingTTS {
|
|
19
21
|
constructor({ ttsUrl, apiKey, voiceId = 'JBFqnCBsd6RMkjVDRZzb' /* George */,
|
|
20
|
-
modelId = 'eleven_flash_v2_5', format = 'mp3_44100_64' } = {}) {
|
|
22
|
+
modelId = 'eleven_flash_v2_5', format = 'mp3_44100_64', headers = {} } = {}) {
|
|
21
23
|
super(voiceId);
|
|
22
24
|
if (!ttsUrl && !apiKey) throw new Error('ElevenLabsTTS needs ttsUrl (backend proxy) or apiKey (direct, dev only)');
|
|
23
25
|
this.ttsUrl = ttsUrl; this.apiKey = apiKey; this.modelId = modelId; this.format = format;
|
|
26
|
+
this.headers = headers; // proxy-path auth (X-API-Key etc.) — never sent to ElevenLabs directly
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
// Listing voices needs the key, so on the proxy path it has to come from YOUR backend: a GET on
|
|
30
|
+
// the same ttsUrl returns [{ id, name }] (see examples/token-worker). Direct mode asks ElevenLabs
|
|
31
|
+
// with the key it already has. Either way a failure is non-fatal — an empty list just means the
|
|
32
|
+
// host keeps whatever voiceId it was constructed with, and synthesis is unaffected.
|
|
33
|
+
async voices() {
|
|
34
|
+
const [url, headers] = this.ttsUrl
|
|
35
|
+
? [this.ttsUrl, this.headers]
|
|
36
|
+
: ['https://api.elevenlabs.io/v2/voices?page_size=100', { 'xi-api-key': this.apiKey }];
|
|
37
|
+
try {
|
|
38
|
+
const r = await fetch(url, { headers, signal: AbortSignal.timeout(5000) });
|
|
39
|
+
if (!r.ok) return [];
|
|
40
|
+
const body = await r.json();
|
|
41
|
+
return (body.voices ?? []).map((v) => ({ id: v.voice_id ?? v.id, name: v.name }));
|
|
42
|
+
} catch { return []; }
|
|
24
43
|
}
|
|
25
44
|
|
|
26
45
|
// Pre-establish the connection (DNS + TLS + any proxy hop) so the first real clip only pays TTFB.
|
|
@@ -37,7 +56,7 @@ export class ElevenLabsTTS extends StreamingTTS {
|
|
|
37
56
|
const url = direct ? `${EL_API}/${this.voiceId}?output_format=${this.format}` : this.ttsUrl;
|
|
38
57
|
const res = await fetch(url, {
|
|
39
58
|
method: 'POST',
|
|
40
|
-
headers: { 'Content-Type': 'application/json', ...(direct ? { 'xi-api-key': this.apiKey } :
|
|
59
|
+
headers: { 'Content-Type': 'application/json', ...(direct ? { 'xi-api-key': this.apiKey } : this.headers) },
|
|
41
60
|
// The proxy form carries voice/model/format in the body so one backend route serves any voice.
|
|
42
61
|
body: JSON.stringify(direct ? { text, model_id: this.modelId }
|
|
43
62
|
: { text, voice_id: this.voiceId, model_id: this.modelId, output_format: this.format }),
|
package/src/voice-agent.js
CHANGED
|
@@ -909,7 +909,7 @@ export class VoiceAgent {
|
|
|
909
909
|
if (this.history[this.history.length - 1]?.role !== 'user') return;
|
|
910
910
|
this._set('thinking'); // LLM stage
|
|
911
911
|
this._abort = pf?.ctl ?? new AbortController(); // adopted prefetch keeps ITS controller so barge-in aborts the right fetch
|
|
912
|
-
await this._speakTurn(this._abort.signal, pf?.gen); // _speakTurn never throws — errors surface via onEvent
|
|
912
|
+
await this._speakTurn(this._abort.signal, pf?.gen, seq); // _speakTurn never throws — errors surface via onEvent
|
|
913
913
|
})());
|
|
914
914
|
}
|
|
915
915
|
|
|
@@ -917,7 +917,9 @@ export class VoiceAgent {
|
|
|
917
917
|
// prefix, not the full generated answer) plus a ledger of the tools it dispatched. Stored as
|
|
918
918
|
// `this._turn` so the next user turn can await it. Never throws (errors surface via onEvent) so
|
|
919
919
|
// the await is safe — and never awaits a tool, so a hung one can't hold the next turn hostage.
|
|
920
|
-
|
|
920
|
+
// `turn` tags every assistant event with the reply it belongs to, so a host can tell a LATE
|
|
921
|
+
// event from a superseded turn apart from one for the current turn (see the demo's transcript).
|
|
922
|
+
async _speakTurn(signal, prefetched = null, turn = 0) {
|
|
921
923
|
// Fresh turn → drop the previous reply's audible text so user speech heard before THIS turn's
|
|
922
924
|
// first audio plays can't be misjudged as echo of the OLD reply — but PARK it in _prevReplyText:
|
|
923
925
|
// its speakers→mic echo tail is still in flight for a moment (acoustic delay + STT latency), and
|
|
@@ -989,7 +991,7 @@ export class VoiceAgent {
|
|
|
989
991
|
// read live each tick), so streaming the whole answer here would yank the cursor to the end.
|
|
990
992
|
// Once the cursor is live we only grow `answer` (the hook reads it); until then (muted, or the
|
|
991
993
|
// first sentence is still synthesizing) we stream the full answer so the bubble isn't empty.
|
|
992
|
-
if (!self._cursorLive) self.onEvent({ type: 'assistant', text: answer, final: false });
|
|
994
|
+
if (!self._cursorLive) self.onEvent({ type: 'assistant', text: answer, final: false, turn });
|
|
993
995
|
yield item.text;
|
|
994
996
|
}
|
|
995
997
|
})(this, stripToolCallMimicry(prefetched ?? this.llm(this.history, this.sysmsg, signal))); // prefetched stream already carries this turn's user text (see _startPrefetch)
|
|
@@ -1010,13 +1012,20 @@ export class VoiceAgent {
|
|
|
1010
1012
|
return answer.slice(0, i);
|
|
1011
1013
|
};
|
|
1012
1014
|
this._cursorLive = false;
|
|
1015
|
+
let lastDraft = ''; // last emitted heard+answer — the rAF ticker repeats itself
|
|
1013
1016
|
this.tts.setOnProgress?.((spokenSoFar, scope) => {
|
|
1014
1017
|
if (signal.aborted) return;
|
|
1015
1018
|
this._cursorLive = true; // hook now drives the draft → LLM stream stops emitting full text
|
|
1016
1019
|
const heard = rawPrefix(spokenSoFar);
|
|
1017
1020
|
this._replyText = heard; // AUDIBLE prefix — drives the spoken cursor
|
|
1018
1021
|
this._replyEcho = scope ? rawPrefix(scope) : heard; // clip-bounded echo reference (cursor lags real audio)
|
|
1019
|
-
|
|
1022
|
+
// The player ticks every animation frame but the time→text estimate only advances a char every
|
|
1023
|
+
// few frames — drop the identical repeats so hosts see one event per visible change, not 60/s.
|
|
1024
|
+
// (`answer` is in the key too: while streaming, `full` still grows between identical cursors.)
|
|
1025
|
+
const draft = heard.length + '\u0000' + answer.length;
|
|
1026
|
+
if (draft === lastDraft) return;
|
|
1027
|
+
lastDraft = draft;
|
|
1028
|
+
this.onEvent({ type: 'assistant', text: heard, full: answer, final: false, turn });
|
|
1020
1029
|
});
|
|
1021
1030
|
try {
|
|
1022
1031
|
// TTS muted: drain the same stream so the transcript + tool calls still run, but never voice
|
|
@@ -1030,7 +1039,9 @@ export class VoiceAgent {
|
|
|
1030
1039
|
} catch (e) {
|
|
1031
1040
|
if (e.name !== 'AbortError') { // real failure (not barge-in) → surface it
|
|
1032
1041
|
this.tts.setOnProgress?.(null); this._cursorLive = false;
|
|
1033
|
-
|
|
1042
|
+
// Tagged with the turn: this error is this reply's TERMINAL event (no assistant final
|
|
1043
|
+
// follows), and a host tracking bubbles per turn needs to know which one just died.
|
|
1044
|
+
this.onEvent({ type: 'error', error: e.message, turn });
|
|
1034
1045
|
// Tools fire DURING streaming, so a reply that dies here may already have dispatched one.
|
|
1035
1046
|
// Record it anyway: the ledger is about what was DONE, not about what got spoken — dropping
|
|
1036
1047
|
// it would let the next turn call the same tool again.
|
|
@@ -1063,11 +1074,11 @@ export class VoiceAgent {
|
|
|
1063
1074
|
// full.slice(text.length) tail aligns. keepFull/normal-finish → whole answer (no tail); a barge-in
|
|
1064
1075
|
// → the heard prefix solid + the unspoken remainder dimmed, showing exactly where TTS got.
|
|
1065
1076
|
const shown = signal.keepFull ? answer : rawPrefix(spoken);
|
|
1066
|
-
this.onEvent({ type: 'assistant', text: shown, full: answer, final: true });
|
|
1077
|
+
this.onEvent({ type: 'assistant', text: shown, full: answer, final: true, turn });
|
|
1067
1078
|
} else if (answer) {
|
|
1068
1079
|
// Interrupted before anything was heard (nothing voiced yet): drop the streamed draft so the
|
|
1069
1080
|
// transcript doesn't keep a dangling unfinished bubble. Empty text → host clears the draft.
|
|
1070
|
-
this.onEvent({ type: 'assistant', text: '', final: false });
|
|
1081
|
+
this.onEvent({ type: 'assistant', text: '', final: false, turn });
|
|
1071
1082
|
}
|
|
1072
1083
|
// Tools are NOT joined here. They were dispatched during streaming (in parallel with TTS) and
|
|
1073
1084
|
// _runTool reports its own outcome via onEvent, so waiting adds nothing — while a tool that
|
|
@@ -1849,7 +1860,18 @@ export class StreamingTTS {
|
|
|
1849
1860
|
if (idx >= tape.length) { await awaitEntry(idx); if (takePending()) continue; if (idx >= tape.length) break; } // wait for the producer to reach this sentence (stream end → done)
|
|
1850
1861
|
const e = tape[idx];
|
|
1851
1862
|
const buf = await e.blobP;
|
|
1852
|
-
|
|
1863
|
+
// A clip that failed to RENDER is a voice problem, not an answer problem: the model already
|
|
1864
|
+
// produced this text and the user is already reading it. Throwing here abandoned every
|
|
1865
|
+
// REMAINING sentence — the reply went silent partway through and speak() resolved with only
|
|
1866
|
+
// the prefix, so history recorded less than what was on screen. Skip the mute sentence
|
|
1867
|
+
// instead, count it as delivered (heardMax), and keep speaking the rest.
|
|
1868
|
+
if (e.err) {
|
|
1869
|
+
// Say it out loud on the same channel as the other TTS diagnostics: a sentence that is
|
|
1870
|
+
// silently skipped is exactly the "it just got lost" symptom this branch exists to avoid.
|
|
1871
|
+
this.onEvent?.({ type: 'error', error: `TTS synth failed, sentence not voiced: ${e.err.message}` });
|
|
1872
|
+
heardMax = Math.max(heardMax, e.nwStart + nw(e.text));
|
|
1873
|
+
idx++; startNw = 0; continue;
|
|
1874
|
+
}
|
|
1853
1875
|
if (takePending()) continue; // a tap during synth/stream gap → reposition
|
|
1854
1876
|
this._curIdx = idx;
|
|
1855
1877
|
const prefix = tape.slice(0, idx).map(x => x.text).join(' ') + (idx ? ' ' : '');
|