@todoforai/voiceloop 0.1.5 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -76,7 +76,8 @@ the place to start reading.
76
76
 
77
77
  ```
78
78
  you stop speaking ──┐
79
- │ end-of-turn debounce (~600ms)
79
+ │ end-of-turn debounce (500ms configured; ~1.5s in practice
80
+ │ on browser STT — see the webspeech note below)
80
81
  LLM prefetch ────┤ ← already streaming since your interim transcript stabilized
81
82
  │
82
83
  first LLM tokens ───┤ first sentence boundary (~8+ chars)
@@ -96,6 +97,7 @@ into the mic (no AEC):
96
97
  | OpenAI Realtime (own LLM)* | 870ms | **0/30** | **17/30** |
97
98
  | **voiceloop** (deepgram + ElevenLabs flash) | **980ms** | **0/30** | **0/30** |
98
99
  | **voiceloop** (deepgram + Piper, free local TTS) | **970ms** | **0/30** | — |
100
+ | **voiceloop** (webspeech + Piper, zero-key default) | 2110ms | — | — |
99
101
  | Pipecat 1.8.1 (same providers) | 1050ms | 2/30 | **20/30** |
100
102
  | ElevenLabs ConvAI | 1450ms | **0/30** | 0/30 |
101
103
 
@@ -114,7 +116,7 @@ Full tables, method and reproduction steps: [`results/RESULTS.md`](https://githu
114
116
  ## STT providers
115
117
 
116
118
  ```js
117
- new VoiceAgent({ sttProvider: 'webspeech' }) // default: free, browser-native (Chrome/Safari)
119
+ new VoiceAgent({ sttProvider: 'webspeech' }) // default: zero-key, browser-native (Chrome/Safari) — but ~1.2s slower to close a turn; see the table below
118
120
  new VoiceAgent({ sttProvider: 'elevenlabs', sttTokenUrl: '/api/stt/token' })
119
121
  new VoiceAgent({ sttProvider: 'deepgram', sttTokenUrl: '/api/stt/token' })
120
122
  new VoiceAgent({ sttProvider: 'speechmatics', sttTokenUrl: '/api/stt/token' })
@@ -134,7 +136,7 @@ On browsers without SpeechRecognition (Firefox, WebKitGTK) `webspeech` falls bac
134
136
 
135
137
  | Provider | End-of-turn | Notes |
136
138
  |---|---|---|
137
- | `webspeech` | Native (browser-managed) | Free, on-device, Chrome/Safari only |
139
+ | `webspeech` | Native (browser-managed) | Zero-key (no token endpoint), Chrome/Safari only. **~2.1s voice→voice** — ~1.2s slower to close a turn than cloud STT ([measured](bench/results/RESULTS.md)); it also owns the mic, so voiceloop's VAD/echo pipeline is bypassed. Recognition is handled by the browser, which may use a vendor cloud service — it is not guaranteed on-device |
138
140
  | `elevenlabs` | VAD-gated | Scribe v2 realtime, great accuracy |
139
141
  | `deepgram` | **Native** (Flux) | Provider's own turn model hears the full stream |
140
142
  | `speechmatics` | VAD-gated | Cheapest per second, locked-words-only finals |
@@ -145,6 +147,7 @@ The default adapter speaks OpenAI's streaming wire format (OpenAI, Groq, Cerebra
145
147
 
146
148
  ```js
147
149
  const agent = new VoiceAgent({
150
+ // TS: annotate as `LLM` (exported) and every parameter is inferred.
148
151
  llm: async function* (history, system, signal, { toolGate } = {}) {
149
152
  // history: [{ role: 'user'|'assistant', content }], system: composed persona+context
150
153
  // toolGate: only for adapters that run tools themselves — see below; ignore it otherwise
@@ -212,6 +215,26 @@ agent.replay(text, fromNw?, onProgress?) // re-voice a past reply; onProgress(
212
215
  agent.dumpAudio() // last 30s of mic audio as WAV + stall report (debugging)
213
216
  ```
214
217
 
218
+ ### Constructor options
219
+
220
+ Full types (with per-option notes) in [`src/index.d.ts`](src/index.d.ts).
221
+
222
+ | | |
223
+ |---|---|
224
+ | `persona`, `sysmsg` | identity (static) and live context (`setSysmsg()` mid-session) |
225
+ | `llmUrl`, `model`, `apiKey`, `maxTokens` | built-in OpenAI-compatible adapter |
226
+ | `llm` | your own adapter — replaces all of the above |
227
+ | `tools` | what the model may call ([Tools](#tools)) |
228
+ | `tts`, `speed` | TTS engine (default Piper) and Piper's speaking rate |
229
+ | `sttProvider`, `sttLang`, `sttModel`, `sttUrl` | which recognizer, which language |
230
+ | `sttTokenUrl` / `getSttToken`, `sttUsageUrl` | token minting ([STT providers](#stt-providers)); usage metering |
231
+ | `keyterms` | domain words to bias STT towards (max 50, ≤20 chars) |
232
+ | `micDeviceId` | input device; also `start(deviceId)` |
233
+ | `turnDetector`, `maxPauseMs`, `sttEotThreshold` | when a user's turn is over |
234
+ | `bargeInMinChars`, `vadOptions`, `preroll` | interruption sensitivity and mic gating |
235
+ | `fetchFn` | transport seam for the built-in adapter (tests, request wrapping) |
236
+ | `onEvent` | every event below |
237
+
215
238
  Events via `onEvent(e)`: `state`, `stt`, `assistant`, `tool`, `vad`, `echo`, `error`, `diag`.
216
239
  A `tool` event carries `{ name, id, args }` plus either `result` (settled) or `running: true` (an [adapter-announced](#adapters-that-run-tools-themselves) call still executing) — render the pair as one chip keyed by `id`.
217
240
 
@@ -245,7 +268,7 @@ Custom engines subclass `StreamingTTS` and implement one method:
245
268
 
246
269
  ```js
247
270
  class MyTTS extends StreamingTTS {
248
- async _synth(text) { return wavBlob; } // sentence in, audio Blob out
271
+ async _synth(text, signal) { return wavBlob; } // sentence in, audio Blob out (null if aborted)
249
272
  }
250
273
  ```
251
274
 
@@ -254,7 +277,7 @@ Sentence chunking, the playback tape, tap-to-seek, barge-in and progress reporti
254
277
  ## Testing
255
278
 
256
279
  ```
257
- npm test # 101 tests, no browser needed
280
+ npm test # no browser needed
258
281
  ```
259
282
 
260
283
  The suite locks in turn serialization, hold/release semantics, tool dedup, self-echo classification, prefetch adoption rules, tape seeking, and each STT provider's turn-boundary state machine.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@todoforai/voiceloop",
3
- "version": "0.1.5",
3
+ "version": "0.1.6",
4
4
  "description": "Fast, interruptible browser voice agent: VAD → STT → LLM → TTS with real barge-in, echo filtering, streaming first-sentence TTS and tap-to-seek. Zero runtime dependencies.",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/index.d.ts CHANGED
@@ -13,6 +13,9 @@ export type VoiceAgentEvent =
13
13
  | { type: 'echo'; text: string } // dropped self-echo (the agent's own TTS leaked into the mic) — diagnostics only
14
14
  | { type: 'diag'; diag: 'aec-fallback'; message: string } // once/session: AEC loopback failed → TTS plays un-cancelled (self-echo only text-filtered)
15
15
  | { type: 'diag'; diag: 'presynth'; text: string } // a reply's first clip was synthesized speculatively (latency diagnostics)
16
+ // Did that speculation pay off? hit = the real reply opened with the presynthesized text (clip
17
+ // reused, latency saved); miss = it didn't (clip discarded). `pre` is what we had guessed.
18
+ | { type: 'diag'; diag: 'presynth-hit' | 'presynth-miss'; text: string; pre: string }
16
19
  | { type: 'error'; error: string };
17
20
 
18
21
  export interface VoiceTool {
@@ -34,21 +37,31 @@ export type LLMChunk =
34
37
  | { text: string }
35
38
  | { tool: string; id?: string; args: Record<string, unknown>; result?: unknown; error?: string; running?: true };
36
39
  // Pluggable LLM: lazy async generator over the chat history (fetch fires on first pull).
37
- // The default (agent-llm.js, vendored Pi loop) accepts an optional 4th arg: `toolGate` is awaited
38
- // before ANY tool executes — speculative prefetch passes one that resolves on turn commit, so a
39
- // speculation streams text early but can never fire a side effect from uncommitted speech.
40
- // Custom LLMs are a low-level seam (tests, embedding hosts): their bare tool chunks are executed
41
- // by the agent, but the results are NOT fed back into that LLM mid-turn — such calls are terminal
42
- // for the turn. They may ignore toolGate.
43
- export type LLM = (
44
- history: Array<{ role: string; content: string }>,
45
- system: string,
46
- signal: AbortSignal,
47
- options?: { toolGate?: Promise<unknown> },
48
- ) => AsyncIterable<LLMChunk>;
40
+ // The default (makeOpenAILLM) yields bare tool chunks that the AGENT executes; the results are not
41
+ // fed back into it mid-turn, so such calls are terminal for the turn. That is the low-level seam.
42
+ // An agent LOOP that runs tools inside its own generator declares the two flags below.
43
+ export interface LLM {
44
+ (
45
+ history: Array<{ role: string; content: string }>,
46
+ system: string,
47
+ signal: AbortSignal,
48
+ options?: { toolGate?: Promise<unknown> },
49
+ ): AsyncIterable<LLMChunk>;
50
+ /** This generator ran the tool itself: chunks carry `result`/`error` and the agent REPORTS them
51
+ * instead of executing again. Without it, an outcome-bearing chunk is re-run. */
52
+ executesTools?: boolean;
53
+ /** This generator awaits `options.toolGate` before ANY tool executes. Required to keep
54
+ * speculative prefetch on: prefetch starts from an uncommitted interim, and the gate is what
55
+ * stops a speculation from firing a real side effect. `executesTools` without this silently
56
+ * disables prefetch (see voice-agent.js:793). */
57
+ acceptsToolGate?: boolean;
58
+ }
49
59
 
50
60
  // Pluggable TTS contract. `speak` resolves with the text actually HEARD (full on completion,
51
61
  // a prefix on barge-in). The optional members back the voice picker / switching.
62
+ /** (spokenSoFar, scope) — see VoiceTTS.setOnProgress. */
63
+ export type TtsProgress = (spokenText: string, scope: string) => void;
64
+
52
65
  export interface VoiceTTS {
53
66
  speak(input: AsyncIterable<string> | string, signal?: AbortSignal, fromNw?: number): Promise<string>;
54
67
  stop?(): void;
@@ -58,8 +71,10 @@ export interface VoiceTTS {
58
71
  setVoice?(voiceId: string): void;
59
72
  setSpeed?(speed: number): void;
60
73
  /** Register a callback fired while a clip plays with the chars spoken SO FAR across the whole
61
- * reply — drives the host's live spoken cursor. Pass null to clear. */
62
- setOnProgress?(fn: ((spokenText: string) => void) | null): void;
74
+ * reply — drives the host's live spoken cursor. Pass null to clear.
75
+ * `scope` is that prefix PLUS the whole clip now playing: the cursor lags real audio, so echo
76
+ * suppression judges against this upper bound rather than the spoken text. */
77
+ setOnProgress?(fn: TtsProgress | null): void;
63
78
  }
64
79
 
65
80
  export interface VoiceAgentOptions {
@@ -70,7 +85,8 @@ export interface VoiceAgentOptions {
70
85
  /** LLM model id sent to the backend `/llm` route; empty → the backend's VOICE_MODEL default. */
71
86
  model?: string;
72
87
  llm?: LLM;
73
- /** Test seam for the agent-loop LLM transport; defaults to global fetch. */
88
+ /** Transport seam for the built-in LLM adapter; defaults to global fetch. For tests and hosts
89
+ * that must wrap every request (auth refresh, tracing). Unused when you pass your own `llm`. */
74
90
  fetchFn?: typeof fetch;
75
91
  apiKey?: string;
76
92
  /** Full endpoint URLs, not a base: voiceloop composes no routes, so a host can point the LLM and
@@ -99,8 +115,15 @@ export interface VoiceAgentOptions {
99
115
  /** Mint a short-TTL STT token yourself, for auth that doesn't fit one POST route. Called with the
100
116
  * provider actually RUNNING (post-downgrade), never the requested one. Wins over sttTokenUrl. */
101
117
  getSttToken?: (provider: SttProvider) => Promise<SttToken>;
118
+ /** POST route that receives metered STT seconds: `{ seconds, provider, ... }`, batched and
119
+ * fire-and-forget (failures are swallowed — billing telemetry never breaks the pipeline).
120
+ * Omit it and nothing is reported. Cloud providers only; the browser's own STT is free. */
102
121
  sttUsageUrl?: string;
122
+ /** Override the provider's realtime WS endpoint (self-hosted/regional gateways). Default: the
123
+ * provider's public URL. */
103
124
  sttUrl?: string;
125
+ /** Provider STT model id (e.g. Deepgram's flux variants); empty → the provider's default.
126
+ * Dropped when the provider downgrades, since a model id is meaningless across providers. */
104
127
  sttModel?: string;
105
128
  /** Deepgram Flux end-of-turn confidence (0.5–0.9) — its model closes the turn itself once its
106
129
  * confidence the user is done exceeds this threshold. Lower = snappier turn ends, higher = more
@@ -109,8 +132,13 @@ export interface VoiceAgentOptions {
109
132
  /** Every tool the model may call. voiceloop ships NO built-in tools — a voice library has no
110
133
  * business knowing what a "todo" is — so product tools (and their `run`) are supplied here. */
111
134
  tools?: Record<string, VoiceTool>;
135
+ /** Domain words to bias cloud STT towards ("Kubernetes", product names) — the fix for jargon
136
+ * transcribed phonetically. Silently trimmed to 50 terms of ≤20 chars. Cloud providers only. */
112
137
  keyterms?: string[];
138
+ /** Audio chunks retained BEFORE the VAD fires, prepended to the utterance so the first syllable
139
+ * isn't clipped (VAD confirms speech a beat after it starts). Default TUNING.PREROLL_CHUNKS. */
113
140
  preroll?: number;
141
+ /** Passed through to Silero VAD (thresholds, frame sizes) — see tuning.js for the defaults. */
114
142
  vadOptions?: Record<string, unknown>;
115
143
  /** Semantic end-of-turn hook run when VAD hears end-of-speech: return false to KEEP listening (the
116
144
  * user only paused mid-thought), true (default, when omitted) to close the turn. Lets you swap the
@@ -279,14 +307,18 @@ export function resolveSttProvider(id: string): SttProvider;
279
307
  // ── TTS ──────────────────────────────────────────────────────────────────────────────────────
280
308
  /** Base class for streaming TTS engines: sentence-splits the incoming token stream, synthesizes
281
309
  * the next clip while the current one plays, and reports the spoken cursor via setOnProgress.
282
- * Subclass and implement `_synth(text)` to add an engine. */
310
+ * Subclass and implement `_synth` to add an engine. */
283
311
  export class StreamingTTS implements VoiceTTS {
284
312
  constructor(voiceId?: string, speed?: number);
313
+ /** THE subclass contract: synthesize one sentence → audio Blob (null when aborted/empty).
314
+ * Honour `signal` — a barge-in aborts mid-clip and the base class won't wait for you. */
315
+ protected _synth(text: string, signal?: AbortSignal): Promise<Blob | null>;
285
316
  speak(input: AsyncIterable<string> | string, signal?: AbortSignal, fromNw?: number): Promise<string>;
286
317
  stop(): void;
287
- setOnProgress(fn: (spoken: string, scope: string) => void): void;
288
- /** Jump playback to a word index within the current reply. Returns false when the seek can't be
289
- * honoured (nothing playing, or past the text streamed so far) — check it before moving a cursor. */
318
+ setOnProgress(fn: TtsProgress | null): void;
319
+ /** Jump playback to a position in the current reply (non-whitespace char count, as everywhere
320
+ * else). Returns false when the seek can't be honoured (nothing playing, or past the text
321
+ * streamed so far) — check it before moving a cursor. */
290
322
  seek(nwIndex: number): boolean;
291
323
  }
292
324
  /** Human-grade cloud voices. Pass `ttsUrl` (your backend proxy, key stays server-side); `apiKey`
package/src/stt.js CHANGED
@@ -567,7 +567,9 @@ export function makeDeepgramSTT(opts) {
567
567
  };
568
568
  }
569
569
 
570
- // ── Browser Web Speech API (local, no cloud, no token) ───────────────────────
570
+ // ── Browser Web Speech API (browser-managed, no token) ───────────────────────
571
+ // "Browser-native" is not "on-device": Chrome may route recognition through a vendor speech
572
+ // service. The win here is zero setup (no key, no token endpoint), not guaranteed privacy.
571
573
  // The browser's own SpeechRecognition engine (Chrome/Safari `webkitSpeechRecognition`). It OWNS the
572
574
  // whole audio path — its own mic capture AND its own silence-based end-of-turn — so the agent builds
573
575
  // NO getUserMedia / AudioWorklet / Silero-VAD pipeline for it (`selfCapture: true`, which the agent
@@ -576,7 +578,7 @@ export function makeDeepgramSTT(opts) {
576
578
  // interim results → onPartial(tail) live draft + word-based barge-in while the agent is speaking
577
579
  // final results → accumulate + (re)arm the end-of-turn timer; if the user keeps talking within
578
580
  // the debounce window the fragments merge, else the timer flushes the turn
579
- // No backend token, no /stt/usage reporting (free + on-device), no socket. Only where the API exists
581
+ // No backend token, no /stt/usage reporting (free, zero-setup), no socket. Only where the API exists
580
582
  // (else open() reports a fatal so the host can fall back / surface it).
581
583
  // Browser Web Speech API STT. The agent hands it nothing (selfCapture) — it owns its mic and its own
582
584
  // end-of-turn (nativeEOT). It reports transcripts via onPartial (live draft, drives the user bubble +
package/src/tuning.js CHANGED
@@ -82,6 +82,14 @@ export const TUNING = {
82
82
  // a dead speech service costs a handful of tries over a few seconds instead of a spin loop.
83
83
  WEBSPEECH_ERROR_BACKOFF_MS: 300,
84
84
 
85
+ // Delay before pre-warming the TTS engine after start() (download+compile the voice model, JIT the
86
+ // WASM inference path). Deferred rather than immediate: Piper's ~60MB model download saturates
87
+ // bandwidth and was measured stretching the Deepgram WS handshake from ~0.5s to ~3s when fired
88
+ // concurrently. TTS isn't needed until the first REPLY (a user utterance + LLM roundtrip away), so
89
+ // waiting costs nothing — while NOT warming at all costs the first reply Piper's full cold start
90
+ // (measured 5.4–10.2s to first audio, vs 637ms warmed).
91
+ TTS_WARM_DELAY_MS: 2500,
92
+
85
93
  // Speculative LLM prefetch: once the live interim has been STABLE this long (no new words), the
86
94
  // agent starts generating a reply for the probable final text — overlapping the LLM's first-token
87
95
  // latency with the rest of the end-of-turn debounce. Exact-match turns adopt the running stream
@@ -407,7 +407,17 @@ export class VoiceAgent {
407
407
  // handled by setMuted() → stt.setEnabled(). No warm-resume machinery: open() is idempotent + cheap.
408
408
  // open() can fail synchronously (Web Speech unsupported → onFatal → stop() sets _closed + idle).
409
409
  // Guard the listening flip so a fatal isn't overwritten with a fake 'listening' state.
410
- if (this.stt.selfCapture) { this.stt.open?.(); if (!this._closed) this._set('listening'); return; }
410
+ // TTS still needs warming though: it's the one cold start a selfCapture STT does NOT own. Same
411
+ // deferral as the pipeline path below — the first reply is a user utterance + LLM roundtrip away,
412
+ // so a late start costs nothing, while skipping it entirely made turn 0 pay Piper's full cold
413
+ // start (measured 5.4–10.2s to first audio, vs 637ms on the warmed pipeline path).
414
+ if (this.stt.selfCapture) {
415
+ this.stt.open?.();
416
+ if (!this._closed) this._set('listening');
417
+ const gen = ++this._pipelineGen;
418
+ setTimeout(() => { if (gen === this._pipelineGen && !this._closed) this.tts.warm?.().catch(() => {}); }, TUNING.TTS_WARM_DELAY_MS);
419
+ return;
420
+ }
411
421
  // WARM RESUME: stop() pauses (not tears down) the pipeline, so a resume of the SAME agent finds the
412
422
  // mic stream + audio graph + VAD model still alive — just re-arm them. This skips the getUserMedia
413
423
  // permission round-trip AND the MicVAD.new() Silero/ONNX cold start (the bulk of "open→listening"
@@ -509,11 +519,9 @@ export class VoiceAgent {
509
519
  this._startVad(gen).then(() => { if (gen === this._pipelineGen && !this._closed) this._swapToSileroWhenReady(); })
510
520
  .catch(() => { if (gen === this._pipelineGen) this._sileroPending = false; });
511
521
  // Warm the TTS engine (Piper: download+compile the ONNX voice model, JIT the WASM path) so the
512
- // first reply isn't stalled by cold start mid-turn — but DEFERRED off the STT connect window:
513
- // Piper's ~60MB HuggingFace download saturates bandwidth and was measured stretching the
514
- // Deepgram WS handshake from ~0.5s to ~3s when fired concurrently. TTS isn't needed until the
515
- // first REPLY (user utterance + LLM roundtrip away), so a short delay costs nothing.
516
- setTimeout(() => { if (gen === this._pipelineGen && !this._closed) this.tts.warm?.().catch(() => {}); }, 2500);
522
+ // first reply isn't stalled by cold start mid-turn — deferred off the STT connect window (see
523
+ // TTS_WARM_DELAY_MS).
524
+ setTimeout(() => { if (gen === this._pipelineGen && !this._closed) this.tts.warm?.().catch(() => {}); }, TUNING.TTS_WARM_DELAY_MS);
517
525
  })();
518
526
  }
519
527