@kuralle-syrinx/realtime 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Kuralle
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
22
+
package/README.md ADDED
@@ -0,0 +1,152 @@
1
+ # @kuralle-syrinx/realtime
2
+
3
+ The **bi-model** front seam for Syrinx (backlog **B-01**): a `RealtimeBridge` `VoicePlugin` that drives a
4
+ full-duplex **s2s "front" model** (OpenAI `gpt-realtime-2`) as the live conversational surface, and
5
+ delegates the "meat" (RAG / reasoning) to **any existing `Reasoner`** (the seam from
6
+ `@kuralle-syrinx/aisdk` / `mastra`) as a tool. The guardian front owns presence + speech understanding +
7
+ turn-taking; the async back owns the facts. See `docs/rfc-realtime-bridge.md` and
8
+ `bi-model-research/` for the design and the prior art (Fin Voice 2 / MoshiRAG / TML interaction models).
9
+
10
+ ## What it is
11
+
12
+ - **`fromOpenAIRealtime(opts)` → `RealtimeAdapter`** — owns the `gpt-realtime-2` WebSocket (over
13
+ `@kuralle-syrinx/ws`'s reconnecting `WebSocketConnection`). Normalizes provider events into a small
14
+ `RealtimeEvent` union (`audio` / `speech_started` / `transcript` / `tool_call` / `response_started` /
15
+ `response_done` / `error`). Audio is `audio/pcm` @ **24 kHz**.
16
+ - **`RealtimeBridge`** — a `VoicePlugin`. Consumes `user.audio_received` (resampled 16k→24k → provider),
17
+ emits `tts.audio` (provider 24k→16k, chunked ≤20 ms) + `tts.end`, mints a **fresh `contextId` per turn**
18
+ (so barge-in never permanently mutes the agent), surfaces `llm.error`, and — when given a `Reasoner` —
19
+ runs the delegate loop on the front model's `ask_university`-style tool call and feeds the answer back
20
+ via `function_call_output` for the front model to voice.
21
+
22
+ ## Two modes (and how the back model plugs in)
23
+
24
+ The `RealtimeBridge` runs in two modes; the back "meat" model plugs in via the **`Reasoner` seam** — the
25
+ *same* framework adapters the cascade `ReasoningBridge` uses (`@kuralle-syrinx/aisdk`'s
26
+ `fromStreamText`/`fromAiSdkAgent`/`fromStreamFactory`, `@kuralle-syrinx/mastra`'s `fromMastraAgent`). You
27
+ pass the **`Reasoner`**, not the `ReasoningBridge` plugin (the bridge runs it as a delegate tool and feeds
28
+ the result back for the front model to voice — using the `ReasoningBridge` plugin here would double-speak).
29
+
30
+ ```ts
31
+ import { RealtimeBridge, fromOpenAIRealtime } from "@kuralle-syrinx/realtime";
32
+ import { createNodeWsSocket } from "@kuralle-syrinx/ws/node";
33
+
34
+ const adapter = fromOpenAIRealtime({
35
+ apiKey: process.env.OPENAI_API_KEY!,
36
+ socketFactory: createNodeWsSocket,
37
+ // turnDetection defaults to semantic_vad; server_vad is more deterministic for tests/telephony.
38
+ // tools: [...] // domain tools the front model may call — supplied by YOU, never hardcoded here.
39
+ });
40
+
41
+ // (1) STANDALONE — pure s2s, the realtime model answers from its own knowledge:
42
+ session.registerPlugin("realtime", new RealtimeBridge(adapter));
43
+
44
+ // (2) BI-MODEL with an AI SDK back (the "meat"):
45
+ import { fromStreamText } from "@kuralle-syrinx/aisdk";
46
+ const aiReasoner = fromStreamText({ model, system, tools: { resolveLateAddRequest } });
47
+ const adapterA = fromOpenAIRealtime({ ...opts, tools: [{ name: "ask_kb", description: "...", parameters: {/*JSON Schema*/} }] });
48
+ session.registerPlugin("realtime", new RealtimeBridge(adapterA, aiReasoner, "ask_kb"));
49
+
50
+ // (3) BI-MODEL with a Mastra back — identical wiring, just a different Reasoner factory:
51
+ import { fromMastraAgent } from "@kuralle-syrinx/mastra";
52
+ const mastraReasoner = fromMastraAgent(myMastraAgent);
53
+ session.registerPlugin("realtime", new RealtimeBridge(adapterA, mastraReasoner, "ask_kb"));
54
+ ```
55
+
56
+ Run the session with `endpointingOwner:"timer"` — the s2s model owns turn detection, so NO STT/VAD/TTS
57
+ plugins are registered on the live path. The **delegate tool is caller-supplied**: pass the tool def to
58
+ the adapter (`tools`) and its name as the bridge's 3rd arg (`delegateToolName`); the adapter is fully
59
+ domain-neutral (it never hardcodes any tool). The same `Reasoner` backends also power the cascade
60
+ `ReasoningBridge` — only the front (s2s vs STT→TTS) differs.
61
+
62
+ ## Deploy on Cloudflare Workers
63
+
64
+ `@kuralle-syrinx/realtime` is **edge-clean**: no `Buffer`, `process`, or `node:crypto` in `src/`. The
65
+ adapter is **provider-socket-agnostic** — inject whichever `@kuralle-syrinx/ws` factory your runtime needs.
66
+ On Workers, outbound provider WebSockets that require auth headers use the fetch-upgrade path via
67
+ `createWorkersSocket` (not the global `WebSocket` constructor, which cannot set headers).
68
+
69
+ Wire secrets through the Worker **`env` binding** (Wrangler secrets / vars), not `process.env`. Pass
70
+ `apiKey` and `debug` as constructor options:
71
+
72
+ ```ts
73
+ import { VoiceAgentSession } from "@kuralle-syrinx/core";
74
+ import { RealtimeBridge, fromOpenAIRealtime } from "@kuralle-syrinx/realtime";
75
+ import { createWorkersSocket } from "@kuralle-syrinx/ws/workers";
76
+
77
+ /** Bound in wrangler.jsonc / dashboard — e.g. OPENAI_API_KEY secret. */
78
+ export interface Env {
79
+ readonly OPENAI_API_KEY: string;
80
+ }
81
+
82
+ export function createRealtimeVoiceSession(env: Env): VoiceAgentSession {
83
+ const adapter = fromOpenAIRealtime({
84
+ apiKey: env.OPENAI_API_KEY,
85
+ socketFactory: createWorkersSocket,
86
+ debug: false,
87
+ turnDetection: { type: "semantic_vad" },
88
+ });
89
+
90
+ const session = new VoiceAgentSession({
91
+ endpointingOwner: "timer",
92
+ plugins: { realtime: {} },
93
+ });
94
+ session.registerPlugin("realtime", new RealtimeBridge(adapter));
95
+ return session;
96
+ }
97
+ ```
98
+
99
+ **Durable Object session shape** (see `packages/server-workers`): the Worker `fetch` handler routes
100
+ `/ws?sessionId=…` to a `VoiceConversation` Durable Object. The DO accepts the client upgrade via
101
+ `WebSocketPair`, constructs the `VoiceAgentSession` (cascade or bi-model realtime — same env-injection
102
+ pattern), and pumps audio over the accepted socket. Provider outbound legs (OpenAI Realtime, Deepgram,
103
+ Cartesia, …) all dial through `createWorkersSocket` so auth headers ride on the fetch upgrade.
104
+
105
+ Regression lock: `edge-safety.test.ts` runs the adapter + bridge with `Buffer` and `process` removed from
106
+ `globalThis`.
107
+
108
+ ## Capability model
109
+
110
+ `RealtimeAdapter.caps` lets the bridge adapt per provider:
111
+
112
+ | cap | gpt-realtime-2 | meaning |
113
+ |---|---|---|
114
+ | `inputSampleRateHz` / `outputSampleRateHz` | 24000 / 24000 | resample boundaries (engine is 16k) |
115
+ | `supportsConcurrentToolAudio` | `true` | native **async function calling** — the model keeps the turn fluid while the delegate runs; no double-audio observed |
116
+ | `supportsTruncate` | `true` | barge-in sends `conversation.item.truncate(audio_end_ms)` (not just `response.cancel`) |
117
+
118
+ A `fromGeminiLive` / `fromMoshi` adapter can follow with different caps (Gemini Live tool calls are
119
+ blocking → `supportsConcurrentToolAudio:false`; Moshi-class owned models could use embedding-sum injection).
120
+
121
+ ## Latency (measured, honest)
122
+
123
+ From the live `gpt-realtime-2` smokes on this branch (one turn, university fixture; `server_vad`):
124
+
125
+ - **Frame round-trip** (`smoke:realtime-frame`): provider audio → resample 24k→16k → Syrinx envelope
126
+ codec, `ok` — proves the rate-handling path adds no decode break.
127
+ - **One-turn audio** (`smoke:realtime-oneturn`): ~3.6 s assistant audio delivered through the standard
128
+ paced `tts.audio` path.
129
+ - **Bi-model turn** (`smoke:realtime-university`): front lead-in onset ≈ first audio; `ask_university`
130
+ tool call at ≈13.9 s into the run; university `Reasoner` answer back ≈4.2 s later; front voiced the
131
+ grounded body — **the reasoner latency was hidden under the lead-in** (the keyword-delay-gap thesis).
132
+
133
+ **Honest characterization (not "~0"):** the bridged topology is `client ↔ Syrinx ↔ gpt-realtime-2` — one
134
+ extra WS hop + input/output resampling + per-frame bus dispatch on top of talking to the provider
135
+ directly. **Not yet measured:** a rigorous *first-audio delta* of direct-gpt-realtime-2 vs
136
+ via-`RealtimeBridge` (the WBS-5 comparison harness). Treat the delta as an open measurement; co-locate
137
+ Syrinx with the provider region to minimize the added leg.
138
+
139
+ ## Status (B-01 build)
140
+
141
+ | Capability | State |
142
+ |---|---|
143
+ | `fromOpenAIRealtime` adapter + ws realtime socket | ✅ live-verified |
144
+ | `RealtimeBridge` live audio loop (fresh contextId/turn) | ✅ live-verified |
145
+ | Delegate → `Reasoner` (bi-model), `function_call_output` injection | ✅ live-verified (university turn) |
146
+ | Barge-in: `speech_started`→interrupt, `cancel`+`truncate`, abort delegate, cancel-when-idle guard | ✅ logic unit-verified + detection live-confirmed; live "resume-after-barge" smoke is flaky (orchestration) |
147
+ | First-audio direct-vs-bridged latency delta harness | ⏳ open (WBS-5) |
148
+ | `fromGeminiLive` / `fromMoshi` adapters | ⏳ future |
149
+
150
+ Tests: `pnpm --filter @kuralle-syrinx/realtime test`. Live gates (need `OPENAI_API_KEY`):
151
+ `smoke:realtime-frame` / `:realtime-oneturn` / `:realtime-university` / `:realtime-bargein` in
152
+ `examples/02-hello-voice-headless`.
package/package.json ADDED
@@ -0,0 +1,22 @@
1
+ {
2
+ "name": "@kuralle-syrinx/realtime",
3
+ "version": "2.1.0",
4
+ "private": false,
5
+ "type": "module",
6
+ "license": "MIT",
7
+ "main": "./src/index.ts",
8
+ "types": "./src/index.ts",
9
+ "dependencies": {
10
+ "@google/genai": "^2.8.0",
11
+ "@kuralle-syrinx/core": "2.1.0",
12
+ "@kuralle-syrinx/ws": "2.1.0"
13
+ },
14
+ "devDependencies": {
15
+ "typescript": "^5.7.0",
16
+ "vitest": "^2.1.0"
17
+ },
18
+ "scripts": {
19
+ "typecheck": "tsc --noEmit",
20
+ "test": "vitest run"
21
+ }
22
+ }
package/src/base64.ts ADDED
@@ -0,0 +1,16 @@
1
+ // SPDX-License-Identifier: MIT
2
+
3
+ export function bytesToBase64(bytes: Uint8Array): string {
4
+ let binary = "";
5
+ for (let i = 0; i < bytes.length; i += 0x8000) {
6
+ binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));
7
+ }
8
+ return globalThis.btoa(binary);
9
+ }
10
+
11
+ export function base64ToBytes(base64: string): Uint8Array {
12
+ const raw = globalThis.atob(base64);
13
+ const bytes = new Uint8Array(raw.length);
14
+ for (let i = 0; i < raw.length; i += 1) bytes[i] = raw.charCodeAt(i);
15
+ return bytes;
16
+ }
@@ -0,0 +1,173 @@
1
+ // SPDX-License-Identifier: MIT
2
+ //
3
+ // R-14: regression gate — the realtime package must run on workerd without
4
+ // Buffer, process, or node:crypto. Reintroducing any Node-only primitive in
5
+ // src/ should fail this test.
6
+
7
+ import { readFileSync, readdirSync } from "node:fs";
8
+ import path from "node:path";
9
+ import { fileURLToPath } from "node:url";
10
+
11
+ import { describe, expect, it } from "vitest";
12
+
13
+ import {
14
+ PipelineBusImpl,
15
+ Route,
16
+ type TextToSpeechAudioPacket,
17
+ type TurnChangePacket,
18
+ } from "@kuralle-syrinx/core";
19
+ import type { ManagedSocket, SocketData, SocketFactory } from "@kuralle-syrinx/ws";
20
+
21
+ import { bytesToBase64, fromOpenAIRealtime } from "./from-openai-realtime.js";
22
+ import { RealtimeBridge } from "./realtime-bridge.js";
23
+
24
+ const srcDir = path.dirname(fileURLToPath(import.meta.url));
25
+
26
+ interface EdgeMockHarness {
27
+ readonly factory: SocketFactory;
28
+ readonly sent: string[];
29
+ inject(msg: Record<string, unknown>): void;
30
+ }
31
+
32
+ function createEdgeMockHarness(): EdgeMockHarness {
33
+ const sent: string[] = [];
34
+ let messageHandler: ((data: SocketData, isBinary: boolean) => void) | null = null;
35
+
36
+ const socket: ManagedSocket = {
37
+ get isOpen() {
38
+ return true;
39
+ },
40
+ send: (data: SocketData) => {
41
+ sent.push(typeof data === "string" ? data : "");
42
+ },
43
+ keepAlivePing: () => {},
44
+ verify: async () => true,
45
+ dispose: () => {},
46
+ onOpen: (handler) => {
47
+ queueMicrotask(() => handler());
48
+ },
49
+ onMessage: (handler) => {
50
+ messageHandler = handler;
51
+ },
52
+ onClose: () => {},
53
+ onError: () => {},
54
+ };
55
+
56
+ return {
57
+ factory: () => socket,
58
+ sent,
59
+ inject: (msg) => messageHandler?.(JSON.stringify(msg), false),
60
+ };
61
+ }
62
+
63
+ function pcmFromSamples(samples: readonly number[]): Uint8Array {
64
+ const pcm = Int16Array.from(samples);
65
+ return new Uint8Array(pcm.buffer, pcm.byteOffset, pcm.byteLength);
66
+ }
67
+
68
+ async function waitFor(predicate: () => boolean, timeoutMs = 2000): Promise<void> {
69
+ const startedAt = Date.now();
70
+ while (Date.now() - startedAt < timeoutMs) {
71
+ if (predicate()) return;
72
+ await new Promise((resolve) => setTimeout(resolve, 5));
73
+ }
74
+ throw new Error("Timed out waiting for condition");
75
+ }
76
+
77
+ function collectSourceFiles(dir: string): string[] {
78
+ const out: string[] = [];
79
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
80
+ const full = path.join(dir, entry.name);
81
+ if (entry.isDirectory()) {
82
+ out.push(...collectSourceFiles(full));
83
+ continue;
84
+ }
85
+ if (entry.name.endsWith(".ts") && !entry.name.endsWith(".test.ts")) {
86
+ out.push(full);
87
+ }
88
+ }
89
+ return out;
90
+ }
91
+
92
+ describe("edge safety (R-14)", () => {
93
+ it("src/ contains no Buffer, process, or node:crypto imports", () => {
94
+ const forbidden = [
95
+ /\bfrom\s+["']node:crypto["']/,
96
+ /\brequire\s*\(\s*["']node:crypto["']\s*\)/,
97
+ /\bglobalThis\.Buffer\b/,
98
+ /\bglobalThis\.process\b/,
99
+ /\bprocess\.env\b/,
100
+ /\bBuffer\.from\b/,
101
+ /\bBuffer\.alloc\b/,
102
+ ];
103
+ const hits: string[] = [];
104
+ for (const file of collectSourceFiles(srcDir)) {
105
+ const text = readFileSync(file, "utf8");
106
+ for (const pattern of forbidden) {
107
+ if (pattern.test(text)) {
108
+ hits.push(`${path.relative(srcDir, file)}: ${pattern.source}`);
109
+ }
110
+ }
111
+ }
112
+ expect(hits).toEqual([]);
113
+ });
114
+
115
+ // NOTE: we do NOT delete globalThis.Buffer/process here — vitest's own worker needs `process`
116
+ // (its uncaughtException handler calls process.listeners), so deleting it crashes the runner.
117
+ // Edge-safety is enforced statically by the source-scan above; this test proves the audio path
118
+ // actually runs end-to-end using only the runtime-agnostic helpers.
119
+ it("fromOpenAIRealtime + RealtimeBridge round-trip audio via runtime-agnostic helpers", async () => {
120
+ const mock = createEdgeMockHarness();
121
+ const adapter = fromOpenAIRealtime({
122
+ apiKey: "edge-test-key",
123
+ socketFactory: mock.factory,
124
+ url: () => "wss://example.test/realtime?model=gpt-realtime-2",
125
+ });
126
+ const bridge = new RealtimeBridge(adapter);
127
+ const bus = new PipelineBusImpl();
128
+ const turnChanges: TurnChangePacket[] = [];
129
+ const ttsAudio: TextToSpeechAudioPacket[] = [];
130
+ bus.on("turn.change", (pkt) => { turnChanges.push(pkt as TurnChangePacket); });
131
+ bus.on("tts.audio", (pkt) => { ttsAudio.push(pkt as TextToSpeechAudioPacket); });
132
+
133
+ const started = bus.start();
134
+
135
+ const initTask = bridge.initialize(bus, {});
136
+ await waitFor(() => mock.sent.length > 0);
137
+ mock.inject({ type: "session.updated" });
138
+ await initTask;
139
+
140
+ bus.push(Route.Main, {
141
+ kind: "user.audio_received",
142
+ contextId: "transport-turn",
143
+ timestampMs: Date.now(),
144
+ audio: pcmFromSamples([100, 200, 300, 400]),
145
+ });
146
+
147
+ await waitFor(() =>
148
+ mock.sent.some((raw) => (JSON.parse(raw) as { type: string }).type === "input_audio_buffer.append"),
149
+ );
150
+
151
+ const providerPcm = pcmFromSamples(Array.from({ length: 960 }, (_, i) => i));
152
+ mock.inject({ type: "response.created" });
153
+ mock.inject({
154
+ type: "response.output_audio.delta",
155
+ delta: bytesToBase64(providerPcm),
156
+ });
157
+ mock.inject({ type: "response.done" });
158
+
159
+ await waitFor(() => ttsAudio.length > 0 && turnChanges.length > 0);
160
+
161
+ const contextId = turnChanges[0]!.contextId;
162
+ expect(contextId).toMatch(
163
+ /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
164
+ );
165
+ expect(ttsAudio[0]!.contextId).toBe(contextId);
166
+ expect(ttsAudio[0]!.sampleRateHz).toBe(16_000);
167
+ expect(ttsAudio[0]!.audio.byteLength).toBeGreaterThan(0);
168
+
169
+ await bridge.close();
170
+ bus.stop();
171
+ await started;
172
+ });
173
+ });
@@ -0,0 +1,197 @@
1
+ // SPDX-License-Identifier: MIT
2
+
3
+ import { afterEach, describe, expect, it, vi } from "vitest";
4
+
5
+ import type { LiveServerMessage } from "@google/genai";
6
+
7
+ import type { RealtimeEvent } from "./realtime-adapter.js";
8
+ import { bytesToBase64 } from "./base64.js";
9
+ import { fromGeminiLive } from "./from-gemini-live.js";
10
+
11
+ const sendRealtimeInput = vi.fn();
12
+ const sendToolResponse = vi.fn();
13
+ const closeSession = vi.fn();
14
+
15
+ let onopen: (() => void) | null = null;
16
+ let onmessage: ((msg: LiveServerMessage) => void) | null = null;
17
+
18
+ const liveConnect = vi.fn().mockImplementation(async ({ callbacks }: {
19
+ callbacks: {
20
+ onopen?: () => void;
21
+ onmessage?: (msg: LiveServerMessage) => void;
22
+ };
23
+ }) => {
24
+ onopen = callbacks.onopen ?? null;
25
+ onmessage = callbacks.onmessage ?? null;
26
+ queueMicrotask(() => callbacks.onopen?.());
27
+ return {
28
+ sendRealtimeInput,
29
+ sendToolResponse,
30
+ close: closeSession,
31
+ };
32
+ });
33
+
34
+ vi.mock("@google/genai", () => ({
35
+ GoogleGenAI: vi.fn().mockImplementation(() => ({
36
+ live: { connect: liveConnect },
37
+ })),
38
+ Modality: { AUDIO: "AUDIO" },
39
+ }));
40
+
41
+ afterEach(() => {
42
+ sendRealtimeInput.mockClear();
43
+ sendToolResponse.mockClear();
44
+ closeSession.mockClear();
45
+ liveConnect.mockClear();
46
+ onopen = null;
47
+ onmessage = null;
48
+ });
49
+
50
+ async function collectEvents(
51
+ events: AsyncIterable<RealtimeEvent>,
52
+ max = 12,
53
+ ): Promise<RealtimeEvent[]> {
54
+ const out: RealtimeEvent[] = [];
55
+ for await (const event of events) {
56
+ out.push(event);
57
+ if (out.length >= max) break;
58
+ }
59
+ return out;
60
+ }
61
+
62
+ function inject(msg: Partial<LiveServerMessage> & Record<string, unknown>): void {
63
+ if (!onmessage) throw new Error("mock session onmessage not wired");
64
+ onmessage(msg as LiveServerMessage);
65
+ }
66
+
67
+ describe("fromGeminiLive", () => {
68
+ it("emits client calls for open, audio, and tool result", async () => {
69
+ const adapter = fromGeminiLive({
70
+ apiKey: "test-key",
71
+ tools: [{
72
+ name: "consult_knowledge",
73
+ description: "Answer knowledge questions.",
74
+ parameters: { type: "object", properties: { query: { type: "string" } }, required: ["query"] },
75
+ }],
76
+ });
77
+
78
+ await adapter.open(new AbortController().signal);
79
+
80
+ expect(liveConnect).toHaveBeenCalledTimes(1);
81
+ const connectArg = liveConnect.mock.calls[0]![0] as {
82
+ model: string;
83
+ config: Record<string, unknown>;
84
+ };
85
+ expect(connectArg.model).toBe("gemini-3.1-flash-live-preview");
86
+ expect(connectArg.config["tools"]).toEqual([{
87
+ functionDeclarations: [{
88
+ name: "consult_knowledge",
89
+ description: "Answer knowledge questions.",
90
+ parametersJsonSchema: {
91
+ type: "object",
92
+ properties: { query: { type: "string" } },
93
+ required: ["query"],
94
+ },
95
+ }],
96
+ }]);
97
+
98
+ const pcm = new Uint8Array([0, 1, 2, 3]);
99
+ adapter.sendAudio(pcm);
100
+ expect(sendRealtimeInput).toHaveBeenCalledWith({
101
+ audio: {
102
+ data: bytesToBase64(pcm),
103
+ mimeType: "audio/pcm;rate=16000",
104
+ },
105
+ });
106
+
107
+ inject({
108
+ toolCall: {
109
+ functionCalls: [{
110
+ id: "call_abc",
111
+ name: "consult_knowledge",
112
+ args: { query: "late add" },
113
+ }],
114
+ },
115
+ });
116
+ adapter.injectToolResult("call_abc", "Late Add Petition required.");
117
+ expect(sendToolResponse).toHaveBeenCalledWith({
118
+ functionResponses: [{
119
+ id: "call_abc",
120
+ name: "consult_knowledge",
121
+ response: { result: "Late Add Petition required." },
122
+ }],
123
+ });
124
+
125
+ adapter.cancelResponse(420);
126
+ expect(sendRealtimeInput).toHaveBeenCalledTimes(1);
127
+ });
128
+
129
+ it("normalizes provider server messages into RealtimeEvent", async () => {
130
+ const adapter = fromGeminiLive({ apiKey: "test-key" });
131
+ const eventsTask = collectEvents(adapter.events, 7);
132
+ await adapter.open(new AbortController().signal);
133
+
134
+ const audioBytes = new Uint8Array([9, 10, 11, 12]);
135
+ inject({ setupComplete: {} });
136
+ inject({
137
+ serverContent: {
138
+ modelTurn: {
139
+ parts: [{
140
+ inlineData: {
141
+ data: bytesToBase64(audioBytes),
142
+ mimeType: "audio/pcm;rate=24000",
143
+ },
144
+ }],
145
+ },
146
+ },
147
+ });
148
+ inject({ serverContent: { interrupted: true } });
149
+ inject({
150
+ serverContent: {
151
+ inputTranscription: { text: "Can I add Biology?", finished: true },
152
+ },
153
+ });
154
+ inject({
155
+ serverContent: {
156
+ outputTranscription: { text: "Let me check that.", finished: true },
157
+ },
158
+ });
159
+ inject({
160
+ toolCall: {
161
+ functionCalls: [{
162
+ id: "call_123",
163
+ name: "consult_knowledge",
164
+ args: { query: "late add biology" },
165
+ }],
166
+ },
167
+ });
168
+ inject({ serverContent: { turnComplete: true } });
169
+
170
+ const events = await eventsTask;
171
+ expect(events.slice(0, 7)).toEqual([
172
+ { type: "response_started" },
173
+ { type: "audio", pcm16: audioBytes, sampleRateHz: 24000 },
174
+ { type: "speech_started" },
175
+ { type: "transcript", role: "user", text: "Can I add Biology?", final: true },
176
+ { type: "transcript", role: "assistant", text: "Let me check that.", final: true },
177
+ {
178
+ type: "tool_call",
179
+ toolId: "call_123",
180
+ toolName: "consult_knowledge",
181
+ args: { query: "late add biology" },
182
+ },
183
+ { type: "response_done" },
184
+ ]);
185
+ });
186
+
187
+ it("exposes Gemini Live capability flags", () => {
188
+ const adapter = fromGeminiLive({ apiKey: "test-key" });
189
+ expect(adapter.caps).toEqual({
190
+ inputSampleRateHz: 16_000,
191
+ outputSampleRateHz: 24_000,
192
+ supportsConcurrentToolAudio: false,
193
+ supportsTruncate: false,
194
+ emitsServerSpeechStarted: true,
195
+ });
196
+ });
197
+ });