@charivo/realtime 0.18.1 → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +83 -26
  2. package/dist/chunk-3YPRYU2M.mjs +1251 -0
  3. package/dist/chunk-3YPRYU2M.mjs.map +1 -0
  4. package/dist/chunk-5OO737VD.mjs +33 -0
  5. package/dist/chunk-5OO737VD.mjs.map +1 -0
  6. package/dist/{chunk-U7ER2VBF.mjs → chunk-EIYYLUWF.mjs} +10 -17
  7. package/dist/chunk-EIYYLUWF.mjs.map +1 -0
  8. package/dist/{chunk-VFIMTFOP.mjs → chunk-JEJGTULX.mjs} +24 -31
  9. package/dist/chunk-JEJGTULX.mjs.map +1 -0
  10. package/dist/{chunk-EXLERHST.mjs → chunk-VTHBTUH3.mjs} +7 -10
  11. package/dist/chunk-VTHBTUH3.mjs.map +1 -0
  12. package/dist/gemini/index.d.mts +22 -0
  13. package/dist/gemini/index.d.ts +22 -0
  14. package/dist/gemini/index.js +1282 -0
  15. package/dist/gemini/index.js.map +1 -0
  16. package/dist/gemini/index.mjs +4 -0
  17. package/dist/gemini/index.mjs.map +1 -0
  18. package/dist/index.d.mts +2 -2
  19. package/dist/index.d.ts +2 -2
  20. package/dist/index.js.map +1 -1
  21. package/dist/index.mjs.map +1 -1
  22. package/dist/openai/index.js +22 -7
  23. package/dist/openai/index.js.map +1 -1
  24. package/dist/openai/index.mjs +3 -2
  25. package/dist/openai-agents/index.js +25 -14
  26. package/dist/openai-agents/index.js.map +1 -1
  27. package/dist/openai-agents/index.mjs +3 -2
  28. package/dist/remote/index.d.mts +1 -0
  29. package/dist/remote/index.d.ts +1 -0
  30. package/dist/remote/index.js +1296 -37
  31. package/dist/remote/index.js.map +1 -1
  32. package/dist/remote/index.mjs +13 -4
  33. package/dist/remote/index.mjs.map +1 -1
  34. package/package.json +7 -2
  35. package/dist/chunk-EXLERHST.mjs.map +0 -1
  36. package/dist/chunk-U7ER2VBF.mjs.map +0 -1
  37. package/dist/chunk-VFIMTFOP.mjs.map +0 -1
package/README.md CHANGED
@@ -50,11 +50,23 @@ await manager.startSession({
50
50
  provider: "openai",
51
51
  model: "gpt-realtime-2.1-mini",
52
52
  });
53
+
54
+ // Or Gemini Live, over the same remote client and route:
55
+ await manager.startSession({ provider: "gemini", transport: "websocket" });
53
56
  ```
54
57
 
55
- `gpt-realtime-2.1-mini` is the default realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://openai.com/api/pricing/) before switching.
58
+ The remote client picks its browser adapter from that `provider`/`transport`
59
+ pair — OpenAI Realtime over WebRTC (the default transport) or Gemini Live over
60
+ WebSocket — and your route dispatches to `@charivo/server/openai` or
61
+ `@charivo/server/gemini` on the same field.
62
+
63
+ `gpt-realtime-2.1-mini` is the default OpenAI realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing) before switching. The Gemini default is `gemini-3.1-flash-live-preview`, the model the transport was measured against.
56
64
 
57
- For iOS-safe lip-sync, call `prepareAudio?.()` from the same user gesture that
65
+ `createRealtimeManager` guarantees `prepareAudio` and `setEventEmitter`, so
66
+ calls on its result need no `?.`. A variable typed as the core
67
+ `RealtimeManager` still does — a third-party manager may omit them.
68
+
69
+ For iOS-safe lip-sync, call `prepareAudio()` from the same user gesture that
58
70
  starts the session, before `startSession(...)`. Pass the same session config to
59
71
  both calls — `prepareAudio` needs it to resolve which remote adapter to warm up:
60
72
 
@@ -64,10 +76,13 @@ const sessionConfig = {
64
76
  model: "gpt-realtime-2.1-mini",
65
77
  };
66
78
 
67
- await manager.prepareAudio?.(sessionConfig);
79
+ await manager.prepareAudio(sessionConfig);
68
80
  await manager.startSession(sessionConfig);
69
81
  ```
70
82
 
83
+ On an OpenAI session, `updateSession(...)` patches the live session in place
84
+ (Gemini Live refuses live patches — see [Session Refresh](#session-refresh)):
85
+
71
86
  ```ts
72
87
  await manager.updateSession({
73
88
  voice: "alloy",
@@ -89,29 +104,39 @@ session active and drives reconnect attempts internally. During that window
89
104
  `state.session.status` stays `"active"` while `state.connection` moves back to
90
105
  `"connecting"`.
91
106
 
92
- ## No-server dev (OpenAI Agents transport)
107
+ ## No-server dev
93
108
 
94
- For local development you can skip the server route: pass an OpenAI API key to
95
- the direct Agents transport client and it mints a short-lived realtime client
96
- secret in the browser, mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
109
+ For local development you can skip the server route: pass a provider API key to
110
+ the direct transport client and it mints the short-lived credential in the
111
+ browser — OpenAI's realtime client secret, or Google's single-use ephemeral
112
+ token — mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
97
113
 
98
114
  ```ts
99
115
  import { createRealtimeManager } from "@charivo/realtime";
100
116
  import { createOpenAIRealtimeAgentsClient } from "@charivo/realtime/openai-agents";
117
+ import { createGeminiLiveClient } from "@charivo/realtime/gemini";
101
118
 
102
119
  // Dev/testing only: the API key is exposed in the browser.
103
- const client = createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." });
104
- const manager = createRealtimeManager(client);
105
-
120
+ const manager = createRealtimeManager(
121
+ createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." }),
122
+ );
106
123
  await manager.startSession({ provider: "openai", model: "gpt-realtime-2.1-mini" });
124
+
125
+ // Or Gemini Live, the same way.
126
+ const geminiManager = createRealtimeManager(
127
+ createGeminiLiveClient({ apiKey: "AIza..." }),
128
+ );
129
+ await geminiManager.startSession({ provider: "gemini", transport: "websocket" });
107
130
  ```
108
131
 
109
- `createOpenAIRealtimeAgentsClient` option precedence is `sessionBootstrap` >
110
- `apiEndpoint` > `apiKey`. The `apiKey` path is dev/testing only and additionally
111
- requires microphone permission, a secure context (`localhost` or `https`), and a
112
- user gesture to start; the minted client secret is short-lived (re-minted per
113
- session). For production, use the server-mediated `@charivo/realtime/remote`
114
- client shown above.
132
+ Both clients share the option precedence `sessionBootstrap` > `apiEndpoint` >
133
+ `apiKey`. The `apiKey` path is dev/testing only and additionally requires
134
+ microphone permission, a secure context (`localhost` or `https`), and a user
135
+ gesture to start; the minted credential is short-lived (re-minted per session).
136
+ The Gemini token still carries the whole session, exactly as the server
137
+ provider mints it, so nothing downstream differs from production. For
138
+ production, use the server-mediated `@charivo/realtime/remote` client shown
139
+ above.
115
140
 
116
141
  ## Exports
117
142
 
@@ -123,6 +148,20 @@ client shown above.
123
148
  `resultProjectors`
124
149
  - `RealtimeLogger`
125
150
 
151
+ Transport clients live on subpaths:
152
+
153
+ - `@charivo/realtime/remote`: `createRemoteRealtimeClient(config?)`, plus
154
+ `DEFAULT_REMOTE_REALTIME_ADAPTERS` — the registry it resolves from, keyed by
155
+ the adapter ids exported from `@charivo/core`
156
+ - `@charivo/realtime/openai-agents`: `createOpenAIRealtimeAgentsClient(options?)`
157
+ — OpenAI Realtime over WebRTC through the OpenAI Agents SDK
158
+ - `@charivo/realtime/openai`: `createOpenAIRealtimeClient(options?)` — the
159
+ legacy low-level OpenAI WebRTC transport
160
+ - `@charivo/realtime/gemini`: `createGeminiLiveClient(options?)` — Gemini Live
161
+ over WebSocket (`apiEndpoint`, `sessionBootstrap`, or a dev/testing
162
+ `apiKey`); captures the microphone at 16 kHz and plays the model's 24 kHz PCM
163
+ through its own scheduler with a lip-sync tap
164
+
126
165
  ## Instruction Layering
127
166
 
128
167
  `@charivo/realtime` keeps its default instructions generic: spoken-output
@@ -156,8 +195,10 @@ await manager.startSession({
156
195
  `buildRealtimeSessionConfig(...)` does not fill provider-specific fields such
157
196
  as `provider` or `model`. Prefer building on top of it for instructions and
158
197
  character voice, then pass provider/model explicitly at `startSession(...)`.
159
- OpenAI transport packages and `@charivo/server/openai` keep their own
160
- OpenAI-specific fallbacks for omitted model or voice values.
198
+ Each transport/provider pair keeps its own fallbacks for omitted model or voice
199
+ values: the OpenAI packages theirs, and `@charivo/server/gemini`
200
+ `gemini-3.1-flash-live-preview` with the `Kore` voice — which is also what a
201
+ voice id it does not recognise (an OpenAI one, say) silently becomes.
161
202
 
162
203
  Avatar-specific realtime tools and avatar-specific instruction addenda now live
163
204
  in `@charivo/avatar` (works with both `@charivo/realtime` and `@charivo/llm`;
@@ -216,7 +257,12 @@ latest requested config, current character, and current tool registry.
216
257
  - inactive managers only cache the requested base config for the next
217
258
  `startSession(...)`
218
259
  - active managers keep the current connection open and issue a transport-level
219
- session patch
260
+ session patch — which `@charivo/realtime/gemini` rejects outright: a Gemini
261
+ Live session is fixed when its token is minted (the token's setup replaces
262
+ the browser's, measured), and the Live API has no `session.update`. The call
263
+ rejects, `realtime:error` carries the same error, and the live session keeps
264
+ the configuration it started with; stop and start a new session to change it,
265
+ and register tools before `startSession(...)`
220
266
  - successful patches update `realtime:state` only and do not emit synthetic
221
267
  `realtime:session:end/start` refresh boundaries
222
268
  - patch failures keep the current live session and previous
@@ -237,6 +283,12 @@ Successful reconnects are treated as a continuation of the same live session.
237
283
  - the next reconnect attempt always rebuilds from the latest effective config
238
284
  - in-flight assistant responses are marked as interrupted and are not resumed
239
285
  - old tool-call ids are not replayed after reconnect
286
+ - on Gemini Live every attempt re-mints, through your route or (on the dev
287
+ `apiKey` path) straight from Google's token endpoint — tokens are `uses: 1`,
288
+ and replaying one closes the socket with `1011` (measured) — and
289
+ the recovered session is a fresh one: `sessionResumptionUpdate` handles are
290
+ not spent, `goAway` is not acted on (the close after it drives this same
291
+ recovery), and the transport's microphone gate re-arms
240
292
 
241
293
  Observability events emitted by the manager:
242
294
 
@@ -279,7 +331,7 @@ manager.registerTool({
279
331
  Live registration:
280
332
 
281
333
  - `registerTool(...)` and `unregisterTool(...)` update the local registry immediately
282
- - an active session is patched automatically, so no explicit `updateSession(...)` is needed
334
+ - an active OpenAI session is patched automatically, so no explicit `updateSession(...)` is needed; a live Gemini Live session refuses the patch (see [Session Refresh](#session-refresh)), so register tools before starting it
283
335
  - a change made while a response is in flight is deferred to just after it, and several deferred changes collapse into one refresh
284
336
  - both return `void`; a failed patch surfaces as `realtime:error`, not a rejection
285
337
  - until that patch lands, an already-active session may still call a just-unregistered tool, which returns a failure result
@@ -309,10 +361,14 @@ When connected, the manager relays:
309
361
  - `tts:audio:end`
310
362
 
311
363
  `tts:audio:end` reports that playback finished, not that the server finished
312
- sending — send completion can precede it by a wide margin. The transports derive
313
- it from
314
- `output_audio_buffer.stopped` (and `output_audio_buffer.cleared` on an
315
- interruption), never from `response.audio.done` / `response.output_audio.done`.
364
+ sending — send completion can precede it by a wide margin. The OpenAI
365
+ transports derive it from `output_audio_buffer.stopped` (and
366
+ `output_audio_buffer.cleared` on an interruption), never from
367
+ `response.audio.done` / `response.output_audio.done`. The Gemini transport
368
+ plays the audio itself and ends it only when its scheduler has drained AND the
369
+ server's `turnComplete` has arrived — neither alone, because the scheduler
370
+ drains spuriously at the start of a turn and `turnComplete` is paced by a
371
+ server clock rather than observed from playback (both measured).
316
372
  Consumers act on this event — `RenderManager` releases a held expression and
317
373
  stops lip-sync there — so a custom transport that reports it at send completion
318
374
  will reset the avatar's face mid-reply. See
@@ -321,7 +377,8 @@ will reset the avatar's face mid-reply. See
321
377
  `RealtimeState.audioPlaying` exposes the same playback segment as state, which is
322
378
  what to read when you need to know whether the character is still talking.
323
379
  `response.status` answers a different question — it completes when the provider
324
- stops sending, while playback runs on — so anything gated on it is wrong for the
325
- tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
380
+ closes the response, at send completion on OpenAI and at the clock-paced
381
+ `turnComplete` on Gemini, while playback runs on — so anything gated on it is
382
+ wrong for the tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
326
383
  still expected, which is the response-in-progress condition `sendMessage`
327
384
  refuses on.