@charivo/realtime 0.19.0 → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -50,9 +50,17 @@ await manager.startSession({
50
50
  provider: "openai",
51
51
  model: "gpt-realtime-2.1-mini",
52
52
  });
53
+
54
+ // Or Gemini Live, over the same remote client and route:
55
+ await manager.startSession({ provider: "gemini", transport: "websocket" });
53
56
  ```
54
57
 
55
- `gpt-realtime-2.1-mini` is the default realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://openai.com/api/pricing/) before switching.
58
+ The remote client picks its browser adapter from that `provider`/`transport`
59
+ pair — OpenAI Realtime over WebRTC (the default transport) or Gemini Live over
60
+ WebSocket — and your route dispatches to `@charivo/server/openai` or
61
+ `@charivo/server/gemini` on the same field.
62
+
63
+ `gpt-realtime-2.1-mini` is the default OpenAI realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing) before switching. The Gemini default is `gemini-3.1-flash-live-preview`, the model the transport was measured against.
56
64
 
57
65
  `createRealtimeManager` guarantees `prepareAudio` and `setEventEmitter`, so
58
66
  calls on its result need no `?.`. A variable typed as the core
@@ -72,6 +80,9 @@ await manager.prepareAudio(sessionConfig);
72
80
  await manager.startSession(sessionConfig);
73
81
  ```
74
82
 
83
+ On an OpenAI session, `updateSession(...)` patches the live session in place
84
+ (Gemini Live refuses live patches — see [Session Refresh](#session-refresh)):
85
+
75
86
  ```ts
76
87
  await manager.updateSession({
77
88
  voice: "alloy",
@@ -93,29 +104,39 @@ session active and drives reconnect attempts internally. During that window
93
104
  `state.session.status` stays `"active"` while `state.connection` moves back to
94
105
  `"connecting"`.
95
106
 
96
- ## No-server dev (OpenAI Agents transport)
107
+ ## No-server dev
97
108
 
98
- For local development you can skip the server route: pass an OpenAI API key to
99
- the direct Agents transport client and it mints a short-lived realtime client
100
- secret in the browser, mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
109
+ For local development you can skip the server route: pass a provider API key to
110
+ the direct transport client and it mints the short-lived credential in the
111
+ browser — OpenAI's realtime client secret, or Google's single-use ephemeral
112
+ token — mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
101
113
 
102
114
  ```ts
103
115
  import { createRealtimeManager } from "@charivo/realtime";
104
116
  import { createOpenAIRealtimeAgentsClient } from "@charivo/realtime/openai-agents";
117
+ import { createGeminiLiveClient } from "@charivo/realtime/gemini";
105
118
 
106
119
  // Dev/testing only: the API key is exposed in the browser.
107
- const client = createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." });
108
- const manager = createRealtimeManager(client);
109
-
120
+ const manager = createRealtimeManager(
121
+ createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." }),
122
+ );
110
123
  await manager.startSession({ provider: "openai", model: "gpt-realtime-2.1-mini" });
124
+
125
+ // Or Gemini Live, the same way.
126
+ const geminiManager = createRealtimeManager(
127
+ createGeminiLiveClient({ apiKey: "AIza..." }),
128
+ );
129
+ await geminiManager.startSession({ provider: "gemini", transport: "websocket" });
111
130
  ```
112
131
 
113
- `createOpenAIRealtimeAgentsClient` option precedence is `sessionBootstrap` >
114
- `apiEndpoint` > `apiKey`. The `apiKey` path is dev/testing only and additionally
115
- requires microphone permission, a secure context (`localhost` or `https`), and a
116
- user gesture to start; the minted client secret is short-lived (re-minted per
117
- session). For production, use the server-mediated `@charivo/realtime/remote`
118
- client shown above.
132
+ Both clients share the option precedence `sessionBootstrap` > `apiEndpoint` >
133
+ `apiKey`. The `apiKey` path is dev/testing only and additionally requires
134
+ microphone permission, a secure context (`localhost` or `https`), and a user
135
+ gesture to start; the minted credential is short-lived (re-minted per session).
136
+ The Gemini token still carries the whole session, exactly as the server
137
+ provider mints it, so nothing downstream differs from production. For
138
+ production, use the server-mediated `@charivo/realtime/remote` client shown
139
+ above.
119
140
 
120
141
  ## Exports
121
142
 
@@ -127,6 +148,20 @@ client shown above.
127
148
  `resultProjectors`
128
149
  - `RealtimeLogger`
129
150
 
151
+ Transport clients live on subpaths:
152
+
153
+ - `@charivo/realtime/remote`: `createRemoteRealtimeClient(config?)`, plus
154
+ `DEFAULT_REMOTE_REALTIME_ADAPTERS` — the registry it resolves from, keyed by
155
+ the adapter ids exported from `@charivo/core`
156
+ - `@charivo/realtime/openai-agents`: `createOpenAIRealtimeAgentsClient(options?)`
157
+ — OpenAI Realtime over WebRTC through the OpenAI Agents SDK
158
+ - `@charivo/realtime/openai`: `createOpenAIRealtimeClient(options?)` — the
159
+ legacy low-level OpenAI WebRTC transport
160
+ - `@charivo/realtime/gemini`: `createGeminiLiveClient(options?)` — Gemini Live
161
+ over WebSocket (`apiEndpoint`, `sessionBootstrap`, or a dev/testing
162
+ `apiKey`); captures the microphone at 16 kHz and plays the model's 24 kHz PCM
163
+ through its own scheduler with a lip-sync tap
164
+
130
165
  ## Instruction Layering
131
166
 
132
167
  `@charivo/realtime` keeps its default instructions generic: spoken-output
@@ -160,8 +195,10 @@ await manager.startSession({
160
195
  `buildRealtimeSessionConfig(...)` does not fill provider-specific fields such
161
196
  as `provider` or `model`. Prefer building on top of it for instructions and
162
197
  character voice, then pass provider/model explicitly at `startSession(...)`.
163
- OpenAI transport packages and `@charivo/server/openai` keep their own
164
- OpenAI-specific fallbacks for omitted model or voice values.
198
+ Each transport/provider pair keeps its own fallbacks for omitted model or voice
199
+ values: the OpenAI packages theirs, and `@charivo/server/gemini`
200
+ `gemini-3.1-flash-live-preview` with the `Kore` voice — which is also what a
201
+ voice id it does not recognise (an OpenAI one, say) silently becomes.
165
202
 
166
203
  Avatar-specific realtime tools and avatar-specific instruction addenda now live
167
204
  in `@charivo/avatar` (works with both `@charivo/realtime` and `@charivo/llm`;
@@ -220,7 +257,12 @@ latest requested config, current character, and current tool registry.
220
257
  - inactive managers only cache the requested base config for the next
221
258
  `startSession(...)`
222
259
  - active managers keep the current connection open and issue a transport-level
223
- session patch
260
+ session patch — which `@charivo/realtime/gemini` rejects outright: a Gemini
261
+ Live session is fixed when its token is minted (the token's setup replaces
262
+ the browser's, measured), and the Live API has no `session.update`. The call
263
+ rejects, `realtime:error` carries the same error, and the live session keeps
264
+ the configuration it started with; stop and start a new session to change it,
265
+ and register tools before `startSession(...)`
224
266
  - successful patches update `realtime:state` only and do not emit synthetic
225
267
  `realtime:session:end/start` refresh boundaries
226
268
  - patch failures keep the current live session and previous
@@ -241,6 +283,12 @@ Successful reconnects are treated as a continuation of the same live session.
241
283
  - the next reconnect attempt always rebuilds from the latest effective config
242
284
  - in-flight assistant responses are marked as interrupted and are not resumed
243
285
  - old tool-call ids are not replayed after reconnect
286
+ - on Gemini Live every attempt re-mints, through your route or (on the dev
287
+ `apiKey` path) straight from Google's token endpoint — tokens are `uses: 1`,
288
+ and replaying one closes the socket with `1011` (measured) — and
289
+ the recovered session is a fresh one: `sessionResumptionUpdate` handles are
290
+ not spent, `goAway` is not acted on (the close after it drives this same
291
+ recovery), and the transport's microphone gate re-arms
244
292
 
245
293
  Observability events emitted by the manager:
246
294
 
@@ -283,7 +331,7 @@ manager.registerTool({
283
331
  Live registration:
284
332
 
285
333
  - `registerTool(...)` and `unregisterTool(...)` update the local registry immediately
286
- - an active session is patched automatically, so no explicit `updateSession(...)` is needed
334
+ - an active OpenAI session is patched automatically, so no explicit `updateSession(...)` is needed; a live Gemini Live session refuses the patch (see [Session Refresh](#session-refresh)), so register tools before starting it
287
335
  - a change made while a response is in flight is deferred to just after it, and several deferred changes collapse into one refresh
288
336
  - both return `void`; a failed patch surfaces as `realtime:error`, not a rejection
289
337
  - until that patch lands, an already-active session may still call a just-unregistered tool, which returns a failure result
@@ -313,10 +361,14 @@ When connected, the manager relays:
313
361
  - `tts:audio:end`
314
362
 
315
363
  `tts:audio:end` reports that playback finished, not that the server finished
316
- sending — send completion can precede it by a wide margin. The transports derive
317
- it from
318
- `output_audio_buffer.stopped` (and `output_audio_buffer.cleared` on an
319
- interruption), never from `response.audio.done` / `response.output_audio.done`.
364
+ sending — send completion can precede it by a wide margin. The OpenAI
365
+ transports derive it from `output_audio_buffer.stopped` (and
366
+ `output_audio_buffer.cleared` on an interruption), never from
367
+ `response.audio.done` / `response.output_audio.done`. The Gemini transport
368
+ plays the audio itself and ends it only when its scheduler has drained AND the
369
+ server's `turnComplete` has arrived — neither alone, because the scheduler
370
+ drains spuriously at the start of a turn and `turnComplete` is paced by a
371
+ server clock rather than observed from playback (both measured).
320
372
  Consumers act on this event — `RenderManager` releases a held expression and
321
373
  stops lip-sync there — so a custom transport that reports it at send completion
322
374
  will reset the avatar's face mid-reply. See
@@ -325,7 +377,8 @@ will reset the avatar's face mid-reply. See
325
377
  `RealtimeState.audioPlaying` exposes the same playback segment as state, which is
326
378
  what to read when you need to know whether the character is still talking.
327
379
  `response.status` answers a different question — it completes when the provider
328
- stops sending, while playback runs on — so anything gated on it is wrong for the
329
- tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
380
+ closes the response, at send completion on OpenAI and at the clock-paced
381
+ `turnComplete` on Gemini, while playback runs on — so anything gated on it is
382
+ wrong for the tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
330
383
  still expected, which is the response-in-progress condition `sendMessage`
331
384
  refuses on.