@charivo/realtime 0.18.1 → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +83 -26
- package/dist/chunk-3YPRYU2M.mjs +1251 -0
- package/dist/chunk-3YPRYU2M.mjs.map +1 -0
- package/dist/chunk-5OO737VD.mjs +33 -0
- package/dist/chunk-5OO737VD.mjs.map +1 -0
- package/dist/{chunk-U7ER2VBF.mjs → chunk-EIYYLUWF.mjs} +10 -17
- package/dist/chunk-EIYYLUWF.mjs.map +1 -0
- package/dist/{chunk-VFIMTFOP.mjs → chunk-JEJGTULX.mjs} +24 -31
- package/dist/chunk-JEJGTULX.mjs.map +1 -0
- package/dist/{chunk-EXLERHST.mjs → chunk-VTHBTUH3.mjs} +7 -10
- package/dist/chunk-VTHBTUH3.mjs.map +1 -0
- package/dist/gemini/index.d.mts +22 -0
- package/dist/gemini/index.d.ts +22 -0
- package/dist/gemini/index.js +1282 -0
- package/dist/gemini/index.js.map +1 -0
- package/dist/gemini/index.mjs +4 -0
- package/dist/gemini/index.mjs.map +1 -0
- package/dist/index.d.mts +2 -2
- package/dist/index.d.ts +2 -2
- package/dist/index.js.map +1 -1
- package/dist/index.mjs.map +1 -1
- package/dist/openai/index.js +22 -7
- package/dist/openai/index.js.map +1 -1
- package/dist/openai/index.mjs +3 -2
- package/dist/openai-agents/index.js +25 -14
- package/dist/openai-agents/index.js.map +1 -1
- package/dist/openai-agents/index.mjs +3 -2
- package/dist/remote/index.d.mts +1 -0
- package/dist/remote/index.d.ts +1 -0
- package/dist/remote/index.js +1296 -37
- package/dist/remote/index.js.map +1 -1
- package/dist/remote/index.mjs +13 -4
- package/dist/remote/index.mjs.map +1 -1
- package/package.json +7 -2
- package/dist/chunk-EXLERHST.mjs.map +0 -1
- package/dist/chunk-U7ER2VBF.mjs.map +0 -1
- package/dist/chunk-VFIMTFOP.mjs.map +0 -1
package/README.md
CHANGED
|
@@ -50,11 +50,23 @@ await manager.startSession({
|
|
|
50
50
|
provider: "openai",
|
|
51
51
|
model: "gpt-realtime-2.1-mini",
|
|
52
52
|
});
|
|
53
|
+
|
|
54
|
+
// Or Gemini Live, over the same remote client and route:
|
|
55
|
+
await manager.startSession({ provider: "gemini", transport: "websocket" });
|
|
53
56
|
```
|
|
54
57
|
|
|
55
|
-
|
|
58
|
+
The remote client picks its browser adapter from that `provider`/`transport`
|
|
59
|
+
pair — OpenAI Realtime over WebRTC (the default transport) or Gemini Live over
|
|
60
|
+
WebSocket — and your route dispatches to `@charivo/server/openai` or
|
|
61
|
+
`@charivo/server/gemini` on the same field.
|
|
62
|
+
|
|
63
|
+
`gpt-realtime-2.1-mini` is the default OpenAI realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing) before switching. The Gemini default is `gemini-3.1-flash-live-preview`, the model the transport was measured against.
|
|
56
64
|
|
|
57
|
-
|
|
65
|
+
`createRealtimeManager` guarantees `prepareAudio` and `setEventEmitter`, so
|
|
66
|
+
calls on its result need no `?.`. A variable typed as the core
|
|
67
|
+
`RealtimeManager` still does — a third-party manager may omit them.
|
|
68
|
+
|
|
69
|
+
For iOS-safe lip-sync, call `prepareAudio()` from the same user gesture that
|
|
58
70
|
starts the session, before `startSession(...)`. Pass the same session config to
|
|
59
71
|
both calls — `prepareAudio` needs it to resolve which remote adapter to warm up:
|
|
60
72
|
|
|
@@ -64,10 +76,13 @@ const sessionConfig = {
|
|
|
64
76
|
model: "gpt-realtime-2.1-mini",
|
|
65
77
|
};
|
|
66
78
|
|
|
67
|
-
await manager.prepareAudio
|
|
79
|
+
await manager.prepareAudio(sessionConfig);
|
|
68
80
|
await manager.startSession(sessionConfig);
|
|
69
81
|
```
|
|
70
82
|
|
|
83
|
+
On an OpenAI session, `updateSession(...)` patches the live session in place
|
|
84
|
+
(Gemini Live refuses live patches — see [Session Refresh](#session-refresh)):
|
|
85
|
+
|
|
71
86
|
```ts
|
|
72
87
|
await manager.updateSession({
|
|
73
88
|
voice: "alloy",
|
|
@@ -89,29 +104,39 @@ session active and drives reconnect attempts internally. During that window
|
|
|
89
104
|
`state.session.status` stays `"active"` while `state.connection` moves back to
|
|
90
105
|
`"connecting"`.
|
|
91
106
|
|
|
92
|
-
## No-server dev
|
|
107
|
+
## No-server dev
|
|
93
108
|
|
|
94
|
-
For local development you can skip the server route: pass
|
|
95
|
-
the direct
|
|
96
|
-
|
|
109
|
+
For local development you can skip the server route: pass a provider API key to
|
|
110
|
+
the direct transport client and it mints the short-lived credential in the
|
|
111
|
+
browser — OpenAI's realtime client secret, or Google's single-use ephemeral
|
|
112
|
+
token — mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
|
|
97
113
|
|
|
98
114
|
```ts
|
|
99
115
|
import { createRealtimeManager } from "@charivo/realtime";
|
|
100
116
|
import { createOpenAIRealtimeAgentsClient } from "@charivo/realtime/openai-agents";
|
|
117
|
+
import { createGeminiLiveClient } from "@charivo/realtime/gemini";
|
|
101
118
|
|
|
102
119
|
// Dev/testing only: the API key is exposed in the browser.
|
|
103
|
-
const
|
|
104
|
-
|
|
105
|
-
|
|
120
|
+
const manager = createRealtimeManager(
|
|
121
|
+
createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." }),
|
|
122
|
+
);
|
|
106
123
|
await manager.startSession({ provider: "openai", model: "gpt-realtime-2.1-mini" });
|
|
124
|
+
|
|
125
|
+
// Or Gemini Live, the same way.
|
|
126
|
+
const geminiManager = createRealtimeManager(
|
|
127
|
+
createGeminiLiveClient({ apiKey: "AIza..." }),
|
|
128
|
+
);
|
|
129
|
+
await geminiManager.startSession({ provider: "gemini", transport: "websocket" });
|
|
107
130
|
```
|
|
108
131
|
|
|
109
|
-
|
|
110
|
-
`
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
132
|
+
Both clients share the option precedence `sessionBootstrap` > `apiEndpoint` >
|
|
133
|
+
`apiKey`. The `apiKey` path is dev/testing only and additionally requires
|
|
134
|
+
microphone permission, a secure context (`localhost` or `https`), and a user
|
|
135
|
+
gesture to start; the minted credential is short-lived (re-minted per session).
|
|
136
|
+
The Gemini token still carries the whole session, exactly as the server
|
|
137
|
+
provider mints it, so nothing downstream differs from production. For
|
|
138
|
+
production, use the server-mediated `@charivo/realtime/remote` client shown
|
|
139
|
+
above.
|
|
115
140
|
|
|
116
141
|
## Exports
|
|
117
142
|
|
|
@@ -123,6 +148,20 @@ client shown above.
|
|
|
123
148
|
`resultProjectors`
|
|
124
149
|
- `RealtimeLogger`
|
|
125
150
|
|
|
151
|
+
Transport clients live on subpaths:
|
|
152
|
+
|
|
153
|
+
- `@charivo/realtime/remote`: `createRemoteRealtimeClient(config?)`, plus
|
|
154
|
+
`DEFAULT_REMOTE_REALTIME_ADAPTERS` — the registry it resolves from, keyed by
|
|
155
|
+
the adapter ids exported from `@charivo/core`
|
|
156
|
+
- `@charivo/realtime/openai-agents`: `createOpenAIRealtimeAgentsClient(options?)`
|
|
157
|
+
— OpenAI Realtime over WebRTC through the OpenAI Agents SDK
|
|
158
|
+
- `@charivo/realtime/openai`: `createOpenAIRealtimeClient(options?)` — the
|
|
159
|
+
legacy low-level OpenAI WebRTC transport
|
|
160
|
+
- `@charivo/realtime/gemini`: `createGeminiLiveClient(options?)` — Gemini Live
|
|
161
|
+
over WebSocket (`apiEndpoint`, `sessionBootstrap`, or a dev/testing
|
|
162
|
+
`apiKey`); captures the microphone at 16 kHz and plays the model's 24 kHz PCM
|
|
163
|
+
through its own scheduler with a lip-sync tap
|
|
164
|
+
|
|
126
165
|
## Instruction Layering
|
|
127
166
|
|
|
128
167
|
`@charivo/realtime` keeps its default instructions generic: spoken-output
|
|
@@ -156,8 +195,10 @@ await manager.startSession({
|
|
|
156
195
|
`buildRealtimeSessionConfig(...)` does not fill provider-specific fields such
|
|
157
196
|
as `provider` or `model`. Prefer building on top of it for instructions and
|
|
158
197
|
character voice, then pass provider/model explicitly at `startSession(...)`.
|
|
159
|
-
|
|
160
|
-
|
|
198
|
+
Each transport/provider pair keeps its own fallbacks for omitted model or voice
|
|
199
|
+
values: the OpenAI packages theirs, and `@charivo/server/gemini`
|
|
200
|
+
`gemini-3.1-flash-live-preview` with the `Kore` voice — which is also what a
|
|
201
|
+
voice id it does not recognise (an OpenAI one, say) silently becomes.
|
|
161
202
|
|
|
162
203
|
Avatar-specific realtime tools and avatar-specific instruction addenda now live
|
|
163
204
|
in `@charivo/avatar` (works with both `@charivo/realtime` and `@charivo/llm`;
|
|
@@ -216,7 +257,12 @@ latest requested config, current character, and current tool registry.
|
|
|
216
257
|
- inactive managers only cache the requested base config for the next
|
|
217
258
|
`startSession(...)`
|
|
218
259
|
- active managers keep the current connection open and issue a transport-level
|
|
219
|
-
session patch
|
|
260
|
+
session patch — which `@charivo/realtime/gemini` rejects outright: a Gemini
|
|
261
|
+
Live session is fixed when its token is minted (the token's setup replaces
|
|
262
|
+
the browser's, measured), and the Live API has no `session.update`. The call
|
|
263
|
+
rejects, `realtime:error` carries the same error, and the live session keeps
|
|
264
|
+
the configuration it started with; stop and start a new session to change it,
|
|
265
|
+
and register tools before `startSession(...)`
|
|
220
266
|
- successful patches update `realtime:state` only and do not emit synthetic
|
|
221
267
|
`realtime:session:end/start` refresh boundaries
|
|
222
268
|
- patch failures keep the current live session and previous
|
|
@@ -237,6 +283,12 @@ Successful reconnects are treated as a continuation of the same live session.
|
|
|
237
283
|
- the next reconnect attempt always rebuilds from the latest effective config
|
|
238
284
|
- in-flight assistant responses are marked as interrupted and are not resumed
|
|
239
285
|
- old tool-call ids are not replayed after reconnect
|
|
286
|
+
- on Gemini Live every attempt re-mints, through your route or (on the dev
|
|
287
|
+
`apiKey` path) straight from Google's token endpoint — tokens are `uses: 1`,
|
|
288
|
+
and replaying one closes the socket with `1011` (measured) — and
|
|
289
|
+
the recovered session is a fresh one: `sessionResumptionUpdate` handles are
|
|
290
|
+
not spent, `goAway` is not acted on (the close after it drives this same
|
|
291
|
+
recovery), and the transport's microphone gate re-arms
|
|
240
292
|
|
|
241
293
|
Observability events emitted by the manager:
|
|
242
294
|
|
|
@@ -279,7 +331,7 @@ manager.registerTool({
|
|
|
279
331
|
Live registration:
|
|
280
332
|
|
|
281
333
|
- `registerTool(...)` and `unregisterTool(...)` update the local registry immediately
|
|
282
|
-
- an active session is patched automatically, so no explicit `updateSession(...)` is needed
|
|
334
|
+
- an active OpenAI session is patched automatically, so no explicit `updateSession(...)` is needed; a live Gemini Live session refuses the patch (see [Session Refresh](#session-refresh)), so register tools before starting it
|
|
283
335
|
- a change made while a response is in flight is deferred to just after it, and several deferred changes collapse into one refresh
|
|
284
336
|
- both return `void`; a failed patch surfaces as `realtime:error`, not a rejection
|
|
285
337
|
- until that patch lands, an already-active session may still call a just-unregistered tool, which returns a failure result
|
|
@@ -309,10 +361,14 @@ When connected, the manager relays:
|
|
|
309
361
|
- `tts:audio:end`
|
|
310
362
|
|
|
311
363
|
`tts:audio:end` reports that playback finished, not that the server finished
|
|
312
|
-
sending — send completion can precede it by a wide margin. The
|
|
313
|
-
it from
|
|
314
|
-
`output_audio_buffer.
|
|
315
|
-
|
|
364
|
+
sending — send completion can precede it by a wide margin. The OpenAI
|
|
365
|
+
transports derive it from `output_audio_buffer.stopped` (and
|
|
366
|
+
`output_audio_buffer.cleared` on an interruption), never from
|
|
367
|
+
`response.audio.done` / `response.output_audio.done`. The Gemini transport
|
|
368
|
+
plays the audio itself and ends it only when its scheduler has drained AND the
|
|
369
|
+
server's `turnComplete` has arrived — neither alone, because the scheduler
|
|
370
|
+
drains spuriously at the start of a turn and `turnComplete` is paced by a
|
|
371
|
+
server clock rather than observed from playback (both measured).
|
|
316
372
|
Consumers act on this event — `RenderManager` releases a held expression and
|
|
317
373
|
stops lip-sync there — so a custom transport that reports it at send completion
|
|
318
374
|
will reset the avatar's face mid-reply. See
|
|
@@ -321,7 +377,8 @@ will reset the avatar's face mid-reply. See
|
|
|
321
377
|
`RealtimeState.audioPlaying` exposes the same playback segment as state, which is
|
|
322
378
|
what to read when you need to know whether the character is still talking.
|
|
323
379
|
`response.status` answers a different question — it completes when the provider
|
|
324
|
-
|
|
325
|
-
|
|
380
|
+
closes the response, at send completion on OpenAI and at the clock-paced
|
|
381
|
+
`turnComplete` on Gemini, while playback runs on — so anything gated on it is
|
|
382
|
+
wrong for the tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
|
|
326
383
|
still expected, which is the response-in-progress condition `sendMessage`
|
|
327
384
|
refuses on.
|