@charivo/realtime 0.19.0 → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -24
- package/dist/chunk-3YPRYU2M.mjs +1251 -0
- package/dist/chunk-3YPRYU2M.mjs.map +1 -0
- package/dist/chunk-5OO737VD.mjs +33 -0
- package/dist/chunk-5OO737VD.mjs.map +1 -0
- package/dist/{chunk-U7ER2VBF.mjs → chunk-EIYYLUWF.mjs} +10 -17
- package/dist/chunk-EIYYLUWF.mjs.map +1 -0
- package/dist/{chunk-VFIMTFOP.mjs → chunk-JEJGTULX.mjs} +24 -31
- package/dist/chunk-JEJGTULX.mjs.map +1 -0
- package/dist/{chunk-EXLERHST.mjs → chunk-VTHBTUH3.mjs} +7 -10
- package/dist/chunk-VTHBTUH3.mjs.map +1 -0
- package/dist/gemini/index.d.mts +22 -0
- package/dist/gemini/index.d.ts +22 -0
- package/dist/gemini/index.js +1282 -0
- package/dist/gemini/index.js.map +1 -0
- package/dist/gemini/index.mjs +4 -0
- package/dist/gemini/index.mjs.map +1 -0
- package/dist/openai/index.js +22 -7
- package/dist/openai/index.js.map +1 -1
- package/dist/openai/index.mjs +3 -2
- package/dist/openai-agents/index.js +25 -14
- package/dist/openai-agents/index.js.map +1 -1
- package/dist/openai-agents/index.mjs +3 -2
- package/dist/remote/index.d.mts +1 -0
- package/dist/remote/index.d.ts +1 -0
- package/dist/remote/index.js +1296 -37
- package/dist/remote/index.js.map +1 -1
- package/dist/remote/index.mjs +13 -4
- package/dist/remote/index.mjs.map +1 -1
- package/package.json +7 -2
- package/dist/chunk-EXLERHST.mjs.map +0 -1
- package/dist/chunk-U7ER2VBF.mjs.map +0 -1
- package/dist/chunk-VFIMTFOP.mjs.map +0 -1
package/README.md
CHANGED
|
@@ -50,9 +50,17 @@ await manager.startSession({
|
|
|
50
50
|
provider: "openai",
|
|
51
51
|
model: "gpt-realtime-2.1-mini",
|
|
52
52
|
});
|
|
53
|
+
|
|
54
|
+
// Or Gemini Live, over the same remote client and route:
|
|
55
|
+
await manager.startSession({ provider: "gemini", transport: "websocket" });
|
|
53
56
|
```
|
|
54
57
|
|
|
55
|
-
|
|
58
|
+
The remote client picks its browser adapter from that `provider`/`transport`
|
|
59
|
+
pair — OpenAI Realtime over WebRTC (the default transport) or Gemini Live over
|
|
60
|
+
WebSocket — and your route dispatches to `@charivo/server/openai` or
|
|
61
|
+
`@charivo/server/gemini` on the same field.
|
|
62
|
+
|
|
63
|
+
`gpt-realtime-2.1-mini` is the default OpenAI realtime model; the full `gpt-realtime-2.1` is available but meaningfully more expensive — consult [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing) before switching. The Gemini default is `gemini-3.1-flash-live-preview`, the model the transport was measured against.
|
|
56
64
|
|
|
57
65
|
`createRealtimeManager` guarantees `prepareAudio` and `setEventEmitter`, so
|
|
58
66
|
calls on its result need no `?.`. A variable typed as the core
|
|
@@ -72,6 +80,9 @@ await manager.prepareAudio(sessionConfig);
|
|
|
72
80
|
await manager.startSession(sessionConfig);
|
|
73
81
|
```
|
|
74
82
|
|
|
83
|
+
On an OpenAI session, `updateSession(...)` patches the live session in place
|
|
84
|
+
(Gemini Live refuses live patches — see [Session Refresh](#session-refresh)):
|
|
85
|
+
|
|
75
86
|
```ts
|
|
76
87
|
await manager.updateSession({
|
|
77
88
|
voice: "alloy",
|
|
@@ -93,29 +104,39 @@ session active and drives reconnect attempts internally. During that window
|
|
|
93
104
|
`state.session.status` stays `"active"` while `state.connection` moves back to
|
|
94
105
|
`"connecting"`.
|
|
95
106
|
|
|
96
|
-
## No-server dev
|
|
107
|
+
## No-server dev
|
|
97
108
|
|
|
98
|
-
For local development you can skip the server route: pass
|
|
99
|
-
the direct
|
|
100
|
-
|
|
109
|
+
For local development you can skip the server route: pass a provider API key to
|
|
110
|
+
the direct transport client and it mints the short-lived credential in the
|
|
111
|
+
browser — OpenAI's realtime client secret, or Google's single-use ephemeral
|
|
112
|
+
token — mirroring `@charivo/llm/openai` and `@charivo/tts/openai`.
|
|
101
113
|
|
|
102
114
|
```ts
|
|
103
115
|
import { createRealtimeManager } from "@charivo/realtime";
|
|
104
116
|
import { createOpenAIRealtimeAgentsClient } from "@charivo/realtime/openai-agents";
|
|
117
|
+
import { createGeminiLiveClient } from "@charivo/realtime/gemini";
|
|
105
118
|
|
|
106
119
|
// Dev/testing only: the API key is exposed in the browser.
|
|
107
|
-
const
|
|
108
|
-
|
|
109
|
-
|
|
120
|
+
const manager = createRealtimeManager(
|
|
121
|
+
createOpenAIRealtimeAgentsClient({ apiKey: "sk-..." }),
|
|
122
|
+
);
|
|
110
123
|
await manager.startSession({ provider: "openai", model: "gpt-realtime-2.1-mini" });
|
|
124
|
+
|
|
125
|
+
// Or Gemini Live, the same way.
|
|
126
|
+
const geminiManager = createRealtimeManager(
|
|
127
|
+
createGeminiLiveClient({ apiKey: "AIza..." }),
|
|
128
|
+
);
|
|
129
|
+
await geminiManager.startSession({ provider: "gemini", transport: "websocket" });
|
|
111
130
|
```
|
|
112
131
|
|
|
113
|
-
|
|
114
|
-
`
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
132
|
+
Both clients share the option precedence `sessionBootstrap` > `apiEndpoint` >
|
|
133
|
+
`apiKey`. The `apiKey` path is dev/testing only and additionally requires
|
|
134
|
+
microphone permission, a secure context (`localhost` or `https`), and a user
|
|
135
|
+
gesture to start; the minted credential is short-lived (re-minted per session).
|
|
136
|
+
The Gemini token still carries the whole session, exactly as the server
|
|
137
|
+
provider mints it, so nothing downstream differs from production. For
|
|
138
|
+
production, use the server-mediated `@charivo/realtime/remote` client shown
|
|
139
|
+
above.
|
|
119
140
|
|
|
120
141
|
## Exports
|
|
121
142
|
|
|
@@ -127,6 +148,20 @@ client shown above.
|
|
|
127
148
|
`resultProjectors`
|
|
128
149
|
- `RealtimeLogger`
|
|
129
150
|
|
|
151
|
+
Transport clients live on subpaths:
|
|
152
|
+
|
|
153
|
+
- `@charivo/realtime/remote`: `createRemoteRealtimeClient(config?)`, plus
|
|
154
|
+
`DEFAULT_REMOTE_REALTIME_ADAPTERS` — the registry it resolves from, keyed by
|
|
155
|
+
the adapter ids exported from `@charivo/core`
|
|
156
|
+
- `@charivo/realtime/openai-agents`: `createOpenAIRealtimeAgentsClient(options?)`
|
|
157
|
+
— OpenAI Realtime over WebRTC through the OpenAI Agents SDK
|
|
158
|
+
- `@charivo/realtime/openai`: `createOpenAIRealtimeClient(options?)` — the
|
|
159
|
+
legacy low-level OpenAI WebRTC transport
|
|
160
|
+
- `@charivo/realtime/gemini`: `createGeminiLiveClient(options?)` — Gemini Live
|
|
161
|
+
over WebSocket (`apiEndpoint`, `sessionBootstrap`, or a dev/testing
|
|
162
|
+
`apiKey`); captures the microphone at 16 kHz and plays the model's 24 kHz PCM
|
|
163
|
+
through its own scheduler with a lip-sync tap
|
|
164
|
+
|
|
130
165
|
## Instruction Layering
|
|
131
166
|
|
|
132
167
|
`@charivo/realtime` keeps its default instructions generic: spoken-output
|
|
@@ -160,8 +195,10 @@ await manager.startSession({
|
|
|
160
195
|
`buildRealtimeSessionConfig(...)` does not fill provider-specific fields such
|
|
161
196
|
as `provider` or `model`. Prefer building on top of it for instructions and
|
|
162
197
|
character voice, then pass provider/model explicitly at `startSession(...)`.
|
|
163
|
-
|
|
164
|
-
|
|
198
|
+
Each transport/provider pair keeps its own fallbacks for omitted model or voice
|
|
199
|
+
values: the OpenAI packages theirs, and `@charivo/server/gemini`
|
|
200
|
+
`gemini-3.1-flash-live-preview` with the `Kore` voice — which is also what a
|
|
201
|
+
voice id it does not recognise (an OpenAI one, say) silently becomes.
|
|
165
202
|
|
|
166
203
|
Avatar-specific realtime tools and avatar-specific instruction addenda now live
|
|
167
204
|
in `@charivo/avatar` (works with both `@charivo/realtime` and `@charivo/llm`;
|
|
@@ -220,7 +257,12 @@ latest requested config, current character, and current tool registry.
|
|
|
220
257
|
- inactive managers only cache the requested base config for the next
|
|
221
258
|
`startSession(...)`
|
|
222
259
|
- active managers keep the current connection open and issue a transport-level
|
|
223
|
-
session patch
|
|
260
|
+
session patch — which `@charivo/realtime/gemini` rejects outright: a Gemini
|
|
261
|
+
Live session is fixed when its token is minted (the token's setup replaces
|
|
262
|
+
the browser's, measured), and the Live API has no `session.update`. The call
|
|
263
|
+
rejects, `realtime:error` carries the same error, and the live session keeps
|
|
264
|
+
the configuration it started with; stop and start a new session to change it,
|
|
265
|
+
and register tools before `startSession(...)`
|
|
224
266
|
- successful patches update `realtime:state` only and do not emit synthetic
|
|
225
267
|
`realtime:session:end/start` refresh boundaries
|
|
226
268
|
- patch failures keep the current live session and previous
|
|
@@ -241,6 +283,12 @@ Successful reconnects are treated as a continuation of the same live session.
|
|
|
241
283
|
- the next reconnect attempt always rebuilds from the latest effective config
|
|
242
284
|
- in-flight assistant responses are marked as interrupted and are not resumed
|
|
243
285
|
- old tool-call ids are not replayed after reconnect
|
|
286
|
+
- on Gemini Live every attempt re-mints, through your route or (on the dev
|
|
287
|
+
`apiKey` path) straight from Google's token endpoint — tokens are `uses: 1`,
|
|
288
|
+
and replaying one closes the socket with `1011` (measured) — and
|
|
289
|
+
the recovered session is a fresh one: `sessionResumptionUpdate` handles are
|
|
290
|
+
not spent, `goAway` is not acted on (the close after it drives this same
|
|
291
|
+
recovery), and the transport's microphone gate re-arms
|
|
244
292
|
|
|
245
293
|
Observability events emitted by the manager:
|
|
246
294
|
|
|
@@ -283,7 +331,7 @@ manager.registerTool({
|
|
|
283
331
|
Live registration:
|
|
284
332
|
|
|
285
333
|
- `registerTool(...)` and `unregisterTool(...)` update the local registry immediately
|
|
286
|
-
- an active session is patched automatically, so no explicit `updateSession(...)` is needed
|
|
334
|
+
- an active OpenAI session is patched automatically, so no explicit `updateSession(...)` is needed; a live Gemini Live session refuses the patch (see [Session Refresh](#session-refresh)), so register tools before starting it
|
|
287
335
|
- a change made while a response is in flight is deferred to just after it, and several deferred changes collapse into one refresh
|
|
288
336
|
- both return `void`; a failed patch surfaces as `realtime:error`, not a rejection
|
|
289
337
|
- until that patch lands, an already-active session may still call a just-unregistered tool, which returns a failure result
|
|
@@ -313,10 +361,14 @@ When connected, the manager relays:
|
|
|
313
361
|
- `tts:audio:end`
|
|
314
362
|
|
|
315
363
|
`tts:audio:end` reports that playback finished, not that the server finished
|
|
316
|
-
sending — send completion can precede it by a wide margin. The
|
|
317
|
-
it from
|
|
318
|
-
`output_audio_buffer.
|
|
319
|
-
|
|
364
|
+
sending — send completion can precede it by a wide margin. The OpenAI
|
|
365
|
+
transports derive it from `output_audio_buffer.stopped` (and
|
|
366
|
+
`output_audio_buffer.cleared` on an interruption), never from
|
|
367
|
+
`response.audio.done` / `response.output_audio.done`. The Gemini transport
|
|
368
|
+
plays the audio itself and ends it only when its scheduler has drained AND the
|
|
369
|
+
server's `turnComplete` has arrived — neither alone, because the scheduler
|
|
370
|
+
drains spuriously at the start of a turn and `turnComplete` is paced by a
|
|
371
|
+
server clock rather than observed from playback (both measured).
|
|
320
372
|
Consumers act on this event — `RenderManager` releases a held expression and
|
|
321
373
|
stops lip-sync there — so a custom transport that reports it at send completion
|
|
322
374
|
will reset the avatar's face mid-reply. See
|
|
@@ -325,7 +377,8 @@ will reset the avatar's face mid-reply. See
|
|
|
325
377
|
`RealtimeState.audioPlaying` exposes the same playback segment as state, which is
|
|
326
378
|
what to read when you need to know whether the character is still talking.
|
|
327
379
|
`response.status` answers a different question — it completes when the provider
|
|
328
|
-
|
|
329
|
-
|
|
380
|
+
closes the response, at send completion on OpenAI and at the clock-paced
|
|
381
|
+
`turnComplete` on Gemini, while playback runs on — so anything gated on it is
|
|
382
|
+
wrong for the tail of every turn. `RealtimeState.awaitingResponse` reports whether a reply is
|
|
330
383
|
still expected, which is the response-in-progress condition `sendMessage`
|
|
331
384
|
refuses on.
|