realtime-avatar 0.25.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -0
- package/dist/browser.d.ts +2 -87
- package/dist/browser.js +6 -1
- package/dist/{chunk-GMYGHDY2.js → chunk-HMESE54B.js} +22 -4
- package/dist/{chunk-DNHHQ5GZ.js → chunk-XJPGQMEW.js} +28 -10
- package/dist/express.js +2 -2
- package/dist/hono.js +2 -2
- package/dist/index.d.ts +12 -0
- package/dist/index.js +1 -1
- package/dist/nextjs.js +2 -2
- package/dist/playout-delay-CtRZdtsj.d.ts +168 -0
- package/dist/{proxy-client-O-Mu9JKt.d.ts → proxy-client-C8M_G_Kk.d.ts} +163 -17
- package/dist/react-native.d.ts +3 -3
- package/dist/react-native.js +341 -26
- package/dist/react.d.ts +75 -14
- package/dist/react.js +532 -83
- package/dist/server.js +1 -1
- package/dist/tanstack-start.js +2 -2
- package/dist/tools.d.ts +17 -1
- package/dist/tools.js +28 -8
- package/package.json +1 -1
- package/dist/playout-delay-DWOc8iCl.d.ts +0 -65
package/README.md
CHANGED
|
@@ -224,6 +224,7 @@ all), and the `video` policy types are deliberately not one-to-one with the wire
|
|
|
224
224
|
// calls
|
|
225
225
|
rta.startCall({ avatarId, mode?, instructions?, context?, maxSeconds?, video?, recording?, connectionHistory?, transcript?, metadata? })
|
|
226
226
|
rta.endCall(sessionId, { reason? }) // free an abandoned call's slot; idempotent, never throws
|
|
227
|
+
rta.leaveQueue(queueTicketId, { reason? }) // give up a queued call's place in line; idempotent, never throws
|
|
227
228
|
|
|
228
229
|
// optional recordings; server only, requires recordings:read
|
|
229
230
|
rta.listRecordings({ sessionId, limit?, cursor? })
|
|
@@ -294,6 +295,61 @@ call. Policy — `instructions`, `maxSeconds`, `voice`, `video` — is decided i
|
|
|
294
295
|
your server. A route that spreads the request body into `startCall` hands your caller your
|
|
295
296
|
system prompt and your bill.
|
|
296
297
|
|
|
298
|
+
### Styling
|
|
299
|
+
|
|
300
|
+
`AvatarCall` and `AvatarVideoSurface` lay themselves out with inline styles, so they need no
|
|
301
|
+
CSS build and no configuration: no stylesheet to import and no Tailwind `@source` for this
|
|
302
|
+
package. The surface fills its container; give that container a definite size (an aspect
|
|
303
|
+
ratio plus a height, say). `className` on the surface adds your own classes; to change a
|
|
304
|
+
property the surface sets itself, such as its 100% size, pass `style`.
|
|
305
|
+
|
|
306
|
+
`AvatarCall`'s `children` render in a layer that fills the call's box above the video and
|
|
307
|
+
the live badge, so an absolutely positioned control places itself against the box and stays
|
|
308
|
+
clickable while she is live.
|
|
309
|
+
|
|
310
|
+
### Microphone and sound
|
|
311
|
+
|
|
312
|
+
The handle reports the two local failures a "live" call can hide. `call.microphone` says
|
|
313
|
+
whether she can hear the user (`off`, `pending`, `on`, `muted`, `blocked`, `unavailable`; the
|
|
314
|
+
last two carry `reason`, `message` and `hint`), and `onMicrophoneProblem` fires once per new
|
|
315
|
+
problem. A device that stops mid-call reads `pending` while LiveKit retries the default device,
|
|
316
|
+
and `unavailable` (`device-lost`) only once that fails. `call.retryMicrophone()` asks again after
|
|
317
|
+
the user fixes it, and `call.setMicrophoneEnabled(enabled)` mutes and unmutes. While the call is
|
|
318
|
+
waiting or connecting those only record the choice (nothing is captured before the room
|
|
319
|
+
connects), a mute made then is kept when the call goes live, and after `end()` they do nothing.
|
|
320
|
+
|
|
321
|
+
`call.audio` says whether the user can hear her (`unknown`, `allowed`, `blocked`; `unknown` again
|
|
322
|
+
once the call ended). While it is `blocked`, `AvatarCall` shows a "Tap to turn on sound" button,
|
|
323
|
+
top-centre in a polite live region, that calls `call.startAudio()`. Pass `audioUnlockPrompt` a
|
|
324
|
+
function to render your own (your words, your placement), or `false` for nothing.
|
|
325
|
+
`startAudio()` must run inside a click or tap handler; it never rejects and resolves whether
|
|
326
|
+
playback is allowed afterwards.
|
|
327
|
+
|
|
328
|
+
Every state is derived from LiveKit's public events, and LiveKit still owns the device: it
|
|
329
|
+
captures on connect and stops when the room leaves. A microphone error from an earlier call on
|
|
330
|
+
the same room is not reported for the next one. A publish that fails after the microphone was
|
|
331
|
+
captured reaches only the room's `onError` and stays `pending`: that callback also carries
|
|
332
|
+
unrelated failures, so it is not guessed into a microphone problem.
|
|
333
|
+
|
|
334
|
+
### Ending a call
|
|
335
|
+
|
|
336
|
+
`call.end()` (and `useRealtimeSession().end()`) is terminal from every phase, including
|
|
337
|
+
`waiting` in the queue and `connecting`. It stops the queue retry and the reconnect ladder,
|
|
338
|
+
releases the held session or the queue ticket, leaves the room (which stops the microphone),
|
|
339
|
+
moves `status` to `"ended"` and fires `onEnded({ reason: "user_ended" })` once. Ending a call
|
|
340
|
+
that is already over does nothing. `sayAndEnd(text)` is the graceful path when she should speak
|
|
341
|
+
a last line first.
|
|
342
|
+
|
|
343
|
+
**After `end()`, nothing starts a call except you, explicitly.** Changing `avatarId`, `mode` or
|
|
344
|
+
`listen` on an ended `AvatarCall` does not redial; remount it, for example with a `key`:
|
|
345
|
+
|
|
346
|
+
```tsx
|
|
347
|
+
<AvatarCall key={callId} client={client} avatarId={avatarId} /> // new callId → new call
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
On the lower-level hooks, `reconnect()` redials (a double tap mints once), and so does setting
|
|
351
|
+
`active` to false and back. `reset()` does not undo an end.
|
|
352
|
+
|
|
297
353
|
### Optional connection details
|
|
298
354
|
|
|
299
355
|
`AvatarCall` provides call status, actions and end reasons for your default UI. To show
|
|
@@ -390,6 +446,28 @@ room-wide to whatever you render inside it (`playoutDelaySeconds={false}` opts o
|
|
|
390
446
|
`attachAvatarTools` runs your functions in the page. Nothing is executed on the platform, and
|
|
391
447
|
a tool has **2.5 seconds** to answer before the call gives up on it and tells her it failed.
|
|
392
448
|
|
|
449
|
+
A failed registration throws `ToolRegistrationError`, whose `retryable` says whether trying again
|
|
450
|
+
can help: the agent not there or not armed by the deadline (a slow start, or a session minted
|
|
451
|
+
without `client_tools`; one attempt cannot tell which), or an RPC transport failure, is retryable;
|
|
452
|
+
an oversized manifest or a request LiveKit or the agent refused is not.
|
|
453
|
+
|
|
454
|
+
In React, `useCharacterTools(tools)` registers them while the room is connected and returns
|
|
455
|
+
their state: `idle`, `registering` (with `attempt`), `ready` (`registered`, `rejected`) or
|
|
456
|
+
`error`. A retryable failure is retried twice (after 1s, then 3s) while the room stays
|
|
457
|
+
connected; only then is it `error`. It re-registers when the manifest changes (names,
|
|
458
|
+
descriptions, parameters), not when the `tools` object does, so an inline object is fine and
|
|
459
|
+
each call reaches your latest `execute`. With `AvatarCall`, call it from a component you render
|
|
460
|
+
as a child, which runs inside the call's room, and show its state rather than ignore it:
|
|
461
|
+
|
|
462
|
+
```tsx
|
|
463
|
+
function Tools() {
|
|
464
|
+
const tools = useCharacterTools(myTools);
|
|
465
|
+
return tools.status === "error" ? <p role="status">Tools are off for this call: {tools.error}</p> : null;
|
|
466
|
+
}
|
|
467
|
+
|
|
468
|
+
<AvatarCall client={client} avatarId="ava_…">{() => <Tools />}</AvatarCall>
|
|
469
|
+
```
|
|
470
|
+
|
|
393
471
|
## What `/react` exports, and what it deliberately does not
|
|
394
472
|
|
|
395
473
|
31 names, down from 82 on 2026-08-26. Two groups came out and are not coming back:
|
package/dist/browser.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { D as DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS, P as PlayoutDelayTarget,
|
|
1
|
+
export { D as DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS, E as EnableMicrophoneOptions, M as MicrophoneCapableRoom, a as MicrophoneFailureReason, b as MicrophoneProblem, c as MicrophoneProblemReason, d as MicrophoneResult, P as PlayoutDelayTarget, e as applyAvatarPlayoutDelay, f as applyPlayoutDelay, g as describeMicrophoneFailure, h as enableMicrophone } from './playout-delay-CtRZdtsj.js';
|
|
2
2
|
|
|
3
3
|
type VideoSupport = {
|
|
4
4
|
supported: true;
|
|
@@ -10,91 +10,6 @@ type VideoSupport = {
|
|
|
10
10
|
/** Check before starting a billed video session; this does not prove negotiated playback. */
|
|
11
11
|
declare function checkVideoSupport(codec?: string): VideoSupport;
|
|
12
12
|
|
|
13
|
-
/**
|
|
14
|
-
* Turning the microphone on, and saying what went wrong when it does not.
|
|
15
|
-
*
|
|
16
|
-
* WHY THIS EXISTS. `room.localParticipant.setMicrophoneEnabled(true)` is one line, so every
|
|
17
|
-
* integration writes it inline and none of them handle its rejection — the reject path is
|
|
18
|
-
* invisible until a real user is on a real machine. When it fails there is nothing to see:
|
|
19
|
-
* the call is already connected, the page is mid-`await`, and the rejection becomes an
|
|
20
|
-
* unhandled promise nobody is looking at. The reported symptom is always the same sentence —
|
|
21
|
-
* "the mic won't start" — and it covers at least six unrelated causes with different fixes,
|
|
22
|
-
* one of which is not in the browser at all.
|
|
23
|
-
*
|
|
24
|
-
* So this returns a RESULT rather than throwing: the cause becomes a value the caller has to
|
|
25
|
-
* handle, and each one carries the sentence that names its actual fix. `message` is what the
|
|
26
|
-
* browser said; `hint` is what the person should do.
|
|
27
|
-
*
|
|
28
|
-
* It is deliberately structural — it never imports `livekit-client`, so it pins no version
|
|
29
|
-
* and adds no bytes. Anything with a `localParticipant.setMicrophoneEnabled` satisfies it.
|
|
30
|
-
*/
|
|
31
|
-
/** The sliver of livekit-client's `Room` this needs. Structural on purpose — see above. */
|
|
32
|
-
interface MicrophoneCapableRoom {
|
|
33
|
-
localParticipant: {
|
|
34
|
-
setMicrophoneEnabled(enabled: boolean): Promise<unknown>;
|
|
35
|
-
};
|
|
36
|
-
}
|
|
37
|
-
/**
|
|
38
|
-
* Why the microphone did not start. Every variant has a DIFFERENT fix, which is the whole
|
|
39
|
-
* reason this is an enum and not a boolean.
|
|
40
|
-
*/
|
|
41
|
-
type MicrophoneFailureReason =
|
|
42
|
-
/** No `navigator.mediaDevices` — the page is not a secure context. */
|
|
43
|
-
"insecure-origin"
|
|
44
|
-
/** The permission prompt was never answered. getUserMedia may never settle; we time out. */
|
|
45
|
-
| "no-answer"
|
|
46
|
-
/** The BROWSER denied it — the site's own permission, resettable from the address bar. */
|
|
47
|
-
| "denied-by-browser"
|
|
48
|
-
/** The OS denied it — macOS System Settings, nothing the page or the site can change. */
|
|
49
|
-
| "denied-by-os"
|
|
50
|
-
/** No input device is connected, or none matches the constraints. */
|
|
51
|
-
| "no-device"
|
|
52
|
-
/** A device exists but something else holds it — another tab, app, or a stale call. */
|
|
53
|
-
| "device-in-use"
|
|
54
|
-
/** Anything unrecognised. `message` still carries what the browser said. */
|
|
55
|
-
| "unknown";
|
|
56
|
-
type MicrophoneResult = {
|
|
57
|
-
ok: true;
|
|
58
|
-
} | {
|
|
59
|
-
ok: false;
|
|
60
|
-
reason: MicrophoneFailureReason;
|
|
61
|
-
/** What the browser reported, verbatim. Show it — it is often exact. */
|
|
62
|
-
message: string;
|
|
63
|
-
/** What the person should actually do about it. */
|
|
64
|
-
hint: string;
|
|
65
|
-
};
|
|
66
|
-
interface EnableMicrophoneOptions {
|
|
67
|
-
/**
|
|
68
|
-
* How long to wait before calling it a no-answer. Default 15s.
|
|
69
|
-
*
|
|
70
|
-
* This is not belt-and-braces. Per the getUserMedia specification a prompt the user never
|
|
71
|
-
* dismisses may leave the promise permanently pending — "neither resolve nor reject" — and
|
|
72
|
-
* a page that awaits it in its connect path hangs on a state that has no error and no end.
|
|
73
|
-
* A deadline turns that into a reportable outcome.
|
|
74
|
-
*/
|
|
75
|
-
timeoutMs?: number;
|
|
76
|
-
}
|
|
77
|
-
/**
|
|
78
|
-
* Publish the caller's microphone, and report WHY if it does not start.
|
|
79
|
-
*
|
|
80
|
-
* ```ts
|
|
81
|
-
* const mic = await enableMicrophone(room);
|
|
82
|
-
* if (!mic.ok) {
|
|
83
|
-
* status.textContent = `Microphone: ${mic.message}`;
|
|
84
|
-
* help.textContent = mic.hint; // the sentence that names the fix
|
|
85
|
-
* }
|
|
86
|
-
* ```
|
|
87
|
-
*
|
|
88
|
-
* Never throws. A call that cannot start the microphone is an ordinary outcome of asking a
|
|
89
|
-
* person for a device, not a programmer error, so it belongs in the return type.
|
|
90
|
-
*
|
|
91
|
-
* Note it does NOT hang up on failure — the call is still live and still billing, and only
|
|
92
|
-
* the caller knows whether a text-only session is useful to them. If it is not, disconnect
|
|
93
|
-
* in the `!ok` branch; leaving a room connected against a device the user never granted is
|
|
94
|
-
* how the retry ends up contending with the call that is still holding it.
|
|
95
|
-
*/
|
|
96
|
-
declare function enableMicrophone(room: MicrophoneCapableRoom, options?: EnableMicrophoneOptions): Promise<MicrophoneResult>;
|
|
97
|
-
|
|
98
13
|
/**
|
|
99
14
|
* Making the character AUDIBLE — the half of "audio works" that has nothing to do with the
|
|
100
15
|
* microphone.
|
|
@@ -251,4 +166,4 @@ type BoomerangPlayback = {
|
|
|
251
166
|
*/
|
|
252
167
|
declare function attachBoomerangPlayback(video: HTMLVideoElement, options?: BoomerangPlaybackOptions): BoomerangPlayback;
|
|
253
168
|
|
|
254
|
-
export { type AttachRemoteAudioOptions, type AttachableTrack, type AudioCapableRoom, type BoomerangPlayback, type BoomerangPlaybackOptions, type
|
|
169
|
+
export { type AttachRemoteAudioOptions, type AttachableTrack, type AudioCapableRoom, type BoomerangPlayback, type BoomerangPlaybackOptions, type PreparableParticipant, type PreparableRoom, type PrepareAvatarRoomOptions, type PreparedAvatarRoom, type RemoteAudioAttachment, type VideoSupport, attachBoomerangPlayback, attachRemoteAudio, checkVideoSupport, prepareAvatarRoom };
|
package/dist/browser.js
CHANGED
|
@@ -18,6 +18,7 @@ var HINTS = {
|
|
|
18
18
|
"denied-by-os": "The operating system is blocking the browser itself, so no site can record. On macOS: System Settings > Privacy & Security > Microphone, enable your browser, then RESTART it \u2014 the change does not apply to a running browser.",
|
|
19
19
|
"no-device": "No microphone is available. Connect one, and if you are on Bluetooth headphones check they are in a mode that has a microphone.",
|
|
20
20
|
"device-in-use": "Another application or browser tab is holding the microphone. Close it \u2014 including any call from this page you did not hang up \u2014 and try again.",
|
|
21
|
+
"device-lost": "The microphone stopped during the call: it was unplugged, or access to it was turned off. Reconnect it or allow access again, then turn the microphone back on.",
|
|
21
22
|
unknown: "Try again, and if it persists include the message above in your report."
|
|
22
23
|
};
|
|
23
24
|
function isOperatingSystemDenial(message) {
|
|
@@ -54,6 +55,10 @@ function classify(error) {
|
|
|
54
55
|
function fail(reason, message) {
|
|
55
56
|
return { ok: false, reason, message, hint: HINTS[reason] };
|
|
56
57
|
}
|
|
58
|
+
function describeMicrophoneFailure(error) {
|
|
59
|
+
const { reason, message } = classify(error);
|
|
60
|
+
return { reason, message, hint: HINTS[reason] };
|
|
61
|
+
}
|
|
57
62
|
async function enableMicrophone(room, options = {}) {
|
|
58
63
|
const timeoutMs = options.timeoutMs ?? 15e3;
|
|
59
64
|
if (typeof navigator === "undefined" || !navigator.mediaDevices) {
|
|
@@ -253,4 +258,4 @@ function now() {
|
|
|
253
258
|
return typeof performance !== "undefined" ? performance.now() : Date.now();
|
|
254
259
|
}
|
|
255
260
|
|
|
256
|
-
export { DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS, applyAvatarPlayoutDelay, applyPlayoutDelay, attachBoomerangPlayback, attachRemoteAudio, checkVideoSupport, enableMicrophone, prepareAvatarRoom };
|
|
261
|
+
export { DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS, applyAvatarPlayoutDelay, applyPlayoutDelay, attachBoomerangPlayback, attachRemoteAudio, checkVideoSupport, describeMicrophoneFailure, enableMicrophone, prepareAvatarRoom };
|
|
@@ -654,7 +654,7 @@ z.object({
|
|
|
654
654
|
queue_position: z.number().int().positive().optional(),
|
|
655
655
|
recommended_retry_ms: z.number().int().nonnegative()
|
|
656
656
|
}).strict();
|
|
657
|
-
z.enum([
|
|
657
|
+
var liveKitSessionReleaseReasonSchema = z.enum([
|
|
658
658
|
"page_hide",
|
|
659
659
|
"disconnected",
|
|
660
660
|
"superseded",
|
|
@@ -665,7 +665,7 @@ z.enum([
|
|
|
665
665
|
|
|
666
666
|
// ../http-client/src/client.ts
|
|
667
667
|
var DEFAULT_BASE_URL = "https://realtimeavatar.ai/api/v1";
|
|
668
|
-
var SDK_VERSION = "0.
|
|
668
|
+
var SDK_VERSION = "0.26.0";
|
|
669
669
|
var RealtimeAvatar = class {
|
|
670
670
|
#apiKey;
|
|
671
671
|
#baseUrl;
|
|
@@ -777,7 +777,25 @@ var RealtimeAvatar = class {
|
|
|
777
777
|
*/
|
|
778
778
|
async endCall(sessionId, options = {}) {
|
|
779
779
|
if (!sessionId) return false;
|
|
780
|
-
|
|
780
|
+
return this.#release({ session_id: sessionId }, options);
|
|
781
|
+
}
|
|
782
|
+
/**
|
|
783
|
+
* Give up a QUEUED call's place in line, by the `queueTicketId` that `startCall` returned.
|
|
784
|
+
*
|
|
785
|
+
* A queued call has no session yet, so {@link endCall} cannot reach it: the release contract
|
|
786
|
+
* carries the ticket in its own field, and a ticket sent as a session id names nothing and is
|
|
787
|
+
* acknowledged as an idempotent no-op. The place is then held until its TTL, at the front of
|
|
788
|
+
* the queue, in front of everyone behind it.
|
|
789
|
+
*
|
|
790
|
+
* Same contract as `endCall`: best-effort, `true` only when the platform acknowledged it,
|
|
791
|
+
* never a throw, and only for tickets YOUR SERVER was handed.
|
|
792
|
+
*/
|
|
793
|
+
async leaveQueue(queueTicketId, options = {}) {
|
|
794
|
+
if (!queueTicketId) return false;
|
|
795
|
+
return this.#release({ queue_ticket_id: queueTicketId }, options);
|
|
796
|
+
}
|
|
797
|
+
async #release(target, options) {
|
|
798
|
+
const body = { ...target };
|
|
781
799
|
if (options.reason !== void 0) body.reason = options.reason;
|
|
782
800
|
if (options.capacityPool !== void 0) body.capacity_pool = options.capacityPool;
|
|
783
801
|
try {
|
|
@@ -1281,4 +1299,4 @@ function isQueued(result) {
|
|
|
1281
1299
|
return "queued" in result;
|
|
1282
1300
|
}
|
|
1283
1301
|
|
|
1284
|
-
export { ROUTE_TIMEOUT_MS, RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, verifyTranscript };
|
|
1302
|
+
export { ROUTE_TIMEOUT_MS, RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, liveKitSessionReleaseReasonSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, sessionModeSchema, verifyTranscript };
|
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import { RealtimeAvatar, ROUTE_TIMEOUT_MS, isQueued, RealtimeAvatarHttpError, RealtimeAvatarError } from './chunk-
|
|
1
|
+
import { sessionModeSchema, liveKitSessionReleaseReasonSchema, RealtimeAvatar, ROUTE_TIMEOUT_MS, isQueued, RealtimeAvatarHttpError, RealtimeAvatarError } from './chunk-HMESE54B.js';
|
|
2
|
+
import { z } from 'zod';
|
|
2
3
|
|
|
3
4
|
// ../http-client/src/generated/error-semantics.ts
|
|
4
5
|
var ERROR_SEMANTICS = {
|
|
@@ -38,6 +39,17 @@ var MESSAGE_SHAPE_PATTERNS = { "machine": { "source": "[_=<>{}\\[\\]|\\\\$`]|\\w
|
|
|
38
39
|
Object.values(MESSAGE_SHAPE_PATTERNS).map(
|
|
39
40
|
(pattern) => new RegExp(pattern.source, pattern.flags)
|
|
40
41
|
);
|
|
42
|
+
var proxyConnectRequestSchema = z.object({
|
|
43
|
+
avatarId: z.string().min(1),
|
|
44
|
+
mode: sessionModeSchema.optional()
|
|
45
|
+
});
|
|
46
|
+
var proxyEndRequestSchema = z.object({
|
|
47
|
+
session_id: z.string().min(1).optional(),
|
|
48
|
+
queue_ticket_id: z.string().min(1).optional(),
|
|
49
|
+
reason: liveKitSessionReleaseReasonSchema.optional().catch(void 0)
|
|
50
|
+
}).strict().refine((body) => body.session_id !== void 0 || body.queue_ticket_id !== void 0, {
|
|
51
|
+
message: "session_id or queue_ticket_id is required"
|
|
52
|
+
});
|
|
41
53
|
|
|
42
54
|
// ../proxy/src/config.ts
|
|
43
55
|
var MISCONFIGURED = "internal_error";
|
|
@@ -97,19 +109,25 @@ function createProxyHandler(config) {
|
|
|
97
109
|
try {
|
|
98
110
|
if (operation === "avatars") return json({ data: await rta.listAvatars() });
|
|
99
111
|
if (operation === "credits") return json(await rta.creditBalance());
|
|
112
|
+
const raw = await request.json().catch(() => void 0);
|
|
100
113
|
if (operation === "end") {
|
|
101
|
-
const
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
const owns = config.ownsSession ? await config.ownsSession({ request, sessionId }) : minted.delete(sessionId);
|
|
105
|
-
if (!owns) return new Response(null, { status: 204 });
|
|
114
|
+
const parsed2 = proxyEndRequestSchema.safeParse(raw);
|
|
115
|
+
if (!parsed2.success) return json({ error: "session_id or queue_ticket_id is required" }, 422);
|
|
116
|
+
const ended = parsed2.data;
|
|
106
117
|
const reason = ended.reason === "page_hide" || ended.reason === "unmount" ? ended.reason : "manual";
|
|
107
|
-
await
|
|
118
|
+
const owned = async (id) => config.ownsSession ? await config.ownsSession({ request, sessionId: id }) : minted.delete(id);
|
|
119
|
+
if (ended.session_id !== void 0 && await owned(ended.session_id)) {
|
|
120
|
+
await rta.endCall(ended.session_id, { reason });
|
|
121
|
+
}
|
|
122
|
+
if (ended.queue_ticket_id !== void 0 && await owned(ended.queue_ticket_id)) {
|
|
123
|
+
await rta.leaveQueue(ended.queue_ticket_id, { reason });
|
|
124
|
+
}
|
|
108
125
|
return new Response(null, { status: 204 });
|
|
109
126
|
}
|
|
110
|
-
const
|
|
111
|
-
if (!
|
|
112
|
-
const
|
|
127
|
+
const parsed = proxyConnectRequestSchema.safeParse(raw);
|
|
128
|
+
if (!parsed.success) return json({ error: "avatarId is required" }, 422);
|
|
129
|
+
const body = parsed.data;
|
|
130
|
+
const mode = body.mode ?? "avatar";
|
|
113
131
|
const decided = await config.session?.({ request, avatarId: body.avatarId, mode });
|
|
114
132
|
if (decided instanceof Response) return decided;
|
|
115
133
|
const call = await rta.startCall({ avatarId: body.avatarId, mode, ...decided ?? {} });
|
package/dist/express.js
CHANGED
package/dist/hono.js
CHANGED
package/dist/index.d.ts
CHANGED
|
@@ -82,6 +82,18 @@ declare class RealtimeAvatar {
|
|
|
82
82
|
* request body lets any visitor hang up any call on your account.
|
|
83
83
|
*/
|
|
84
84
|
endCall(sessionId: string, options?: EndCallOptions): Promise<boolean>;
|
|
85
|
+
/**
|
|
86
|
+
* Give up a QUEUED call's place in line, by the `queueTicketId` that `startCall` returned.
|
|
87
|
+
*
|
|
88
|
+
* A queued call has no session yet, so {@link endCall} cannot reach it: the release contract
|
|
89
|
+
* carries the ticket in its own field, and a ticket sent as a session id names nothing and is
|
|
90
|
+
* acknowledged as an idempotent no-op. The place is then held until its TTL, at the front of
|
|
91
|
+
* the queue, in front of everyone behind it.
|
|
92
|
+
*
|
|
93
|
+
* Same contract as `endCall`: best-effort, `true` only when the platform acknowledged it,
|
|
94
|
+
* never a throw, and only for tickets YOUR SERVER was handed.
|
|
95
|
+
*/
|
|
96
|
+
leaveQueue(queueTicketId: string, options?: Pick<EndCallOptions, "reason">): Promise<boolean>;
|
|
85
97
|
/**
|
|
86
98
|
* Register a character from a looping clip you host.
|
|
87
99
|
*
|
package/dist/index.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
export { RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, verifyTranscript } from './chunk-
|
|
1
|
+
export { RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, verifyTranscript } from './chunk-HMESE54B.js';
|
|
2
2
|
import './chunk-MCIXU6XP.js';
|
package/dist/nextjs.js
CHANGED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Turning the microphone on, and saying what went wrong when it does not.
|
|
3
|
+
*
|
|
4
|
+
* WHY THIS EXISTS. `room.localParticipant.setMicrophoneEnabled(true)` is one line, so every
|
|
5
|
+
* integration writes it inline and none of them handle its rejection — the reject path is
|
|
6
|
+
* invisible until a real user is on a real machine. When it fails there is nothing to see:
|
|
7
|
+
* the call is already connected, the page is mid-`await`, and the rejection becomes an
|
|
8
|
+
* unhandled promise nobody is looking at. The reported symptom is always the same sentence —
|
|
9
|
+
* "the mic won't start" — and it covers at least six unrelated causes with different fixes,
|
|
10
|
+
* one of which is not in the browser at all.
|
|
11
|
+
*
|
|
12
|
+
* So this returns a RESULT rather than throwing: the cause becomes a value the caller has to
|
|
13
|
+
* handle, and each one carries the sentence that names its actual fix. `message` is what the
|
|
14
|
+
* browser said; `hint` is what the person should do.
|
|
15
|
+
*
|
|
16
|
+
* It is deliberately structural — it never imports `livekit-client`, so it pins no version
|
|
17
|
+
* and adds no bytes. Anything with a `localParticipant.setMicrophoneEnabled` satisfies it.
|
|
18
|
+
*/
|
|
19
|
+
/** The sliver of livekit-client's `Room` this needs. Structural on purpose — see above. */
|
|
20
|
+
interface MicrophoneCapableRoom {
|
|
21
|
+
localParticipant: {
|
|
22
|
+
setMicrophoneEnabled(enabled: boolean): Promise<unknown>;
|
|
23
|
+
};
|
|
24
|
+
}
|
|
25
|
+
/**
|
|
26
|
+
* Why the microphone did not start. Every variant has a DIFFERENT fix, which is the whole
|
|
27
|
+
* reason this is an enum and not a boolean.
|
|
28
|
+
*/
|
|
29
|
+
type MicrophoneFailureReason =
|
|
30
|
+
/** No `navigator.mediaDevices` — the page is not a secure context. */
|
|
31
|
+
"insecure-origin"
|
|
32
|
+
/** The permission prompt was never answered. getUserMedia may never settle; we time out. */
|
|
33
|
+
| "no-answer"
|
|
34
|
+
/** The BROWSER denied it — the site's own permission, resettable from the address bar. */
|
|
35
|
+
| "denied-by-browser"
|
|
36
|
+
/** The OS denied it — macOS System Settings, nothing the page or the site can change. */
|
|
37
|
+
| "denied-by-os"
|
|
38
|
+
/** No input device is connected, or none matches the constraints. */
|
|
39
|
+
| "no-device"
|
|
40
|
+
/** A device exists but something else holds it — another tab, app, or a stale call. */
|
|
41
|
+
| "device-in-use"
|
|
42
|
+
/** Anything unrecognised. `message` still carries what the browser said. */
|
|
43
|
+
| "unknown";
|
|
44
|
+
type MicrophoneResult = {
|
|
45
|
+
ok: true;
|
|
46
|
+
} | {
|
|
47
|
+
ok: false;
|
|
48
|
+
reason: MicrophoneFailureReason;
|
|
49
|
+
/** What the browser reported, verbatim. Show it — it is often exact. */
|
|
50
|
+
message: string;
|
|
51
|
+
/** What the person should actually do about it. */
|
|
52
|
+
hint: string;
|
|
53
|
+
};
|
|
54
|
+
interface EnableMicrophoneOptions {
|
|
55
|
+
/**
|
|
56
|
+
* How long to wait before calling it a no-answer. Default 15s.
|
|
57
|
+
*
|
|
58
|
+
* This is not belt-and-braces. Per the getUserMedia specification a prompt the user never
|
|
59
|
+
* dismisses may leave the promise permanently pending — "neither resolve nor reject" — and
|
|
60
|
+
* a page that awaits it in its connect path hangs on a state that has no error and no end.
|
|
61
|
+
* A deadline turns that into a reportable outcome.
|
|
62
|
+
*/
|
|
63
|
+
timeoutMs?: number;
|
|
64
|
+
}
|
|
65
|
+
/**
|
|
66
|
+
* Why a microphone that was WORKING stopped: the device was unplugged, or access was revoked
|
|
67
|
+
* mid-call. LiveKit tries the default device first and mutes the track when that fails too.
|
|
68
|
+
* Only a live call can report this; `enableMicrophone` never does.
|
|
69
|
+
*/
|
|
70
|
+
type MicrophoneProblemReason = MicrophoneFailureReason | "device-lost";
|
|
71
|
+
/** A microphone problem as a value: the cause, what the browser said, and what to do. */
|
|
72
|
+
type MicrophoneProblem = {
|
|
73
|
+
reason: MicrophoneProblemReason;
|
|
74
|
+
message: string;
|
|
75
|
+
hint: string;
|
|
76
|
+
};
|
|
77
|
+
/**
|
|
78
|
+
* Describe a microphone failure, whoever caught it. The one classifier behind both
|
|
79
|
+
* {@link enableMicrophone} and the React call handle's `microphone` state, so the two can never
|
|
80
|
+
* give the same failure different advice.
|
|
81
|
+
*/
|
|
82
|
+
declare function describeMicrophoneFailure(error: unknown): MicrophoneProblem;
|
|
83
|
+
/**
|
|
84
|
+
* Publish the caller's microphone, and report WHY if it does not start.
|
|
85
|
+
*
|
|
86
|
+
* ```ts
|
|
87
|
+
* const mic = await enableMicrophone(room);
|
|
88
|
+
* if (!mic.ok) {
|
|
89
|
+
* status.textContent = `Microphone: ${mic.message}`;
|
|
90
|
+
* help.textContent = mic.hint; // the sentence that names the fix
|
|
91
|
+
* }
|
|
92
|
+
* ```
|
|
93
|
+
*
|
|
94
|
+
* Never throws. A call that cannot start the microphone is an ordinary outcome of asking a
|
|
95
|
+
* person for a device, not a programmer error, so it belongs in the return type.
|
|
96
|
+
*
|
|
97
|
+
* Note it does NOT hang up on failure — the call is still live and still billing, and only
|
|
98
|
+
* the caller knows whether a text-only session is useful to them. If it is not, disconnect
|
|
99
|
+
* in the `!ok` branch; leaving a room connected against a device the user never granted is
|
|
100
|
+
* how the retry ends up contending with the call that is still holding it.
|
|
101
|
+
*/
|
|
102
|
+
declare function enableMicrophone(room: MicrophoneCapableRoom, options?: EnableMicrophoneOptions): Promise<MicrophoneResult>;
|
|
103
|
+
|
|
104
|
+
/**
|
|
105
|
+
* Receiver-side playout delay (SECONDS) for the avatar media tracks.
|
|
106
|
+
*
|
|
107
|
+
* The avatar is published from a GPU datacenter and reaches viewers over the
|
|
108
|
+
* public internet, where a few percent of packet loss is normal. A shallow
|
|
109
|
+
* receiver buffer stalls (freezes) on every loss while it waits for a
|
|
110
|
+
* retransmit; a deeper buffer recovers the lost packets BEFORE playout, so the
|
|
111
|
+
* viewer sees smooth 25fps instead of freezing. Measured at ~5% loss this took
|
|
112
|
+
* the stream from ~11fps with multi-second freezes to a steady 25fps with zero
|
|
113
|
+
* freezes.
|
|
114
|
+
*
|
|
115
|
+
* 0.5s is applied via the native `RemoteTrack.setPlayoutDelay`, which sets the
|
|
116
|
+
* receiver's `playoutDelayHint` — the same Chromium jitter-buffer knob the old
|
|
117
|
+
* hand-rolled `jitterBufferTarget` reached into (the spec renamed
|
|
118
|
+
* playoutDelayHint→jitterBufferTarget; both influence the same buffer depth),
|
|
119
|
+
* but typed and SDK-owned so there is no cast or feature-probe. The hint is NOT
|
|
120
|
+
* free at the open: the receiver holds the FIRST video frame toward the target
|
|
121
|
+
* (~250-450ms of measured TTFF), a deliberate cost paid once for the freeze-free
|
|
122
|
+
* steady state above. The open-small-then-ramp variant that tried to dodge that
|
|
123
|
+
* cost was shipped, failed twice (the rVFC ramp trigger is throttled on the
|
|
124
|
+
* opacity:0-mounted element, stranding the shallow buffer), and was deliberately
|
|
125
|
+
* removed in 2262e4c7b — do not re-propose it. No-op on browsers that don't
|
|
126
|
+
* support the hint (the SDK warns and moves on).
|
|
127
|
+
*
|
|
128
|
+
* This lives in the vanilla browser package, not the React one, because the
|
|
129
|
+
* React surface applies it for you and a plain-DOM page has to be given it.
|
|
130
|
+
* Until this file existed the helper was only reachable through
|
|
131
|
+
* `realtime-avatar/react`, so every plain-DOM adopter — every one of this repo's
|
|
132
|
+
* own demos included — ran with the browser's default shallow buffer and froze on
|
|
133
|
+
* every lost packet. Nobody should have to know this knob exists: `prepareAvatarRoom`
|
|
134
|
+
* (and `attachRemoteAudio`, which calls it) applies it to every track the room
|
|
135
|
+
* subscribes, and `RealtimeAvatarLiveKitRoom` does the same for React. This file is
|
|
136
|
+
* the one place the number and its reasoning live.
|
|
137
|
+
* Structural like its siblings: no `livekit-client` import, no version pin.
|
|
138
|
+
*/
|
|
139
|
+
declare const DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS = 0.5;
|
|
140
|
+
/**
|
|
141
|
+
* A track that can take a native playout-delay hint — livekit-client's subscribed
|
|
142
|
+
* `RemoteTrack`. Duck-typed on the method rather than `instanceof`, so a
|
|
143
|
+
* placeholder or local track is skipped and the helper is testable with a fake.
|
|
144
|
+
*/
|
|
145
|
+
interface PlayoutDelayTarget {
|
|
146
|
+
setPlayoutDelay(delayInSeconds: number): void;
|
|
147
|
+
}
|
|
148
|
+
/**
|
|
149
|
+
* Apply the receiver playout delay to ONE subscribed track.
|
|
150
|
+
*
|
|
151
|
+
* Call it for every track `RoomEvent.TrackSubscribed` hands you, audio and video
|
|
152
|
+
* alike, before or after `attach()` — WebRTC syncs the pair to the LARGER hint,
|
|
153
|
+
* so applying it to one stream and not the other lets the lips lag the voice.
|
|
154
|
+
* Returns `true` when the hint was applied and `false` when the object cannot take
|
|
155
|
+
* one (a local track, a placeholder, a not-yet-subscribed publication), so a
|
|
156
|
+
* caller can log the miss without a try/catch. The delay is floored at 0 so a
|
|
157
|
+
* negative input can never throw.
|
|
158
|
+
*/
|
|
159
|
+
declare function applyPlayoutDelay(track: unknown, delaySeconds?: number): boolean;
|
|
160
|
+
/**
|
|
161
|
+
* Apply the SAME native playout delay to the avatar's audio + video tracks
|
|
162
|
+
* (kept equal so the a/v sync stays lip-locked). Pure and exported so the clamp
|
|
163
|
+
* and the both-tracks-equal invariant are unit-testable without a DOM; the React
|
|
164
|
+
* hook `useAvatarPlayoutDelay` is a thin effect wrapper around this.
|
|
165
|
+
*/
|
|
166
|
+
declare function applyAvatarPlayoutDelay(videoTrack: unknown, audioTrack: unknown, delaySeconds?: number): void;
|
|
167
|
+
|
|
168
|
+
export { DEFAULT_AVATAR_PLAYOUT_DELAY_SECONDS as D, type EnableMicrophoneOptions as E, type MicrophoneCapableRoom as M, type PlayoutDelayTarget as P, type MicrophoneFailureReason as a, type MicrophoneProblem as b, type MicrophoneProblemReason as c, type MicrophoneResult as d, applyAvatarPlayoutDelay as e, applyPlayoutDelay as f, describeMicrophoneFailure as g, enableMicrophone as h };
|