realtime-avatar 0.22.0 → 0.24.0

package/README.md

@@ -25,6 +25,47 @@ return call.raw;   // relay to the browser byte-for-byte
 
 That is the whole server half. The client joins with the payload and renders her.
 
+## React Native compatibility
+
+The next release requires **LiveKit React Native 3** for `/react-native`; it does not claim
+compatibility with LiveKit React Native 2. Install the tested group together:
+
+```bash
+npm install --save-exact @livekit/react-native@3.0.0 @livekit/react-native-webrtc@144.2.0 livekit-client@2.22.3 @livekit/components-react@2.9.21
+```
+
+`@livekit/components-react@2.9.21` retains `@livekit/components-core@0.12.13`; both satisfy the
+native 3.0.0 peer contracts. Keep a single resolved components-react instance: the native room
+provides its context and the SDK's shared hooks consume that same context. SDK checks use React
+19.2.7 and React Native 0.81.5. The web binding also uses client 2.22.3; no components upgrade or
+stream-handling patch is required for this tuple. Peer ranges allow compatible updates, but only
+this exact tuple has been checked here. Client 2.22.3 introduces `machina@7.0.1`, whose Node
+engine is `>=22.22`; use Node 22.22 or newer for client/native tooling (verified on 22.23.1).
+This does not change the dependency-free server entry's Node floor. Node 20 tooling is not
+verified for the new LiveKit group.
+
+Call the SDK's re-exported `registerGlobals()` once at app startup, before creating rooms. RN3
+configures iOS audio natively by default; do not reintroduce the deprecated JavaScript audio
+management callback. `RealtimeAvatarLiveKitRoom` retains the existing `AudioSession.startAudioSession`
+and `stopAudioSession` lifecycle while a grant and `connect` intent are present. Apps that own audio
+configuration or coordinate overlapping calls must pass `manageAudioSession={false}` and own that
+lifecycle; the hook is not a shared audio-session lease. Camera publication still requires both the
+server grant's permission and the caller's opt-in. Room callbacks/options continue to use LiveKit's
+public types and implementation; no separate connection state machine is introduced.
+
+**This is a native build upgrade, not an OTA-only change.** Rebuild iOS and Android binaries with
+RN3/WebRTC 144.2.0 before shipping JavaScript that requires them. Retain the previous SDK group for
+installed binaries that still use RN2, and isolate their OTA runtime/channel appropriately.
+
+The packaged-entry regression suite mounts the built `/react-native` adapter with the installed
+RN3 room and audio implementations, real React/LiveKit hooks, and shared contexts. It checks room
+props, microphone/camera policy, callbacks, start/stop intent, redial, and listener cleanup. It mocks
+native module calls, network connection and media capture; it does **not** prove native binary
+linking, `registerGlobals`, RTCView rendering, permissions, Bluetooth/speaker routing, interruption
+recovery, asynchronous OS audio teardown, or overlapping-room ownership. Test these on iOS and
+Android devices, including repeated call/redial and background/foreground transitions, before a
+consumer release.
+
 ## Optional call recordings
 
 Your server decides whether to record after your application obtains consent:
@@ -143,6 +184,32 @@ Two facts follow from that picture and drive everything else:
 
 ---
 
+## Input source on room messages
+
+On React and React Native, `session.sendTurn(text)` automatically observes text.
+For already recognized speech, bind
+`session.createTranscriptSender({ inputSource: "client_stt" })` once per session.
+The adapter defaults to `client_stt`; it captures its own default, and accepts a
+per-send `{ inputSource: "text" }` override. `sendTurn` accepts the same optional
+declaration. Only `text` and `client_stt` are valid client declarations. No speech
+recognition engine is included. `instructions` still work on either sender.
+
+`retryTurn()` preserves the resolved source and declaration scope after a timeout,
+with a new `turn_id` and `retry_of_turn_id` pointing to the previous attempt.
+Room consumers receive the attributes through the existing `lk.chat` text stream.
+`useChat().chatMessages` exposes them as `message.attributes`; an imperative receiver
+can read `reader.info.attributes` in `registerTextStreamHandler("lk.chat", handler)`.
+The observation is `rta.observed_input_source`; the optional declaration and scope
+are `rta.declared_input_source` and `rta.input_source_declaration_scope`. Missing
+legacy attributes remain unknown. Use one handler owner per topic; a `useChat`
+consumer should not register a duplicate raw handler on the same room.
+
+This path needs no inference or platform changes. The consumer must already be
+connected to the room, and text-stream messages are not durable webhook deliveries.
+The legacy `lk-chat-topic` compatibility path omits attributes. Automatic RTA server
+STT attribution and final transcript webhook provenance are separate work. Existing
+webhook types and behavior are unchanged.
+
 ## API
 
 Everything is on one class. The full types are in
@@ -359,3 +426,34 @@ for building your own queue UI, and the zod schemas `sessionBehaviorSchema` / `s
 - The API is versioned at `/api/v1`; breaking changes get a new version, not a silent edit.
 
 MIT licensed.
+
+
+### Continuous participant recordings
+
+Use `recording: "participants"` on your server to retain the two participants
+separately. A camera-enabled call has four logical media tracks in two MP4 files:
+user camera/microphone and character video/voice. Each file is continuous through
+camera mute, unpublish and republish; the user's microphone continues while their
+camera is off. `camera: true` is still a separate publication permission, and
+recording alone never opens a camera. Voice-only participants get audio-only MP4s.
+
+The returned `call.recordings` contains both pending artifacts, while legacy
+modes retain `call.recording`. Once finalized, use `listRecordings({ sessionId })`
+and `getRecordingAccess(recordingId)` to retrieve authorized, renewable URLs.
+`recording.participant.role` identifies the user or character; media timestamps
+align the files without fixing their visual layout. Keep failed/missing files
+visible instead of treating the surviving participant as a complete recording.
+
+```tsx
+import { RecordingPlayer } from "realtime-avatar/react";
+
+// assets comes from your authenticated application endpoint.
+// Each item is { recording, url }; refresh expiring URLs through your server.
+<RecordingPlayer assets={assets} />
+```
+
+The player provides one play/pause control and one seek bar for both files. It
+plays each file's audio once, preserves start offsets, waits for buffering and
+refuses to guess synchronization when media timestamps are missing. Original
+files remain independently playable and editable. Participant departure ends
+that file; a new call is a new session, not an automatic concatenation.

package/dist/{chunk-T3RZSVRI.js → chunk-FUJI3KIH.js}

@@ -1,4 +1,4 @@
-import { RealtimeAvatar, isQueued, RealtimeAvatarHttpError } from './chunk-N76M4MEU.js';
+import { RealtimeAvatar, isQueued, RealtimeAvatarHttpError } from './chunk-ZXPLFSG7.js';
 
 // ../proxy/src/config.ts
 var json = (body, status = 200) => new Response(JSON.stringify(body), {

package/dist/{chunk-N76M4MEU.js → chunk-ZXPLFSG7.js}

@@ -234,15 +234,22 @@ var schema26 = z.strictObject({
 var clipLibraryResponseSchema = schema24;
 var clipLibraryUpdateSchema = schema26;
 var RECORDED_MEDIA_MODES = ["audio", "video", "audio_video"];
-var recordingModeSchema = z.enum(["off", ...RECORDED_MEDIA_MODES]).describe("Server-owned recording policy; omitted means off. Audio includes user and avatar audio.");
+var recordingModeSchema = z.enum(["off", ...RECORDED_MEDIA_MODES, "participants"]).describe("Server-owned recording policy; omitted means off. Legacy audio includes user and avatar audio. participants saves one continuous audio/video file per participant, without mixing their voices.");
 var RECORDING_STATUSES = ["pending", "recording", "processing", "ready", "failed", "expired"];
 z.enum(RECORDING_STATUSES);
+var recordingParticipantSchema = z.object({
+  participantIdentity: z.string().min(1).max(160),
+  role: z.enum(["user", "avatar"])
+}).strict();
 var recordingMetadataSchema = z.object({
   sessionId: z.string().min(1),
   recordingId: z.string().min(1),
-  mode: recordingModeSchema.exclude(["off"]),
+  mode: z.enum(RECORDED_MEDIA_MODES),
   createdAt: z.string().datetime({ offset: true }),
-  retainedUntil: z.string().datetime({ offset: true })
+  retainedUntil: z.string().datetime({ offset: true }),
+  participant: recordingParticipantSchema.optional(),
+  mediaStartedAt: z.string().datetime({ offset: true }).optional(),
+  mediaEndedAt: z.string().datetime({ offset: true }).optional()
 });
 var recordingArtifactSchema = z.discriminatedUnion("status", [
   recordingMetadataSchema.extend({ status: z.enum(["pending", "recording", "processing", "expired"]) }).strict(),
@@ -548,6 +555,8 @@ z.object({
 }).strict();
 var liveKitSessionGrantSchema = z.object({
   recording: recordingArtifactSchema.optional(),
+  recording_mode: recordingModeSchema.optional(),
+  recordings: z.array(recordingArtifactSchema).max(2).optional(),
   connection_history: connectionHistoryGrantSchema.optional(),
   status: z.literal("ready").default("ready"),
   session_id: z.string().min(1),
@@ -645,7 +654,7 @@ z.enum([
 
 // ../http-client/src/client.ts
 var DEFAULT_BASE_URL = "https://realtimeavatar.ai/api/v1";
-var SDK_VERSION = "0.22.0";
+var SDK_VERSION = "0.24.0";
 var RealtimeAvatar = class {
   #apiKey;
   #baseUrl;
@@ -718,6 +727,8 @@ var RealtimeAvatar = class {
     return {
       status: "ready",
       ...grant.recording === void 0 ? {} : { recording: grant.recording },
+      ...grant.recording_mode === void 0 ? {} : { recordingMode: grant.recording_mode },
+      ...grant.recordings === void 0 ? {} : { recordings: grant.recordings },
       sessionId: grant.session_id,
       roomName: grant.room_name,
       livekitUrl: grant.livekit_url,

package/dist/express.d.ts

@@ -1,5 +1,5 @@
-import { P as ProxyConfig } from './types-k6pW-wKA.js';
-import './types-Cmto2uGs.js';
+import { P as ProxyConfig } from './types-CpqMHna3.js';
+import './types-D77jgSms.js';
 import 'zod';
 
 type Expressish = {

package/dist/express.js

@@ -1,5 +1,5 @@
-import { createProxyHandler } from './chunk-T3RZSVRI.js';
-import './chunk-N76M4MEU.js';
+import { createProxyHandler } from './chunk-FUJI3KIH.js';
+import './chunk-ZXPLFSG7.js';
 import './chunk-MCIXU6XP.js';
 
 // ../proxy/src/express.ts

package/dist/hono.d.ts

@@ -1,5 +1,5 @@
-import { P as ProxyConfig } from './types-k6pW-wKA.js';
-import './types-Cmto2uGs.js';
+import { P as ProxyConfig } from './types-CpqMHna3.js';
+import './types-D77jgSms.js';
 import 'zod';
 
 /**

package/dist/hono.js

@@ -1,5 +1,5 @@
-import { createProxyHandler } from './chunk-T3RZSVRI.js';
-import './chunk-N76M4MEU.js';
+import { createProxyHandler } from './chunk-FUJI3KIH.js';
+import './chunk-ZXPLFSG7.js';
 import './chunk-MCIXU6XP.js';
 
 // ../proxy/src/hono.ts

package/dist/index.d.ts

@@ -1,5 +1,5 @@
-import { a as CallPolicy, C as CallMode, S as StartCallResult, E as EndCallOptions, A as Avatar, b as AvatarUpdate, c as AvatarSourceSwap, d as ClipLibraryDeclaration, e as ClipLibraryUpdate, L as LoopRedirect, f as ClipLibrary, g as AssetKind, h as Asset, i as ListSessionsOptions, U as UsageSessionPage, j as UsageSession, k as ListRecordingsQuery, l as ListRecordingsResponse, R as RecordingArtifact, m as RecordingAccessResponse, n as CreditBalance, T as TranscriptPayload } from './types-Cmto2uGs.js';
-export { o as AvatarClip, p as CallConnection, q as CallQueued, r as ClipDeclaration, s as ClipLibraryPlan, t as ClipSource, u as ContextMessage, v as EndCallReason, w as RecordingArtifactStatus, x as RecordingMode, V as VideoPolicy, y as isQueued, z as listRecordingsResponseSchema, B as recordingAccessResponseSchema, D as recordingArtifactSchema, F as recordingModeSchema } from './types-Cmto2uGs.js';
+import { a as CallPolicy, C as CallMode, S as StartCallResult, E as EndCallOptions, A as Avatar, b as AvatarUpdate, c as AvatarSourceSwap, d as ClipLibraryDeclaration, e as ClipLibraryUpdate, L as LoopRedirect, f as ClipLibrary, g as AssetKind, h as Asset, i as ListSessionsOptions, U as UsageSessionPage, j as UsageSession, k as ListRecordingsQuery, l as ListRecordingsResponse, R as RecordingArtifact, m as RecordingAccessResponse, n as CreditBalance, T as TranscriptPayload } from './types-D77jgSms.js';
+export { o as AvatarClip, p as CallConnection, q as CallQueued, r as ClipDeclaration, s as ClipLibraryPlan, t as ClipSource, u as ContextMessage, v as EndCallReason, w as RecordingArtifactStatus, x as RecordingMode, V as VideoPolicy, y as isQueued, z as listRecordingsResponseSchema, B as recordingAccessResponseSchema, D as recordingArtifactSchema, F as recordingModeSchema } from './types-D77jgSms.js';
 import { ConnectionHistoryResponse } from './connection-history.js';
 import { z } from 'zod';
 

package/dist/index.js

@@ -1,2 +1,2 @@
-export { RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, verifyTranscript } from './chunk-N76M4MEU.js';
+export { RealtimeAvatar, RealtimeAvatarError, RealtimeAvatarHttpError, clipLibraryDeclarationSchema, isQueued, listRecordingsResponseSchema, recordingAccessResponseSchema, recordingArtifactSchema, recordingModeSchema, verifyTranscript } from './chunk-ZXPLFSG7.js';
 import './chunk-MCIXU6XP.js';

package/dist/nextjs.d.ts

@@ -1,5 +1,5 @@
-import { P as ProxyConfig } from './types-k6pW-wKA.js';
-import './types-Cmto2uGs.js';
+import { P as ProxyConfig } from './types-CpqMHna3.js';
+import './types-D77jgSms.js';
 import 'zod';
 
 /**

package/dist/nextjs.js

@@ -1,5 +1,5 @@
-import { createProxyHandler } from './chunk-T3RZSVRI.js';
-import './chunk-N76M4MEU.js';
+import { createProxyHandler } from './chunk-FUJI3KIH.js';
+import './chunk-ZXPLFSG7.js';
 import './chunk-MCIXU6XP.js';
 
 // ../proxy/src/nextjs.ts

package/dist/{proxy-client-6ndCkiio.d.ts → proxy-client-DlFXaXtn.d.ts}

@@ -135,8 +135,8 @@ declare const liveKitSessionRequestSchema: z.ZodObject<{
     instructions: z.ZodOptional<z.ZodString>;
     initialContext: z.ZodDefault<z.ZodArray<z.ZodObject<{
         role: z.ZodEnum<{
-            system: "system";
             user: "user";
+            system: "system";
             assistant: "assistant";
         }>;
         content: z.ZodString;
@@ -253,6 +253,15 @@ declare const liveKitSessionGrantSchema: z.ZodObject<{
         }>;
         createdAt: z.ZodString;
         retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
         status: z.ZodEnum<{
             pending: "pending";
             recording: "recording";
@@ -269,6 +278,15 @@ declare const liveKitSessionGrantSchema: z.ZodObject<{
         }>;
         createdAt: z.ZodString;
         retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
         status: z.ZodLiteral<"ready">;
         mediaType: z.ZodEnum<{
             "audio/mp4": "audio/mp4";
@@ -286,12 +304,104 @@ declare const liveKitSessionGrantSchema: z.ZodObject<{
         }>;
         createdAt: z.ZodString;
         retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
         status: z.ZodLiteral<"failed">;
         errorCode: z.ZodEnum<{
             recording_failed: "recording_failed";
             recording_unavailable: "recording_unavailable";
         }>;
     }, z.core.$strict>], "status">>;
+    recording_mode: z.ZodOptional<z.ZodEnum<{
+        audio: "audio";
+        video: "video";
+        audio_video: "audio_video";
+        off: "off";
+        participants: "participants";
+    }>>;
+    recordings: z.ZodOptional<z.ZodArray<z.ZodDiscriminatedUnion<[z.ZodObject<{
+        sessionId: z.ZodString;
+        recordingId: z.ZodString;
+        mode: z.ZodEnum<{
+            audio: "audio";
+            video: "video";
+            audio_video: "audio_video";
+        }>;
+        createdAt: z.ZodString;
+        retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
+        status: z.ZodEnum<{
+            pending: "pending";
+            recording: "recording";
+            processing: "processing";
+            expired: "expired";
+        }>;
+    }, z.core.$strict>, z.ZodObject<{
+        sessionId: z.ZodString;
+        recordingId: z.ZodString;
+        mode: z.ZodEnum<{
+            audio: "audio";
+            video: "video";
+            audio_video: "audio_video";
+        }>;
+        createdAt: z.ZodString;
+        retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
+        status: z.ZodLiteral<"ready">;
+        mediaType: z.ZodEnum<{
+            "audio/mp4": "audio/mp4";
+            "video/mp4": "video/mp4";
+        }>;
+        sizeBytes: z.ZodNumber;
+        durationMs: z.ZodNullable<z.ZodNumber>;
+    }, z.core.$strict>, z.ZodObject<{
+        sessionId: z.ZodString;
+        recordingId: z.ZodString;
+        mode: z.ZodEnum<{
+            audio: "audio";
+            video: "video";
+            audio_video: "audio_video";
+        }>;
+        createdAt: z.ZodString;
+        retainedUntil: z.ZodString;
+        participant: z.ZodOptional<z.ZodObject<{
+            participantIdentity: z.ZodString;
+            role: z.ZodEnum<{
+                user: "user";
+                avatar: "avatar";
+            }>;
+        }, z.core.$strict>>;
+        mediaStartedAt: z.ZodOptional<z.ZodString>;
+        mediaEndedAt: z.ZodOptional<z.ZodString>;
+        status: z.ZodLiteral<"failed">;
+        errorCode: z.ZodEnum<{
+            recording_failed: "recording_failed";
+            recording_unavailable: "recording_unavailable";
+        }>;
+    }, z.core.$strict>], "status">>>;
     connection_history: z.ZodOptional<z.ZodObject<{
         endpoint: z.ZodString;
         token: z.ZodString;
@@ -1468,6 +1578,9 @@ type TurnState = "listening" | "thinking" | "speaking" | "quiet";
  */
 declare function mapTurnState(assistantState: string | null | undefined): TurnState;
 
+/** Source an application declares for text sent to an in-room consumer. */
+type DeclaredInputSource = "text" | "client_stt";
+
 /** How early `onApproachingEnd` fires before the hard cap (room to compose the goodbye). */
 declare const DEFAULT_APPROACHING_END_LEAD_SECONDS = 45;
 /** How early the grace window opens before the cap (fits LLM excuse + send RTT + TTS + playout). */
@@ -1486,6 +1599,17 @@ type ClosingTurnResult = {
 type ExtendResult = {
     ok: boolean;
 };
+type SendTurnOptions = {
+    instructions?: string;
+    /** Optional declaration; the SDK still observes a text transport. */
+    inputSource?: DeclaredInputSource;
+};
+type TranscriptSenderOptions = {
+    /** Captured once for this sender. Defaults to client_stt. */
+    inputSource?: DeclaredInputSource;
+};
+/** Send an already recognized transcript. This adapter does not capture or recognize audio. */
+type TranscriptSender = (text: string, opts?: SendTurnOptions) => Promise<void>;
 type ApproachingEndEvent = {
     secondsLeft: number;
     reason: ApproachingEndReason;
@@ -1619,10 +1743,10 @@ type RealtimeSessionApi = SessionLifecycleApi & {
         proof?: string;
     }) => ExtendResult;
     /** Send a normal turn THROUGH the SDK (arms the turn-timeout watchdog + enables retryTurn). */
-    sendTurn: (text: string, opts?: {
-        instructions?: string;
-    }) => Promise<void>;
-    /** Re-send the last turn sent through the SDK (the "no response" recovery). */
+    sendTurn: (text: string, opts?: SendTurnOptions) => Promise<void>;
+    /** Bind a transcript source once; a send's explicit inputSource takes precedence. */
+    createTranscriptSender: (opts?: TranscriptSenderOptions) => TranscriptSender;
+    /** Re-send the last turn with its resolved provenance, a new ID, and retry_of_turn_id. */
     retryTurn: () => void;
     /** End gracefully now (the user tapped End). */
     end: (reason?: EndReason) => void;
@@ -1744,4 +1868,4 @@ interface ProxyClientOptions {
 }
 declare function createProxyClient(options: ProxyClientOptions): AvatarSessionClient;
 
-export { type QualityCap as $, type AvatarVideoFit as A, type BehaviorSnapshot as B, type CallTranscript as C, DEFAULT_APPROACHING_END_LEAD_SECONDS as D, type EndReason as E, type FishTtsModel as F, type Governor as G, type GovernorState as H, type GraceWindowClosedEvent as I, type GraceWindowOpenEvent as J, type GraceWindowState as K, type LiveKitSessionGrant as L, type IdleWarningEvent as M, type InboundRtpCursor as N, type InboundRtpReading as O, type KnownBehaviorState as P, type LLMProvider as Q, type LLMSelection as R, type LiveKitAvatarGrantState as S, type LiveKitAvatarGrantStatus as T, type LiveKitCapacityState as U, type LiveKitConnectionStatus as V, type LiveKitSessionRequest as W, type LiveKitSessionStartResult as X, MAX_RECONNECT_ATTEMPTS as Y, MAX_SESSION_INSTRUCTIONS_CHARS as Z, type ProxyClientOptions as _, AdaptivePlayoutController as a, RECONNECT_BACKOFF_MS as a0, type RealtimeAvatarRequestOptions as a1, type RealtimeSessionApi as a2, type RealtimeSessionMedia as a3, type RealtimeSessionRoomSinks as a4, type ReconnectPolicy as a5, type ReconnectingEvent as a6, type RecoveryState as a7, type RetryStep as a8, type SessionBehavior as a9, splitCallTranscript as aA, useAvatarAdaptivePlayoutDelay as aB, useAvatarCamera as aC, useAvatarPlayoutDelay as aD, useAvatarQualityGovernor as aE, useCallTranscript as aF, useLiveKitAvatarGrant as aG, useLiveTrackProducing as aH, useMicLease as aI, useRealtimeSession as aJ, useReleaseMicLeaseOnTrackEnded as aK, useSessionLifecycle as aL, AvatarVideoSurface as aM, type AvatarVideoSurfaceProps as aN, type LivePlaybackKeeper as aO, MIC_LEASE_ENDED_TIMEOUT_MS as aP, type MicLease as aQ, NETWORK_QUALITY_POLICY as aR, type NetworkQualityStatus as aS, type PlayableVideoElement as aT, RealtimeAvatarCapacityError as aU, RealtimeAvatarLiveKitRoom as aV, type RealtimeAvatarLiveKitRoomProps as aW, shouldReplayPendingTurn as aX, type SessionClip as aa, type SessionClocks as ab, type SessionEndReason as ac, type SessionLifecycleApi as ad, type SessionLifecyclePhase as ae, type SessionLifecyclePhaseKind as af, SessionLifecycleRoomBridge as ag, type SessionLifecycleRoomBridgeProps as ah, type SurfaceLayers as ai, type TurnState as aj, type TurnTimeoutEvent as ak, type UseAvatarQualityGovernorInput as al, type UseLiveKitAvatarGrantInput as am, type UseRealtimeSessionInput as an, type UseSessionLifecycleInput as ao, type VoiceSpec as ap, type VoiceSpecInput as aq, capacityErrorFromBusy as ar, capacityStateFromGrant as as, createProxyClient as at, isNativeLiveTrackSubscribed as au, knownBehaviorStates as av, mapTurnState as aw, readInboundRtp as ax, sessionBehaviorSchema as ay, sessionClipSchema as az, type AdaptivePlayoutDecision as b, type AdaptivePlayoutOptions as c, type AdaptivePlayoutSample as d, type ApproachingEndEvent as e, type ApproachingEndReason as f, type AvatarConnectionDetails as g, type AvatarSessionClient as h, type CallTranscriptSegment as i, type CartesiaTtsModel as j, type ClipResult as k, type ClosingTurnResult as l, type CreditsLowEvent as m, DEFAULT_CREDITS_LOW_LEAD_SECONDS as n, DEFAULT_GOVERNOR_CONFIG as o, DEFAULT_GRACE_CEILING_SECONDS as p, DEFAULT_GRACE_WINDOW_LEAD_SECONDS as q, DEFAULT_IDLE_SECONDS as r, DEFAULT_IDLE_WARN_LEAD_SECONDS as s, DEFAULT_TURN_TIMEOUT_SECONDS as t, type EndedEvent as u, type ExtendResult as v, type FreezeReadingFn as w, type GovernorAction as x, type GovernorConfig as y, type GovernorSignal as z };
+export { type ProxyClientOptions as $, type AvatarVideoFit as A, type BehaviorSnapshot as B, type CallTranscript as C, DEFAULT_APPROACHING_END_LEAD_SECONDS as D, type EndReason as E, type FishTtsModel as F, type Governor as G, type GovernorSignal as H, type GovernorState as I, type GraceWindowClosedEvent as J, type GraceWindowOpenEvent as K, type LiveKitSessionGrant as L, type GraceWindowState as M, type IdleWarningEvent as N, type InboundRtpCursor as O, type InboundRtpReading as P, type KnownBehaviorState as Q, type LLMProvider as R, type LLMSelection as S, type LiveKitAvatarGrantState as T, type LiveKitAvatarGrantStatus as U, type LiveKitCapacityState as V, type LiveKitConnectionStatus as W, type LiveKitSessionRequest as X, type LiveKitSessionStartResult as Y, MAX_RECONNECT_ATTEMPTS as Z, MAX_SESSION_INSTRUCTIONS_CHARS as _, AdaptivePlayoutController as a, shouldReplayPendingTurn as a$, type QualityCap as a0, RECONNECT_BACKOFF_MS as a1, type RealtimeAvatarRequestOptions as a2, type RealtimeSessionApi as a3, type RealtimeSessionMedia as a4, type RealtimeSessionRoomSinks as a5, type ReconnectPolicy as a6, type ReconnectingEvent as a7, type RecoveryState as a8, type RetryStep as a9, mapTurnState as aA, readInboundRtp as aB, sessionBehaviorSchema as aC, sessionClipSchema as aD, splitCallTranscript as aE, useAvatarAdaptivePlayoutDelay as aF, useAvatarCamera as aG, useAvatarPlayoutDelay as aH, useAvatarQualityGovernor as aI, useCallTranscript as aJ, useLiveKitAvatarGrant as aK, useLiveTrackProducing as aL, useMicLease as aM, useRealtimeSession as aN, useReleaseMicLeaseOnTrackEnded as aO, useSessionLifecycle as aP, AvatarVideoSurface as aQ, type AvatarVideoSurfaceProps as aR, type LivePlaybackKeeper as aS, MIC_LEASE_ENDED_TIMEOUT_MS as aT, type MicLease as aU, NETWORK_QUALITY_POLICY as aV, type NetworkQualityStatus as aW, type PlayableVideoElement as aX, RealtimeAvatarCapacityError as aY, RealtimeAvatarLiveKitRoom as aZ, type RealtimeAvatarLiveKitRoomProps as a_, type SendTurnOptions as aa, type SessionBehavior as ab, type SessionClip as ac, type SessionClocks as ad, type SessionEndReason as ae, type SessionLifecycleApi as af, type SessionLifecyclePhase as ag, type SessionLifecyclePhaseKind as ah, SessionLifecycleRoomBridge as ai, type SessionLifecycleRoomBridgeProps as aj, type SurfaceLayers as ak, type TranscriptSender as al, type TranscriptSenderOptions as am, type TurnState as an, type TurnTimeoutEvent as ao, type UseAvatarQualityGovernorInput as ap, type UseLiveKitAvatarGrantInput as aq, type UseRealtimeSessionInput as ar, type UseSessionLifecycleInput as as, type VoiceSpec as at, type VoiceSpecInput as au, capacityErrorFromBusy as av, capacityStateFromGrant as aw, createProxyClient as ax, isNativeLiveTrackSubscribed as ay, knownBehaviorStates as az, type AdaptivePlayoutDecision as b, type AdaptivePlayoutOptions as c, type AdaptivePlayoutSample as d, type ApproachingEndEvent as e, type ApproachingEndReason as f, type AvatarConnectionDetails as g, type AvatarSessionClient as h, type CallTranscriptSegment as i, type CartesiaTtsModel as j, type ClipResult as k, type ClosingTurnResult as l, type CreditsLowEvent as m, DEFAULT_CREDITS_LOW_LEAD_SECONDS as n, DEFAULT_GOVERNOR_CONFIG as o, DEFAULT_GRACE_CEILING_SECONDS as p, DEFAULT_GRACE_WINDOW_LEAD_SECONDS as q, DEFAULT_IDLE_SECONDS as r, DEFAULT_IDLE_WARN_LEAD_SECONDS as s, DEFAULT_TURN_TIMEOUT_SECONDS as t, type DeclaredInputSource as u, type EndedEvent as v, type ExtendResult as w, type FreezeReadingFn as x, type GovernorAction as y, type GovernorConfig as z };

package/dist/react-native.d.ts

@@ -1,7 +1,7 @@
 import { LiveKitRoomProps } from '@livekit/react-native';
 export { AudioSession, VideoTrack, VideoTrackProps, registerGlobals } from '@livekit/react-native';
-import { L as LiveKitSessionGrant, A as AvatarVideoFit } from './proxy-client-6ndCkiio.js';
-export { a as AdaptivePlayoutController, b as AdaptivePlayoutDecision, c as AdaptivePlayoutOptions, d as AdaptivePlayoutSample, e as ApproachingEndEvent, f as ApproachingEndReason, g as AvatarConnectionDetails, h as AvatarSessionClient, B as BehaviorSnapshot, C as CallTranscript, i as CallTranscriptSegment, j as CartesiaTtsModel, k as ClipResult, l as ClosingTurnResult, m as CreditsLowEvent, D as DEFAULT_APPROACHING_END_LEAD_SECONDS, n as DEFAULT_CREDITS_LOW_LEAD_SECONDS, o as DEFAULT_GOVERNOR_CONFIG, p as DEFAULT_GRACE_CEILING_SECONDS, q as DEFAULT_GRACE_WINDOW_LEAD_SECONDS, r as DEFAULT_IDLE_SECONDS, s as DEFAULT_IDLE_WARN_LEAD_SECONDS, t as DEFAULT_TURN_TIMEOUT_SECONDS, E as EndReason, u as EndedEvent, v as ExtendResult, F as FishTtsModel, w as FreezeReadingFn, G as Governor, x as GovernorAction, y as GovernorConfig, z as GovernorSignal, H as GovernorState, I as GraceWindowClosedEvent, J as GraceWindowOpenEvent, K as GraceWindowState, M as IdleWarningEvent, N as InboundRtpCursor, O as InboundRtpReading, P as KnownBehaviorState, Q as LLMProvider, R as LLMSelection, S as LiveKitAvatarGrantState, T as LiveKitAvatarGrantStatus, U as LiveKitCapacityState, V as LiveKitConnectionStatus, W as LiveKitSessionRequest, X as LiveKitSessionStartResult, Y as MAX_RECONNECT_ATTEMPTS, Z as MAX_SESSION_INSTRUCTIONS_CHARS, _ as ProxyClientOptions, $ as QualityCap, a0 as RECONNECT_BACKOFF_MS, a1 as RealtimeAvatarRequestOptions, a2 as RealtimeSessionApi, a3 as RealtimeSessionMedia, a4 as RealtimeSessionRoomSinks, a5 as ReconnectPolicy, a6 as ReconnectingEvent, a7 as RecoveryState, a8 as RetryStep, a9 as SessionBehavior, aa as SessionClip, ab as SessionClocks, ac as SessionEndReason, ad as SessionLifecycleApi, ae as SessionLifecyclePhase, af as SessionLifecyclePhaseKind, ag as SessionLifecycleRoomBridge, ah as SessionLifecycleRoomBridgeProps, ai as SurfaceLayers, aj as TurnState, ak as TurnTimeoutEvent, al as UseAvatarQualityGovernorInput, am as UseLiveKitAvatarGrantInput, an as UseRealtimeSessionInput, ao as UseSessionLifecycleInput, ap as VoiceSpec, aq as VoiceSpecInput, ar as capacityErrorFromBusy, as as capacityStateFromGrant, at as createProxyClient, au as isNativeLiveTrackSubscribed, av as knownBehaviorStates, aw as mapTurnState, ax as readInboundRtp, ay as sessionBehaviorSchema, az as sessionClipSchema, aA as splitCallTranscript, aB as useAvatarAdaptivePlayoutDelay, aC as useAvatarCamera, aD as useAvatarPlayoutDelay, aE as useAvatarQualityGovernor, aF as useCallTranscript, aG as useLiveKitAvatarGrant, aH as useLiveTrackProducing, aI as useMicLease, aJ as useRealtimeSession, aK as useReleaseMicLeaseOnTrackEnded, aL as useSessionLifecycle } from './proxy-client-6ndCkiio.js';
+import { L as LiveKitSessionGrant, A as AvatarVideoFit } from './proxy-client-DlFXaXtn.js';
+export { a as AdaptivePlayoutController, b as AdaptivePlayoutDecision, c as AdaptivePlayoutOptions, d as AdaptivePlayoutSample, e as ApproachingEndEvent, f as ApproachingEndReason, g as AvatarConnectionDetails, h as AvatarSessionClient, B as BehaviorSnapshot, C as CallTranscript, i as CallTranscriptSegment, j as CartesiaTtsModel, k as ClipResult, l as ClosingTurnResult, m as CreditsLowEvent, D as DEFAULT_APPROACHING_END_LEAD_SECONDS, n as DEFAULT_CREDITS_LOW_LEAD_SECONDS, o as DEFAULT_GOVERNOR_CONFIG, p as DEFAULT_GRACE_CEILING_SECONDS, q as DEFAULT_GRACE_WINDOW_LEAD_SECONDS, r as DEFAULT_IDLE_SECONDS, s as DEFAULT_IDLE_WARN_LEAD_SECONDS, t as DEFAULT_TURN_TIMEOUT_SECONDS, u as DeclaredInputSource, E as EndReason, v as EndedEvent, w as ExtendResult, F as FishTtsModel, x as FreezeReadingFn, G as Governor, y as GovernorAction, z as GovernorConfig, H as GovernorSignal, I as GovernorState, J as GraceWindowClosedEvent, K as GraceWindowOpenEvent, M as GraceWindowState, N as IdleWarningEvent, O as InboundRtpCursor, P as InboundRtpReading, Q as KnownBehaviorState, R as LLMProvider, S as LLMSelection, T as LiveKitAvatarGrantState, U as LiveKitAvatarGrantStatus, V as LiveKitCapacityState, W as LiveKitConnectionStatus, X as LiveKitSessionRequest, Y as LiveKitSessionStartResult, Z as MAX_RECONNECT_ATTEMPTS, _ as MAX_SESSION_INSTRUCTIONS_CHARS, $ as ProxyClientOptions, a0 as QualityCap, a1 as RECONNECT_BACKOFF_MS, a2 as RealtimeAvatarRequestOptions, a3 as RealtimeSessionApi, a4 as RealtimeSessionMedia, a5 as RealtimeSessionRoomSinks, a6 as ReconnectPolicy, a7 as ReconnectingEvent, a8 as RecoveryState, a9 as RetryStep, aa as SendTurnOptions, ab as SessionBehavior, ac as SessionClip, ad as SessionClocks, ae as SessionEndReason, af as SessionLifecycleApi, ag as SessionLifecyclePhase, ah as SessionLifecyclePhaseKind, ai as SessionLifecycleRoomBridge, aj as SessionLifecycleRoomBridgeProps, ak as SurfaceLayers, al as TranscriptSender, am as TranscriptSenderOptions, an as TurnState, ao as TurnTimeoutEvent, ap as UseAvatarQualityGovernorInput, aq as UseLiveKitAvatarGrantInput, ar as UseRealtimeSessionInput, as as UseSessionLifecycleInput, at as VoiceSpec, au as VoiceSpecInput, av as capacityErrorFromBusy, aw as capacityStateFromGrant, ax as createProxyClient, ay as isNativeLiveTrackSubscribed, az as knownBehaviorStates, aA as mapTurnState, aB as readInboundRtp, aC as sessionBehaviorSchema, aD as sessionClipSchema, aE as splitCallTranscript, aF as useAvatarAdaptivePlayoutDelay, aG as useAvatarCamera, aH as useAvatarPlayoutDelay, aI as useAvatarQualityGovernor, aJ as useCallTranscript, aK as useLiveKitAvatarGrant, aL as useLiveTrackProducing, aM as useMicLease, aN as useRealtimeSession, aO as useReleaseMicLeaseOnTrackEnded, aP as useSessionLifecycle } from './proxy-client-DlFXaXtn.js';
 import { ReactNode, ReactElement } from 'react';
 import { StyleProp, ViewStyle } from 'react-native';
 export { useChat, useConnectionState, useLocalParticipant, useRoomContext, useTrackToggle, useTranscriptions, useVoiceAssistant } from '@livekit/components-react';

package/dist/react-native.js

@@ -1585,15 +1585,22 @@ var styles = StyleSheet.create({
   }
 });
 var RECORDED_MEDIA_MODES = ["audio", "video", "audio_video"];
-var recordingModeSchema = z.enum(["off", ...RECORDED_MEDIA_MODES]).describe("Server-owned recording policy; omitted means off. Audio includes user and avatar audio.");
+var recordingModeSchema = z.enum(["off", ...RECORDED_MEDIA_MODES, "participants"]).describe("Server-owned recording policy; omitted means off. Legacy audio includes user and avatar audio. participants saves one continuous audio/video file per participant, without mixing their voices.");
 var RECORDING_STATUSES = ["pending", "recording", "processing", "ready", "failed", "expired"];
 z.enum(RECORDING_STATUSES);
+var recordingParticipantSchema = z.object({
+  participantIdentity: z.string().min(1).max(160),
+  role: z.enum(["user", "avatar"])
+}).strict();
 var recordingMetadataSchema = z.object({
   sessionId: z.string().min(1),
   recordingId: z.string().min(1),
-  mode: recordingModeSchema.exclude(["off"]),
+  mode: z.enum(RECORDED_MEDIA_MODES),
   createdAt: z.string().datetime({ offset: true }),
-  retainedUntil: z.string().datetime({ offset: true })
+  retainedUntil: z.string().datetime({ offset: true }),
+  participant: recordingParticipantSchema.optional(),
+  mediaStartedAt: z.string().datetime({ offset: true }).optional(),
+  mediaEndedAt: z.string().datetime({ offset: true }).optional()
 });
 var recordingArtifactSchema = z.discriminatedUnion("status", [
   recordingMetadataSchema.extend({ status: z.enum(["pending", "recording", "processing", "expired"]) }).strict(),
@@ -1900,6 +1907,8 @@ z.object({
 }).strict();
 z.object({
   recording: recordingArtifactSchema.optional(),
+  recording_mode: recordingModeSchema.optional(),
+  recordings: z.array(recordingArtifactSchema).max(2).optional(),
   connection_history: connectionHistoryGrantSchema.optional(),
   status: z.literal("ready").default("ready"),
   session_id: z.string().min(1),
@@ -1920,6 +1929,11 @@ var RTA_LIFECYCLE_TOPIC = "rta.lifecycle";
 var RTA_TURN_INSTRUCTIONS_ATTR = "rta.turn_instructions";
 var RTA_CLOSING_TURN_ATTR = "rta.closing_turn";
 var RTA_TURN_ID_ATTR = "rta.turn_id";
+var RTA_RETRY_OF_TURN_ID_ATTR = "rta.retry_of_turn_id";
+var RTA_INPUT_SOURCE_VERSION_ATTR = "rta.input_source_version";
+var RTA_OBSERVED_INPUT_SOURCE_ATTR = "rta.observed_input_source";
+var RTA_DECLARED_INPUT_SOURCE_ATTR = "rta.declared_input_source";
+var RTA_INPUT_SOURCE_DECLARATION_SCOPE_ATTR = "rta.input_source_declaration_scope";
 var sessionEndReasonSchema = z.enum([
   "user_ended",
   "session_cap",
@@ -2855,6 +2869,11 @@ var DEFAULT_CREDITS_LOW_LEAD_SECONDS = 300;
 function positive(value, fallbackSeconds) {
   return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : fallbackSeconds;
 }
+function validateInputSource(value) {
+  if (value !== void 0 && value !== "text" && value !== "client_stt") {
+    throw new TypeError('inputSource must be "text" or "client_stt"');
+  }
+}
 function useRealtimeSession(input) {
   const {
     session,
@@ -2912,7 +2931,8 @@ function useRealtimeSession(input) {
   const turnSenderRef = useRef(null);
   const dataPublisherRef = useRef(null);
   const lastTurnRef = useRef(null);
-  const [clockTick, setClockTick] = useState(0);
+  const turnSequenceRef = useRef(0);
+  const [, setClockTick] = useState(0);
   const endsAt = endsAtFrom({
     serverStartedAtUnixMs: serverClock?.startedAtUnixMs ?? null,
     maxSessionSeconds: serverClock?.maxSessionSeconds ?? null
@@ -2934,8 +2954,8 @@ function useRealtimeSession(input) {
       if (typeof crypto !== "undefined" && typeof crypto.randomUUID === "function") return crypto.randomUUID();
     } catch {
     }
-    return `t-${Date.now()}-${clockTick}`;
-  }, [clockTick]);
+    return `t-${Date.now()}-${++turnSequenceRef.current}`;
+  }, []);
   const onLifecycleData = useCallback((frame) => {
     const parsed = lifecycleServerFrameSchema.safeParse(frame);
     if (!parsed.success) return;
@@ -3035,8 +3055,8 @@ function useRealtimeSession(input) {
         if (gw === "open" || gw === "delivering") stayConnected();
       }
       const turnInfo = lastTurnRef.current;
-      if (turnInfo && now - turnInfo.sentAt >= turnTimeoutMs) {
-        lastTurnRef.current = null;
+      if (turnInfo && turnInfo.sentAt !== null && now - turnInfo.sentAt >= turnTimeoutMs) {
+        turnInfo.sentAt = null;
         cbRef.current.onTurnTimeout?.({ turnId: turnInfo.id });
       }
       setClockTick((value) => value + 1);
@@ -3065,6 +3085,7 @@ function useRealtimeSession(input) {
   }, [phaseKind, lifecycle.phase, lifecycle.attempt]);
   useEffect(() => {
     if (phaseKind === "ended") {
+      lastTurnRef.current = null;
       if (!endedFiredRef.current) {
         endedFiredRef.current = true;
         const inner2 = lifecycle.phase.kind === "ended" ? lifecycle.phase.reason : void 0;
@@ -3101,23 +3122,47 @@ function useRealtimeSession(input) {
     setGraceWindow({ kind: "closed" });
     return { ok: true };
   }, []);
-  const sendTurn = useCallback(async (text, opts) => {
+  const sendResolvedTurn = useCallback(async (text, attributes, retryOfTurnId) => {
     const trimmed = text.trim();
     const sender = turnSenderRef.current;
     if (!sender || !trimmed) return;
     const turnId = newTurnId();
-    lastTurnRef.current = { text: trimmed, opts, id: turnId, sentAt: Date.now() };
+    const outgoing = { ...attributes, [RTA_TURN_ID_ATTR]: turnId };
+    if (retryOfTurnId) outgoing[RTA_RETRY_OF_TURN_ID_ATTR] = retryOfTurnId;
+    lastTurnRef.current = { text: trimmed, attributes: { ...outgoing }, id: turnId, sentAt: Date.now() };
     lifecycle.markActivity();
-    const attributes = opts?.instructions ? { [RTA_TURN_INSTRUCTIONS_ATTR]: opts.instructions } : void 0;
-    await sender(trimmed, attributes ? { attributes } : void 0);
+    await sender(trimmed, { attributes: outgoing });
   }, [lifecycle, newTurnId]);
+  const sendInput = useCallback(async (text, opts, adapterSource) => {
+    const turnSource = opts?.inputSource;
+    validateInputSource(turnSource);
+    const declaredSource = turnSource ?? adapterSource;
+    const attributes = {
+      [RTA_INPUT_SOURCE_VERSION_ATTR]: "1",
+      [RTA_OBSERVED_INPUT_SOURCE_ATTR]: "text"
+    };
+    if (declaredSource !== void 0) {
+      attributes[RTA_DECLARED_INPUT_SOURCE_ATTR] = declaredSource;
+      attributes[RTA_INPUT_SOURCE_DECLARATION_SCOPE_ATTR] = turnSource !== void 0 ? "turn" : "adapter";
+    }
+    if (opts?.instructions) attributes[RTA_TURN_INSTRUCTIONS_ATTR] = opts.instructions;
+    await sendResolvedTurn(text, attributes);
+  }, [sendResolvedTurn]);
+  const sendTurn = useCallback((text, opts) => sendInput(text, opts), [sendInput]);
+  const createTranscriptSender = useCallback((opts) => {
+    const source = opts?.inputSource;
+    validateInputSource(source);
+    const boundSource = source ?? "client_stt";
+    return (text, turnOpts) => sendInput(text, turnOpts, boundSource);
+  }, [sendInput]);
   const retryTurn = useCallback(() => {
     const last = lastTurnRef.current;
     if (!last) return;
-    void sendTurn(last.text, last.opts);
-  }, [sendTurn]);
+    void sendResolvedTurn(last.text, last.attributes, last.id).catch(() => void 0);
+  }, [sendResolvedTurn]);
   const end = useCallback((reason) => {
     if (reason) lastLabeledEndReasonRef.current = reason;
+    lastTurnRef.current = null;
     requestGracefulClose();
     lifecycle.reset();
   }, [lifecycle, requestGracefulClose]);
@@ -3184,6 +3229,7 @@ function useRealtimeSession(input) {
       requestGracefulClose,
       extend,
       sendTurn,
+      createTranscriptSender,
       retryTurn,
       end,
       behavior,
@@ -3206,6 +3252,7 @@ function useRealtimeSession(input) {
       requestGracefulClose,
       extend,
       sendTurn,
+      createTranscriptSender,
       retryTurn,
       end,
       behavior,