@npcify/web 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -125,10 +125,13 @@ animate()
125
125
  - **Dialogue & Speech State**:
126
126
  - `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
127
127
  - `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
128
+ - `character.onConnectionClosed(callback)`: The server closed the conversation connection ([lifecycle](#runtime-and-lifecycle)).
128
129
  - **Voice Input** ([details](#voice-input-and-input-modes)):
129
130
  - `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
130
131
  - `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
131
132
  - `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
133
+ - **Sight** ([details](#sight)):
134
+ - `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
132
135
 
133
136
  ## Mode 2: Managed Canvas (`mountNpcify`)
134
137
 
@@ -252,6 +255,70 @@ character.onListeningState((state) => {
252
255
  - The character owner pays the recognition at the price of the provider that
253
256
  the owner chose, as for the turn's language model.
254
257
 
258
+ ## Sight
259
+
260
+ The character can see the player's camera when the character owner turned
261
+ sight on (the Sight tab of the character in the dashboard, where the owner
262
+ also chooses the vision provider). The SDK captures small camera frames and
263
+ sends them on the conversation connection. The server sends each frame to the
264
+ vision provider and keeps only a short text observation; it does not keep the
265
+ frames. The character reads the observations of the last minute in its next
266
+ reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
267
+
268
+ ```js
269
+ const see = document.querySelector('#see')
270
+ const report = (error) => { status.textContent = error.message }
271
+ see.hidden = character.sightState === 'unavailable'
272
+ see.onclick = () => {
273
+ if (character.sightState === 'off') {
274
+ character.startSight().catch(report)
275
+ } else {
276
+ character.stopSight()
277
+ }
278
+ }
279
+ character.onSightState((state, error) => {
280
+ see.hidden = state === 'unavailable'
281
+ see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
282
+ if (error) report(error)
283
+ })
284
+ character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
285
+ ```
286
+
287
+ - `startSight({ source, intervalMs, motion, onError })` starts sight in the
288
+ current conversation; call `createConversation()` first. The first call
289
+ asks for the camera permission. `source` is `'camera'` (default) or a
290
+ `MediaStream` that you opened, for example a screen share. `stopSight()`
291
+ closes the camera that the SDK opened. The SDK never stops the tracks of a
292
+ stream that you give.
293
+ - Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
294
+ smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
295
+ it compares a 32×18 grid of the luminance with the grid before. When the
296
+ mean change is more than `motion.threshold` (default 8, on a 0 to 255
297
+ scale), a frame goes, one each second at most. Else a frame goes every
298
+ `intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
299
+ frames only at the interval. One frame at a time waits for the server: a
300
+ frame that comes due during the wait goes at the answer. While the page is
301
+ hidden, the SDK captures nothing.
302
+ - `sightState` is `'unavailable'` (the owner did not turn sight on, or the
303
+ conversation is baked), `'off'`, `'starting'` or `'on'`.
304
+ `onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
305
+ gives each observation of the current conversation:
306
+ `{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
307
+ - Errors are `SightError`s with a `code`. `startSight()` rejects with
308
+ `SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
309
+ `SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
310
+ or a code of the conversation connection.
311
+ - These codes stop sight after it started and go to `onError` and to the
312
+ `error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
313
+ `INSUFFICIENT_POINTS` (the owner has no points),
314
+ `CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
315
+ shared screen stopped). The SDK skips one frame and continues for
316
+ `SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
317
+ `SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
318
+ - `endConversation()`, `dispose()` and `loadAssets()` stop sight.
319
+ - The character owner pays the analysis of each frame at the price of the
320
+ provider that the owner chose. A refused or failed frame costs nothing.
321
+
255
322
  ## Runtime and lifecycle
256
323
 
257
324
  - `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
@@ -277,6 +344,16 @@ character.onListeningState((state) => {
277
344
  baked playback requires browser rendering.
278
345
  - Keys may cover multiple characters. Editing scopes, rotating, or deleting keys
279
346
  affects subsequent requests and disconnects associated conversation sockets.
347
+ - `character.onConnectionClosed(({ reason, code, conversationId }) => …)` reports
348
+ a close of the conversation connection by the server: a refused or failed
349
+ join (`code`), or a key that was edited, rotated or deleted (`code: null`).
350
+ The SDK does not use that connection again: the next `connect`,
351
+ `sendMessage` or `startListening` opens a new one. Joins and requests that
352
+ wait on a closed connection fail at once with a `ConversationConnectionError`
353
+ (`CONVERSATION_DISCONNECTED`); a wait without an answer in time fails with
354
+ `CONVERSATION_TIMEOUT`. A refused conversation fails at once with
355
+ `CONVERSATION_ACCESS_DENIED` until `endConversation()`; after another join
356
+ code (`CONVERSATION_JOIN_FAILED` or a newer one) the next call tries again.
280
357
  - ESM only; `require()` is not a supported package entry point. Browser APIs are
281
358
  used when creating a view; do not call `mountNpcify` during server rendering.
282
359
 
package/dist/index.d.ts CHANGED
@@ -3,6 +3,49 @@ import { VRMAnimation } from '@pixiv/three-vrm-animation';
3
3
  import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
4
4
  import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
5
5
 
6
+ /**
7
+ * Sight (docs/sight.md): a client captures frames, the server sends each
8
+ * frame to the vision provider that the owner of the character chose, and the
9
+ * character reads the text observations in its next replies. Frames are never
10
+ * stored.
11
+ */
12
+ /** The vision providers. No other provider takes over when one fails. */
13
+ declare const SIGHT_PROVIDERS: readonly ["gemini", "openai"];
14
+ type SightProvider = (typeof SIGHT_PROVIDERS)[number];
15
+ /** The codes of a refused frame. A newer server can send other codes. */
16
+ declare const SIGHT_ERRORS: readonly ["SIGHT_DISABLED", "SIGHT_RATE_LIMITED", "SIGHT_FRAME_TOO_LARGE", "SIGHT_FRAME_INVALID", "SIGHT_PROVIDER_FAILED", "INSUFFICIENT_POINTS", "CONVERSATION_ACCESS_DENIED"];
17
+ type SightErrorCode = (typeof SIGHT_ERRORS)[number];
18
+ type SightPersonPosition = 'left' | 'center' | 'right';
19
+ /** One person or avatar in a frame. */
20
+ type SightPerson = {
21
+ position: SightPersonPosition;
22
+ /** What the person does, in five words or fewer. */
23
+ action: string;
24
+ /** The person faces or addresses the camera. */
25
+ facingCamera: boolean;
26
+ };
27
+ /** What the vision model saw in one frame. */
28
+ type SightObservation = {
29
+ /** The number of people or avatars in the frame. */
30
+ people: number;
31
+ persons: SightPerson[];
32
+ /** Each readable piece of text, verbatim. */
33
+ text: string[];
34
+ };
35
+ /** `sight_observation`: an observation of the conversation, sent to its room. */
36
+ type SightObservationEvent = {
37
+ conversationId: string;
38
+ observationId: string;
39
+ /** The capture time that the client sent, as an ISO 8601 string. */
40
+ capturedAt: string;
41
+ observation: SightObservation;
42
+ /** One sentence about the frame, for the character. */
43
+ summary: string;
44
+ model: string;
45
+ /** The points charged to the owner; null while the charge is pending. */
46
+ costPoints: number | null;
47
+ };
48
+
6
49
  type TtsProviderId = string;
7
50
  type NullableTtsProviderId = TtsProviderId | null;
8
51
 
@@ -18,6 +61,14 @@ type Sqt = {
18
61
  quaternion?: [number, number, number, number] | null;
19
62
  translation?: [number, number, number] | null;
20
63
  };
64
+ /**
65
+ * The owner's sight switch and vision provider (docs/sight.md). `enabled` is
66
+ * true only with a provider; an SDK captures frames only then.
67
+ */
68
+ type AvatarSightSettings = {
69
+ enabled: boolean;
70
+ provider: SightProvider | null;
71
+ };
21
72
  interface AvatarPublicInfo {
22
73
  id: string;
23
74
  name: string | null;
@@ -48,8 +99,12 @@ interface AvatarPublicInfo {
48
99
  * `allowCustomReply`. Missing on older servers: off.
49
100
  */
50
101
  speechInputEnabled?: boolean;
102
+ /** Sight (Sight tab). Missing on older servers: off. */
103
+ sight?: AvatarSightSettings;
51
104
  autoPlay: boolean;
52
105
  serverRenderingEnabled: boolean;
106
+ /** The character's body motion is generated live in each turn (missing on older servers). */
107
+ realtimeMotionEnabled?: boolean;
53
108
  cameraSqt?: Sqt | null;
54
109
  }
55
110
 
@@ -192,12 +247,45 @@ interface StagingAnimationClip {
192
247
  animation: string;
193
248
  loop: boolean;
194
249
  }
250
+ /** A part of the staging whose body motion is generated live (NPCP 1.1). */
251
+ interface StagingMotionClip {
252
+ /** Seconds from the start of the staging. */
253
+ startTime: number;
254
+ /** Seconds from the start of the staging. */
255
+ endTime: number;
256
+ /** A short English description of the body motion. */
257
+ prompt: string;
258
+ }
259
+ type StagingMotionFramesSource = {
260
+ type: 's3';
261
+ url: string;
262
+ } | {
263
+ type: 'bytes';
264
+ data: Uint8Array;
265
+ };
266
+ /**
267
+ * The stored live motion of a staging (NPCP 1.1): pose frames on the staging
268
+ * clock, in the layout of `motion_frames` (see `motion-frames.ts`).
269
+ */
270
+ interface StagingMotionTrack {
271
+ fps: number;
272
+ frameCount: number;
273
+ /** VRM humanoid bone names, the order of the rotations of a frame. */
274
+ bones: string[];
275
+ /** Hips height of the source rest pose in meters. */
276
+ restHipsHeight: number;
277
+ frames: StagingMotionFramesSource;
278
+ }
195
279
  interface StagingTimeline {
196
280
  /** Total duration in seconds. */
197
281
  duration: number;
198
282
  dialogue: StagingDialogueBlock[];
199
283
  expressions: StagingExpressionClip[];
200
284
  animations: StagingAnimationClip[];
285
+ /** Live motion prompts of a character with real-time motion (NPCP 1.1). */
286
+ motions?: StagingMotionClip[];
287
+ /** Stored live motion frames, on reads of a completed staging (NPCP 1.1). */
288
+ motion?: StagingMotionTrack;
201
289
  }
202
290
  declare const STAGING_PROTOCOL_VERSION: 1;
203
291
  interface StagingResultPayload {
@@ -262,6 +350,7 @@ interface StagingStreamSpeechErrorEvent {
262
350
  stagingId: string;
263
351
  speechId: string;
264
352
  streamId: string;
353
+ /** Always `SPEECH_FAILED`; the cause stays in the server logs. */
265
354
  message: string;
266
355
  }
267
356
  interface StagingStreamOptionsEvent {
@@ -275,6 +364,10 @@ interface StagingStreamErrorEvent {
275
364
  type: 'error';
276
365
  taskId?: string;
277
366
  stagingId?: string;
367
+ /**
368
+ * A client error of the server, for example `INSUFFICIENT_POINTS`, or
369
+ * `TURN_FAILED`. Vendor text stays in the server logs.
370
+ */
278
371
  message: string;
279
372
  }
280
373
  interface StagingStreamBillingEvent {
@@ -284,7 +377,28 @@ interface StagingStreamBillingEvent {
284
377
  usageCharge?: UsageChargeSummary | null;
285
378
  availablePoints?: number;
286
379
  }
287
- type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent;
380
+ /**
381
+ * Live pose frames of a character with real-time motion (NPCP 1.1). Older
382
+ * receivers ignore the event. Frame `startFrame + i` plays at
383
+ * `(startFrame + i) / fps` seconds of the staging clock. A frame of a later
384
+ * event replaces the frame with the same index unless its `revision` is lower.
385
+ */
386
+ interface StagingStreamMotionFramesEvent {
387
+ type: 'motion_frames';
388
+ taskId: string;
389
+ stagingId: string;
390
+ revision: number;
391
+ fps: number;
392
+ startFrame: number;
393
+ frameCount: number;
394
+ /** VRM humanoid bone names, the order of the rotations of a frame. */
395
+ bones: string[];
396
+ /** Hips height of the source rest pose in meters. */
397
+ restHipsHeight: number;
398
+ /** `frameCount x (3 + 4 x bones.length)` float32 values, little endian. */
399
+ frames: Uint8Array;
400
+ }
401
+ type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent | StagingStreamMotionFramesEvent;
288
402
 
289
403
  /**
290
404
  * Voice input of conversations (docs/voice-input.md): the input modes of the
@@ -341,7 +455,7 @@ interface ConversationTurnOptions {
341
455
  requestId?: string;
342
456
  }
343
457
 
344
- type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled'>>;
458
+ type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled' | 'sight'>>;
345
459
 
346
460
  interface LlmThinkingPayload {
347
461
  taskId?: string | null;
@@ -994,8 +1108,14 @@ declare class StagingPlayer {
994
1108
  private activeExpression;
995
1109
  private activeExpressionMouthMode;
996
1110
  private activeAnimation;
1111
+ private currentTrackTime;
997
1112
  constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
998
1113
  get paused(): boolean;
1114
+ /**
1115
+ * Seconds of the staging clock that the tracks show now, or null when no
1116
+ * staging plays (also while playback waits for the next dialogue block).
1117
+ */
1118
+ get trackTime(): number | null;
999
1119
  pause(): void;
1000
1120
  resume(): Promise<void>;
1001
1121
  private waitUntilResumed;
@@ -1016,6 +1136,55 @@ declare class StagingPlayer {
1016
1136
  private clearPerformance;
1017
1137
  }
1018
1138
 
1139
+ /**
1140
+ * How the host uses the root of live motion. `anchored` (default) removes the
1141
+ * horizontal hips movement of the turn, so that the character stays where the
1142
+ * host placed it; height and turns stay. `world` applies it as generated.
1143
+ */
1144
+ type LiveMotionRootMode = 'anchored' | 'world';
1145
+ interface LiveMotionOptions {
1146
+ rootMode?: LiveMotionRootMode;
1147
+ /** Seconds to blend from idle into live frames. */
1148
+ blendInSeconds?: number;
1149
+ /** Seconds to blend from live frames back to idle. */
1150
+ blendOutSeconds?: number;
1151
+ }
1152
+ /**
1153
+ * Plays live motion frames (ADR 0011) on a VRM's normalized humanoid. Call
1154
+ * `update()` after the animation mixer and before `vrm.update()`: the live
1155
+ * pose blends over the mixer's pose (idle) with its own weight.
1156
+ */
1157
+ declare class LiveMotionPlayer {
1158
+ private vrm;
1159
+ private readonly tracks;
1160
+ private readonly rootMode;
1161
+ private readonly blendIn;
1162
+ private readonly blendOut;
1163
+ private weight;
1164
+ /** The last applied pose, to blend out of it after the frames end. */
1165
+ private lastPose;
1166
+ private readonly scratch;
1167
+ constructor(options?: LiveMotionOptions);
1168
+ setVrm(vrm: VRM | null): void;
1169
+ /** Adds the frames of a `motion_frames` event; invalid events are dropped. */
1170
+ push(received: StagingStreamMotionFramesEvent): boolean;
1171
+ /** Loads the stored frames of a staging (a replay). */
1172
+ load(stagingId: string, track: StagingMotionTrack, fetchBytes?: (url: string) => Promise<Uint8Array>): Promise<void>;
1173
+ has(stagingId: string): boolean;
1174
+ forget(stagingId: string): void;
1175
+ /** The current live weight, 0 (idle) to 1 (live frames). */
1176
+ get liveWeight(): number;
1177
+ /**
1178
+ * Applies the live pose of `stagingId` at `time` seconds of its staging
1179
+ * clock; `time` null means that no staging plays now.
1180
+ */
1181
+ update(delta: number, stagingId: string | null, time: number | null): void;
1182
+ private track;
1183
+ /** The interpolated frame at `time`, or null when its frames are missing. */
1184
+ private sample;
1185
+ private apply;
1186
+ }
1187
+
1019
1188
  type Vector3Tuple = readonly [number, number, number];
1020
1189
  interface Vector3Like {
1021
1190
  readonly x: number;
@@ -1151,6 +1320,11 @@ interface NpcifyOptions {
1151
1320
  * `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
1152
1321
  */
1153
1322
  normalizeIdleOrientation?: boolean;
1323
+ /**
1324
+ * Playback of live motion frames (characters with real-time motion). The
1325
+ * default `rootMode: 'anchored'` keeps the character where it stands.
1326
+ */
1327
+ liveMotion?: LiveMotionOptions;
1154
1328
  /**
1155
1329
  * How the player can reply: `'options'`, `'text'` and `'voice'`, in any
1156
1330
  * combination; one mode forces it. Without it, every mode that the
@@ -1188,6 +1362,10 @@ interface StreamPlaybackDependencies {
1188
1362
  handleCallbackError: (context: string, error: unknown) => void;
1189
1363
  reportError: (error: Error, context?: string) => void;
1190
1364
  recordEvent?: (event: PerformanceEventPayload) => void;
1365
+ /** Live motion frames of the playing staging (NPCP 1.1). */
1366
+ onMotionFrames?: (event: StagingStreamMotionFramesEvent) => void;
1367
+ /** The stored live motion of a staging that starts to play (a replay). */
1368
+ onMotionTrack?: (stagingId: string, track: StagingMotionTrack) => void;
1191
1369
  }
1192
1370
  declare class StreamPlaybackController {
1193
1371
  private readonly dependencies;
@@ -1214,6 +1392,8 @@ declare class StreamPlaybackController {
1214
1392
  get revealedText(): string;
1215
1393
  get options(): StagingResultPayload['options'];
1216
1394
  get cursor(): CharacterPlaybackCursor | null;
1395
+ /** The staging that plays or waits to play, or null. */
1396
+ get activeStagingId(): string | null;
1217
1397
  onStreamEvent(callback: (event: StagingStreamEvent) => Promise<void> | void): () => void;
1218
1398
  onSpeechPlaybackStart(callback: (speech: StagingSpeechPlaybackEvent) => Promise<void> | void): () => void;
1219
1399
  onCharacterReveal(callback: (speech: StagingSpeechPlaybackEvent, characterIndex: number, character: string) => void): () => void;
@@ -1300,6 +1480,7 @@ declare class CharacterRuntime {
1300
1480
  protected readonly performanceRecorder: PerformanceRecorder;
1301
1481
  protected readonly telemetry: ClientTelemetry;
1302
1482
  protected readonly streamPlayback: StreamPlaybackController;
1483
+ protected readonly liveMotion: LiveMotionPlayer;
1303
1484
  protected defaultParent: Object3D | null;
1304
1485
  protected currentGazeTarget: Object3D | null;
1305
1486
  /** The current gaze target came from `lookAt()`. */
@@ -1436,6 +1617,85 @@ declare class CharacterRuntime {
1436
1617
  protected readonly reportError: (error: Error, context?: string) => void;
1437
1618
  }
1438
1619
 
1620
+ /**
1621
+ * The code of a join that the server did not accept. Only
1622
+ * `CONVERSATION_ACCESS_DENIED` is final; the next join tries any other code
1623
+ * again. Another string is a code of a newer server.
1624
+ */
1625
+ type ConversationJoinFailure = 'CONVERSATION_ACCESS_DENIED' | 'CONVERSATION_JOIN_FAILED' | (string & {});
1626
+ /**
1627
+ * `CONVERSATION_DISCONNECTED`: the connection closed, was lost or is not
1628
+ * open. `CONVERSATION_TIMEOUT`: no acknowledgement came in time.
1629
+ */
1630
+ type ConversationConnectionErrorCode = ConversationJoinFailure | 'CONVERSATION_DISCONNECTED' | 'CONVERSATION_TIMEOUT';
1631
+ /**
1632
+ * A join or a request that the conversation connection could not complete
1633
+ * (docs/streaming_reply.md#web-sdk).
1634
+ */
1635
+ declare class ConversationConnectionError extends Error {
1636
+ readonly code: ConversationConnectionErrorCode;
1637
+ /** The close or loss that failed the wait (`io server disconnect`, `transport close`, `connect_error`, ...), or null. */
1638
+ readonly reason: string | null;
1639
+ constructor(code: ConversationConnectionErrorCode, message: string,
1640
+ /** The close or loss that failed the wait (`io server disconnect`, `transport close`, `connect_error`, ...), or null. */
1641
+ reason?: string | null);
1642
+ }
1643
+ /** The server closed the conversation connection, or refused to open it. */
1644
+ interface ConversationConnectionClosedEvent {
1645
+ /** `io server disconnect`, or `connect_error` when the server refused the connection. */
1646
+ reason: string;
1647
+ /**
1648
+ * The join that made the server close the connection; null for another
1649
+ * close, for example after the API key was rotated, edited or deleted.
1650
+ */
1651
+ code: ConversationJoinFailure | null;
1652
+ /** The conversation of that join, or null. */
1653
+ conversationId: string | null;
1654
+ }
1655
+
1656
+ /**
1657
+ * The code of a `SightError`: a refusal of the server (`SIGHT_ERRORS` of
1658
+ * `@npcify/shared`), a code of the conversation connection
1659
+ * (`CONVERSATION_DISCONNECTED`, `CONVERSATION_TIMEOUT`, ...), or a code of
1660
+ * the SDK. Another string is a code of a newer server.
1661
+ */
1662
+ type SightFailureCode = SightErrorCode | 'CAMERA_PERMISSION_DENIED' | 'CAMERA_UNAVAILABLE' | 'SIGHT_SOURCE_ENDED' | 'SIGHT_ALREADY_STARTED' | 'CONVERSATION_REQUIRED' | (string & {});
1663
+ declare class SightError extends Error {
1664
+ readonly code: SightFailureCode;
1665
+ constructor(code: SightFailureCode, message?: string);
1666
+ }
1667
+
1668
+ /** `'camera'` opens the camera; a `MediaStream` is a video of the caller. */
1669
+ type SightSourceOption = MediaStream | 'camera';
1670
+
1671
+ /**
1672
+ * `'unavailable'`: the character info did not turn sight on, or the
1673
+ * conversation is baked. `'off'`: sight can start. `'starting'`: the camera
1674
+ * and the connection open. `'on'`: frames go to the server.
1675
+ */
1676
+ type SightState = 'unavailable' | 'off' | 'starting' | 'on';
1677
+ interface SightOptions {
1678
+ /**
1679
+ * `'camera'` (default) opens the camera, and `stopSight()` closes it. A
1680
+ * `MediaStream` (a camera or a screen share of your page) stays yours:
1681
+ * the SDK never stops its tracks.
1682
+ */
1683
+ source?: SightSourceOption;
1684
+ /** The longest time between two frames. Default 5000 ms; the minimum is 1000 ms. */
1685
+ intervalMs?: number;
1686
+ /**
1687
+ * A change of the picture sends a frame at once (one each second at most).
1688
+ * `threshold` is the mean change of the 32 × 18 luma grid between two
1689
+ * checks, on a 0 to 255 scale (default 8). `false` sends only at the
1690
+ * interval.
1691
+ */
1692
+ motion?: {
1693
+ threshold?: number;
1694
+ } | false;
1695
+ /** The error that stopped sight after `startSight()` resolved. */
1696
+ onError?: (error: SightError) => void;
1697
+ }
1698
+
1439
1699
  declare class MicrophoneError extends Error {
1440
1700
  readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
1441
1701
  constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
@@ -1536,18 +1796,18 @@ declare class Character extends CharacterRuntime {
1536
1796
  private readonly notifiedStagingIds;
1537
1797
  private readonly liveConversation;
1538
1798
  private baked;
1799
+ /** The conversation whose live motion was warmed up. */
1800
+ private liveMotionPrepared;
1539
1801
  private readonly authoring;
1540
1802
  protected readonly normalizeIdleOrientation: boolean;
1541
1803
  private readonly bakedAsset?;
1542
1804
  private animationRefreshId;
1543
1805
  private readonly voice;
1544
- private readonly configuredInputModes;
1545
- private resolvedInputModes;
1546
- /** The character's voice input switch, after the character loads. */
1547
- private speechInput;
1548
- private latestOptions;
1806
+ private readonly sight;
1807
+ private readonly inputs;
1549
1808
  private listeningStateCallback;
1550
1809
  private transcriptCallback;
1810
+ private connectionClosedCallback;
1551
1811
  conversationId: string | null;
1552
1812
  constructor(options: NpcifyOptions);
1553
1813
  load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
@@ -1555,6 +1815,13 @@ declare class Character extends CharacterRuntime {
1555
1815
  reloadAnimations(info?: NpcifyAssetInfo): Promise<void>;
1556
1816
  reloadAssets(): Promise<void>;
1557
1817
  initializeRemote(): Promise<void>;
1818
+ /**
1819
+ * Warms up live motion for the current conversation, once, on the
1820
+ * player's intent: call it at the first focus or key in a reply field or
1821
+ * when a talk key goes down. It does nothing for a character without live
1822
+ * motion; the server limits warm-ups per player (ADR 0011).
1823
+ */
1824
+ prepareLiveMotion(): Promise<boolean>;
1558
1825
  createConversation(options?: {
1559
1826
  autoConnect?: boolean;
1560
1827
  autoPlay?: boolean;
@@ -1594,6 +1861,32 @@ declare class Character extends CharacterRuntime {
1594
1861
  onListeningState(callback: (state: ListeningState) => void): () => void;
1595
1862
  /** Partial text while the player speaks (`final: false`), then the final text. */
1596
1863
  onTranscript(callback: (event: TranscriptEvent) => void): () => void;
1864
+ /**
1865
+ * `'unavailable'` until the character info turns sight on (its owner's
1866
+ * Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
1867
+ * `'on'`.
1868
+ */
1869
+ get sightState(): SightState;
1870
+ /**
1871
+ * Lets the character see the player's camera (docs/sight.md). Needs a
1872
+ * conversation; the first call asks for the camera permission. Rejects
1873
+ * with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
1874
+ * `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
1875
+ */
1876
+ startSight(options?: SightOptions): Promise<void>;
1877
+ /** Stops the frames. Closes the camera that `startSight` opened. */
1878
+ stopSight(): void;
1879
+ /** What the character saw: each observation of the current conversation. */
1880
+ onSight(callback: (event: SightObservationEvent) => void): () => void;
1881
+ /** The sight state. `error` is the `SightError` that stopped sight. */
1882
+ onSightState(callback: (state: SightState, error?: SightError) => void): () => void;
1883
+ /**
1884
+ * The server closed the conversation connection: a refused or failed join,
1885
+ * or an API key that was rotated, edited or deleted. The next `connect`,
1886
+ * `sendMessage` or `startListening` opens a new connection; a refused
1887
+ * conversation fails at once with `CONVERSATION_ACCESS_DENIED`.
1888
+ */
1889
+ onConnectionClosed(callback: (event: ConversationConnectionClosedEvent) => void): () => void;
1597
1890
  getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
1598
1891
  acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
1599
1892
  playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
@@ -1608,14 +1901,6 @@ declare class Character extends CharacterRuntime {
1608
1901
  private resetAssets;
1609
1902
  private readonly getAssetInfo;
1610
1903
  private handleStagingResult;
1611
- /** Throws when the input modes do not include `mode`. */
1612
- private assertInputMode;
1613
- /**
1614
- * A typed reply needs the `text` mode; one of the current reply options
1615
- * needs `options` or `text`. Without a configuration the server's rules
1616
- * apply alone, as before.
1617
- */
1618
- private assertReplyMode;
1619
1904
  private invokeThinkingCallback;
1620
1905
  }
1621
1906
  declare class Npcify extends Character {
@@ -1727,4 +2012,4 @@ interface NpcifyView {
1727
2012
  /** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
1728
2013
  declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
1729
2014
 
1730
- export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
2015
+ export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SightError, type SightFailureCode, type SightObservation, type SightObservationEvent, type SightOptions, type SightPerson, type SightState, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };