@npcify/web 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -98,7 +98,9 @@ animate()
98
98
  - `character.attachTo(parent)` / `character.detach()`: Attach or detach from any `THREE.Object3D`.
99
99
  - `character.setShadows({ cast, receive })`: Configure shadow flags across all character meshes.
100
100
  - `character.setTransform({ position, rotation, scale })`: Set root transform with `Vector3`/`Euler` or `[x, y, z]` tuples.
101
- - `character.setGazeTarget(object3d)`: Make template-face eyes follow a `THREE.Camera` or `THREE.Object3D`.
101
+ - `character.setGazeTarget(object3d)`: Make template-face eyes follow a `THREE.Camera` or `THREE.Object3D` with their gaze cone and idle eye moves. Other faces do not follow it. `mountNpcify` sets its camera.
102
+ - `character.lookAt(object3d | [x, y, z] | { x, y, z } | x, y, z)`: Look at an object or a world position. A template face keeps its gaze cone, idle eye moves and staging look cues; other faces turn their eyes with three-vrm `vrm.lookAt.target`. Coordinates that are not finite numbers throw a `RangeError`.
103
+ - `character.clearLookAt()`: Remove the gaze target, so that the eyes look ahead.
102
104
  - **Audio & Web Audio (`character.audio`)**:
103
105
  - `character.audio.element`: Underlying `HTMLAudioElement | null`.
104
106
  - `character.audio.playing`, `paused`, `ended`, `currentTime`, `duration`, `unlocked`.
@@ -123,6 +125,9 @@ animate()
123
125
  - **Dialogue & Speech State**:
124
126
  - `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
125
127
  - `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
128
+ - **Voice Input** ([details](#voice-input-and-input-modes)):
129
+ - `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
130
+ - `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
126
131
  - `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
127
132
 
128
133
  ## Mode 2: Managed Canvas (`mountNpcify`)
@@ -186,13 +191,72 @@ with any ESM bundler after replacing their configuration placeholders. Keys in b
186
191
  code are visible to visitors. Use narrow scopes; keep keys in your backend when they
187
192
  must remain private. Calls made with a key are billed to its owner.
188
193
 
194
+ ## Voice input and input modes
195
+
196
+ The player can speak a reply. The SDK records the microphone and streams the
197
+ audio to the Npcify server. At the release, the server recognises the speech
198
+ and starts the turn. The recognised text is the player's message, as a typed
199
+ reply.
200
+
201
+ ```js
202
+ // Hold the button, or Space, to talk. The release sends the speech.
203
+ const talk = document.querySelector('#talk')
204
+ talk.hidden = !character.inputModes.includes('voice')
205
+ const unbindTalk = character.bindPushToTalk({
206
+ element: talk,
207
+ onResult: ({ text }) => { if (!text) status.textContent = 'No speech was heard.' },
208
+ onShortPress: () => { status.textContent = 'Hold to talk.' },
209
+ onError: (error) => { status.textContent = error.message },
210
+ })
211
+ character.onListeningState((state) => {
212
+ talk.textContent =
213
+ { listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
214
+ })
215
+ ```
216
+
217
+ - `bindPushToTalk({ element, key, minHoldMs, listening, enabled, onResult, onShortPress, onError })`:
218
+ press and hold an element, a key (`KeyboardEvent.code`, default `'Space'`;
219
+ `null` binds no key), or both. The key does not start while the focus is in
220
+ a field or on another control, and key repeat is ignored. A press shorter
221
+ than `minHoldMs` (300 ms), or a release before the microphone gave any
222
+ audio, sends nothing and calls `onShortPress`. Escape, window blur and a
223
+ hidden tab cancel. A long press on the element does not open a menu, select
224
+ text or scroll, and a release outside the element ends the press. Returns
225
+ an unbind function.
226
+ - `startListening({ language, send, maxDurationMs, endOnSilenceMs })` starts
227
+ listening from code, for example when the character stops speaking.
228
+ `endOnSilenceMs` stops after that much silence that follows speech, and
229
+ `maxDurationMs` (30 s at most) always stops. `stopListening()` resolves with
230
+ `{ text, language, durationMs, sent, turn, send() }`, or with null when the
231
+ microphone gave no audio yet. `cancelListening()` discards the recording;
232
+ nothing is charged.
233
+ - `onTranscript(({ text, final }) => …)` gives the final text, and partial
234
+ text when the server's speech model gives it.
235
+ - The first press or `startListening()` asks for the microphone permission.
236
+ An empty result sends nothing.
237
+ - `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
238
+ the server detects the language.
239
+ - `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
240
+ an array of `'options'`, `'text'` and `'voice'`; one mode forces it. Without
241
+ it, every mode that the character permits is available. A baked
242
+ conversation permits only `['options']`, and a character without custom
243
+ replies permits only `'options'`. Another configuration throws an
244
+ `InputModeError` when the character loads. With `inputModes` set,
245
+ `sendMessage` with typed text needs `'text'`, a reply option needs
246
+ `'options'` or `'text'`, and listening needs `'voice'`.
247
+ `character.inputModes` gives the modes in use.
248
+ - The character owner pays the recognition at the provider's price, as for
249
+ the turn's language model.
250
+
189
251
  ## Runtime and lifecycle
190
252
 
191
253
  - `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
192
- `{ character, npcify, scene, camera, renderer, controls, dispose }` after loading.
193
- Give the canvas a nonzero CSS size.
254
+ `{ character, npcify, scene, camera, renderer, controls, dispose }` after loading,
255
+ and normalizes the idle animation orientation to face `+Z` by default
256
+ (`normalizeIdleOrientation: true`). Give the canvas a nonzero CSS size.
194
257
  - `loadCharacter({ baseUrl, characterId, apiKey, parent, gazeTarget, shadows })`
195
- loads and returns a `Character` instance for custom Three.js scenes and render loops.
258
+ loads and returns a `Character` instance for custom Three.js scenes and render loops
259
+ (`normalizeIdleOrientation` defaults to `false`).
196
260
  - Call `unlockAudio()` (or `character.audio.unlock()`) during a user gesture before
197
261
  starting playback.
198
262
  - `onOptionsReady` delivers choices after playback finishes. `sendMessage(label)`
package/dist/index.d.ts CHANGED
@@ -1,10 +1,9 @@
1
1
  import { VRM, VRMExpressionPresetName, VRMHumanoid, VRMHumanBoneName } from '@pixiv/three-vrm';
2
2
  import { VRMAnimation } from '@pixiv/three-vrm-animation';
3
- import { AnimationMixer, AnimationAction, Object3D, Group, SkinnedMesh, Mesh, Material, Skeleton, Vector3, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
3
+ import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
4
4
  import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
5
5
 
6
- declare const TTS_PROVIDERS: readonly ["11labs"];
7
- type TtsProviderId = (typeof TTS_PROVIDERS)[number];
6
+ type TtsProviderId = string;
8
7
  type NullableTtsProviderId = TtsProviderId | null;
9
8
 
10
9
  declare const EXPRESSION_CALL_RANGES: readonly ["linear", "bool"];
@@ -40,6 +39,7 @@ interface AvatarPublicInfo {
40
39
  freeFirstNMessages?: number | null;
41
40
  hostedCharacterEnabled?: boolean;
42
41
  listedPublicly?: boolean;
42
+ r18?: boolean;
43
43
  bakedConversationEnabled?: boolean;
44
44
  bakedConversationRevision?: number;
45
45
  allowCustomReply?: boolean;
@@ -281,6 +281,18 @@ interface StagingStreamBillingEvent {
281
281
  }
282
282
  type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent;
283
283
 
284
+ /**
285
+ * Voice input of conversations (docs/voice-input.md): the input modes of the
286
+ * SDKs, and the messages of a speech session between an SDK and the server.
287
+ */
288
+ /** How a player can reply: a reply option, typed text, or speech. */
289
+ declare const INPUT_MODES: readonly ["options", "text", "voice"];
290
+ type InputMode = (typeof INPUT_MODES)[number];
291
+ /** A configuration of input modes that is invalid or that the character does not permit. */
292
+ declare class InputModeError extends Error {
293
+ constructor(message: string);
294
+ }
295
+
284
296
  declare const BAKED_CONVERSATION_VERSION: 1;
285
297
  /** Portable, language-neutral asset. Audio is embedded binary, never a signed URL. */
286
298
  interface BakedConversationAsset {
@@ -320,9 +332,11 @@ declare const decodePerformance: <T = unknown>(input: ArrayBuffer | Uint8Array,
320
332
  interface ConversationTurnOptions {
321
333
  silent?: boolean;
322
334
  parentStagingId?: string | null;
335
+ /** The turn's `Idempotency-Key`; a new UUID when absent. */
336
+ requestId?: string;
323
337
  }
324
338
 
325
- type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'>;
339
+ type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply'>>;
326
340
 
327
341
  interface LlmThinkingPayload {
328
342
  taskId?: string | null;
@@ -587,6 +601,13 @@ interface AnimationConfig {
587
601
  loop?: boolean;
588
602
  weight?: number;
589
603
  }
604
+ interface AnimationControllerOptions {
605
+ /**
606
+ * Normalize the configured idle animation clip so its average torso heading
607
+ * and neck/head orientation face canonical `+Z`.
608
+ */
609
+ normalizeIdleOrientation?: boolean;
610
+ }
590
611
  interface CharacterAnimationPlayOptions {
591
612
  fadeDuration?: number;
592
613
  loop?: boolean;
@@ -619,14 +640,17 @@ declare class AnimationController implements AnimationControllerLike {
619
640
  prepareAnimations?: (ids: string[]) => Promise<void>;
620
641
  private readonly mixer;
621
642
  private readonly clips;
643
+ private readonly rawClips;
644
+ private readonly normalizedIdleClips;
622
645
  private readonly configs;
646
+ private readonly normalizeIdleOrientation;
623
647
  private previewMixer;
624
648
  private restorePreview;
625
649
  private activeAction;
626
650
  private activeAnimationId;
627
651
  private idleAnimationId;
628
652
  private pendingReturnToIdleAction;
629
- constructor(vrm: VRM, idleAnimationId?: string | null);
653
+ constructor(vrm: VRM, idleAnimationId?: string | null, options?: AnimationControllerOptions);
630
654
  getAnimationMixer(): AnimationMixer;
631
655
  getActiveAction(): AnimationAction | null;
632
656
  getActiveAnimationId(): string | null;
@@ -637,6 +661,7 @@ declare class AnimationController implements AnimationControllerLike {
637
661
  previewAnimation(animation: VRMAnimation): () => void;
638
662
  setIdleAnimation(animationId: string | null): void;
639
663
  clearAnimations(): void;
664
+ private resolveClipForId;
640
665
  playAnimation(animationId: string, fadeDuration?: number, loopOverride?: boolean): void;
641
666
  playIdle(fadeDuration?: number): void;
642
667
  stopAnimation(fadeDuration?: number): void;
@@ -745,6 +770,7 @@ declare const FacialPerformanceLayer: {
745
770
  readonly Viseme: "viseme";
746
771
  readonly Blink: "blink";
747
772
  readonly Gaze: "gaze";
773
+ readonly Overlay: "overlay";
748
774
  };
749
775
  type FacialPerformanceLayerId = (typeof FacialPerformanceLayer)[keyof typeof FacialPerformanceLayer];
750
776
  type FacialMouthMode = 'expression' | 'viseme';
@@ -800,6 +826,9 @@ declare class FacialPerformanceMixer {
800
826
  playCue(channel: FacialChannelName, options?: FacialCueOptions): void;
801
827
  setEmotion(channel: FacialChannelName, weight: number, mouthMode: FacialMouthMode): void;
802
828
  clearEmotion(mouthMode?: FacialMouthMode): void;
829
+ /** Blend temporary emotions without changing timeline targets or mouth ownership. */
830
+ setOverlay(weights: Readonly<Record<string, number>>): void;
831
+ clearOverlay(): void;
803
832
  releaseVisemeMouthMode(): void;
804
833
  /**
805
834
  * Ends the speech source without disturbing any other performance layer.
@@ -949,6 +978,8 @@ declare class StagingPlayer {
949
978
  private readonly audioPlayer;
950
979
  private readonly facialMixer;
951
980
  private readonly animationController;
981
+ private readonly clock;
982
+ private pauseRevision;
952
983
  private playbackController;
953
984
  private generation;
954
985
  private incremental;
@@ -959,6 +990,10 @@ declare class StagingPlayer {
959
990
  private activeExpressionMouthMode;
960
991
  private activeAnimation;
961
992
  constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
993
+ get paused(): boolean;
994
+ pause(): void;
995
+ resume(): Promise<void>;
996
+ private waitUntilResumed;
962
997
  play(timeline: StagingTimeline, options?: StagingPlaybackOptions): Promise<void>;
963
998
  updateTimeline(timeline: StagingTimeline, complete?: boolean): void;
964
999
  private startCursor;
@@ -977,6 +1012,12 @@ declare class StagingPlayer {
977
1012
  }
978
1013
 
979
1014
  type Vector3Tuple = readonly [number, number, number];
1015
+ interface Vector3Like {
1016
+ readonly x: number;
1017
+ readonly y: number;
1018
+ readonly z: number;
1019
+ }
1020
+ type CharacterLookAtTarget = Object3D | Vector3 | Vector3Tuple | Vector3Like | null;
980
1021
  interface CharacterShadowOptions {
981
1022
  cast?: boolean;
982
1023
  receive?: boolean;
@@ -994,8 +1035,26 @@ declare class CharacterSceneRig {
994
1035
  private cachedSkeletons;
995
1036
  private shadowOptions;
996
1037
  private transformOptions;
1038
+ private lookAtAnchor;
1039
+ /** The target that setVrmLookAtTarget set on the current model. */
1040
+ private vrmLookAtTarget;
997
1041
  get vrm(): VRM | null;
998
1042
  setVrm(vrm: VRM | null): void;
1043
+ /**
1044
+ * The `Object3D` of a `lookAt()` argument: an `Object3D` as it is, a point
1045
+ * or three coordinates as a world position that this rig keeps, or null.
1046
+ * Coordinates that are not finite numbers throw a `RangeError` and change
1047
+ * nothing.
1048
+ */
1049
+ resolveLookAtTarget(targetOrX: CharacterLookAtTarget | number, y?: number, z?: number): Object3D | null;
1050
+ /**
1051
+ * Sets three-vrm's own look-at target (`vrm.lookAt.target`) of the current
1052
+ * model. Null removes the target that this method set; a target that other
1053
+ * code set stays. The IdleFace of a template face does not move the eyes
1054
+ * while `vrm.lookAt.target` is set.
1055
+ */
1056
+ setVrmLookAtTarget(target: Object3D | null): void;
1057
+ private updateLookAtAnchor;
999
1058
  /** Root Three.js group containing the VRM hierarchy (`vrm.scene`). */
1000
1059
  get scene(): Group | null;
1001
1060
  /** Alias for `scene` for Three.js `Object3D` workflows. */
@@ -1080,6 +1139,21 @@ interface NpcifyOptions {
1080
1139
  volume?: number;
1081
1140
  /** Initial speech audio muted flag. */
1082
1141
  muted?: boolean;
1142
+ /**
1143
+ * Normalize the character's idle animation orientation so the torso and head
1144
+ * face canonical `+Z` on average while preserving idle motion.
1145
+ * Defaults to `true` for `Npcify` (Web Chats and Canvas API) and `false` for
1146
+ * `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
1147
+ */
1148
+ normalizeIdleOrientation?: boolean;
1149
+ /**
1150
+ * How the player can reply: `'options'`, `'text'` and `'voice'`, in any
1151
+ * combination; one mode forces it. Without it, every mode that the
1152
+ * character permits. A baked conversation permits only `['options']`; a
1153
+ * character without custom replies permits only `options`. A mode that the
1154
+ * character does not permit throws an `InputModeError` on load.
1155
+ */
1156
+ inputModes?: InputMode[];
1083
1157
  /** Optional error callback. */
1084
1158
  onError?: (error: Error) => void;
1085
1159
  }
@@ -1146,7 +1220,8 @@ declare class StreamPlaybackController {
1146
1220
  force?: boolean;
1147
1221
  timeline?: StagingTimeline;
1148
1222
  }): Promise<boolean>;
1149
- handleEvent(event: StagingStreamEvent): void;
1223
+ /** Returns the playback result for a timeline event. */
1224
+ handleEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
1150
1225
  cleanup(): void;
1151
1226
  private resetRealtimeSpeech;
1152
1227
  private playTimeline;
@@ -1187,6 +1262,10 @@ declare class CharacterExpressions {
1187
1262
  setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
1188
1263
  /** Clear the active emotion expression layer. */
1189
1264
  clearEmotion(mouthMode?: FacialMouthMode): void;
1265
+ /** Set a temporary emotion overlay. Timeline emotions and visemes keep their state. */
1266
+ setOverlay(weights: Readonly<Record<string, number>>): void;
1267
+ /** Ease the temporary overlay back to the current timeline or idle face. */
1268
+ clearOverlay(): void;
1190
1269
  /** Reset all expression layer targets to zero. */
1191
1270
  reset(): void;
1192
1271
  }
@@ -1217,6 +1296,8 @@ declare class CharacterRuntime {
1217
1296
  protected readonly streamPlayback: StreamPlaybackController;
1218
1297
  protected defaultParent: Object3D | null;
1219
1298
  protected currentGazeTarget: Object3D | null;
1299
+ /** The current gaze target came from `lookAt()`. */
1300
+ protected lookAtActive: boolean;
1220
1301
  protected thinkingState: boolean;
1221
1302
  protected errorCallback: ((error: Error) => void) | null;
1222
1303
  constructor(options: CharacterRuntimeInitOptions);
@@ -1288,6 +1369,12 @@ declare class CharacterRuntime {
1288
1369
  /** Stop the currently playing skeletal animation. */
1289
1370
  stopAnimation(fadeDuration?: number): void;
1290
1371
  unlockAudio(): Promise<boolean>;
1372
+ /** Freeze the current NPCP timeline, audio, face, and animation. */
1373
+ pausePlayback(): void;
1374
+ /** Continue the same timeline. Call from a gesture if the browser requires it. */
1375
+ resumePlayback(): Promise<void>;
1376
+ /** Cancel current output and pending audio/animation loads. */
1377
+ interruptPlayback(): void;
1291
1378
  getExpressions(): string[];
1292
1379
  getMouthExpressionNames(): string[];
1293
1380
  getExpressionNames(): string[];
@@ -1296,8 +1383,24 @@ declare class CharacterRuntime {
1296
1383
  getExpressionWeight(name: FacialChannelName): number;
1297
1384
  setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
1298
1385
  clearEmotion(mouthMode?: FacialMouthMode): void;
1299
- /** The object the eyes of a template face follow (the camera); null looks ahead. */
1386
+ /**
1387
+ * The object the eyes of a template face follow (the camera); null looks
1388
+ * ahead. It replaces a `lookAt()` target. Other faces do not follow it.
1389
+ */
1300
1390
  setGazeTarget(target: Object3D | null): void;
1391
+ /**
1392
+ * Direct the character's gaze toward a Three.js `Object3D` or world
1393
+ * coordinates. A template face keeps its gaze cone, idle eye moves and the
1394
+ * look cues of staging; other faces turn their eyes with three-vrm's
1395
+ * `vrm.lookAt.target`. Coordinates must be finite numbers, else this throws
1396
+ * a `RangeError` and keeps the current target.
1397
+ */
1398
+ lookAt(target: CharacterLookAtTarget): void;
1399
+ lookAt(x: number, y: number, z: number): void;
1400
+ /** Reset the character's gaze target so the eyes look straight ahead. */
1401
+ clearLookAt(): void;
1402
+ /** Briefly look at an object; release restores the prior target and gaze mode. */
1403
+ lookAtTemporarily(target: Object3D): () => void;
1301
1404
  startPerformanceRecording(options?: PerformanceRecordingStartOptions): void;
1302
1405
  stopPerformanceRecording(): PerformanceRecording | null;
1303
1406
  getPerformanceRecording(): PerformanceRecording | null;
@@ -1314,12 +1417,110 @@ declare class CharacterRuntime {
1314
1417
  update(delta: number): void;
1315
1418
  protected readonly getAudioPlayer: () => AudioPlayer;
1316
1419
  protected readonly getStagingPlayer: () => StagingPlayer;
1420
+ /**
1421
+ * A template face follows the gaze target with its IdleFace. Only a
1422
+ * `lookAt()` target moves the eyes of another face, through three-vrm's
1423
+ * `vrm.lookAt.target`; on a template face that target would stop the
1424
+ * IdleFace (its gaze cone, idle eye moves and look cues).
1425
+ */
1426
+ protected applyGaze(): void;
1317
1427
  protected resetRuntimeAssets(): void;
1318
1428
  protected disposeRuntimeAssets(): void;
1319
1429
  protected readonly handleCallbackError: (context: string, error: unknown) => void;
1320
1430
  protected readonly reportError: (error: Error, context?: string) => void;
1321
1431
  }
1322
1432
 
1433
+ declare class MicrophoneError extends Error {
1434
+ readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
1435
+ constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
1436
+ }
1437
+ /** One frame of 16 kHz mono PCM and its loudness (RMS, 0 to 1). */
1438
+ type CaptureFrame = {
1439
+ pcm: Int16Array;
1440
+ level: number;
1441
+ };
1442
+ /** A source of 16 kHz frames. The SDK uses the microphone; tests use a fake. */
1443
+ interface AudioCapture {
1444
+ start(onFrame: (frame: CaptureFrame) => void): Promise<void>;
1445
+ /** Ends the capture; the last partial frame arrives before this resolves. */
1446
+ stop(): Promise<void>;
1447
+ /** Ends the capture without its last frame. */
1448
+ cancel(): void;
1449
+ }
1450
+
1451
+ type ListeningState = 'idle' | 'starting' | 'listening' | 'recognizing';
1452
+ interface ListeningOptions {
1453
+ /** BCP 47 hint, for example `ja`, `zh` or `en`. Without it the language is detected. */
1454
+ language?: string | null;
1455
+ /** `false`: recognise only; send the text later with `result.send()`. Default `true`. */
1456
+ send?: boolean;
1457
+ /** The recording stops at this time. Default and maximum: 30 seconds. */
1458
+ maxDurationMs?: number;
1459
+ /** For listening without a release: stop after this much silence that follows speech. */
1460
+ endOnSilenceMs?: number | null;
1461
+ parentStagingId?: string | null;
1462
+ silent?: boolean;
1463
+ }
1464
+ interface SpeechInputResult {
1465
+ /** The recognised text; empty when no speech was recognised (then no turn starts). */
1466
+ text: string;
1467
+ /** The detected language, when the recognition vendor gives it. */
1468
+ language: string | null;
1469
+ durationMs: number;
1470
+ /** The server started the turn with this text. */
1471
+ sent: boolean;
1472
+ /** The final payload of the turn, when it was sent. */
1473
+ turn: Promise<StagingResultPayload> | null;
1474
+ /** Sends the text as the player's message, when it was not sent. */
1475
+ send(): Promise<StagingResultPayload>;
1476
+ }
1477
+ interface TranscriptEvent {
1478
+ text: string;
1479
+ /** False for the text so far while the player speaks. */
1480
+ final: boolean;
1481
+ }
1482
+ declare class SpeechInputError extends Error {
1483
+ readonly code: string;
1484
+ constructor(code: string, message?: string);
1485
+ }
1486
+
1487
+ /** What push-to-talk drives: a `Character` or `Npcify` instance. */
1488
+ interface PushToTalkTarget {
1489
+ readonly listeningState: ListeningState;
1490
+ startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
1491
+ stopListening(): Promise<SpeechInputResult | null>;
1492
+ cancelListening(): void;
1493
+ }
1494
+ interface PushToTalkOptions {
1495
+ /** A button or any element to press and hold. */
1496
+ element?: HTMLElement | null;
1497
+ /** A `KeyboardEvent.code` to hold. Default `'Space'`; `null` binds no key. */
1498
+ key?: string | null;
1499
+ /** A shorter press sends nothing. Default 300 ms. */
1500
+ minHoldMs?: number;
1501
+ /** The options of each listening, or a function that gives them at each press. */
1502
+ listening?: ListeningOptions | (() => ListeningOptions);
1503
+ /** False while the player cannot reply (for example while options are locked). */
1504
+ enabled?: () => boolean;
1505
+ onResult?: (result: SpeechInputResult) => void;
1506
+ /**
1507
+ * A press sent nothing: it was shorter than `minHoldMs`, or it ended before
1508
+ * the microphone gave any audio (a permission prompt, a slow device). A UI
1509
+ * can say "hold to talk".
1510
+ */
1511
+ onShortPress?: () => void;
1512
+ onError?: (error: unknown) => void;
1513
+ }
1514
+ /**
1515
+ * Press and hold to talk (docs/voice-input.md): an element, a key, or both.
1516
+ * The key does not start while the focus is in a field or on another
1517
+ * control, key repeat is ignored, a short press sends nothing, and window
1518
+ * blur, a hidden tab and Escape cancel. A long press on the element does not
1519
+ * open a menu, select text or scroll. A release anywhere ends the press.
1520
+ * Returns a function that unbinds.
1521
+ */
1522
+ declare const bindPushToTalk: (target: PushToTalkTarget, options?: PushToTalkOptions) => () => void;
1523
+
1323
1524
  declare class Character extends CharacterRuntime {
1324
1525
  private readonly conversationSocket;
1325
1526
  private renderSessionId;
@@ -1330,8 +1531,15 @@ declare class Character extends CharacterRuntime {
1330
1531
  private readonly liveConversation;
1331
1532
  private baked;
1332
1533
  private readonly authoring;
1534
+ protected readonly normalizeIdleOrientation: boolean;
1333
1535
  private readonly bakedAsset?;
1334
1536
  private animationRefreshId;
1537
+ private readonly voice;
1538
+ private readonly configuredInputModes;
1539
+ private resolvedInputModes;
1540
+ private latestOptions;
1541
+ private listeningStateCallback;
1542
+ private transcriptCallback;
1335
1543
  conversationId: string | null;
1336
1544
  constructor(options: NpcifyOptions);
1337
1545
  load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
@@ -1353,18 +1561,57 @@ declare class Character extends CharacterRuntime {
1353
1561
  replayStaging(stagingId: string): Promise<StagingResultPayload>;
1354
1562
  replayStaging(conversationId: string, stagingId: string, options?: PlayStagingOptions): Promise<StagingResultPayload>;
1355
1563
  get isBakedConversation(): boolean;
1564
+ /**
1565
+ * The input modes in use: the configured modes, or every mode that the
1566
+ * character permits. Before `loadAssets`, the configuration or all modes.
1567
+ */
1568
+ get inputModes(): readonly InputMode[];
1569
+ get listeningState(): ListeningState;
1570
+ /**
1571
+ * Opens the microphone and streams the speech to the server. Resolves with
1572
+ * the result when listening ends (`stopListening`, `endOnSilenceMs` or
1573
+ * `maxDurationMs`), or with null when it is cancelled. The first call asks
1574
+ * for the microphone permission.
1575
+ */
1576
+ startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
1577
+ /**
1578
+ * Ends the recording; resolves as soon as the server recognised the text.
1579
+ * Before the microphone gave any audio, it cancels and resolves null.
1580
+ */
1581
+ stopListening(): Promise<SpeechInputResult | null>;
1582
+ /** Discards the recording. Nothing is recognised or charged. */
1583
+ cancelListening(): void;
1584
+ /** Press and hold an element or a key (default Space) to talk. Returns an unbind function. */
1585
+ bindPushToTalk(options?: PushToTalkOptions): () => void;
1586
+ onListeningState(callback: (state: ListeningState) => void): () => void;
1587
+ /** Partial text while the player speaks (`final: false`), then the final text. */
1588
+ onTranscript(callback: (event: TranscriptEvent) => void): () => void;
1356
1589
  getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
1357
1590
  acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
1358
1591
  playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
1592
+ /**
1593
+ * Plays one live staging event from another transport, such as an actor
1594
+ * stage. A timeline event returns its playback result.
1595
+ */
1596
+ playStreamEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
1359
1597
  setRenderSessionId(renderSessionId: string | null): void;
1360
1598
  onStagingResult(callback: (payload: StagingResultPayload) => Promise<void> | void): () => void;
1361
1599
  onThinking(callback: (payload: LlmThinkingPayload) => Promise<void> | void): () => void;
1362
1600
  private resetAssets;
1363
1601
  private readonly getAssetInfo;
1364
1602
  private handleStagingResult;
1603
+ /** Throws when the input modes do not include `mode`. */
1604
+ private assertInputMode;
1605
+ /**
1606
+ * A typed reply needs the `text` mode; one of the current reply options
1607
+ * needs `options` or `text`. Without a configuration the server's rules
1608
+ * apply alone, as before.
1609
+ */
1610
+ private assertReplyMode;
1365
1611
  private invokeThinkingCallback;
1366
1612
  }
1367
1613
  declare class Npcify extends Character {
1614
+ constructor(options: NpcifyOptions);
1368
1615
  }
1369
1616
 
1370
1617
  interface LoadCharacterOptions extends CharacterOptions {
@@ -1455,6 +1702,9 @@ interface MountNpcifyOptions {
1455
1702
  apiToken?: string;
1456
1703
  serverRendering?: boolean;
1457
1704
  assetProfile?: 'original' | 'web-optimized-v1';
1705
+ normalizeIdleOrientation?: boolean;
1706
+ /** How the player can reply: `'options'`, `'text'`, `'voice'` (docs/voice-input.md). */
1707
+ inputModes?: InputMode[];
1458
1708
  onError?: (error: Error) => void;
1459
1709
  }
1460
1710
  interface NpcifyView {
@@ -1469,4 +1719,4 @@ interface NpcifyView {
1469
1719
  /** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
1470
1720
  declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
1471
1721
 
1472
- export { AnimationController, type AnimationControllerLike, type AssetLoadProgress, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, type LlmThinkingPayload, type LoadCharacterOptions, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
1722
+ export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };