@npcify/web 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -4
- package/dist/index.d.ts +258 -8
- package/dist/index.js +2085 -824
- package/examples/index.html +1 -0
- package/examples/main.js +40 -12
- package/examples/three-game.js +13 -3
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -98,7 +98,9 @@ animate()
|
|
|
98
98
|
- `character.attachTo(parent)` / `character.detach()`: Attach or detach from any `THREE.Object3D`.
|
|
99
99
|
- `character.setShadows({ cast, receive })`: Configure shadow flags across all character meshes.
|
|
100
100
|
- `character.setTransform({ position, rotation, scale })`: Set root transform with `Vector3`/`Euler` or `[x, y, z]` tuples.
|
|
101
|
-
- `character.setGazeTarget(object3d)`: Make template-face eyes follow a `THREE.Camera` or `THREE.Object3D
|
|
101
|
+
- `character.setGazeTarget(object3d)`: Make template-face eyes follow a `THREE.Camera` or `THREE.Object3D` with their gaze cone and idle eye moves. Other faces do not follow it. `mountNpcify` sets its camera.
|
|
102
|
+
- `character.lookAt(object3d | [x, y, z] | { x, y, z } | x, y, z)`: Look at an object or a world position. A template face keeps its gaze cone, idle eye moves and staging look cues; other faces turn their eyes with three-vrm `vrm.lookAt.target`. Coordinates that are not finite numbers throw a `RangeError`.
|
|
103
|
+
- `character.clearLookAt()`: Remove the gaze target, so that the eyes look ahead.
|
|
102
104
|
- **Audio & Web Audio (`character.audio`)**:
|
|
103
105
|
- `character.audio.element`: Underlying `HTMLAudioElement | null`.
|
|
104
106
|
- `character.audio.playing`, `paused`, `ended`, `currentTime`, `duration`, `unlocked`.
|
|
@@ -123,6 +125,9 @@ animate()
|
|
|
123
125
|
- **Dialogue & Speech State**:
|
|
124
126
|
- `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
|
|
125
127
|
- `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
|
|
128
|
+
- **Voice Input** ([details](#voice-input-and-input-modes)):
|
|
129
|
+
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
130
|
+
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
126
131
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
127
132
|
|
|
128
133
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
@@ -186,13 +191,72 @@ with any ESM bundler after replacing their configuration placeholders. Keys in b
|
|
|
186
191
|
code are visible to visitors. Use narrow scopes; keep keys in your backend when they
|
|
187
192
|
must remain private. Calls made with a key are billed to its owner.
|
|
188
193
|
|
|
194
|
+
## Voice input and input modes
|
|
195
|
+
|
|
196
|
+
The player can speak a reply. The SDK records the microphone and streams the
|
|
197
|
+
audio to the Npcify server. At the release, the server recognises the speech
|
|
198
|
+
and starts the turn. The recognised text is the player's message, as a typed
|
|
199
|
+
reply.
|
|
200
|
+
|
|
201
|
+
```js
|
|
202
|
+
// Hold the button, or Space, to talk. The release sends the speech.
|
|
203
|
+
const talk = document.querySelector('#talk')
|
|
204
|
+
talk.hidden = !character.inputModes.includes('voice')
|
|
205
|
+
const unbindTalk = character.bindPushToTalk({
|
|
206
|
+
element: talk,
|
|
207
|
+
onResult: ({ text }) => { if (!text) status.textContent = 'No speech was heard.' },
|
|
208
|
+
onShortPress: () => { status.textContent = 'Hold to talk.' },
|
|
209
|
+
onError: (error) => { status.textContent = error.message },
|
|
210
|
+
})
|
|
211
|
+
character.onListeningState((state) => {
|
|
212
|
+
talk.textContent =
|
|
213
|
+
{ listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
|
|
214
|
+
})
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
- `bindPushToTalk({ element, key, minHoldMs, listening, enabled, onResult, onShortPress, onError })`:
|
|
218
|
+
press and hold an element, a key (`KeyboardEvent.code`, default `'Space'`;
|
|
219
|
+
`null` binds no key), or both. The key does not start while the focus is in
|
|
220
|
+
a field or on another control, and key repeat is ignored. A press shorter
|
|
221
|
+
than `minHoldMs` (300 ms), or a release before the microphone gave any
|
|
222
|
+
audio, sends nothing and calls `onShortPress`. Escape, window blur and a
|
|
223
|
+
hidden tab cancel. A long press on the element does not open a menu, select
|
|
224
|
+
text or scroll, and a release outside the element ends the press. Returns
|
|
225
|
+
an unbind function.
|
|
226
|
+
- `startListening({ language, send, maxDurationMs, endOnSilenceMs })` starts
|
|
227
|
+
listening from code, for example when the character stops speaking.
|
|
228
|
+
`endOnSilenceMs` stops after that much silence that follows speech, and
|
|
229
|
+
`maxDurationMs` (30 s at most) always stops. `stopListening()` resolves with
|
|
230
|
+
`{ text, language, durationMs, sent, turn, send() }`, or with null when the
|
|
231
|
+
microphone gave no audio yet. `cancelListening()` discards the recording;
|
|
232
|
+
nothing is charged.
|
|
233
|
+
- `onTranscript(({ text, final }) => …)` gives the final text, and partial
|
|
234
|
+
text when the server's speech model gives it.
|
|
235
|
+
- The first press or `startListening()` asks for the microphone permission.
|
|
236
|
+
An empty result sends nothing.
|
|
237
|
+
- `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
|
|
238
|
+
the server detects the language.
|
|
239
|
+
- `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
|
|
240
|
+
an array of `'options'`, `'text'` and `'voice'`; one mode forces it. Without
|
|
241
|
+
it, every mode that the character permits is available. A baked
|
|
242
|
+
conversation permits only `['options']`, and a character without custom
|
|
243
|
+
replies permits only `'options'`. Another configuration throws an
|
|
244
|
+
`InputModeError` when the character loads. With `inputModes` set,
|
|
245
|
+
`sendMessage` with typed text needs `'text'`, a reply option needs
|
|
246
|
+
`'options'` or `'text'`, and listening needs `'voice'`.
|
|
247
|
+
`character.inputModes` gives the modes in use.
|
|
248
|
+
- The character owner pays the recognition at the provider's price, as for
|
|
249
|
+
the turn's language model.
|
|
250
|
+
|
|
189
251
|
## Runtime and lifecycle
|
|
190
252
|
|
|
191
253
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|
|
192
|
-
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading
|
|
193
|
-
|
|
254
|
+
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading,
|
|
255
|
+
and normalizes the idle animation orientation to face `+Z` by default
|
|
256
|
+
(`normalizeIdleOrientation: true`). Give the canvas a nonzero CSS size.
|
|
194
257
|
- `loadCharacter({ baseUrl, characterId, apiKey, parent, gazeTarget, shadows })`
|
|
195
|
-
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
258
|
+
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
259
|
+
(`normalizeIdleOrientation` defaults to `false`).
|
|
196
260
|
- Call `unlockAudio()` (or `character.audio.unlock()`) during a user gesture before
|
|
197
261
|
starting playback.
|
|
198
262
|
- `onOptionsReady` delivers choices after playback finishes. `sendMessage(label)`
|
package/dist/index.d.ts
CHANGED
|
@@ -1,10 +1,9 @@
|
|
|
1
1
|
import { VRM, VRMExpressionPresetName, VRMHumanoid, VRMHumanBoneName } from '@pixiv/three-vrm';
|
|
2
2
|
import { VRMAnimation } from '@pixiv/three-vrm-animation';
|
|
3
|
-
import { AnimationMixer, AnimationAction, Object3D, Group, SkinnedMesh, Mesh, Material, Skeleton,
|
|
3
|
+
import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
|
|
4
4
|
import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
|
|
5
5
|
|
|
6
|
-
|
|
7
|
-
type TtsProviderId = (typeof TTS_PROVIDERS)[number];
|
|
6
|
+
type TtsProviderId = string;
|
|
8
7
|
type NullableTtsProviderId = TtsProviderId | null;
|
|
9
8
|
|
|
10
9
|
declare const EXPRESSION_CALL_RANGES: readonly ["linear", "bool"];
|
|
@@ -40,6 +39,7 @@ interface AvatarPublicInfo {
|
|
|
40
39
|
freeFirstNMessages?: number | null;
|
|
41
40
|
hostedCharacterEnabled?: boolean;
|
|
42
41
|
listedPublicly?: boolean;
|
|
42
|
+
r18?: boolean;
|
|
43
43
|
bakedConversationEnabled?: boolean;
|
|
44
44
|
bakedConversationRevision?: number;
|
|
45
45
|
allowCustomReply?: boolean;
|
|
@@ -281,6 +281,18 @@ interface StagingStreamBillingEvent {
|
|
|
281
281
|
}
|
|
282
282
|
type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent;
|
|
283
283
|
|
|
284
|
+
/**
|
|
285
|
+
* Voice input of conversations (docs/voice-input.md): the input modes of the
|
|
286
|
+
* SDKs, and the messages of a speech session between an SDK and the server.
|
|
287
|
+
*/
|
|
288
|
+
/** How a player can reply: a reply option, typed text, or speech. */
|
|
289
|
+
declare const INPUT_MODES: readonly ["options", "text", "voice"];
|
|
290
|
+
type InputMode = (typeof INPUT_MODES)[number];
|
|
291
|
+
/** A configuration of input modes that is invalid or that the character does not permit. */
|
|
292
|
+
declare class InputModeError extends Error {
|
|
293
|
+
constructor(message: string);
|
|
294
|
+
}
|
|
295
|
+
|
|
284
296
|
declare const BAKED_CONVERSATION_VERSION: 1;
|
|
285
297
|
/** Portable, language-neutral asset. Audio is embedded binary, never a signed URL. */
|
|
286
298
|
interface BakedConversationAsset {
|
|
@@ -320,9 +332,11 @@ declare const decodePerformance: <T = unknown>(input: ArrayBuffer | Uint8Array,
|
|
|
320
332
|
interface ConversationTurnOptions {
|
|
321
333
|
silent?: boolean;
|
|
322
334
|
parentStagingId?: string | null;
|
|
335
|
+
/** The turn's `Idempotency-Key`; a new UUID when absent. */
|
|
336
|
+
requestId?: string;
|
|
323
337
|
}
|
|
324
338
|
|
|
325
|
-
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'
|
|
339
|
+
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply'>>;
|
|
326
340
|
|
|
327
341
|
interface LlmThinkingPayload {
|
|
328
342
|
taskId?: string | null;
|
|
@@ -587,6 +601,13 @@ interface AnimationConfig {
|
|
|
587
601
|
loop?: boolean;
|
|
588
602
|
weight?: number;
|
|
589
603
|
}
|
|
604
|
+
interface AnimationControllerOptions {
|
|
605
|
+
/**
|
|
606
|
+
* Normalize the configured idle animation clip so its average torso heading
|
|
607
|
+
* and neck/head orientation face canonical `+Z`.
|
|
608
|
+
*/
|
|
609
|
+
normalizeIdleOrientation?: boolean;
|
|
610
|
+
}
|
|
590
611
|
interface CharacterAnimationPlayOptions {
|
|
591
612
|
fadeDuration?: number;
|
|
592
613
|
loop?: boolean;
|
|
@@ -619,14 +640,17 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
619
640
|
prepareAnimations?: (ids: string[]) => Promise<void>;
|
|
620
641
|
private readonly mixer;
|
|
621
642
|
private readonly clips;
|
|
643
|
+
private readonly rawClips;
|
|
644
|
+
private readonly normalizedIdleClips;
|
|
622
645
|
private readonly configs;
|
|
646
|
+
private readonly normalizeIdleOrientation;
|
|
623
647
|
private previewMixer;
|
|
624
648
|
private restorePreview;
|
|
625
649
|
private activeAction;
|
|
626
650
|
private activeAnimationId;
|
|
627
651
|
private idleAnimationId;
|
|
628
652
|
private pendingReturnToIdleAction;
|
|
629
|
-
constructor(vrm: VRM, idleAnimationId?: string | null);
|
|
653
|
+
constructor(vrm: VRM, idleAnimationId?: string | null, options?: AnimationControllerOptions);
|
|
630
654
|
getAnimationMixer(): AnimationMixer;
|
|
631
655
|
getActiveAction(): AnimationAction | null;
|
|
632
656
|
getActiveAnimationId(): string | null;
|
|
@@ -637,6 +661,7 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
637
661
|
previewAnimation(animation: VRMAnimation): () => void;
|
|
638
662
|
setIdleAnimation(animationId: string | null): void;
|
|
639
663
|
clearAnimations(): void;
|
|
664
|
+
private resolveClipForId;
|
|
640
665
|
playAnimation(animationId: string, fadeDuration?: number, loopOverride?: boolean): void;
|
|
641
666
|
playIdle(fadeDuration?: number): void;
|
|
642
667
|
stopAnimation(fadeDuration?: number): void;
|
|
@@ -745,6 +770,7 @@ declare const FacialPerformanceLayer: {
|
|
|
745
770
|
readonly Viseme: "viseme";
|
|
746
771
|
readonly Blink: "blink";
|
|
747
772
|
readonly Gaze: "gaze";
|
|
773
|
+
readonly Overlay: "overlay";
|
|
748
774
|
};
|
|
749
775
|
type FacialPerformanceLayerId = (typeof FacialPerformanceLayer)[keyof typeof FacialPerformanceLayer];
|
|
750
776
|
type FacialMouthMode = 'expression' | 'viseme';
|
|
@@ -800,6 +826,9 @@ declare class FacialPerformanceMixer {
|
|
|
800
826
|
playCue(channel: FacialChannelName, options?: FacialCueOptions): void;
|
|
801
827
|
setEmotion(channel: FacialChannelName, weight: number, mouthMode: FacialMouthMode): void;
|
|
802
828
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
829
|
+
/** Blend temporary emotions without changing timeline targets or mouth ownership. */
|
|
830
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
831
|
+
clearOverlay(): void;
|
|
803
832
|
releaseVisemeMouthMode(): void;
|
|
804
833
|
/**
|
|
805
834
|
* Ends the speech source without disturbing any other performance layer.
|
|
@@ -949,6 +978,8 @@ declare class StagingPlayer {
|
|
|
949
978
|
private readonly audioPlayer;
|
|
950
979
|
private readonly facialMixer;
|
|
951
980
|
private readonly animationController;
|
|
981
|
+
private readonly clock;
|
|
982
|
+
private pauseRevision;
|
|
952
983
|
private playbackController;
|
|
953
984
|
private generation;
|
|
954
985
|
private incremental;
|
|
@@ -959,6 +990,10 @@ declare class StagingPlayer {
|
|
|
959
990
|
private activeExpressionMouthMode;
|
|
960
991
|
private activeAnimation;
|
|
961
992
|
constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
|
|
993
|
+
get paused(): boolean;
|
|
994
|
+
pause(): void;
|
|
995
|
+
resume(): Promise<void>;
|
|
996
|
+
private waitUntilResumed;
|
|
962
997
|
play(timeline: StagingTimeline, options?: StagingPlaybackOptions): Promise<void>;
|
|
963
998
|
updateTimeline(timeline: StagingTimeline, complete?: boolean): void;
|
|
964
999
|
private startCursor;
|
|
@@ -977,6 +1012,12 @@ declare class StagingPlayer {
|
|
|
977
1012
|
}
|
|
978
1013
|
|
|
979
1014
|
type Vector3Tuple = readonly [number, number, number];
|
|
1015
|
+
interface Vector3Like {
|
|
1016
|
+
readonly x: number;
|
|
1017
|
+
readonly y: number;
|
|
1018
|
+
readonly z: number;
|
|
1019
|
+
}
|
|
1020
|
+
type CharacterLookAtTarget = Object3D | Vector3 | Vector3Tuple | Vector3Like | null;
|
|
980
1021
|
interface CharacterShadowOptions {
|
|
981
1022
|
cast?: boolean;
|
|
982
1023
|
receive?: boolean;
|
|
@@ -994,8 +1035,26 @@ declare class CharacterSceneRig {
|
|
|
994
1035
|
private cachedSkeletons;
|
|
995
1036
|
private shadowOptions;
|
|
996
1037
|
private transformOptions;
|
|
1038
|
+
private lookAtAnchor;
|
|
1039
|
+
/** The target that setVrmLookAtTarget set on the current model. */
|
|
1040
|
+
private vrmLookAtTarget;
|
|
997
1041
|
get vrm(): VRM | null;
|
|
998
1042
|
setVrm(vrm: VRM | null): void;
|
|
1043
|
+
/**
|
|
1044
|
+
* The `Object3D` of a `lookAt()` argument: an `Object3D` as it is, a point
|
|
1045
|
+
* or three coordinates as a world position that this rig keeps, or null.
|
|
1046
|
+
* Coordinates that are not finite numbers throw a `RangeError` and change
|
|
1047
|
+
* nothing.
|
|
1048
|
+
*/
|
|
1049
|
+
resolveLookAtTarget(targetOrX: CharacterLookAtTarget | number, y?: number, z?: number): Object3D | null;
|
|
1050
|
+
/**
|
|
1051
|
+
* Sets three-vrm's own look-at target (`vrm.lookAt.target`) of the current
|
|
1052
|
+
* model. Null removes the target that this method set; a target that other
|
|
1053
|
+
* code set stays. The IdleFace of a template face does not move the eyes
|
|
1054
|
+
* while `vrm.lookAt.target` is set.
|
|
1055
|
+
*/
|
|
1056
|
+
setVrmLookAtTarget(target: Object3D | null): void;
|
|
1057
|
+
private updateLookAtAnchor;
|
|
999
1058
|
/** Root Three.js group containing the VRM hierarchy (`vrm.scene`). */
|
|
1000
1059
|
get scene(): Group | null;
|
|
1001
1060
|
/** Alias for `scene` for Three.js `Object3D` workflows. */
|
|
@@ -1080,6 +1139,21 @@ interface NpcifyOptions {
|
|
|
1080
1139
|
volume?: number;
|
|
1081
1140
|
/** Initial speech audio muted flag. */
|
|
1082
1141
|
muted?: boolean;
|
|
1142
|
+
/**
|
|
1143
|
+
* Normalize the character's idle animation orientation so the torso and head
|
|
1144
|
+
* face canonical `+Z` on average while preserving idle motion.
|
|
1145
|
+
* Defaults to `true` for `Npcify` (Web Chats and Canvas API) and `false` for
|
|
1146
|
+
* `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
|
|
1147
|
+
*/
|
|
1148
|
+
normalizeIdleOrientation?: boolean;
|
|
1149
|
+
/**
|
|
1150
|
+
* How the player can reply: `'options'`, `'text'` and `'voice'`, in any
|
|
1151
|
+
* combination; one mode forces it. Without it, every mode that the
|
|
1152
|
+
* character permits. A baked conversation permits only `['options']`; a
|
|
1153
|
+
* character without custom replies permits only `options`. A mode that the
|
|
1154
|
+
* character does not permit throws an `InputModeError` on load.
|
|
1155
|
+
*/
|
|
1156
|
+
inputModes?: InputMode[];
|
|
1083
1157
|
/** Optional error callback. */
|
|
1084
1158
|
onError?: (error: Error) => void;
|
|
1085
1159
|
}
|
|
@@ -1146,7 +1220,8 @@ declare class StreamPlaybackController {
|
|
|
1146
1220
|
force?: boolean;
|
|
1147
1221
|
timeline?: StagingTimeline;
|
|
1148
1222
|
}): Promise<boolean>;
|
|
1149
|
-
|
|
1223
|
+
/** Returns the playback result for a timeline event. */
|
|
1224
|
+
handleEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1150
1225
|
cleanup(): void;
|
|
1151
1226
|
private resetRealtimeSpeech;
|
|
1152
1227
|
private playTimeline;
|
|
@@ -1187,6 +1262,10 @@ declare class CharacterExpressions {
|
|
|
1187
1262
|
setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
|
|
1188
1263
|
/** Clear the active emotion expression layer. */
|
|
1189
1264
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
1265
|
+
/** Set a temporary emotion overlay. Timeline emotions and visemes keep their state. */
|
|
1266
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
1267
|
+
/** Ease the temporary overlay back to the current timeline or idle face. */
|
|
1268
|
+
clearOverlay(): void;
|
|
1190
1269
|
/** Reset all expression layer targets to zero. */
|
|
1191
1270
|
reset(): void;
|
|
1192
1271
|
}
|
|
@@ -1217,6 +1296,8 @@ declare class CharacterRuntime {
|
|
|
1217
1296
|
protected readonly streamPlayback: StreamPlaybackController;
|
|
1218
1297
|
protected defaultParent: Object3D | null;
|
|
1219
1298
|
protected currentGazeTarget: Object3D | null;
|
|
1299
|
+
/** The current gaze target came from `lookAt()`. */
|
|
1300
|
+
protected lookAtActive: boolean;
|
|
1220
1301
|
protected thinkingState: boolean;
|
|
1221
1302
|
protected errorCallback: ((error: Error) => void) | null;
|
|
1222
1303
|
constructor(options: CharacterRuntimeInitOptions);
|
|
@@ -1288,6 +1369,12 @@ declare class CharacterRuntime {
|
|
|
1288
1369
|
/** Stop the currently playing skeletal animation. */
|
|
1289
1370
|
stopAnimation(fadeDuration?: number): void;
|
|
1290
1371
|
unlockAudio(): Promise<boolean>;
|
|
1372
|
+
/** Freeze the current NPCP timeline, audio, face, and animation. */
|
|
1373
|
+
pausePlayback(): void;
|
|
1374
|
+
/** Continue the same timeline. Call from a gesture if the browser requires it. */
|
|
1375
|
+
resumePlayback(): Promise<void>;
|
|
1376
|
+
/** Cancel current output and pending audio/animation loads. */
|
|
1377
|
+
interruptPlayback(): void;
|
|
1291
1378
|
getExpressions(): string[];
|
|
1292
1379
|
getMouthExpressionNames(): string[];
|
|
1293
1380
|
getExpressionNames(): string[];
|
|
@@ -1296,8 +1383,24 @@ declare class CharacterRuntime {
|
|
|
1296
1383
|
getExpressionWeight(name: FacialChannelName): number;
|
|
1297
1384
|
setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
|
|
1298
1385
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
1299
|
-
/**
|
|
1386
|
+
/**
|
|
1387
|
+
* The object the eyes of a template face follow (the camera); null looks
|
|
1388
|
+
* ahead. It replaces a `lookAt()` target. Other faces do not follow it.
|
|
1389
|
+
*/
|
|
1300
1390
|
setGazeTarget(target: Object3D | null): void;
|
|
1391
|
+
/**
|
|
1392
|
+
* Direct the character's gaze toward a Three.js `Object3D` or world
|
|
1393
|
+
* coordinates. A template face keeps its gaze cone, idle eye moves and the
|
|
1394
|
+
* look cues of staging; other faces turn their eyes with three-vrm's
|
|
1395
|
+
* `vrm.lookAt.target`. Coordinates must be finite numbers, else this throws
|
|
1396
|
+
* a `RangeError` and keeps the current target.
|
|
1397
|
+
*/
|
|
1398
|
+
lookAt(target: CharacterLookAtTarget): void;
|
|
1399
|
+
lookAt(x: number, y: number, z: number): void;
|
|
1400
|
+
/** Reset the character's gaze target so the eyes look straight ahead. */
|
|
1401
|
+
clearLookAt(): void;
|
|
1402
|
+
/** Briefly look at an object; release restores the prior target and gaze mode. */
|
|
1403
|
+
lookAtTemporarily(target: Object3D): () => void;
|
|
1301
1404
|
startPerformanceRecording(options?: PerformanceRecordingStartOptions): void;
|
|
1302
1405
|
stopPerformanceRecording(): PerformanceRecording | null;
|
|
1303
1406
|
getPerformanceRecording(): PerformanceRecording | null;
|
|
@@ -1314,12 +1417,110 @@ declare class CharacterRuntime {
|
|
|
1314
1417
|
update(delta: number): void;
|
|
1315
1418
|
protected readonly getAudioPlayer: () => AudioPlayer;
|
|
1316
1419
|
protected readonly getStagingPlayer: () => StagingPlayer;
|
|
1420
|
+
/**
|
|
1421
|
+
* A template face follows the gaze target with its IdleFace. Only a
|
|
1422
|
+
* `lookAt()` target moves the eyes of another face, through three-vrm's
|
|
1423
|
+
* `vrm.lookAt.target`; on a template face that target would stop the
|
|
1424
|
+
* IdleFace (its gaze cone, idle eye moves and look cues).
|
|
1425
|
+
*/
|
|
1426
|
+
protected applyGaze(): void;
|
|
1317
1427
|
protected resetRuntimeAssets(): void;
|
|
1318
1428
|
protected disposeRuntimeAssets(): void;
|
|
1319
1429
|
protected readonly handleCallbackError: (context: string, error: unknown) => void;
|
|
1320
1430
|
protected readonly reportError: (error: Error, context?: string) => void;
|
|
1321
1431
|
}
|
|
1322
1432
|
|
|
1433
|
+
declare class MicrophoneError extends Error {
|
|
1434
|
+
readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
|
|
1435
|
+
constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
|
|
1436
|
+
}
|
|
1437
|
+
/** One frame of 16 kHz mono PCM and its loudness (RMS, 0 to 1). */
|
|
1438
|
+
type CaptureFrame = {
|
|
1439
|
+
pcm: Int16Array;
|
|
1440
|
+
level: number;
|
|
1441
|
+
};
|
|
1442
|
+
/** A source of 16 kHz frames. The SDK uses the microphone; tests use a fake. */
|
|
1443
|
+
interface AudioCapture {
|
|
1444
|
+
start(onFrame: (frame: CaptureFrame) => void): Promise<void>;
|
|
1445
|
+
/** Ends the capture; the last partial frame arrives before this resolves. */
|
|
1446
|
+
stop(): Promise<void>;
|
|
1447
|
+
/** Ends the capture without its last frame. */
|
|
1448
|
+
cancel(): void;
|
|
1449
|
+
}
|
|
1450
|
+
|
|
1451
|
+
type ListeningState = 'idle' | 'starting' | 'listening' | 'recognizing';
|
|
1452
|
+
interface ListeningOptions {
|
|
1453
|
+
/** BCP 47 hint, for example `ja`, `zh` or `en`. Without it the language is detected. */
|
|
1454
|
+
language?: string | null;
|
|
1455
|
+
/** `false`: recognise only; send the text later with `result.send()`. Default `true`. */
|
|
1456
|
+
send?: boolean;
|
|
1457
|
+
/** The recording stops at this time. Default and maximum: 30 seconds. */
|
|
1458
|
+
maxDurationMs?: number;
|
|
1459
|
+
/** For listening without a release: stop after this much silence that follows speech. */
|
|
1460
|
+
endOnSilenceMs?: number | null;
|
|
1461
|
+
parentStagingId?: string | null;
|
|
1462
|
+
silent?: boolean;
|
|
1463
|
+
}
|
|
1464
|
+
interface SpeechInputResult {
|
|
1465
|
+
/** The recognised text; empty when no speech was recognised (then no turn starts). */
|
|
1466
|
+
text: string;
|
|
1467
|
+
/** The detected language, when the recognition vendor gives it. */
|
|
1468
|
+
language: string | null;
|
|
1469
|
+
durationMs: number;
|
|
1470
|
+
/** The server started the turn with this text. */
|
|
1471
|
+
sent: boolean;
|
|
1472
|
+
/** The final payload of the turn, when it was sent. */
|
|
1473
|
+
turn: Promise<StagingResultPayload> | null;
|
|
1474
|
+
/** Sends the text as the player's message, when it was not sent. */
|
|
1475
|
+
send(): Promise<StagingResultPayload>;
|
|
1476
|
+
}
|
|
1477
|
+
interface TranscriptEvent {
|
|
1478
|
+
text: string;
|
|
1479
|
+
/** False for the text so far while the player speaks. */
|
|
1480
|
+
final: boolean;
|
|
1481
|
+
}
|
|
1482
|
+
declare class SpeechInputError extends Error {
|
|
1483
|
+
readonly code: string;
|
|
1484
|
+
constructor(code: string, message?: string);
|
|
1485
|
+
}
|
|
1486
|
+
|
|
1487
|
+
/** What push-to-talk drives: a `Character` or `Npcify` instance. */
|
|
1488
|
+
interface PushToTalkTarget {
|
|
1489
|
+
readonly listeningState: ListeningState;
|
|
1490
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1491
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1492
|
+
cancelListening(): void;
|
|
1493
|
+
}
|
|
1494
|
+
interface PushToTalkOptions {
|
|
1495
|
+
/** A button or any element to press and hold. */
|
|
1496
|
+
element?: HTMLElement | null;
|
|
1497
|
+
/** A `KeyboardEvent.code` to hold. Default `'Space'`; `null` binds no key. */
|
|
1498
|
+
key?: string | null;
|
|
1499
|
+
/** A shorter press sends nothing. Default 300 ms. */
|
|
1500
|
+
minHoldMs?: number;
|
|
1501
|
+
/** The options of each listening, or a function that gives them at each press. */
|
|
1502
|
+
listening?: ListeningOptions | (() => ListeningOptions);
|
|
1503
|
+
/** False while the player cannot reply (for example while options are locked). */
|
|
1504
|
+
enabled?: () => boolean;
|
|
1505
|
+
onResult?: (result: SpeechInputResult) => void;
|
|
1506
|
+
/**
|
|
1507
|
+
* A press sent nothing: it was shorter than `minHoldMs`, or it ended before
|
|
1508
|
+
* the microphone gave any audio (a permission prompt, a slow device). A UI
|
|
1509
|
+
* can say "hold to talk".
|
|
1510
|
+
*/
|
|
1511
|
+
onShortPress?: () => void;
|
|
1512
|
+
onError?: (error: unknown) => void;
|
|
1513
|
+
}
|
|
1514
|
+
/**
|
|
1515
|
+
* Press and hold to talk (docs/voice-input.md): an element, a key, or both.
|
|
1516
|
+
* The key does not start while the focus is in a field or on another
|
|
1517
|
+
* control, key repeat is ignored, a short press sends nothing, and window
|
|
1518
|
+
* blur, a hidden tab and Escape cancel. A long press on the element does not
|
|
1519
|
+
* open a menu, select text or scroll. A release anywhere ends the press.
|
|
1520
|
+
* Returns a function that unbinds.
|
|
1521
|
+
*/
|
|
1522
|
+
declare const bindPushToTalk: (target: PushToTalkTarget, options?: PushToTalkOptions) => () => void;
|
|
1523
|
+
|
|
1323
1524
|
declare class Character extends CharacterRuntime {
|
|
1324
1525
|
private readonly conversationSocket;
|
|
1325
1526
|
private renderSessionId;
|
|
@@ -1330,8 +1531,15 @@ declare class Character extends CharacterRuntime {
|
|
|
1330
1531
|
private readonly liveConversation;
|
|
1331
1532
|
private baked;
|
|
1332
1533
|
private readonly authoring;
|
|
1534
|
+
protected readonly normalizeIdleOrientation: boolean;
|
|
1333
1535
|
private readonly bakedAsset?;
|
|
1334
1536
|
private animationRefreshId;
|
|
1537
|
+
private readonly voice;
|
|
1538
|
+
private readonly configuredInputModes;
|
|
1539
|
+
private resolvedInputModes;
|
|
1540
|
+
private latestOptions;
|
|
1541
|
+
private listeningStateCallback;
|
|
1542
|
+
private transcriptCallback;
|
|
1335
1543
|
conversationId: string | null;
|
|
1336
1544
|
constructor(options: NpcifyOptions);
|
|
1337
1545
|
load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
|
|
@@ -1353,18 +1561,57 @@ declare class Character extends CharacterRuntime {
|
|
|
1353
1561
|
replayStaging(stagingId: string): Promise<StagingResultPayload>;
|
|
1354
1562
|
replayStaging(conversationId: string, stagingId: string, options?: PlayStagingOptions): Promise<StagingResultPayload>;
|
|
1355
1563
|
get isBakedConversation(): boolean;
|
|
1564
|
+
/**
|
|
1565
|
+
* The input modes in use: the configured modes, or every mode that the
|
|
1566
|
+
* character permits. Before `loadAssets`, the configuration or all modes.
|
|
1567
|
+
*/
|
|
1568
|
+
get inputModes(): readonly InputMode[];
|
|
1569
|
+
get listeningState(): ListeningState;
|
|
1570
|
+
/**
|
|
1571
|
+
* Opens the microphone and streams the speech to the server. Resolves with
|
|
1572
|
+
* the result when listening ends (`stopListening`, `endOnSilenceMs` or
|
|
1573
|
+
* `maxDurationMs`), or with null when it is cancelled. The first call asks
|
|
1574
|
+
* for the microphone permission.
|
|
1575
|
+
*/
|
|
1576
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1577
|
+
/**
|
|
1578
|
+
* Ends the recording; resolves as soon as the server recognised the text.
|
|
1579
|
+
* Before the microphone gave any audio, it cancels and resolves null.
|
|
1580
|
+
*/
|
|
1581
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1582
|
+
/** Discards the recording. Nothing is recognised or charged. */
|
|
1583
|
+
cancelListening(): void;
|
|
1584
|
+
/** Press and hold an element or a key (default Space) to talk. Returns an unbind function. */
|
|
1585
|
+
bindPushToTalk(options?: PushToTalkOptions): () => void;
|
|
1586
|
+
onListeningState(callback: (state: ListeningState) => void): () => void;
|
|
1587
|
+
/** Partial text while the player speaks (`final: false`), then the final text. */
|
|
1588
|
+
onTranscript(callback: (event: TranscriptEvent) => void): () => void;
|
|
1356
1589
|
getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
|
|
1357
1590
|
acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
|
|
1358
1591
|
playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
|
|
1592
|
+
/**
|
|
1593
|
+
* Plays one live staging event from another transport, such as an actor
|
|
1594
|
+
* stage. A timeline event returns its playback result.
|
|
1595
|
+
*/
|
|
1596
|
+
playStreamEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1359
1597
|
setRenderSessionId(renderSessionId: string | null): void;
|
|
1360
1598
|
onStagingResult(callback: (payload: StagingResultPayload) => Promise<void> | void): () => void;
|
|
1361
1599
|
onThinking(callback: (payload: LlmThinkingPayload) => Promise<void> | void): () => void;
|
|
1362
1600
|
private resetAssets;
|
|
1363
1601
|
private readonly getAssetInfo;
|
|
1364
1602
|
private handleStagingResult;
|
|
1603
|
+
/** Throws when the input modes do not include `mode`. */
|
|
1604
|
+
private assertInputMode;
|
|
1605
|
+
/**
|
|
1606
|
+
* A typed reply needs the `text` mode; one of the current reply options
|
|
1607
|
+
* needs `options` or `text`. Without a configuration the server's rules
|
|
1608
|
+
* apply alone, as before.
|
|
1609
|
+
*/
|
|
1610
|
+
private assertReplyMode;
|
|
1365
1611
|
private invokeThinkingCallback;
|
|
1366
1612
|
}
|
|
1367
1613
|
declare class Npcify extends Character {
|
|
1614
|
+
constructor(options: NpcifyOptions);
|
|
1368
1615
|
}
|
|
1369
1616
|
|
|
1370
1617
|
interface LoadCharacterOptions extends CharacterOptions {
|
|
@@ -1455,6 +1702,9 @@ interface MountNpcifyOptions {
|
|
|
1455
1702
|
apiToken?: string;
|
|
1456
1703
|
serverRendering?: boolean;
|
|
1457
1704
|
assetProfile?: 'original' | 'web-optimized-v1';
|
|
1705
|
+
normalizeIdleOrientation?: boolean;
|
|
1706
|
+
/** How the player can reply: `'options'`, `'text'`, `'voice'` (docs/voice-input.md). */
|
|
1707
|
+
inputModes?: InputMode[];
|
|
1458
1708
|
onError?: (error: Error) => void;
|
|
1459
1709
|
}
|
|
1460
1710
|
interface NpcifyView {
|
|
@@ -1469,4 +1719,4 @@ interface NpcifyView {
|
|
|
1469
1719
|
/** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
|
|
1470
1720
|
declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
|
|
1471
1721
|
|
|
1472
|
-
export { AnimationController, type AnimationControllerLike, type AssetLoadProgress, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, type LlmThinkingPayload, type LoadCharacterOptions, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
|
1722
|
+
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|