@npcify/web 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +69 -3
- package/dist/index.d.ts +216 -4
- package/dist/index.js +1999 -106
- package/examples/index.html +1 -0
- package/examples/main.js +43 -12
- package/examples/three-game.js +13 -3
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -125,6 +125,9 @@ animate()
|
|
|
125
125
|
- **Dialogue & Speech State**:
|
|
126
126
|
- `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
|
|
127
127
|
- `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
|
|
128
|
+
- **Voice Input** ([details](#voice-input-and-input-modes)):
|
|
129
|
+
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
130
|
+
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
128
131
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
129
132
|
|
|
130
133
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
@@ -188,13 +191,76 @@ with any ESM bundler after replacing their configuration placeholders. Keys in b
|
|
|
188
191
|
code are visible to visitors. Use narrow scopes; keep keys in your backend when they
|
|
189
192
|
must remain private. Calls made with a key are billed to its owner.
|
|
190
193
|
|
|
194
|
+
## Voice input and input modes
|
|
195
|
+
|
|
196
|
+
The player can speak a reply when the character owner turned voice input on
|
|
197
|
+
(the STT tab of the character in the dashboard, where the owner also chooses
|
|
198
|
+
the speech recognition provider). The SDK records the microphone and streams
|
|
199
|
+
the audio to the Npcify server. At the release, the server recognises the
|
|
200
|
+
speech and starts the turn. The recognised text is the player's message, as a
|
|
201
|
+
typed reply.
|
|
202
|
+
|
|
203
|
+
```js
|
|
204
|
+
// Hold the button, or Space, to talk. The release sends the speech.
|
|
205
|
+
const talk = document.querySelector('#talk')
|
|
206
|
+
talk.hidden = !character.inputModes.includes('voice')
|
|
207
|
+
const unbindTalk = character.bindPushToTalk({
|
|
208
|
+
element: talk,
|
|
209
|
+
onResult: ({ text }) => { if (!text) status.textContent = 'No speech was heard.' },
|
|
210
|
+
onShortPress: () => { status.textContent = 'Hold to talk.' },
|
|
211
|
+
onError: (error) => { status.textContent = error.message },
|
|
212
|
+
})
|
|
213
|
+
character.onListeningState((state) => {
|
|
214
|
+
talk.textContent =
|
|
215
|
+
{ listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
|
|
216
|
+
})
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
- `bindPushToTalk({ element, key, minHoldMs, listening, enabled, onResult, onShortPress, onError })`:
|
|
220
|
+
press and hold an element, a key (`KeyboardEvent.code`, default `'Space'`;
|
|
221
|
+
`null` binds no key), or both. The key does not start while the focus is in
|
|
222
|
+
a field or on another control, and key repeat is ignored. A press shorter
|
|
223
|
+
than `minHoldMs` (300 ms), or a release before the microphone gave any
|
|
224
|
+
audio, sends nothing and calls `onShortPress`. Escape, window blur and a
|
|
225
|
+
hidden tab cancel. A long press on the element does not open a menu, select
|
|
226
|
+
text or scroll, and a release outside the element ends the press. Returns
|
|
227
|
+
an unbind function.
|
|
228
|
+
- `startListening({ language, send, maxDurationMs, endOnSilenceMs })` starts
|
|
229
|
+
listening from code, for example when the character stops speaking.
|
|
230
|
+
`endOnSilenceMs` stops after that much silence that follows speech, and
|
|
231
|
+
`maxDurationMs` (30 s at most) always stops. `stopListening()` resolves with
|
|
232
|
+
`{ text, language, durationMs, sent, turn, send() }`, or with null when the
|
|
233
|
+
microphone gave no audio yet. `cancelListening()` discards the recording;
|
|
234
|
+
the owner pays only the audio that already went to a streaming provider.
|
|
235
|
+
- `onTranscript(({ text, final }) => …)` gives the final text, and partial
|
|
236
|
+
text when the server's speech model gives it.
|
|
237
|
+
- The first press or `startListening()` asks for the microphone permission.
|
|
238
|
+
An empty result sends nothing.
|
|
239
|
+
- `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
|
|
240
|
+
the server detects the language.
|
|
241
|
+
- `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
|
|
242
|
+
an array of `'options'`, `'text'` and `'voice'`; one mode forces it. Without
|
|
243
|
+
it, every mode that the character permits is available. A baked
|
|
244
|
+
conversation permits only `['options']`, a character without custom
|
|
245
|
+
replies permits only `'options'`, and `'voice'` needs the character's voice
|
|
246
|
+
input (`speechInputEnabled` in the character info). Another configuration
|
|
247
|
+
throws an `InputModeError` when the character loads, and listening on a
|
|
248
|
+
character with voice input off throws one too. With `inputModes` set,
|
|
249
|
+
`sendMessage` with typed text needs `'text'`, a reply option needs
|
|
250
|
+
`'options'` or `'text'`, and listening needs `'voice'`.
|
|
251
|
+
`character.inputModes` gives the modes in use.
|
|
252
|
+
- The character owner pays the recognition at the price of the provider that
|
|
253
|
+
the owner chose, as for the turn's language model.
|
|
254
|
+
|
|
191
255
|
## Runtime and lifecycle
|
|
192
256
|
|
|
193
257
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|
|
194
|
-
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading
|
|
195
|
-
|
|
258
|
+
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading,
|
|
259
|
+
and normalizes the idle animation orientation to face `+Z` by default
|
|
260
|
+
(`normalizeIdleOrientation: true`). Give the canvas a nonzero CSS size.
|
|
196
261
|
- `loadCharacter({ baseUrl, characterId, apiKey, parent, gazeTarget, shadows })`
|
|
197
|
-
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
262
|
+
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
263
|
+
(`normalizeIdleOrientation` defaults to `false`).
|
|
198
264
|
- Call `unlockAudio()` (or `character.audio.unlock()`) during a user gesture before
|
|
199
265
|
starting playback.
|
|
200
266
|
- `onOptionsReady` delivers choices after playback finishes. `sendMessage(label)`
|
package/dist/index.d.ts
CHANGED
|
@@ -39,9 +39,15 @@ interface AvatarPublicInfo {
|
|
|
39
39
|
freeFirstNMessages?: number | null;
|
|
40
40
|
hostedCharacterEnabled?: boolean;
|
|
41
41
|
listedPublicly?: boolean;
|
|
42
|
+
r18?: boolean;
|
|
42
43
|
bakedConversationEnabled?: boolean;
|
|
43
44
|
bakedConversationRevision?: number;
|
|
44
45
|
allowCustomReply?: boolean;
|
|
46
|
+
/**
|
|
47
|
+
* The owner turned voice input on and chose a provider. Voice also needs
|
|
48
|
+
* `allowCustomReply`. Missing on older servers: off.
|
|
49
|
+
*/
|
|
50
|
+
speechInputEnabled?: boolean;
|
|
45
51
|
autoPlay: boolean;
|
|
46
52
|
serverRenderingEnabled: boolean;
|
|
47
53
|
cameraSqt?: Sqt | null;
|
|
@@ -280,6 +286,18 @@ interface StagingStreamBillingEvent {
|
|
|
280
286
|
}
|
|
281
287
|
type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent;
|
|
282
288
|
|
|
289
|
+
/**
|
|
290
|
+
* Voice input of conversations (docs/voice-input.md): the input modes of the
|
|
291
|
+
* SDKs, and the messages of a speech session between an SDK and the server.
|
|
292
|
+
*/
|
|
293
|
+
/** How a player can reply: a reply option, typed text, or speech. */
|
|
294
|
+
declare const INPUT_MODES: readonly ["options", "text", "voice"];
|
|
295
|
+
type InputMode = (typeof INPUT_MODES)[number];
|
|
296
|
+
/** A configuration of input modes that is invalid or that the character does not permit. */
|
|
297
|
+
declare class InputModeError extends Error {
|
|
298
|
+
constructor(message: string);
|
|
299
|
+
}
|
|
300
|
+
|
|
283
301
|
declare const BAKED_CONVERSATION_VERSION: 1;
|
|
284
302
|
/** Portable, language-neutral asset. Audio is embedded binary, never a signed URL. */
|
|
285
303
|
interface BakedConversationAsset {
|
|
@@ -319,9 +337,11 @@ declare const decodePerformance: <T = unknown>(input: ArrayBuffer | Uint8Array,
|
|
|
319
337
|
interface ConversationTurnOptions {
|
|
320
338
|
silent?: boolean;
|
|
321
339
|
parentStagingId?: string | null;
|
|
340
|
+
/** The turn's `Idempotency-Key`; a new UUID when absent. */
|
|
341
|
+
requestId?: string;
|
|
322
342
|
}
|
|
323
343
|
|
|
324
|
-
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'
|
|
344
|
+
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled'>>;
|
|
325
345
|
|
|
326
346
|
interface LlmThinkingPayload {
|
|
327
347
|
taskId?: string | null;
|
|
@@ -586,6 +606,13 @@ interface AnimationConfig {
|
|
|
586
606
|
loop?: boolean;
|
|
587
607
|
weight?: number;
|
|
588
608
|
}
|
|
609
|
+
interface AnimationControllerOptions {
|
|
610
|
+
/**
|
|
611
|
+
* Normalize the configured idle animation clip so its average torso heading
|
|
612
|
+
* and neck/head orientation face canonical `+Z`.
|
|
613
|
+
*/
|
|
614
|
+
normalizeIdleOrientation?: boolean;
|
|
615
|
+
}
|
|
589
616
|
interface CharacterAnimationPlayOptions {
|
|
590
617
|
fadeDuration?: number;
|
|
591
618
|
loop?: boolean;
|
|
@@ -618,14 +645,17 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
618
645
|
prepareAnimations?: (ids: string[]) => Promise<void>;
|
|
619
646
|
private readonly mixer;
|
|
620
647
|
private readonly clips;
|
|
648
|
+
private readonly rawClips;
|
|
649
|
+
private readonly normalizedIdleClips;
|
|
621
650
|
private readonly configs;
|
|
651
|
+
private readonly normalizeIdleOrientation;
|
|
622
652
|
private previewMixer;
|
|
623
653
|
private restorePreview;
|
|
624
654
|
private activeAction;
|
|
625
655
|
private activeAnimationId;
|
|
626
656
|
private idleAnimationId;
|
|
627
657
|
private pendingReturnToIdleAction;
|
|
628
|
-
constructor(vrm: VRM, idleAnimationId?: string | null);
|
|
658
|
+
constructor(vrm: VRM, idleAnimationId?: string | null, options?: AnimationControllerOptions);
|
|
629
659
|
getAnimationMixer(): AnimationMixer;
|
|
630
660
|
getActiveAction(): AnimationAction | null;
|
|
631
661
|
getActiveAnimationId(): string | null;
|
|
@@ -636,6 +666,7 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
636
666
|
previewAnimation(animation: VRMAnimation): () => void;
|
|
637
667
|
setIdleAnimation(animationId: string | null): void;
|
|
638
668
|
clearAnimations(): void;
|
|
669
|
+
private resolveClipForId;
|
|
639
670
|
playAnimation(animationId: string, fadeDuration?: number, loopOverride?: boolean): void;
|
|
640
671
|
playIdle(fadeDuration?: number): void;
|
|
641
672
|
stopAnimation(fadeDuration?: number): void;
|
|
@@ -744,6 +775,7 @@ declare const FacialPerformanceLayer: {
|
|
|
744
775
|
readonly Viseme: "viseme";
|
|
745
776
|
readonly Blink: "blink";
|
|
746
777
|
readonly Gaze: "gaze";
|
|
778
|
+
readonly Overlay: "overlay";
|
|
747
779
|
};
|
|
748
780
|
type FacialPerformanceLayerId = (typeof FacialPerformanceLayer)[keyof typeof FacialPerformanceLayer];
|
|
749
781
|
type FacialMouthMode = 'expression' | 'viseme';
|
|
@@ -799,6 +831,9 @@ declare class FacialPerformanceMixer {
|
|
|
799
831
|
playCue(channel: FacialChannelName, options?: FacialCueOptions): void;
|
|
800
832
|
setEmotion(channel: FacialChannelName, weight: number, mouthMode: FacialMouthMode): void;
|
|
801
833
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
834
|
+
/** Blend temporary emotions without changing timeline targets or mouth ownership. */
|
|
835
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
836
|
+
clearOverlay(): void;
|
|
802
837
|
releaseVisemeMouthMode(): void;
|
|
803
838
|
/**
|
|
804
839
|
* Ends the speech source without disturbing any other performance layer.
|
|
@@ -948,6 +983,8 @@ declare class StagingPlayer {
|
|
|
948
983
|
private readonly audioPlayer;
|
|
949
984
|
private readonly facialMixer;
|
|
950
985
|
private readonly animationController;
|
|
986
|
+
private readonly clock;
|
|
987
|
+
private pauseRevision;
|
|
951
988
|
private playbackController;
|
|
952
989
|
private generation;
|
|
953
990
|
private incremental;
|
|
@@ -958,6 +995,10 @@ declare class StagingPlayer {
|
|
|
958
995
|
private activeExpressionMouthMode;
|
|
959
996
|
private activeAnimation;
|
|
960
997
|
constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
|
|
998
|
+
get paused(): boolean;
|
|
999
|
+
pause(): void;
|
|
1000
|
+
resume(): Promise<void>;
|
|
1001
|
+
private waitUntilResumed;
|
|
961
1002
|
play(timeline: StagingTimeline, options?: StagingPlaybackOptions): Promise<void>;
|
|
962
1003
|
updateTimeline(timeline: StagingTimeline, complete?: boolean): void;
|
|
963
1004
|
private startCursor;
|
|
@@ -1103,6 +1144,22 @@ interface NpcifyOptions {
|
|
|
1103
1144
|
volume?: number;
|
|
1104
1145
|
/** Initial speech audio muted flag. */
|
|
1105
1146
|
muted?: boolean;
|
|
1147
|
+
/**
|
|
1148
|
+
* Normalize the character's idle animation orientation so the torso and head
|
|
1149
|
+
* face canonical `+Z` on average while preserving idle motion.
|
|
1150
|
+
* Defaults to `true` for `Npcify` (Web Chats and Canvas API) and `false` for
|
|
1151
|
+
* `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
|
|
1152
|
+
*/
|
|
1153
|
+
normalizeIdleOrientation?: boolean;
|
|
1154
|
+
/**
|
|
1155
|
+
* How the player can reply: `'options'`, `'text'` and `'voice'`, in any
|
|
1156
|
+
* combination; one mode forces it. Without it, every mode that the
|
|
1157
|
+
* character permits. A baked conversation permits only `['options']`; a
|
|
1158
|
+
* character without custom replies permits only `options`; `voice` needs
|
|
1159
|
+
* the character's voice input (`speechInputEnabled`, the STT tab). A mode
|
|
1160
|
+
* that the character does not permit throws an `InputModeError` on load.
|
|
1161
|
+
*/
|
|
1162
|
+
inputModes?: InputMode[];
|
|
1106
1163
|
/** Optional error callback. */
|
|
1107
1164
|
onError?: (error: Error) => void;
|
|
1108
1165
|
}
|
|
@@ -1169,7 +1226,8 @@ declare class StreamPlaybackController {
|
|
|
1169
1226
|
force?: boolean;
|
|
1170
1227
|
timeline?: StagingTimeline;
|
|
1171
1228
|
}): Promise<boolean>;
|
|
1172
|
-
|
|
1229
|
+
/** Returns the playback result for a timeline event. */
|
|
1230
|
+
handleEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1173
1231
|
cleanup(): void;
|
|
1174
1232
|
private resetRealtimeSpeech;
|
|
1175
1233
|
private playTimeline;
|
|
@@ -1210,6 +1268,10 @@ declare class CharacterExpressions {
|
|
|
1210
1268
|
setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
|
|
1211
1269
|
/** Clear the active emotion expression layer. */
|
|
1212
1270
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
1271
|
+
/** Set a temporary emotion overlay. Timeline emotions and visemes keep their state. */
|
|
1272
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
1273
|
+
/** Ease the temporary overlay back to the current timeline or idle face. */
|
|
1274
|
+
clearOverlay(): void;
|
|
1213
1275
|
/** Reset all expression layer targets to zero. */
|
|
1214
1276
|
reset(): void;
|
|
1215
1277
|
}
|
|
@@ -1313,6 +1375,12 @@ declare class CharacterRuntime {
|
|
|
1313
1375
|
/** Stop the currently playing skeletal animation. */
|
|
1314
1376
|
stopAnimation(fadeDuration?: number): void;
|
|
1315
1377
|
unlockAudio(): Promise<boolean>;
|
|
1378
|
+
/** Freeze the current NPCP timeline, audio, face, and animation. */
|
|
1379
|
+
pausePlayback(): void;
|
|
1380
|
+
/** Continue the same timeline. Call from a gesture if the browser requires it. */
|
|
1381
|
+
resumePlayback(): Promise<void>;
|
|
1382
|
+
/** Cancel current output and pending audio/animation loads. */
|
|
1383
|
+
interruptPlayback(): void;
|
|
1316
1384
|
getExpressions(): string[];
|
|
1317
1385
|
getMouthExpressionNames(): string[];
|
|
1318
1386
|
getExpressionNames(): string[];
|
|
@@ -1337,6 +1405,8 @@ declare class CharacterRuntime {
|
|
|
1337
1405
|
lookAt(x: number, y: number, z: number): void;
|
|
1338
1406
|
/** Reset the character's gaze target so the eyes look straight ahead. */
|
|
1339
1407
|
clearLookAt(): void;
|
|
1408
|
+
/** Briefly look at an object; release restores the prior target and gaze mode. */
|
|
1409
|
+
lookAtTemporarily(target: Object3D): () => void;
|
|
1340
1410
|
startPerformanceRecording(options?: PerformanceRecordingStartOptions): void;
|
|
1341
1411
|
stopPerformanceRecording(): PerformanceRecording | null;
|
|
1342
1412
|
getPerformanceRecording(): PerformanceRecording | null;
|
|
@@ -1366,6 +1436,97 @@ declare class CharacterRuntime {
|
|
|
1366
1436
|
protected readonly reportError: (error: Error, context?: string) => void;
|
|
1367
1437
|
}
|
|
1368
1438
|
|
|
1439
|
+
declare class MicrophoneError extends Error {
|
|
1440
|
+
readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
|
|
1441
|
+
constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
|
|
1442
|
+
}
|
|
1443
|
+
/** One frame of 16 kHz mono PCM and its loudness (RMS, 0 to 1). */
|
|
1444
|
+
type CaptureFrame = {
|
|
1445
|
+
pcm: Int16Array;
|
|
1446
|
+
level: number;
|
|
1447
|
+
};
|
|
1448
|
+
/** A source of 16 kHz frames. The SDK uses the microphone; tests use a fake. */
|
|
1449
|
+
interface AudioCapture {
|
|
1450
|
+
start(onFrame: (frame: CaptureFrame) => void): Promise<void>;
|
|
1451
|
+
/** Ends the capture; the last partial frame arrives before this resolves. */
|
|
1452
|
+
stop(): Promise<void>;
|
|
1453
|
+
/** Ends the capture without its last frame. */
|
|
1454
|
+
cancel(): void;
|
|
1455
|
+
}
|
|
1456
|
+
|
|
1457
|
+
type ListeningState = 'idle' | 'starting' | 'listening' | 'recognizing';
|
|
1458
|
+
interface ListeningOptions {
|
|
1459
|
+
/** BCP 47 hint, for example `ja`, `zh` or `en`. Without it the language is detected. */
|
|
1460
|
+
language?: string | null;
|
|
1461
|
+
/** `false`: recognise only; send the text later with `result.send()`. Default `true`. */
|
|
1462
|
+
send?: boolean;
|
|
1463
|
+
/** The recording stops at this time. Default and maximum: 30 seconds. */
|
|
1464
|
+
maxDurationMs?: number;
|
|
1465
|
+
/** For listening without a release: stop after this much silence that follows speech. */
|
|
1466
|
+
endOnSilenceMs?: number | null;
|
|
1467
|
+
parentStagingId?: string | null;
|
|
1468
|
+
silent?: boolean;
|
|
1469
|
+
}
|
|
1470
|
+
interface SpeechInputResult {
|
|
1471
|
+
/** The recognised text; empty when no speech was recognised (then no turn starts). */
|
|
1472
|
+
text: string;
|
|
1473
|
+
/** The detected language, when the recognition vendor gives it. */
|
|
1474
|
+
language: string | null;
|
|
1475
|
+
durationMs: number;
|
|
1476
|
+
/** The server started the turn with this text. */
|
|
1477
|
+
sent: boolean;
|
|
1478
|
+
/** The final payload of the turn, when it was sent. */
|
|
1479
|
+
turn: Promise<StagingResultPayload> | null;
|
|
1480
|
+
/** Sends the text as the player's message, when it was not sent. */
|
|
1481
|
+
send(): Promise<StagingResultPayload>;
|
|
1482
|
+
}
|
|
1483
|
+
interface TranscriptEvent {
|
|
1484
|
+
text: string;
|
|
1485
|
+
/** False for the text so far while the player speaks. */
|
|
1486
|
+
final: boolean;
|
|
1487
|
+
}
|
|
1488
|
+
declare class SpeechInputError extends Error {
|
|
1489
|
+
readonly code: string;
|
|
1490
|
+
constructor(code: string, message?: string);
|
|
1491
|
+
}
|
|
1492
|
+
|
|
1493
|
+
/** What push-to-talk drives: a `Character` or `Npcify` instance. */
|
|
1494
|
+
interface PushToTalkTarget {
|
|
1495
|
+
readonly listeningState: ListeningState;
|
|
1496
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1497
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1498
|
+
cancelListening(): void;
|
|
1499
|
+
}
|
|
1500
|
+
interface PushToTalkOptions {
|
|
1501
|
+
/** A button or any element to press and hold. */
|
|
1502
|
+
element?: HTMLElement | null;
|
|
1503
|
+
/** A `KeyboardEvent.code` to hold. Default `'Space'`; `null` binds no key. */
|
|
1504
|
+
key?: string | null;
|
|
1505
|
+
/** A shorter press sends nothing. Default 300 ms. */
|
|
1506
|
+
minHoldMs?: number;
|
|
1507
|
+
/** The options of each listening, or a function that gives them at each press. */
|
|
1508
|
+
listening?: ListeningOptions | (() => ListeningOptions);
|
|
1509
|
+
/** False while the player cannot reply (for example while options are locked). */
|
|
1510
|
+
enabled?: () => boolean;
|
|
1511
|
+
onResult?: (result: SpeechInputResult) => void;
|
|
1512
|
+
/**
|
|
1513
|
+
* A press sent nothing: it was shorter than `minHoldMs`, or it ended before
|
|
1514
|
+
* the microphone gave any audio (a permission prompt, a slow device). A UI
|
|
1515
|
+
* can say "hold to talk".
|
|
1516
|
+
*/
|
|
1517
|
+
onShortPress?: () => void;
|
|
1518
|
+
onError?: (error: unknown) => void;
|
|
1519
|
+
}
|
|
1520
|
+
/**
|
|
1521
|
+
* Press and hold to talk (docs/voice-input.md): an element, a key, or both.
|
|
1522
|
+
* The key does not start while the focus is in a field or on another
|
|
1523
|
+
* control, key repeat is ignored, a short press sends nothing, and window
|
|
1524
|
+
* blur, a hidden tab and Escape cancel. A long press on the element does not
|
|
1525
|
+
* open a menu, select text or scroll. A release anywhere ends the press.
|
|
1526
|
+
* Returns a function that unbinds.
|
|
1527
|
+
*/
|
|
1528
|
+
declare const bindPushToTalk: (target: PushToTalkTarget, options?: PushToTalkOptions) => () => void;
|
|
1529
|
+
|
|
1369
1530
|
declare class Character extends CharacterRuntime {
|
|
1370
1531
|
private readonly conversationSocket;
|
|
1371
1532
|
private renderSessionId;
|
|
@@ -1376,8 +1537,17 @@ declare class Character extends CharacterRuntime {
|
|
|
1376
1537
|
private readonly liveConversation;
|
|
1377
1538
|
private baked;
|
|
1378
1539
|
private readonly authoring;
|
|
1540
|
+
protected readonly normalizeIdleOrientation: boolean;
|
|
1379
1541
|
private readonly bakedAsset?;
|
|
1380
1542
|
private animationRefreshId;
|
|
1543
|
+
private readonly voice;
|
|
1544
|
+
private readonly configuredInputModes;
|
|
1545
|
+
private resolvedInputModes;
|
|
1546
|
+
/** The character's voice input switch, after the character loads. */
|
|
1547
|
+
private speechInput;
|
|
1548
|
+
private latestOptions;
|
|
1549
|
+
private listeningStateCallback;
|
|
1550
|
+
private transcriptCallback;
|
|
1381
1551
|
conversationId: string | null;
|
|
1382
1552
|
constructor(options: NpcifyOptions);
|
|
1383
1553
|
load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
|
|
@@ -1399,18 +1569,57 @@ declare class Character extends CharacterRuntime {
|
|
|
1399
1569
|
replayStaging(stagingId: string): Promise<StagingResultPayload>;
|
|
1400
1570
|
replayStaging(conversationId: string, stagingId: string, options?: PlayStagingOptions): Promise<StagingResultPayload>;
|
|
1401
1571
|
get isBakedConversation(): boolean;
|
|
1572
|
+
/**
|
|
1573
|
+
* The input modes in use: the configured modes, or every mode that the
|
|
1574
|
+
* character permits. Before `loadAssets`, the configuration or all modes.
|
|
1575
|
+
*/
|
|
1576
|
+
get inputModes(): readonly InputMode[];
|
|
1577
|
+
get listeningState(): ListeningState;
|
|
1578
|
+
/**
|
|
1579
|
+
* Opens the microphone and streams the speech to the server. Resolves with
|
|
1580
|
+
* the result when listening ends (`stopListening`, `endOnSilenceMs` or
|
|
1581
|
+
* `maxDurationMs`), or with null when it is cancelled. The first call asks
|
|
1582
|
+
* for the microphone permission.
|
|
1583
|
+
*/
|
|
1584
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1585
|
+
/**
|
|
1586
|
+
* Ends the recording; resolves as soon as the server recognised the text.
|
|
1587
|
+
* Before the microphone gave any audio, it cancels and resolves null.
|
|
1588
|
+
*/
|
|
1589
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1590
|
+
/** Discards the recording. Nothing is recognised or charged. */
|
|
1591
|
+
cancelListening(): void;
|
|
1592
|
+
/** Press and hold an element or a key (default Space) to talk. Returns an unbind function. */
|
|
1593
|
+
bindPushToTalk(options?: PushToTalkOptions): () => void;
|
|
1594
|
+
onListeningState(callback: (state: ListeningState) => void): () => void;
|
|
1595
|
+
/** Partial text while the player speaks (`final: false`), then the final text. */
|
|
1596
|
+
onTranscript(callback: (event: TranscriptEvent) => void): () => void;
|
|
1402
1597
|
getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
|
|
1403
1598
|
acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
|
|
1404
1599
|
playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
|
|
1600
|
+
/**
|
|
1601
|
+
* Plays one live staging event from another transport, such as an actor
|
|
1602
|
+
* stage. A timeline event returns its playback result.
|
|
1603
|
+
*/
|
|
1604
|
+
playStreamEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1405
1605
|
setRenderSessionId(renderSessionId: string | null): void;
|
|
1406
1606
|
onStagingResult(callback: (payload: StagingResultPayload) => Promise<void> | void): () => void;
|
|
1407
1607
|
onThinking(callback: (payload: LlmThinkingPayload) => Promise<void> | void): () => void;
|
|
1408
1608
|
private resetAssets;
|
|
1409
1609
|
private readonly getAssetInfo;
|
|
1410
1610
|
private handleStagingResult;
|
|
1611
|
+
/** Throws when the input modes do not include `mode`. */
|
|
1612
|
+
private assertInputMode;
|
|
1613
|
+
/**
|
|
1614
|
+
* A typed reply needs the `text` mode; one of the current reply options
|
|
1615
|
+
* needs `options` or `text`. Without a configuration the server's rules
|
|
1616
|
+
* apply alone, as before.
|
|
1617
|
+
*/
|
|
1618
|
+
private assertReplyMode;
|
|
1411
1619
|
private invokeThinkingCallback;
|
|
1412
1620
|
}
|
|
1413
1621
|
declare class Npcify extends Character {
|
|
1622
|
+
constructor(options: NpcifyOptions);
|
|
1414
1623
|
}
|
|
1415
1624
|
|
|
1416
1625
|
interface LoadCharacterOptions extends CharacterOptions {
|
|
@@ -1501,6 +1710,9 @@ interface MountNpcifyOptions {
|
|
|
1501
1710
|
apiToken?: string;
|
|
1502
1711
|
serverRendering?: boolean;
|
|
1503
1712
|
assetProfile?: 'original' | 'web-optimized-v1';
|
|
1713
|
+
normalizeIdleOrientation?: boolean;
|
|
1714
|
+
/** How the player can reply: `'options'`, `'text'`, `'voice'` (docs/voice-input.md). */
|
|
1715
|
+
inputModes?: InputMode[];
|
|
1504
1716
|
onError?: (error: Error) => void;
|
|
1505
1717
|
}
|
|
1506
1718
|
interface NpcifyView {
|
|
@@ -1515,4 +1727,4 @@ interface NpcifyView {
|
|
|
1515
1727
|
/** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
|
|
1516
1728
|
declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
|
|
1517
1729
|
|
|
1518
|
-
export { AnimationController, type AnimationControllerLike, type AssetLoadProgress, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, type LlmThinkingPayload, type LoadCharacterOptions, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
|
1730
|
+
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|