@npcify/web 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +65 -3
- package/dist/index.d.ts +208 -4
- package/dist/index.js +1960 -106
- package/examples/index.html +1 -0
- package/examples/main.js +40 -12
- package/examples/three-game.js +13 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -125,6 +125,9 @@ animate()
|
|
|
125
125
|
- **Dialogue & Speech State**:
|
|
126
126
|
- `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
|
|
127
127
|
- `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
|
|
128
|
+
- **Voice Input** ([details](#voice-input-and-input-modes)):
|
|
129
|
+
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
130
|
+
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
128
131
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
129
132
|
|
|
130
133
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
@@ -188,13 +191,72 @@ with any ESM bundler after replacing their configuration placeholders. Keys in b
|
|
|
188
191
|
code are visible to visitors. Use narrow scopes; keep keys in your backend when they
|
|
189
192
|
must remain private. Calls made with a key are billed to its owner.
|
|
190
193
|
|
|
194
|
+
## Voice input and input modes
|
|
195
|
+
|
|
196
|
+
The player can speak a reply. The SDK records the microphone and streams the
|
|
197
|
+
audio to the Npcify server. At the release, the server recognises the speech
|
|
198
|
+
and starts the turn. The recognised text is the player's message, as a typed
|
|
199
|
+
reply.
|
|
200
|
+
|
|
201
|
+
```js
|
|
202
|
+
// Hold the button, or Space, to talk. The release sends the speech.
|
|
203
|
+
const talk = document.querySelector('#talk')
|
|
204
|
+
talk.hidden = !character.inputModes.includes('voice')
|
|
205
|
+
const unbindTalk = character.bindPushToTalk({
|
|
206
|
+
element: talk,
|
|
207
|
+
onResult: ({ text }) => { if (!text) status.textContent = 'No speech was heard.' },
|
|
208
|
+
onShortPress: () => { status.textContent = 'Hold to talk.' },
|
|
209
|
+
onError: (error) => { status.textContent = error.message },
|
|
210
|
+
})
|
|
211
|
+
character.onListeningState((state) => {
|
|
212
|
+
talk.textContent =
|
|
213
|
+
{ listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
|
|
214
|
+
})
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
- `bindPushToTalk({ element, key, minHoldMs, listening, enabled, onResult, onShortPress, onError })`:
|
|
218
|
+
press and hold an element, a key (`KeyboardEvent.code`, default `'Space'`;
|
|
219
|
+
`null` binds no key), or both. The key does not start while the focus is in
|
|
220
|
+
a field or on another control, and key repeat is ignored. A press shorter
|
|
221
|
+
than `minHoldMs` (300 ms), or a release before the microphone gave any
|
|
222
|
+
audio, sends nothing and calls `onShortPress`. Escape, window blur and a
|
|
223
|
+
hidden tab cancel. A long press on the element does not open a menu, select
|
|
224
|
+
text or scroll, and a release outside the element ends the press. Returns
|
|
225
|
+
an unbind function.
|
|
226
|
+
- `startListening({ language, send, maxDurationMs, endOnSilenceMs })` starts
|
|
227
|
+
listening from code, for example when the character stops speaking.
|
|
228
|
+
`endOnSilenceMs` stops after that much silence that follows speech, and
|
|
229
|
+
`maxDurationMs` (30 s at most) always stops. `stopListening()` resolves with
|
|
230
|
+
`{ text, language, durationMs, sent, turn, send() }`, or with null when the
|
|
231
|
+
microphone gave no audio yet. `cancelListening()` discards the recording;
|
|
232
|
+
nothing is charged.
|
|
233
|
+
- `onTranscript(({ text, final }) => …)` gives the final text, and partial
|
|
234
|
+
text when the server's speech model gives it.
|
|
235
|
+
- The first press or `startListening()` asks for the microphone permission.
|
|
236
|
+
An empty result sends nothing.
|
|
237
|
+
- `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
|
|
238
|
+
the server detects the language.
|
|
239
|
+
- `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
|
|
240
|
+
an array of `'options'`, `'text'` and `'voice'`; one mode forces it. Without
|
|
241
|
+
it, every mode that the character permits is available. A baked
|
|
242
|
+
conversation permits only `['options']`, and a character without custom
|
|
243
|
+
replies permits only `'options'`. Another configuration throws an
|
|
244
|
+
`InputModeError` when the character loads. With `inputModes` set,
|
|
245
|
+
`sendMessage` with typed text needs `'text'`, a reply option needs
|
|
246
|
+
`'options'` or `'text'`, and listening needs `'voice'`.
|
|
247
|
+
`character.inputModes` gives the modes in use.
|
|
248
|
+
- The character owner pays the recognition at the provider's price, as for
|
|
249
|
+
the turn's language model.
|
|
250
|
+
|
|
191
251
|
## Runtime and lifecycle
|
|
192
252
|
|
|
193
253
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|
|
194
|
-
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading
|
|
195
|
-
|
|
254
|
+
`{ character, npcify, scene, camera, renderer, controls, dispose }` after loading,
|
|
255
|
+
and normalizes the idle animation orientation to face `+Z` by default
|
|
256
|
+
(`normalizeIdleOrientation: true`). Give the canvas a nonzero CSS size.
|
|
196
257
|
- `loadCharacter({ baseUrl, characterId, apiKey, parent, gazeTarget, shadows })`
|
|
197
|
-
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
258
|
+
loads and returns a `Character` instance for custom Three.js scenes and render loops
|
|
259
|
+
(`normalizeIdleOrientation` defaults to `false`).
|
|
198
260
|
- Call `unlockAudio()` (or `character.audio.unlock()`) during a user gesture before
|
|
199
261
|
starting playback.
|
|
200
262
|
- `onOptionsReady` delivers choices after playback finishes. `sendMessage(label)`
|
package/dist/index.d.ts
CHANGED
|
@@ -39,6 +39,7 @@ interface AvatarPublicInfo {
|
|
|
39
39
|
freeFirstNMessages?: number | null;
|
|
40
40
|
hostedCharacterEnabled?: boolean;
|
|
41
41
|
listedPublicly?: boolean;
|
|
42
|
+
r18?: boolean;
|
|
42
43
|
bakedConversationEnabled?: boolean;
|
|
43
44
|
bakedConversationRevision?: number;
|
|
44
45
|
allowCustomReply?: boolean;
|
|
@@ -280,6 +281,18 @@ interface StagingStreamBillingEvent {
|
|
|
280
281
|
}
|
|
281
282
|
type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent;
|
|
282
283
|
|
|
284
|
+
/**
|
|
285
|
+
* Voice input of conversations (docs/voice-input.md): the input modes of the
|
|
286
|
+
* SDKs, and the messages of a speech session between an SDK and the server.
|
|
287
|
+
*/
|
|
288
|
+
/** How a player can reply: a reply option, typed text, or speech. */
|
|
289
|
+
declare const INPUT_MODES: readonly ["options", "text", "voice"];
|
|
290
|
+
type InputMode = (typeof INPUT_MODES)[number];
|
|
291
|
+
/** A configuration of input modes that is invalid or that the character does not permit. */
|
|
292
|
+
declare class InputModeError extends Error {
|
|
293
|
+
constructor(message: string);
|
|
294
|
+
}
|
|
295
|
+
|
|
283
296
|
declare const BAKED_CONVERSATION_VERSION: 1;
|
|
284
297
|
/** Portable, language-neutral asset. Audio is embedded binary, never a signed URL. */
|
|
285
298
|
interface BakedConversationAsset {
|
|
@@ -319,9 +332,11 @@ declare const decodePerformance: <T = unknown>(input: ArrayBuffer | Uint8Array,
|
|
|
319
332
|
interface ConversationTurnOptions {
|
|
320
333
|
silent?: boolean;
|
|
321
334
|
parentStagingId?: string | null;
|
|
335
|
+
/** The turn's `Idempotency-Key`; a new UUID when absent. */
|
|
336
|
+
requestId?: string;
|
|
322
337
|
}
|
|
323
338
|
|
|
324
|
-
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'
|
|
339
|
+
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply'>>;
|
|
325
340
|
|
|
326
341
|
interface LlmThinkingPayload {
|
|
327
342
|
taskId?: string | null;
|
|
@@ -586,6 +601,13 @@ interface AnimationConfig {
|
|
|
586
601
|
loop?: boolean;
|
|
587
602
|
weight?: number;
|
|
588
603
|
}
|
|
604
|
+
interface AnimationControllerOptions {
|
|
605
|
+
/**
|
|
606
|
+
* Normalize the configured idle animation clip so its average torso heading
|
|
607
|
+
* and neck/head orientation face canonical `+Z`.
|
|
608
|
+
*/
|
|
609
|
+
normalizeIdleOrientation?: boolean;
|
|
610
|
+
}
|
|
589
611
|
interface CharacterAnimationPlayOptions {
|
|
590
612
|
fadeDuration?: number;
|
|
591
613
|
loop?: boolean;
|
|
@@ -618,14 +640,17 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
618
640
|
prepareAnimations?: (ids: string[]) => Promise<void>;
|
|
619
641
|
private readonly mixer;
|
|
620
642
|
private readonly clips;
|
|
643
|
+
private readonly rawClips;
|
|
644
|
+
private readonly normalizedIdleClips;
|
|
621
645
|
private readonly configs;
|
|
646
|
+
private readonly normalizeIdleOrientation;
|
|
622
647
|
private previewMixer;
|
|
623
648
|
private restorePreview;
|
|
624
649
|
private activeAction;
|
|
625
650
|
private activeAnimationId;
|
|
626
651
|
private idleAnimationId;
|
|
627
652
|
private pendingReturnToIdleAction;
|
|
628
|
-
constructor(vrm: VRM, idleAnimationId?: string | null);
|
|
653
|
+
constructor(vrm: VRM, idleAnimationId?: string | null, options?: AnimationControllerOptions);
|
|
629
654
|
getAnimationMixer(): AnimationMixer;
|
|
630
655
|
getActiveAction(): AnimationAction | null;
|
|
631
656
|
getActiveAnimationId(): string | null;
|
|
@@ -636,6 +661,7 @@ declare class AnimationController implements AnimationControllerLike {
|
|
|
636
661
|
previewAnimation(animation: VRMAnimation): () => void;
|
|
637
662
|
setIdleAnimation(animationId: string | null): void;
|
|
638
663
|
clearAnimations(): void;
|
|
664
|
+
private resolveClipForId;
|
|
639
665
|
playAnimation(animationId: string, fadeDuration?: number, loopOverride?: boolean): void;
|
|
640
666
|
playIdle(fadeDuration?: number): void;
|
|
641
667
|
stopAnimation(fadeDuration?: number): void;
|
|
@@ -744,6 +770,7 @@ declare const FacialPerformanceLayer: {
|
|
|
744
770
|
readonly Viseme: "viseme";
|
|
745
771
|
readonly Blink: "blink";
|
|
746
772
|
readonly Gaze: "gaze";
|
|
773
|
+
readonly Overlay: "overlay";
|
|
747
774
|
};
|
|
748
775
|
type FacialPerformanceLayerId = (typeof FacialPerformanceLayer)[keyof typeof FacialPerformanceLayer];
|
|
749
776
|
type FacialMouthMode = 'expression' | 'viseme';
|
|
@@ -799,6 +826,9 @@ declare class FacialPerformanceMixer {
|
|
|
799
826
|
playCue(channel: FacialChannelName, options?: FacialCueOptions): void;
|
|
800
827
|
setEmotion(channel: FacialChannelName, weight: number, mouthMode: FacialMouthMode): void;
|
|
801
828
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
829
|
+
/** Blend temporary emotions without changing timeline targets or mouth ownership. */
|
|
830
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
831
|
+
clearOverlay(): void;
|
|
802
832
|
releaseVisemeMouthMode(): void;
|
|
803
833
|
/**
|
|
804
834
|
* Ends the speech source without disturbing any other performance layer.
|
|
@@ -948,6 +978,8 @@ declare class StagingPlayer {
|
|
|
948
978
|
private readonly audioPlayer;
|
|
949
979
|
private readonly facialMixer;
|
|
950
980
|
private readonly animationController;
|
|
981
|
+
private readonly clock;
|
|
982
|
+
private pauseRevision;
|
|
951
983
|
private playbackController;
|
|
952
984
|
private generation;
|
|
953
985
|
private incremental;
|
|
@@ -958,6 +990,10 @@ declare class StagingPlayer {
|
|
|
958
990
|
private activeExpressionMouthMode;
|
|
959
991
|
private activeAnimation;
|
|
960
992
|
constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
|
|
993
|
+
get paused(): boolean;
|
|
994
|
+
pause(): void;
|
|
995
|
+
resume(): Promise<void>;
|
|
996
|
+
private waitUntilResumed;
|
|
961
997
|
play(timeline: StagingTimeline, options?: StagingPlaybackOptions): Promise<void>;
|
|
962
998
|
updateTimeline(timeline: StagingTimeline, complete?: boolean): void;
|
|
963
999
|
private startCursor;
|
|
@@ -1103,6 +1139,21 @@ interface NpcifyOptions {
|
|
|
1103
1139
|
volume?: number;
|
|
1104
1140
|
/** Initial speech audio muted flag. */
|
|
1105
1141
|
muted?: boolean;
|
|
1142
|
+
/**
|
|
1143
|
+
* Normalize the character's idle animation orientation so the torso and head
|
|
1144
|
+
* face canonical `+Z` on average while preserving idle motion.
|
|
1145
|
+
* Defaults to `true` for `Npcify` (Web Chats and Canvas API) and `false` for
|
|
1146
|
+
* `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
|
|
1147
|
+
*/
|
|
1148
|
+
normalizeIdleOrientation?: boolean;
|
|
1149
|
+
/**
|
|
1150
|
+
* How the player can reply: `'options'`, `'text'` and `'voice'`, in any
|
|
1151
|
+
* combination; one mode forces it. Without it, every mode that the
|
|
1152
|
+
* character permits. A baked conversation permits only `['options']`; a
|
|
1153
|
+
* character without custom replies permits only `options`. A mode that the
|
|
1154
|
+
* character does not permit throws an `InputModeError` on load.
|
|
1155
|
+
*/
|
|
1156
|
+
inputModes?: InputMode[];
|
|
1106
1157
|
/** Optional error callback. */
|
|
1107
1158
|
onError?: (error: Error) => void;
|
|
1108
1159
|
}
|
|
@@ -1169,7 +1220,8 @@ declare class StreamPlaybackController {
|
|
|
1169
1220
|
force?: boolean;
|
|
1170
1221
|
timeline?: StagingTimeline;
|
|
1171
1222
|
}): Promise<boolean>;
|
|
1172
|
-
|
|
1223
|
+
/** Returns the playback result for a timeline event. */
|
|
1224
|
+
handleEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1173
1225
|
cleanup(): void;
|
|
1174
1226
|
private resetRealtimeSpeech;
|
|
1175
1227
|
private playTimeline;
|
|
@@ -1210,6 +1262,10 @@ declare class CharacterExpressions {
|
|
|
1210
1262
|
setEmotion(name: FacialChannelName, weight?: number, mouthMode?: FacialMouthMode): void;
|
|
1211
1263
|
/** Clear the active emotion expression layer. */
|
|
1212
1264
|
clearEmotion(mouthMode?: FacialMouthMode): void;
|
|
1265
|
+
/** Set a temporary emotion overlay. Timeline emotions and visemes keep their state. */
|
|
1266
|
+
setOverlay(weights: Readonly<Record<string, number>>): void;
|
|
1267
|
+
/** Ease the temporary overlay back to the current timeline or idle face. */
|
|
1268
|
+
clearOverlay(): void;
|
|
1213
1269
|
/** Reset all expression layer targets to zero. */
|
|
1214
1270
|
reset(): void;
|
|
1215
1271
|
}
|
|
@@ -1313,6 +1369,12 @@ declare class CharacterRuntime {
|
|
|
1313
1369
|
/** Stop the currently playing skeletal animation. */
|
|
1314
1370
|
stopAnimation(fadeDuration?: number): void;
|
|
1315
1371
|
unlockAudio(): Promise<boolean>;
|
|
1372
|
+
/** Freeze the current NPCP timeline, audio, face, and animation. */
|
|
1373
|
+
pausePlayback(): void;
|
|
1374
|
+
/** Continue the same timeline. Call from a gesture if the browser requires it. */
|
|
1375
|
+
resumePlayback(): Promise<void>;
|
|
1376
|
+
/** Cancel current output and pending audio/animation loads. */
|
|
1377
|
+
interruptPlayback(): void;
|
|
1316
1378
|
getExpressions(): string[];
|
|
1317
1379
|
getMouthExpressionNames(): string[];
|
|
1318
1380
|
getExpressionNames(): string[];
|
|
@@ -1337,6 +1399,8 @@ declare class CharacterRuntime {
|
|
|
1337
1399
|
lookAt(x: number, y: number, z: number): void;
|
|
1338
1400
|
/** Reset the character's gaze target so the eyes look straight ahead. */
|
|
1339
1401
|
clearLookAt(): void;
|
|
1402
|
+
/** Briefly look at an object; release restores the prior target and gaze mode. */
|
|
1403
|
+
lookAtTemporarily(target: Object3D): () => void;
|
|
1340
1404
|
startPerformanceRecording(options?: PerformanceRecordingStartOptions): void;
|
|
1341
1405
|
stopPerformanceRecording(): PerformanceRecording | null;
|
|
1342
1406
|
getPerformanceRecording(): PerformanceRecording | null;
|
|
@@ -1366,6 +1430,97 @@ declare class CharacterRuntime {
|
|
|
1366
1430
|
protected readonly reportError: (error: Error, context?: string) => void;
|
|
1367
1431
|
}
|
|
1368
1432
|
|
|
1433
|
+
declare class MicrophoneError extends Error {
|
|
1434
|
+
readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
|
|
1435
|
+
constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
|
|
1436
|
+
}
|
|
1437
|
+
/** One frame of 16 kHz mono PCM and its loudness (RMS, 0 to 1). */
|
|
1438
|
+
type CaptureFrame = {
|
|
1439
|
+
pcm: Int16Array;
|
|
1440
|
+
level: number;
|
|
1441
|
+
};
|
|
1442
|
+
/** A source of 16 kHz frames. The SDK uses the microphone; tests use a fake. */
|
|
1443
|
+
interface AudioCapture {
|
|
1444
|
+
start(onFrame: (frame: CaptureFrame) => void): Promise<void>;
|
|
1445
|
+
/** Ends the capture; the last partial frame arrives before this resolves. */
|
|
1446
|
+
stop(): Promise<void>;
|
|
1447
|
+
/** Ends the capture without its last frame. */
|
|
1448
|
+
cancel(): void;
|
|
1449
|
+
}
|
|
1450
|
+
|
|
1451
|
+
type ListeningState = 'idle' | 'starting' | 'listening' | 'recognizing';
|
|
1452
|
+
interface ListeningOptions {
|
|
1453
|
+
/** BCP 47 hint, for example `ja`, `zh` or `en`. Without it the language is detected. */
|
|
1454
|
+
language?: string | null;
|
|
1455
|
+
/** `false`: recognise only; send the text later with `result.send()`. Default `true`. */
|
|
1456
|
+
send?: boolean;
|
|
1457
|
+
/** The recording stops at this time. Default and maximum: 30 seconds. */
|
|
1458
|
+
maxDurationMs?: number;
|
|
1459
|
+
/** For listening without a release: stop after this much silence that follows speech. */
|
|
1460
|
+
endOnSilenceMs?: number | null;
|
|
1461
|
+
parentStagingId?: string | null;
|
|
1462
|
+
silent?: boolean;
|
|
1463
|
+
}
|
|
1464
|
+
interface SpeechInputResult {
|
|
1465
|
+
/** The recognised text; empty when no speech was recognised (then no turn starts). */
|
|
1466
|
+
text: string;
|
|
1467
|
+
/** The detected language, when the recognition vendor gives it. */
|
|
1468
|
+
language: string | null;
|
|
1469
|
+
durationMs: number;
|
|
1470
|
+
/** The server started the turn with this text. */
|
|
1471
|
+
sent: boolean;
|
|
1472
|
+
/** The final payload of the turn, when it was sent. */
|
|
1473
|
+
turn: Promise<StagingResultPayload> | null;
|
|
1474
|
+
/** Sends the text as the player's message, when it was not sent. */
|
|
1475
|
+
send(): Promise<StagingResultPayload>;
|
|
1476
|
+
}
|
|
1477
|
+
interface TranscriptEvent {
|
|
1478
|
+
text: string;
|
|
1479
|
+
/** False for the text so far while the player speaks. */
|
|
1480
|
+
final: boolean;
|
|
1481
|
+
}
|
|
1482
|
+
declare class SpeechInputError extends Error {
|
|
1483
|
+
readonly code: string;
|
|
1484
|
+
constructor(code: string, message?: string);
|
|
1485
|
+
}
|
|
1486
|
+
|
|
1487
|
+
/** What push-to-talk drives: a `Character` or `Npcify` instance. */
|
|
1488
|
+
interface PushToTalkTarget {
|
|
1489
|
+
readonly listeningState: ListeningState;
|
|
1490
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1491
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1492
|
+
cancelListening(): void;
|
|
1493
|
+
}
|
|
1494
|
+
interface PushToTalkOptions {
|
|
1495
|
+
/** A button or any element to press and hold. */
|
|
1496
|
+
element?: HTMLElement | null;
|
|
1497
|
+
/** A `KeyboardEvent.code` to hold. Default `'Space'`; `null` binds no key. */
|
|
1498
|
+
key?: string | null;
|
|
1499
|
+
/** A shorter press sends nothing. Default 300 ms. */
|
|
1500
|
+
minHoldMs?: number;
|
|
1501
|
+
/** The options of each listening, or a function that gives them at each press. */
|
|
1502
|
+
listening?: ListeningOptions | (() => ListeningOptions);
|
|
1503
|
+
/** False while the player cannot reply (for example while options are locked). */
|
|
1504
|
+
enabled?: () => boolean;
|
|
1505
|
+
onResult?: (result: SpeechInputResult) => void;
|
|
1506
|
+
/**
|
|
1507
|
+
* A press sent nothing: it was shorter than `minHoldMs`, or it ended before
|
|
1508
|
+
* the microphone gave any audio (a permission prompt, a slow device). A UI
|
|
1509
|
+
* can say "hold to talk".
|
|
1510
|
+
*/
|
|
1511
|
+
onShortPress?: () => void;
|
|
1512
|
+
onError?: (error: unknown) => void;
|
|
1513
|
+
}
|
|
1514
|
+
/**
|
|
1515
|
+
* Press and hold to talk (docs/voice-input.md): an element, a key, or both.
|
|
1516
|
+
* The key does not start while the focus is in a field or on another
|
|
1517
|
+
* control, key repeat is ignored, a short press sends nothing, and window
|
|
1518
|
+
* blur, a hidden tab and Escape cancel. A long press on the element does not
|
|
1519
|
+
* open a menu, select text or scroll. A release anywhere ends the press.
|
|
1520
|
+
* Returns a function that unbinds.
|
|
1521
|
+
*/
|
|
1522
|
+
declare const bindPushToTalk: (target: PushToTalkTarget, options?: PushToTalkOptions) => () => void;
|
|
1523
|
+
|
|
1369
1524
|
declare class Character extends CharacterRuntime {
|
|
1370
1525
|
private readonly conversationSocket;
|
|
1371
1526
|
private renderSessionId;
|
|
@@ -1376,8 +1531,15 @@ declare class Character extends CharacterRuntime {
|
|
|
1376
1531
|
private readonly liveConversation;
|
|
1377
1532
|
private baked;
|
|
1378
1533
|
private readonly authoring;
|
|
1534
|
+
protected readonly normalizeIdleOrientation: boolean;
|
|
1379
1535
|
private readonly bakedAsset?;
|
|
1380
1536
|
private animationRefreshId;
|
|
1537
|
+
private readonly voice;
|
|
1538
|
+
private readonly configuredInputModes;
|
|
1539
|
+
private resolvedInputModes;
|
|
1540
|
+
private latestOptions;
|
|
1541
|
+
private listeningStateCallback;
|
|
1542
|
+
private transcriptCallback;
|
|
1381
1543
|
conversationId: string | null;
|
|
1382
1544
|
constructor(options: NpcifyOptions);
|
|
1383
1545
|
load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
|
|
@@ -1399,18 +1561,57 @@ declare class Character extends CharacterRuntime {
|
|
|
1399
1561
|
replayStaging(stagingId: string): Promise<StagingResultPayload>;
|
|
1400
1562
|
replayStaging(conversationId: string, stagingId: string, options?: PlayStagingOptions): Promise<StagingResultPayload>;
|
|
1401
1563
|
get isBakedConversation(): boolean;
|
|
1564
|
+
/**
|
|
1565
|
+
* The input modes in use: the configured modes, or every mode that the
|
|
1566
|
+
* character permits. Before `loadAssets`, the configuration or all modes.
|
|
1567
|
+
*/
|
|
1568
|
+
get inputModes(): readonly InputMode[];
|
|
1569
|
+
get listeningState(): ListeningState;
|
|
1570
|
+
/**
|
|
1571
|
+
* Opens the microphone and streams the speech to the server. Resolves with
|
|
1572
|
+
* the result when listening ends (`stopListening`, `endOnSilenceMs` or
|
|
1573
|
+
* `maxDurationMs`), or with null when it is cancelled. The first call asks
|
|
1574
|
+
* for the microphone permission.
|
|
1575
|
+
*/
|
|
1576
|
+
startListening(options?: ListeningOptions): Promise<SpeechInputResult | null>;
|
|
1577
|
+
/**
|
|
1578
|
+
* Ends the recording; resolves as soon as the server recognised the text.
|
|
1579
|
+
* Before the microphone gave any audio, it cancels and resolves null.
|
|
1580
|
+
*/
|
|
1581
|
+
stopListening(): Promise<SpeechInputResult | null>;
|
|
1582
|
+
/** Discards the recording. Nothing is recognised or charged. */
|
|
1583
|
+
cancelListening(): void;
|
|
1584
|
+
/** Press and hold an element or a key (default Space) to talk. Returns an unbind function. */
|
|
1585
|
+
bindPushToTalk(options?: PushToTalkOptions): () => void;
|
|
1586
|
+
onListeningState(callback: (state: ListeningState) => void): () => void;
|
|
1587
|
+
/** Partial text while the player speaks (`final: false`), then the final text. */
|
|
1588
|
+
onTranscript(callback: (event: TranscriptEvent) => void): () => void;
|
|
1402
1589
|
getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
|
|
1403
1590
|
acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
|
|
1404
1591
|
playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
|
|
1592
|
+
/**
|
|
1593
|
+
* Plays one live staging event from another transport, such as an actor
|
|
1594
|
+
* stage. A timeline event returns its playback result.
|
|
1595
|
+
*/
|
|
1596
|
+
playStreamEvent(event: StagingStreamEvent): Promise<boolean> | undefined;
|
|
1405
1597
|
setRenderSessionId(renderSessionId: string | null): void;
|
|
1406
1598
|
onStagingResult(callback: (payload: StagingResultPayload) => Promise<void> | void): () => void;
|
|
1407
1599
|
onThinking(callback: (payload: LlmThinkingPayload) => Promise<void> | void): () => void;
|
|
1408
1600
|
private resetAssets;
|
|
1409
1601
|
private readonly getAssetInfo;
|
|
1410
1602
|
private handleStagingResult;
|
|
1603
|
+
/** Throws when the input modes do not include `mode`. */
|
|
1604
|
+
private assertInputMode;
|
|
1605
|
+
/**
|
|
1606
|
+
* A typed reply needs the `text` mode; one of the current reply options
|
|
1607
|
+
* needs `options` or `text`. Without a configuration the server's rules
|
|
1608
|
+
* apply alone, as before.
|
|
1609
|
+
*/
|
|
1610
|
+
private assertReplyMode;
|
|
1411
1611
|
private invokeThinkingCallback;
|
|
1412
1612
|
}
|
|
1413
1613
|
declare class Npcify extends Character {
|
|
1614
|
+
constructor(options: NpcifyOptions);
|
|
1414
1615
|
}
|
|
1415
1616
|
|
|
1416
1617
|
interface LoadCharacterOptions extends CharacterOptions {
|
|
@@ -1501,6 +1702,9 @@ interface MountNpcifyOptions {
|
|
|
1501
1702
|
apiToken?: string;
|
|
1502
1703
|
serverRendering?: boolean;
|
|
1503
1704
|
assetProfile?: 'original' | 'web-optimized-v1';
|
|
1705
|
+
normalizeIdleOrientation?: boolean;
|
|
1706
|
+
/** How the player can reply: `'options'`, `'text'`, `'voice'` (docs/voice-input.md). */
|
|
1707
|
+
inputModes?: InputMode[];
|
|
1504
1708
|
onError?: (error: Error) => void;
|
|
1505
1709
|
}
|
|
1506
1710
|
interface NpcifyView {
|
|
@@ -1515,4 +1719,4 @@ interface NpcifyView {
|
|
|
1515
1719
|
/** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
|
|
1516
1720
|
declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
|
|
1517
1721
|
|
|
1518
|
-
export { AnimationController, type AnimationControllerLike, type AssetLoadProgress, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, type LlmThinkingPayload, type LoadCharacterOptions, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
|
1722
|
+
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|