@npcify/web 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -0
- package/dist/index.d.ts +301 -16
- package/dist/index.js +1506 -240
- package/examples/index.html +1 -0
- package/examples/main.js +22 -0
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -125,10 +125,13 @@ animate()
|
|
|
125
125
|
- **Dialogue & Speech State**:
|
|
126
126
|
- `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
|
|
127
127
|
- `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
|
|
128
|
+
- `character.onConnectionClosed(callback)`: The server closed the conversation connection ([lifecycle](#runtime-and-lifecycle)).
|
|
128
129
|
- **Voice Input** ([details](#voice-input-and-input-modes)):
|
|
129
130
|
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
130
131
|
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
131
132
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
133
|
+
- **Sight** ([details](#sight)):
|
|
134
|
+
- `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
|
|
132
135
|
|
|
133
136
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
134
137
|
|
|
@@ -252,6 +255,70 @@ character.onListeningState((state) => {
|
|
|
252
255
|
- The character owner pays the recognition at the price of the provider that
|
|
253
256
|
the owner chose, as for the turn's language model.
|
|
254
257
|
|
|
258
|
+
## Sight
|
|
259
|
+
|
|
260
|
+
The character can see the player's camera when the character owner turned
|
|
261
|
+
sight on (the Sight tab of the character in the dashboard, where the owner
|
|
262
|
+
also chooses the vision provider). The SDK captures small camera frames and
|
|
263
|
+
sends them on the conversation connection. The server sends each frame to the
|
|
264
|
+
vision provider and keeps only a short text observation; it does not keep the
|
|
265
|
+
frames. The character reads the observations of the last minute in its next
|
|
266
|
+
reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
|
|
267
|
+
|
|
268
|
+
```js
|
|
269
|
+
const see = document.querySelector('#see')
|
|
270
|
+
const report = (error) => { status.textContent = error.message }
|
|
271
|
+
see.hidden = character.sightState === 'unavailable'
|
|
272
|
+
see.onclick = () => {
|
|
273
|
+
if (character.sightState === 'off') {
|
|
274
|
+
character.startSight().catch(report)
|
|
275
|
+
} else {
|
|
276
|
+
character.stopSight()
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
character.onSightState((state, error) => {
|
|
280
|
+
see.hidden = state === 'unavailable'
|
|
281
|
+
see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
|
|
282
|
+
if (error) report(error)
|
|
283
|
+
})
|
|
284
|
+
character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
- `startSight({ source, intervalMs, motion, onError })` starts sight in the
|
|
288
|
+
current conversation; call `createConversation()` first. The first call
|
|
289
|
+
asks for the camera permission. `source` is `'camera'` (default) or a
|
|
290
|
+
`MediaStream` that you opened, for example a screen share. `stopSight()`
|
|
291
|
+
closes the camera that the SDK opened. The SDK never stops the tracks of a
|
|
292
|
+
stream that you give.
|
|
293
|
+
- Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
|
|
294
|
+
smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
|
|
295
|
+
it compares a 32×18 grid of the luminance with the grid before. When the
|
|
296
|
+
mean change is more than `motion.threshold` (default 8, on a 0 to 255
|
|
297
|
+
scale), a frame goes, one each second at most. Else a frame goes every
|
|
298
|
+
`intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
|
|
299
|
+
frames only at the interval. One frame at a time waits for the server: a
|
|
300
|
+
frame that comes due during the wait goes at the answer. While the page is
|
|
301
|
+
hidden, the SDK captures nothing.
|
|
302
|
+
- `sightState` is `'unavailable'` (the owner did not turn sight on, or the
|
|
303
|
+
conversation is baked), `'off'`, `'starting'` or `'on'`.
|
|
304
|
+
`onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
|
|
305
|
+
gives each observation of the current conversation:
|
|
306
|
+
`{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
|
|
307
|
+
- Errors are `SightError`s with a `code`. `startSight()` rejects with
|
|
308
|
+
`SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
|
|
309
|
+
`SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
|
|
310
|
+
or a code of the conversation connection.
|
|
311
|
+
- These codes stop sight after it started and go to `onError` and to the
|
|
312
|
+
`error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
|
|
313
|
+
`INSUFFICIENT_POINTS` (the owner has no points),
|
|
314
|
+
`CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
|
|
315
|
+
shared screen stopped). The SDK skips one frame and continues for
|
|
316
|
+
`SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
|
|
317
|
+
`SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
|
|
318
|
+
- `endConversation()`, `dispose()` and `loadAssets()` stop sight.
|
|
319
|
+
- The character owner pays the analysis of each frame at the price of the
|
|
320
|
+
provider that the owner chose. A refused or failed frame costs nothing.
|
|
321
|
+
|
|
255
322
|
## Runtime and lifecycle
|
|
256
323
|
|
|
257
324
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|
|
@@ -277,6 +344,16 @@ character.onListeningState((state) => {
|
|
|
277
344
|
baked playback requires browser rendering.
|
|
278
345
|
- Keys may cover multiple characters. Editing scopes, rotating, or deleting keys
|
|
279
346
|
affects subsequent requests and disconnects associated conversation sockets.
|
|
347
|
+
- `character.onConnectionClosed(({ reason, code, conversationId }) => …)` reports
|
|
348
|
+
a close of the conversation connection by the server: a refused or failed
|
|
349
|
+
join (`code`), or a key that was edited, rotated or deleted (`code: null`).
|
|
350
|
+
The SDK does not use that connection again: the next `connect`,
|
|
351
|
+
`sendMessage` or `startListening` opens a new one. Joins and requests that
|
|
352
|
+
wait on a closed connection fail at once with a `ConversationConnectionError`
|
|
353
|
+
(`CONVERSATION_DISCONNECTED`); a wait without an answer in time fails with
|
|
354
|
+
`CONVERSATION_TIMEOUT`. A refused conversation fails at once with
|
|
355
|
+
`CONVERSATION_ACCESS_DENIED` until `endConversation()`; after another join
|
|
356
|
+
code (`CONVERSATION_JOIN_FAILED` or a newer one) the next call tries again.
|
|
280
357
|
- ESM only; `require()` is not a supported package entry point. Browser APIs are
|
|
281
358
|
used when creating a view; do not call `mountNpcify` during server rendering.
|
|
282
359
|
|
package/dist/index.d.ts
CHANGED
|
@@ -3,6 +3,49 @@ import { VRMAnimation } from '@pixiv/three-vrm-animation';
|
|
|
3
3
|
import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
|
|
4
4
|
import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
|
|
5
5
|
|
|
6
|
+
/**
|
|
7
|
+
* Sight (docs/sight.md): a client captures frames, the server sends each
|
|
8
|
+
* frame to the vision provider that the owner of the character chose, and the
|
|
9
|
+
* character reads the text observations in its next replies. Frames are never
|
|
10
|
+
* stored.
|
|
11
|
+
*/
|
|
12
|
+
/** The vision providers. No other provider takes over when one fails. */
|
|
13
|
+
declare const SIGHT_PROVIDERS: readonly ["gemini", "openai"];
|
|
14
|
+
type SightProvider = (typeof SIGHT_PROVIDERS)[number];
|
|
15
|
+
/** The codes of a refused frame. A newer server can send other codes. */
|
|
16
|
+
declare const SIGHT_ERRORS: readonly ["SIGHT_DISABLED", "SIGHT_RATE_LIMITED", "SIGHT_FRAME_TOO_LARGE", "SIGHT_FRAME_INVALID", "SIGHT_PROVIDER_FAILED", "INSUFFICIENT_POINTS", "CONVERSATION_ACCESS_DENIED"];
|
|
17
|
+
type SightErrorCode = (typeof SIGHT_ERRORS)[number];
|
|
18
|
+
type SightPersonPosition = 'left' | 'center' | 'right';
|
|
19
|
+
/** One person or avatar in a frame. */
|
|
20
|
+
type SightPerson = {
|
|
21
|
+
position: SightPersonPosition;
|
|
22
|
+
/** What the person does, in five words or fewer. */
|
|
23
|
+
action: string;
|
|
24
|
+
/** The person faces or addresses the camera. */
|
|
25
|
+
facingCamera: boolean;
|
|
26
|
+
};
|
|
27
|
+
/** What the vision model saw in one frame. */
|
|
28
|
+
type SightObservation = {
|
|
29
|
+
/** The number of people or avatars in the frame. */
|
|
30
|
+
people: number;
|
|
31
|
+
persons: SightPerson[];
|
|
32
|
+
/** Each readable piece of text, verbatim. */
|
|
33
|
+
text: string[];
|
|
34
|
+
};
|
|
35
|
+
/** `sight_observation`: an observation of the conversation, sent to its room. */
|
|
36
|
+
type SightObservationEvent = {
|
|
37
|
+
conversationId: string;
|
|
38
|
+
observationId: string;
|
|
39
|
+
/** The capture time that the client sent, as an ISO 8601 string. */
|
|
40
|
+
capturedAt: string;
|
|
41
|
+
observation: SightObservation;
|
|
42
|
+
/** One sentence about the frame, for the character. */
|
|
43
|
+
summary: string;
|
|
44
|
+
model: string;
|
|
45
|
+
/** The points charged to the owner; null while the charge is pending. */
|
|
46
|
+
costPoints: number | null;
|
|
47
|
+
};
|
|
48
|
+
|
|
6
49
|
type TtsProviderId = string;
|
|
7
50
|
type NullableTtsProviderId = TtsProviderId | null;
|
|
8
51
|
|
|
@@ -18,6 +61,14 @@ type Sqt = {
|
|
|
18
61
|
quaternion?: [number, number, number, number] | null;
|
|
19
62
|
translation?: [number, number, number] | null;
|
|
20
63
|
};
|
|
64
|
+
/**
|
|
65
|
+
* The owner's sight switch and vision provider (docs/sight.md). `enabled` is
|
|
66
|
+
* true only with a provider; an SDK captures frames only then.
|
|
67
|
+
*/
|
|
68
|
+
type AvatarSightSettings = {
|
|
69
|
+
enabled: boolean;
|
|
70
|
+
provider: SightProvider | null;
|
|
71
|
+
};
|
|
21
72
|
interface AvatarPublicInfo {
|
|
22
73
|
id: string;
|
|
23
74
|
name: string | null;
|
|
@@ -48,8 +99,12 @@ interface AvatarPublicInfo {
|
|
|
48
99
|
* `allowCustomReply`. Missing on older servers: off.
|
|
49
100
|
*/
|
|
50
101
|
speechInputEnabled?: boolean;
|
|
102
|
+
/** Sight (Sight tab). Missing on older servers: off. */
|
|
103
|
+
sight?: AvatarSightSettings;
|
|
51
104
|
autoPlay: boolean;
|
|
52
105
|
serverRenderingEnabled: boolean;
|
|
106
|
+
/** The character's body motion is generated live in each turn (missing on older servers). */
|
|
107
|
+
realtimeMotionEnabled?: boolean;
|
|
53
108
|
cameraSqt?: Sqt | null;
|
|
54
109
|
}
|
|
55
110
|
|
|
@@ -192,12 +247,45 @@ interface StagingAnimationClip {
|
|
|
192
247
|
animation: string;
|
|
193
248
|
loop: boolean;
|
|
194
249
|
}
|
|
250
|
+
/** A part of the staging whose body motion is generated live (NPCP 1.1). */
|
|
251
|
+
interface StagingMotionClip {
|
|
252
|
+
/** Seconds from the start of the staging. */
|
|
253
|
+
startTime: number;
|
|
254
|
+
/** Seconds from the start of the staging. */
|
|
255
|
+
endTime: number;
|
|
256
|
+
/** A short English description of the body motion. */
|
|
257
|
+
prompt: string;
|
|
258
|
+
}
|
|
259
|
+
type StagingMotionFramesSource = {
|
|
260
|
+
type: 's3';
|
|
261
|
+
url: string;
|
|
262
|
+
} | {
|
|
263
|
+
type: 'bytes';
|
|
264
|
+
data: Uint8Array;
|
|
265
|
+
};
|
|
266
|
+
/**
|
|
267
|
+
* The stored live motion of a staging (NPCP 1.1): pose frames on the staging
|
|
268
|
+
* clock, in the layout of `motion_frames` (see `motion-frames.ts`).
|
|
269
|
+
*/
|
|
270
|
+
interface StagingMotionTrack {
|
|
271
|
+
fps: number;
|
|
272
|
+
frameCount: number;
|
|
273
|
+
/** VRM humanoid bone names, the order of the rotations of a frame. */
|
|
274
|
+
bones: string[];
|
|
275
|
+
/** Hips height of the source rest pose in meters. */
|
|
276
|
+
restHipsHeight: number;
|
|
277
|
+
frames: StagingMotionFramesSource;
|
|
278
|
+
}
|
|
195
279
|
interface StagingTimeline {
|
|
196
280
|
/** Total duration in seconds. */
|
|
197
281
|
duration: number;
|
|
198
282
|
dialogue: StagingDialogueBlock[];
|
|
199
283
|
expressions: StagingExpressionClip[];
|
|
200
284
|
animations: StagingAnimationClip[];
|
|
285
|
+
/** Live motion prompts of a character with real-time motion (NPCP 1.1). */
|
|
286
|
+
motions?: StagingMotionClip[];
|
|
287
|
+
/** Stored live motion frames, on reads of a completed staging (NPCP 1.1). */
|
|
288
|
+
motion?: StagingMotionTrack;
|
|
201
289
|
}
|
|
202
290
|
declare const STAGING_PROTOCOL_VERSION: 1;
|
|
203
291
|
interface StagingResultPayload {
|
|
@@ -262,6 +350,7 @@ interface StagingStreamSpeechErrorEvent {
|
|
|
262
350
|
stagingId: string;
|
|
263
351
|
speechId: string;
|
|
264
352
|
streamId: string;
|
|
353
|
+
/** Always `SPEECH_FAILED`; the cause stays in the server logs. */
|
|
265
354
|
message: string;
|
|
266
355
|
}
|
|
267
356
|
interface StagingStreamOptionsEvent {
|
|
@@ -275,6 +364,10 @@ interface StagingStreamErrorEvent {
|
|
|
275
364
|
type: 'error';
|
|
276
365
|
taskId?: string;
|
|
277
366
|
stagingId?: string;
|
|
367
|
+
/**
|
|
368
|
+
* A client error of the server, for example `INSUFFICIENT_POINTS`, or
|
|
369
|
+
* `TURN_FAILED`. Vendor text stays in the server logs.
|
|
370
|
+
*/
|
|
278
371
|
message: string;
|
|
279
372
|
}
|
|
280
373
|
interface StagingStreamBillingEvent {
|
|
@@ -284,7 +377,28 @@ interface StagingStreamBillingEvent {
|
|
|
284
377
|
usageCharge?: UsageChargeSummary | null;
|
|
285
378
|
availablePoints?: number;
|
|
286
379
|
}
|
|
287
|
-
|
|
380
|
+
/**
|
|
381
|
+
* Live pose frames of a character with real-time motion (NPCP 1.1). Older
|
|
382
|
+
* receivers ignore the event. Frame `startFrame + i` plays at
|
|
383
|
+
* `(startFrame + i) / fps` seconds of the staging clock. A frame of a later
|
|
384
|
+
* event replaces the frame with the same index unless its `revision` is lower.
|
|
385
|
+
*/
|
|
386
|
+
interface StagingStreamMotionFramesEvent {
|
|
387
|
+
type: 'motion_frames';
|
|
388
|
+
taskId: string;
|
|
389
|
+
stagingId: string;
|
|
390
|
+
revision: number;
|
|
391
|
+
fps: number;
|
|
392
|
+
startFrame: number;
|
|
393
|
+
frameCount: number;
|
|
394
|
+
/** VRM humanoid bone names, the order of the rotations of a frame. */
|
|
395
|
+
bones: string[];
|
|
396
|
+
/** Hips height of the source rest pose in meters. */
|
|
397
|
+
restHipsHeight: number;
|
|
398
|
+
/** `frameCount x (3 + 4 x bones.length)` float32 values, little endian. */
|
|
399
|
+
frames: Uint8Array;
|
|
400
|
+
}
|
|
401
|
+
type StagingStreamEvent = StagingStreamStartEvent | StagingStreamTextDeltaEvent | StagingStreamTimelineEvent | StagingStreamPrefixEvent | StagingStreamLiveEvent | StagingStreamSpeechChunkEvent | StagingStreamSpeechCompleteEvent | StagingStreamSpeechErrorEvent | StagingStreamOptionsEvent | StagingStreamErrorEvent | StagingStreamBillingEvent | StagingStreamMotionFramesEvent;
|
|
288
402
|
|
|
289
403
|
/**
|
|
290
404
|
* Voice input of conversations (docs/voice-input.md): the input modes of the
|
|
@@ -341,7 +455,7 @@ interface ConversationTurnOptions {
|
|
|
341
455
|
requestId?: string;
|
|
342
456
|
}
|
|
343
457
|
|
|
344
|
-
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled'>>;
|
|
458
|
+
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled' | 'sight'>>;
|
|
345
459
|
|
|
346
460
|
interface LlmThinkingPayload {
|
|
347
461
|
taskId?: string | null;
|
|
@@ -994,8 +1108,14 @@ declare class StagingPlayer {
|
|
|
994
1108
|
private activeExpression;
|
|
995
1109
|
private activeExpressionMouthMode;
|
|
996
1110
|
private activeAnimation;
|
|
1111
|
+
private currentTrackTime;
|
|
997
1112
|
constructor(audioPlayer: AudioPlayer, facialMixer: FacialPerformanceMixer, animationController: AnimationControllerLike);
|
|
998
1113
|
get paused(): boolean;
|
|
1114
|
+
/**
|
|
1115
|
+
* Seconds of the staging clock that the tracks show now, or null when no
|
|
1116
|
+
* staging plays (also while playback waits for the next dialogue block).
|
|
1117
|
+
*/
|
|
1118
|
+
get trackTime(): number | null;
|
|
999
1119
|
pause(): void;
|
|
1000
1120
|
resume(): Promise<void>;
|
|
1001
1121
|
private waitUntilResumed;
|
|
@@ -1016,6 +1136,55 @@ declare class StagingPlayer {
|
|
|
1016
1136
|
private clearPerformance;
|
|
1017
1137
|
}
|
|
1018
1138
|
|
|
1139
|
+
/**
|
|
1140
|
+
* How the host uses the root of live motion. `anchored` (default) removes the
|
|
1141
|
+
* horizontal hips movement of the turn, so that the character stays where the
|
|
1142
|
+
* host placed it; height and turns stay. `world` applies it as generated.
|
|
1143
|
+
*/
|
|
1144
|
+
type LiveMotionRootMode = 'anchored' | 'world';
|
|
1145
|
+
interface LiveMotionOptions {
|
|
1146
|
+
rootMode?: LiveMotionRootMode;
|
|
1147
|
+
/** Seconds to blend from idle into live frames. */
|
|
1148
|
+
blendInSeconds?: number;
|
|
1149
|
+
/** Seconds to blend from live frames back to idle. */
|
|
1150
|
+
blendOutSeconds?: number;
|
|
1151
|
+
}
|
|
1152
|
+
/**
|
|
1153
|
+
* Plays live motion frames (ADR 0011) on a VRM's normalized humanoid. Call
|
|
1154
|
+
* `update()` after the animation mixer and before `vrm.update()`: the live
|
|
1155
|
+
* pose blends over the mixer's pose (idle) with its own weight.
|
|
1156
|
+
*/
|
|
1157
|
+
declare class LiveMotionPlayer {
|
|
1158
|
+
private vrm;
|
|
1159
|
+
private readonly tracks;
|
|
1160
|
+
private readonly rootMode;
|
|
1161
|
+
private readonly blendIn;
|
|
1162
|
+
private readonly blendOut;
|
|
1163
|
+
private weight;
|
|
1164
|
+
/** The last applied pose, to blend out of it after the frames end. */
|
|
1165
|
+
private lastPose;
|
|
1166
|
+
private readonly scratch;
|
|
1167
|
+
constructor(options?: LiveMotionOptions);
|
|
1168
|
+
setVrm(vrm: VRM | null): void;
|
|
1169
|
+
/** Adds the frames of a `motion_frames` event; invalid events are dropped. */
|
|
1170
|
+
push(received: StagingStreamMotionFramesEvent): boolean;
|
|
1171
|
+
/** Loads the stored frames of a staging (a replay). */
|
|
1172
|
+
load(stagingId: string, track: StagingMotionTrack, fetchBytes?: (url: string) => Promise<Uint8Array>): Promise<void>;
|
|
1173
|
+
has(stagingId: string): boolean;
|
|
1174
|
+
forget(stagingId: string): void;
|
|
1175
|
+
/** The current live weight, 0 (idle) to 1 (live frames). */
|
|
1176
|
+
get liveWeight(): number;
|
|
1177
|
+
/**
|
|
1178
|
+
* Applies the live pose of `stagingId` at `time` seconds of its staging
|
|
1179
|
+
* clock; `time` null means that no staging plays now.
|
|
1180
|
+
*/
|
|
1181
|
+
update(delta: number, stagingId: string | null, time: number | null): void;
|
|
1182
|
+
private track;
|
|
1183
|
+
/** The interpolated frame at `time`, or null when its frames are missing. */
|
|
1184
|
+
private sample;
|
|
1185
|
+
private apply;
|
|
1186
|
+
}
|
|
1187
|
+
|
|
1019
1188
|
type Vector3Tuple = readonly [number, number, number];
|
|
1020
1189
|
interface Vector3Like {
|
|
1021
1190
|
readonly x: number;
|
|
@@ -1151,6 +1320,11 @@ interface NpcifyOptions {
|
|
|
1151
1320
|
* `Character` (`loadCharacter` / `createCharacter` Three.js game mode).
|
|
1152
1321
|
*/
|
|
1153
1322
|
normalizeIdleOrientation?: boolean;
|
|
1323
|
+
/**
|
|
1324
|
+
* Playback of live motion frames (characters with real-time motion). The
|
|
1325
|
+
* default `rootMode: 'anchored'` keeps the character where it stands.
|
|
1326
|
+
*/
|
|
1327
|
+
liveMotion?: LiveMotionOptions;
|
|
1154
1328
|
/**
|
|
1155
1329
|
* How the player can reply: `'options'`, `'text'` and `'voice'`, in any
|
|
1156
1330
|
* combination; one mode forces it. Without it, every mode that the
|
|
@@ -1188,6 +1362,10 @@ interface StreamPlaybackDependencies {
|
|
|
1188
1362
|
handleCallbackError: (context: string, error: unknown) => void;
|
|
1189
1363
|
reportError: (error: Error, context?: string) => void;
|
|
1190
1364
|
recordEvent?: (event: PerformanceEventPayload) => void;
|
|
1365
|
+
/** Live motion frames of the playing staging (NPCP 1.1). */
|
|
1366
|
+
onMotionFrames?: (event: StagingStreamMotionFramesEvent) => void;
|
|
1367
|
+
/** The stored live motion of a staging that starts to play (a replay). */
|
|
1368
|
+
onMotionTrack?: (stagingId: string, track: StagingMotionTrack) => void;
|
|
1191
1369
|
}
|
|
1192
1370
|
declare class StreamPlaybackController {
|
|
1193
1371
|
private readonly dependencies;
|
|
@@ -1214,6 +1392,8 @@ declare class StreamPlaybackController {
|
|
|
1214
1392
|
get revealedText(): string;
|
|
1215
1393
|
get options(): StagingResultPayload['options'];
|
|
1216
1394
|
get cursor(): CharacterPlaybackCursor | null;
|
|
1395
|
+
/** The staging that plays or waits to play, or null. */
|
|
1396
|
+
get activeStagingId(): string | null;
|
|
1217
1397
|
onStreamEvent(callback: (event: StagingStreamEvent) => Promise<void> | void): () => void;
|
|
1218
1398
|
onSpeechPlaybackStart(callback: (speech: StagingSpeechPlaybackEvent) => Promise<void> | void): () => void;
|
|
1219
1399
|
onCharacterReveal(callback: (speech: StagingSpeechPlaybackEvent, characterIndex: number, character: string) => void): () => void;
|
|
@@ -1300,6 +1480,7 @@ declare class CharacterRuntime {
|
|
|
1300
1480
|
protected readonly performanceRecorder: PerformanceRecorder;
|
|
1301
1481
|
protected readonly telemetry: ClientTelemetry;
|
|
1302
1482
|
protected readonly streamPlayback: StreamPlaybackController;
|
|
1483
|
+
protected readonly liveMotion: LiveMotionPlayer;
|
|
1303
1484
|
protected defaultParent: Object3D | null;
|
|
1304
1485
|
protected currentGazeTarget: Object3D | null;
|
|
1305
1486
|
/** The current gaze target came from `lookAt()`. */
|
|
@@ -1436,6 +1617,85 @@ declare class CharacterRuntime {
|
|
|
1436
1617
|
protected readonly reportError: (error: Error, context?: string) => void;
|
|
1437
1618
|
}
|
|
1438
1619
|
|
|
1620
|
+
/**
|
|
1621
|
+
* The code of a join that the server did not accept. Only
|
|
1622
|
+
* `CONVERSATION_ACCESS_DENIED` is final; the next join tries any other code
|
|
1623
|
+
* again. Another string is a code of a newer server.
|
|
1624
|
+
*/
|
|
1625
|
+
type ConversationJoinFailure = 'CONVERSATION_ACCESS_DENIED' | 'CONVERSATION_JOIN_FAILED' | (string & {});
|
|
1626
|
+
/**
|
|
1627
|
+
* `CONVERSATION_DISCONNECTED`: the connection closed, was lost or is not
|
|
1628
|
+
* open. `CONVERSATION_TIMEOUT`: no acknowledgement came in time.
|
|
1629
|
+
*/
|
|
1630
|
+
type ConversationConnectionErrorCode = ConversationJoinFailure | 'CONVERSATION_DISCONNECTED' | 'CONVERSATION_TIMEOUT';
|
|
1631
|
+
/**
|
|
1632
|
+
* A join or a request that the conversation connection could not complete
|
|
1633
|
+
* (docs/streaming_reply.md#web-sdk).
|
|
1634
|
+
*/
|
|
1635
|
+
declare class ConversationConnectionError extends Error {
|
|
1636
|
+
readonly code: ConversationConnectionErrorCode;
|
|
1637
|
+
/** The close or loss that failed the wait (`io server disconnect`, `transport close`, `connect_error`, ...), or null. */
|
|
1638
|
+
readonly reason: string | null;
|
|
1639
|
+
constructor(code: ConversationConnectionErrorCode, message: string,
|
|
1640
|
+
/** The close or loss that failed the wait (`io server disconnect`, `transport close`, `connect_error`, ...), or null. */
|
|
1641
|
+
reason?: string | null);
|
|
1642
|
+
}
|
|
1643
|
+
/** The server closed the conversation connection, or refused to open it. */
|
|
1644
|
+
interface ConversationConnectionClosedEvent {
|
|
1645
|
+
/** `io server disconnect`, or `connect_error` when the server refused the connection. */
|
|
1646
|
+
reason: string;
|
|
1647
|
+
/**
|
|
1648
|
+
* The join that made the server close the connection; null for another
|
|
1649
|
+
* close, for example after the API key was rotated, edited or deleted.
|
|
1650
|
+
*/
|
|
1651
|
+
code: ConversationJoinFailure | null;
|
|
1652
|
+
/** The conversation of that join, or null. */
|
|
1653
|
+
conversationId: string | null;
|
|
1654
|
+
}
|
|
1655
|
+
|
|
1656
|
+
/**
|
|
1657
|
+
* The code of a `SightError`: a refusal of the server (`SIGHT_ERRORS` of
|
|
1658
|
+
* `@npcify/shared`), a code of the conversation connection
|
|
1659
|
+
* (`CONVERSATION_DISCONNECTED`, `CONVERSATION_TIMEOUT`, ...), or a code of
|
|
1660
|
+
* the SDK. Another string is a code of a newer server.
|
|
1661
|
+
*/
|
|
1662
|
+
type SightFailureCode = SightErrorCode | 'CAMERA_PERMISSION_DENIED' | 'CAMERA_UNAVAILABLE' | 'SIGHT_SOURCE_ENDED' | 'SIGHT_ALREADY_STARTED' | 'CONVERSATION_REQUIRED' | (string & {});
|
|
1663
|
+
declare class SightError extends Error {
|
|
1664
|
+
readonly code: SightFailureCode;
|
|
1665
|
+
constructor(code: SightFailureCode, message?: string);
|
|
1666
|
+
}
|
|
1667
|
+
|
|
1668
|
+
/** `'camera'` opens the camera; a `MediaStream` is a video of the caller. */
|
|
1669
|
+
type SightSourceOption = MediaStream | 'camera';
|
|
1670
|
+
|
|
1671
|
+
/**
|
|
1672
|
+
* `'unavailable'`: the character info did not turn sight on, or the
|
|
1673
|
+
* conversation is baked. `'off'`: sight can start. `'starting'`: the camera
|
|
1674
|
+
* and the connection open. `'on'`: frames go to the server.
|
|
1675
|
+
*/
|
|
1676
|
+
type SightState = 'unavailable' | 'off' | 'starting' | 'on';
|
|
1677
|
+
interface SightOptions {
|
|
1678
|
+
/**
|
|
1679
|
+
* `'camera'` (default) opens the camera, and `stopSight()` closes it. A
|
|
1680
|
+
* `MediaStream` (a camera or a screen share of your page) stays yours:
|
|
1681
|
+
* the SDK never stops its tracks.
|
|
1682
|
+
*/
|
|
1683
|
+
source?: SightSourceOption;
|
|
1684
|
+
/** The longest time between two frames. Default 5000 ms; the minimum is 1000 ms. */
|
|
1685
|
+
intervalMs?: number;
|
|
1686
|
+
/**
|
|
1687
|
+
* A change of the picture sends a frame at once (one each second at most).
|
|
1688
|
+
* `threshold` is the mean change of the 32 × 18 luma grid between two
|
|
1689
|
+
* checks, on a 0 to 255 scale (default 8). `false` sends only at the
|
|
1690
|
+
* interval.
|
|
1691
|
+
*/
|
|
1692
|
+
motion?: {
|
|
1693
|
+
threshold?: number;
|
|
1694
|
+
} | false;
|
|
1695
|
+
/** The error that stopped sight after `startSight()` resolved. */
|
|
1696
|
+
onError?: (error: SightError) => void;
|
|
1697
|
+
}
|
|
1698
|
+
|
|
1439
1699
|
declare class MicrophoneError extends Error {
|
|
1440
1700
|
readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
|
|
1441
1701
|
constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
|
|
@@ -1536,18 +1796,18 @@ declare class Character extends CharacterRuntime {
|
|
|
1536
1796
|
private readonly notifiedStagingIds;
|
|
1537
1797
|
private readonly liveConversation;
|
|
1538
1798
|
private baked;
|
|
1799
|
+
/** The conversation whose live motion was warmed up. */
|
|
1800
|
+
private liveMotionPrepared;
|
|
1539
1801
|
private readonly authoring;
|
|
1540
1802
|
protected readonly normalizeIdleOrientation: boolean;
|
|
1541
1803
|
private readonly bakedAsset?;
|
|
1542
1804
|
private animationRefreshId;
|
|
1543
1805
|
private readonly voice;
|
|
1544
|
-
private readonly
|
|
1545
|
-
private
|
|
1546
|
-
/** The character's voice input switch, after the character loads. */
|
|
1547
|
-
private speechInput;
|
|
1548
|
-
private latestOptions;
|
|
1806
|
+
private readonly sight;
|
|
1807
|
+
private readonly inputs;
|
|
1549
1808
|
private listeningStateCallback;
|
|
1550
1809
|
private transcriptCallback;
|
|
1810
|
+
private connectionClosedCallback;
|
|
1551
1811
|
conversationId: string | null;
|
|
1552
1812
|
constructor(options: NpcifyOptions);
|
|
1553
1813
|
load(callback?: (info: AssetLoadProgress) => void): Promise<this>;
|
|
@@ -1555,6 +1815,13 @@ declare class Character extends CharacterRuntime {
|
|
|
1555
1815
|
reloadAnimations(info?: NpcifyAssetInfo): Promise<void>;
|
|
1556
1816
|
reloadAssets(): Promise<void>;
|
|
1557
1817
|
initializeRemote(): Promise<void>;
|
|
1818
|
+
/**
|
|
1819
|
+
* Warms up live motion for the current conversation, once, on the
|
|
1820
|
+
* player's intent: call it at the first focus or key in a reply field or
|
|
1821
|
+
* when a talk key goes down. It does nothing for a character without live
|
|
1822
|
+
* motion; the server limits warm-ups per player (ADR 0011).
|
|
1823
|
+
*/
|
|
1824
|
+
prepareLiveMotion(): Promise<boolean>;
|
|
1558
1825
|
createConversation(options?: {
|
|
1559
1826
|
autoConnect?: boolean;
|
|
1560
1827
|
autoPlay?: boolean;
|
|
@@ -1594,6 +1861,32 @@ declare class Character extends CharacterRuntime {
|
|
|
1594
1861
|
onListeningState(callback: (state: ListeningState) => void): () => void;
|
|
1595
1862
|
/** Partial text while the player speaks (`final: false`), then the final text. */
|
|
1596
1863
|
onTranscript(callback: (event: TranscriptEvent) => void): () => void;
|
|
1864
|
+
/**
|
|
1865
|
+
* `'unavailable'` until the character info turns sight on (its owner's
|
|
1866
|
+
* Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
|
|
1867
|
+
* `'on'`.
|
|
1868
|
+
*/
|
|
1869
|
+
get sightState(): SightState;
|
|
1870
|
+
/**
|
|
1871
|
+
* Lets the character see the player's camera (docs/sight.md). Needs a
|
|
1872
|
+
* conversation; the first call asks for the camera permission. Rejects
|
|
1873
|
+
* with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
|
|
1874
|
+
* `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
|
|
1875
|
+
*/
|
|
1876
|
+
startSight(options?: SightOptions): Promise<void>;
|
|
1877
|
+
/** Stops the frames. Closes the camera that `startSight` opened. */
|
|
1878
|
+
stopSight(): void;
|
|
1879
|
+
/** What the character saw: each observation of the current conversation. */
|
|
1880
|
+
onSight(callback: (event: SightObservationEvent) => void): () => void;
|
|
1881
|
+
/** The sight state. `error` is the `SightError` that stopped sight. */
|
|
1882
|
+
onSightState(callback: (state: SightState, error?: SightError) => void): () => void;
|
|
1883
|
+
/**
|
|
1884
|
+
* The server closed the conversation connection: a refused or failed join,
|
|
1885
|
+
* or an API key that was rotated, edited or deleted. The next `connect`,
|
|
1886
|
+
* `sendMessage` or `startListening` opens a new connection; a refused
|
|
1887
|
+
* conversation fails at once with `CONVERSATION_ACCESS_DENIED`.
|
|
1888
|
+
*/
|
|
1889
|
+
onConnectionClosed(callback: (event: ConversationConnectionClosedEvent) => void): () => void;
|
|
1597
1890
|
getConversationWorldline(conversationId?: string | null): Promise<ConversationWorldline>;
|
|
1598
1891
|
acceptAuthoringResult(payload: StagingResultPayload): Promise<void>;
|
|
1599
1892
|
playStaging(payload: StagingResultPayload, options?: PlayStagingOptions): Promise<boolean>;
|
|
@@ -1608,14 +1901,6 @@ declare class Character extends CharacterRuntime {
|
|
|
1608
1901
|
private resetAssets;
|
|
1609
1902
|
private readonly getAssetInfo;
|
|
1610
1903
|
private handleStagingResult;
|
|
1611
|
-
/** Throws when the input modes do not include `mode`. */
|
|
1612
|
-
private assertInputMode;
|
|
1613
|
-
/**
|
|
1614
|
-
* A typed reply needs the `text` mode; one of the current reply options
|
|
1615
|
-
* needs `options` or `text`. Without a configuration the server's rules
|
|
1616
|
-
* apply alone, as before.
|
|
1617
|
-
*/
|
|
1618
|
-
private assertReplyMode;
|
|
1619
1904
|
private invokeThinkingCallback;
|
|
1620
1905
|
}
|
|
1621
1906
|
declare class Npcify extends Character {
|
|
@@ -1727,4 +2012,4 @@ interface NpcifyView {
|
|
|
1727
2012
|
/** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
|
|
1728
2013
|
declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
|
|
1729
2014
|
|
|
1730
|
-
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
|
2015
|
+
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SightError, type SightFailureCode, type SightObservation, type SightObservationEvent, type SightOptions, type SightPerson, type SightState, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|