@npcify/web 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +66 -0
- package/dist/index.d.ts +119 -15
- package/dist/index.js +718 -70
- package/examples/index.html +1 -0
- package/examples/main.js +22 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -130,6 +130,8 @@ animate()
|
|
|
130
130
|
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
131
131
|
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
132
132
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
133
|
+
- **Sight** ([details](#sight)):
|
|
134
|
+
- `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
|
|
133
135
|
|
|
134
136
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
135
137
|
|
|
@@ -253,6 +255,70 @@ character.onListeningState((state) => {
|
|
|
253
255
|
- The character owner pays the recognition at the price of the provider that
|
|
254
256
|
the owner chose, as for the turn's language model.
|
|
255
257
|
|
|
258
|
+
## Sight
|
|
259
|
+
|
|
260
|
+
The character can see the player's camera when the character owner turned
|
|
261
|
+
sight on (the Sight tab of the character in the dashboard, where the owner
|
|
262
|
+
also chooses the vision provider). The SDK captures small camera frames and
|
|
263
|
+
sends them on the conversation connection. The server sends each frame to the
|
|
264
|
+
vision provider and keeps only a short text observation; it does not keep the
|
|
265
|
+
frames. The character reads the observations of the last minute in its next
|
|
266
|
+
reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
|
|
267
|
+
|
|
268
|
+
```js
|
|
269
|
+
const see = document.querySelector('#see')
|
|
270
|
+
const report = (error) => { status.textContent = error.message }
|
|
271
|
+
see.hidden = character.sightState === 'unavailable'
|
|
272
|
+
see.onclick = () => {
|
|
273
|
+
if (character.sightState === 'off') {
|
|
274
|
+
character.startSight().catch(report)
|
|
275
|
+
} else {
|
|
276
|
+
character.stopSight()
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
character.onSightState((state, error) => {
|
|
280
|
+
see.hidden = state === 'unavailable'
|
|
281
|
+
see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
|
|
282
|
+
if (error) report(error)
|
|
283
|
+
})
|
|
284
|
+
character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
- `startSight({ source, intervalMs, motion, onError })` starts sight in the
|
|
288
|
+
current conversation; call `createConversation()` first. The first call
|
|
289
|
+
asks for the camera permission. `source` is `'camera'` (default) or a
|
|
290
|
+
`MediaStream` that you opened, for example a screen share. `stopSight()`
|
|
291
|
+
closes the camera that the SDK opened. The SDK never stops the tracks of a
|
|
292
|
+
stream that you give.
|
|
293
|
+
- Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
|
|
294
|
+
smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
|
|
295
|
+
it compares a 32×18 grid of the luminance with the grid before. When the
|
|
296
|
+
mean change is more than `motion.threshold` (default 8, on a 0 to 255
|
|
297
|
+
scale), a frame goes, one each second at most. Else a frame goes every
|
|
298
|
+
`intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
|
|
299
|
+
frames only at the interval. One frame at a time waits for the server: a
|
|
300
|
+
frame that comes due during the wait goes at the answer. While the page is
|
|
301
|
+
hidden, the SDK captures nothing.
|
|
302
|
+
- `sightState` is `'unavailable'` (the owner did not turn sight on, or the
|
|
303
|
+
conversation is baked), `'off'`, `'starting'` or `'on'`.
|
|
304
|
+
`onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
|
|
305
|
+
gives each observation of the current conversation:
|
|
306
|
+
`{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
|
|
307
|
+
- Errors are `SightError`s with a `code`. `startSight()` rejects with
|
|
308
|
+
`SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
|
|
309
|
+
`SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
|
|
310
|
+
or a code of the conversation connection.
|
|
311
|
+
- These codes stop sight after it started and go to `onError` and to the
|
|
312
|
+
`error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
|
|
313
|
+
`INSUFFICIENT_POINTS` (the owner has no points),
|
|
314
|
+
`CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
|
|
315
|
+
shared screen stopped). The SDK skips one frame and continues for
|
|
316
|
+
`SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
|
|
317
|
+
`SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
|
|
318
|
+
- `endConversation()`, `dispose()` and `loadAssets()` stop sight.
|
|
319
|
+
- The character owner pays the analysis of each frame at the price of the
|
|
320
|
+
provider that the owner chose. A refused or failed frame costs nothing.
|
|
321
|
+
|
|
256
322
|
## Runtime and lifecycle
|
|
257
323
|
|
|
258
324
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|
package/dist/index.d.ts
CHANGED
|
@@ -3,6 +3,49 @@ import { VRMAnimation } from '@pixiv/three-vrm-animation';
|
|
|
3
3
|
import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
|
|
4
4
|
import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
|
|
5
5
|
|
|
6
|
+
/**
|
|
7
|
+
* Sight (docs/sight.md): a client captures frames, the server sends each
|
|
8
|
+
* frame to the vision provider that the owner of the character chose, and the
|
|
9
|
+
* character reads the text observations in its next replies. Frames are never
|
|
10
|
+
* stored.
|
|
11
|
+
*/
|
|
12
|
+
/** The vision providers. No other provider takes over when one fails. */
|
|
13
|
+
declare const SIGHT_PROVIDERS: readonly ["gemini", "openai"];
|
|
14
|
+
type SightProvider = (typeof SIGHT_PROVIDERS)[number];
|
|
15
|
+
/** The codes of a refused frame. A newer server can send other codes. */
|
|
16
|
+
declare const SIGHT_ERRORS: readonly ["SIGHT_DISABLED", "SIGHT_RATE_LIMITED", "SIGHT_FRAME_TOO_LARGE", "SIGHT_FRAME_INVALID", "SIGHT_PROVIDER_FAILED", "INSUFFICIENT_POINTS", "CONVERSATION_ACCESS_DENIED"];
|
|
17
|
+
type SightErrorCode = (typeof SIGHT_ERRORS)[number];
|
|
18
|
+
type SightPersonPosition = 'left' | 'center' | 'right';
|
|
19
|
+
/** One person or avatar in a frame. */
|
|
20
|
+
type SightPerson = {
|
|
21
|
+
position: SightPersonPosition;
|
|
22
|
+
/** What the person does, in five words or fewer. */
|
|
23
|
+
action: string;
|
|
24
|
+
/** The person faces or addresses the camera. */
|
|
25
|
+
facingCamera: boolean;
|
|
26
|
+
};
|
|
27
|
+
/** What the vision model saw in one frame. */
|
|
28
|
+
type SightObservation = {
|
|
29
|
+
/** The number of people or avatars in the frame. */
|
|
30
|
+
people: number;
|
|
31
|
+
persons: SightPerson[];
|
|
32
|
+
/** Each readable piece of text, verbatim. */
|
|
33
|
+
text: string[];
|
|
34
|
+
};
|
|
35
|
+
/** `sight_observation`: an observation of the conversation, sent to its room. */
|
|
36
|
+
type SightObservationEvent = {
|
|
37
|
+
conversationId: string;
|
|
38
|
+
observationId: string;
|
|
39
|
+
/** The capture time that the client sent, as an ISO 8601 string. */
|
|
40
|
+
capturedAt: string;
|
|
41
|
+
observation: SightObservation;
|
|
42
|
+
/** One sentence about the frame, for the character. */
|
|
43
|
+
summary: string;
|
|
44
|
+
model: string;
|
|
45
|
+
/** The points charged to the owner; null while the charge is pending. */
|
|
46
|
+
costPoints: number | null;
|
|
47
|
+
};
|
|
48
|
+
|
|
6
49
|
type TtsProviderId = string;
|
|
7
50
|
type NullableTtsProviderId = TtsProviderId | null;
|
|
8
51
|
|
|
@@ -18,6 +61,14 @@ type Sqt = {
|
|
|
18
61
|
quaternion?: [number, number, number, number] | null;
|
|
19
62
|
translation?: [number, number, number] | null;
|
|
20
63
|
};
|
|
64
|
+
/**
|
|
65
|
+
* The owner's sight switch and vision provider (docs/sight.md). `enabled` is
|
|
66
|
+
* true only with a provider; an SDK captures frames only then.
|
|
67
|
+
*/
|
|
68
|
+
type AvatarSightSettings = {
|
|
69
|
+
enabled: boolean;
|
|
70
|
+
provider: SightProvider | null;
|
|
71
|
+
};
|
|
21
72
|
interface AvatarPublicInfo {
|
|
22
73
|
id: string;
|
|
23
74
|
name: string | null;
|
|
@@ -48,6 +99,8 @@ interface AvatarPublicInfo {
|
|
|
48
99
|
* `allowCustomReply`. Missing on older servers: off.
|
|
49
100
|
*/
|
|
50
101
|
speechInputEnabled?: boolean;
|
|
102
|
+
/** Sight (Sight tab). Missing on older servers: off. */
|
|
103
|
+
sight?: AvatarSightSettings;
|
|
51
104
|
autoPlay: boolean;
|
|
52
105
|
serverRenderingEnabled: boolean;
|
|
53
106
|
/** The character's body motion is generated live in each turn (missing on older servers). */
|
|
@@ -402,7 +455,7 @@ interface ConversationTurnOptions {
|
|
|
402
455
|
requestId?: string;
|
|
403
456
|
}
|
|
404
457
|
|
|
405
|
-
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled'>>;
|
|
458
|
+
type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled' | 'sight'>>;
|
|
406
459
|
|
|
407
460
|
interface LlmThinkingPayload {
|
|
408
461
|
taskId?: string | null;
|
|
@@ -1600,6 +1653,49 @@ interface ConversationConnectionClosedEvent {
|
|
|
1600
1653
|
conversationId: string | null;
|
|
1601
1654
|
}
|
|
1602
1655
|
|
|
1656
|
+
/**
|
|
1657
|
+
* The code of a `SightError`: a refusal of the server (`SIGHT_ERRORS` of
|
|
1658
|
+
* `@npcify/shared`), a code of the conversation connection
|
|
1659
|
+
* (`CONVERSATION_DISCONNECTED`, `CONVERSATION_TIMEOUT`, ...), or a code of
|
|
1660
|
+
* the SDK. Another string is a code of a newer server.
|
|
1661
|
+
*/
|
|
1662
|
+
type SightFailureCode = SightErrorCode | 'CAMERA_PERMISSION_DENIED' | 'CAMERA_UNAVAILABLE' | 'SIGHT_SOURCE_ENDED' | 'SIGHT_ALREADY_STARTED' | 'CONVERSATION_REQUIRED' | (string & {});
|
|
1663
|
+
declare class SightError extends Error {
|
|
1664
|
+
readonly code: SightFailureCode;
|
|
1665
|
+
constructor(code: SightFailureCode, message?: string);
|
|
1666
|
+
}
|
|
1667
|
+
|
|
1668
|
+
/** `'camera'` opens the camera; a `MediaStream` is a video of the caller. */
|
|
1669
|
+
type SightSourceOption = MediaStream | 'camera';
|
|
1670
|
+
|
|
1671
|
+
/**
|
|
1672
|
+
* `'unavailable'`: the character info did not turn sight on, or the
|
|
1673
|
+
* conversation is baked. `'off'`: sight can start. `'starting'`: the camera
|
|
1674
|
+
* and the connection open. `'on'`: frames go to the server.
|
|
1675
|
+
*/
|
|
1676
|
+
type SightState = 'unavailable' | 'off' | 'starting' | 'on';
|
|
1677
|
+
interface SightOptions {
|
|
1678
|
+
/**
|
|
1679
|
+
* `'camera'` (default) opens the camera, and `stopSight()` closes it. A
|
|
1680
|
+
* `MediaStream` (a camera or a screen share of your page) stays yours:
|
|
1681
|
+
* the SDK never stops its tracks.
|
|
1682
|
+
*/
|
|
1683
|
+
source?: SightSourceOption;
|
|
1684
|
+
/** The longest time between two frames. Default 5000 ms; the minimum is 1000 ms. */
|
|
1685
|
+
intervalMs?: number;
|
|
1686
|
+
/**
|
|
1687
|
+
* A change of the picture sends a frame at once (one each second at most).
|
|
1688
|
+
* `threshold` is the mean change of the 32 × 18 luma grid between two
|
|
1689
|
+
* checks, on a 0 to 255 scale (default 8). `false` sends only at the
|
|
1690
|
+
* interval.
|
|
1691
|
+
*/
|
|
1692
|
+
motion?: {
|
|
1693
|
+
threshold?: number;
|
|
1694
|
+
} | false;
|
|
1695
|
+
/** The error that stopped sight after `startSight()` resolved. */
|
|
1696
|
+
onError?: (error: SightError) => void;
|
|
1697
|
+
}
|
|
1698
|
+
|
|
1603
1699
|
declare class MicrophoneError extends Error {
|
|
1604
1700
|
readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
|
|
1605
1701
|
constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
|
|
@@ -1707,11 +1803,8 @@ declare class Character extends CharacterRuntime {
|
|
|
1707
1803
|
private readonly bakedAsset?;
|
|
1708
1804
|
private animationRefreshId;
|
|
1709
1805
|
private readonly voice;
|
|
1710
|
-
private readonly
|
|
1711
|
-
private
|
|
1712
|
-
/** The character's voice input switch, after the character loads. */
|
|
1713
|
-
private speechInput;
|
|
1714
|
-
private latestOptions;
|
|
1806
|
+
private readonly sight;
|
|
1807
|
+
private readonly inputs;
|
|
1715
1808
|
private listeningStateCallback;
|
|
1716
1809
|
private transcriptCallback;
|
|
1717
1810
|
private connectionClosedCallback;
|
|
@@ -1768,6 +1861,25 @@ declare class Character extends CharacterRuntime {
|
|
|
1768
1861
|
onListeningState(callback: (state: ListeningState) => void): () => void;
|
|
1769
1862
|
/** Partial text while the player speaks (`final: false`), then the final text. */
|
|
1770
1863
|
onTranscript(callback: (event: TranscriptEvent) => void): () => void;
|
|
1864
|
+
/**
|
|
1865
|
+
* `'unavailable'` until the character info turns sight on (its owner's
|
|
1866
|
+
* Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
|
|
1867
|
+
* `'on'`.
|
|
1868
|
+
*/
|
|
1869
|
+
get sightState(): SightState;
|
|
1870
|
+
/**
|
|
1871
|
+
* Lets the character see the player's camera (docs/sight.md). Needs a
|
|
1872
|
+
* conversation; the first call asks for the camera permission. Rejects
|
|
1873
|
+
* with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
|
|
1874
|
+
* `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
|
|
1875
|
+
*/
|
|
1876
|
+
startSight(options?: SightOptions): Promise<void>;
|
|
1877
|
+
/** Stops the frames. Closes the camera that `startSight` opened. */
|
|
1878
|
+
stopSight(): void;
|
|
1879
|
+
/** What the character saw: each observation of the current conversation. */
|
|
1880
|
+
onSight(callback: (event: SightObservationEvent) => void): () => void;
|
|
1881
|
+
/** The sight state. `error` is the `SightError` that stopped sight. */
|
|
1882
|
+
onSightState(callback: (state: SightState, error?: SightError) => void): () => void;
|
|
1771
1883
|
/**
|
|
1772
1884
|
* The server closed the conversation connection: a refused or failed join,
|
|
1773
1885
|
* or an API key that was rotated, edited or deleted. The next `connect`,
|
|
@@ -1789,14 +1901,6 @@ declare class Character extends CharacterRuntime {
|
|
|
1789
1901
|
private resetAssets;
|
|
1790
1902
|
private readonly getAssetInfo;
|
|
1791
1903
|
private handleStagingResult;
|
|
1792
|
-
/** Throws when the input modes do not include `mode`. */
|
|
1793
|
-
private assertInputMode;
|
|
1794
|
-
/**
|
|
1795
|
-
* A typed reply needs the `text` mode; one of the current reply options
|
|
1796
|
-
* needs `options` or `text`. Without a configuration the server's rules
|
|
1797
|
-
* apply alone, as before.
|
|
1798
|
-
*/
|
|
1799
|
-
private assertReplyMode;
|
|
1800
1904
|
private invokeThinkingCallback;
|
|
1801
1905
|
}
|
|
1802
1906
|
declare class Npcify extends Character {
|
|
@@ -1908,4 +2012,4 @@ interface NpcifyView {
|
|
|
1908
2012
|
/** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
|
|
1909
2013
|
declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
|
|
1910
2014
|
|
|
1911
|
-
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
|
2015
|
+
export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SightError, type SightFailureCode, type SightObservation, type SightObservationEvent, type SightOptions, type SightPerson, type SightState, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
|
package/dist/index.js
CHANGED
|
@@ -88,6 +88,20 @@ var AI_ROLES = {
|
|
|
88
88
|
models: ["gemini-3.8-flash-tts", "gemini-3.8-flash-lite-tts"],
|
|
89
89
|
options: []
|
|
90
90
|
},
|
|
91
|
+
// Sight (docs/sight.md): one role for each provider that an owner can
|
|
92
|
+
// choose. Priced by VISION_PRICES of the server.
|
|
93
|
+
vision_gemini: {
|
|
94
|
+
name: "Gemini sight",
|
|
95
|
+
source: "gemini",
|
|
96
|
+
models: ["gemini-3.5-flash-lite"],
|
|
97
|
+
options: []
|
|
98
|
+
},
|
|
99
|
+
vision_openai: {
|
|
100
|
+
name: "OpenAI sight",
|
|
101
|
+
source: "openai",
|
|
102
|
+
models: ["gpt-6-luna"],
|
|
103
|
+
options: []
|
|
104
|
+
},
|
|
91
105
|
// Priced by OPENAI_PRICES.text of the generator.
|
|
92
106
|
generator_agent: {
|
|
93
107
|
name: "Generator agent",
|
|
@@ -242,6 +256,43 @@ var SPEECH_INPUT_PROVIDER_PROFILES = Object.freeze({
|
|
|
242
256
|
}
|
|
243
257
|
});
|
|
244
258
|
|
|
259
|
+
// ../ts-shared/src/sight.ts
|
|
260
|
+
var SIGHT_EVENTS = Object.freeze({
|
|
261
|
+
/** Client to server: one JPEG frame, acknowledged with a `SightFrameAck`. */
|
|
262
|
+
frame: "conversations:sight",
|
|
263
|
+
/** Server to the conversation's room: one `SightObservationEvent`. */
|
|
264
|
+
observation: "sight_observation"
|
|
265
|
+
});
|
|
266
|
+
var SIGHT_LIMITS = Object.freeze({
|
|
267
|
+
/** The largest JPEG frame, in bytes (200 KiB). */
|
|
268
|
+
maxFrameBytes: 200 * 1024,
|
|
269
|
+
/** The longest edge of a frame or a crop, in pixels. */
|
|
270
|
+
maxEdgePx: 1280,
|
|
271
|
+
/** The shortest time between two frames of one conversation. */
|
|
272
|
+
minFrameIntervalMs: 1e3,
|
|
273
|
+
maxCrops: 4,
|
|
274
|
+
/** The largest JPEG crop, in bytes (100 KiB). */
|
|
275
|
+
maxCropBytes: 100 * 1024,
|
|
276
|
+
maxCropLabelLength: 40
|
|
277
|
+
});
|
|
278
|
+
var SIGHT_CAPTURE_SIZE = Object.freeze({
|
|
279
|
+
width: 640,
|
|
280
|
+
height: 360,
|
|
281
|
+
jpegQuality: 0.7
|
|
282
|
+
});
|
|
283
|
+
var SIGHT_PROVIDER_PROFILES = Object.freeze({
|
|
284
|
+
gemini: {
|
|
285
|
+
model: "gemini-3.5-flash-lite",
|
|
286
|
+
frameToObservationMs: 1280,
|
|
287
|
+
usdPerFrame: 29e-5
|
|
288
|
+
},
|
|
289
|
+
openai: {
|
|
290
|
+
model: "gpt-6-luna",
|
|
291
|
+
frameToObservationMs: 2140,
|
|
292
|
+
usdPerFrame: 74e-6
|
|
293
|
+
}
|
|
294
|
+
});
|
|
295
|
+
|
|
245
296
|
// ../ts-shared/src/product-usage.ts
|
|
246
297
|
var productUsageEvents = [
|
|
247
298
|
"page_viewed",
|
|
@@ -10211,6 +10262,522 @@ var CharacterRuntime = class {
|
|
|
10211
10262
|
|
|
10212
10263
|
// ../../clients/web-client/src/conversation-socket.ts
|
|
10213
10264
|
import { io } from "socket.io-client";
|
|
10265
|
+
|
|
10266
|
+
// ../../clients/web-client/src/sight/sight-error.ts
|
|
10267
|
+
var SightError = class extends Error {
|
|
10268
|
+
constructor(code, message) {
|
|
10269
|
+
super(message ?? code);
|
|
10270
|
+
this.code = code;
|
|
10271
|
+
this.name = "NpcifySightError";
|
|
10272
|
+
}
|
|
10273
|
+
};
|
|
10274
|
+
var SIGHT_STOP_CODES = /* @__PURE__ */ new Set([
|
|
10275
|
+
"SIGHT_DISABLED",
|
|
10276
|
+
"INSUFFICIENT_POINTS",
|
|
10277
|
+
"CONVERSATION_ACCESS_DENIED"
|
|
10278
|
+
]);
|
|
10279
|
+
var SIGHT_OFF_MESSAGE = "Sight is off for this character. Its owner turns it on in the Sight tab.";
|
|
10280
|
+
var STOP_MESSAGES = {
|
|
10281
|
+
SIGHT_DISABLED: SIGHT_OFF_MESSAGE,
|
|
10282
|
+
INSUFFICIENT_POINTS: "The owner of the character has no points to pay for sight.",
|
|
10283
|
+
CONVERSATION_ACCESS_DENIED: "This connection has no access to the conversation."
|
|
10284
|
+
};
|
|
10285
|
+
var sightStopError = function(code) {
|
|
10286
|
+
return new SightError(code, STOP_MESSAGES[code] ?? code);
|
|
10287
|
+
};
|
|
10288
|
+
var toSightError = function(error, fallback) {
|
|
10289
|
+
if (error instanceof SightError) {
|
|
10290
|
+
return error;
|
|
10291
|
+
}
|
|
10292
|
+
const code = error?.code;
|
|
10293
|
+
return new SightError(
|
|
10294
|
+
typeof code === "string" && code ? code : fallback,
|
|
10295
|
+
error instanceof Error ? error.message : void 0
|
|
10296
|
+
);
|
|
10297
|
+
};
|
|
10298
|
+
|
|
10299
|
+
// ../../clients/web-client/src/sight/frame-source.ts
|
|
10300
|
+
var LUMA_GRID = Object.freeze({ width: 32, height: 18 });
|
|
10301
|
+
var SMALLER_JPEG_QUALITY = 0.5;
|
|
10302
|
+
var HAVE_CURRENT_DATA = 2;
|
|
10303
|
+
var fitCaptureSize = function(width, height, limit = SIGHT_CAPTURE_SIZE) {
|
|
10304
|
+
const scale = Math.min(1, limit.width / width, limit.height / height);
|
|
10305
|
+
return {
|
|
10306
|
+
width: Math.max(1, Math.round(width * scale)),
|
|
10307
|
+
height: Math.max(1, Math.round(height * scale))
|
|
10308
|
+
};
|
|
10309
|
+
};
|
|
10310
|
+
var lumaOf = function(rgba) {
|
|
10311
|
+
const luma = new Uint8Array(Math.floor(rgba.length / 4));
|
|
10312
|
+
for (let index = 0; index < luma.length; index += 1) {
|
|
10313
|
+
const offset = index * 4;
|
|
10314
|
+
luma[index] = Math.round(
|
|
10315
|
+
0.299 * (rgba[offset] ?? 0) + 0.587 * (rgba[offset + 1] ?? 0) + 0.114 * (rgba[offset + 2] ?? 0)
|
|
10316
|
+
);
|
|
10317
|
+
}
|
|
10318
|
+
return luma;
|
|
10319
|
+
};
|
|
10320
|
+
var meanLumaChange = function(previous, next) {
|
|
10321
|
+
const length = Math.min(previous.length, next.length);
|
|
10322
|
+
if (length === 0) {
|
|
10323
|
+
return 0;
|
|
10324
|
+
}
|
|
10325
|
+
let sum = 0;
|
|
10326
|
+
for (let index = 0; index < length; index += 1) {
|
|
10327
|
+
sum += Math.abs((next[index] ?? 0) - (previous[index] ?? 0));
|
|
10328
|
+
}
|
|
10329
|
+
return sum / length;
|
|
10330
|
+
};
|
|
10331
|
+
var encodeJpeg = function(canvas, quality) {
|
|
10332
|
+
return new Promise((resolve) => {
|
|
10333
|
+
canvas.toBlob(resolve, "image/jpeg", quality);
|
|
10334
|
+
});
|
|
10335
|
+
};
|
|
10336
|
+
var VideoFrameSource = class {
|
|
10337
|
+
constructor(stream, owned) {
|
|
10338
|
+
this.stream = stream;
|
|
10339
|
+
this.owned = owned;
|
|
10340
|
+
const settings = stream.getVideoTracks()[0]?.getSettings?.();
|
|
10341
|
+
this.kind = settings?.displaySurface ? "screen" : "camera";
|
|
10342
|
+
const video = document.createElement("video");
|
|
10343
|
+
video.muted = true;
|
|
10344
|
+
video.playsInline = true;
|
|
10345
|
+
video.setAttribute("playsinline", "");
|
|
10346
|
+
video.srcObject = stream;
|
|
10347
|
+
void video.play().catch(() => void 0);
|
|
10348
|
+
this.video = video;
|
|
10349
|
+
this.frameCanvas = document.createElement("canvas");
|
|
10350
|
+
const grid = document.createElement("canvas");
|
|
10351
|
+
grid.width = LUMA_GRID.width;
|
|
10352
|
+
grid.height = LUMA_GRID.height;
|
|
10353
|
+
this.gridContext = grid.getContext("2d", { willReadFrequently: true });
|
|
10354
|
+
if (this.gridContext) {
|
|
10355
|
+
this.gridContext.imageSmoothingQuality = "high";
|
|
10356
|
+
}
|
|
10357
|
+
}
|
|
10358
|
+
kind;
|
|
10359
|
+
video;
|
|
10360
|
+
frameCanvas;
|
|
10361
|
+
frameContext = null;
|
|
10362
|
+
gridContext;
|
|
10363
|
+
stopped = false;
|
|
10364
|
+
get live() {
|
|
10365
|
+
return !this.stopped && this.stream.getVideoTracks().some((track) => track.readyState === "live");
|
|
10366
|
+
}
|
|
10367
|
+
async capture() {
|
|
10368
|
+
const picture = this.pictureSize();
|
|
10369
|
+
if (!picture) {
|
|
10370
|
+
return null;
|
|
10371
|
+
}
|
|
10372
|
+
const size = fitCaptureSize(picture.width, picture.height);
|
|
10373
|
+
const canvas = this.frameCanvas;
|
|
10374
|
+
if (canvas.width !== size.width || canvas.height !== size.height) {
|
|
10375
|
+
canvas.width = size.width;
|
|
10376
|
+
canvas.height = size.height;
|
|
10377
|
+
}
|
|
10378
|
+
this.frameContext ??= canvas.getContext("2d");
|
|
10379
|
+
const context2 = this.frameContext;
|
|
10380
|
+
if (!context2) {
|
|
10381
|
+
return null;
|
|
10382
|
+
}
|
|
10383
|
+
context2.imageSmoothingQuality = "high";
|
|
10384
|
+
context2.drawImage(this.video, 0, 0, size.width, size.height);
|
|
10385
|
+
let blob = await encodeJpeg(canvas, SIGHT_CAPTURE_SIZE.jpegQuality);
|
|
10386
|
+
if (blob && blob.size > SIGHT_LIMITS.maxFrameBytes) {
|
|
10387
|
+
blob = await encodeJpeg(canvas, SMALLER_JPEG_QUALITY);
|
|
10388
|
+
}
|
|
10389
|
+
if (!blob || blob.size > SIGHT_LIMITS.maxFrameBytes) {
|
|
10390
|
+
return null;
|
|
10391
|
+
}
|
|
10392
|
+
return { jpeg: await blob.arrayBuffer(), ...size };
|
|
10393
|
+
}
|
|
10394
|
+
luminance() {
|
|
10395
|
+
const context2 = this.gridContext;
|
|
10396
|
+
if (!context2 || !this.pictureSize()) {
|
|
10397
|
+
return null;
|
|
10398
|
+
}
|
|
10399
|
+
context2.drawImage(this.video, 0, 0, LUMA_GRID.width, LUMA_GRID.height);
|
|
10400
|
+
return lumaOf(
|
|
10401
|
+
context2.getImageData(0, 0, LUMA_GRID.width, LUMA_GRID.height).data
|
|
10402
|
+
);
|
|
10403
|
+
}
|
|
10404
|
+
stop() {
|
|
10405
|
+
if (this.stopped) {
|
|
10406
|
+
return;
|
|
10407
|
+
}
|
|
10408
|
+
this.stopped = true;
|
|
10409
|
+
this.video.pause();
|
|
10410
|
+
this.video.srcObject = null;
|
|
10411
|
+
if (this.owned) {
|
|
10412
|
+
this.stream.getTracks().forEach((track) => track.stop());
|
|
10413
|
+
}
|
|
10414
|
+
}
|
|
10415
|
+
pictureSize() {
|
|
10416
|
+
const { video } = this;
|
|
10417
|
+
if (this.stopped || video.readyState < HAVE_CURRENT_DATA || !video.videoWidth || !video.videoHeight) {
|
|
10418
|
+
return null;
|
|
10419
|
+
}
|
|
10420
|
+
return { width: video.videoWidth, height: video.videoHeight };
|
|
10421
|
+
}
|
|
10422
|
+
};
|
|
10423
|
+
var openFrameSource = async function(source) {
|
|
10424
|
+
if (source !== "camera") {
|
|
10425
|
+
if (typeof source?.getVideoTracks !== "function") {
|
|
10426
|
+
throw new SightError(
|
|
10427
|
+
"CAMERA_UNAVAILABLE",
|
|
10428
|
+
'The sight source must be "camera" or a MediaStream.'
|
|
10429
|
+
);
|
|
10430
|
+
}
|
|
10431
|
+
if (source.getVideoTracks().length === 0) {
|
|
10432
|
+
throw new SightError(
|
|
10433
|
+
"CAMERA_UNAVAILABLE",
|
|
10434
|
+
"The stream has no video track."
|
|
10435
|
+
);
|
|
10436
|
+
}
|
|
10437
|
+
return new VideoFrameSource(source, false);
|
|
10438
|
+
}
|
|
10439
|
+
const devices = typeof navigator === "undefined" ? void 0 : navigator.mediaDevices;
|
|
10440
|
+
if (!devices?.getUserMedia) {
|
|
10441
|
+
throw new SightError(
|
|
10442
|
+
"CAMERA_UNAVAILABLE",
|
|
10443
|
+
"This browser cannot open the camera."
|
|
10444
|
+
);
|
|
10445
|
+
}
|
|
10446
|
+
let stream;
|
|
10447
|
+
try {
|
|
10448
|
+
stream = await devices.getUserMedia({
|
|
10449
|
+
video: {
|
|
10450
|
+
width: { ideal: SIGHT_CAPTURE_SIZE.width },
|
|
10451
|
+
height: { ideal: SIGHT_CAPTURE_SIZE.height }
|
|
10452
|
+
},
|
|
10453
|
+
audio: false
|
|
10454
|
+
});
|
|
10455
|
+
} catch (error) {
|
|
10456
|
+
const name = error?.name;
|
|
10457
|
+
throw name === "NotAllowedError" || name === "SecurityError" ? new SightError("CAMERA_PERMISSION_DENIED", "Camera access was denied.") : new SightError("CAMERA_UNAVAILABLE", "No camera is available.");
|
|
10458
|
+
}
|
|
10459
|
+
return new VideoFrameSource(stream, true);
|
|
10460
|
+
};
|
|
10461
|
+
|
|
10462
|
+
// ../../clients/web-client/src/sight/sight-input.ts
|
|
10463
|
+
var SIGHT_CHECK_MS = 250;
|
|
10464
|
+
var SIGHT_INTERVAL_MS = 5e3;
|
|
10465
|
+
var SIGHT_MOTION_THRESHOLD = 8;
|
|
10466
|
+
var ACK_TIMEOUT_MS = 2e4;
|
|
10467
|
+
var systemClock = {
|
|
10468
|
+
now: () => Date.now(),
|
|
10469
|
+
setInterval: (callback, ms) => setInterval(callback, ms),
|
|
10470
|
+
clearInterval: (handle) => clearInterval(handle)
|
|
10471
|
+
};
|
|
10472
|
+
var settle = async function(run) {
|
|
10473
|
+
return run();
|
|
10474
|
+
};
|
|
10475
|
+
var documentHidden = function() {
|
|
10476
|
+
return typeof document !== "undefined" && document.hidden === true;
|
|
10477
|
+
};
|
|
10478
|
+
var intervalOf = function(value) {
|
|
10479
|
+
return typeof value === "number" && value > 0 ? Math.max(SIGHT_LIMITS.minFrameIntervalMs, value) : SIGHT_INTERVAL_MS;
|
|
10480
|
+
};
|
|
10481
|
+
var thresholdOf = function(motion) {
|
|
10482
|
+
if (motion === false) {
|
|
10483
|
+
return null;
|
|
10484
|
+
}
|
|
10485
|
+
const threshold = motion?.threshold;
|
|
10486
|
+
return typeof threshold === "number" && Number.isFinite(threshold) ? Math.max(0, threshold) : SIGHT_MOTION_THRESHOLD;
|
|
10487
|
+
};
|
|
10488
|
+
var isSightObservationEvent = function(value) {
|
|
10489
|
+
const event = value;
|
|
10490
|
+
const observation = event?.observation;
|
|
10491
|
+
return typeof event?.conversationId === "string" && typeof event.observationId === "string" && typeof event.summary === "string" && typeof observation === "object" && observation !== null && typeof observation.people === "number" && Array.isArray(observation.persons) && Array.isArray(observation.text);
|
|
10492
|
+
};
|
|
10493
|
+
var SightInput = class {
|
|
10494
|
+
constructor(deps) {
|
|
10495
|
+
this.deps = deps;
|
|
10496
|
+
this.clock = deps.clock ?? systemClock;
|
|
10497
|
+
this.hidden = deps.hidden ?? documentHidden;
|
|
10498
|
+
}
|
|
10499
|
+
clock;
|
|
10500
|
+
hidden;
|
|
10501
|
+
session = null;
|
|
10502
|
+
/** Why sight cannot start; null when it can. */
|
|
10503
|
+
unavailable = "Sight needs the character info: load the character first.";
|
|
10504
|
+
current = "unavailable";
|
|
10505
|
+
stateCallback = null;
|
|
10506
|
+
observationCallback = null;
|
|
10507
|
+
get state() {
|
|
10508
|
+
return this.current;
|
|
10509
|
+
}
|
|
10510
|
+
/**
|
|
10511
|
+
* The character info was read. Sight is available only when the owner
|
|
10512
|
+
* turned it on, and never in a baked conversation. Sight that becomes
|
|
10513
|
+
* unavailable stops.
|
|
10514
|
+
*/
|
|
10515
|
+
configure(settings) {
|
|
10516
|
+
this.unavailable = settings.baked ? "Sight is not available in a baked (offline) conversation." : settings.enabled ? null : SIGHT_OFF_MESSAGE;
|
|
10517
|
+
if (this.unavailable) {
|
|
10518
|
+
this.stop();
|
|
10519
|
+
} else {
|
|
10520
|
+
this.publish();
|
|
10521
|
+
}
|
|
10522
|
+
}
|
|
10523
|
+
/**
|
|
10524
|
+
* Opens the source and joins the conversation. Resolves when frames can
|
|
10525
|
+
* go; the first frame goes when the video has a picture.
|
|
10526
|
+
*/
|
|
10527
|
+
async start(options = {}) {
|
|
10528
|
+
if (this.unavailable) {
|
|
10529
|
+
throw new SightError("SIGHT_DISABLED", this.unavailable);
|
|
10530
|
+
}
|
|
10531
|
+
if (this.session) {
|
|
10532
|
+
throw new SightError(
|
|
10533
|
+
"SIGHT_ALREADY_STARTED",
|
|
10534
|
+
"Sight has already started."
|
|
10535
|
+
);
|
|
10536
|
+
}
|
|
10537
|
+
const conversationId = this.deps.conversationId();
|
|
10538
|
+
if (!conversationId) {
|
|
10539
|
+
throw new SightError(
|
|
10540
|
+
"CONVERSATION_REQUIRED",
|
|
10541
|
+
"Create a conversation before sight starts."
|
|
10542
|
+
);
|
|
10543
|
+
}
|
|
10544
|
+
const session = {
|
|
10545
|
+
intervalMs: intervalOf(options.intervalMs),
|
|
10546
|
+
threshold: thresholdOf(options.motion),
|
|
10547
|
+
onError: options.onError,
|
|
10548
|
+
source: null,
|
|
10549
|
+
timer: null,
|
|
10550
|
+
grid: null,
|
|
10551
|
+
lastSentAt: Number.NEGATIVE_INFINITY,
|
|
10552
|
+
motion: false,
|
|
10553
|
+
inFlight: false,
|
|
10554
|
+
rejoin: false
|
|
10555
|
+
};
|
|
10556
|
+
this.session = session;
|
|
10557
|
+
this.publish();
|
|
10558
|
+
const open = this.deps.openSource ?? openFrameSource;
|
|
10559
|
+
const [joined, opened] = await Promise.allSettled([
|
|
10560
|
+
settle(() => this.deps.socket().ensureConnected(conversationId)),
|
|
10561
|
+
settle(() => open(options.source ?? "camera"))
|
|
10562
|
+
]);
|
|
10563
|
+
const source = opened.status === "fulfilled" ? opened.value : null;
|
|
10564
|
+
if (session !== this.session) {
|
|
10565
|
+
source?.stop();
|
|
10566
|
+
return;
|
|
10567
|
+
}
|
|
10568
|
+
const failure = opened.status === "rejected" ? toSightError(opened.reason, "CAMERA_UNAVAILABLE") : joined.status === "rejected" ? toSightError(joined.reason, "CONVERSATION_DISCONNECTED") : null;
|
|
10569
|
+
if (failure || !source) {
|
|
10570
|
+
source?.stop();
|
|
10571
|
+
this.session = null;
|
|
10572
|
+
this.publish();
|
|
10573
|
+
throw failure ?? new SightError("CAMERA_UNAVAILABLE");
|
|
10574
|
+
}
|
|
10575
|
+
session.source = source;
|
|
10576
|
+
session.timer = this.clock.setInterval(
|
|
10577
|
+
() => this.check(session),
|
|
10578
|
+
SIGHT_CHECK_MS
|
|
10579
|
+
);
|
|
10580
|
+
this.publish();
|
|
10581
|
+
this.check(session);
|
|
10582
|
+
}
|
|
10583
|
+
/** Stops the frames and releases the source. Does nothing when sight is off. */
|
|
10584
|
+
stop() {
|
|
10585
|
+
if (this.session) {
|
|
10586
|
+
this.end(this.session);
|
|
10587
|
+
}
|
|
10588
|
+
this.publish();
|
|
10589
|
+
}
|
|
10590
|
+
/** An observation of the current conversation (the socket checked the ID). */
|
|
10591
|
+
handleObservation(event) {
|
|
10592
|
+
if (this.observationCallback) {
|
|
10593
|
+
invokeSafeCallback(
|
|
10594
|
+
"onSight",
|
|
10595
|
+
this.deps.handleCallbackError,
|
|
10596
|
+
() => this.observationCallback?.(event)
|
|
10597
|
+
);
|
|
10598
|
+
}
|
|
10599
|
+
}
|
|
10600
|
+
onObservation(callback) {
|
|
10601
|
+
return bindCallbackSlot(
|
|
10602
|
+
() => this.observationCallback,
|
|
10603
|
+
(next) => this.observationCallback = next,
|
|
10604
|
+
callback
|
|
10605
|
+
);
|
|
10606
|
+
}
|
|
10607
|
+
onState(callback) {
|
|
10608
|
+
return bindCallbackSlot(
|
|
10609
|
+
() => this.stateCallback,
|
|
10610
|
+
(next) => this.stateCallback = next,
|
|
10611
|
+
callback
|
|
10612
|
+
);
|
|
10613
|
+
}
|
|
10614
|
+
/** One check: the end of the source, a hidden page, motion, then a due frame. */
|
|
10615
|
+
check(session) {
|
|
10616
|
+
const source = session.source;
|
|
10617
|
+
if (session !== this.session || !source) {
|
|
10618
|
+
return;
|
|
10619
|
+
}
|
|
10620
|
+
if (!source.live) {
|
|
10621
|
+
this.fail(
|
|
10622
|
+
session,
|
|
10623
|
+
new SightError("SIGHT_SOURCE_ENDED", "The video of sight ended.")
|
|
10624
|
+
);
|
|
10625
|
+
return;
|
|
10626
|
+
}
|
|
10627
|
+
if (this.hidden()) {
|
|
10628
|
+
session.grid = null;
|
|
10629
|
+
return;
|
|
10630
|
+
}
|
|
10631
|
+
if (session.threshold !== null) {
|
|
10632
|
+
const grid = source.luminance();
|
|
10633
|
+
if (grid && session.grid && meanLumaChange(session.grid, grid) > session.threshold) {
|
|
10634
|
+
session.motion = true;
|
|
10635
|
+
}
|
|
10636
|
+
session.grid = grid;
|
|
10637
|
+
}
|
|
10638
|
+
this.sendWhenDue(session);
|
|
10639
|
+
}
|
|
10640
|
+
/**
|
|
10641
|
+
* A frame is due at the interval, or after motion when one second passed
|
|
10642
|
+
* since the last frame. While a frame waits for its acknowledgement, a due
|
|
10643
|
+
* frame waits too: the acknowledgement sends it.
|
|
10644
|
+
*/
|
|
10645
|
+
sendWhenDue(session) {
|
|
10646
|
+
if (session !== this.session || session.inFlight || this.hidden()) {
|
|
10647
|
+
return;
|
|
10648
|
+
}
|
|
10649
|
+
const elapsed = this.clock.now() - session.lastSentAt;
|
|
10650
|
+
if (elapsed >= session.intervalMs || session.motion && elapsed >= SIGHT_LIMITS.minFrameIntervalMs) {
|
|
10651
|
+
void this.send(session);
|
|
10652
|
+
}
|
|
10653
|
+
}
|
|
10654
|
+
async send(session) {
|
|
10655
|
+
const source = session.source;
|
|
10656
|
+
if (!source) {
|
|
10657
|
+
return;
|
|
10658
|
+
}
|
|
10659
|
+
session.inFlight = true;
|
|
10660
|
+
let sent = false;
|
|
10661
|
+
try {
|
|
10662
|
+
const capturedAt = this.clock.now();
|
|
10663
|
+
let frame;
|
|
10664
|
+
try {
|
|
10665
|
+
frame = await source.capture();
|
|
10666
|
+
} catch (error) {
|
|
10667
|
+
this.fail(
|
|
10668
|
+
session,
|
|
10669
|
+
new SightError(
|
|
10670
|
+
"CAMERA_UNAVAILABLE",
|
|
10671
|
+
error instanceof Error ? error.message : "The video gave no picture."
|
|
10672
|
+
)
|
|
10673
|
+
);
|
|
10674
|
+
return;
|
|
10675
|
+
}
|
|
10676
|
+
if (!frame || session !== this.session) {
|
|
10677
|
+
return;
|
|
10678
|
+
}
|
|
10679
|
+
const conversationId = this.deps.conversationId();
|
|
10680
|
+
if (!conversationId) {
|
|
10681
|
+
this.stop();
|
|
10682
|
+
return;
|
|
10683
|
+
}
|
|
10684
|
+
session.lastSentAt = this.clock.now();
|
|
10685
|
+
session.motion = false;
|
|
10686
|
+
sent = true;
|
|
10687
|
+
const socket = this.deps.socket();
|
|
10688
|
+
if (session.rejoin) {
|
|
10689
|
+
await socket.ensureConnected(conversationId);
|
|
10690
|
+
session.rejoin = false;
|
|
10691
|
+
}
|
|
10692
|
+
const request = {
|
|
10693
|
+
conversationId,
|
|
10694
|
+
capturedAt,
|
|
10695
|
+
width: frame.width,
|
|
10696
|
+
height: frame.height,
|
|
10697
|
+
source: source.kind,
|
|
10698
|
+
jpeg: frame.jpeg
|
|
10699
|
+
};
|
|
10700
|
+
const ack = await socket.request(
|
|
10701
|
+
SIGHT_EVENTS.frame,
|
|
10702
|
+
request,
|
|
10703
|
+
ACK_TIMEOUT_MS
|
|
10704
|
+
);
|
|
10705
|
+
const code = ack && !ack.ok ? ack.error : null;
|
|
10706
|
+
if (code && SIGHT_STOP_CODES.has(code)) {
|
|
10707
|
+
this.fail(session, sightStopError(code));
|
|
10708
|
+
}
|
|
10709
|
+
} catch (error) {
|
|
10710
|
+
this.failed(session, error);
|
|
10711
|
+
} finally {
|
|
10712
|
+
session.inFlight = false;
|
|
10713
|
+
if (sent) {
|
|
10714
|
+
this.sendWhenDue(session);
|
|
10715
|
+
}
|
|
10716
|
+
}
|
|
10717
|
+
}
|
|
10718
|
+
/**
|
|
10719
|
+
* A frame that did not reach the server or got no acknowledgement. A
|
|
10720
|
+
* refused conversation stops sight. A timeout skips the frame; a lost
|
|
10721
|
+
* connection also joins the conversation again before the next frame.
|
|
10722
|
+
*/
|
|
10723
|
+
failed(session, error) {
|
|
10724
|
+
const code = error?.code;
|
|
10725
|
+
if (typeof code === "string" && SIGHT_STOP_CODES.has(code)) {
|
|
10726
|
+
this.fail(session, sightStopError(code));
|
|
10727
|
+
return;
|
|
10728
|
+
}
|
|
10729
|
+
if (code !== "CONVERSATION_TIMEOUT") {
|
|
10730
|
+
session.rejoin = true;
|
|
10731
|
+
}
|
|
10732
|
+
}
|
|
10733
|
+
/** Stops sight because of `error`, and reports it. */
|
|
10734
|
+
fail(session, error) {
|
|
10735
|
+
if (session !== this.session) {
|
|
10736
|
+
return;
|
|
10737
|
+
}
|
|
10738
|
+
this.end(session);
|
|
10739
|
+
if (error.code === "SIGHT_DISABLED") {
|
|
10740
|
+
this.unavailable = SIGHT_OFF_MESSAGE;
|
|
10741
|
+
}
|
|
10742
|
+
this.publish(error);
|
|
10743
|
+
const onError = session.onError;
|
|
10744
|
+
if (onError) {
|
|
10745
|
+
invokeSafeCallback(
|
|
10746
|
+
"startSight onError",
|
|
10747
|
+
this.deps.handleCallbackError,
|
|
10748
|
+
() => onError(error)
|
|
10749
|
+
);
|
|
10750
|
+
}
|
|
10751
|
+
}
|
|
10752
|
+
end(session) {
|
|
10753
|
+
if (this.session !== session) {
|
|
10754
|
+
return;
|
|
10755
|
+
}
|
|
10756
|
+
this.session = null;
|
|
10757
|
+
if (session.timer !== null) {
|
|
10758
|
+
this.clock.clearInterval(session.timer);
|
|
10759
|
+
session.timer = null;
|
|
10760
|
+
}
|
|
10761
|
+
session.source?.stop();
|
|
10762
|
+
}
|
|
10763
|
+
publish(error) {
|
|
10764
|
+
const session = this.session;
|
|
10765
|
+
const next = session ? session.source ? "on" : "starting" : this.unavailable ? "unavailable" : "off";
|
|
10766
|
+
if (next === this.current && !error) {
|
|
10767
|
+
return;
|
|
10768
|
+
}
|
|
10769
|
+
this.current = next;
|
|
10770
|
+
if (this.stateCallback) {
|
|
10771
|
+
invokeSafeCallback(
|
|
10772
|
+
"onSightState",
|
|
10773
|
+
this.deps.handleCallbackError,
|
|
10774
|
+
() => this.stateCallback?.(next, error)
|
|
10775
|
+
);
|
|
10776
|
+
}
|
|
10777
|
+
}
|
|
10778
|
+
};
|
|
10779
|
+
|
|
10780
|
+
// ../../clients/web-client/src/conversation-socket.ts
|
|
10214
10781
|
var ConversationConnectionError = class extends Error {
|
|
10215
10782
|
constructor(code, message, reason = null) {
|
|
10216
10783
|
super(message);
|
|
@@ -10563,6 +11130,11 @@ var ConversationSocket = class {
|
|
|
10563
11130
|
this.options.handleSpeechEvent?.({ type: "turnFailed", event });
|
|
10564
11131
|
}
|
|
10565
11132
|
});
|
|
11133
|
+
socket.on(SIGHT_EVENTS.observation, (payload) => {
|
|
11134
|
+
if (isSightObservationEvent(payload) && payload.conversationId === this.options.getConversationId()) {
|
|
11135
|
+
this.options.handleSightEvent?.(payload);
|
|
11136
|
+
}
|
|
11137
|
+
});
|
|
10566
11138
|
socket.on("conversations:turn", (payload) => {
|
|
10567
11139
|
let candidate;
|
|
10568
11140
|
try {
|
|
@@ -10591,6 +11163,96 @@ var ConversationSocket = class {
|
|
|
10591
11163
|
}
|
|
10592
11164
|
};
|
|
10593
11165
|
|
|
11166
|
+
// ../../clients/web-client/src/input-mode-rules.ts
|
|
11167
|
+
var InputModeRules = class {
|
|
11168
|
+
configured;
|
|
11169
|
+
resolved = null;
|
|
11170
|
+
/** The character's voice input switch, after the character loads. */
|
|
11171
|
+
speechInput = null;
|
|
11172
|
+
latestOptions = [];
|
|
11173
|
+
/**
|
|
11174
|
+
* Throws an `InputModeError` at once for a malformed configuration, and
|
|
11175
|
+
* for a baked asset with a mode other than `options`.
|
|
11176
|
+
*/
|
|
11177
|
+
constructor(inputModes, bakedAsset) {
|
|
11178
|
+
this.configured = parseInputModes(inputModes);
|
|
11179
|
+
if (this.configured && bakedAsset) {
|
|
11180
|
+
resolveInputModes(this.configured, {
|
|
11181
|
+
allowCustomReply: false,
|
|
11182
|
+
baked: true,
|
|
11183
|
+
speechInput: false
|
|
11184
|
+
});
|
|
11185
|
+
}
|
|
11186
|
+
}
|
|
11187
|
+
/**
|
|
11188
|
+
* The character loaded. Throws an `InputModeError` when the configuration
|
|
11189
|
+
* has a mode that the character does not permit.
|
|
11190
|
+
*/
|
|
11191
|
+
load(permissions) {
|
|
11192
|
+
this.speechInput = permissions.speechInput;
|
|
11193
|
+
this.resolved = resolveInputModes(this.configured, permissions);
|
|
11194
|
+
}
|
|
11195
|
+
/**
|
|
11196
|
+
* The input modes in use: the configured modes, or every mode that the
|
|
11197
|
+
* character permits. Before the character loads, the configuration or all
|
|
11198
|
+
* modes.
|
|
11199
|
+
*/
|
|
11200
|
+
get modes() {
|
|
11201
|
+
return this.resolved ?? this.configured ?? permittedInputModes({
|
|
11202
|
+
allowCustomReply: true,
|
|
11203
|
+
baked: false,
|
|
11204
|
+
speechInput: true
|
|
11205
|
+
});
|
|
11206
|
+
}
|
|
11207
|
+
/** The reply options of the latest result, for `assertReply`. */
|
|
11208
|
+
setOptions(options) {
|
|
11209
|
+
this.latestOptions = options;
|
|
11210
|
+
}
|
|
11211
|
+
/** Throws when the input modes do not include `mode`. */
|
|
11212
|
+
assertMode(mode, baked) {
|
|
11213
|
+
if (mode === "voice" && baked) {
|
|
11214
|
+
throw new InputModeError(
|
|
11215
|
+
"Voice input is not available in a baked (offline) conversation."
|
|
11216
|
+
);
|
|
11217
|
+
}
|
|
11218
|
+
if (mode === "voice" && this.speechInput === false) {
|
|
11219
|
+
throw new InputModeError(
|
|
11220
|
+
"Voice input is off for this character. Its owner turns it on in the STT tab."
|
|
11221
|
+
);
|
|
11222
|
+
}
|
|
11223
|
+
const modes = this.modes;
|
|
11224
|
+
if (!modes.includes(mode)) {
|
|
11225
|
+
throw new InputModeError(
|
|
11226
|
+
`The "${mode}" input mode is off: inputModes is ${JSON.stringify(modes)}.`
|
|
11227
|
+
);
|
|
11228
|
+
}
|
|
11229
|
+
}
|
|
11230
|
+
/**
|
|
11231
|
+
* A typed reply needs the `text` mode; one of the current reply options
|
|
11232
|
+
* needs `options` or `text`. Without a configuration the server's rules
|
|
11233
|
+
* apply alone, as before.
|
|
11234
|
+
*/
|
|
11235
|
+
assertReply(message) {
|
|
11236
|
+
if (!this.configured) {
|
|
11237
|
+
return;
|
|
11238
|
+
}
|
|
11239
|
+
const modes = this.modes;
|
|
11240
|
+
if (modes.includes("text")) {
|
|
11241
|
+
return;
|
|
11242
|
+
}
|
|
11243
|
+
const reply = message.trim();
|
|
11244
|
+
const isOption = this.latestOptions.some(
|
|
11245
|
+
(option) => option.label.trim() === reply
|
|
11246
|
+
);
|
|
11247
|
+
if (isOption && modes.includes("options")) {
|
|
11248
|
+
return;
|
|
11249
|
+
}
|
|
11250
|
+
throw new InputModeError(
|
|
11251
|
+
isOption ? `Reply options are off: inputModes is ${JSON.stringify(modes)}.` : `Typed replies are off: inputModes is ${JSON.stringify(modes)}. Send one of the reply options or use startListening().`
|
|
11252
|
+
);
|
|
11253
|
+
}
|
|
11254
|
+
};
|
|
11255
|
+
|
|
10594
11256
|
// ../../clients/web-client/src/voice/push-to-talk.ts
|
|
10595
11257
|
var INTERACTIVE = 'input, textarea, select, button, a[href], summary, [contenteditable]:not([contenteditable="false"]), [role="button"], [role="textbox"], [role="checkbox"], [role="switch"], [role="menuitem"], [role="tab"], [role="slider"], [role="option"]';
|
|
10596
11258
|
var elementOf = function(target) {
|
|
@@ -11550,11 +12212,8 @@ var Character = class extends CharacterRuntime {
|
|
|
11550
12212
|
bakedAsset;
|
|
11551
12213
|
animationRefreshId = 0;
|
|
11552
12214
|
voice;
|
|
11553
|
-
|
|
11554
|
-
|
|
11555
|
-
/** The character's voice input switch, after the character loads. */
|
|
11556
|
-
speechInput = null;
|
|
11557
|
-
latestOptions = [];
|
|
12215
|
+
sight;
|
|
12216
|
+
inputs;
|
|
11558
12217
|
listeningStateCallback = null;
|
|
11559
12218
|
transcriptCallback = null;
|
|
11560
12219
|
connectionClosedCallback = null;
|
|
@@ -11578,14 +12237,10 @@ var Character = class extends CharacterRuntime {
|
|
|
11578
12237
|
this.normalizeIdleOrientation = options.normalizeIdleOrientation ?? false;
|
|
11579
12238
|
this.bakedAsset = options.bakedConversation;
|
|
11580
12239
|
this.initialAssetInfo = options.initialAssetInfo ?? null;
|
|
11581
|
-
this.
|
|
11582
|
-
|
|
11583
|
-
|
|
11584
|
-
|
|
11585
|
-
baked: true,
|
|
11586
|
-
speechInput: false
|
|
11587
|
-
});
|
|
11588
|
-
}
|
|
12240
|
+
this.inputs = new InputModeRules(
|
|
12241
|
+
options.inputModes,
|
|
12242
|
+
Boolean(this.bakedAsset) && !this.authoring
|
|
12243
|
+
);
|
|
11589
12244
|
this.conversationSocket = new ConversationSocket({
|
|
11590
12245
|
baseUrl: this.baseUrl,
|
|
11591
12246
|
apiToken: this.apiToken,
|
|
@@ -11601,17 +12256,23 @@ var Character = class extends CharacterRuntime {
|
|
|
11601
12256
|
},
|
|
11602
12257
|
handleCallbackError: this.handleCallbackError,
|
|
11603
12258
|
handleSpeechEvent: (event) => this.voice.handleSocketEvent(event),
|
|
12259
|
+
handleSightEvent: (event) => this.sight.handleObservation(event),
|
|
11604
12260
|
handleConnectionClosed: (event) => invokeSafeCallback(
|
|
11605
12261
|
"onConnectionClosed",
|
|
11606
12262
|
this.handleCallbackError,
|
|
11607
12263
|
() => this.connectionClosedCallback?.(event)
|
|
11608
12264
|
)
|
|
11609
12265
|
});
|
|
12266
|
+
this.sight = new SightInput({
|
|
12267
|
+
socket: () => this.conversationSocket,
|
|
12268
|
+
conversationId: () => this.conversationId,
|
|
12269
|
+
handleCallbackError: this.handleCallbackError
|
|
12270
|
+
});
|
|
11610
12271
|
this.voice = new VoiceInput({
|
|
11611
12272
|
socket: () => this.conversationSocket,
|
|
11612
12273
|
conversationId: () => this.conversationId,
|
|
11613
12274
|
renderSessionId: () => this.renderSessionId,
|
|
11614
|
-
assertVoiceAllowed: () => this.
|
|
12275
|
+
assertVoiceAllowed: () => this.inputs.assertMode("voice", this.baked !== null),
|
|
11615
12276
|
turnInFlight: () => this.liveConversation.inFlight,
|
|
11616
12277
|
beginTurn: () => {
|
|
11617
12278
|
if (this.conversationId) {
|
|
@@ -11675,12 +12336,12 @@ var Character = class extends CharacterRuntime {
|
|
|
11675
12336
|
const progressReporter = createAssetProgressReporter(callback);
|
|
11676
12337
|
const info = await this.getAssetInfo();
|
|
11677
12338
|
const baked = !this.authoring && Boolean(this.bakedAsset || info.bakedConversationEnabled);
|
|
11678
|
-
this.
|
|
11679
|
-
this.resolvedInputModes = resolveInputModes(this.configuredInputModes, {
|
|
12339
|
+
this.inputs.load({
|
|
11680
12340
|
allowCustomReply: info.allowCustomReply !== false,
|
|
11681
12341
|
baked,
|
|
11682
|
-
speechInput:
|
|
12342
|
+
speechInput: info.speechInputEnabled === true
|
|
11683
12343
|
});
|
|
12344
|
+
this.sight.configure({ enabled: info.sight?.enabled === true, baked });
|
|
11684
12345
|
this.baked = baked ? await loadBakedConversation({
|
|
11685
12346
|
baseUrl: this.baseUrl,
|
|
11686
12347
|
characterId: this.characterId,
|
|
@@ -11750,11 +12411,14 @@ var Character = class extends CharacterRuntime {
|
|
|
11750
12411
|
if (info.bakedConversationEnabled || this.bakedAsset) {
|
|
11751
12412
|
throw new Error("Baked conversation requires local rendering.");
|
|
11752
12413
|
}
|
|
11753
|
-
this.
|
|
11754
|
-
this.resolvedInputModes = resolveInputModes(this.configuredInputModes, {
|
|
12414
|
+
this.inputs.load({
|
|
11755
12415
|
allowCustomReply: info.allowCustomReply !== false,
|
|
11756
12416
|
baked: false,
|
|
11757
|
-
speechInput:
|
|
12417
|
+
speechInput: info.speechInputEnabled === true
|
|
12418
|
+
});
|
|
12419
|
+
this.sight.configure({
|
|
12420
|
+
enabled: info.sight?.enabled === true,
|
|
12421
|
+
baked: false
|
|
11758
12422
|
});
|
|
11759
12423
|
this.animationController = new RemoteAnimationController();
|
|
11760
12424
|
}
|
|
@@ -11818,6 +12482,7 @@ var Character = class extends CharacterRuntime {
|
|
|
11818
12482
|
}
|
|
11819
12483
|
async disconnect() {
|
|
11820
12484
|
this.voice.reset();
|
|
12485
|
+
this.sight.stop();
|
|
11821
12486
|
this.thinkingState = false;
|
|
11822
12487
|
if (this.animationController) {
|
|
11823
12488
|
this.streamPlayback.cleanup();
|
|
@@ -11853,7 +12518,7 @@ var Character = class extends CharacterRuntime {
|
|
|
11853
12518
|
maybeMessageOrOptions,
|
|
11854
12519
|
maybeOptions
|
|
11855
12520
|
);
|
|
11856
|
-
this.
|
|
12521
|
+
this.inputs.assertReply(message);
|
|
11857
12522
|
if (this.voice.turnPending) {
|
|
11858
12523
|
throw new Error("A reply is already being generated. Please wait.");
|
|
11859
12524
|
}
|
|
@@ -11901,11 +12566,7 @@ var Character = class extends CharacterRuntime {
|
|
|
11901
12566
|
* character permits. Before `loadAssets`, the configuration or all modes.
|
|
11902
12567
|
*/
|
|
11903
12568
|
get inputModes() {
|
|
11904
|
-
return this.
|
|
11905
|
-
allowCustomReply: true,
|
|
11906
|
-
baked: false,
|
|
11907
|
-
speechInput: true
|
|
11908
|
-
});
|
|
12569
|
+
return this.inputs.modes;
|
|
11909
12570
|
}
|
|
11910
12571
|
get listeningState() {
|
|
11911
12572
|
return this.voice.listeningState;
|
|
@@ -11949,6 +12610,35 @@ var Character = class extends CharacterRuntime {
|
|
|
11949
12610
|
callback
|
|
11950
12611
|
);
|
|
11951
12612
|
}
|
|
12613
|
+
/**
|
|
12614
|
+
* `'unavailable'` until the character info turns sight on (its owner's
|
|
12615
|
+
* Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
|
|
12616
|
+
* `'on'`.
|
|
12617
|
+
*/
|
|
12618
|
+
get sightState() {
|
|
12619
|
+
return this.sight.state;
|
|
12620
|
+
}
|
|
12621
|
+
/**
|
|
12622
|
+
* Lets the character see the player's camera (docs/sight.md). Needs a
|
|
12623
|
+
* conversation; the first call asks for the camera permission. Rejects
|
|
12624
|
+
* with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
|
|
12625
|
+
* `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
|
|
12626
|
+
*/
|
|
12627
|
+
startSight(options) {
|
|
12628
|
+
return this.sight.start(options);
|
|
12629
|
+
}
|
|
12630
|
+
/** Stops the frames. Closes the camera that `startSight` opened. */
|
|
12631
|
+
stopSight() {
|
|
12632
|
+
this.sight.stop();
|
|
12633
|
+
}
|
|
12634
|
+
/** What the character saw: each observation of the current conversation. */
|
|
12635
|
+
onSight(callback) {
|
|
12636
|
+
return this.sight.onObservation(callback);
|
|
12637
|
+
}
|
|
12638
|
+
/** The sight state. `error` is the `SightError` that stopped sight. */
|
|
12639
|
+
onSightState(callback) {
|
|
12640
|
+
return this.sight.onState(callback);
|
|
12641
|
+
}
|
|
11952
12642
|
/**
|
|
11953
12643
|
* The server closed the conversation connection: a refused or failed join,
|
|
11954
12644
|
* or an API key that was rotated, edited or deleted. The next `connect`,
|
|
@@ -12025,7 +12715,7 @@ var Character = class extends CharacterRuntime {
|
|
|
12025
12715
|
async handleStagingResult(payload, options) {
|
|
12026
12716
|
this.thinkingState = false;
|
|
12027
12717
|
const normalized = normalizeStagingResultPayload(payload);
|
|
12028
|
-
this.
|
|
12718
|
+
this.inputs.setOptions(normalized.options);
|
|
12029
12719
|
if (options.source === "turn") {
|
|
12030
12720
|
this.liveConversation.received(normalized);
|
|
12031
12721
|
}
|
|
@@ -12052,49 +12742,6 @@ var Character = class extends CharacterRuntime {
|
|
|
12052
12742
|
void this.streamPlayback.playPayload(normalized);
|
|
12053
12743
|
}
|
|
12054
12744
|
}
|
|
12055
|
-
/** Throws when the input modes do not include `mode`. */
|
|
12056
|
-
assertInputMode(mode) {
|
|
12057
|
-
if (mode === "voice" && this.baked) {
|
|
12058
|
-
throw new InputModeError(
|
|
12059
|
-
"Voice input is not available in a baked (offline) conversation."
|
|
12060
|
-
);
|
|
12061
|
-
}
|
|
12062
|
-
if (mode === "voice" && this.speechInput === false) {
|
|
12063
|
-
throw new InputModeError(
|
|
12064
|
-
"Voice input is off for this character. Its owner turns it on in the STT tab."
|
|
12065
|
-
);
|
|
12066
|
-
}
|
|
12067
|
-
const modes = this.inputModes;
|
|
12068
|
-
if (!modes.includes(mode)) {
|
|
12069
|
-
throw new InputModeError(
|
|
12070
|
-
`The "${mode}" input mode is off: inputModes is ${JSON.stringify(modes)}.`
|
|
12071
|
-
);
|
|
12072
|
-
}
|
|
12073
|
-
}
|
|
12074
|
-
/**
|
|
12075
|
-
* A typed reply needs the `text` mode; one of the current reply options
|
|
12076
|
-
* needs `options` or `text`. Without a configuration the server's rules
|
|
12077
|
-
* apply alone, as before.
|
|
12078
|
-
*/
|
|
12079
|
-
assertReplyMode(message) {
|
|
12080
|
-
if (!this.configuredInputModes) {
|
|
12081
|
-
return;
|
|
12082
|
-
}
|
|
12083
|
-
const modes = this.inputModes;
|
|
12084
|
-
if (modes.includes("text")) {
|
|
12085
|
-
return;
|
|
12086
|
-
}
|
|
12087
|
-
const reply = message.trim();
|
|
12088
|
-
const isOption = this.latestOptions.some(
|
|
12089
|
-
(option) => option.label.trim() === reply
|
|
12090
|
-
);
|
|
12091
|
-
if (isOption && modes.includes("options")) {
|
|
12092
|
-
return;
|
|
12093
|
-
}
|
|
12094
|
-
throw new InputModeError(
|
|
12095
|
-
isOption ? `Reply options are off: inputModes is ${JSON.stringify(modes)}.` : `Typed replies are off: inputModes is ${JSON.stringify(modes)}. Send one of the reply options or use startListening().`
|
|
12096
|
-
);
|
|
12097
|
-
}
|
|
12098
12745
|
invokeThinkingCallback(payload) {
|
|
12099
12746
|
this.thinkingState = true;
|
|
12100
12747
|
if (this.thinkingCallback) {
|
|
@@ -12704,6 +13351,7 @@ export {
|
|
|
12704
13351
|
PERFORMANCE_MEDIA_TYPE,
|
|
12705
13352
|
PerformanceRecordingSchemaVersion,
|
|
12706
13353
|
RemoteAnimationController,
|
|
13354
|
+
SightError,
|
|
12707
13355
|
SpeechInputError,
|
|
12708
13356
|
analyzePerformanceRecording,
|
|
12709
13357
|
bindPushToTalk,
|
package/examples/index.html
CHANGED
|
@@ -2,5 +2,6 @@
|
|
|
2
2
|
<button id="start" disabled>Start conversation</button>
|
|
3
3
|
<div id="replies"></div>
|
|
4
4
|
<button id="talk" type="button" hidden disabled>Hold to talk</button>
|
|
5
|
+
<button id="see" type="button" hidden disabled>Let the character see you</button>
|
|
5
6
|
<p id="status" role="status"></p>
|
|
6
7
|
<script type="module" src="/main.js"></script>
|
package/examples/main.js
CHANGED
|
@@ -10,6 +10,7 @@ const config = {
|
|
|
10
10
|
const start = document.querySelector('#start')
|
|
11
11
|
const replies = document.querySelector('#replies')
|
|
12
12
|
const talk = document.querySelector('#talk')
|
|
13
|
+
const see = document.querySelector('#see')
|
|
13
14
|
const status = document.querySelector('#status')
|
|
14
15
|
const report = (error) => { status.textContent = error.message }
|
|
15
16
|
const view = await mountNpcify({
|
|
@@ -56,12 +57,33 @@ character.onListeningState((state) => {
|
|
|
56
57
|
{ listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
|
|
57
58
|
})
|
|
58
59
|
character.onTranscript(({ text }) => { status.textContent = text })
|
|
60
|
+
// The character sees the camera when its owner turned sight on (the Sight
|
|
61
|
+
// tab). The first click asks for the camera permission.
|
|
62
|
+
const showSight = (state) => {
|
|
63
|
+
see.hidden = state === 'unavailable'
|
|
64
|
+
see.textContent =
|
|
65
|
+
{ starting: 'Opening the camera…', on: 'Stop the camera' }[state] ?? 'Let the character see you'
|
|
66
|
+
}
|
|
67
|
+
showSight(character.sightState)
|
|
68
|
+
character.onSightState((state, error) => {
|
|
69
|
+
showSight(state)
|
|
70
|
+
if (error) report(error)
|
|
71
|
+
})
|
|
72
|
+
character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
|
|
73
|
+
see.onclick = () => {
|
|
74
|
+
if (character.sightState === 'off') {
|
|
75
|
+
character.startSight().catch(report)
|
|
76
|
+
} else {
|
|
77
|
+
character.stopSight()
|
|
78
|
+
}
|
|
79
|
+
}
|
|
59
80
|
start.disabled = false
|
|
60
81
|
start.onclick = async () => {
|
|
61
82
|
start.disabled = true
|
|
62
83
|
try {
|
|
63
84
|
await character.unlockAudio()
|
|
64
85
|
await character.createConversation()
|
|
86
|
+
see.disabled = false
|
|
65
87
|
} catch (error) {
|
|
66
88
|
report(error)
|
|
67
89
|
start.disabled = false
|