@npcify/web 0.4.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -130,6 +130,8 @@ animate()
130
130
  - `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
131
131
  - `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
132
132
  - `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
133
+ - **Sight** ([details](#sight)):
134
+ - `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
133
135
 
134
136
  ## Mode 2: Managed Canvas (`mountNpcify`)
135
137
 
@@ -253,6 +255,70 @@ character.onListeningState((state) => {
253
255
  - The character owner pays the recognition at the price of the provider that
254
256
  the owner chose, as for the turn's language model.
255
257
 
258
+ ## Sight
259
+
260
+ The character can see the player's camera when the character owner turned
261
+ sight on (the Sight tab of the character in the dashboard, where the owner
262
+ also chooses the vision provider). The SDK captures small camera frames and
263
+ sends them on the conversation connection. The server sends each frame to the
264
+ vision provider and keeps only a short text observation; it does not keep the
265
+ frames. The character reads the observations of the last minute in its next
266
+ reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
267
+
268
+ ```js
269
+ const see = document.querySelector('#see')
270
+ const report = (error) => { status.textContent = error.message }
271
+ see.hidden = character.sightState === 'unavailable'
272
+ see.onclick = () => {
273
+ if (character.sightState === 'off') {
274
+ character.startSight().catch(report)
275
+ } else {
276
+ character.stopSight()
277
+ }
278
+ }
279
+ character.onSightState((state, error) => {
280
+ see.hidden = state === 'unavailable'
281
+ see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
282
+ if (error) report(error)
283
+ })
284
+ character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
285
+ ```
286
+
287
+ - `startSight({ source, intervalMs, motion, onError })` starts sight in the
288
+ current conversation; call `createConversation()` first. The first call
289
+ asks for the camera permission. `source` is `'camera'` (default) or a
290
+ `MediaStream` that you opened, for example a screen share. `stopSight()`
291
+ closes the camera that the SDK opened. The SDK never stops the tracks of a
292
+ stream that you give.
293
+ - Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
294
+ smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
295
+ it compares a 32×18 grid of the luminance with the grid before. When the
296
+ mean change is more than `motion.threshold` (default 8, on a 0 to 255
297
+ scale), a frame goes, one each second at most. Else a frame goes every
298
+ `intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
299
+ frames only at the interval. One frame at a time waits for the server: a
300
+ frame that comes due during the wait goes at the answer. While the page is
301
+ hidden, the SDK captures nothing.
302
+ - `sightState` is `'unavailable'` (the owner did not turn sight on, or the
303
+ conversation is baked), `'off'`, `'starting'` or `'on'`.
304
+ `onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
305
+ gives each observation of the current conversation:
306
+ `{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
307
+ - Errors are `SightError`s with a `code`. `startSight()` rejects with
308
+ `SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
309
+ `SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
310
+ or a code of the conversation connection.
311
+ - These codes stop sight after it started and go to `onError` and to the
312
+ `error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
313
+ `INSUFFICIENT_POINTS` (the owner has no points),
314
+ `CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
315
+ shared screen stopped). The SDK skips one frame and continues for
316
+ `SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
317
+ `SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
318
+ - `endConversation()`, `dispose()` and `loadAssets()` stop sight.
319
+ - The character owner pays the analysis of each frame at the price of the
320
+ provider that the owner chose. A refused or failed frame costs nothing.
321
+
256
322
  ## Runtime and lifecycle
257
323
 
258
324
  - `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
package/dist/index.d.ts CHANGED
@@ -3,6 +3,49 @@ import { VRMAnimation } from '@pixiv/three-vrm-animation';
3
3
  import { AnimationMixer, AnimationAction, Object3D, Vector3, Group, SkinnedMesh, Mesh, Material, Skeleton, Euler, Scene, PerspectiveCamera, WebGLRenderer } from 'three';
4
4
  import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls.js';
5
5
 
6
+ /**
7
+ * Sight (docs/sight.md): a client captures frames, the server sends each
8
+ * frame to the vision provider that the owner of the character chose, and the
9
+ * character reads the text observations in its next replies. Frames are never
10
+ * stored.
11
+ */
12
+ /** The vision providers. No other provider takes over when one fails. */
13
+ declare const SIGHT_PROVIDERS: readonly ["gemini", "openai"];
14
+ type SightProvider = (typeof SIGHT_PROVIDERS)[number];
15
+ /** The codes of a refused frame. A newer server can send other codes. */
16
+ declare const SIGHT_ERRORS: readonly ["SIGHT_DISABLED", "SIGHT_RATE_LIMITED", "SIGHT_FRAME_TOO_LARGE", "SIGHT_FRAME_INVALID", "SIGHT_PROVIDER_FAILED", "INSUFFICIENT_POINTS", "CONVERSATION_ACCESS_DENIED"];
17
+ type SightErrorCode = (typeof SIGHT_ERRORS)[number];
18
+ type SightPersonPosition = 'left' | 'center' | 'right';
19
+ /** One person or avatar in a frame. */
20
+ type SightPerson = {
21
+ position: SightPersonPosition;
22
+ /** What the person does, in five words or fewer. */
23
+ action: string;
24
+ /** The person faces or addresses the camera. */
25
+ facingCamera: boolean;
26
+ };
27
+ /** What the vision model saw in one frame. */
28
+ type SightObservation = {
29
+ /** The number of people or avatars in the frame. */
30
+ people: number;
31
+ persons: SightPerson[];
32
+ /** Each readable piece of text, verbatim. */
33
+ text: string[];
34
+ };
35
+ /** `sight_observation`: an observation of the conversation, sent to its room. */
36
+ type SightObservationEvent = {
37
+ conversationId: string;
38
+ observationId: string;
39
+ /** The capture time that the client sent, as an ISO 8601 string. */
40
+ capturedAt: string;
41
+ observation: SightObservation;
42
+ /** One sentence about the frame, for the character. */
43
+ summary: string;
44
+ model: string;
45
+ /** The points charged to the owner; null while the charge is pending. */
46
+ costPoints: number | null;
47
+ };
48
+
6
49
  type TtsProviderId = string;
7
50
  type NullableTtsProviderId = TtsProviderId | null;
8
51
 
@@ -18,6 +61,14 @@ type Sqt = {
18
61
  quaternion?: [number, number, number, number] | null;
19
62
  translation?: [number, number, number] | null;
20
63
  };
64
+ /**
65
+ * The owner's sight switch and vision provider (docs/sight.md). `enabled` is
66
+ * true only with a provider; an SDK captures frames only then.
67
+ */
68
+ type AvatarSightSettings = {
69
+ enabled: boolean;
70
+ provider: SightProvider | null;
71
+ };
21
72
  interface AvatarPublicInfo {
22
73
  id: string;
23
74
  name: string | null;
@@ -48,6 +99,8 @@ interface AvatarPublicInfo {
48
99
  * `allowCustomReply`. Missing on older servers: off.
49
100
  */
50
101
  speechInputEnabled?: boolean;
102
+ /** Sight (Sight tab). Missing on older servers: off. */
103
+ sight?: AvatarSightSettings;
51
104
  autoPlay: boolean;
52
105
  serverRenderingEnabled: boolean;
53
106
  /** The character's body motion is generated live in each turn (missing on older servers). */
@@ -402,7 +455,7 @@ interface ConversationTurnOptions {
402
455
  requestId?: string;
403
456
  }
404
457
 
405
- type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled'>>;
458
+ type NpcifyAssetInfo = Pick<AvatarPublicInfo, 'animations' | 'idleAnimationId' | 'bakedConversationEnabled' | 'bakedConversationRevision'> & Partial<Pick<AvatarPublicInfo, 'allowCustomReply' | 'speechInputEnabled' | 'sight'>>;
406
459
 
407
460
  interface LlmThinkingPayload {
408
461
  taskId?: string | null;
@@ -1600,6 +1653,49 @@ interface ConversationConnectionClosedEvent {
1600
1653
  conversationId: string | null;
1601
1654
  }
1602
1655
 
1656
+ /**
1657
+ * The code of a `SightError`: a refusal of the server (`SIGHT_ERRORS` of
1658
+ * `@npcify/shared`), a code of the conversation connection
1659
+ * (`CONVERSATION_DISCONNECTED`, `CONVERSATION_TIMEOUT`, ...), or a code of
1660
+ * the SDK. Another string is a code of a newer server.
1661
+ */
1662
+ type SightFailureCode = SightErrorCode | 'CAMERA_PERMISSION_DENIED' | 'CAMERA_UNAVAILABLE' | 'SIGHT_SOURCE_ENDED' | 'SIGHT_ALREADY_STARTED' | 'CONVERSATION_REQUIRED' | (string & {});
1663
+ declare class SightError extends Error {
1664
+ readonly code: SightFailureCode;
1665
+ constructor(code: SightFailureCode, message?: string);
1666
+ }
1667
+
1668
+ /** `'camera'` opens the camera; a `MediaStream` is a video of the caller. */
1669
+ type SightSourceOption = MediaStream | 'camera';
1670
+
1671
+ /**
1672
+ * `'unavailable'`: the character info did not turn sight on, or the
1673
+ * conversation is baked. `'off'`: sight can start. `'starting'`: the camera
1674
+ * and the connection open. `'on'`: frames go to the server.
1675
+ */
1676
+ type SightState = 'unavailable' | 'off' | 'starting' | 'on';
1677
+ interface SightOptions {
1678
+ /**
1679
+ * `'camera'` (default) opens the camera, and `stopSight()` closes it. A
1680
+ * `MediaStream` (a camera or a screen share of your page) stays yours:
1681
+ * the SDK never stops its tracks.
1682
+ */
1683
+ source?: SightSourceOption;
1684
+ /** The longest time between two frames. Default 5000 ms; the minimum is 1000 ms. */
1685
+ intervalMs?: number;
1686
+ /**
1687
+ * A change of the picture sends a frame at once (one each second at most).
1688
+ * `threshold` is the mean change of the 32 × 18 luma grid between two
1689
+ * checks, on a 0 to 255 scale (default 8). `false` sends only at the
1690
+ * interval.
1691
+ */
1692
+ motion?: {
1693
+ threshold?: number;
1694
+ } | false;
1695
+ /** The error that stopped sight after `startSight()` resolved. */
1696
+ onError?: (error: SightError) => void;
1697
+ }
1698
+
1603
1699
  declare class MicrophoneError extends Error {
1604
1700
  readonly code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE';
1605
1701
  constructor(code: 'MICROPHONE_PERMISSION_DENIED' | 'MICROPHONE_UNAVAILABLE', message: string);
@@ -1707,11 +1803,8 @@ declare class Character extends CharacterRuntime {
1707
1803
  private readonly bakedAsset?;
1708
1804
  private animationRefreshId;
1709
1805
  private readonly voice;
1710
- private readonly configuredInputModes;
1711
- private resolvedInputModes;
1712
- /** The character's voice input switch, after the character loads. */
1713
- private speechInput;
1714
- private latestOptions;
1806
+ private readonly sight;
1807
+ private readonly inputs;
1715
1808
  private listeningStateCallback;
1716
1809
  private transcriptCallback;
1717
1810
  private connectionClosedCallback;
@@ -1768,6 +1861,25 @@ declare class Character extends CharacterRuntime {
1768
1861
  onListeningState(callback: (state: ListeningState) => void): () => void;
1769
1862
  /** Partial text while the player speaks (`final: false`), then the final text. */
1770
1863
  onTranscript(callback: (event: TranscriptEvent) => void): () => void;
1864
+ /**
1865
+ * `'unavailable'` until the character info turns sight on (its owner's
1866
+ * Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
1867
+ * `'on'`.
1868
+ */
1869
+ get sightState(): SightState;
1870
+ /**
1871
+ * Lets the character see the player's camera (docs/sight.md). Needs a
1872
+ * conversation; the first call asks for the camera permission. Rejects
1873
+ * with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
1874
+ * `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
1875
+ */
1876
+ startSight(options?: SightOptions): Promise<void>;
1877
+ /** Stops the frames. Closes the camera that `startSight` opened. */
1878
+ stopSight(): void;
1879
+ /** What the character saw: each observation of the current conversation. */
1880
+ onSight(callback: (event: SightObservationEvent) => void): () => void;
1881
+ /** The sight state. `error` is the `SightError` that stopped sight. */
1882
+ onSightState(callback: (state: SightState, error?: SightError) => void): () => void;
1771
1883
  /**
1772
1884
  * The server closed the conversation connection: a refused or failed join,
1773
1885
  * or an API key that was rotated, edited or deleted. The next `connect`,
@@ -1789,14 +1901,6 @@ declare class Character extends CharacterRuntime {
1789
1901
  private resetAssets;
1790
1902
  private readonly getAssetInfo;
1791
1903
  private handleStagingResult;
1792
- /** Throws when the input modes do not include `mode`. */
1793
- private assertInputMode;
1794
- /**
1795
- * A typed reply needs the `text` mode; one of the current reply options
1796
- * needs `options` or `text`. Without a configuration the server's rules
1797
- * apply alone, as before.
1798
- */
1799
- private assertReplyMode;
1800
1904
  private invokeThinkingCallback;
1801
1905
  }
1802
1906
  declare class Npcify extends Character {
@@ -1908,4 +2012,4 @@ interface NpcifyView {
1908
2012
  /** Mount into an existing canvas. Its CSS must give it a nonzero width and height. */
1909
2013
  declare const mountNpcify: (options: MountNpcifyOptions) => Promise<NpcifyView>;
1910
2014
 
1911
- export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
2015
+ export { AnimationController, type AnimationControllerLike, type AnimationControllerOptions, type AssetLoadProgress, type AudioCapture, type AudioPerformanceFrameSnapshot, AudioPlayer, BakedConversation, type BakedConversationAsset, type CaptureFrame, Character, CharacterAnimation, type CharacterAnimationPlayOptions, CharacterAudio, type CharacterDisposeOptions, CharacterExpressions, type CharacterLookAtTarget, type CharacterOptions, type CharacterPlaybackCursor, CharacterRuntime, type CharacterRuntimeInitOptions, CharacterSceneRig, type CharacterShadowOptions, type CharacterSpeechState, type CharacterTransformOptions, type ConversationConnectionClosedEvent, ConversationConnectionError, type ConversationConnectionErrorCode, type ConversationJoinFailure, type ConversationTurnOptions, type FacialChannelName, type FacialCueOptions, type FacialMouthMode, type FacialPerformanceFrameSnapshot, FacialPerformanceLayer, type FacialPerformanceLayerId, FacialPerformanceMixer, type FacialPerformanceModelSnapshot, INPUT_MODES, type InputMode, InputModeError, type ListeningOptions, type ListeningState, type LiveMotionOptions, LiveMotionPlayer, type LiveMotionRootMode, type LlmThinkingPayload, type LoadCharacterOptions, MicrophoneError, type MountNpcifyOptions, Npcify, type NpcifyAssetInfo, Character as NpcifyCharacter, type NpcifyOptions, type NpcifyView, PERFORMANCE_FILE_EXTENSION, PERFORMANCE_MEDIA_TYPE, type PerformanceAnalysis, type PerformanceAnalysisOptions, type PerformanceEvent, type PerformanceFrame, type PerformanceIssue, type PerformanceKind, type PerformanceRecording, PerformanceRecordingSchemaVersion, type PerformanceRecordingStartOptions, type PerformanceTimelineSnapshot, type PlayStagingOptions, type PushToTalkOptions, type PushToTalkTarget, RemoteAnimationController, type RepeatedVisemeTransitionAnalysis, SightError, type SightFailureCode, type SightObservation, type SightObservationEvent, type SightOptions, type SightPerson, type SightState, SpeechInputError, type SpeechInputResult, type StagingOptionsReadyPayload, type StagingPlaybackStart, type StagingSpeechPlaybackEvent, type TranscriptEvent, type Vector3Like, type Vector3Tuple, type VisemePerformanceFrameSnapshot, analyzePerformanceRecording, bindPushToTalk, createCharacter, decodePerformance, encodePerformance, loadBakedConversation, loadCharacter, mountNpcify };
package/dist/index.js CHANGED
@@ -88,6 +88,20 @@ var AI_ROLES = {
88
88
  models: ["gemini-3.8-flash-tts", "gemini-3.8-flash-lite-tts"],
89
89
  options: []
90
90
  },
91
+ // Sight (docs/sight.md): one role for each provider that an owner can
92
+ // choose. Priced by VISION_PRICES of the server.
93
+ vision_gemini: {
94
+ name: "Gemini sight",
95
+ source: "gemini",
96
+ models: ["gemini-3.5-flash-lite"],
97
+ options: []
98
+ },
99
+ vision_openai: {
100
+ name: "OpenAI sight",
101
+ source: "openai",
102
+ models: ["gpt-6-luna"],
103
+ options: []
104
+ },
91
105
  // Priced by OPENAI_PRICES.text of the generator.
92
106
  generator_agent: {
93
107
  name: "Generator agent",
@@ -242,6 +256,43 @@ var SPEECH_INPUT_PROVIDER_PROFILES = Object.freeze({
242
256
  }
243
257
  });
244
258
 
259
+ // ../ts-shared/src/sight.ts
260
+ var SIGHT_EVENTS = Object.freeze({
261
+ /** Client to server: one JPEG frame, acknowledged with a `SightFrameAck`. */
262
+ frame: "conversations:sight",
263
+ /** Server to the conversation's room: one `SightObservationEvent`. */
264
+ observation: "sight_observation"
265
+ });
266
+ var SIGHT_LIMITS = Object.freeze({
267
+ /** The largest JPEG frame, in bytes (200 KiB). */
268
+ maxFrameBytes: 200 * 1024,
269
+ /** The longest edge of a frame or a crop, in pixels. */
270
+ maxEdgePx: 1280,
271
+ /** The shortest time between two frames of one conversation. */
272
+ minFrameIntervalMs: 1e3,
273
+ maxCrops: 4,
274
+ /** The largest JPEG crop, in bytes (100 KiB). */
275
+ maxCropBytes: 100 * 1024,
276
+ maxCropLabelLength: 40
277
+ });
278
+ var SIGHT_CAPTURE_SIZE = Object.freeze({
279
+ width: 640,
280
+ height: 360,
281
+ jpegQuality: 0.7
282
+ });
283
+ var SIGHT_PROVIDER_PROFILES = Object.freeze({
284
+ gemini: {
285
+ model: "gemini-3.5-flash-lite",
286
+ frameToObservationMs: 1280,
287
+ usdPerFrame: 29e-5
288
+ },
289
+ openai: {
290
+ model: "gpt-6-luna",
291
+ frameToObservationMs: 2140,
292
+ usdPerFrame: 74e-6
293
+ }
294
+ });
295
+
245
296
  // ../ts-shared/src/product-usage.ts
246
297
  var productUsageEvents = [
247
298
  "page_viewed",
@@ -10211,6 +10262,522 @@ var CharacterRuntime = class {
10211
10262
 
10212
10263
  // ../../clients/web-client/src/conversation-socket.ts
10213
10264
  import { io } from "socket.io-client";
10265
+
10266
+ // ../../clients/web-client/src/sight/sight-error.ts
10267
+ var SightError = class extends Error {
10268
+ constructor(code, message) {
10269
+ super(message ?? code);
10270
+ this.code = code;
10271
+ this.name = "NpcifySightError";
10272
+ }
10273
+ };
10274
+ var SIGHT_STOP_CODES = /* @__PURE__ */ new Set([
10275
+ "SIGHT_DISABLED",
10276
+ "INSUFFICIENT_POINTS",
10277
+ "CONVERSATION_ACCESS_DENIED"
10278
+ ]);
10279
+ var SIGHT_OFF_MESSAGE = "Sight is off for this character. Its owner turns it on in the Sight tab.";
10280
+ var STOP_MESSAGES = {
10281
+ SIGHT_DISABLED: SIGHT_OFF_MESSAGE,
10282
+ INSUFFICIENT_POINTS: "The owner of the character has no points to pay for sight.",
10283
+ CONVERSATION_ACCESS_DENIED: "This connection has no access to the conversation."
10284
+ };
10285
+ var sightStopError = function(code) {
10286
+ return new SightError(code, STOP_MESSAGES[code] ?? code);
10287
+ };
10288
+ var toSightError = function(error, fallback) {
10289
+ if (error instanceof SightError) {
10290
+ return error;
10291
+ }
10292
+ const code = error?.code;
10293
+ return new SightError(
10294
+ typeof code === "string" && code ? code : fallback,
10295
+ error instanceof Error ? error.message : void 0
10296
+ );
10297
+ };
10298
+
10299
+ // ../../clients/web-client/src/sight/frame-source.ts
10300
+ var LUMA_GRID = Object.freeze({ width: 32, height: 18 });
10301
+ var SMALLER_JPEG_QUALITY = 0.5;
10302
+ var HAVE_CURRENT_DATA = 2;
10303
+ var fitCaptureSize = function(width, height, limit = SIGHT_CAPTURE_SIZE) {
10304
+ const scale = Math.min(1, limit.width / width, limit.height / height);
10305
+ return {
10306
+ width: Math.max(1, Math.round(width * scale)),
10307
+ height: Math.max(1, Math.round(height * scale))
10308
+ };
10309
+ };
10310
+ var lumaOf = function(rgba) {
10311
+ const luma = new Uint8Array(Math.floor(rgba.length / 4));
10312
+ for (let index = 0; index < luma.length; index += 1) {
10313
+ const offset = index * 4;
10314
+ luma[index] = Math.round(
10315
+ 0.299 * (rgba[offset] ?? 0) + 0.587 * (rgba[offset + 1] ?? 0) + 0.114 * (rgba[offset + 2] ?? 0)
10316
+ );
10317
+ }
10318
+ return luma;
10319
+ };
10320
+ var meanLumaChange = function(previous, next) {
10321
+ const length = Math.min(previous.length, next.length);
10322
+ if (length === 0) {
10323
+ return 0;
10324
+ }
10325
+ let sum = 0;
10326
+ for (let index = 0; index < length; index += 1) {
10327
+ sum += Math.abs((next[index] ?? 0) - (previous[index] ?? 0));
10328
+ }
10329
+ return sum / length;
10330
+ };
10331
+ var encodeJpeg = function(canvas, quality) {
10332
+ return new Promise((resolve) => {
10333
+ canvas.toBlob(resolve, "image/jpeg", quality);
10334
+ });
10335
+ };
10336
+ var VideoFrameSource = class {
10337
+ constructor(stream, owned) {
10338
+ this.stream = stream;
10339
+ this.owned = owned;
10340
+ const settings = stream.getVideoTracks()[0]?.getSettings?.();
10341
+ this.kind = settings?.displaySurface ? "screen" : "camera";
10342
+ const video = document.createElement("video");
10343
+ video.muted = true;
10344
+ video.playsInline = true;
10345
+ video.setAttribute("playsinline", "");
10346
+ video.srcObject = stream;
10347
+ void video.play().catch(() => void 0);
10348
+ this.video = video;
10349
+ this.frameCanvas = document.createElement("canvas");
10350
+ const grid = document.createElement("canvas");
10351
+ grid.width = LUMA_GRID.width;
10352
+ grid.height = LUMA_GRID.height;
10353
+ this.gridContext = grid.getContext("2d", { willReadFrequently: true });
10354
+ if (this.gridContext) {
10355
+ this.gridContext.imageSmoothingQuality = "high";
10356
+ }
10357
+ }
10358
+ kind;
10359
+ video;
10360
+ frameCanvas;
10361
+ frameContext = null;
10362
+ gridContext;
10363
+ stopped = false;
10364
+ get live() {
10365
+ return !this.stopped && this.stream.getVideoTracks().some((track) => track.readyState === "live");
10366
+ }
10367
+ async capture() {
10368
+ const picture = this.pictureSize();
10369
+ if (!picture) {
10370
+ return null;
10371
+ }
10372
+ const size = fitCaptureSize(picture.width, picture.height);
10373
+ const canvas = this.frameCanvas;
10374
+ if (canvas.width !== size.width || canvas.height !== size.height) {
10375
+ canvas.width = size.width;
10376
+ canvas.height = size.height;
10377
+ }
10378
+ this.frameContext ??= canvas.getContext("2d");
10379
+ const context2 = this.frameContext;
10380
+ if (!context2) {
10381
+ return null;
10382
+ }
10383
+ context2.imageSmoothingQuality = "high";
10384
+ context2.drawImage(this.video, 0, 0, size.width, size.height);
10385
+ let blob = await encodeJpeg(canvas, SIGHT_CAPTURE_SIZE.jpegQuality);
10386
+ if (blob && blob.size > SIGHT_LIMITS.maxFrameBytes) {
10387
+ blob = await encodeJpeg(canvas, SMALLER_JPEG_QUALITY);
10388
+ }
10389
+ if (!blob || blob.size > SIGHT_LIMITS.maxFrameBytes) {
10390
+ return null;
10391
+ }
10392
+ return { jpeg: await blob.arrayBuffer(), ...size };
10393
+ }
10394
+ luminance() {
10395
+ const context2 = this.gridContext;
10396
+ if (!context2 || !this.pictureSize()) {
10397
+ return null;
10398
+ }
10399
+ context2.drawImage(this.video, 0, 0, LUMA_GRID.width, LUMA_GRID.height);
10400
+ return lumaOf(
10401
+ context2.getImageData(0, 0, LUMA_GRID.width, LUMA_GRID.height).data
10402
+ );
10403
+ }
10404
+ stop() {
10405
+ if (this.stopped) {
10406
+ return;
10407
+ }
10408
+ this.stopped = true;
10409
+ this.video.pause();
10410
+ this.video.srcObject = null;
10411
+ if (this.owned) {
10412
+ this.stream.getTracks().forEach((track) => track.stop());
10413
+ }
10414
+ }
10415
+ pictureSize() {
10416
+ const { video } = this;
10417
+ if (this.stopped || video.readyState < HAVE_CURRENT_DATA || !video.videoWidth || !video.videoHeight) {
10418
+ return null;
10419
+ }
10420
+ return { width: video.videoWidth, height: video.videoHeight };
10421
+ }
10422
+ };
10423
+ var openFrameSource = async function(source) {
10424
+ if (source !== "camera") {
10425
+ if (typeof source?.getVideoTracks !== "function") {
10426
+ throw new SightError(
10427
+ "CAMERA_UNAVAILABLE",
10428
+ 'The sight source must be "camera" or a MediaStream.'
10429
+ );
10430
+ }
10431
+ if (source.getVideoTracks().length === 0) {
10432
+ throw new SightError(
10433
+ "CAMERA_UNAVAILABLE",
10434
+ "The stream has no video track."
10435
+ );
10436
+ }
10437
+ return new VideoFrameSource(source, false);
10438
+ }
10439
+ const devices = typeof navigator === "undefined" ? void 0 : navigator.mediaDevices;
10440
+ if (!devices?.getUserMedia) {
10441
+ throw new SightError(
10442
+ "CAMERA_UNAVAILABLE",
10443
+ "This browser cannot open the camera."
10444
+ );
10445
+ }
10446
+ let stream;
10447
+ try {
10448
+ stream = await devices.getUserMedia({
10449
+ video: {
10450
+ width: { ideal: SIGHT_CAPTURE_SIZE.width },
10451
+ height: { ideal: SIGHT_CAPTURE_SIZE.height }
10452
+ },
10453
+ audio: false
10454
+ });
10455
+ } catch (error) {
10456
+ const name = error?.name;
10457
+ throw name === "NotAllowedError" || name === "SecurityError" ? new SightError("CAMERA_PERMISSION_DENIED", "Camera access was denied.") : new SightError("CAMERA_UNAVAILABLE", "No camera is available.");
10458
+ }
10459
+ return new VideoFrameSource(stream, true);
10460
+ };
10461
+
10462
+ // ../../clients/web-client/src/sight/sight-input.ts
10463
+ var SIGHT_CHECK_MS = 250;
10464
+ var SIGHT_INTERVAL_MS = 5e3;
10465
+ var SIGHT_MOTION_THRESHOLD = 8;
10466
+ var ACK_TIMEOUT_MS = 2e4;
10467
+ var systemClock = {
10468
+ now: () => Date.now(),
10469
+ setInterval: (callback, ms) => setInterval(callback, ms),
10470
+ clearInterval: (handle) => clearInterval(handle)
10471
+ };
10472
+ var settle = async function(run) {
10473
+ return run();
10474
+ };
10475
+ var documentHidden = function() {
10476
+ return typeof document !== "undefined" && document.hidden === true;
10477
+ };
10478
+ var intervalOf = function(value) {
10479
+ return typeof value === "number" && value > 0 ? Math.max(SIGHT_LIMITS.minFrameIntervalMs, value) : SIGHT_INTERVAL_MS;
10480
+ };
10481
+ var thresholdOf = function(motion) {
10482
+ if (motion === false) {
10483
+ return null;
10484
+ }
10485
+ const threshold = motion?.threshold;
10486
+ return typeof threshold === "number" && Number.isFinite(threshold) ? Math.max(0, threshold) : SIGHT_MOTION_THRESHOLD;
10487
+ };
10488
+ var isSightObservationEvent = function(value) {
10489
+ const event = value;
10490
+ const observation = event?.observation;
10491
+ return typeof event?.conversationId === "string" && typeof event.observationId === "string" && typeof event.summary === "string" && typeof observation === "object" && observation !== null && typeof observation.people === "number" && Array.isArray(observation.persons) && Array.isArray(observation.text);
10492
+ };
10493
+ var SightInput = class {
10494
+ constructor(deps) {
10495
+ this.deps = deps;
10496
+ this.clock = deps.clock ?? systemClock;
10497
+ this.hidden = deps.hidden ?? documentHidden;
10498
+ }
10499
+ clock;
10500
+ hidden;
10501
+ session = null;
10502
+ /** Why sight cannot start; null when it can. */
10503
+ unavailable = "Sight needs the character info: load the character first.";
10504
+ current = "unavailable";
10505
+ stateCallback = null;
10506
+ observationCallback = null;
10507
+ get state() {
10508
+ return this.current;
10509
+ }
10510
+ /**
10511
+ * The character info was read. Sight is available only when the owner
10512
+ * turned it on, and never in a baked conversation. Sight that becomes
10513
+ * unavailable stops.
10514
+ */
10515
+ configure(settings) {
10516
+ this.unavailable = settings.baked ? "Sight is not available in a baked (offline) conversation." : settings.enabled ? null : SIGHT_OFF_MESSAGE;
10517
+ if (this.unavailable) {
10518
+ this.stop();
10519
+ } else {
10520
+ this.publish();
10521
+ }
10522
+ }
10523
+ /**
10524
+ * Opens the source and joins the conversation. Resolves when frames can
10525
+ * go; the first frame goes when the video has a picture.
10526
+ */
10527
+ async start(options = {}) {
10528
+ if (this.unavailable) {
10529
+ throw new SightError("SIGHT_DISABLED", this.unavailable);
10530
+ }
10531
+ if (this.session) {
10532
+ throw new SightError(
10533
+ "SIGHT_ALREADY_STARTED",
10534
+ "Sight has already started."
10535
+ );
10536
+ }
10537
+ const conversationId = this.deps.conversationId();
10538
+ if (!conversationId) {
10539
+ throw new SightError(
10540
+ "CONVERSATION_REQUIRED",
10541
+ "Create a conversation before sight starts."
10542
+ );
10543
+ }
10544
+ const session = {
10545
+ intervalMs: intervalOf(options.intervalMs),
10546
+ threshold: thresholdOf(options.motion),
10547
+ onError: options.onError,
10548
+ source: null,
10549
+ timer: null,
10550
+ grid: null,
10551
+ lastSentAt: Number.NEGATIVE_INFINITY,
10552
+ motion: false,
10553
+ inFlight: false,
10554
+ rejoin: false
10555
+ };
10556
+ this.session = session;
10557
+ this.publish();
10558
+ const open = this.deps.openSource ?? openFrameSource;
10559
+ const [joined, opened] = await Promise.allSettled([
10560
+ settle(() => this.deps.socket().ensureConnected(conversationId)),
10561
+ settle(() => open(options.source ?? "camera"))
10562
+ ]);
10563
+ const source = opened.status === "fulfilled" ? opened.value : null;
10564
+ if (session !== this.session) {
10565
+ source?.stop();
10566
+ return;
10567
+ }
10568
+ const failure = opened.status === "rejected" ? toSightError(opened.reason, "CAMERA_UNAVAILABLE") : joined.status === "rejected" ? toSightError(joined.reason, "CONVERSATION_DISCONNECTED") : null;
10569
+ if (failure || !source) {
10570
+ source?.stop();
10571
+ this.session = null;
10572
+ this.publish();
10573
+ throw failure ?? new SightError("CAMERA_UNAVAILABLE");
10574
+ }
10575
+ session.source = source;
10576
+ session.timer = this.clock.setInterval(
10577
+ () => this.check(session),
10578
+ SIGHT_CHECK_MS
10579
+ );
10580
+ this.publish();
10581
+ this.check(session);
10582
+ }
10583
+ /** Stops the frames and releases the source. Does nothing when sight is off. */
10584
+ stop() {
10585
+ if (this.session) {
10586
+ this.end(this.session);
10587
+ }
10588
+ this.publish();
10589
+ }
10590
+ /** An observation of the current conversation (the socket checked the ID). */
10591
+ handleObservation(event) {
10592
+ if (this.observationCallback) {
10593
+ invokeSafeCallback(
10594
+ "onSight",
10595
+ this.deps.handleCallbackError,
10596
+ () => this.observationCallback?.(event)
10597
+ );
10598
+ }
10599
+ }
10600
+ onObservation(callback) {
10601
+ return bindCallbackSlot(
10602
+ () => this.observationCallback,
10603
+ (next) => this.observationCallback = next,
10604
+ callback
10605
+ );
10606
+ }
10607
+ onState(callback) {
10608
+ return bindCallbackSlot(
10609
+ () => this.stateCallback,
10610
+ (next) => this.stateCallback = next,
10611
+ callback
10612
+ );
10613
+ }
10614
+ /** One check: the end of the source, a hidden page, motion, then a due frame. */
10615
+ check(session) {
10616
+ const source = session.source;
10617
+ if (session !== this.session || !source) {
10618
+ return;
10619
+ }
10620
+ if (!source.live) {
10621
+ this.fail(
10622
+ session,
10623
+ new SightError("SIGHT_SOURCE_ENDED", "The video of sight ended.")
10624
+ );
10625
+ return;
10626
+ }
10627
+ if (this.hidden()) {
10628
+ session.grid = null;
10629
+ return;
10630
+ }
10631
+ if (session.threshold !== null) {
10632
+ const grid = source.luminance();
10633
+ if (grid && session.grid && meanLumaChange(session.grid, grid) > session.threshold) {
10634
+ session.motion = true;
10635
+ }
10636
+ session.grid = grid;
10637
+ }
10638
+ this.sendWhenDue(session);
10639
+ }
10640
+ /**
10641
+ * A frame is due at the interval, or after motion when one second passed
10642
+ * since the last frame. While a frame waits for its acknowledgement, a due
10643
+ * frame waits too: the acknowledgement sends it.
10644
+ */
10645
+ sendWhenDue(session) {
10646
+ if (session !== this.session || session.inFlight || this.hidden()) {
10647
+ return;
10648
+ }
10649
+ const elapsed = this.clock.now() - session.lastSentAt;
10650
+ if (elapsed >= session.intervalMs || session.motion && elapsed >= SIGHT_LIMITS.minFrameIntervalMs) {
10651
+ void this.send(session);
10652
+ }
10653
+ }
10654
+ async send(session) {
10655
+ const source = session.source;
10656
+ if (!source) {
10657
+ return;
10658
+ }
10659
+ session.inFlight = true;
10660
+ let sent = false;
10661
+ try {
10662
+ const capturedAt = this.clock.now();
10663
+ let frame;
10664
+ try {
10665
+ frame = await source.capture();
10666
+ } catch (error) {
10667
+ this.fail(
10668
+ session,
10669
+ new SightError(
10670
+ "CAMERA_UNAVAILABLE",
10671
+ error instanceof Error ? error.message : "The video gave no picture."
10672
+ )
10673
+ );
10674
+ return;
10675
+ }
10676
+ if (!frame || session !== this.session) {
10677
+ return;
10678
+ }
10679
+ const conversationId = this.deps.conversationId();
10680
+ if (!conversationId) {
10681
+ this.stop();
10682
+ return;
10683
+ }
10684
+ session.lastSentAt = this.clock.now();
10685
+ session.motion = false;
10686
+ sent = true;
10687
+ const socket = this.deps.socket();
10688
+ if (session.rejoin) {
10689
+ await socket.ensureConnected(conversationId);
10690
+ session.rejoin = false;
10691
+ }
10692
+ const request = {
10693
+ conversationId,
10694
+ capturedAt,
10695
+ width: frame.width,
10696
+ height: frame.height,
10697
+ source: source.kind,
10698
+ jpeg: frame.jpeg
10699
+ };
10700
+ const ack = await socket.request(
10701
+ SIGHT_EVENTS.frame,
10702
+ request,
10703
+ ACK_TIMEOUT_MS
10704
+ );
10705
+ const code = ack && !ack.ok ? ack.error : null;
10706
+ if (code && SIGHT_STOP_CODES.has(code)) {
10707
+ this.fail(session, sightStopError(code));
10708
+ }
10709
+ } catch (error) {
10710
+ this.failed(session, error);
10711
+ } finally {
10712
+ session.inFlight = false;
10713
+ if (sent) {
10714
+ this.sendWhenDue(session);
10715
+ }
10716
+ }
10717
+ }
10718
+ /**
10719
+ * A frame that did not reach the server or got no acknowledgement. A
10720
+ * refused conversation stops sight. A timeout skips the frame; a lost
10721
+ * connection also joins the conversation again before the next frame.
10722
+ */
10723
+ failed(session, error) {
10724
+ const code = error?.code;
10725
+ if (typeof code === "string" && SIGHT_STOP_CODES.has(code)) {
10726
+ this.fail(session, sightStopError(code));
10727
+ return;
10728
+ }
10729
+ if (code !== "CONVERSATION_TIMEOUT") {
10730
+ session.rejoin = true;
10731
+ }
10732
+ }
10733
+ /** Stops sight because of `error`, and reports it. */
10734
+ fail(session, error) {
10735
+ if (session !== this.session) {
10736
+ return;
10737
+ }
10738
+ this.end(session);
10739
+ if (error.code === "SIGHT_DISABLED") {
10740
+ this.unavailable = SIGHT_OFF_MESSAGE;
10741
+ }
10742
+ this.publish(error);
10743
+ const onError = session.onError;
10744
+ if (onError) {
10745
+ invokeSafeCallback(
10746
+ "startSight onError",
10747
+ this.deps.handleCallbackError,
10748
+ () => onError(error)
10749
+ );
10750
+ }
10751
+ }
10752
+ end(session) {
10753
+ if (this.session !== session) {
10754
+ return;
10755
+ }
10756
+ this.session = null;
10757
+ if (session.timer !== null) {
10758
+ this.clock.clearInterval(session.timer);
10759
+ session.timer = null;
10760
+ }
10761
+ session.source?.stop();
10762
+ }
10763
+ publish(error) {
10764
+ const session = this.session;
10765
+ const next = session ? session.source ? "on" : "starting" : this.unavailable ? "unavailable" : "off";
10766
+ if (next === this.current && !error) {
10767
+ return;
10768
+ }
10769
+ this.current = next;
10770
+ if (this.stateCallback) {
10771
+ invokeSafeCallback(
10772
+ "onSightState",
10773
+ this.deps.handleCallbackError,
10774
+ () => this.stateCallback?.(next, error)
10775
+ );
10776
+ }
10777
+ }
10778
+ };
10779
+
10780
+ // ../../clients/web-client/src/conversation-socket.ts
10214
10781
  var ConversationConnectionError = class extends Error {
10215
10782
  constructor(code, message, reason = null) {
10216
10783
  super(message);
@@ -10563,6 +11130,11 @@ var ConversationSocket = class {
10563
11130
  this.options.handleSpeechEvent?.({ type: "turnFailed", event });
10564
11131
  }
10565
11132
  });
11133
+ socket.on(SIGHT_EVENTS.observation, (payload) => {
11134
+ if (isSightObservationEvent(payload) && payload.conversationId === this.options.getConversationId()) {
11135
+ this.options.handleSightEvent?.(payload);
11136
+ }
11137
+ });
10566
11138
  socket.on("conversations:turn", (payload) => {
10567
11139
  let candidate;
10568
11140
  try {
@@ -10591,6 +11163,96 @@ var ConversationSocket = class {
10591
11163
  }
10592
11164
  };
10593
11165
 
11166
+ // ../../clients/web-client/src/input-mode-rules.ts
11167
+ var InputModeRules = class {
11168
+ configured;
11169
+ resolved = null;
11170
+ /** The character's voice input switch, after the character loads. */
11171
+ speechInput = null;
11172
+ latestOptions = [];
11173
+ /**
11174
+ * Throws an `InputModeError` at once for a malformed configuration, and
11175
+ * for a baked asset with a mode other than `options`.
11176
+ */
11177
+ constructor(inputModes, bakedAsset) {
11178
+ this.configured = parseInputModes(inputModes);
11179
+ if (this.configured && bakedAsset) {
11180
+ resolveInputModes(this.configured, {
11181
+ allowCustomReply: false,
11182
+ baked: true,
11183
+ speechInput: false
11184
+ });
11185
+ }
11186
+ }
11187
+ /**
11188
+ * The character loaded. Throws an `InputModeError` when the configuration
11189
+ * has a mode that the character does not permit.
11190
+ */
11191
+ load(permissions) {
11192
+ this.speechInput = permissions.speechInput;
11193
+ this.resolved = resolveInputModes(this.configured, permissions);
11194
+ }
11195
+ /**
11196
+ * The input modes in use: the configured modes, or every mode that the
11197
+ * character permits. Before the character loads, the configuration or all
11198
+ * modes.
11199
+ */
11200
+ get modes() {
11201
+ return this.resolved ?? this.configured ?? permittedInputModes({
11202
+ allowCustomReply: true,
11203
+ baked: false,
11204
+ speechInput: true
11205
+ });
11206
+ }
11207
+ /** The reply options of the latest result, for `assertReply`. */
11208
+ setOptions(options) {
11209
+ this.latestOptions = options;
11210
+ }
11211
+ /** Throws when the input modes do not include `mode`. */
11212
+ assertMode(mode, baked) {
11213
+ if (mode === "voice" && baked) {
11214
+ throw new InputModeError(
11215
+ "Voice input is not available in a baked (offline) conversation."
11216
+ );
11217
+ }
11218
+ if (mode === "voice" && this.speechInput === false) {
11219
+ throw new InputModeError(
11220
+ "Voice input is off for this character. Its owner turns it on in the STT tab."
11221
+ );
11222
+ }
11223
+ const modes = this.modes;
11224
+ if (!modes.includes(mode)) {
11225
+ throw new InputModeError(
11226
+ `The "${mode}" input mode is off: inputModes is ${JSON.stringify(modes)}.`
11227
+ );
11228
+ }
11229
+ }
11230
+ /**
11231
+ * A typed reply needs the `text` mode; one of the current reply options
11232
+ * needs `options` or `text`. Without a configuration the server's rules
11233
+ * apply alone, as before.
11234
+ */
11235
+ assertReply(message) {
11236
+ if (!this.configured) {
11237
+ return;
11238
+ }
11239
+ const modes = this.modes;
11240
+ if (modes.includes("text")) {
11241
+ return;
11242
+ }
11243
+ const reply = message.trim();
11244
+ const isOption = this.latestOptions.some(
11245
+ (option) => option.label.trim() === reply
11246
+ );
11247
+ if (isOption && modes.includes("options")) {
11248
+ return;
11249
+ }
11250
+ throw new InputModeError(
11251
+ isOption ? `Reply options are off: inputModes is ${JSON.stringify(modes)}.` : `Typed replies are off: inputModes is ${JSON.stringify(modes)}. Send one of the reply options or use startListening().`
11252
+ );
11253
+ }
11254
+ };
11255
+
10594
11256
  // ../../clients/web-client/src/voice/push-to-talk.ts
10595
11257
  var INTERACTIVE = 'input, textarea, select, button, a[href], summary, [contenteditable]:not([contenteditable="false"]), [role="button"], [role="textbox"], [role="checkbox"], [role="switch"], [role="menuitem"], [role="tab"], [role="slider"], [role="option"]';
10596
11258
  var elementOf = function(target) {
@@ -11550,11 +12212,8 @@ var Character = class extends CharacterRuntime {
11550
12212
  bakedAsset;
11551
12213
  animationRefreshId = 0;
11552
12214
  voice;
11553
- configuredInputModes;
11554
- resolvedInputModes = null;
11555
- /** The character's voice input switch, after the character loads. */
11556
- speechInput = null;
11557
- latestOptions = [];
12215
+ sight;
12216
+ inputs;
11558
12217
  listeningStateCallback = null;
11559
12218
  transcriptCallback = null;
11560
12219
  connectionClosedCallback = null;
@@ -11578,14 +12237,10 @@ var Character = class extends CharacterRuntime {
11578
12237
  this.normalizeIdleOrientation = options.normalizeIdleOrientation ?? false;
11579
12238
  this.bakedAsset = options.bakedConversation;
11580
12239
  this.initialAssetInfo = options.initialAssetInfo ?? null;
11581
- this.configuredInputModes = parseInputModes(options.inputModes);
11582
- if (this.configuredInputModes && this.bakedAsset && !this.authoring) {
11583
- resolveInputModes(this.configuredInputModes, {
11584
- allowCustomReply: false,
11585
- baked: true,
11586
- speechInput: false
11587
- });
11588
- }
12240
+ this.inputs = new InputModeRules(
12241
+ options.inputModes,
12242
+ Boolean(this.bakedAsset) && !this.authoring
12243
+ );
11589
12244
  this.conversationSocket = new ConversationSocket({
11590
12245
  baseUrl: this.baseUrl,
11591
12246
  apiToken: this.apiToken,
@@ -11601,17 +12256,23 @@ var Character = class extends CharacterRuntime {
11601
12256
  },
11602
12257
  handleCallbackError: this.handleCallbackError,
11603
12258
  handleSpeechEvent: (event) => this.voice.handleSocketEvent(event),
12259
+ handleSightEvent: (event) => this.sight.handleObservation(event),
11604
12260
  handleConnectionClosed: (event) => invokeSafeCallback(
11605
12261
  "onConnectionClosed",
11606
12262
  this.handleCallbackError,
11607
12263
  () => this.connectionClosedCallback?.(event)
11608
12264
  )
11609
12265
  });
12266
+ this.sight = new SightInput({
12267
+ socket: () => this.conversationSocket,
12268
+ conversationId: () => this.conversationId,
12269
+ handleCallbackError: this.handleCallbackError
12270
+ });
11610
12271
  this.voice = new VoiceInput({
11611
12272
  socket: () => this.conversationSocket,
11612
12273
  conversationId: () => this.conversationId,
11613
12274
  renderSessionId: () => this.renderSessionId,
11614
- assertVoiceAllowed: () => this.assertInputMode("voice"),
12275
+ assertVoiceAllowed: () => this.inputs.assertMode("voice", this.baked !== null),
11615
12276
  turnInFlight: () => this.liveConversation.inFlight,
11616
12277
  beginTurn: () => {
11617
12278
  if (this.conversationId) {
@@ -11675,12 +12336,12 @@ var Character = class extends CharacterRuntime {
11675
12336
  const progressReporter = createAssetProgressReporter(callback);
11676
12337
  const info = await this.getAssetInfo();
11677
12338
  const baked = !this.authoring && Boolean(this.bakedAsset || info.bakedConversationEnabled);
11678
- this.speechInput = info.speechInputEnabled === true;
11679
- this.resolvedInputModes = resolveInputModes(this.configuredInputModes, {
12339
+ this.inputs.load({
11680
12340
  allowCustomReply: info.allowCustomReply !== false,
11681
12341
  baked,
11682
- speechInput: this.speechInput
12342
+ speechInput: info.speechInputEnabled === true
11683
12343
  });
12344
+ this.sight.configure({ enabled: info.sight?.enabled === true, baked });
11684
12345
  this.baked = baked ? await loadBakedConversation({
11685
12346
  baseUrl: this.baseUrl,
11686
12347
  characterId: this.characterId,
@@ -11750,11 +12411,14 @@ var Character = class extends CharacterRuntime {
11750
12411
  if (info.bakedConversationEnabled || this.bakedAsset) {
11751
12412
  throw new Error("Baked conversation requires local rendering.");
11752
12413
  }
11753
- this.speechInput = info.speechInputEnabled === true;
11754
- this.resolvedInputModes = resolveInputModes(this.configuredInputModes, {
12414
+ this.inputs.load({
11755
12415
  allowCustomReply: info.allowCustomReply !== false,
11756
12416
  baked: false,
11757
- speechInput: this.speechInput
12417
+ speechInput: info.speechInputEnabled === true
12418
+ });
12419
+ this.sight.configure({
12420
+ enabled: info.sight?.enabled === true,
12421
+ baked: false
11758
12422
  });
11759
12423
  this.animationController = new RemoteAnimationController();
11760
12424
  }
@@ -11818,6 +12482,7 @@ var Character = class extends CharacterRuntime {
11818
12482
  }
11819
12483
  async disconnect() {
11820
12484
  this.voice.reset();
12485
+ this.sight.stop();
11821
12486
  this.thinkingState = false;
11822
12487
  if (this.animationController) {
11823
12488
  this.streamPlayback.cleanup();
@@ -11853,7 +12518,7 @@ var Character = class extends CharacterRuntime {
11853
12518
  maybeMessageOrOptions,
11854
12519
  maybeOptions
11855
12520
  );
11856
- this.assertReplyMode(message);
12521
+ this.inputs.assertReply(message);
11857
12522
  if (this.voice.turnPending) {
11858
12523
  throw new Error("A reply is already being generated. Please wait.");
11859
12524
  }
@@ -11901,11 +12566,7 @@ var Character = class extends CharacterRuntime {
11901
12566
  * character permits. Before `loadAssets`, the configuration or all modes.
11902
12567
  */
11903
12568
  get inputModes() {
11904
- return this.resolvedInputModes ?? this.configuredInputModes ?? permittedInputModes({
11905
- allowCustomReply: true,
11906
- baked: false,
11907
- speechInput: true
11908
- });
12569
+ return this.inputs.modes;
11909
12570
  }
11910
12571
  get listeningState() {
11911
12572
  return this.voice.listeningState;
@@ -11949,6 +12610,35 @@ var Character = class extends CharacterRuntime {
11949
12610
  callback
11950
12611
  );
11951
12612
  }
12613
+ /**
12614
+ * `'unavailable'` until the character info turns sight on (its owner's
12615
+ * Sight tab), and in a baked conversation; then `'off'`, `'starting'` or
12616
+ * `'on'`.
12617
+ */
12618
+ get sightState() {
12619
+ return this.sight.state;
12620
+ }
12621
+ /**
12622
+ * Lets the character see the player's camera (docs/sight.md). Needs a
12623
+ * conversation; the first call asks for the camera permission. Rejects
12624
+ * with a `SightError`: `SIGHT_DISABLED` when sight is unavailable,
12625
+ * `CONVERSATION_REQUIRED`, `CAMERA_PERMISSION_DENIED`, ...
12626
+ */
12627
+ startSight(options) {
12628
+ return this.sight.start(options);
12629
+ }
12630
+ /** Stops the frames. Closes the camera that `startSight` opened. */
12631
+ stopSight() {
12632
+ this.sight.stop();
12633
+ }
12634
+ /** What the character saw: each observation of the current conversation. */
12635
+ onSight(callback) {
12636
+ return this.sight.onObservation(callback);
12637
+ }
12638
+ /** The sight state. `error` is the `SightError` that stopped sight. */
12639
+ onSightState(callback) {
12640
+ return this.sight.onState(callback);
12641
+ }
11952
12642
  /**
11953
12643
  * The server closed the conversation connection: a refused or failed join,
11954
12644
  * or an API key that was rotated, edited or deleted. The next `connect`,
@@ -12025,7 +12715,7 @@ var Character = class extends CharacterRuntime {
12025
12715
  async handleStagingResult(payload, options) {
12026
12716
  this.thinkingState = false;
12027
12717
  const normalized = normalizeStagingResultPayload(payload);
12028
- this.latestOptions = normalized.options;
12718
+ this.inputs.setOptions(normalized.options);
12029
12719
  if (options.source === "turn") {
12030
12720
  this.liveConversation.received(normalized);
12031
12721
  }
@@ -12052,49 +12742,6 @@ var Character = class extends CharacterRuntime {
12052
12742
  void this.streamPlayback.playPayload(normalized);
12053
12743
  }
12054
12744
  }
12055
- /** Throws when the input modes do not include `mode`. */
12056
- assertInputMode(mode) {
12057
- if (mode === "voice" && this.baked) {
12058
- throw new InputModeError(
12059
- "Voice input is not available in a baked (offline) conversation."
12060
- );
12061
- }
12062
- if (mode === "voice" && this.speechInput === false) {
12063
- throw new InputModeError(
12064
- "Voice input is off for this character. Its owner turns it on in the STT tab."
12065
- );
12066
- }
12067
- const modes = this.inputModes;
12068
- if (!modes.includes(mode)) {
12069
- throw new InputModeError(
12070
- `The "${mode}" input mode is off: inputModes is ${JSON.stringify(modes)}.`
12071
- );
12072
- }
12073
- }
12074
- /**
12075
- * A typed reply needs the `text` mode; one of the current reply options
12076
- * needs `options` or `text`. Without a configuration the server's rules
12077
- * apply alone, as before.
12078
- */
12079
- assertReplyMode(message) {
12080
- if (!this.configuredInputModes) {
12081
- return;
12082
- }
12083
- const modes = this.inputModes;
12084
- if (modes.includes("text")) {
12085
- return;
12086
- }
12087
- const reply = message.trim();
12088
- const isOption = this.latestOptions.some(
12089
- (option) => option.label.trim() === reply
12090
- );
12091
- if (isOption && modes.includes("options")) {
12092
- return;
12093
- }
12094
- throw new InputModeError(
12095
- isOption ? `Reply options are off: inputModes is ${JSON.stringify(modes)}.` : `Typed replies are off: inputModes is ${JSON.stringify(modes)}. Send one of the reply options or use startListening().`
12096
- );
12097
- }
12098
12745
  invokeThinkingCallback(payload) {
12099
12746
  this.thinkingState = true;
12100
12747
  if (this.thinkingCallback) {
@@ -12704,6 +13351,7 @@ export {
12704
13351
  PERFORMANCE_MEDIA_TYPE,
12705
13352
  PerformanceRecordingSchemaVersion,
12706
13353
  RemoteAnimationController,
13354
+ SightError,
12707
13355
  SpeechInputError,
12708
13356
  analyzePerformanceRecording,
12709
13357
  bindPushToTalk,
@@ -2,5 +2,6 @@
2
2
  <button id="start" disabled>Start conversation</button>
3
3
  <div id="replies"></div>
4
4
  <button id="talk" type="button" hidden disabled>Hold to talk</button>
5
+ <button id="see" type="button" hidden disabled>Let the character see you</button>
5
6
  <p id="status" role="status"></p>
6
7
  <script type="module" src="/main.js"></script>
package/examples/main.js CHANGED
@@ -10,6 +10,7 @@ const config = {
10
10
  const start = document.querySelector('#start')
11
11
  const replies = document.querySelector('#replies')
12
12
  const talk = document.querySelector('#talk')
13
+ const see = document.querySelector('#see')
13
14
  const status = document.querySelector('#status')
14
15
  const report = (error) => { status.textContent = error.message }
15
16
  const view = await mountNpcify({
@@ -56,12 +57,33 @@ character.onListeningState((state) => {
56
57
  { listening: 'Listening…', recognizing: 'Recognizing…' }[state] ?? 'Hold to talk'
57
58
  })
58
59
  character.onTranscript(({ text }) => { status.textContent = text })
60
+ // The character sees the camera when its owner turned sight on (the Sight
61
+ // tab). The first click asks for the camera permission.
62
+ const showSight = (state) => {
63
+ see.hidden = state === 'unavailable'
64
+ see.textContent =
65
+ { starting: 'Opening the camera…', on: 'Stop the camera' }[state] ?? 'Let the character see you'
66
+ }
67
+ showSight(character.sightState)
68
+ character.onSightState((state, error) => {
69
+ showSight(state)
70
+ if (error) report(error)
71
+ })
72
+ character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
73
+ see.onclick = () => {
74
+ if (character.sightState === 'off') {
75
+ character.startSight().catch(report)
76
+ } else {
77
+ character.stopSight()
78
+ }
79
+ }
59
80
  start.disabled = false
60
81
  start.onclick = async () => {
61
82
  start.disabled = true
62
83
  try {
63
84
  await character.unlockAudio()
64
85
  await character.createConversation()
86
+ see.disabled = false
65
87
  } catch (error) {
66
88
  report(error)
67
89
  start.disabled = false
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@npcify/web",
3
- "version": "0.4.1",
3
+ "version": "0.5.0",
4
4
  "description": "Framework-independent Npcify characters, live conversations and baked performances for the web",
5
5
  "type": "module",
6
6
  "license": "UNLICENSED",