@npcify/web 0.4.1 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -125,11 +125,15 @@ animate()
125
125
  - **Dialogue & Speech State**:
126
126
  - `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
127
127
  - `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
128
+ - `character.setMemoryIdentifier(id)` / `character.getMemoryIdentifier()`: Set or read the SDK player memory identifier (`string | null | undefined`, up to 128 characters). When **SDK Player Memory** is enabled in **Export → Web SDK**, passing a non-empty identifier isolates memories per player; passing an empty string, `null`, or omitting `memoryIdentifier` stores memories in the character owner's dashboard bucket (**Dashboard → Memory**).
128
129
  - `character.onConnectionClosed(callback)`: The server closed the conversation connection ([lifecycle](#runtime-and-lifecycle)).
129
130
  - **Voice Input** ([details](#voice-input-and-input-modes)):
130
131
  - `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
131
132
  - `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
133
+ - `character.notifyUserTyping(active)`, `character.playerActivity`, `character.onPlayerActivity(callback)`: listening reflexes ([details](#voice-input-and-input-modes)).
132
134
  - `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
135
+ - **Sight** ([details](#sight)):
136
+ - `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
133
137
 
134
138
  ## Mode 2: Managed Canvas (`mountNpcify`)
135
139
 
@@ -237,6 +241,13 @@ character.onListeningState((state) => {
237
241
  text when the server's speech model gives it.
238
242
  - The first press or `startListening()` asks for the microphone permission.
239
243
  An empty result sends nothing.
244
+ - Listening reflexes: while the player holds to talk, the character looks at
245
+ the player, shows a subtle attentive face and nods lightly at pauses. Call
246
+ `notifyUserTyping(true)` from the text field's `input` event (and `false`
247
+ after a send) for the same gaze and face while the player types; typing
248
+ ends 1.5 s after the last input. A reply takes over at once. This is local:
249
+ no request and no cost. `playerActivity` is `'idle'`, `'typing'` or
250
+ `'speaking'`.
240
251
  - `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
241
252
  the server detects the language.
242
253
  - `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
@@ -253,6 +264,70 @@ character.onListeningState((state) => {
253
264
  - The character owner pays the recognition at the price of the provider that
254
265
  the owner chose, as for the turn's language model.
255
266
 
267
+ ## Sight
268
+
269
+ The character can see the player's camera when the character owner turned
270
+ sight on (the Sight tab of the character in the dashboard, where the owner
271
+ also chooses the vision provider). The SDK captures small camera frames and
272
+ sends them on the conversation connection. The server sends each frame to the
273
+ vision provider and keeps only a short text observation; it does not keep the
274
+ frames. The character reads the observations of the last minute in its next
275
+ reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
276
+
277
+ ```js
278
+ const see = document.querySelector('#see')
279
+ const report = (error) => { status.textContent = error.message }
280
+ see.hidden = character.sightState === 'unavailable'
281
+ see.onclick = () => {
282
+ if (character.sightState === 'off') {
283
+ character.startSight().catch(report)
284
+ } else {
285
+ character.stopSight()
286
+ }
287
+ }
288
+ character.onSightState((state, error) => {
289
+ see.hidden = state === 'unavailable'
290
+ see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
291
+ if (error) report(error)
292
+ })
293
+ character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
294
+ ```
295
+
296
+ - `startSight({ source, intervalMs, motion, onError })` starts sight in the
297
+ current conversation; call `createConversation()` first. The first call
298
+ asks for the camera permission. `source` is `'camera'` (default) or a
299
+ `MediaStream` that you opened, for example a screen share. `stopSight()`
300
+ closes the camera that the SDK opened. The SDK never stops the tracks of a
301
+ stream that you give.
302
+ - Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
303
+ smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
304
+ it compares a 32×18 grid of the luminance with the grid before. When the
305
+ mean change is more than `motion.threshold` (default 8, on a 0 to 255
306
+ scale), a frame goes, one each second at most. Else a frame goes every
307
+ `intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
308
+ frames only at the interval. One frame at a time waits for the server: a
309
+ frame that comes due during the wait goes at the answer. While the page is
310
+ hidden, the SDK captures nothing.
311
+ - `sightState` is `'unavailable'` (the owner did not turn sight on, or the
312
+ conversation is baked), `'off'`, `'starting'` or `'on'`.
313
+ `onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
314
+ gives each observation of the current conversation:
315
+ `{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
316
+ - Errors are `SightError`s with a `code`. `startSight()` rejects with
317
+ `SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
318
+ `SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
319
+ or a code of the conversation connection.
320
+ - These codes stop sight after it started and go to `onError` and to the
321
+ `error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
322
+ `INSUFFICIENT_POINTS` (the owner has no points),
323
+ `CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
324
+ shared screen stopped). The SDK skips one frame and continues for
325
+ `SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
326
+ `SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
327
+ - `endConversation()`, `dispose()` and `loadAssets()` stop sight.
328
+ - The character owner pays the analysis of each frame at the price of the
329
+ provider that the owner chose. A refused or failed frame costs nothing.
330
+
256
331
  ## Runtime and lifecycle
257
332
 
258
333
  - `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns