@npcify/web 0.4.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +75 -0
- package/dist/index.d.ts +463 -57
- package/dist/index.js +2757 -398
- package/examples/index.html +1 -0
- package/examples/main.js +22 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -125,11 +125,15 @@ animate()
|
|
|
125
125
|
- **Dialogue & Speech State**:
|
|
126
126
|
- `character.speaking`, `character.thinking`, `character.currentSpeech`, `character.currentText`, `character.revealedText`, `character.options`, `character.cursor`, `character.speech`.
|
|
127
127
|
- `await character.createConversation()`, `await character.sendMessage(text)`, `await character.endConversation()`.
|
|
128
|
+
- `character.setMemoryIdentifier(id)` / `character.getMemoryIdentifier()`: Set or read the SDK player memory identifier (`string | null | undefined`, up to 128 characters). When **SDK Player Memory** is enabled in **Export → Web SDK**, passing a non-empty identifier isolates memories per player; passing an empty string, `null`, or omitting `memoryIdentifier` stores memories in the character owner's dashboard bucket (**Dashboard → Memory**).
|
|
128
129
|
- `character.onConnectionClosed(callback)`: The server closed the conversation connection ([lifecycle](#runtime-and-lifecycle)).
|
|
129
130
|
- **Voice Input** ([details](#voice-input-and-input-modes)):
|
|
130
131
|
- `character.bindPushToTalk(options)`, `character.startListening(options)`, `character.stopListening()`, `character.cancelListening()`.
|
|
131
132
|
- `character.listeningState`, `character.onListeningState(callback)`, `character.onTranscript(callback)`, `character.inputModes`.
|
|
133
|
+
- `character.notifyUserTyping(active)`, `character.playerActivity`, `character.onPlayerActivity(callback)`: listening reflexes ([details](#voice-input-and-input-modes)).
|
|
132
134
|
- `await character.dispose()`: Ends conversation, detaches from the scene, removes the audio element, and disposes VRM GPU resources.
|
|
135
|
+
- **Sight** ([details](#sight)):
|
|
136
|
+
- `character.startSight(options)`, `character.stopSight()`, `character.sightState`, `character.onSightState(callback)`, `character.onSight(callback)`.
|
|
133
137
|
|
|
134
138
|
## Mode 2: Managed Canvas (`mountNpcify`)
|
|
135
139
|
|
|
@@ -237,6 +241,13 @@ character.onListeningState((state) => {
|
|
|
237
241
|
text when the server's speech model gives it.
|
|
238
242
|
- The first press or `startListening()` asks for the microphone permission.
|
|
239
243
|
An empty result sends nothing.
|
|
244
|
+
- Listening reflexes: while the player holds to talk, the character looks at
|
|
245
|
+
the player, shows a subtle attentive face and nods lightly at pauses. Call
|
|
246
|
+
`notifyUserTyping(true)` from the text field's `input` event (and `false`
|
|
247
|
+
after a send) for the same gaze and face while the player types; typing
|
|
248
|
+
ends 1.5 s after the last input. A reply takes over at once. This is local:
|
|
249
|
+
no request and no cost. `playerActivity` is `'idle'`, `'typing'` or
|
|
250
|
+
`'speaking'`.
|
|
240
251
|
- `language` is an optional BCP 47 hint (`'zh'`, `'ja'`, `'en'`). Without it,
|
|
241
252
|
the server detects the language.
|
|
242
253
|
- `inputModes` (an option of `mountNpcify`, `loadCharacter` and `Npcify`) is
|
|
@@ -253,6 +264,70 @@ character.onListeningState((state) => {
|
|
|
253
264
|
- The character owner pays the recognition at the price of the provider that
|
|
254
265
|
the owner chose, as for the turn's language model.
|
|
255
266
|
|
|
267
|
+
## Sight
|
|
268
|
+
|
|
269
|
+
The character can see the player's camera when the character owner turned
|
|
270
|
+
sight on (the Sight tab of the character in the dashboard, where the owner
|
|
271
|
+
also chooses the vision provider). The SDK captures small camera frames and
|
|
272
|
+
sends them on the conversation connection. The server sends each frame to the
|
|
273
|
+
vision provider and keeps only a short text observation; it does not keep the
|
|
274
|
+
frames. The character reads the observations of the last minute in its next
|
|
275
|
+
reply. See the [server contract](https://github.com/midaikidev/npcify/blob/develop/docs/sight.md).
|
|
276
|
+
|
|
277
|
+
```js
|
|
278
|
+
const see = document.querySelector('#see')
|
|
279
|
+
const report = (error) => { status.textContent = error.message }
|
|
280
|
+
see.hidden = character.sightState === 'unavailable'
|
|
281
|
+
see.onclick = () => {
|
|
282
|
+
if (character.sightState === 'off') {
|
|
283
|
+
character.startSight().catch(report)
|
|
284
|
+
} else {
|
|
285
|
+
character.stopSight()
|
|
286
|
+
}
|
|
287
|
+
}
|
|
288
|
+
character.onSightState((state, error) => {
|
|
289
|
+
see.hidden = state === 'unavailable'
|
|
290
|
+
see.textContent = state === 'on' ? 'Stop the camera' : 'Let the character see you'
|
|
291
|
+
if (error) report(error)
|
|
292
|
+
})
|
|
293
|
+
character.onSight(({ summary }) => { status.textContent = `The character sees: ${summary}` })
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
- `startSight({ source, intervalMs, motion, onError })` starts sight in the
|
|
297
|
+
current conversation; call `createConversation()` first. The first call
|
|
298
|
+
asks for the camera permission. `source` is `'camera'` (default) or a
|
|
299
|
+
`MediaStream` that you opened, for example a screen share. `stopSight()`
|
|
300
|
+
closes the camera that the SDK opened. The SDK never stops the tracks of a
|
|
301
|
+
stream that you give.
|
|
302
|
+
- Frames: the SDK makes each frame fit in 640×360 pixels (it does not make a
|
|
303
|
+
smaller picture larger) and sends it as a JPEG at quality 0.7. Every 250 ms
|
|
304
|
+
it compares a 32×18 grid of the luminance with the grid before. When the
|
|
305
|
+
mean change is more than `motion.threshold` (default 8, on a 0 to 255
|
|
306
|
+
scale), a frame goes, one each second at most. Else a frame goes every
|
|
307
|
+
`intervalMs` (default 5000 ms, minimum 1000 ms). `motion: false` sends
|
|
308
|
+
frames only at the interval. One frame at a time waits for the server: a
|
|
309
|
+
frame that comes due during the wait goes at the answer. While the page is
|
|
310
|
+
hidden, the SDK captures nothing.
|
|
311
|
+
- `sightState` is `'unavailable'` (the owner did not turn sight on, or the
|
|
312
|
+
conversation is baked), `'off'`, `'starting'` or `'on'`.
|
|
313
|
+
`onSightState((state, error) => …)` gives each change. `onSight((event) => …)`
|
|
314
|
+
gives each observation of the current conversation:
|
|
315
|
+
`{ observation: { people, persons, text }, summary, capturedAt, model, costPoints }`.
|
|
316
|
+
- Errors are `SightError`s with a `code`. `startSight()` rejects with
|
|
317
|
+
`SIGHT_DISABLED` (sight is unavailable), `CONVERSATION_REQUIRED`,
|
|
318
|
+
`SIGHT_ALREADY_STARTED`, `CAMERA_PERMISSION_DENIED`, `CAMERA_UNAVAILABLE`
|
|
319
|
+
or a code of the conversation connection.
|
|
320
|
+
- These codes stop sight after it started and go to `onError` and to the
|
|
321
|
+
`error` of `onSightState`: `SIGHT_DISABLED` (the owner turned sight off),
|
|
322
|
+
`INSUFFICIENT_POINTS` (the owner has no points),
|
|
323
|
+
`CONVERSATION_ACCESS_DENIED` and `SIGHT_SOURCE_ENDED` (the camera or the
|
|
324
|
+
shared screen stopped). The SDK skips one frame and continues for
|
|
325
|
+
`SIGHT_RATE_LIMITED`, `SIGHT_FRAME_TOO_LARGE`, `SIGHT_FRAME_INVALID`,
|
|
326
|
+
`SIGHT_PROVIDER_FAILED`, a lost connection and a timeout.
|
|
327
|
+
- `endConversation()`, `dispose()` and `loadAssets()` stop sight.
|
|
328
|
+
- The character owner pays the analysis of each frame at the price of the
|
|
329
|
+
provider that the owner chose. A refused or failed frame costs nothing.
|
|
330
|
+
|
|
256
331
|
## Runtime and lifecycle
|
|
257
332
|
|
|
258
333
|
- `mountNpcify({ canvas, baseUrl, characterId, apiKey, serverRendering })` returns
|