@thattobi/narrator-avatar 1.1.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/PUBLISHING.md +62 -0
- package/README.md +352 -0
- package/THIRD_PARTY_NOTICES.md +27 -0
- package/dist/NarratorAvatarFull-DhBLV6kE.js +37063 -0
- package/dist/NarratorAvatarFull-DhBLV6kE.js.map +1 -0
- package/dist/index-xRyAk0sW.js +734 -0
- package/dist/index-xRyAk0sW.js.map +1 -0
- package/dist/index.d.ts +72 -0
- package/dist/narrator-avatar.js +6 -0
- package/dist/narrator-avatar.js.map +1 -0
- package/dist/narrator-avatar.umd.cjs +4071 -0
- package/dist/narrator-avatar.umd.cjs.map +1 -0
- package/package.json +59 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 narrator-avatar contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/PUBLISHING.md
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Publishing @thattobi/narrator-avatar to npm
|
|
2
|
+
|
|
3
|
+
Run these commands from this project's root in PowerShell. This is a **new scoped package**, separate from the existing unscoped `narrator-avatar` package. The npm package does not contain avatar `.glb` files or Deepgram credentials.
|
|
4
|
+
|
|
5
|
+
## 1. Check your account and package name
|
|
6
|
+
|
|
7
|
+
Your npm username must be `thattobi` to publish under this scope. Sign in and confirm it:
|
|
8
|
+
|
|
9
|
+
```powershell
|
|
10
|
+
npm login
|
|
11
|
+
npm whoami
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`npm whoami` must print `thattobi`. Then check the registry and this checkout:
|
|
15
|
+
|
|
16
|
+
```powershell
|
|
17
|
+
npm view @thattobi/narrator-avatar version
|
|
18
|
+
node -p "require('./package.json').name + '@' + require('./package.json').version"
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Before the first release, `npm view` should return **404 Not Found**; that is expected. As of 24 September 2026, this scoped name was not published, and this checkout is set to `1.1.10`. For later releases, npm requires a version not already published under this scoped name. Bump both `package.json` and `package-lock.json` with:
|
|
22
|
+
|
|
23
|
+
```powershell
|
|
24
|
+
npm version patch --no-git-tag-version
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Use `minor` for a backward-compatible feature or `major` for a breaking API change. Do not bump just because you rebuilt.
|
|
28
|
+
|
|
29
|
+
## 2. Build and inspect locally
|
|
30
|
+
|
|
31
|
+
```powershell
|
|
32
|
+
npm install
|
|
33
|
+
npm run check
|
|
34
|
+
npm install --prefix playground
|
|
35
|
+
npm run build --prefix playground
|
|
36
|
+
npm pack --dry-run
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Inspect the `npm pack --dry-run` list before proceeding. It should contain `README.md`, `PUBLISHING.md`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, `package.json`, `dist/index.d.ts`, and the built JavaScript files. It must **not** contain `.env`, API keys, `node_modules`, playground assets, or avatar models. Check the actual output even though the `files` allowlist restricts it.
|
|
40
|
+
|
|
41
|
+
For a visual and speech smoke test, run `npm run dev`, open `http://localhost:5173`, select both avatars in turn, and test Speak, Stop, Pause/Resume, and teaching gestures. The playground's `VITE_DEEPGRAM_API_KEY` is visible in the browser; do not use a production key there.
|
|
42
|
+
|
|
43
|
+
## 3. Publish publicly
|
|
44
|
+
|
|
45
|
+
```powershell
|
|
46
|
+
npm publish --access public
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Complete npm's authentication prompt (for example, two-factor authentication). Do not paste npm tokens into the repository or a chat. `--access public` is needed for the first public release of a scoped package.
|
|
50
|
+
|
|
51
|
+
If npm reports that the scope belongs to another account, stop and confirm you logged in as `thattobi`.
|
|
52
|
+
|
|
53
|
+
## 4. Verify the release
|
|
54
|
+
|
|
55
|
+
```powershell
|
|
56
|
+
npm view @thattobi/narrator-avatar version
|
|
57
|
+
npm view @thattobi/narrator-avatar dist-tags
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Check the [npm package page](https://www.npmjs.com/package/@thattobi/narrator-avatar) and its rendered README. In a separate React app, run `npm install @thattobi/narrator-avatar` and follow the README's usage example with a hosted compatible GLB and a server-side Deepgram proxy. Apps currently importing `narrator-avatar` will **not** switch automatically; change their dependency and import to `@thattobi/narrator-avatar` when ready. Publishing does not deploy the playground or change any consuming app.
|
|
61
|
+
|
|
62
|
+
Official references: [npm's scoped public package guide](https://docs.npmjs.com/creating-and-publishing-scoped-public-packages/) and [npm pack documentation](https://docs.npmjs.com/cli/pack/).
|
package/README.md
ADDED
|
@@ -0,0 +1,352 @@
|
|
|
1
|
+
# @thattobi/narrator-avatar
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/@thattobi/narrator-avatar)
|
|
4
|
+
[](https://www.npmjs.com/package/@thattobi/narrator-avatar)
|
|
5
|
+
|
|
6
|
+
React component for 3D talking avatars with lip-sync, Deepgram or Google TTS, content-aware hand gestures, and pause/resume. Built on [@met4citizen/talkinghead](https://www.npmjs.com/package/@met4citizen/talkinghead).
|
|
7
|
+
|
|
8
|
+
## Start here
|
|
9
|
+
|
|
10
|
+
1. Install `@thattobi/narrator-avatar` in a React 18+ app: `npm install @thattobi/narrator-avatar`.
|
|
11
|
+
2. Put your compatible `.glb` in your app's public assets, for example `public/avatars/teacher.glb`. **The npm package does not include an avatar model.**
|
|
12
|
+
3. Render `<NarratorAvatar avatarMode="full" avatarUrl="/avatars/teacher.glb" ... />` in a container with an explicit width and height.
|
|
13
|
+
4. For speech, configure Deepgram through a same-origin server endpoint (recommended), or use a restricted test key locally. See [TTS credentials and server proxy](#tts-credentials-and-server-proxy).
|
|
14
|
+
5. Call `avatarRef.current?.speakText('Hello!')` from a user click so the browser can unlock audio.
|
|
15
|
+
|
|
16
|
+
For a ready-to-run local demo, see [Local development (playground)](#local-development-playground). For releasing a new version, see [PUBLISHING.md](PUBLISHING.md).
|
|
17
|
+
|
|
18
|
+
## Features
|
|
19
|
+
|
|
20
|
+
- **3D avatars** – Ready Player Me–compatible GLB models (full-body)
|
|
21
|
+
- **Lip-sync** – Deepgram audio-driven mouth shapes when AudioWorklet is available; estimated word timing is the fallback (English bundled; other languages need extra setup)
|
|
22
|
+
- **TTS** – Deepgram (streaming) or Google Cloud Text-to-Speech
|
|
23
|
+
- **Gestures** – 8 built-in hand poses plus 25 teaching gestures and combined sequences
|
|
24
|
+
- **Playback** – Speak, Pause, Resume, Stop (mid-phrase resume in accurate mode)
|
|
25
|
+
- **Speech reliability** – Prefetches upcoming phrases, retries TTS, resumes AudioContext on tab focus (reduces lag and silent dropouts)
|
|
26
|
+
- **Avatar Studio controls** – Visual presets, skin gloss/pores/warmth, eye contact, and mood
|
|
27
|
+
- **Accessibility** – Subtitle callback for closed captions
|
|
28
|
+
|
|
29
|
+
## Table of contents
|
|
30
|
+
|
|
31
|
+
- [Local development (playground)](#local-development-playground)
|
|
32
|
+
- [Install](#install)
|
|
33
|
+
- [Usage](#usage)
|
|
34
|
+
- [Props](#props)
|
|
35
|
+
- [Ref API](#ref-api)
|
|
36
|
+
- [Gesture names](#gesture-names)
|
|
37
|
+
- [TTS credentials and server proxy](#tts-credentials-and-server-proxy)
|
|
38
|
+
- [Environment variables](#environment-variables)
|
|
39
|
+
- [TypeScript](#typescript)
|
|
40
|
+
- [Troubleshooting](#troubleshooting)
|
|
41
|
+
- [Publishing](#publishing)
|
|
42
|
+
- [License](#license)
|
|
43
|
+
|
|
44
|
+
## Local development (playground)
|
|
45
|
+
|
|
46
|
+
Preview library changes with hot reload before publishing:
|
|
47
|
+
|
|
48
|
+
From the package root:
|
|
49
|
+
|
|
50
|
+
```powershell
|
|
51
|
+
npm install
|
|
52
|
+
npm install --prefix playground
|
|
53
|
+
Copy-Item playground/.env.example playground/.env
|
|
54
|
+
# Add your Deepgram key to playground/.env, then:
|
|
55
|
+
npm run dev
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
On macOS/Linux, use `cp playground/.env.example playground/.env` instead of `Copy-Item`. Open `http://localhost:5173`. Select `avatar.glb` or `male.glb`, wait for the avatar to load, then click **Speak**. The playground's `VITE_DEEPGRAM_API_KEY` is **local testing only**: it is visible in the browser. Do not deploy this setup unchanged.
|
|
59
|
+
|
|
60
|
+
The playground imports live source from `src/` (not `dist/`). Edit components under `src/` and the Vite app at http://localhost:5173 will pick up changes.
|
|
61
|
+
|
|
62
|
+
### Teacher experience
|
|
63
|
+
|
|
64
|
+
The playground defaults to your Avaturn **`avatar.glb`** and offers **`male.glb`** as the other local choice. Both models have the same teaching skeleton and all Oculus viseme targets required for lip-sync. MetaPerson is not loaded by the playground.
|
|
65
|
+
|
|
66
|
+
Teaching behavior (library tutor mode):
|
|
67
|
+
|
|
68
|
+
- Clearer phrase breaks and longer breaths after questions / key points
|
|
69
|
+
- Avatar-matched Deepgram voices: Thalia for `avatar.glb`, Arcas for `male.glb`
|
|
70
|
+
- Slightly slower speech and 700ms base phrase gaps
|
|
71
|
+
- Tutor gestures: deliberate hand poses with longer cooldowns and smooth transitions
|
|
72
|
+
- Audio-driven visemes from the actual speech waveform when AudioWorklet is available
|
|
73
|
+
|
|
74
|
+
Select the other GLB from the dropdown anytime.
|
|
75
|
+
|
|
76
|
+
### Custom Ready Player Me avatar
|
|
77
|
+
|
|
78
|
+
Stock `brunette.glb` is fine for smoke tests. For a more personal tutor mesh:
|
|
79
|
+
|
|
80
|
+
1. Create a full-body avatar at [readyplayer.me/avatar](https://readyplayer.me/avatar/)
|
|
81
|
+
2. Copy the avatar id from the share URL (24 hex characters)
|
|
82
|
+
3. Fetch it with TalkingHead morph targets into the playground:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
cd playground
|
|
86
|
+
npm run avatar:fetch -- <rpm-avatar-id> teacher
|
|
87
|
+
# writes public/avatars/teacher.glb and updates public/avatars/manifest.json
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
4. Refresh the playground and choose `teacher.glb` in **Local avatar**
|
|
91
|
+
|
|
92
|
+
You can also drop any compatible `.glb` into `playground/public/avatars/` and add its filename to `manifest.json`. Prefer local files — the Ready Player Me CDN is often blocked.
|
|
93
|
+
|
|
94
|
+
## Install
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
npm install @thattobi/narrator-avatar
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
React 18+ and React DOM 18+ are peer dependencies; install them in your app if they are not already present. The package bundles its renderer and English lip-sync—**no import maps or special Vite config required.** Your app supplies the avatar `.glb` and TTS endpoint.
|
|
101
|
+
|
|
102
|
+
## Usage
|
|
103
|
+
|
|
104
|
+
```jsx
|
|
105
|
+
import { useRef } from 'react';
|
|
106
|
+
import NarratorAvatar from '@thattobi/narrator-avatar';
|
|
107
|
+
|
|
108
|
+
function MyPage() {
|
|
109
|
+
const avatarRef = useRef(null);
|
|
110
|
+
|
|
111
|
+
return (
|
|
112
|
+
<div style={{ width: '400px', height: '500px' }}>
|
|
113
|
+
<NarratorAvatar
|
|
114
|
+
ref={avatarRef}
|
|
115
|
+
avatarMode="full"
|
|
116
|
+
avatarUrl="/avatars/teacher.glb"
|
|
117
|
+
avatarBody="F"
|
|
118
|
+
ttsService="deepgram"
|
|
119
|
+
ttsVoice="aura-2-thalia-en"
|
|
120
|
+
deepgramEndpoint="/api/deepgram/v1/speak"
|
|
121
|
+
accurateLipSync={true}
|
|
122
|
+
audioDrivenLipSync={true}
|
|
123
|
+
speechRate={0.9}
|
|
124
|
+
visualPreset="beauty"
|
|
125
|
+
skinGloss={1.5}
|
|
126
|
+
skinPores={2}
|
|
127
|
+
onReady={() => {}}
|
|
128
|
+
onSpeechStart={() => {}}
|
|
129
|
+
onSpeechEnd={() => {}}
|
|
130
|
+
onSubtitle={() => {}}
|
|
131
|
+
/>
|
|
132
|
+
<button onClick={() => avatarRef.current?.speakText('Hello! How are you?')}>
|
|
133
|
+
Speak
|
|
134
|
+
</button>
|
|
135
|
+
<button onClick={() => avatarRef.current?.pauseSpeaking()}>Pause</button>
|
|
136
|
+
<button onClick={() => avatarRef.current?.resumeSpeaking()}>Resume</button>
|
|
137
|
+
<button onClick={() => avatarRef.current?.stopSpeaking()}>Stop</button>
|
|
138
|
+
<button onClick={() => avatarRef.current?.makeEyeContact(2000)}>
|
|
139
|
+
Eye contact
|
|
140
|
+
</button>
|
|
141
|
+
</div>
|
|
142
|
+
);
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Place `teacher.glb` in your app's `public/avatars/` directory. The example expects your server to handle `/api/deepgram/v1/speak` and attach its Deepgram key. Do not put a production key in this browser component. In Next.js, put `"use client";` at the top of the component file.
|
|
147
|
+
|
|
148
|
+
## Props
|
|
149
|
+
|
|
150
|
+
| Prop | Description |
|
|
151
|
+
|------|-------------|
|
|
152
|
+
| `avatarUrl` | URL to GLB model (e.g. `/avatars/brunette.glb`) |
|
|
153
|
+
| `avatarBody` | `'M'` or `'F'` for posture |
|
|
154
|
+
| `cameraView` | Camera framing: `'full'`, `'mid'`, `'upper'`, `'head'` (default `'mid'`) |
|
|
155
|
+
| `cameraDistance`, `cameraY` | Optional distance and vertical offsets for precise framing; the playground uses them for a head-to-waist view |
|
|
156
|
+
| `cameraRotateEnable` | Allow mouse drag to rotate view (default `false`). Set `true` to enable. |
|
|
157
|
+
| `cameraZoomEnable` | Allow mouse wheel to zoom (default `false`). Set `true` to enable. |
|
|
158
|
+
| `cameraPanEnable` | Allow mouse to pan (default `false`) |
|
|
159
|
+
| `ttsService` | `'google'` or `'deepgram'` |
|
|
160
|
+
| `ttsVoice` | Deepgram: e.g. `aura-2-mars-en`, `aura-2-aurora-en`. Google: e.g. `en-GB-Standard-A` |
|
|
161
|
+
| `ttsApiKey` | Provider key. Avoid using this in a browser: expose a server-side TTS proxy instead. |
|
|
162
|
+
| `deepgramEndpoint` | Deepgram REST endpoint. Set this to a same-origin server-side proxy in production. |
|
|
163
|
+
| `lipsyncModules` | Array of language codes (default `['en']`) |
|
|
164
|
+
| `lipsyncLang` | Lip-sync language (default `'en'`) |
|
|
165
|
+
| `visualPreset` | Lighting preset: `'cinematic'`, `'beauty'`, `'studio'`, `'sunset'`, `'broadcast'` |
|
|
166
|
+
| `visualQuality` | `'auto'` (default): ultra on desktop, balanced on mobile/slow network. `'ultra'` \| `'balanced'` \| `'performance'` |
|
|
167
|
+
| `lazyMount` | `true` \| `false` \| `'auto'` (default): defer WebGL init until visible on mobile |
|
|
168
|
+
| `skinGloss` | Skin sheen/sweat intensity from `0` to `2` |
|
|
169
|
+
| `skinPores` | Procedural pore/normal intensity from `0` to `2` |
|
|
170
|
+
| `skinWarmth` | Warm skin tone blend from `0` to `2` |
|
|
171
|
+
| `eyeContactIntensity` | Eye openness/contact strength from `0` to `2` |
|
|
172
|
+
| `modelFPS`, `modelPixelRatio` | Render cadence and resolution multiplier. **TalkingHead also multiplies `modelPixelRatio` by `devicePixelRatio`**, so use values near `1` (not `4`). Mobile is capped automatically. |
|
|
173
|
+
| `dracoEnabled`, `dracoDecoderPath` | Enable Draco-compressed avatar loading |
|
|
174
|
+
| `modelDynamicBones` | TalkingHead dynamic-bone config for rigged hair/body parts |
|
|
175
|
+
| `update` | Per-frame callback `(dt, talkingHead)` |
|
|
176
|
+
| `accurateLipSync` | `true` = REST per phrase for stable playback and mid-phrase pause/resume. Default `true` |
|
|
177
|
+
| `audioDrivenLipSync` | `true` (default) = drive Deepgram mouth shapes from the audio waveform through HeadAudio; falls back to estimated word visemes if AudioWorklet cannot load |
|
|
178
|
+
| `speechRate` | For Deepgram Aura-2, `0.7`–`1.5`; sent as Deepgram's native `speed` control, preserving the voice's natural pitch and prosody. |
|
|
179
|
+
| `phraseGapMs` | Base teacher breath between speech units (default `560`). Adapts longer after `?` / long beats. Complete sentences are not glued together so breaths can land. |
|
|
180
|
+
| `speechGestures` | Content-aware hand gestures (default `true`) |
|
|
181
|
+
| `teachingAnimations` | Optional full-body FBX URLs keyed by the semantic teaching beats. Use this to replace a compact hand pose with a retargeted motion-capture clip. |
|
|
182
|
+
| `speechGestureStyle` | `'tutor'` (default): purposeful LMS gestures. `'kids'`: high-energy hands + face emoji reactions for children. `'expressive'`: legacy busier keyword mapping |
|
|
183
|
+
| `getGestureForPhrase` | `(phrase) => { name, dur?, mirror?, eyeContactMs? } \| null` to override per phrase |
|
|
184
|
+
| `avatarMode` | `'auto'` (default): **voice-only on mobile** (~22KB entry, no GLB/THREE); full 3D lazy-loaded on desktop. `'voice'` / `'full'` to force |
|
|
185
|
+
| `preloadText` | Lesson text to warm into the TTS cache when the avatar is ready (phrase 0 first, then ahead window) |
|
|
186
|
+
| `onPreloadReady` | Fires when phrase 0 is cached — Speak can start without waiting on Deepgram |
|
|
187
|
+
| `pauseWhenHidden` | `true` (default): pause on tab hide / phone call; resume on return. Leave/unmount always stops |
|
|
188
|
+
| `subtitleMaxWords` | Cap word-follow captions to last N words (`0` = full phrase). Reduces cut-off / layout shake |
|
|
189
|
+
| `onReady`, `onError`, `onSpeechStart`, `onSpeechEnd` | Lifecycle callbacks |
|
|
190
|
+
| `onSubtitle` | Word-follow caption callback as speech plays `(text: string) => void` |
|
|
191
|
+
|
|
192
|
+
## Ref API
|
|
193
|
+
|
|
194
|
+
| Method / property | Description |
|
|
195
|
+
|-------------------|-------------|
|
|
196
|
+
| `speakText(text, options?)` | Speak text via TTS |
|
|
197
|
+
| `preloadSpeech(text)` | Prefetch first phrases into TTS cache (call when lesson text is known) |
|
|
198
|
+
| `pauseSpeaking()` | Pause mid-phrase when `accurateLipSync` is true |
|
|
199
|
+
| `resumeSpeaking()` | Resume from the same point in the current phrase |
|
|
200
|
+
| `stopSpeaking()` | Stop and clear |
|
|
201
|
+
| `makeEyeContact(durationMs?)` | Ask the avatar to hold stronger eye contact |
|
|
202
|
+
| `setMood(mood)` | Change TalkingHead mood |
|
|
203
|
+
| `setLighting(options)` | Pass lighting options to TalkingHead |
|
|
204
|
+
| `setView(view, options?)` | Change camera view |
|
|
205
|
+
| `playGesture(name, dur?, mirror?, ms?)` | Play a built-in gesture |
|
|
206
|
+
| `playTeachingGesture(name, options?)` | Play a teaching gesture or a combined gesture sequence (see below) |
|
|
207
|
+
| `playAnimation(url, onprogress?, dur?, ndx?, scale?)` | Play a Mixamo/RPM FBX animation |
|
|
208
|
+
| `playPose(url, onprogress?, dur?, ndx?, scale?)` | Play a Mixamo/RPM FBX pose |
|
|
209
|
+
| `stopAnimation()`, `stopPose()` | Stop active animation or pose |
|
|
210
|
+
| `setMixerGain(speech, background?, fadeSecs?)` | Adjust speech/background audio gain |
|
|
211
|
+
| `playBackgroundAudio(url)`, `stopBackgroundAudio()` | Control background audio |
|
|
212
|
+
| `isReady` | Whether the avatar has finished loading |
|
|
213
|
+
| `isSpeaking` | Whether the avatar is currently speaking |
|
|
214
|
+
|
|
215
|
+
## Gesture names
|
|
216
|
+
|
|
217
|
+
Use `avatarRef.current?.playGesture(name, dur?, mirror?, ms?)` for a built-in pose. `dur` is in seconds, `ms` is the transition time in milliseconds, and `mirror` switches hands.
|
|
218
|
+
|
|
219
|
+
| Built-in pose names |
|
|
220
|
+
|---------------------|
|
|
221
|
+
| `handup`, `index`, `ok`, `thumbup`, `thumbdown`, `side`, `shrug`, `namaste` |
|
|
222
|
+
|
|
223
|
+
Use `avatarRef.current?.playTeachingGesture(name, { dur?, mirror?, blendMs? })` for a semantic teaching beat. It returns `false` if the full avatar is not ready or the name is unknown.
|
|
224
|
+
|
|
225
|
+
| Teaching beats | Extra reactions | Combined sequences |
|
|
226
|
+
|----------------|-----------------|--------------------|
|
|
227
|
+
| `greet`, `invite`, `explain`, `count`, `emphasize`, `celebrate`, `encourage`, `think`, `caution`, `thanks`, `recap`, `transition` | `wave`, `bow`, `nod`, `disagree`, `present`, `whiteboard` | `welcomeSequence`, `boardExplainSequence`, `guidedQuestionSequence`, `celebrateSequence`, `gentleCorrectionSequence`, `recapSequence`, `goodbyeSequence` |
|
|
228
|
+
|
|
229
|
+
```js
|
|
230
|
+
avatarRef.current?.playGesture('thumbup', 2, false, 850);
|
|
231
|
+
avatarRef.current?.playTeachingGesture('explain');
|
|
232
|
+
avatarRef.current?.playTeachingGesture('boardExplainSequence', { blendMs: 850 });
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
These are two different methods: pass `greet` to `playTeachingGesture`, not `playGesture`. Set `avatarMode="full"` if gestures must also appear on phones; `auto` switches Deepgram users to voice-only on mobile.
|
|
236
|
+
|
|
237
|
+
### Teaching gestures and sequences
|
|
238
|
+
|
|
239
|
+
The playground uses TalkingHead's built-in hand and face gestures with a tuned
|
|
240
|
+
head-to-waist camera view. Its teaching beats are `greet`, `invite`, `explain`, `count`,
|
|
241
|
+
`emphasize`, `celebrate`, `encourage`, `think`, `caution`, `thanks`, `recap`, and
|
|
242
|
+
`transition`. The extra reactions `wave`, `bow`, `nod`, `disagree`, `present`,
|
|
243
|
+
and `whiteboard` use the same built-in rig-safe gesture system.
|
|
244
|
+
|
|
245
|
+
For a single teaching performance rather than an isolated pose, call one of
|
|
246
|
+
the built-in sequences: `welcomeSequence`, `boardExplainSequence`,
|
|
247
|
+
`guidedQuestionSequence`, `celebrateSequence`, `gentleCorrectionSequence`,
|
|
248
|
+
`recapSequence`, or `goodbyeSequence`. Each sequence combines hand poses,
|
|
249
|
+
facial reactions, and eye contact. For example:
|
|
250
|
+
|
|
251
|
+
```js
|
|
252
|
+
avatarRef.current?.playTeachingGesture('boardExplainSequence');
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
The playground uses only the built-in rig-safe gestures. An unrelated FBX
|
|
256
|
+
motion pack previously bent an avatar's arms because its shoulder rest pose did
|
|
257
|
+
not match. The playground does not include a full-body motion pack.
|
|
258
|
+
|
|
259
|
+
The optional `teachingAnimations` prop is for FBX clips that you have already
|
|
260
|
+
retargeted and tested against your avatar. For example:
|
|
261
|
+
|
|
262
|
+
```jsx
|
|
263
|
+
<NarratorAvatar
|
|
264
|
+
teachingAnimations={{
|
|
265
|
+
greet: '/animations/wave.fbx',
|
|
266
|
+
explain: '/animations/explain.fbx',
|
|
267
|
+
count: '/animations/counting.fbx',
|
|
268
|
+
celebrate: '/animations/celebrate.fbx',
|
|
269
|
+
think: '/animations/thinking.fbx',
|
|
270
|
+
}}
|
|
271
|
+
/>
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Without a compatible mapping, `playTeachingGesture('explain')` uses the built-in
|
|
275
|
+
gesture. Test any custom animation against the exact avatar before adding it to
|
|
276
|
+
the mapping.
|
|
277
|
+
|
|
278
|
+
## TTS credentials and server proxy
|
|
279
|
+
|
|
280
|
+
Do **not** put Deepgram or Google credentials in `VITE_*` variables or pass a production secret through `ttsApiKey`: Vite embeds those values in the browser bundle. Route TTS through your server instead, enforce authentication and rate limits there, and keep provider credentials only on that server.
|
|
281
|
+
|
|
282
|
+
For Deepgram REST speech, set `deepgramEndpoint="/api/deepgram/v1/speak"`, leave `ttsApiKey` unset, and keep `accurateLipSync={true}`. Your server endpoint should accept `POST` with JSON `{ "text": "..." }` and the `model`, `speed`, `encoding`, `container`, and `sample_rate` query parameters, then forward the request to `https://api.deepgram.com/v1/speak` with `Authorization: Token <server-side key>`. Return the audio response with its content type. Reject unauthenticated or excessive requests before forwarding. The browser will not send the provider key in this setup.
|
|
283
|
+
|
|
284
|
+
The direct Deepgram WebSocket mode (`accurateLipSync={false}`) still needs browser-side credentials; use the REST proxy mode for production. The same-origin proxy works in both full-avatar and mobile voice-only modes.
|
|
285
|
+
|
|
286
|
+
The component retains `ttsApiKey` only for local experimentation and restricted test keys. It is not a secure production credential boundary.
|
|
287
|
+
|
|
288
|
+
## Environment variables
|
|
289
|
+
|
|
290
|
+
| Variable | Use |
|
|
291
|
+
|----------|-----|
|
|
292
|
+
| `VITE_DEEPGRAM_API_KEY` | Local development only; never deploy a provider secret with this prefix. |
|
|
293
|
+
| `VITE_GOOGLE_TTS_API_KEY` | Local development only; never deploy a provider secret with this prefix. |
|
|
294
|
+
|
|
295
|
+
## TypeScript
|
|
296
|
+
|
|
297
|
+
Type declarations ship with the npm package; no separate declaration file is needed:
|
|
298
|
+
|
|
299
|
+
```tsx
|
|
300
|
+
import { useRef } from 'react';
|
|
301
|
+
import NarratorAvatar, { type NarratorAvatarRef } from '@thattobi/narrator-avatar';
|
|
302
|
+
|
|
303
|
+
const avatarRef = useRef<NarratorAvatarRef>(null);
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
## Troubleshooting
|
|
307
|
+
|
|
308
|
+
| Symptom | Check |
|
|
309
|
+
|---------|-------|
|
|
310
|
+
| Avatar does not appear | Confirm `avatarUrl` returns your `.glb`, the parent has a nonzero width and height, and the model has a TalkingHead-compatible rig/visemes. The npm package does not ship a GLB. |
|
|
311
|
+
| Speak produces no audio | Wait for `onReady`, call `speakText` from a click/tap, and verify your Deepgram key (local only) or authenticated same-origin proxy endpoint. |
|
|
312
|
+
| CORS / failed to fetch | Use a same-origin server endpoint for production. The Vite proxy in the playground is for development only. |
|
|
313
|
+
| Gesture has no visible effect | Use `playGesture` for the eight built-in names, `playTeachingGesture` for teaching names, and `avatarMode="full"` on mobile. |
|
|
314
|
+
| Audio-driven mouth movement falls back | AudioWorklet must load in a secure context (`https://` or localhost) and be allowed by your Content Security Policy. Speech still works with estimated visemes when it cannot load. |
|
|
315
|
+
|
|
316
|
+
## Performance (web and mobile, one GLB per avatar)
|
|
317
|
+
|
|
318
|
+
Use **one** model URL per avatar (e.g. `/avatars/tutor.glb`). The component adapts at runtime — no `-mobile` / `-web` copies.
|
|
319
|
+
|
|
320
|
+
| Mechanism | Desktop | Mobile / Save-Data / low-end |
|
|
321
|
+
|-----------|---------|------------------------------|
|
|
322
|
+
| `visualQuality="auto"` | `ultra` — 60 FPS, studio lights, 4K shadows | Phones → `balanced` (20 FPS, capped DPR ≤~1.35, no shadows/studio lights). Low-RAM / 2G / Save-Data → `performance` (15 FPS, minimal lights) |
|
|
323
|
+
| Pixel ratio | ~1.25 × DPR | Multiplier auto-capped so total device pixels stay ~1.15–1.35 |
|
|
324
|
+
| Motion | Full idle/speaking head sway | Reduced `modelMovementFactor` + head move |
|
|
325
|
+
| TTS preload | First 2–3 phrases | First phrase only; cache capped |
|
|
326
|
+
| `lazyMount="auto"` | init immediately | init when scrolled into view |
|
|
327
|
+
| Load timeout | 30s | 120s (large GLB on cellular) |
|
|
328
|
+
| Tab hidden | animation paused | animation paused |
|
|
329
|
+
|
|
330
|
+
```jsx
|
|
331
|
+
<NarratorAvatar
|
|
332
|
+
avatarUrl="/avatars/test.glb"
|
|
333
|
+
visualQuality="auto"
|
|
334
|
+
lazyMount="auto"
|
|
335
|
+
/>
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
Force a lighter tier anytime: `visualQuality="performance"`.
|
|
339
|
+
|
|
340
|
+
**App tips:** mount **only one** avatar on narrow viewports; enable gzip/brotli (and Draco if you ship compressed GLBs) on your CDN.
|
|
341
|
+
|
|
342
|
+
Keep `modelPixelRatio` near **0.45–0.75 on mobile** if you override — TalkingHead multiplies it by `devicePixelRatio`.
|
|
343
|
+
|
|
344
|
+
**Next.js:** render the component from a client component (`"use client";`). Keep the Deepgram key in a server route, never in `NEXT_PUBLIC_*` variables.
|
|
345
|
+
|
|
346
|
+
## Publishing
|
|
347
|
+
|
|
348
|
+
Maintainers: follow [PUBLISHING.md](PUBLISHING.md) for the exact build, tarball inspection, version bump, npm login, and publish commands.
|
|
349
|
+
|
|
350
|
+
## License
|
|
351
|
+
|
|
352
|
+
[MIT](LICENSE)
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
The built JavaScript may contain code from the following MIT-licensed projects:
|
|
4
|
+
|
|
5
|
+
- TalkingHead — Copyright (c) 2023-2024 Mika Suominen
|
|
6
|
+
- HeadAudio — Copyright (c) 2025 Mika Suominen
|
|
7
|
+
- three.js — Copyright © 2010-2026 three.js authors
|
|
8
|
+
|
|
9
|
+
For each project, the following MIT license applies:
|
|
10
|
+
|
|
11
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
12
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
13
|
+
in the Software without restriction, including without limitation the rights
|
|
14
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
15
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
16
|
+
furnished to do so, subject to the following conditions:
|
|
17
|
+
|
|
18
|
+
The above copyright notice and this permission notice shall be included in all
|
|
19
|
+
copies or substantial portions of the Software.
|
|
20
|
+
|
|
21
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
22
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
23
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
24
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
25
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
26
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
27
|
+
SOFTWARE.
|