@facepe/widget-mobile-app 1.0.0 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. package/README.md +293 -176
  2. package/dist/FacePeMobileAvatar.d.ts +1 -6
  3. package/dist/FacePeMobileAvatar.js +667 -95
  4. package/dist/FacePeMobileProvider.d.ts +2 -2
  5. package/dist/types.d.ts +35 -7
  6. package/dist/useFacePeAvatar.d.ts +3 -1
  7. package/dist/useFacePeAvatar.js +42 -1
  8. package/package.json +13 -5
package/README.md CHANGED
@@ -1,176 +1,293 @@
1
- # @facepe/widget-mobile-app
2
-
3
- The official **FacePe AI 3D Avatar Live Streaming SDK** for **React Native** (supporting both **iOS** and **Android**).
4
-
5
- Embed interactive, photorealistic, lip-synced 3D digital avatars directly inside your native mobile apps with real-time speech recognition, natural audio dialog, and event callbacks.
6
-
7
- ---
8
-
9
- ## 📱 Features
10
-
11
- - **⚡ Low-Latency Live Video & Audio**: Real-time WebRTC media streaming for avatar rendering.
12
- - **🗣️ Two-Way Audio Conversation**: Bidirectional voice talk with interrupt handling & VAD.
13
- - **🍎 Full iOS Support**: Compatible with iOS 13.0+ (iPhone & iPad).
14
- - **🤖 Full Android Support**: Compatible with Android 7.0+ (API 24+).
15
- - **💬 Real-Time Transcripts**: Captions and user transcripts via `onTranscript`.
16
- - **🛒 Conversational Ordering**: Extract order items and payment intents directly in your app.
17
-
18
- ---
19
-
20
- ## 📦 Installation
21
-
22
- In your React Native / Expo project:
23
-
24
- ```bash
25
- npm install @facepe/widget-mobile-app react-native-webview
26
- ```
27
- *(or using yarn / pnpm)*:
28
- ```bash
29
- yarn add @facepe/widget-mobile-app react-native-webview
30
- ```
31
-
32
- ---
33
-
34
- ## 🛠️ Platform Permissions Setup
35
-
36
- ### iOS Setup (`ios/Podfile` & `Info.plist`)
37
-
38
- 1. Open `ios/YourAppName/Info.plist` and add permission strings for microphone and camera:
39
-
40
- ```xml
41
- <key>NSMicrophoneUsageDescription</key>
42
- <string>We need access to your microphone to converse with the AI avatar.</string>
43
- <key>NSCameraUsageDescription</key>
44
- <string>We need access to your camera for interactive features.</string>
45
- ```
46
-
47
- 2. Run CocoaPods:
48
- ```bash
49
- cd ios && pod install && cd ..
50
- ```
51
-
52
- ---
53
-
54
- ### Android Setup (`android/app/src/main/AndroidManifest.xml`)
55
-
56
- Open `android/app/src/main/AndroidManifest.xml` and include permissions:
57
-
58
- ```xml
59
- <uses-permission android:name="android.permission.INTERNET" />
60
- <uses-permission android:name="android.permission.RECORD_AUDIO" />
61
- <uses-permission android:name="android.permission.CAMERA" />
62
- <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
63
- ```
64
-
65
- ---
66
-
67
- ## 🚀 Quick Start Example
68
-
69
- ### 1. Server-side Token Endpoint
70
-
71
- Exchange your `accessKey` and `secretKey` on your secure backend (Express / Next.js / Python / Go) so secrets are never exposed on mobile:
72
-
73
- ```javascript
74
- // POST /api/facepe-token
75
- app.get('/api/facepe-token', async (req, res) => {
76
- const response = await fetch('https://api.facepe.ai/api/v1/sdk/token', {
77
- method: 'POST',
78
- headers: { 'Content-Type': 'application/json' },
79
- body: JSON.stringify({
80
- accessKey: process.env.FACEPE_SDK_ACCESS_KEY,
81
- secretKey: process.env.FACEPE_SDK_SECRET_KEY,
82
- channel: 'sdk',
83
- }),
84
- });
85
- const data = await response.json();
86
- res.send(data.data.token);
87
- });
88
- ```
89
-
90
- ---
91
-
92
- ### 2. React Native Component
93
-
94
- ```tsx
95
- import React from 'react';
96
- import { StyleSheet, View, SafeAreaView, Text } from 'react-native';
97
- import {
98
- FacePeMobileProvider,
99
- FacePeMobileAvatar,
100
- } from '@facepe/widget-mobile-app';
101
-
102
- export default function App() {
103
- return (
104
- <SafeAreaView style={styles.container}>
105
- <Text style={styles.title}>FacePe AI Digital Human</Text>
106
-
107
- <FacePeMobileProvider
108
- apiBaseUrl="https://api.facepe.ai/api/v1/sdk"
109
- getToken={async () => {
110
- const res = await fetch('https://your-backend.com/api/facepe-token');
111
- return await res.text();
112
- }}
113
- >
114
- <View style={styles.avatarContainer}>
115
- <FacePeMobileAvatar
116
- language="en"
117
- onTranscript={({ payload }) => {
118
- console.log(`[${payload.role}]:`, payload.text);
119
- }}
120
- onSpeakStart={() => console.log('Avatar speaking')}
121
- onSpeakEnd={() => console.log('Avatar stopped speaking')}
122
- onOrderUpdate={({ items }) => console.log('Order update:', items)}
123
- onCheckout={() => console.log('Checkout requested!')}
124
- />
125
- </View>
126
- </FacePeMobileProvider>
127
- </SafeAreaView>
128
- );
129
- }
130
-
131
- const styles = StyleSheet.create({
132
- container: {
133
- flex: 1,
134
- backgroundColor: '#0F172A',
135
- },
136
- title: {
137
- color: '#FFFFFF',
138
- fontSize: 20,
139
- fontWeight: 'bold',
140
- textAlign: 'center',
141
- marginVertical: 16,
142
- },
143
- avatarContainer: {
144
- flex: 1,
145
- marginHorizontal: 16,
146
- marginBottom: 24,
147
- borderRadius: 24,
148
- overflow: 'hidden',
149
- },
150
- });
151
- ```
152
-
153
- ---
154
-
155
- ## ⚙️ Props & Configuration
156
-
157
- ### `<FacePeMobileAvatar />`
158
-
159
- | Prop | Type | Description |
160
- | :--- | :--- | :--- |
161
- | `avatarKey` | `string` | Optional specific avatar key assigned to your company. |
162
- | `language` | `string` | Spoken language (`'en'`, `'hi'`, `'es'`, `'ta'`, `'te'`, etc.). |
163
- | `style` | `ViewStyle` | Style applied to container element. |
164
- | `onTranscript` | `function` | Fires whenever the user or avatar utters a phrase. |
165
- | `onSpeakStart` | `function` | Fires when the avatar begins vocal response. |
166
- | `onSpeakEnd` | `function` | Fires when the avatar stops speaking. |
167
- | `onUserSpeakStart` | `function` | Fires when the user starts speaking (voice activity). |
168
- | `onUserSpeakEnd` | `function` | Fires when the user pauses / stops speaking. |
169
- | `onOrderUpdate` | `function` | Fires when items are detected in conversation. |
170
- | `onCheckout` | `function` | Fires when customer signals intent to pay. |
171
- | `onError` | `function` | Error callback `{ code, message }`. |
172
-
173
- ---
174
-
175
- ## 📄 License
176
- MIT © FacePe AI
1
+ # @facepe/widget-mobile-app
2
+
3
+ The official **FacePe AI Digital Human Live Streaming SDK** for **React Native** (supporting both **iOS** and **Android**).
4
+
5
+ Embed interactive, photorealistic, lip-synced 3D digital avatars directly inside your native mobile apps with pure WebRTC media streaming, native microphone capture, bidirectional voice dialog, and live data events — **without webviews**.
6
+
7
+ ---
8
+
9
+ ## 📱 Features
10
+
11
+ - **⚡ Native WebRTC Streaming**: Pure native media streaming powered by `@livekit/react-native` (no WebViews or embedded iframes).
12
+ - **🗣️ Natural Two-Way Voice Dialog**: Real-time microphone capture with hardware echo cancellation, noise suppression, and server VAD.
13
+ - **🍎 Full iOS Support**: Hardware-accelerated video decoding on iOS 13.0+ (iPhone & iPad).
14
+ - **🤖 Full Android Support**: Hardware-accelerated video decoding on Android 7.0+ (API 24+).
15
+ - **💬 Real-Time Transcriptions**: Bidirectional captions for avatar speech and user speech.
16
+ - **🛒 Conversational Commerce**: Automatic detection of cart items, orders, and payment intents.
17
+ - **🎛️ Imperative Controls**: Ref methods to start/stop stream, mute/unmute mic, speak text, or interrupt speech.
18
+
19
+ ---
20
+
21
+ ## 📦 Installation
22
+
23
+ In your React Native project:
24
+
25
+ ```bash
26
+ npm install @facepe/widget-mobile-app @livekit/react-native @livekit/react-native-webrtc livekit-client @anam-ai/js-sdk
27
+ ```
28
+
29
+ *(or using yarn / pnpm)*:
30
+
31
+ ```bash
32
+ yarn add @facepe/widget-mobile-app @livekit/react-native @livekit/react-native-webrtc livekit-client @anam-ai/js-sdk
33
+ ```
34
+
35
+ All avatars play natively (no WebView). The streaming libraries are loaded
36
+ only when a conversation starts. The app needs a development build (Expo Go
37
+ cannot load the native WebRTC module).
38
+
39
+ ---
40
+
41
+ ## 🛠️ Platform Permissions Setup
42
+
43
+ ### iOS Setup (`ios/Podfile` & `Info.plist`)
44
+
45
+ 1. Open `ios/YourAppName/Info.plist` and add permission strings for microphone and audio:
46
+
47
+ ```xml
48
+ <key>NSMicrophoneUsageDescription</key>
49
+ <string>We need access to your microphone to converse with the FacePe AI avatar.</string>
50
+ <key>NSCameraUsageDescription</key>
51
+ <string>Camera access for interactive features.</string>
52
+ ```
53
+
54
+ 2. Run CocoaPods:
55
+ ```bash
56
+ cd ios && pod install && cd ..
57
+ ```
58
+
59
+ ---
60
+
61
+ ### Android Setup (`android/app/src/main/AndroidManifest.xml`)
62
+
63
+ Open `android/app/src/main/AndroidManifest.xml` and include permissions:
64
+
65
+ ```xml
66
+ <uses-permission android:name="android.permission.INTERNET" />
67
+ <uses-permission android:name="android.permission.RECORD_AUDIO" />
68
+ <uses-permission android:name="android.permission.CAMERA" />
69
+ <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
70
+ <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
71
+ ```
72
+
73
+ ---
74
+
75
+ ## 🚀 Quick Start Example
76
+
77
+ ### 1. Server-side Token Endpoint
78
+
79
+ Exchange your `accessKey` and `secretKey` on your secure backend (Node.js / Express / Next.js / Python / Go) so credentials are never exposed inside mobile apps:
80
+
81
+ ```javascript
82
+ // Express server route: GET /api/facepe-token
83
+ app.get('/api/facepe-token', async (req, res) => {
84
+ const response = await fetch('https://api.facepe.ai/api/v1/sdk/token', {
85
+ method: 'POST',
86
+ headers: { 'Content-Type': 'application/json' },
87
+ body: JSON.stringify({
88
+ accessKey: process.env.FACEPE_SDK_ACCESS_KEY,
89
+ secretKey: process.env.FACEPE_SDK_SECRET_KEY,
90
+ channel: 'sdk',
91
+ }),
92
+ });
93
+ const data = await response.json();
94
+ res.send(data.data.token);
95
+ });
96
+ ```
97
+
98
+ ---
99
+
100
+ ### 2. React Native Component
101
+
102
+ ```tsx
103
+ import React, { useRef } from 'react';
104
+ import { StyleSheet, View, SafeAreaView, Text, TouchableOpacity } from 'react-native';
105
+ import {
106
+ FacePeMobileProvider,
107
+ FacePeMobileAvatar,
108
+ FacePeMobileAvatarRef,
109
+ useFacePeAvatar,
110
+ } from '@facepe/widget-mobile-app';
111
+
112
+ export default function App() {
113
+ const avatarRef = useRef<FacePeMobileAvatarRef>(null);
114
+
115
+ return (
116
+ <SafeAreaView style={styles.container}>
117
+ <Text style={styles.title}>FacePe AI Native Digital Human</Text>
118
+
119
+ <FacePeMobileProvider
120
+ apiBaseUrl="https://api.facepe.ai/api/v1/sdk"
121
+ getToken={async () => {
122
+ const res = await fetch('https://your-backend.com/api/facepe-token');
123
+ return await res.text();
124
+ }}
125
+ >
126
+ <AvatarScreen avatarRef={avatarRef} />
127
+ </FacePeMobileProvider>
128
+ </SafeAreaView>
129
+ );
130
+ }
131
+
132
+ function AvatarScreen({ avatarRef }: { avatarRef: React.RefObject<FacePeMobileAvatarRef> }) {
133
+ const { transcripts, isAvatarSpeaking, avatarProps } = useFacePeAvatar();
134
+
135
+ return (
136
+ <View style={styles.content}>
137
+ <View style={styles.avatarContainer}>
138
+ <FacePeMobileAvatar
139
+ ref={avatarRef}
140
+ language="en"
141
+ objectFit="cover"
142
+ autoStart={true}
143
+ enableMic={true}
144
+ {...avatarProps}
145
+ />
146
+ </View>
147
+
148
+ {/* Control bar */}
149
+ <View style={styles.controls}>
150
+ <TouchableOpacity
151
+ style={styles.button}
152
+ onPress={() => avatarRef.current?.toggleMic()}
153
+ >
154
+ <Text style={styles.buttonText}>Toggle Mic</Text>
155
+ </TouchableOpacity>
156
+
157
+ <TouchableOpacity
158
+ style={styles.button}
159
+ onPress={() => avatarRef.current?.interrupt()}
160
+ >
161
+ <Text style={styles.buttonText}>Interrupt</Text>
162
+ </TouchableOpacity>
163
+ </View>
164
+
165
+ {/* Real-time speech transcript log */}
166
+ <View style={styles.transcriptBox}>
167
+ {transcripts.slice(-3).map((item, idx) => (
168
+ <Text key={idx} style={styles.transcriptLine}>
169
+ <Text style={item.role === 'assistant' ? styles.avatarTag : styles.userTag}>
170
+ {item.role === 'assistant' ? 'Avatar: ' : 'You: '}
171
+ </Text>
172
+ {item.text}
173
+ </Text>
174
+ ))}
175
+ </View>
176
+ </View>
177
+ );
178
+ }
179
+
180
+ const styles = StyleSheet.create({
181
+ container: {
182
+ flex: 1,
183
+ backgroundColor: '#0F172A',
184
+ },
185
+ content: {
186
+ flex: 1,
187
+ padding: 16,
188
+ },
189
+ title: {
190
+ color: '#FFFFFF',
191
+ fontSize: 20,
192
+ fontWeight: 'bold',
193
+ textAlign: 'center',
194
+ marginVertical: 12,
195
+ },
196
+ avatarContainer: {
197
+ flex: 1,
198
+ borderRadius: 24,
199
+ overflow: 'hidden',
200
+ backgroundColor: '#1E293B',
201
+ },
202
+ controls: {
203
+ flexDirection: 'row',
204
+ justifyContent: 'center',
205
+ gap: 12,
206
+ marginVertical: 14,
207
+ },
208
+ button: {
209
+ backgroundColor: '#7C3AED',
210
+ paddingHorizontal: 20,
211
+ paddingVertical: 10,
212
+ borderRadius: 12,
213
+ },
214
+ buttonText: {
215
+ color: '#FFFFFF',
216
+ fontWeight: '600',
217
+ fontSize: 14,
218
+ },
219
+ transcriptBox: {
220
+ backgroundColor: '#1E293B',
221
+ padding: 12,
222
+ borderRadius: 12,
223
+ minHeight: 80,
224
+ },
225
+ transcriptLine: {
226
+ color: '#E2E8F0',
227
+ fontSize: 13,
228
+ marginBottom: 4,
229
+ },
230
+ avatarTag: {
231
+ color: '#A78BFA',
232
+ fontWeight: '700',
233
+ },
234
+ userTag: {
235
+ color: '#60A5FA',
236
+ fontWeight: '700',
237
+ },
238
+ });
239
+ ```
240
+
241
+ ---
242
+
243
+ ## ⚙️ Props & Configuration
244
+
245
+ ### `<FacePeMobileAvatar />`
246
+
247
+ | Prop | Type | Default | Description |
248
+ | :--- | :--- | :--- | :--- |
249
+ | `avatarKey` | `string` | *(optional)* | Specific avatar key or ID assigned to your company. |
250
+ | `language` | `string` | `'en'` | Spoken language code (`'en'`, `'hi'`, `'es'`, `'ta'`, `'te'`, etc.). |
251
+ | `style` | `ViewStyle` | `undefined` | Style applied to avatar container. |
252
+ | `videoStyle` | `ViewStyle` | `undefined` | Style applied to native video view. |
253
+ | `objectFit` | `'cover' \| 'contain'` | `'cover'` | Video aspect scaling mode. |
254
+ | `mirror` | `boolean` | `false` | Mirror video orientation. |
255
+ | `autoStart` | `boolean` | `true` | Automatically connect and start stream on mount. |
256
+ | `enableMic` | `boolean` | `true` | Automatically enable microphone on connect. |
257
+ | `onTranscript` | `function` | `undefined` | Fires when speech transcription arrives. |
258
+ | `onSpeakStart` | `function` | `undefined` | Fires when avatar starts speaking. |
259
+ | `onSpeakEnd` | `function` | `undefined` | Fires when avatar finishes speaking. |
260
+ | `onUserSpeakStart` | `function` | `undefined` | Fires when user voice activity begins. |
261
+ | `onUserSpeakEnd` | `function` | `undefined` | Fires when user voice activity ends. |
262
+ | `onOrderUpdate` | `function` | `undefined` | Fires when cart items or intents are recognized. |
263
+ | `onCheckout` | `function` | `undefined` | Fires when customer requests checkout / payment. |
264
+ | `onStatusChange` | `function` | `undefined` | Stream lifecycle status: `'idle' \| 'connecting' \| 'connected' \| 'reconnecting' \| 'disconnected' \| 'error'`. |
265
+ | `onError` | `function` | `undefined` | Stream error callback `{ code, message }`. |
266
+
267
+ ---
268
+
269
+ ### Imperative Ref (`FacePeMobileAvatarRef`)
270
+
271
+ ```tsx
272
+ const avatarRef = useRef<FacePeMobileAvatarRef>(null);
273
+
274
+ // Start streaming session
275
+ await avatarRef.current?.start();
276
+
277
+ // Stop streaming session & finalize billing usage
278
+ await avatarRef.current?.stop();
279
+
280
+ // Toggle mic on/off (returns new state boolean)
281
+ const isNowMuted = await avatarRef.current?.toggleMic();
282
+
283
+ // Make avatar speak programmatically
284
+ await avatarRef.current?.speak("Hello! How can I assist you today?");
285
+
286
+ // Interrupt avatar speech immediately
287
+ await avatarRef.current?.interrupt();
288
+ ```
289
+
290
+ ---
291
+
292
+ ## 📄 License
293
+ MIT © FacePe AI
package/dist/FacePeMobileAvatar.d.ts CHANGED
@@ -1,8 +1,3 @@
1
1
  import React from 'react';
2
- import type { FacePeMobileAvatarProps } from './types';
3
- export interface FacePeMobileAvatarRef {
4
- start: () => Promise<void>;
5
- stop: () => Promise<void>;
6
- toggleMic: () => Promise<void>;
7
- }
2
+ import type { FacePeMobileAvatarProps, FacePeMobileAvatarRef } from './types';
8
3
  export declare const FacePeMobileAvatar: React.ForwardRefExoticComponent<FacePeMobileAvatarProps & React.RefAttributes<FacePeMobileAvatarRef>>;