@mobileai/react-native 0.4.5 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +80 -15
  2. package/lib/module/components/AIAgent.js +181 -38
  3. package/lib/module/components/AIAgent.js.map +1 -1
  4. package/lib/module/components/AgentChatBar.js +53 -29
  5. package/lib/module/components/AgentChatBar.js.map +1 -1
  6. package/lib/module/components/Icons.js +337 -0
  7. package/lib/module/components/Icons.js.map +1 -0
  8. package/lib/module/core/AgentRuntime.js +74 -3
  9. package/lib/module/core/AgentRuntime.js.map +1 -1
  10. package/lib/module/core/systemPrompt.js +87 -34
  11. package/lib/module/core/systemPrompt.js.map +1 -1
  12. package/lib/module/services/AudioInputService.js +73 -2
  13. package/lib/module/services/AudioInputService.js.map +1 -1
  14. package/lib/module/services/AudioOutputService.js +58 -5
  15. package/lib/module/services/AudioOutputService.js.map +1 -1
  16. package/lib/module/services/VoiceService.js +284 -239
  17. package/lib/module/services/VoiceService.js.map +1 -1
  18. package/lib/typescript/src/components/AIAgent.d.ts.map +1 -1
  19. package/lib/typescript/src/components/AgentChatBar.d.ts.map +1 -1
  20. package/lib/typescript/src/components/Icons.d.ts +43 -0
  21. package/lib/typescript/src/components/Icons.d.ts.map +1 -0
  22. package/lib/typescript/src/core/AgentRuntime.d.ts +12 -0
  23. package/lib/typescript/src/core/AgentRuntime.d.ts.map +1 -1
  24. package/lib/typescript/src/core/systemPrompt.d.ts +7 -4
  25. package/lib/typescript/src/core/systemPrompt.d.ts.map +1 -1
  26. package/lib/typescript/src/services/AudioInputService.d.ts +13 -0
  27. package/lib/typescript/src/services/AudioInputService.d.ts.map +1 -1
  28. package/lib/typescript/src/services/AudioOutputService.d.ts.map +1 -1
  29. package/lib/typescript/src/services/VoiceService.d.ts +41 -24
  30. package/lib/typescript/src/services/VoiceService.d.ts.map +1 -1
  31. package/package.json +1 -1
  32. package/src/components/AIAgent.tsx +194 -38
  33. package/src/components/AgentChatBar.tsx +44 -25
  34. package/src/components/Icons.tsx +253 -0
  35. package/src/core/AgentRuntime.ts +70 -3
  36. package/src/core/systemPrompt.ts +87 -34
  37. package/src/services/AudioInputService.ts +77 -2
  38. package/src/services/AudioOutputService.ts +59 -5
  39. package/src/services/VoiceService.ts +280 -252
package/README.md CHANGED
@@ -16,17 +16,27 @@ Wrap your navigation with `<AIAgent>`. The AI automatically understands your ent
16
16
 
17
17
  ## ✨ Features
18
18
 
19
+ ### Text Mode
19
20
  - 🤖 **Zero-config UI understanding** — No annotations needed. The AI sees and understands your entire screen automatically.
20
21
  - 🎯 **Works with every component** — Buttons, switches, inputs, custom components — all work out of the box.
21
22
  - 🖼️ **Sees images & videos** — The AI knows what media is on screen and can describe it.
22
23
  - 🧭 **Auto-navigation** — Navigates between screens to complete multi-step tasks.
23
- - 🔒 **Production-grade security** — Element gating, content masking, lifecycle hooks, human-in-the-loop confirmation.
24
24
  - 🧩 **Custom actions** — Expose any business logic (checkout, API calls) as AI-callable tools with `useAction`.
25
25
  - 🌐 **MCP bridge** — Let external AI agents (OpenClaw, Claude Desktop) control your app remotely.
26
- - 🎙️ **Live Agent (Voice Mode)** — Real-time conversational AI that sees your screen and takes action while you speak.
27
- - 🎤 **Voice Dictation** — Built-in on-device speech-to-text for the text chat mode.
28
26
  - 🌍 **Bilingual** — English and Arabic support built-in.
29
27
 
28
+ ### 🎤 Voice Mode (Live Agent)
29
+ - 🗣️ **Real-time voice chat** — Bidirectional audio with Gemini Live API. Speak naturally, the agent responds with voice.
30
+ - 🔄 **Screen change detection** — The agent automatically detects when the screen changes (e.g., loading finishes) and updates its context — no polling tool needed.
31
+ - 🛡️ **Tool-first protocol** — Tool calls are emitted before speech to prevent server crashes (Gemini Live API limitation).
32
+ - 🔇 **Audio gating** — Mic is automatically paused during tool execution and resumed after, preventing race conditions.
33
+ - 🚫 **Auto-navigation guard** — Code-level gate rejects tool calls before the user speaks, preventing the model from acting on screen context alone.
34
+
35
+ ### Security & Production
36
+ - 🔒 **Production-grade security** — Element gating, content masking, lifecycle hooks, human-in-the-loop confirmation.
37
+
38
+ > **Provider support:** Currently supports **Google Gemini** only (`gemini-2.5-flash` for text, `gemini-2.5-flash-native-audio-preview` for voice). Additional providers may be added in future releases.
39
+
30
40
  ## 📦 Installation
31
41
 
32
42
  ```bash
@@ -39,20 +49,74 @@ No native modules required by default. Works with Expo managed workflow out of t
39
49
 
40
50
  ### Optional Native Dependencies
41
51
 
42
- If you want to use **Live Agent (Voice)**, **Text Mode Dictation**, or **Screenshots** (for image/video content), install these optional peer dependencies:
52
+ #### Screenshots
53
+
54
+ If you want to use **Screenshots** (for image/video content), install this optional peer dependency:
43
55
 
44
56
  ```bash
45
- # For Voice / Live Agent mode:
46
- npx expo install react-native-audio-api
57
+ npx expo install react-native-view-shot
58
+ ```
47
59
 
48
- # For Text mode voice dictation (mic button):
49
- npx expo install expo-speech-recognition
60
+ #### 🎤 Voice Mode (Real-time Voice Chat)
50
61
 
51
- # For Screenshots (when asking AI "what do you see in this image?"):
52
- npx expo install react-native-view-shot
62
+ Voice mode enables real-time bidirectional audio with the Gemini Live API. It requires one native module:
63
+
64
+ ```bash
65
+ # Audio capture + playback (required for voice mode):
66
+ npm install react-native-audio-api
67
+ ```
68
+
69
+ **After installing, you need native configuration based on your setup:**
70
+
71
+ <details>
72
+ <summary><b>Expo Managed Workflow</b></summary>
73
+
74
+ Add permissions to your `app.json`:
75
+
76
+ ```json
77
+ {
78
+ "expo": {
79
+ "android": {
80
+ "permissions": [
81
+ "RECORD_AUDIO",
82
+ "MODIFY_AUDIO_SETTINGS"
83
+ ]
84
+ },
85
+ "ios": {
86
+ "infoPlist": {
87
+ "NSMicrophoneUsageDescription": "Required for voice chat with AI assistant"
88
+ }
89
+ }
90
+ }
91
+ }
53
92
  ```
54
93
 
55
- > **Note on Permissions**: Ensure you add Microphone permissions (`NSMicrophoneUsageDescription` / `RECORD_AUDIO`) to your `app.json` or `Info.plist`/`AndroidManifest.xml` if using the audio packages.
94
+ Then rebuild: `npx expo prebuild && npx expo run:android` (or `run:ios`)
95
+
96
+ </details>
97
+
98
+ <details>
99
+ <summary><b>Expo Bare / React Native CLI</b></summary>
100
+
101
+ **Android** — add to `android/app/src/main/AndroidManifest.xml`:
102
+
103
+ ```xml
104
+ <uses-permission android:name="android.permission.RECORD_AUDIO"/>
105
+ <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
106
+ ```
107
+
108
+ **iOS** — add to `ios/YourApp/Info.plist`:
109
+
110
+ ```xml
111
+ <key>NSMicrophoneUsageDescription</key>
112
+ <string>Required for voice chat with AI assistant</string>
113
+ ```
114
+
115
+ Then rebuild: `npx react-native run-android` (or `run-ios`)
116
+
117
+ </details>
118
+
119
+ > **Note:** Hardware echo cancellation (AEC) is automatically enabled through `react-native-audio-api`'s AudioManager — no extra setup needed.
56
120
 
57
121
  ## 🚀 Quick Start
58
122
 
@@ -80,16 +144,17 @@ A floating chat bar appears automatically. Ask the AI to navigate, tap buttons,
80
144
 
81
145
  ### `<AIAgent>` Component
82
146
 
83
- The root provider. Wrap your app once at the top level. You can toggle between Text and Live Agent (Voice) modes.
147
+ The root provider. Wrap your app once at the top level.
84
148
 
85
149
  | Prop | Type | Default | Description |
86
150
  |------|------|---------|-------------|
87
151
  | `apiKey` | `string` | — | **Required.** Gemini API key. |
88
- | `model` | `string` | `'gemini-2.5-flash'` | Gemini model name. |
152
+ | `model` | `string` | `'gemini-2.5-flash'` | Gemini model name for text mode. |
89
153
  | `navRef` | `NavigationContainerRef` | — | Navigation ref for auto-navigation. |
90
- | `language` | `'en' \| 'ar'` | `'en'` | UI language. Forces Gemini Voice to speak the chosen dialect natively. |
91
- | `maxSteps` | `number` | `10` | Max steps per task. |
154
+ | `maxSteps` | `number` | `10` | Max steps per task (text mode). |
92
155
  | `showChatBar` | `boolean` | `true` | Show the floating chat bar. |
156
+ | `enableVoice` | `boolean` | `true` | Enable voice mode tab in the chat bar. |
157
+ | `language` | `'en' \| 'ar'` | `'en'` | Agent language (English/Arabic). |
93
158
  | `onResult` | `(result) => void` | — | Called when the agent finishes. |
94
159
 
95
160
  ### `useAction(name, description, params, handler)`
@@ -80,6 +80,10 @@ export function AIAgent({
80
80
  const voiceServiceRef = useRef(null);
81
81
  const audioInputRef = useRef(null);
82
82
  const audioOutputRef = useRef(null);
83
+ const toolLockRef = useRef(false);
84
+ const userHasSpokenRef = useRef(false);
85
+ const lastScreenContextRef = useRef('');
86
+ const screenPollIntervalRef = useRef(null);
83
87
 
84
88
  // Compute available modes from props
85
89
  const availableModes = useMemo(() => {
@@ -105,8 +109,10 @@ export function AIAgent({
105
109
  onAfterStep,
106
110
  onBeforeTask,
107
111
  onAfterTask,
108
- transformScreenContent,
109
- customTools,
112
+ customTools: mode === 'voice' ? {
113
+ ...customTools,
114
+ ask_user: null
115
+ } : customTools,
110
116
  instructions,
111
117
  stepDelay,
112
118
  mcpServerUrl,
@@ -115,7 +121,7 @@ export function AIAgent({
115
121
  onStatusUpdate: setStatusText,
116
122
  onTokenUsage,
117
123
  // Page-agent pattern: block the agent loop until user responds
118
- onAskUser: question => {
124
+ onAskUser: mode === 'voice' ? undefined : question => {
119
125
  return new Promise(resolve => {
120
126
  askUserResolverRef.current = resolve;
121
127
  // Show question in chat bar, allow user input
@@ -128,7 +134,7 @@ export function AIAgent({
128
134
  setStatusText('');
129
135
  });
130
136
  }
131
- }), [apiKey, model, language, maxSteps, interactiveBlacklist, interactiveWhitelist, onBeforeStep, onAfterStep, onBeforeTask, onAfterTask, transformScreenContent, customTools, instructions, stepDelay, mcpServerUrl, router, pathname, onTokenUsage]);
137
+ }), [mode, apiKey, model, language, maxSteps, interactiveBlacklist, interactiveWhitelist, onBeforeStep, onAfterStep, onBeforeTask, onAfterTask, transformScreenContent, customTools, instructions, stepDelay, mcpServerUrl, router, pathname, onTokenUsage]);
132
138
  const provider = useMemo(() => new GeminiProvider(apiKey, model), [apiKey, model]);
133
139
  const runtime = useMemo(() => new AgentRuntime(provider, config, rootViewRef.current, navRef),
134
140
  // eslint-disable-next-line react-hooks/exhaustive-deps
@@ -160,21 +166,25 @@ export function AIAgent({
160
166
  }
161
167
  logger.info('AIAgent', `Mode changed to "${mode}" — initializing voice services...`);
162
168
 
169
+ // Track async audio output init — mic MUST wait for this
170
+ let audioOutputInitPromise = Promise.resolve();
171
+
163
172
  // Create VoiceService with runtime's built-in tools (navigate, tap, type, done, etc.)
164
173
  if (!voiceServiceRef.current) {
165
174
  logger.info('AIAgent', 'Creating VoiceService...');
166
175
  const runtimeTools = runtime.getTools();
167
176
  logger.info('AIAgent', `Registering ${runtimeTools.length} tools with VoiceService: ${runtimeTools.map(t => t.name).join(', ')}`);
168
- // Build the full voice system prompt (screen format + tool descriptions + guardrails)
169
- // This gives voice mode the same screen understanding as text mode
177
+ // Use voice-adapted system prompt same core rules as text mode
178
+ // but without agent-loop directives that trigger autonomous actions
170
179
  const voicePrompt = buildVoiceSystemPrompt(language, instructions?.system);
180
+ logger.info('AIAgent', `📝 Voice system prompt (${voicePrompt.length} chars):\n${voicePrompt}`);
171
181
  voiceServiceRef.current = new VoiceService({
172
182
  apiKey,
173
183
  systemPrompt: voicePrompt,
174
184
  tools: runtimeTools,
175
185
  language
176
186
  });
177
- logger.info('AIAgent', 'VoiceService created with full voice system prompt and tools');
187
+ logger.info('AIAgent', `VoiceService created with ${runtimeTools.length} tools: ${runtimeTools.map(t => t.name).join(', ')}`);
178
188
  }
179
189
 
180
190
  // Create AudioOutputService if not exists
@@ -183,7 +193,11 @@ export function AIAgent({
183
193
  audioOutputRef.current = new AudioOutputService({
184
194
  onError: err => logger.error('AIAgent', `AudioOutput error: ${err}`)
185
195
  });
186
- audioOutputRef.current.initialize().then(ok => {
196
+ // IMPORTANT: Must await initialize() BEFORE starting mic.
197
+ // initialize() calls setAudioSessionOptions which reconfigures the
198
+ // audio hardware. If the mic starts before this finishes, the native
199
+ // audio session change kills the recorder's device handle.
200
+ audioOutputInitPromise = audioOutputRef.current.initialize().then(ok => {
187
201
  logger.info('AIAgent', `AudioOutputService initialized: ${ok}`);
188
202
  });
189
203
  }
@@ -194,7 +208,7 @@ export function AIAgent({
194
208
  audioInputRef.current = new AudioInputService({
195
209
  // Default 16kHz — Gemini Live API input standard
196
210
  onAudioChunk: chunk => {
197
- logger.debug('AIAgent', `Mic chunk: ${chunk.length} chars`);
211
+ logger.info('AIAgent', `🎤 onAudioChunk: ${chunk.length} chars, voiceService=${!!voiceServiceRef.current}, connected=${voiceServiceRef.current?.isConnected}`);
198
212
  voiceServiceRef.current?.sendAudio(chunk);
199
213
  },
200
214
  onError: err => logger.error('AIAgent', `AudioInput error: ${err}`),
@@ -202,65 +216,194 @@ export function AIAgent({
202
216
  });
203
217
  }
204
218
 
205
- // Connect VoiceService
219
+ // Connect VoiceService (async — SDK's ai.live.connect returns a Promise)
206
220
  logger.info('AIAgent', 'Connecting VoiceService...');
207
- voiceServiceRef.current.connect({
221
+ void voiceServiceRef.current.connect({
208
222
  onAudioResponse: audio => {
209
- logger.info('AIAgent', `Received audio response (${audio.length} chars)`);
223
+ logger.info('AIAgent', `🔊 Audio response: ${audio.length} chars, audioOutputRef=${!!audioOutputRef.current}`);
210
224
  setIsAISpeaking(true);
211
- audioOutputRef.current?.enqueue(audio);
225
+ if (!audioOutputRef.current) {
226
+ logger.error('AIAgent', '❌ audioOutputRef.current is NULL — cannot play audio!');
227
+ return;
228
+ }
229
+ audioOutputRef.current.enqueue(audio);
212
230
  },
213
231
  onStatusChange: status => {
214
232
  logger.info('AIAgent', `Voice status: ${status}`);
215
233
  const connected = status === 'connected';
216
234
  setIsVoiceConnected(connected);
217
235
  if (connected) {
218
- logger.info('AIAgent', '✅ VoiceService connected — auto-starting mic...');
219
- // Auto-start mic streaming once WebSocket is ready
220
- audioInputRef.current?.start().then(ok => {
221
- if (ok) {
222
- setIsMicActive(true);
223
- logger.info('AIAgent', '🎙️ Mic auto-started after connection');
224
- }
236
+ logger.info('AIAgent', '✅ VoiceService connected — waiting for audio session init before starting mic...');
237
+ // Wait for audio session config to finish BEFORE starting mic.
238
+ // If mic starts while setAudioSessionOptions is in flight,
239
+ // the native audio device gets killed (AudioDeviceStop error).
240
+ audioOutputInitPromise.then(() => {
241
+ logger.info('AIAgent', ' Audio session ready — starting mic now...');
242
+ audioInputRef.current?.start().then(ok => {
243
+ if (ok) {
244
+ setIsMicActive(true);
245
+ logger.info('AIAgent', '🎙️ Mic auto-started after connection');
246
+ }
247
+ });
225
248
  });
226
- // Send initial screen context (tree) so the model knows what's on screen
227
- const initialContext = runtime.getScreenContext();
228
- voiceServiceRef.current?.sendScreenContext(initialContext);
229
- logger.info('AIAgent', '📡 Initial screen context sent to voice model');
249
+ }
250
+
251
+ // Handle unexpected disconnection — auto-reconnect ONLY if not intentional
252
+ if (status === 'disconnected' && mode === 'voice' && voiceServiceRef.current && !voiceServiceRef.current.intentionalDisconnect) {
253
+ // Stop mic & audio immediately so user isn't stuck
254
+ audioInputRef.current?.stop();
255
+ audioOutputRef.current?.stop();
256
+ setIsMicActive(false);
257
+ setIsAISpeaking(false);
258
+ logger.warn('AIAgent', '⚠️ Voice connection dropped unexpectedly — attempting reconnect in 2s...');
259
+
260
+ // Auto-reconnect after short delay
261
+ setTimeout(() => {
262
+ if (mode === 'voice' && voiceServiceRef.current && !voiceServiceRef.current.intentionalDisconnect && voiceServiceRef.current.lastCallbacks) {
263
+ logger.info('AIAgent', '🔄 Reconnecting VoiceService...');
264
+ void voiceServiceRef.current.connect(voiceServiceRef.current.lastCallbacks);
265
+ }
266
+ }, 2000);
230
267
  }
231
268
  },
232
269
  onTranscript: (text, isFinal, role) => {
233
270
  logger.info('AIAgent', `Transcript [${role}] (final=${isFinal}): "${text}"`);
271
+ if (role === 'user') {
272
+ userHasSpokenRef.current = true;
273
+ }
274
+ },
275
+ onSetupComplete: () => {
276
+ logger.info('AIAgent', '✅ SDK setup complete — sending initial screen context now');
277
+ userHasSpokenRef.current = false;
278
+ const initialContext = runtime.getScreenContext();
279
+ lastScreenContextRef.current = initialContext; // Seed for screen change detection
280
+ const passiveContext = `[SYSTEM CONTEXT — THIS IS NOT A USER REQUEST. DO NOT RESPOND. DO NOT NAVIGATE. DO NOT CALL ANY TOOLS. DO NOT SPEAK. Just silently read and memorize this screen layout. You will use it when the user speaks their first voice command.]\n\n${initialContext}`;
281
+ logger.info('AIAgent', `📡 Initial screen context (${passiveContext.length} chars):\n${passiveContext}`);
282
+ voiceServiceRef.current?.sendScreenContext(passiveContext);
283
+ logger.info('AIAgent', '📡 Initial screen context sent (turnComplete=true)');
234
284
  },
235
285
  onToolCall: async toolCall => {
236
- logger.info('AIAgent', `Voice tool call: ${toolCall.name}(${JSON.stringify(toolCall.args)})`);
237
- // Execute the tool via AgentRuntime and send result back to Gemini
238
- const result = await runtime.executeTool(toolCall.name, toolCall.args);
239
- logger.info('AIAgent', `Voice tool result: ${result}`);
240
- voiceServiceRef.current?.sendFunctionResponse(toolCall.name, toolCall.id, {
241
- result
242
- });
286
+ logger.info('AIAgent', `🔧 Voice tool call: ${toolCall.name}(${JSON.stringify(toolCall.args)}) [id=${toolCall.id}]`);
287
+
288
+ // Code-level gate: reject tool calls before the user has spoken.
289
+ // The model sometimes auto-navigates on receiving screen context.
290
+ if (!userHasSpokenRef.current) {
291
+ logger.warn('AIAgent', `🚫 Rejected tool call ${toolCall.name} — user hasn't spoken yet`);
292
+ voiceServiceRef.current?.sendFunctionResponse(toolCall.name, toolCall.id, {
293
+ result: 'Action rejected: wait for the user to speak before performing any actions.'
294
+ });
295
+ return;
296
+ }
297
+
298
+ // CRITICAL: Gate audio input during tool execution.
299
+ // The Gemini Live API crashes (code 1008) if sendRealtimeInput
300
+ // (audio) is called while a tool call is pending. Stop the mic
301
+ // before executing the tool and resume after the response is sent.
302
+ audioInputRef.current?.stop();
303
+ logger.info('AIAgent', `🔇 Mic paused for tool execution: ${toolCall.name}`);
304
+
305
+ // One-tool-at-a-time enforcement (mirrors text mode's line 752).
306
+ if (toolLockRef.current) {
307
+ logger.warn('AIAgent', `⏳ Tool locked — waiting for previous tool to finish before executing ${toolCall.name}`);
308
+ while (toolLockRef.current) {
309
+ await new Promise(resolve => setTimeout(resolve, 50));
310
+ }
311
+ }
312
+ toolLockRef.current = true;
313
+ try {
314
+ // Execute the tool via AgentRuntime and send result back to Gemini
315
+ const result = await runtime.executeTool(toolCall.name, toolCall.args);
316
+ logger.info('AIAgent', `🔧 Tool result for ${toolCall.name}: ${result}`);
243
317
 
244
- // After tool execution, push updated screen context
245
- // (the screen may have changed from tap/type/navigate)
246
- const updatedContext = runtime.getScreenContext();
247
- voiceServiceRef.current?.sendScreenContext(updatedContext);
248
- logger.info('AIAgent', '📡 Updated screen context sent after tool call');
318
+ // Step delay matches text mode's stepDelay (line 820 in AgentRuntime).
319
+ await new Promise(resolve => setTimeout(resolve, 300));
320
+
321
+ // Include updated screen context IN the tool response
322
+ const updatedContext = runtime.getScreenContext();
323
+ lastScreenContextRef.current = updatedContext; // Sync with poll tracker
324
+ logger.info('AIAgent', `📡 Updated screen context after ${toolCall.name} (${updatedContext.length} chars):\n${updatedContext}`);
325
+ const enrichedResult = `${result}\n\n<updated_screen>\n${updatedContext}\n</updated_screen>`;
326
+ logger.info('AIAgent', `📡 Enriched tool response (${enrichedResult.length} chars):\n${enrichedResult}`);
327
+ voiceServiceRef.current?.sendFunctionResponse(toolCall.name, toolCall.id, {
328
+ result: enrichedResult
329
+ });
330
+ logger.info('AIAgent', `📡 Tool response sent for ${toolCall.name} [id=${toolCall.id}]`);
331
+ } finally {
332
+ toolLockRef.current = false;
333
+ // Resume mic after tool response is sent
334
+ if (voiceServiceRef.current?.isConnected) {
335
+ audioInputRef.current?.start().then(ok => {
336
+ if (ok) {
337
+ setIsMicActive(true);
338
+ logger.info('AIAgent', `🔊 Mic resumed after tool execution: ${toolCall.name}`);
339
+ }
340
+ });
341
+ }
342
+ }
249
343
  },
250
344
  onError: err => {
251
345
  logger.error('AIAgent', `VoiceService error: ${err}`);
346
+ // Stop mic & audio on error to prevent stale state
347
+ audioInputRef.current?.stop();
348
+ audioOutputRef.current?.stop();
349
+ setIsMicActive(false);
350
+ setIsAISpeaking(false);
252
351
  },
253
352
  onTurnComplete: () => {
254
353
  logger.info('AIAgent', 'AI turn complete');
255
354
  setIsAISpeaking(false);
355
+ // No cool-down or echo gate needed — hardware AEC handles everything.
356
+ // Mic stays active and ready for the next voice command immediately.
256
357
  }
257
358
  });
258
359
 
360
+ // ─── Screen Change Detection ───────────────────────────────
361
+ // Poll the Fiber tree every 5s and resend context if the screen meaningfully changed.
362
+ // This gives voice mode the same screen-awareness as text mode's per-step re-read.
363
+ const SCREEN_POLL_INTERVAL = 5000;
364
+ const MIN_DIFF_RATIO = 0.05; // Ignore changes smaller than 5% of total length (animation flicker)
365
+
366
+ screenPollIntervalRef.current = setInterval(() => {
367
+ if (!voiceServiceRef.current?.isConnected) return;
368
+ // Skip during tool execution — the enriched tool response handles that
369
+ if (toolLockRef.current) {
370
+ logger.debug('AIAgent', '🔄 Screen poll skipped — tool lock active');
371
+ return;
372
+ }
373
+ try {
374
+ const currentContext = runtime.getScreenContext();
375
+ if (currentContext === lastScreenContextRef.current) return; // No change
376
+
377
+ // Check if the change is meaningful (not just animation/cursor flicker)
378
+ const lastLen = lastScreenContextRef.current.length;
379
+ const diff = Math.abs(currentContext.length - lastLen);
380
+ const diffRatio = lastLen > 0 ? diff / lastLen : 1;
381
+ if (diffRatio < MIN_DIFF_RATIO) {
382
+ logger.debug('AIAgent', `🔄 Screen poll: minor change ignored (${diff} chars, ${(diffRatio * 100).toFixed(1)}% < ${MIN_DIFF_RATIO * 100}% threshold)`);
383
+ return;
384
+ }
385
+ logger.info('AIAgent', `🔄 Screen change detected (${lastLen} → ${currentContext.length} chars, ${(diffRatio * 100).toFixed(1)}% diff)`);
386
+ lastScreenContextRef.current = currentContext;
387
+ const passiveUpdate = `[SCREEN UPDATE — The UI has changed. Here is the current screen layout. This is not a user request — do not act unless the user asks.]\n\n${currentContext}`;
388
+ voiceServiceRef.current?.sendScreenContext(passiveUpdate);
389
+ logger.info('AIAgent', '🔄 Updated screen context sent to voice model');
390
+ } catch (err) {
391
+ logger.warn('AIAgent', `🔄 Screen poll error: ${err}`);
392
+ }
393
+ }, SCREEN_POLL_INTERVAL);
394
+
259
395
  // Cleanup on mode change back to text
260
396
  return () => {
261
397
  logger.info('AIAgent', `Cleaning up voice services (leaving "${mode}" mode)`);
398
+ // Stop screen change polling
399
+ if (screenPollIntervalRef.current) {
400
+ clearInterval(screenPollIntervalRef.current);
401
+ screenPollIntervalRef.current = null;
402
+ logger.info('AIAgent', '🔄 Screen poll stopped');
403
+ }
404
+ lastScreenContextRef.current = '';
262
405
  voiceServiceRef.current?.disconnect();
263
- voiceServiceRef.current = null; // Ensure fresh instance on next connect
406
+ voiceServiceRef.current = null;
264
407
  audioInputRef.current?.stop();
265
408
  setIsMicActive(false);
266
409
  setIsAISpeaking(false);
@@ -284,7 +427,7 @@ export function AIAgent({
284
427
  setIsMicActive(false);
285
428
  setIsAISpeaking(false);
286
429
  setIsVoiceConnected(false);
287
- // 5. Switch back to text mode (triggers cleanup effect naturally)
430
+ // 6. Switch back to text mode (triggers cleanup effect naturally)
288
431
  setMode('text');
289
432
  logger.info('AIAgent', '🛑 Voice session fully stopped');
290
433
  }, [runtime]);
@@ -1 +1 @@
1
- {"version":3,"names":["React","useCallback","useEffect","useMemo","useRef","useState","View","StyleSheet","AgentRuntime","GeminiProvider","AgentContext","AgentChatBar","AgentOverlay","logger","buildVoiceSystemPrompt","MCPBridge","VoiceService","AudioInputService","AudioOutputService","jsx","_jsx","jsxs","_jsxs","console","log","AIAgent","apiKey","model","navRef","language","maxSteps","showChatBar","children","onResult","interactiveBlacklist","interactiveWhitelist","onBeforeStep","onAfterStep","onBeforeTask","onAfterTask","transformScreenContent","customTools","instructions","stepDelay","mcpServerUrl","router","pathname","enableVoice","onTokenUsage","debug","setEnabled","info","rootViewRef","isThinking","setIsThinking","statusText","setStatusText","lastResult","setLastResult","mode","setMode","isMicActive","setIsMicActive","isSpeakerMuted","setIsSpeakerMuted","isAISpeaking","setIsAISpeaking","isVoiceConnected","setIsVoiceConnected","voiceServiceRef","audioInputRef","audioOutputRef","availableModes","modes","push","join","askUserResolverRef","config","onStatusUpdate","onAskUser","question","Promise","resolve","current","success","message","steps","provider","runtime","updateRefs","bridge","destroy","runtimeTools","getTools","length","map","t","name","voicePrompt","system","systemPrompt","tools","onError","err","error","initialize","then","ok","onAudioChunk","chunk","sendAudio","onPermissionDenied","warn","connect","onAudioResponse","audio","enqueue","onStatusChange","status","connected","start","initialContext","getScreenContext","sendScreenContext","onTranscript","text","isFinal","role","onToolCall","toolCall","JSON","stringify","args","result","executeTool","sendFunctionResponse","id","updatedContext","onTurnComplete","disconnect","stop","stopVoiceSession","handleSend","trim","resolver","execute","Provider","value","ref","style","styles","root","collapsable","visible","onSend","onDismiss","onModeChange","newMode","onStopSession","onMicToggle","active","onSpeakerToggle","muted","mute","unmute","create","flex"],"sourceRoot":"../../../src","sources":["components/AIAgent.tsx"],"mappings":";;AAAA;AACA;AACA;AACA;AACA;AACA;AACA;AACA;AACA;;AAEA,OAAOA,KAAK,IACVC,WAAW,EACXC,SAAS,EACTC,OAAO,EACPC,MAAM,EACNC,QAAQ,QACH,OAAO;AACd,SAASC,IAAI,EAAEC,UAAU,QAAQ,cAAc;AAC/C,SAASC,YAAY,QAAQ,yBAAsB;AACnD,SAASC,cAAc,QAAQ,gCAA6B;AAC5D,SAASC,YAAY,QAAQ,uBAAoB;AACjD,SAASC,YAAY,QAAQ,mBAAgB;AAC7C,SAASC,YAAY,QAAQ,mBAAgB;AAC7C,SAASC,MAAM,QAAQ,oBAAiB;AACxC,SAASC,sBAAsB,QAAQ,yBAAsB;AAC7D,SAASC,SAAS,QAAQ,sBAAmB;AAC7C,SAASC,YAAY,QAAQ,6BAA0B;AACvD,SAASC,iBAAiB,QAAQ,kCAA+B;AACjE,SAASC,kBAAkB,QAAQ,mCAAgC;AAAC,SAAAC,GAAA,IAAAC,IAAA,EAAAC,IAAA,IAAAC,KAAA;AAGpE;AACAC,OAAO,CAACC,GAAG,CAAC,8BAA8B,CAAC;;AAE3C;;AA+DA;;AAEA,OAAO,SAASC,OAAOA,CAAC;EACtBC,MAAM;EACNC,KAAK,GAAG,kBAAkB;EAC1BC,MAAM;EACNC,QAAQ,GAAG,IAAI;EACfC,QAAQ,GAAG,EAAE;EACbC,WAAW,GAAG,IAAI;EAClBC,QAAQ;EACRC,QAAQ;EACR;EACAC,oBAAoB;EACpBC,oBAAoB;EACpBC,YAAY;EACZC,WAAW;EACXC,YAAY;EACZC,WAAW;EACXC,sBAAsB;EACtBC,WAAW;EACXC,YAAY;EACZC,SAAS;EACTC,YAAY;EACZC,MAAM;EACNC,QAAQ;EACRC,WAAW,GAAG,KAAK;EACnBC,YAAY;EACZC,KAAK,GAAG;AACI,CAAC,EAAE;EACf;EACAjD,KAAK,CAACE,SAAS,CAAC,MAAM;IACpBqB,OAAO,CAACC,GAAG,CAAC,wBAAwB,EAAEyB,KAAK,EAAE,mBAAmB,CAAC;IACjEpC,MAAM,CAACqC,UAAU,CAACD,KAAK,CAAC;IACxB,IAAIA,KAAK,EAAE;MACTpC,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,0BAA0B,CAAC;IACpD;EACF,CAAC,EAAE,CAACF,KAAK,CAAC,CAAC;EAEX,MAAMG,WAAW,GAAGhD,MAAM,CAAM,IAAI,CAAC;EACrC,MAAM,CAACiD,UAAU,EAAEC,aAAa,CAAC,GAAGjD,QAAQ,CAAC,KAAK,CAAC;EACnD,MAAM,CAACkD,UAAU,EAAEC,aAAa,CAAC,GAAGnD,QAAQ,CAAC,EAAE,CAAC;EAChD,MAAM,CAACoD,UAAU,EAAEC,aAAa,CAAC,GAAGrD,QAAQ,CAAyB,IAAI,CAAC;;EAE1E;EACA,MAAM,CAACsD,IAAI,EAAEC,OAAO,CAAC,GAAGvD,QAAQ,CAAY,MAAM,CAAC;EACnD,MAAM,CAACwD,WAAW,EAAEC,cAAc,CAAC,GAAGzD,QAAQ,CAAC,KAAK,CAAC;EACrD,MAAM,CAAC0D,cAAc,EAAEC,iBAAiB,CAAC,GAAG3D,QAAQ,CAAC,KAAK,CAAC;EAC3D,MAAM,CAAC4D,YAAY,EAAEC,eAAe,CAAC,GAAG7D,QAAQ,CAAC,KAAK,CAAC;EACvD,MAAM,CAAC8D,gBAAgB,EAAEC,mBAAmB,CAAC,GAAG/D,QAAQ,CAAC,KAAK,CAAC;EAE/D,MAAMgE,eAAe,GAAGjE,MAAM,CAAsB,IAAI,CAAC;EACzD,MAAMkE,aAAa,GAAGlE,MAAM,CAA2B,IAAI,CAAC;EAC5D,MAAMmE,cAAc,GAAGnE,MAAM,CAA4B,IAAI,CAAC;;EAE9D;EACA,MAAMoE,cAA2B,GAAGrE,OAAO,CAAC,MAAM;IAChD,MAAMsE,KAAkB,GAAG,CAAC,MAAM,CAAC;IACnC,IAAI1B,WAAW,EAAE0B,KAAK,CAACC,IAAI,CAAC,OAAO,CAAC;IACpC7D,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoBsB,KAAK,CAACE,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;IAC9D,OAAOF,KAAK;EACd,CAAC,EAAE,CAAC1B,WAAW,CAAC,CAAC;;EAEjB;EACA,MAAM6B,kBAAkB,GAAGxE,MAAM,CAAoC,IAAI,CAAC;;EAE1E;;EAEA,MAAMyE,MAAmB,GAAG1E,OAAO,CAAC,OAAO;IACzCuB,MAAM;IACNC,KAAK;IACLE,QAAQ;IACRC,QAAQ;IACRI,oBAAoB;IACpBC,oBAAoB;IACpBC,YAAY;IACZC,WAAW;IACXC,YAAY;IACZC,WAAW;IACXC,sBAAsB;IACtBC,WAAW;IACXC,YAAY;IACZC,SAAS;IACTC,YAAY;IACZC,MAAM;IACNC,QAAQ;IACRgC,cAAc,EAAEtB,aAAa;IAC7BR,YAAY;IACZ;IACA+B,SAAS,EAAGC,QAAgB,IAAK;MAC/B,OAAO,IAAIC,OAAO,CAAUC,OAAO,IAAK;QACtCN,kBAAkB,CAACO,OAAO,GAAGD,OAAO;QACpC;QACAxB,aAAa,CAAC;UAAE0B,OAAO,EAAE,IAAI;UAAEC,OAAO,EAAE,KAAKL,QAAQ,EAAE;UAAEM,KAAK,EAAE;QAAG,CAAC,CAAC;QACrEhC,aAAa,CAAC,KAAK,CAAC;QACpBE,aAAa,CAAC,EAAE,CAAC;MACnB,CAAC,CAAC;IACJ;EACF,CAAC,CAAC,EAAE,CACF9B,MAAM,EAAEC,KAAK,EAAEE,QAAQ,EAAEC,QAAQ,EACjCI,oBAAoB,EAAEC,oBAAoB,EAC1CC,YAAY,EAAEC,WAAW,EAAEC,YAAY,EAAEC,WAAW,EACpDC,sBAAsB,EAAEC,WAAW,EAAEC,YAAY,EAAEC,SAAS,EAC5DC,YAAY,EAAEC,MAAM,EAAEC,QAAQ,EAAEE,YAAY,CAC7C,CAAC;EAEF,MAAMuC,QAAQ,GAAGpF,OAAO,CAAC,MAAM,IAAIM,cAAc,CAACiB,MAAM,EAAEC,KAAK,CAAC,EAAE,CAACD,MAAM,EAAEC,KAAK,CAAC,CAAC;EAElF,MAAM6D,OAAO,GAAGrF,OAAO,CACrB,MAAM,IAAIK,YAAY,CAAC+E,QAAQ,EAAEV,MAAM,EAAEzB,WAAW,CAAC+B,OAAO,EAAEvD,MAAM,CAAC;EACrE;EACA,CAAC2D,QAAQ,EAAEV,MAAM,CACnB,CAAC;;EAED;EACA3E,SAAS,CAAC,MAAM;IACdsF,OAAO,CAACC,UAAU,CAACrC,WAAW,CAAC+B,OAAO,EAAEvD,MAAM,CAAC;EACjD,CAAC,EAAE,CAAC4D,OAAO,EAAE5D,MAAM,CAAC,CAAC;;EAErB;;EAEA1B,SAAS,CAAC,MAAM;IACd,IAAI,CAAC0C,YAAY,EAAE;IAEnB/B,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4BP,YAAY,EAAE,CAAC;IAClE,MAAM8C,MAAM,GAAG,IAAI3E,SAAS,CAAC6B,YAAY,EAAE4C,OAAO,CAAC;IAEnD,OAAO,MAAM;MACXE,MAAM,CAACC,OAAO,CAAC,CAAC;IAClB,CAAC;EACH,CAAC,EAAE,CAAC/C,YAAY,EAAE4C,OAAO,CAAC,CAAC;;EAE3B;;EAEA;EACAtF,SAAS,CAAC,MAAM;IACd,IAAIyD,IAAI,KAAK,MAAM,EAAE;MACnB9C,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,yCAAyC,CAAC;MACjE;IACF;IAEAtC,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoBQ,IAAI,oCAAoC,CAAC;;IAEpF;IACA,IAAI,CAACU,eAAe,CAACc,OAAO,EAAE;MAC5BtE,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,0BAA0B,CAAC;MAClD,MAAMyC,YAAY,GAAGJ,OAAO,CAACK,QAAQ,CAAC,CAAC;MACvChF,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAeyC,YAAY,CAACE,MAAM,6BAA6BF,YAAY,CAACG,GAAG,CAACC,CAAC,IAAIA,CAAC,CAACC,IAAI,CAAC,CAACtB,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;MACjI;MACA;MACA,MAAMuB,WAAW,GAAGpF,sBAAsB,CAACe,QAAQ,EAAEa,YAAY,EAAEyD,MAAM,CAAC;MAC1E9B,eAAe,CAACc,OAAO,GAAG,IAAInE,YAAY,CAAC;QACzCU,MAAM;QACN0E,YAAY,EAAEF,WAAW;QACzBG,KAAK,EAAET,YAAY;QACnB/D;MACF,CAAC,CAAC;MACFhB,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,8DAA8D,CAAC;IACxF;;IAEA;IACA,IAAI,CAACoB,cAAc,CAACY,OAAO,EAAE;MAC3BtE,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gCAAgC,CAAC;MACxDoB,cAAc,CAACY,OAAO,GAAG,IAAIjE,kBAAkB,CAAC;QAC9CoF,OAAO,EAAGC,GAAG,IAAK1F,MAAM,CAAC2F,KAAK,CAAC,SAAS,EAAE,sBAAsBD,GAAG,EAAE;MACvE,CAAC,CAAC;MACFhC,cAAc,CAACY,OAAO,CAACsB,UAAU,CAAC,CAAC,CAACC,IAAI,CAAEC,EAAE,IAAK;QAC/C9F,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,mCAAmCwD,EAAE,EAAE,CAAC;MACjE,CAAC,CAAC;IACJ;;IAEA;IACA,IAAI,CAACrC,aAAa,CAACa,OAAO,EAAE;MAC1BtE,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,+BAA+B,CAAC;MACvDmB,aAAa,CAACa,OAAO,GAAG,IAAIlE,iBAAiB,CAAC;QAC5C;QACA2F,YAAY,EAAGC,KAAK,IAAK;UACvBhG,MAAM,CAACoC,KAAK,CAAC,SAAS,EAAE,cAAc4D,KAAK,CAACf,MAAM,QAAQ,CAAC;UAC3DzB,eAAe,CAACc,OAAO,EAAE2B,SAAS,CAACD,KAAK,CAAC;QAC3C,CAAC;QACDP,OAAO,EAAGC,GAAG,IAAK1F,MAAM,CAAC2F,KAAK,CAAC,SAAS,EAAE,qBAAqBD,GAAG,EAAE,CAAC;QACrEQ,kBAAkB,EAAEA,CAAA,KAAMlG,MAAM,CAACmG,IAAI,CAAC,SAAS,EAAE,+BAA+B;MAClF,CAAC,CAAC;IACJ;;IAEA;IACAnG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4B,CAAC;IACpDkB,eAAe,CAACc,OAAO,CAAC8B,OAAO,CAAC;MAC9BC,eAAe,EAAGC,KAAK,IAAK;QAC1BtG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4BgE,KAAK,CAACrB,MAAM,SAAS,CAAC;QACzE5B,eAAe,CAAC,IAAI,CAAC;QACrBK,cAAc,CAACY,OAAO,EAAEiC,OAAO,CAACD,KAAK,CAAC;MACxC,CAAC;MACDE,cAAc,EAAGC,MAAM,IAAK;QAC1BzG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,iBAAiBmE,MAAM,EAAE,CAAC;QACjD,MAAMC,SAAS,GAAGD,MAAM,KAAK,WAAW;QACxClD,mBAAmB,CAACmD,SAAS,CAAC;QAC9B,IAAIA,SAAS,EAAE;UACb1G,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,iDAAiD,CAAC;UACzE;UACAmB,aAAa,CAACa,OAAO,EAAEqC,KAAK,CAAC,CAAC,CAACd,IAAI,CAAEC,EAAE,IAAK;YAC1C,IAAIA,EAAE,EAAE;cACN7C,cAAc,CAAC,IAAI,CAAC;cACpBjD,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,uCAAuC,CAAC;YACjE;UACF,CAAC,CAAC;UACF;UACA,MAAMsE,cAAc,GAAGjC,OAAO,CAACkC,gBAAgB,CAAC,CAAC;UACjDrD,eAAe,CAACc,OAAO,EAAEwC,iBAAiB,CAACF,cAAc,CAAC;UAC1D5G,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,+CAA+C,CAAC;QACzE;MACF,CAAC;MACDyE,YAAY,EAAEA,CAACC,IAAI,EAAEC,OAAO,EAAEC,IAAI,KAAK;QACrClH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAe4E,IAAI,YAAYD,OAAO,OAAOD,IAAI,GAAG,CAAC;MAC9E,CAAC;MACDG,UAAU,EAAE,MAAOC,QAAQ,IAAK;QAC9BpH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoB8E,QAAQ,CAAChC,IAAI,IAAIiC,IAAI,CAACC,SAAS,CAACF,QAAQ,CAACG,IAAI,CAAC,GAAG,CAAC;QAC7F;QACA,MAAMC,MAAM,GAAG,MAAM7C,OAAO,CAAC8C,WAAW,CAACL,QAAQ,CAAChC,IAAI,EAAEgC,QAAQ,CAACG,IAAI,CAAC;QACtEvH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,sBAAsBkF,MAAM,EAAE,CAAC;QACtDhE,eAAe,CAACc,OAAO,EAAEoD,oBAAoB,CAACN,QAAQ,CAAChC,IAAI,EAAEgC,QAAQ,CAACO,EAAE,EAAE;UAAEH;QAAO,CAAC,CAAC;;QAErF;QACA;QACA,MAAMI,cAAc,GAAGjD,OAAO,CAACkC,gBAAgB,CAAC,CAAC;QACjDrD,eAAe,CAACc,OAAO,EAAEwC,iBAAiB,CAACc,cAAc,CAAC;QAC1D5H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gDAAgD,CAAC;MAC1E,CAAC;MACDmD,OAAO,EAAGC,GAAG,IAAK;QAChB1F,MAAM,CAAC2F,KAAK,CAAC,SAAS,EAAE,uBAAuBD,GAAG,EAAE,CAAC;MACvD,CAAC;MACDmC,cAAc,EAAEA,CAAA,KAAM;QACpB7H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,kBAAkB,CAAC;QAC1Ce,eAAe,CAAC,KAAK,CAAC;MACxB;IACF,CAAC,CAAC;;IAEF;IACA,OAAO,MAAM;MACXrD,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wCAAwCQ,IAAI,SAAS,CAAC;MAC7EU,eAAe,CAACc,OAAO,EAAEwD,UAAU,CAAC,CAAC;MACrCtE,eAAe,CAACc,OAAO,GAAG,IAAI,CAAC,CAAC;MAChCb,aAAa,CAACa,OAAO,EAAEyD,IAAI,CAAC,CAAC;MAC7B9E,cAAc,CAAC,KAAK,CAAC;MACrBI,eAAe,CAAC,KAAK,CAAC;MACtBE,mBAAmB,CAAC,KAAK,CAAC;IAC5B,CAAC;IACD;EACF,CAAC,EAAE,CAACT,IAAI,EAAEjC,MAAM,EAAE8D,OAAO,EAAE3D,QAAQ,EAAEa,YAAY,CAAC,CAAC;;EAEnD;;EAEA,MAAMmG,gBAAgB,GAAG5I,WAAW,CAAC,MAAM;IACzCY,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,6CAA6C,CAAC;IACrE;IACAmB,aAAa,CAACa,OAAO,EAAEyD,IAAI,CAAC,CAAC;IAC7B;IACArE,cAAc,CAACY,OAAO,EAAEyD,IAAI,CAAC,CAAC;IAC9B;IACAvE,eAAe,CAACc,OAAO,EAAEwD,UAAU,CAAC,CAAC;IACrCtE,eAAe,CAACc,OAAO,GAAG,IAAI;IAC9B;IACArB,cAAc,CAAC,KAAK,CAAC;IACrBI,eAAe,CAAC,KAAK,CAAC;IACtBE,mBAAmB,CAAC,KAAK,CAAC;IAC1B;IACAR,OAAO,CAAC,MAAM,CAAC;IACf/C,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gCAAgC,CAAC;EAC1D,CAAC,EAAE,CAACqC,OAAO,CAAC,CAAC;;EAEb;;EAEA,MAAMsD,UAAU,GAAG7I,WAAW,CAAC,MAAOoF,OAAe,IAAK;IACxD,IAAI,CAACA,OAAO,CAAC0D,IAAI,CAAC,CAAC,EAAE;IAErBlI,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,kBAAkBkC,OAAO,GAAG,CAAC;;IAEpD;IACA,IAAIT,kBAAkB,CAACO,OAAO,EAAE;MAC9B,MAAM6D,QAAQ,GAAGpE,kBAAkB,CAACO,OAAO;MAC3CP,kBAAkB,CAACO,OAAO,GAAG,IAAI;MACjC7B,aAAa,CAAC,IAAI,CAAC;MACnBE,aAAa,CAAC,2BAA2B,CAAC;MAC1CE,aAAa,CAAC,IAAI,CAAC;MACnBsF,QAAQ,CAAC3D,OAAO,CAAC;MACjB;IACF;;IAEA;IACA/B,aAAa,CAAC,IAAI,CAAC;IACnBE,aAAa,CAAC,aAAa,CAAC;IAC5BE,aAAa,CAAC,IAAI,CAAC;IAEnB,IAAI;MACF;MACA8B,OAAO,CAACC,UAAU,CAACrC,WAAW,CAAC+B,OAAO,EAAEvD,MAAM,CAAC;MAE/C,MAAMyG,MAAM,GAAG,MAAM7C,OAAO,CAACyD,OAAO,CAAC5D,OAAO,CAAC;MAE7C3B,aAAa,CAAC2E,MAAM,CAAC;MACrBpG,QAAQ,GAAGoG,MAAM,CAAC;MAElBxH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,WAAWkF,MAAM,CAACjD,OAAO,GAAG,GAAG,GAAG,GAAG,IAAIiD,MAAM,CAAChD,OAAO,EAAE,CAAC;IACnF,CAAC,CAAC,OAAOmB,KAAU,EAAE;MACnB3F,MAAM,CAAC2F,KAAK,CAAC,SAAS,EAAE,mBAAmB,EAAEA,KAAK,CAAC;MACnD9C,aAAa,CAAC;QACZ0B,OAAO,EAAE,KAAK;QACdC,OAAO,EAAE,UAAUmB,KAAK,CAACnB,OAAO,EAAE;QAClCC,KAAK,EAAE;MACT,CAAC,CAAC;IACJ,CAAC,SAAS;MACRhC,aAAa,CAAC,KAAK,CAAC;MACpBE,aAAa,CAAC,EAAE,CAAC;IACnB;EACF,CAAC,EAAE,CAACgC,OAAO,EAAE5D,MAAM,EAAEK,QAAQ,CAAC,CAAC;;EAE/B;;EAEA,oBACEX,KAAA,CAACZ,YAAY,CAACwI,QAAQ;IAACC,KAAK,EAAE3D,OAAQ;IAAAxD,QAAA,gBACpCZ,IAAA,CAACd,IAAI;MAAC8I,GAAG,EAAEhG,WAAY;MAACiG,KAAK,EAAEC,MAAM,CAACC,IAAK;MAACC,WAAW,EAAE,KAAM;MAAAxH,QAAA,EAC5DA;IAAQ,CACL,CAAC,eAGPZ,IAAA,CAACR,YAAY;MAAC6I,OAAO,EAAEpG,UAAW;MAACE,UAAU,EAAEA;IAAW,CAAE,CAAC,EAG5DxB,WAAW,iBACVX,IAAA,CAACT,YAAY;MACX+I,MAAM,EAAEZ,UAAW;MACnBzF,UAAU,EAAEA,UAAW;MACvBI,UAAU,EAAEA,UAAW;MACvB5B,QAAQ,EAAEA,QAAS;MACnB8H,SAAS,EAAEA,CAAA,KAAMjG,aAAa,CAAC,IAAI,CAAE;MACrCc,cAAc,EAAEA,cAAe;MAC/Bb,IAAI,EAAEA,IAAK;MACXiG,YAAY,EAAGC,OAAO,IAAK;QACzBhJ,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gBAAgBQ,IAAI,MAAMkG,OAAO,EAAE,CAAC;QAC3DjG,OAAO,CAACiG,OAAO,CAAC;MAClB,CAAE;MACFhG,WAAW,EAAEA,WAAY;MACzBE,cAAc,EAAEA,cAAe;MAC/BE,YAAY,EAAEA,YAAa;MAC3B6F,aAAa,EAAEjB,gBAAiB;MAChC1E,gBAAgB,EAAEA,gBAAiB;MACnC4F,WAAW,EAAGC,MAAM,IAAK;QACvB,IAAIA,MAAM,IAAI,CAAC7F,gBAAgB,EAAE;UAC/BtD,MAAM,CAACmG,IAAI,CAAC,SAAS,EAAE,oDAAoD,CAAC;UAC5E;QACF;QACAnG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAe6G,MAAM,GAAG,IAAI,GAAG,KAAK,EAAE,CAAC;QAC9DlG,cAAc,CAACkG,MAAM,CAAC;QACtB,IAAIA,MAAM,EAAE;UACVnJ,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wBAAwB,CAAC;UAChDmB,aAAa,CAACa,OAAO,EAAEqC,KAAK,CAAC,CAAC,CAACd,IAAI,CAAEC,EAAE,IAAK;YAC1C9F,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4BwD,EAAE,EAAE,CAAC;UAC1D,CAAC,CAAC;QACJ,CAAC,MAAM;UACL9F,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wBAAwB,CAAC;UAChDmB,aAAa,CAACa,OAAO,EAAEyD,IAAI,CAAC,CAAC;QAC/B;MACF,CAAE;MACFqB,eAAe,EAAGC,KAAK,IAAK;QAC1BrJ,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,mBAAmB+G,KAAK,GAAG,OAAO,GAAG,SAAS,EAAE,CAAC;QACxElG,iBAAiB,CAACkG,KAAK,CAAC;QACxB,IAAIA,KAAK,EAAE;UACT3F,cAAc,CAACY,OAAO,EAAEgF,IAAI,CAAC,CAAC;QAChC,CAAC,MAAM;UACL5F,cAAc,CAACY,OAAO,EAAEiF,MAAM,CAAC,CAAC;QAClC;MACF;IAAE,CAEH,CACF;EAAA,CACoB,CAAC;AAE5B;AAGA,MAAMd,MAAM,GAAG/I,UAAU,CAAC8J,MAAM,CAAC;EAC/Bd,IAAI,EAAE;IACJe,IAAI,EAAE;EACR;AACF,CAAC,CAAC","ignoreList":[]}
1
+ {"version":3,"names":["React","useCallback","useEffect","useMemo","useRef","useState","View","StyleSheet","AgentRuntime","GeminiProvider","AgentContext","AgentChatBar","AgentOverlay","logger","buildVoiceSystemPrompt","MCPBridge","VoiceService","AudioInputService","AudioOutputService","jsx","_jsx","jsxs","_jsxs","console","log","AIAgent","apiKey","model","navRef","language","maxSteps","showChatBar","children","onResult","interactiveBlacklist","interactiveWhitelist","onBeforeStep","onAfterStep","onBeforeTask","onAfterTask","transformScreenContent","customTools","instructions","stepDelay","mcpServerUrl","router","pathname","enableVoice","onTokenUsage","debug","setEnabled","info","rootViewRef","isThinking","setIsThinking","statusText","setStatusText","lastResult","setLastResult","mode","setMode","isMicActive","setIsMicActive","isSpeakerMuted","setIsSpeakerMuted","isAISpeaking","setIsAISpeaking","isVoiceConnected","setIsVoiceConnected","voiceServiceRef","audioInputRef","audioOutputRef","toolLockRef","userHasSpokenRef","lastScreenContextRef","screenPollIntervalRef","availableModes","modes","push","join","askUserResolverRef","config","ask_user","onStatusUpdate","onAskUser","undefined","question","Promise","resolve","current","success","message","steps","provider","runtime","updateRefs","bridge","destroy","audioOutputInitPromise","runtimeTools","getTools","length","map","t","name","voicePrompt","system","systemPrompt","tools","onError","err","error","initialize","then","ok","onAudioChunk","chunk","isConnected","sendAudio","onPermissionDenied","warn","connect","onAudioResponse","audio","enqueue","onStatusChange","status","connected","start","intentionalDisconnect","stop","setTimeout","lastCallbacks","onTranscript","text","isFinal","role","onSetupComplete","initialContext","getScreenContext","passiveContext","sendScreenContext","onToolCall","toolCall","JSON","stringify","args","id","sendFunctionResponse","result","executeTool","updatedContext","enrichedResult","onTurnComplete","SCREEN_POLL_INTERVAL","MIN_DIFF_RATIO","setInterval","currentContext","lastLen","diff","Math","abs","diffRatio","toFixed","passiveUpdate","clearInterval","disconnect","stopVoiceSession","handleSend","trim","resolver","execute","Provider","value","ref","style","styles","root","collapsable","visible","onSend","onDismiss","onModeChange","newMode","onStopSession","onMicToggle","active","onSpeakerToggle","muted","mute","unmute","create","flex"],"sourceRoot":"../../../src","sources":["components/AIAgent.tsx"],"mappings":";;AAAA;AACA;AACA;AACA;AACA;AACA;AACA;AACA;AACA;;AAEA,OAAOA,KAAK,IACVC,WAAW,EACXC,SAAS,EACTC,OAAO,EACPC,MAAM,EACNC,QAAQ,QACH,OAAO;AACd,SAASC,IAAI,EAAEC,UAAU,QAAQ,cAAc;AAC/C,SAASC,YAAY,QAAQ,yBAAsB;AACnD,SAASC,cAAc,QAAQ,gCAA6B;AAC5D,SAASC,YAAY,QAAQ,uBAAoB;AACjD,SAASC,YAAY,QAAQ,mBAAgB;AAC7C,SAASC,YAAY,QAAQ,mBAAgB;AAC7C,SAASC,MAAM,QAAQ,oBAAiB;AACxC,SAASC,sBAAsB,QAAQ,yBAAsB;AAC7D,SAASC,SAAS,QAAQ,sBAAmB;AAC7C,SAASC,YAAY,QAAQ,6BAA0B;AACvD,SAASC,iBAAiB,QAAQ,kCAA+B;AACjE,SAASC,kBAAkB,QAAQ,mCAAgC;AAAC,SAAAC,GAAA,IAAAC,IAAA,EAAAC,IAAA,IAAAC,KAAA;AAGpE;AACAC,OAAO,CAACC,GAAG,CAAC,8BAA8B,CAAC;;AAE3C;;AA+DA;;AAEA,OAAO,SAASC,OAAOA,CAAC;EACtBC,MAAM;EACNC,KAAK,GAAG,kBAAkB;EAC1BC,MAAM;EACNC,QAAQ,GAAG,IAAI;EACfC,QAAQ,GAAG,EAAE;EACbC,WAAW,GAAG,IAAI;EAClBC,QAAQ;EACRC,QAAQ;EACR;EACAC,oBAAoB;EACpBC,oBAAoB;EACpBC,YAAY;EACZC,WAAW;EACXC,YAAY;EACZC,WAAW;EACXC,sBAAsB;EACtBC,WAAW;EACXC,YAAY;EACZC,SAAS;EACTC,YAAY;EACZC,MAAM;EACNC,QAAQ;EACRC,WAAW,GAAG,KAAK;EACnBC,YAAY;EACZC,KAAK,GAAG;AACI,CAAC,EAAE;EACf;EACAjD,KAAK,CAACE,SAAS,CAAC,MAAM;IACpBqB,OAAO,CAACC,GAAG,CAAC,wBAAwB,EAAEyB,KAAK,EAAE,mBAAmB,CAAC;IACjEpC,MAAM,CAACqC,UAAU,CAACD,KAAK,CAAC;IACxB,IAAIA,KAAK,EAAE;MACTpC,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,0BAA0B,CAAC;IACpD;EACF,CAAC,EAAE,CAACF,KAAK,CAAC,CAAC;EAEX,MAAMG,WAAW,GAAGhD,MAAM,CAAM,IAAI,CAAC;EACrC,MAAM,CAACiD,UAAU,EAAEC,aAAa,CAAC,GAAGjD,QAAQ,CAAC,KAAK,CAAC;EACnD,MAAM,CAACkD,UAAU,EAAEC,aAAa,CAAC,GAAGnD,QAAQ,CAAC,EAAE,CAAC;EAChD,MAAM,CAACoD,UAAU,EAAEC,aAAa,CAAC,GAAGrD,QAAQ,CAAyB,IAAI,CAAC;;EAE1E;EACA,MAAM,CAACsD,IAAI,EAAEC,OAAO,CAAC,GAAGvD,QAAQ,CAAY,MAAM,CAAC;EACnD,MAAM,CAACwD,WAAW,EAAEC,cAAc,CAAC,GAAGzD,QAAQ,CAAC,KAAK,CAAC;EACrD,MAAM,CAAC0D,cAAc,EAAEC,iBAAiB,CAAC,GAAG3D,QAAQ,CAAC,KAAK,CAAC;EAC3D,MAAM,CAAC4D,YAAY,EAAEC,eAAe,CAAC,GAAG7D,QAAQ,CAAC,KAAK,CAAC;EACvD,MAAM,CAAC8D,gBAAgB,EAAEC,mBAAmB,CAAC,GAAG/D,QAAQ,CAAC,KAAK,CAAC;EAE/D,MAAMgE,eAAe,GAAGjE,MAAM,CAAsB,IAAI,CAAC;EACzD,MAAMkE,aAAa,GAAGlE,MAAM,CAA2B,IAAI,CAAC;EAC5D,MAAMmE,cAAc,GAAGnE,MAAM,CAA4B,IAAI,CAAC;EAC9D,MAAMoE,WAAW,GAAGpE,MAAM,CAAU,KAAK,CAAC;EAC1C,MAAMqE,gBAAgB,GAAGrE,MAAM,CAAU,KAAK,CAAC;EAC/C,MAAMsE,oBAAoB,GAAGtE,MAAM,CAAS,EAAE,CAAC;EAC/C,MAAMuE,qBAAqB,GAAGvE,MAAM,CAAwC,IAAI,CAAC;;EAEjF;EACA,MAAMwE,cAA2B,GAAGzE,OAAO,CAAC,MAAM;IAChD,MAAM0E,KAAkB,GAAG,CAAC,MAAM,CAAC;IACnC,IAAI9B,WAAW,EAAE8B,KAAK,CAACC,IAAI,CAAC,OAAO,CAAC;IACpCjE,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoB0B,KAAK,CAACE,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;IAC9D,OAAOF,KAAK;EACd,CAAC,EAAE,CAAC9B,WAAW,CAAC,CAAC;;EAEjB;EACA,MAAMiC,kBAAkB,GAAG5E,MAAM,CAAoC,IAAI,CAAC;;EAE1E;;EAEA,MAAM6E,MAAmB,GAAG9E,OAAO,CAAC,OAAO;IACzCuB,MAAM;IACNC,KAAK;IACLE,QAAQ;IACRC,QAAQ;IACRI,oBAAoB;IACpBC,oBAAoB;IACpBC,YAAY;IACZC,WAAW;IACXC,YAAY;IACZC,WAAW;IACXE,WAAW,EAAEkB,IAAI,KAAK,OAAO,GAAG;MAAE,GAAGlB,WAAW;MAAEyC,QAAQ,EAAE;IAAK,CAAC,GAAGzC,WAAW;IAChFC,YAAY;IACZC,SAAS;IACTC,YAAY;IACZC,MAAM;IACNC,QAAQ;IACRqC,cAAc,EAAE3B,aAAa;IAC7BR,YAAY;IACZ;IACAoC,SAAS,EAAEzB,IAAI,KAAK,OAAO,GAAG0B,SAAS,GAAKC,QAAgB,IAAK;MAC/D,OAAO,IAAIC,OAAO,CAAUC,OAAO,IAAK;QACtCR,kBAAkB,CAACS,OAAO,GAAGD,OAAO;QACpC;QACA9B,aAAa,CAAC;UAAEgC,OAAO,EAAE,IAAI;UAAEC,OAAO,EAAE,KAAKL,QAAQ,EAAE;UAAEM,KAAK,EAAE;QAAG,CAAC,CAAC;QACrEtC,aAAa,CAAC,KAAK,CAAC;QACpBE,aAAa,CAAC,EAAE,CAAC;MACnB,CAAC,CAAC;IACJ;EACF,CAAC,CAAC,EAAE,CACFG,IAAI,EAAEjC,MAAM,EAAEC,KAAK,EAAEE,QAAQ,EAAEC,QAAQ,EACvCI,oBAAoB,EAAEC,oBAAoB,EAC1CC,YAAY,EAAEC,WAAW,EAAEC,YAAY,EAAEC,WAAW,EACpDC,sBAAsB,EAAEC,WAAW,EAAEC,YAAY,EAAEC,SAAS,EAC5DC,YAAY,EAAEC,MAAM,EAAEC,QAAQ,EAAEE,YAAY,CAC7C,CAAC;EAEF,MAAM6C,QAAQ,GAAG1F,OAAO,CAAC,MAAM,IAAIM,cAAc,CAACiB,MAAM,EAAEC,KAAK,CAAC,EAAE,CAACD,MAAM,EAAEC,KAAK,CAAC,CAAC;EAElF,MAAMmE,OAAO,GAAG3F,OAAO,CACrB,MAAM,IAAIK,YAAY,CAACqF,QAAQ,EAAEZ,MAAM,EAAE7B,WAAW,CAACqC,OAAO,EAAE7D,MAAM,CAAC;EACrE;EACA,CAACiE,QAAQ,EAAEZ,MAAM,CACnB,CAAC;;EAED;EACA/E,SAAS,CAAC,MAAM;IACd4F,OAAO,CAACC,UAAU,CAAC3C,WAAW,CAACqC,OAAO,EAAE7D,MAAM,CAAC;EACjD,CAAC,EAAE,CAACkE,OAAO,EAAElE,MAAM,CAAC,CAAC;;EAErB;;EAEA1B,SAAS,CAAC,MAAM;IACd,IAAI,CAAC0C,YAAY,EAAE;IAEnB/B,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4BP,YAAY,EAAE,CAAC;IAClE,MAAMoD,MAAM,GAAG,IAAIjF,SAAS,CAAC6B,YAAY,EAAEkD,OAAO,CAAC;IAEnD,OAAO,MAAM;MACXE,MAAM,CAACC,OAAO,CAAC,CAAC;IAClB,CAAC;EACH,CAAC,EAAE,CAACrD,YAAY,EAAEkD,OAAO,CAAC,CAAC;;EAE3B;;EAEA;EACA5F,SAAS,CAAC,MAAM;IACd,IAAIyD,IAAI,KAAK,MAAM,EAAE;MACnB9C,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,yCAAyC,CAAC;MACjE;IACF;IAEAtC,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoBQ,IAAI,oCAAoC,CAAC;;IAEpF;IACA,IAAIuC,sBAAqC,GAAGX,OAAO,CAACC,OAAO,CAAC,CAAC;;IAE7D;IACA,IAAI,CAACnB,eAAe,CAACoB,OAAO,EAAE;MAC5B5E,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,0BAA0B,CAAC;MAClD,MAAMgD,YAAY,GAAGL,OAAO,CAACM,QAAQ,CAAC,CAAC;MACvCvF,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAegD,YAAY,CAACE,MAAM,6BAA6BF,YAAY,CAACG,GAAG,CAACC,CAAC,IAAIA,CAAC,CAACC,IAAI,CAAC,CAACzB,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;MACjI;MACA;MACA,MAAM0B,WAAW,GAAG3F,sBAAsB,CAACe,QAAQ,EAAEa,YAAY,EAAEgE,MAAM,CAAC;MAC1E7F,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,2BAA2BsD,WAAW,CAACJ,MAAM,aAAaI,WAAW,EAAE,CAAC;MAC/FpC,eAAe,CAACoB,OAAO,GAAG,IAAIzE,YAAY,CAAC;QACzCU,MAAM;QACNiF,YAAY,EAAEF,WAAW;QACzBG,KAAK,EAAET,YAAY;QACnBtE;MACF,CAAC,CAAC;MACFhB,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,6BAA6BgD,YAAY,CAACE,MAAM,WAAWF,YAAY,CAACG,GAAG,CAACC,CAAC,IAAIA,CAAC,CAACC,IAAI,CAAC,CAACzB,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;IAC/H;;IAEA;IACA,IAAI,CAACR,cAAc,CAACkB,OAAO,EAAE;MAC3B5E,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gCAAgC,CAAC;MACxDoB,cAAc,CAACkB,OAAO,GAAG,IAAIvE,kBAAkB,CAAC;QAC9C2F,OAAO,EAAGC,GAAG,IAAKjG,MAAM,CAACkG,KAAK,CAAC,SAAS,EAAE,sBAAsBD,GAAG,EAAE;MACvE,CAAC,CAAC;MACF;MACA;MACA;MACA;MACAZ,sBAAsB,GAAG3B,cAAc,CAACkB,OAAO,CAACuB,UAAU,CAAC,CAAC,CAACC,IAAI,CAAEC,EAAE,IAAK;QACxErG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,mCAAmC+D,EAAE,EAAE,CAAC;MACjE,CAAC,CAAC;IACJ;;IAEA;IACA,IAAI,CAAC5C,aAAa,CAACmB,OAAO,EAAE;MAC1B5E,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,+BAA+B,CAAC;MACvDmB,aAAa,CAACmB,OAAO,GAAG,IAAIxE,iBAAiB,CAAC;QAC5C;QACAkG,YAAY,EAAGC,KAAK,IAAK;UACvBvG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oBAAoBiE,KAAK,CAACf,MAAM,wBAAwB,CAAC,CAAChC,eAAe,CAACoB,OAAO,eAAepB,eAAe,CAACoB,OAAO,EAAE4B,WAAW,EAAE,CAAC;UAC9JhD,eAAe,CAACoB,OAAO,EAAE6B,SAAS,CAACF,KAAK,CAAC;QAC3C,CAAC;QACDP,OAAO,EAAGC,GAAG,IAAKjG,MAAM,CAACkG,KAAK,CAAC,SAAS,EAAE,qBAAqBD,GAAG,EAAE,CAAC;QACrES,kBAAkB,EAAEA,CAAA,KAAM1G,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,+BAA+B;MAClF,CAAC,CAAC;IACJ;;IAEA;IACA3G,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4B,CAAC;IACpD,KAAKkB,eAAe,CAACoB,OAAO,CAACgC,OAAO,CAAC;MACnCC,eAAe,EAAGC,KAAK,IAAK;QAC1B9G,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,sBAAsBwE,KAAK,CAACtB,MAAM,0BAA0B,CAAC,CAAC9B,cAAc,CAACkB,OAAO,EAAE,CAAC;QAC9GvB,eAAe,CAAC,IAAI,CAAC;QACrB,IAAI,CAACK,cAAc,CAACkB,OAAO,EAAE;UAC3B5E,MAAM,CAACkG,KAAK,CAAC,SAAS,EAAE,uDAAuD,CAAC;UAChF;QACF;QACAxC,cAAc,CAACkB,OAAO,CAACmC,OAAO,CAACD,KAAK,CAAC;MACvC,CAAC;MACDE,cAAc,EAAGC,MAAM,IAAK;QAC1BjH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,iBAAiB2E,MAAM,EAAE,CAAC;QACjD,MAAMC,SAAS,GAAGD,MAAM,KAAK,WAAW;QACxC1D,mBAAmB,CAAC2D,SAAS,CAAC;QAC9B,IAAIA,SAAS,EAAE;UACblH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,kFAAkF,CAAC;UAC1G;UACA;UACA;UACA+C,sBAAsB,CAACe,IAAI,CAAC,MAAM;YAChCpG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,6CAA6C,CAAC;YACrEmB,aAAa,CAACmB,OAAO,EAAEuC,KAAK,CAAC,CAAC,CAACf,IAAI,CAAEC,EAAE,IAAK;cAC1C,IAAIA,EAAE,EAAE;gBACNpD,cAAc,CAAC,IAAI,CAAC;gBACpBjD,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,uCAAuC,CAAC;cACjE;YACF,CAAC,CAAC;UACJ,CAAC,CAAC;QACJ;;QAEA;QACA,IACE2E,MAAM,KAAK,cAAc,IACzBnE,IAAI,KAAK,OAAO,IAChBU,eAAe,CAACoB,OAAO,IACvB,CAACpB,eAAe,CAACoB,OAAO,CAACwC,qBAAqB,EAC9C;UACA;UACA3D,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;UAC7B3D,cAAc,CAACkB,OAAO,EAAEyC,IAAI,CAAC,CAAC;UAC9BpE,cAAc,CAAC,KAAK,CAAC;UACrBI,eAAe,CAAC,KAAK,CAAC;UAEtBrD,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,0EAA0E,CAAC;;UAElG;UACAW,UAAU,CAAC,MAAM;YACf,IACExE,IAAI,KAAK,OAAO,IAChBU,eAAe,CAACoB,OAAO,IACvB,CAACpB,eAAe,CAACoB,OAAO,CAACwC,qBAAqB,IAC9C5D,eAAe,CAACoB,OAAO,CAAC2C,aAAa,EACrC;cACAvH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,iCAAiC,CAAC;cACzD,KAAKkB,eAAe,CAACoB,OAAO,CAACgC,OAAO,CAACpD,eAAe,CAACoB,OAAO,CAAC2C,aAAa,CAAC;YAC7E;UACF,CAAC,EAAE,IAAI,CAAC;QACV;MACF,CAAC;MACDC,YAAY,EAAEA,CAACC,IAAI,EAAEC,OAAO,EAAEC,IAAI,KAAK;QACrC3H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAeqF,IAAI,YAAYD,OAAO,OAAOD,IAAI,GAAG,CAAC;QAC5E,IAAIE,IAAI,KAAK,MAAM,EAAE;UACnB/D,gBAAgB,CAACgB,OAAO,GAAG,IAAI;QACjC;MACF,CAAC;MACDgD,eAAe,EAAEA,CAAA,KAAM;QACrB5H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,2DAA2D,CAAC;QACnFsB,gBAAgB,CAACgB,OAAO,GAAG,KAAK;QAChC,MAAMiD,cAAc,GAAG5C,OAAO,CAAC6C,gBAAgB,CAAC,CAAC;QACjDjE,oBAAoB,CAACe,OAAO,GAAGiD,cAAc,CAAC,CAAC;QAC/C,MAAME,cAAc,GAAG,+OAA+OF,cAAc,EAAE;QACtR7H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,8BAA8ByF,cAAc,CAACvC,MAAM,aAAauC,cAAc,EAAE,CAAC;QACxGvE,eAAe,CAACoB,OAAO,EAAEoD,iBAAiB,CAACD,cAAc,CAAC;QAC1D/H,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,oDAAoD,CAAC;MAC9E,CAAC;MACD2F,UAAU,EAAE,MAAOC,QAAQ,IAAK;QAC9BlI,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,uBAAuB4F,QAAQ,CAACvC,IAAI,IAAIwC,IAAI,CAACC,SAAS,CAACF,QAAQ,CAACG,IAAI,CAAC,SAASH,QAAQ,CAACI,EAAE,GAAG,CAAC;;QAEpH;QACA;QACA,IAAI,CAAC1E,gBAAgB,CAACgB,OAAO,EAAE;UAC7B5E,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,yBAAyBuB,QAAQ,CAACvC,IAAI,2BAA2B,CAAC;UACzFnC,eAAe,CAACoB,OAAO,EAAE2D,oBAAoB,CAACL,QAAQ,CAACvC,IAAI,EAAEuC,QAAQ,CAACI,EAAE,EAAE;YACxEE,MAAM,EAAE;UACV,CAAC,CAAC;UACF;QACF;;QAEA;QACA;QACA;QACA;QACA/E,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;QAC7BrH,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,qCAAqC4F,QAAQ,CAACvC,IAAI,EAAE,CAAC;;QAE5E;QACA,IAAIhC,WAAW,CAACiB,OAAO,EAAE;UACvB5E,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,wEAAwEuB,QAAQ,CAACvC,IAAI,EAAE,CAAC;UAC/G,OAAOhC,WAAW,CAACiB,OAAO,EAAE;YAC1B,MAAM,IAAIF,OAAO,CAACC,OAAO,IAAI2C,UAAU,CAAC3C,OAAO,EAAE,EAAE,CAAC,CAAC;UACvD;QACF;QACAhB,WAAW,CAACiB,OAAO,GAAG,IAAI;QAE1B,IAAI;UACF;UACA,MAAM4D,MAAM,GAAG,MAAMvD,OAAO,CAACwD,WAAW,CAACP,QAAQ,CAACvC,IAAI,EAAEuC,QAAQ,CAACG,IAAI,CAAC;UACtErI,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,sBAAsB4F,QAAQ,CAACvC,IAAI,KAAK6C,MAAM,EAAE,CAAC;;UAExE;UACA,MAAM,IAAI9D,OAAO,CAACC,OAAO,IAAI2C,UAAU,CAAC3C,OAAO,EAAE,GAAG,CAAC,CAAC;;UAEtD;UACA,MAAM+D,cAAc,GAAGzD,OAAO,CAAC6C,gBAAgB,CAAC,CAAC;UACjDjE,oBAAoB,CAACe,OAAO,GAAG8D,cAAc,CAAC,CAAC;UAC/C1I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,mCAAmC4F,QAAQ,CAACvC,IAAI,KAAK+C,cAAc,CAAClD,MAAM,aAAakD,cAAc,EAAE,CAAC;UAC/H,MAAMC,cAAc,GAAG,GAAGH,MAAM,yBAAyBE,cAAc,qBAAqB;UAC5F1I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,8BAA8BqG,cAAc,CAACnD,MAAM,aAAamD,cAAc,EAAE,CAAC;UAExGnF,eAAe,CAACoB,OAAO,EAAE2D,oBAAoB,CAACL,QAAQ,CAACvC,IAAI,EAAEuC,QAAQ,CAACI,EAAE,EAAE;YAAEE,MAAM,EAAEG;UAAe,CAAC,CAAC;UACrG3I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,6BAA6B4F,QAAQ,CAACvC,IAAI,QAAQuC,QAAQ,CAACI,EAAE,GAAG,CAAC;QAC1F,CAAC,SAAS;UACR3E,WAAW,CAACiB,OAAO,GAAG,KAAK;UAC3B;UACA,IAAIpB,eAAe,CAACoB,OAAO,EAAE4B,WAAW,EAAE;YACxC/C,aAAa,CAACmB,OAAO,EAAEuC,KAAK,CAAC,CAAC,CAACf,IAAI,CAAEC,EAAE,IAAK;cAC1C,IAAIA,EAAE,EAAE;gBACNpD,cAAc,CAAC,IAAI,CAAC;gBACpBjD,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wCAAwC4F,QAAQ,CAACvC,IAAI,EAAE,CAAC;cACjF;YACF,CAAC,CAAC;UACJ;QACF;MACF,CAAC;MACDK,OAAO,EAAGC,GAAG,IAAK;QAChBjG,MAAM,CAACkG,KAAK,CAAC,SAAS,EAAE,uBAAuBD,GAAG,EAAE,CAAC;QACrD;QACAxC,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;QAC7B3D,cAAc,CAACkB,OAAO,EAAEyC,IAAI,CAAC,CAAC;QAC9BpE,cAAc,CAAC,KAAK,CAAC;QACrBI,eAAe,CAAC,KAAK,CAAC;MACxB,CAAC;MACDuF,cAAc,EAAEA,CAAA,KAAM;QACpB5I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,kBAAkB,CAAC;QAC1Ce,eAAe,CAAC,KAAK,CAAC;QACtB;QACA;MACF;IACF,CAAC,CAAC;;IAEF;IACA;IACA;IACA,MAAMwF,oBAAoB,GAAG,IAAI;IACjC,MAAMC,cAAc,GAAG,IAAI,CAAC,CAAC;;IAE7BhF,qBAAqB,CAACc,OAAO,GAAGmE,WAAW,CAAC,MAAM;MAChD,IAAI,CAACvF,eAAe,CAACoB,OAAO,EAAE4B,WAAW,EAAE;MAC3C;MACA,IAAI7C,WAAW,CAACiB,OAAO,EAAE;QACvB5E,MAAM,CAACoC,KAAK,CAAC,SAAS,EAAE,2CAA2C,CAAC;QACpE;MACF;MAEA,IAAI;QACF,MAAM4G,cAAc,GAAG/D,OAAO,CAAC6C,gBAAgB,CAAC,CAAC;QACjD,IAAIkB,cAAc,KAAKnF,oBAAoB,CAACe,OAAO,EAAE,OAAO,CAAC;;QAE7D;QACA,MAAMqE,OAAO,GAAGpF,oBAAoB,CAACe,OAAO,CAACY,MAAM;QACnD,MAAM0D,IAAI,GAAGC,IAAI,CAACC,GAAG,CAACJ,cAAc,CAACxD,MAAM,GAAGyD,OAAO,CAAC;QACtD,MAAMI,SAAS,GAAGJ,OAAO,GAAG,CAAC,GAAGC,IAAI,GAAGD,OAAO,GAAG,CAAC;QAElD,IAAII,SAAS,GAAGP,cAAc,EAAE;UAC9B9I,MAAM,CAACoC,KAAK,CAAC,SAAS,EAAE,yCAAyC8G,IAAI,WAAW,CAACG,SAAS,GAAG,GAAG,EAAEC,OAAO,CAAC,CAAC,CAAC,OAAOR,cAAc,GAAG,GAAG,cAAc,CAAC;UACtJ;QACF;QAEA9I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,8BAA8B2G,OAAO,MAAMD,cAAc,CAACxD,MAAM,WAAW,CAAC6D,SAAS,GAAG,GAAG,EAAEC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;QACxIzF,oBAAoB,CAACe,OAAO,GAAGoE,cAAc;QAC7C,MAAMO,aAAa,GAAG,6IAA6IP,cAAc,EAAE;QACnLxF,eAAe,CAACoB,OAAO,EAAEoD,iBAAiB,CAACuB,aAAa,CAAC;QACzDvJ,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,+CAA+C,CAAC;MACzE,CAAC,CAAC,OAAO2D,GAAG,EAAE;QACZjG,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,yBAAyBV,GAAG,EAAE,CAAC;MACxD;IACF,CAAC,EAAE4C,oBAAoB,CAAC;;IAExB;IACA,OAAO,MAAM;MACX7I,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wCAAwCQ,IAAI,SAAS,CAAC;MAC7E;MACA,IAAIgB,qBAAqB,CAACc,OAAO,EAAE;QACjC4E,aAAa,CAAC1F,qBAAqB,CAACc,OAAO,CAAC;QAC5Cd,qBAAqB,CAACc,OAAO,GAAG,IAAI;QACpC5E,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wBAAwB,CAAC;MAClD;MACAuB,oBAAoB,CAACe,OAAO,GAAG,EAAE;MACjCpB,eAAe,CAACoB,OAAO,EAAE6E,UAAU,CAAC,CAAC;MACrCjG,eAAe,CAACoB,OAAO,GAAG,IAAI;MAC9BnB,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;MAC7BpE,cAAc,CAAC,KAAK,CAAC;MACrBI,eAAe,CAAC,KAAK,CAAC;MACtBE,mBAAmB,CAAC,KAAK,CAAC;IAC5B,CAAC;IACD;EACF,CAAC,EAAE,CAACT,IAAI,EAAEjC,MAAM,EAAEoE,OAAO,EAAEjE,QAAQ,EAAEa,YAAY,CAAC,CAAC;;EAEnD;;EAEA,MAAM6H,gBAAgB,GAAGtK,WAAW,CAAC,MAAM;IACzCY,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,6CAA6C,CAAC;IACrE;IACAmB,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;IAC7B;IACA3D,cAAc,CAACkB,OAAO,EAAEyC,IAAI,CAAC,CAAC;IAC9B;IACA7D,eAAe,CAACoB,OAAO,EAAE6E,UAAU,CAAC,CAAC;IACrCjG,eAAe,CAACoB,OAAO,GAAG,IAAI;IAC9B;IACA3B,cAAc,CAAC,KAAK,CAAC;IACrBI,eAAe,CAAC,KAAK,CAAC;IACtBE,mBAAmB,CAAC,KAAK,CAAC;IAC1B;IACAR,OAAO,CAAC,MAAM,CAAC;IACf/C,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gCAAgC,CAAC;EAC1D,CAAC,EAAE,CAAC2C,OAAO,CAAC,CAAC;;EAEb;;EAEA,MAAM0E,UAAU,GAAGvK,WAAW,CAAC,MAAO0F,OAAe,IAAK;IACxD,IAAI,CAACA,OAAO,CAAC8E,IAAI,CAAC,CAAC,EAAE;IAErB5J,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,kBAAkBwC,OAAO,GAAG,CAAC;;IAEpD;IACA,IAAIX,kBAAkB,CAACS,OAAO,EAAE;MAC9B,MAAMiF,QAAQ,GAAG1F,kBAAkB,CAACS,OAAO;MAC3CT,kBAAkB,CAACS,OAAO,GAAG,IAAI;MACjCnC,aAAa,CAAC,IAAI,CAAC;MACnBE,aAAa,CAAC,2BAA2B,CAAC;MAC1CE,aAAa,CAAC,IAAI,CAAC;MACnBgH,QAAQ,CAAC/E,OAAO,CAAC;MACjB;IACF;;IAEA;IACArC,aAAa,CAAC,IAAI,CAAC;IACnBE,aAAa,CAAC,aAAa,CAAC;IAC5BE,aAAa,CAAC,IAAI,CAAC;IAEnB,IAAI;MACF;MACAoC,OAAO,CAACC,UAAU,CAAC3C,WAAW,CAACqC,OAAO,EAAE7D,MAAM,CAAC;MAE/C,MAAMyH,MAAM,GAAG,MAAMvD,OAAO,CAAC6E,OAAO,CAAChF,OAAO,CAAC;MAE7CjC,aAAa,CAAC2F,MAAM,CAAC;MACrBpH,QAAQ,GAAGoH,MAAM,CAAC;MAElBxI,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,WAAWkG,MAAM,CAAC3D,OAAO,GAAG,GAAG,GAAG,GAAG,IAAI2D,MAAM,CAAC1D,OAAO,EAAE,CAAC;IACnF,CAAC,CAAC,OAAOoB,KAAU,EAAE;MACnBlG,MAAM,CAACkG,KAAK,CAAC,SAAS,EAAE,mBAAmB,EAAEA,KAAK,CAAC;MACnDrD,aAAa,CAAC;QACZgC,OAAO,EAAE,KAAK;QACdC,OAAO,EAAE,UAAUoB,KAAK,CAACpB,OAAO,EAAE;QAClCC,KAAK,EAAE;MACT,CAAC,CAAC;IACJ,CAAC,SAAS;MACRtC,aAAa,CAAC,KAAK,CAAC;MACpBE,aAAa,CAAC,EAAE,CAAC;IACnB;EACF,CAAC,EAAE,CAACsC,OAAO,EAAElE,MAAM,EAAEK,QAAQ,CAAC,CAAC;;EAE/B;;EAEA,oBACEX,KAAA,CAACZ,YAAY,CAACkK,QAAQ;IAACC,KAAK,EAAE/E,OAAQ;IAAA9D,QAAA,gBACpCZ,IAAA,CAACd,IAAI;MAACwK,GAAG,EAAE1H,WAAY;MAAC2H,KAAK,EAAEC,MAAM,CAACC,IAAK;MAACC,WAAW,EAAE,KAAM;MAAAlJ,QAAA,EAC5DA;IAAQ,CACL,CAAC,eAGPZ,IAAA,CAACR,YAAY;MAACuK,OAAO,EAAE9H,UAAW;MAACE,UAAU,EAAEA;IAAW,CAAE,CAAC,EAG5DxB,WAAW,iBACVX,IAAA,CAACT,YAAY;MACXyK,MAAM,EAAEZ,UAAW;MACnBnH,UAAU,EAAEA,UAAW;MACvBI,UAAU,EAAEA,UAAW;MACvB5B,QAAQ,EAAEA,QAAS;MACnBwJ,SAAS,EAAEA,CAAA,KAAM3H,aAAa,CAAC,IAAI,CAAE;MACrCkB,cAAc,EAAEA,cAAe;MAC/BjB,IAAI,EAAEA,IAAK;MACX2H,YAAY,EAAGC,OAAO,IAAK;QACzB1K,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,gBAAgBQ,IAAI,MAAM4H,OAAO,EAAE,CAAC;QAC3D3H,OAAO,CAAC2H,OAAO,CAAC;MAClB,CAAE;MACF1H,WAAW,EAAEA,WAAY;MACzBE,cAAc,EAAEA,cAAe;MAC/BE,YAAY,EAAEA,YAAa;MAC3BuH,aAAa,EAAEjB,gBAAiB;MAChCpG,gBAAgB,EAAEA,gBAAiB;MACnCsH,WAAW,EAAGC,MAAM,IAAK;QACvB,IAAIA,MAAM,IAAI,CAACvH,gBAAgB,EAAE;UAC/BtD,MAAM,CAAC2G,IAAI,CAAC,SAAS,EAAE,oDAAoD,CAAC;UAC5E;QACF;QACA3G,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,eAAeuI,MAAM,GAAG,IAAI,GAAG,KAAK,EAAE,CAAC;QAC9D5H,cAAc,CAAC4H,MAAM,CAAC;QACtB,IAAIA,MAAM,EAAE;UACV7K,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wBAAwB,CAAC;UAChDmB,aAAa,CAACmB,OAAO,EAAEuC,KAAK,CAAC,CAAC,CAACf,IAAI,CAAEC,EAAE,IAAK;YAC1CrG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,4BAA4B+D,EAAE,EAAE,CAAC;UAC1D,CAAC,CAAC;QACJ,CAAC,MAAM;UACLrG,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,wBAAwB,CAAC;UAChDmB,aAAa,CAACmB,OAAO,EAAEyC,IAAI,CAAC,CAAC;QAC/B;MACF,CAAE;MACFyD,eAAe,EAAGC,KAAK,IAAK;QAC1B/K,MAAM,CAACsC,IAAI,CAAC,SAAS,EAAE,mBAAmByI,KAAK,GAAG,OAAO,GAAG,SAAS,EAAE,CAAC;QACxE5H,iBAAiB,CAAC4H,KAAK,CAAC;QACxB,IAAIA,KAAK,EAAE;UACTrH,cAAc,CAACkB,OAAO,EAAEoG,IAAI,CAAC,CAAC;QAChC,CAAC,MAAM;UACLtH,cAAc,CAACkB,OAAO,EAAEqG,MAAM,CAAC,CAAC;QAClC;MACF;IAAE,CAEH,CACF;EAAA,CACoB,CAAC;AAE5B;AAGA,MAAMd,MAAM,GAAGzK,UAAU,CAACwL,MAAM,CAAC;EAC/Bd,IAAI,EAAE;IACJe,IAAI,EAAE;EACR;AACF,CAAC,CAAC","ignoreList":[]}