@mobileai/react-native 0.4.0 → 0.4.1

package/README.md

@@ -16,7 +16,10 @@ Wrap your navigation with `<AIAgent>`. The AI automatically understands your ent
 
 ## ✨ Features
 
-- 🤖 **Zero-config UI understanding** — No need to annotate your views. The AI reads your UI automatically.
+- 🤖 **Zero-config UI understanding** — No annotations, no accessibility labels. The AI reads your entire React Fiber tree automatically.
+- 🎯 **Universal component support** — Works with every React Native component (Pressable, Switch, TextInput, custom components) without modification.
+- 🖼️ **Media inference** — Detects images and videos on screen, captures their URIs, and infers context from parent component names — enabling Gemini's vision capabilities.
+- 📐 **Compact DOM output** — Collapses structural view nesting into flat, token-efficient output. Your 50-level deep navigation bar becomes a single line.
 - 🧭 **Auto-navigation** — Navigates between screens to complete multi-step tasks.
 - 🔒 **Production-grade security** — Element gating, content masking, lifecycle hooks, human-in-the-loop confirmation.
 - 🧩 **Custom actions** — Expose any business logic (checkout, API calls) as AI-callable tools with `useAction`.
@@ -55,6 +58,33 @@ export default function App() {
 
 A floating chat bar appears automatically. Ask the AI to navigate, tap buttons, fill forms — it reads your live UI and acts.
 
+## 🧠 How It Works
+
+The SDK traverses your **React Fiber tree** at runtime and generates a compact, LLM-friendly representation of everything on screen:
+
+```
+Account Settings
+[image in="SettingsScreen" src="https://randomuser.me/api/portraits/men/32.jpg"]
+Demo User
+demo@test.com
+Profile Information
+Full Name
+[0]<text-input>Enter your name />
+Email Address
+[1]<text-input>Enter your email />
+Notifications
+Email Alerts
+[2]<switch value="true"> />
+Push Notifications
+[3]<switch value="false"> />
+```
+
+**Key design decisions:**
+- **Structural views are transparent** — React Navigation's 50+ nested `<View>` layers are collapsed. Only interactive elements and text content appear.
+- **Media is context-aware** — Images emit `[image in="ComponentName" src="url"]`, giving the LLM both *where* the image is and the full URL for Gemini's vision model.
+- **State is explicit** — Switch values, check states, and input values are emitted as attributes so the LLM knows the current state.
+- **Polymorphic tap** — `tap(index)` works on any component: calls `onPress` for buttons, `onValueChange` for switches, and bubbles up the Fiber tree for custom wrapper components.
+
 ## 🔌 API Reference
 
 ### `<AIAgent>` Component
@@ -244,10 +274,10 @@ This starts two servers:
 
 | Tool | Description |
 |------|-------------|
-| `tap(index)` | Tap an interactive element. |
-| `type(index, text)` | Type text into an input. |
+| `tap(index)` | Tap any interactive element. Polymorphic: triggers `onPress` for buttons, `onValueChange` for switches, bubbles up Fiber tree for wrapper components. |
+| `type(index, text)` | Type text into a text-input. |
 | `navigate(screen)` | Navigate to a screen. |
-| `done(text)` | Complete the task. |
+| `done(text)` | Complete the task with a response. |
 | `ask_user(question)` | Ask the user for clarification. |
 
 ## 📋 Requirements

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mobileai/react-native",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "Build autonomous AI agents for React Native and Expo apps. Provides AI-native UI traversal, tool calling, and structured reasoning.",
   "main": "./lib/module/index.js",
   "source": "./src/index.ts",