@loadmill/droid-cua 1.1.2 → 2.0.0

package/README.md

@@ -7,267 +7,141 @@
 <p align="center">
   <a href="#what-is-droid-cua">What is droid-cua?</a> •
   <a href="#quick-start">Quick Start</a> •
+  <a href="#platforms">Platforms</a> •
   <a href="#features">Features</a> •
-  <a href="#usage">Usage</a> •
-  <a href="#assertions">Assertions</a> •
-  <a href="#command-line-options">Command Line Options</a> •
   <a href="#how-it-works">How It Works</a> •
+  <a href="#cli-and-automation">CLI & Automation</a> •
   <a href="#license">License</a>
 </p>
 
 ---
 
-**AI-powered mobile testing using OpenAI's computer-use model**
+**AI-powered mobile testing desktop app for Android and iOS**
 
-Create and run automated Android and iOS tests using natural language. The AI explores your app and generates executable test scripts.
+Create, run, and manage mobile tests with natural language. The desktop app guides setup, connects to your target device or simulator, and turns AI exploration into reusable test scripts.
 
-https://github.com/user-attachments/assets/e6450f45-3700-4cb6-aad5-33ba5f0437c3
+https://github.com/user-attachments/assets/b9e15a1d-8072-4a2f-a4c5-db180ae38620
 
 ---
 
 <h2 id="what-is-droid-cua">💡 What is droid-cua?</h2>
 
-`droid-cua` gives you three core components for mobile testing:
+`droid-cua` is a desktop app for AI-powered mobile testing.
 
-* **Interactive Shell** – Design and run tests with real-time feedback and visual status indicators
-* **Test Scripts** – Simple text files with natural language instructions and assertions
-* **AI Agent** – Autonomous exploration powered by OpenAI's computer-use model
+It helps teams create, edit, and run tests for **Android devices and emulators** and **iOS simulators on macOS** using natural language instead of traditional test code.
 
-Together, these let you create and execute Android and iOS tests without writing traditional test code.
+Under the hood, `droid-cua` uses an AI agent to explore your app, execute actions, and save reusable test scripts that can also be run later in headless workflows.
 
 ---
 
 <h2 id="quick-start">🚀 Quick Start</h2>
 
-**1. Install**
+**1. Download the desktop app**
 
-Globally (recommended):
-```sh
-npm install -g @loadmill/droid-cua
-```
-
-Or from source:
-```sh
-git clone https://github.com/loadmill/droid-cua
-cd droid-cua
-npm install
-npm run build
-```
-
-**2. Set your OpenAI API key**
-
-Using environment variable:
-```sh
-export OPENAI_API_KEY=your-api-key
-```
-
-Or create a `.env` file:
-```sh
-echo "OPENAI_API_KEY=your-api-key" > .env
-```
-
-**3. Setup for your platform**
-
-For Android:
-```sh
-adb version  # Ensure ADB is available
-```
+Get the latest stable desktop build from [GitHub Releases](https://github.com/loadmill/droid-cua/releases).
 
-For iOS (macOS only):
-```sh
-# Install Appium and XCUITest driver
-npm install -g appium
-appium driver install xcuitest
-```
-
-**4. Run**
-
-```sh
-droid-cua
-```
+**2. Launch the app**
 
-An interactive menu will let you select your platform and device:
+Open the desktop app and choose the platform you want to test.
 
-```
-┌──────────────────────────────────────┐
-│ Select Platform                      │
-└──────────────────────────────────────┘
+**3. Add your credentials**
 
-❯ Android (1 running) - 2 emulator(s)
-  iOS - 5 simulator(s)
+Set your OpenAI API key in the app Settings screen, or log in with your Loadmill account.
 
-↑/↓ Navigate  Enter Select  q Quit
-```
+**4. Connect a target device**
 
-The emulator/simulator will auto-launch if not already running.
+- Android: connect a device or select an emulator
+- iOS: choose a simulator on macOS
 
----
+**5. Create or run a test**
 
-<h2 id="features">✨ Features</h2>
+Use the desktop app to create a new test, edit an existing one, or run a saved script with live execution logs.
 
-- **Design Mode** - Describe what to test, AI explores and creates test scripts
-- **Execution Mode** - Run tests with real-time feedback and assertion handling
-- **Headless Mode** - Run tests in CI/CD pipelines
-- **Test Management** - Create, edit, view, and run test scripts
-- **Smart Actions** - Automatic wait detection and coordinate mapping
+You can also keep project run history in a results folder and review past runs from desktop app reports.
 
 ---
 
-<h2 id="usage">📚 Usage</h2>
-
-### Interactive Commands
-
-| Command | Description |
-|---------|-------------|
-| `/create <name>` | Create a new test |
-| `/run <name>` | Execute a test |
-| `/list` | List all tests |
-| `/view <name>` | View test contents |
-| `/edit <name>` | Edit a test |
-| `/help` | Show help |
-| `/exit` | Exit shell |
-
-### Creating Tests
-
-```sh
-droid-cua
-> /create login-test
-> Test the login flow with valid credentials
-```
-
-The AI will explore your app and generate a test script. Review and save it.
-
-### Running Tests
-
-Interactive:
-```sh
-droid-cua
-> /run login-test
-```
-
-Headless (CI/CD):
-```sh
-droid-cua --instructions tests/login-test.dcua
-```
+<h2 id="platforms">📱 Platforms</h2>
 
-### Test Script Format
-
-One instruction per line:
-
-```
-Open the Calculator app
-assert: Calculator app is visible
-Type "2"
-Click the plus button
-Type "3"
-Click the equals button
-assert: result shows 5
-exit
-```
-
-<h3 id="assertions">Assertions</h3>
-
-Assertions validate the app state during test execution. Add them anywhere in your test script.
-
-**Syntax** (all valid):
-```
-assert: the login button is visible
-Assert: error message appears
-ASSERT the result shows 5
-```
-
-**Interactive Mode** - When an assertion fails:
-- `retry` - Retry the same assertion
-- `skip` - Continue to next instruction
-- `stop` - Stop test execution
-
-**Headless Mode** - Assertions fail immediately and exit with code 1.
-
-**Examples**:
-```
-assert: Calculator app is open
-assert: the result shows 8
-assert: error message is displayed in red
-assert: login button is enabled
-```
-
----
-
-<h2 id="command-line-options">💻 Command Line Options</h2>
-
-| Option | Description |
-|--------|-------------|
-| `--avd=NAME` | Specify emulator/simulator name |
-| `--platform=PLATFORM` | Force platform: `android` or `ios` |
-| `--instructions=FILE` | Run test headless |
-| `--record` | Save screenshots |
-| `--debug` | Enable debug logs |
-
-**Examples:**
-```sh
-# Interactive device selection
-droid-cua
-
-# Android emulator
-droid-cua --avd Pixel_8_API_35
-
-# iOS Simulator (auto-detected from name)
-droid-cua --avd "iPhone 16"
-
-# iOS Simulator (explicit platform)
-droid-cua --platform ios --avd "iPhone 16"
-
-# Headless CI mode
-droid-cua --avd "iPhone 16" --instructions tests/login.dcua
-```
-
----
+- **Android** - Physical devices and emulators
+- **iOS** - Simulators on macOS
 
 ## Requirements
 
 **All platforms:**
-- Node.js 18.17.0+
-- OpenAI API Key (Tier 3 for computer-use-preview model)
+- OpenAI API key
 
 **Android:**
 - Android Debug Bridge (ADB)
-- Android Emulator (AVD)
+- Android Emulator CLI for launchable emulators
 
 **iOS (macOS only):**
 - Xcode with iOS Simulator
-- Appium (`npm install -g appium`)
-- XCUITest driver (`appium driver install xcuitest`)
+- Appium
+- XCUITest driver
+
+---
+
+<h2 id="features">✨ Features</h2>
+
+- **Desktop-first workflow** - Create, run, and manage tests from one app
+- **Setup guidance** - Configure API access and platform prerequisites in the app
+- **Device and simulator connection** - Connect Android targets and iOS simulators
+- **Natural-language test creation** - Describe flows in plain English
+- **Test management** - Create, edit, save, and rerun reusable scripts
+- **Live execution logs** - Watch actions and progress as tests run
+- **Reports and history** - Review past runs from a project results folder inside the desktop app
+- **JUnit XML output** - Write standard test reports for CI systems and external tooling
+- **Headless support** - Reuse scripts in CLI and automation workflows
 
 ---
 
 <h2 id="how-it-works">🔧 How It Works</h2>
 
-1. Connects to Android emulator (via ADB) or iOS Simulator (via Appium)
+1. The desktop app connects to an Android device or emulator through ADB, or to an iOS simulator through Appium + XCUITest
 2. Captures full-screen device screenshots
 3. Scales down the screenshots for OpenAI model compatibility
-4. Sends screenshots and user instructions to OpenAI's computer-use-preview model
-5. Receives structured actions (click, scroll, type, keypress, wait, drag)
+4. Sends screenshots and user instructions to OpenAI's computer-use model
+5. Receives structured actions such as click, scroll, type, keypress, wait, and drag
 6. Rescales model outputs back to real device coordinates
-7. Executes the actions on the device
+7. Executes the actions on the device or simulator
 8. Validates assertions and handles failures
 9. Repeats until task completion
 
 ---
 
-## 🎞️ Convert Screenshots to Video
+<h2 id="cli-and-automation">💻 CLI & Automation</h2>
 
-If you run with `--record`, screenshots are saved to:
-```
-droid-cua-recording-<timestamp>/
+The desktop app is the primary way to use `droid-cua`.
+
+For CI, scripting, or advanced workflows, `droid-cua` also includes a CLI for running saved instructions headlessly.
+
+Desktop projects can also keep run reports in a results folder, including JUnit XML output that the app can read back as project history.
+
+Install:
+```sh
+npm install -g @loadmill/droid-cua
 ```
 
-Convert to video with ffmpeg:
+Examples:
 ```sh
-ffmpeg -framerate 1 -pattern_type glob -i 'droid-cua-recording-*/frame_*.png' \
-  -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2" \
-  -c:v libx264 -pix_fmt yuv420p session.mp4
+# Interactive CLI
+droid-cua
+
+# Headless Android run
+droid-cua --avd adb:emulator-5554 --instructions tests/login.dcua
+
+# Headless iOS simulator run
+droid-cua --platform ios --avd "iPhone 16" --instructions tests/login.dcua
 ```
 
+Supported CLI options include:
+- `--avd`
+- `--platform`
+- `--instructions`
+- `--record`
+- `--debug`
+
 ---
 
 <h2 id="license">📄 License</h2>

package/build/index.js

@@ -1,4 +1,5 @@
 import minimist from "minimist";
+import dotenv from "dotenv";
 import path from "path";
 import { mkdir, readFile } from "fs/promises";
 import { connectToDevice, getDeviceInfo } from "./src/device/connection.js";
@@ -9,6 +10,7 @@ import { startInkShell } from "./src/cli/ink-shell.js";
 import { ExecutionMode } from "./src/modes/execution-mode.js";
 import { logger } from "./src/utils/logger.js";
 import { selectDevice } from "./src/cli/device-selector.js";
+dotenv.config();
 const args = minimist(process.argv.slice(2));
 let avdName = args["avd"];
 let platform = args["platform"] || null; // 'ios' or 'android'

package/build/src/cli/app.js

@@ -5,13 +5,27 @@ import { OutputPanel } from './components/OutputPanel.js';
 import { InputPanel } from './components/InputPanel.js';
 import { AgentStatus } from './components/AgentStatus.js';
 import { CommandSuggestions } from './components/CommandSuggestions.js';
-import { COMMANDS } from './command-parser.js';
+import { COMMANDS, getCommandSuggestions } from './command-parser.js';
+import { listTests } from '../test-store/test-manager.js';
+/**
+ * @typedef {Object} CliExecutionOutputItem
+ * @property {string} [type]
+ * @property {string} [text]
+ * @property {string} [eventType]
+ * @property {string} [actionType]
+ * @property {string} [runId]
+ * @property {string} [stepId]
+ * @property {number} [instructionIndex]
+ * @property {Record<string, unknown>} [payload]
+ * @property {unknown} [metadata]
+ */
 /**
  * Main Ink App component - conversational split-pane UI
  */
 export function App({ session, initialMode = 'command', onInput, onExit }) {
     const [mode, setMode] = useState(initialMode);
     const [testName, setTestName] = useState(null);
+    /** @type {[CliExecutionOutputItem[], React.Dispatch<React.SetStateAction<CliExecutionOutputItem[]>>]} */
     const [output, setOutput] = useState([]);
     const [agentWorking, setAgentWorking] = useState(false);
     const [agentMessage, setAgentMessage] = useState('');
@@ -25,9 +39,21 @@ export function App({ session, initialMode = 'command', onInput, onExit }) {
     const [commandHistory, setCommandHistory] = useState([]);
     const [historyIndex, setHistoryIndex] = useState(-1);
     const [tempInput, setTempInput] = useState(''); // Store current typing when navigating history
+    const [availableTests, setAvailableTests] = useState([]); // For tab completion
+    // Load available tests for tab completion
+    const refreshTests = async () => {
+        try {
+            const tests = await listTests();
+            setAvailableTests(tests.map(t => t.name));
+        }
+        catch {
+            setAvailableTests([]);
+        }
+    };
     // Context object passed to modes and commands
     const context = {
         // Output methods
+        /** @param {CliExecutionOutputItem} item */
         addOutput: (item) => {
             setOutput((prev) => [...prev, item]);
         },
@@ -69,11 +95,39 @@ export function App({ session, initialMode = 'command', onInput, onExit }) {
                 setInputResolver(() => resolve);
             });
         },
+        // Refresh tests list (for autocomplete)
+        refreshTests,
     };
-    // Handle up/down arrow keys for command history
+    // Handle up/down arrow keys for command history and Tab for autocomplete
     useInput((input, key) => {
         if (inputDisabled)
             return;
+        // Tab key for autocomplete
+        if (key.tab) {
+            if (inputValue.startsWith('/')) {
+                const parts = inputValue.slice(1).split(' ');
+                const commandPart = parts[0].toLowerCase();
+                if (parts.length === 1 && !inputValue.includes(' ')) {
+                    // Autocomplete command name (e.g., /he -> /help)
+                    const matches = getCommandSuggestions(commandPart);
+                    if (matches.length === 1) {
+                        setInputValue(`/${matches[0]} `);
+                    }
+                }
+                else if (parts.length >= 1) {
+                    // Autocomplete test name for /run, /view, /edit
+                    const testCommands = ['run', 'view', 'edit'];
+                    if (testCommands.includes(commandPart)) {
+                        const testPart = parts.slice(1).join(' ').toLowerCase();
+                        const matchingTests = availableTests.filter(t => t.toLowerCase().startsWith(testPart));
+                        if (matchingTests.length === 1) {
+                            setInputValue(`/${commandPart} ${matchingTests[0]}`);
+                        }
+                    }
+                }
+            }
+            return;
+        }
         if (key.upArrow && commandHistory.length > 0) {
             const newIndex = historyIndex === -1
                 ? commandHistory.length - 1
@@ -104,6 +158,9 @@ export function App({ session, initialMode = 'command', onInput, onExit }) {
             global.inkContext = context;
         }
     }, [context]);
+    useEffect(() => {
+        refreshTests();
+    }, []);
     // Show welcome banner on mount
     useEffect(() => {
         const banner = `
@@ -148,5 +205,5 @@ export function App({ session, initialMode = 'command', onInput, onExit }) {
             React.createElement(AgentStatus, { isWorking: agentWorking, message: agentMessage })),
         React.createElement(Box, { flexDirection: "column" },
             React.createElement(InputPanel, { value: inputValue, onChange: setInputValue, onSubmit: handleInput, placeholder: inputPlaceholder, disabled: inputDisabled }),
-            React.createElement(CommandSuggestions, { input: inputValue, commands: COMMANDS }))));
+            React.createElement(CommandSuggestions, { input: inputValue, commands: COMMANDS, tests: availableTests }))));
 }

package/build/src/cli/components/CommandSuggestions.js

@@ -1,9 +1,9 @@
 import React from 'react';
 import { Box, Text } from 'ink';
 /**
- * Command suggestions shown when user types "/"
+ * Command and test suggestions shown when user types "/" or "/run "
  */
-export function CommandSuggestions({ input, commands }) {
+export function CommandSuggestions({ input, commands, tests = [] }) {
     const MAX_VISIBLE = 6;
     const HEADER_LINES = 1;
     const PANEL_HEIGHT = HEADER_LINES + MAX_VISIBLE;
@@ -11,9 +11,47 @@ export function CommandSuggestions({ input, commands }) {
     if (!input.startsWith('/')) {
         return null;
     }
-    // Extract the command part (without the /)
-    const commandPart = input.slice(1).toLowerCase();
-    // Filter commands that match
+    const parts = input.slice(1).split(' ');
+    const commandPart = parts[0].toLowerCase();
+    const hasSpace = input.includes(' ');
+    // Check if we should show test suggestions
+    const testCommands = ['run', 'view', 'edit'];
+    if (hasSpace && testCommands.includes(commandPart)) {
+        const testPart = parts.slice(1).join(' ').toLowerCase();
+        // Filter tests that match
+        const suggestions = tests
+            .filter(t => t.toLowerCase().startsWith(testPart))
+            .sort()
+            .slice(0, MAX_VISIBLE);
+        if (suggestions.length === 0 && testPart.length === 0) {
+            // Show all tests if nothing typed yet
+            const allTests = tests.slice(0, MAX_VISIBLE);
+            const usedLines = HEADER_LINES + allTests.length;
+            const padLines = Math.max(0, PANEL_HEIGHT - usedLines);
+            return (React.createElement(Box, { flexDirection: "column", paddingX: 1, height: PANEL_HEIGHT },
+                React.createElement(Text, { dimColor: true },
+                    "Available tests: ",
+                    React.createElement(Text, { color: "gray" }, "(Tab to complete)")),
+                allTests.map((test) => (React.createElement(Box, { key: test },
+                    React.createElement(Text, { color: "green" },
+                        "  ",
+                        test)))),
+                allTests.length === 0 && (React.createElement(Text, { dimColor: true }, "  No tests found. Use /create to make one.")),
+                Array.from({ length: padLines }).map((_, i) => (React.createElement(Text, { key: `pad-${i}` }, " ")))));
+        }
+        const usedLines = HEADER_LINES + suggestions.length;
+        const padLines = Math.max(0, PANEL_HEIGHT - usedLines);
+        return (React.createElement(Box, { flexDirection: "column", paddingX: 1, height: PANEL_HEIGHT },
+            React.createElement(Text, { dimColor: true },
+                "Matching tests: ",
+                React.createElement(Text, { color: "gray" }, "(Tab to complete)")),
+            suggestions.map((test) => (React.createElement(Box, { key: test },
+                React.createElement(Text, { color: "green" },
+                    "  ",
+                    test)))),
+            Array.from({ length: padLines }).map((_, i) => (React.createElement(Text, { key: `pad-${i}` }, " ")))));
+    }
+    // Show command suggestions
     const suggestions = Object.entries(commands)
         .filter(([cmd]) => cmd.toLowerCase().startsWith(commandPart))
         .sort()
@@ -21,7 +59,9 @@ export function CommandSuggestions({ input, commands }) {
     const usedLines = HEADER_LINES + suggestions.length;
     const padLines = Math.max(0, PANEL_HEIGHT - usedLines);
     return (React.createElement(Box, { flexDirection: "column", paddingX: 1, height: PANEL_HEIGHT },
-        React.createElement(Text, { dimColor: true }, "Available commands:"),
+        React.createElement(Text, { dimColor: true },
+            "Available commands: ",
+            React.createElement(Text, { color: "gray" }, "(Tab to complete)")),
         suggestions.map(([cmd, description]) => (React.createElement(Box, { key: cmd },
             React.createElement(Text, { color: "cyan" },
                 "  /",

package/build/src/cli/components/OutputPanel.js

@@ -1,7 +1,20 @@
 import React from 'react';
 import { Box, Text } from 'ink';
+/**
+ * @typedef {Object} CliExecutionOutputItem
+ * @property {string} [type]
+ * @property {string} [text]
+ * @property {string} [eventType]
+ * @property {string} [actionType]
+ * @property {string} [runId]
+ * @property {string} [stepId]
+ * @property {number} [instructionIndex]
+ * @property {Record<string, unknown>} [payload]
+ * @property {unknown} [metadata]
+ */
 /**
  * Scrollable output panel for agent actions and reasoning
+ * @param {{ items: CliExecutionOutputItem[] }} props
  */
 export function OutputPanel({ items }) {
     if (items.length === 0) {
@@ -10,6 +23,9 @@ export function OutputPanel({ items }) {
     }
     return (React.createElement(Box, { flexDirection: "column", paddingX: 1, paddingY: 1 }, items.map((item, index) => (React.createElement(OutputItem, { key: index, item: item })))));
 }
+/**
+ * @param {{ item: CliExecutionOutputItem }} props
+ */
 function OutputItem({ item }) {
     switch (item.type) {
         case 'reasoning':