preflite 1.1.0 → 1.1.3

package/README.md

@@ -1,39 +1,36 @@
 # Preflight
 
-> **Give your coding AI a real phone to test on.**
+Local MCP server for testing mobile apps on real Android, iOS, and HarmonyOS devices with AI coding assistants.
 
-Preflight is an MCP (Model Context Protocol) server that connects AI coding assistants (Claude Code, Cursor, Codex) to real Android, iOS, and HarmonyOS devices for interactive testing — entirely on your local machine.
+[![npm version](https://img.shields.io/npm/v/preflite.svg)](https://www.npmjs.com/package/preflite)
+[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](./LICENSE)
+[![Release](https://github.com/zifengjiang/Preflight/actions/workflows/release.yml/badge.svg)](https://github.com/zifengjiang/Preflight/actions/workflows/release.yml)
+
+Preflight gives Claude Code, Cursor, Codex, and other MCP clients a real mobile device to inspect, operate, and test. It turns natural-language testing requests into structured visual-flow runs with live viewing, screenshots, and saved reports.
 
 <p align="center">
-  <img src="docs/demo.gif" alt="Preflight in action" width="720">
+  <img src="docs/demo.gif" alt="Preflight running a mobile test through an AI coding assistant" width="720">
 </p>
 
-## Setup
+## Why Preflight?
 
-### Prerequisites
+- Test on real devices instead of mocked browser views or screenshots.
+- Let coding agents inspect screens, tap, type, swipe, wait, and assert UI state.
+- Generate visual-flow tests without fragile XPath selectors, accessibility-id assumptions, or fixed coordinates.
+- Keep the runtime local: devices, reports, configuration, and the MCP server stay on your machine.
+- Use the same workflow across Android, iOS, and HarmonyOS.
 
-| Dependency | Required for | Notes |
-|-----------|-------------|-------|
-| **Node.js** ≥ 20.11 | All platforms | Install via [nvm](https://github.com/nvm-sh/nvm) or [nodejs.org](https://nodejs.org/) |
-| **adb** | Android | Ship with Android SDK / [platform-tools](https://developer.android.com/studio/releases/platform-tools). Ensure `adb` is on your `PATH`, or set `ADB_BINARY_PATH` in config. |
-| **Xcode** + **iproxy** | iOS | Xcode from [Mac App Store](https://apps.apple.com/app/xcode/id497799835). `iproxy` ships with `brew install libimobiledevice`. |
-| **WebDriverAgent** | iOS | [Facebook WebDriverAgent](https://github.com/facebook/WebDriverAgent) — build and deploy to your device from Xcode. Set `WDA_PROJECT_ROOT` in config to point at your local copy. |
-| **hdc** | HarmonyOS | Ships with [DevEco Studio](https://developer.huawei.com/consumer/cn/deveco-studio/). Ensure `hdc` is on your `PATH`. |
-| **AI model API key** | All platforms | A [Midscene](https://midscenejs.com/)-compatible vision model. Supports OpenAI, Anthropic, Doubao, and more. |
+## Quick Start
 
-### 1. Install
+Install the local runtime and register the MCP server:
 
 ```bash
-npx preflite setup
+npx -y preflite@latest setup
 ```
 
-This downloads the runtime, compiles all assets, and registers the `Preflight` MCP server in your editor's config (`.cursor/mcp.json` or `~/.codex/config.toml`).
+This installs the packaged runtime under `~/.preflight/runtime`, writes a Preflight MCP entry for Codex (`~/.codex/config.toml`), and writes a Cursor project entry when you run it inside a repository (`.cursor/mcp.json`).
 
-> Alternatively, clone the repo and run `npm run mcp:setup` if you want to hack on Preflight itself.
-
-### 2. Configure your AI model
-
-Create `~/.preflight/config.json`:
+Create `~/.preflight/config.json` with a Midscene-compatible vision model:
 
 ```json
 {
@@ -41,24 +38,50 @@ Create `~/.preflight/config.json`:
     "MIDSCENE_MODEL_BASE_URL": "https://ark.cn-beijing.volces.com/api/v3",
     "MIDSCENE_MODEL_API_KEY": "sk-xxxxxxxxxxxxxxxx",
     "MIDSCENE_MODEL_NAME": "doubao-seed-2-0-lite-260215",
-    "MIDSCENE_MODEL_FAMILY": "doubao-seed"
+    "MIDSCENE_MODEL_FAMILY": "doubao-seed",
+    "MIDSCENE_MODEL_REASONING_ENABLED": "false"
   }
 }
 ```
 
-**Supported providers** — any of these works:
+Restart your AI coding assistant, then ask it to use Preflight:
+
+> Check my connected devices and run a smoke test on the app.
 
-| Provider | `MIDSCENE_MODEL_BASE_URL` | `MIDSCENE_MODEL_NAME` |
-|----------|--------------------------|-----------------------|
+## What You Can Ask
+
+Once the MCP server is available, your coding agent can work with real devices through requests like:
+
+- "List my connected Android and iOS devices."
+- "Open the app and verify the login flow."
+- "Explore the settings screen and turn it into a visual-flow test."
+- "Install this APK and run a smoke test."
+- "Run the flow, watch it live, and save the report."
+
+## Requirements
+
+| Dependency | Required for | Notes |
+|-----------|-------------|-------|
+| Node.js >= 20.11 | All platforms | Install via [nvm](https://github.com/nvm-sh/nvm) or [nodejs.org](https://nodejs.org/). |
+| AI model API key | All platforms | Any [Midscene](https://midscenejs.com/)-compatible vision model. |
+| adb | Android | Ships with Android SDK [platform-tools](https://developer.android.com/studio/releases/platform-tools). Ensure `adb` is on your `PATH`, or set `ADB_BINARY_PATH`. |
+| Xcode + iproxy | iOS | Xcode from the [Mac App Store](https://apps.apple.com/app/xcode/id497799835). `iproxy` ships with `brew install libimobiledevice`. |
+| WebDriverAgent | iOS | Build and deploy [WebDriverAgent](https://github.com/facebook/WebDriverAgent) to your device, then set `WDA_PROJECT_ROOT`. |
+| hdc | HarmonyOS | Ships with [DevEco Studio](https://developer.huawei.com/consumer/cn/deveco-studio/). Ensure `hdc` is on your `PATH`. |
+
+## Model Configuration
+
+Preflight loads model configuration from `~/.preflight/config.json`, `~/.preflight/config.yaml`, or `~/.preflight/config.yml`. You can also rely on standard provider environment variables such as `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`.
+
+| Provider | `MIDSCENE_MODEL_BASE_URL` | Example `MIDSCENE_MODEL_NAME` |
+|----------|--------------------------|-------------------------------|
 | OpenAI | `https://api.openai.com/v1` | `gpt-4o` |
 | Anthropic | `https://api.anthropic.com/v1` | `claude-sonnet-4-20250514` |
-| Doubao (Volcengine) | `https://ark.cn-beijing.volces.com/api/v3` | `doubao-seed-2-0-lite-260215` |
-
-You can also set standard env vars (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) — Preflight picks them up automatically.
+| Doubao / Volcengine | `https://ark.cn-beijing.volces.com/api/v3` | `doubao-seed-2-0-lite-260215` |
 
-### (Optional) iOS WebDriverAgent
+## iOS Setup
 
-iOS testing requires [WebDriverAgent](https://github.com/facebook/WebDriverAgent) running on your device:
+iOS testing requires WebDriverAgent running on your device:
 
 ```bash
 git clone https://github.com/facebook/WebDriverAgent.git
@@ -67,7 +90,7 @@ cd WebDriverAgent
 open WebDriverAgent.xcodeproj
 ```
 
-Build the `WebDriverAgentRunner` scheme targeting your device. Then add to `~/.preflight/config.json`:
+Build the `WebDriverAgentRunner` scheme targeting your device. Then add the project path to `~/.preflight/config.json`:
 
 ```json
 {
@@ -78,38 +101,28 @@ Build the `WebDriverAgentRunner` scheme targeting your device. Then add to `~/.p
 }
 ```
 
-### 4. Restart and test
-
-Restart your AI coding assistant. Now you can say things like:
-
-> *"Check my devices, explore the Notes app, and write a test for creating a new note."*
-
-The AI calls `list_devices` → `exploration_start` → explores your app naturally → generates and runs a visual-flow test on the real device.
-
-## How It Uses Midscene
-
-[**Midscene**](https://midscenejs.com/) is the visual AI engine that powers Preflight's ability to see and act on mobile screens. It works by feeding screenshots and structured instructions to a multimodal LLM, then parsing the model's response into concrete UI actions.
+## How It Works
 
-Preflight wraps Midscene in three layers:
+Preflight connects your AI assistant to local device automation through MCP. The assistant does not need to write platform-specific automation scripts; it asks Preflight for device state, explores screens, validates visual-flow JSON, and runs the flow through the local runtime.
 
 ```mermaid
 flowchart LR
-    subgraph AI["Your AI Assistant<br/>(Claude Code / Cursor)"]
-        MCP["MCP Protocol\nstdio"]
+    subgraph AI["AI Coding Assistant"]
+        MCP["MCP Client"]
     end
 
     subgraph Preflight["Preflight MCP Server"]
-        EXP["Exploration Layer\napp exploration &\nscreen understanding"]
-        IR["Visual Flow IR\nstructured test\nstep definitions"]
-        AGENT["automation-agent\ndevice lifecycle &\ntask orchestration"]
+        EXP["Exploration Tools"]
+        IR["Visual Flow IR"]
+        AGENT["Local Runtime"]
     end
 
     subgraph Midscene["Midscene SDK"]
-        MS["Midscene Agent\nscreenshot → LLM →\nUI actions"]
+        MS["Vision Agent"]
     end
 
-    subgraph Device["Mobile Device"]
-        SCREEN["Screen & UI\nElement Tree"]
+    subgraph Device["Real Device"]
+        SCREEN["Screen and UI Tree"]
     end
 
     AI <-->|stdio| MCP
@@ -122,22 +135,60 @@ flowchart LR
     SCREEN -.->|screenshots| MS
 ```
 
-**How the layers work together:**
+The main pieces are:
 
-1. **Exploration Layer** — Your AI assistant uses natural-language tools (`exploration_get_page_summary`, `exploration_ai_act`) to understand the app, navigate screens, and decide what to test.
-2. **Visual Flow IR** — Once the flow is understood, it's captured as a structured JSON — a sequence of steps with assertions, free of fragile selectors or coordinates.
-3. **Midscene SDK** — Each step is compiled into a Midscene plan: the SDK takes a screenshot, sends it with instructions to the vision model, and performs the returned action (tap, type, swipe, assert) on the device via adb/WDA/hdc.
+1. **Exploration tools** help the assistant understand the current screen and choose what to test.
+2. **Visual Flow IR** captures a test as structured JSON with steps, assertions, and app context.
+3. **Midscene SDK** converts high-level visual instructions into device actions such as tap, type, swipe, wait, and assert.
 
-This design means tests survive UI reshuffles (Midscene targets *what* to interact with, not *where*), and the AI never needs to write fragile XPath or accessibility-id selectors.
-
-## Tools Overview
+## MCP Tools
 
 | Category | Tools |
 |----------|-------|
-| **Agent** | `agent_health` · `start_agent` · `stop_agent` · `doctor` · `config_status` |
-| **Device** | `list_devices` · `install_app` |
-| **Exploration** | `exploration_start` · `exploration_end` · `exploration_get_page_summary` · `exploration_ai_act` · `exploration_ask_about_screen` · `exploration_screenshot` · `exploration_type` · `exploration_wait` |
-| **Visual-Flow** | `get_visual_flow_ir_rules` · `validate_visual_flow` · `run_flow` · `watch_run` · `cancel_run` · `save_report` · `read_report` |
+| Agent | `agent_health` · `start_agent` · `stop_agent` · `doctor` · `config_status` |
+| Device | `list_devices` · `install_app` |
+| Exploration | `exploration_start` · `exploration_end` · `exploration_get_page_summary` · `exploration_ai_act` · `exploration_ask_about_screen` · `exploration_screenshot` · `exploration_type` · `exploration_wait` |
+| Visual Flow | `get_visual_flow_ir_rules` · `validate_visual_flow` · `run_flow` · `watch_run` · `cancel_run` · `save_report` · `read_report` |
+
+## Reports
+
+Runs write report assets under:
+
+```text
+~/.preflight/midscene_run/report/<reportName>/
+```
+
+A report can include the HTML summary, execution JSON, screenshots, and compressed recordings when the platform recorder and `ffmpeg` are available.
+
+## Development
+
+Clone the repository when you want to work on Preflight itself:
+
+```bash
+git clone https://github.com/zifengjiang/Preflight.git
+cd Preflight
+npm install
+npm test
+npm run check
+npm run build
+```
+
+Register the local development MCP server:
+
+```bash
+npm run mcp:setup
+```
+
+## Release
+
+Releases are triggered by version tags:
+
+```bash
+git tag v1.1.1
+git push origin v1.1.1
+```
+
+The release workflow publishes the npm package and creates a GitHub Release.
 
 ## License
 

package/dist/infrastructure/midscene/MidsceneRuntimeReal.js

@@ -1,4 +1,4 @@
-import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
+import { mkdir, readFile, readdir, stat, writeFile } from "node:fs/promises";
 import net from "node:net";
 import { tmpdir } from "node:os";
 import path from "node:path";
@@ -542,9 +542,47 @@ export class MidsceneRuntimeReal {
                 console.warn(`[MidsceneRuntimeReal] report artifact missing or unreadable: ${err instanceof Error ? err.message : String(err)}`);
             }
         }
+        let stepFailureMessage;
+        if (reportPaths && result.ok) {
+            const execDir = reportPaths.bundleDir ?? path.dirname(reportPaths.reportHtmlPath);
+            try {
+                const dirEntries = await readdir(execDir);
+                for (const name of dirEntries) {
+                    if (!/^\d+\.execution\.json$/.test(name))
+                        continue;
+                    try {
+                        const raw = await readFile(path.join(execDir, name), "utf8");
+                        const parsed = JSON.parse(raw);
+                        const executions = parsed.executions;
+                        if (!Array.isArray(executions))
+                            continue;
+                        for (const exec of executions) {
+                            if (!exec || typeof exec !== "object")
+                                continue;
+                            const tasks = exec.tasks;
+                            if (!Array.isArray(tasks))
+                                continue;
+                            for (const task of tasks) {
+                                if (task && typeof task === "object" && task.status === "failed") {
+                                    const msg = task.errorMessage;
+                                    stepFailureMessage = typeof msg === "string" && msg.trim() ? msg.trim() : "midscene step execution failed";
+                                }
+                            }
+                        }
+                    }
+                    catch {
+                        // skip unparseable files
+                    }
+                }
+            }
+            catch {
+                // directory not found or unreadable
+            }
+        }
+        const ok = result.ok && stepFailureMessage === undefined;
         return {
-            ok: result.ok,
-            message: result.ok ? "midscene real execution success" : result.stderr || "midscene execution failed",
+            ok,
+            message: stepFailureMessage ?? (result.ok ? "midscene real execution success" : result.stderr || "midscene execution failed"),
             artifacts,
             reportInfo,
         };

package/dist/mcp/agentHttpClient.js

@@ -64,13 +64,26 @@ export class AgentHttpClient {
         const headers = new Headers(init.headers);
         if (this.config.token)
             headers.set("Authorization", `Bearer ${this.config.token}`);
-        const resp = await fetch(url, { ...init, headers });
-        const text = await resp.text();
-        const body = text ? JSON.parse(text) : null;
-        if (!resp.ok) {
-            throw new AgentHttpError(resp.status, body && typeof body === "object" ? JSON.stringify(body) : text);
+        const controller = new AbortController();
+        const timeout = setTimeout(() => controller.abort(), 8_000);
+        try {
+            const resp = await fetch(url, { ...init, headers, signal: controller.signal });
+            const text = await resp.text();
+            const body = text ? JSON.parse(text) : null;
+            if (!resp.ok) {
+                throw new AgentHttpError(resp.status, body && typeof body === "object" ? JSON.stringify(body) : text);
+            }
+            return body;
+        }
+        catch (error) {
+            if (controller.signal.aborted) {
+                throw new AgentHttpError(408, `Agent HTTP request timed out after 8000ms: ${url.pathname}`);
+            }
+            throw error;
+        }
+        finally {
+            clearTimeout(timeout);
         }
-        return body;
     }
 }
 export class AgentHttpError extends Error {

package/dist/mcp/cli.js

@@ -1,5 +1,8 @@
 #!/usr/bin/env node
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { access } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
 import { createPreflightMcpServer } from "./server.js";
 import { setupLocalMcp } from "./setup.js";
 // Prevent unhandled rejections / exceptions from crashing the server mid-session
@@ -17,11 +20,12 @@ if (command === "serve") {
 }
 else if (command === "setup") {
     const projectRoot = argValue("--project-root") ?? process.cwd();
+    const runtimeSourceRoot = argValue("--runtime-source-root") ?? await findPackageRoot();
     const agentBaseUrl = argValue("--agent-base-url") ?? process.env.AGENT_BASE_URL ?? "http://127.0.0.1:18998";
     const livePort = Number(argValue("--live-port") ?? process.env.MCP_LIVE_PORT ?? "18999");
     const runtimeRoot = argValue("--runtime-root") ?? process.env.AGENT_RUNTIME_ROOT?.trim() ?? undefined;
     const installRuntime = !process.argv.includes("--no-install-runtime");
-    const result = await setupLocalMcp({ projectRoot, agentBaseUrl, livePort, runtimeRoot, installRuntime });
+    const result = await setupLocalMcp({ projectRoot, runtimeSourceRoot, agentBaseUrl, livePort, runtimeRoot, installRuntime });
     console.log(JSON.stringify({ ok: true, ...result }, null, 2));
 }
 else {
@@ -34,3 +38,25 @@ function argValue(name) {
         return undefined;
     return process.argv[idx + 1];
 }
+async function findPackageRoot() {
+    let current = dirname(fileURLToPath(import.meta.url));
+    while (true) {
+        if (await exists(join(current, "package.json"))) {
+            return current;
+        }
+        const parent = dirname(current);
+        if (parent === current) {
+            throw new Error("Could not locate the preflite package root.");
+        }
+        current = parent;
+    }
+}
+async function exists(path) {
+    try {
+        await access(path);
+        return true;
+    }
+    catch {
+        return false;
+    }
+}

package/dist/mcp/exploration/tools-intelligent.js

@@ -58,17 +58,60 @@ async function detectForegroundApp(session) {
     return null;
 }
 // ---------------------------------------------------------------------------
+// Safe aiAsk: uses aiQuery under the hood to avoid doubao key-mismatch errors
+// ---------------------------------------------------------------------------
+async function aiAskSafe(session, prompt) {
+    const result = await session.agent.aiQuery(`${prompt}, string`);
+    if (typeof result === "string")
+        return result;
+    if (result && typeof result === "object") {
+        const values = Object.values(result);
+        const firstString = values.find((v) => typeof v === "string");
+        if (firstString)
+            return firstString;
+        return JSON.stringify(result);
+    }
+    return String(result);
+}
+// ---------------------------------------------------------------------------
+// Retry wrapper for transient Midscene model errors
+// ---------------------------------------------------------------------------
+const RETRY_MAX = 3;
+const RETRY_DELAY_MS = 2000;
+async function withRetry(fn, label) {
+    let lastError;
+    for (let attempt = 1; attempt <= RETRY_MAX; attempt++) {
+        try {
+            return await fn();
+        }
+        catch (err) {
+            lastError = err;
+            const msg = err instanceof Error ? err.message : String(err);
+            const retryable = msg.includes("No result in query data") ||
+                msg.includes("BadRequestError") ||
+                msg.includes("RateLimitError") ||
+                msg.includes("ServiceUnavailableError") ||
+                msg.includes("InternalServerError");
+            if (!retryable || attempt === RETRY_MAX)
+                throw err;
+            // Midscene model偶发错误，短暂延迟后重试
+            await new Promise((r) => setTimeout(r, RETRY_DELAY_MS * attempt));
+        }
+    }
+    throw lastError;
+}
+// ---------------------------------------------------------------------------
 // Tool handlers
 // ---------------------------------------------------------------------------
 export function getPageSummaryHandler(ctx) {
     return async (input) => {
         const session = await resolveSession(input.sessionId, ctx);
-        const summary = await session.agent.aiAsk("详细描述当前页面。请按从上到下的顺序列出所有可见区域、交互元素和文案。\n" +
+        const summary = await withRetry(() => aiAskSafe(session, "详细描述当前页面。请按从上到下的顺序列出所有可见区域、交互元素和文案。\n" +
             "特别注意：\n" +
             "1) 页面底部是否有更多内容（是否可滚动）？如果底部紧贴导航栏/状态栏则说明是固定单屏布局\n" +
             "2) 是否有弹窗、广告或遮挡物？\n" +
             "3) 整体布局类型：固定单屏 / 可滚动长页面 / 多Tab / 列表\n" +
-            "先判断布局类型，再逐一描述每个区域的内容。");
+            "先判断布局类型，再逐一描述每个区域的内容。"), "getPageSummary");
         const state = getSession(input.sessionId);
         const foregroundApp = await detectForegroundApp(session);
         const appRef = state.appRef;
@@ -88,7 +131,7 @@ export function getPageSummaryHandler(ctx) {
 export function askAboutScreenHandler(ctx) {
     return async (input) => {
         const session = await resolveSession(input.sessionId, ctx);
-        const answer = await session.agent.aiAsk(input.question);
+        const answer = await withRetry(() => aiAskSafe(session, input.question), "askAboutScreen");
         return { answer };
     };
 }
@@ -97,16 +140,16 @@ export function aiActHandler(ctx) {
         const session = await resolveSession(input.sessionId, ctx);
         const state = getSession(input.sessionId);
         // Before-state: reuse from get_page_summary if available, otherwise grab a quick one
-        const beforeSummary = state.lastPageSummary ?? await session.agent.aiAsk("用一句话描述当前页面的最关键特征：什么类型的页面（列表/表单/弹窗/首页等），最显著的内容是什么。");
-        await session.agent.aiAct(input.intent);
-        const afterSummary = await session.agent.aiAsk(`操作前的页面：${beforeSummary}\n` +
+        const beforeSummary = state.lastPageSummary ?? await withRetry(() => aiAskSafe(session, "用一句话描述当前页面的最关键特征：什么类型的页面（列表/表单/弹窗/首页等），最显著的内容是什么。"), "aiAct.beforeSummary");
+        await withRetry(() => session.agent.aiAct(input.intent), "aiAct");
+        const afterSummary = await withRetry(() => aiAskSafe(session, `操作前的页面：${beforeSummary}\n` +
             `执行的操作：${input.intent}\n` +
             "请判断操作结果：\n" +
             "1) 页面内容是否发生了变化？（进入了新页面、弹出弹窗、滚动到底部、输入框获得焦点等）\n" +
             "2) 如果操作是滑动，是否已到达底部或页面没有变化？\n" +
             "3) 当前页面布局类型是固定单屏还是可滚动长页面？\n" +
             "4) 当前页面最关键的变化是什么？\n" +
-            "如果操作没有产生任何实际变化（比如反复滑动但没有新内容），请明确指出\"页面没有变化\"。");
+            "如果操作没有产生任何实际变化（比如反复滑动但没有新内容），请明确指出\"页面没有变化\"。"), "aiAct.afterSummary");
         // Update for next aiAct call
         state.lastPageSummary = afterSummary;
         const foregroundApp = await detectForegroundApp(session);

package/dist/mcp/exploration/tools-session.js

@@ -64,6 +64,7 @@ export async function ensureIosWdaStarted(resourceId, runtimeEnv, projectRoot) {
         const child = spawn("/bin/bash", [scriptPath, udid, String(port)], {
             stdio: ["ignore", "pipe", "pipe"],
             timeout: 120_000,
+            env: { ...process.env, ...runtimeEnv },
         });
         let stdout = "";
         let stderr = "";