@loadmill/droid-cua 2.40.0 → 2.42.0

package/README.md

@@ -97,6 +97,7 @@ Desktop app support:
 - **Desktop-first workflow** - Create, run, and manage tests from one app
 - **Setup guidance** - Configure API access and platform prerequisites in the app
 - **Device and simulator connection** - Connect Android targets and iOS simulators
+- **Network certificate preparation** - Manage one active root CA and install it on a connected local simulator or supported emulator from the desktop app's Tools menu
 - **Natural-language test creation** - Describe flows in plain English
 - **Test management** - Create, edit, save, and rerun reusable scripts, including tests organized in nested project folders
 - **Live execution logs** - Watch actions and progress as tests run
@@ -105,6 +106,22 @@ Desktop app support:
 - **Run video export** - Generate and download a polished MP4 replay from a desktop run's screenshots and instruction timeline
 - **Headless support** - Reuse scripts in CLI and automation workflows
 
+**Network certificate preparation:** Use **Tools → Manage Certificates** to see the active CA, upload a single `.pem` containing a CA certificate and its matching unencrypted private key, generate a new Droid CA, or remove the active CA. Droid stores one active pair in `~/.droid-cua/network/`; upload copies the PEM material without changing the source file, and removal does not change device trust. Connect a local iOS Simulator or a rootable Android Emulator, then choose **Tools → Install Network Certificate**. This installs the active CA, generating a Droid CA if none exists. Devices prepared with a previous CA may need installation again. Droid may reapply Android system trust when a prepared emulator reconnects after a cold boot. Restart an already running Android app after installation so it loads the updated trust store. Android certificate setup supports capable rootable images through Android 16; Google Play images without root access and Android 17 are not supported.
+
+**Network capture:** Enable **Network Capture** in desktop Settings or pass `--capture-network` for CLI tests. Local Chrome and Edge capture through Playwright with no certificate or proxy setup, using the same network queries and report cards as mobile. Capture covers the test browser context, including frames and popups, from run start. Previously loaded traffic is not recovered and existing pages are not automatically reloaded. CLI `--url` navigation happens after capture starts. Cloud browser capture, WebSocket messages and HAR export are not supported.
+
+**Mobile setup:** Manually route a prepared local simulator or emulator to `127.0.0.1:8765` (`10.0.2.2:8765` from Android Emulator). Enable **Network Capture** in desktop Settings to keep the proxy forwarding while Droid is open and automatically capture every local mobile test, including each project or scenario case. Existing **Network Proxy** settings do not turn capture on; opt in to the new setting. For CLI tests, pass `--capture-network`; this starts the proxy and writes one JSONL log per test. `--network-proxy` remains a forwarding-only CLI flag. Droid prints or logs each capture path and a best-effort health summary. Desktop artifacts are in the project's results folder; CLI artifacts are beside `--report` output when enabled, or under `logs/`. A missing CA stops mobile capture before test actions. No observed traffic or failed HTTPS decryption is reported separately from test assertions, but capture health does not prove that all traffic was observed. WebSocket messages are unsupported. Manual proxy setup is the V1 workflow. Droid does not change device or Mac proxy settings; turn off manually configured routing when finished. If Droid exits while routing remains enabled, restart Droid or clear the proxy to restore connectivity.
+
+Captures stop recording at 100 MiB per run, retaining a partial JSONL with a `CAPTURE_LIMIT` error. Traffic continues forwarding; live network queries report incomplete evidence. Captured bodies are limited to 1 MiB, both on the wire and after decompression. Compressed bodies that expand beyond that limit are omitted with `decoded-body-too-large`; this is incomplete evidence, not a missing response. The separate query evidence limit is 1,000,000 serialized characters. A capture limit or missing traffic is not proof that an application request never occurred.
+
+For troubleshooting, “No traffic observed” points to routing or an app that did not send traffic; it does not prove an expected request was absent. HTTPS connections with no decrypted requests point to certificate trust or interception compatibility. A port-in-use error means another Droid instance or proxy already owns the listener. Stop that listener before starting the CLI. A capture-size or write failure retains only partial evidence. iOS network validation is deferred; ordinary iOS runs without capture do not require proxy or certificate preparation.
+
+**Network questions and assertions:** With capture enabled, tests can ask Droid to inspect HTTP traffic and compare values across requests in natural language. For example: “Assert that the POST to `/pageview` after opening Home contains the `item_id` returned by `/home`.” The `query_network` tool searches by URL substring, method, status, flow IDs, request timestamps, or a starting step. It considers all matching exchanges in time order and sends only selected headers/bodies to one side call using the selected run model. Large results require a narrower search; they are not silently truncated. Optional observation windows wait the full duration, up to 30 seconds per query. Absence means only “no matching traffic observed during this window”; unavailable capture or insufficient evidence cannot establish absence. Desktop and HTML reports show one collapsed Network query card per call; expand it to see the question and answer. The agent receives structured flow references and capture diagnostics, while model details stay out of the timeline.
+
+To inspect a saved capture without a model, run `node scripts/query-network.mjs path/to/network.jsonl '{"url_contains":"/pageview","method":"POST","sections":["request_body"]}'` from the repository. Select `request_headers`, `response_headers`, `request_body`, or `response_body`; use `sections: []` for metadata. Saved-log queries accept `after` and `before` timestamps, but not live step IDs or observation waits.
+
+**Developer network capture:** `node scripts/network-capture-smoke.mjs` still offers a standalone proxy checkpoint. Leave desktop **Network Capture** off and omit CLI network flags while it owns `127.0.0.1:8765`. Press Ctrl+C when finished; use `--output path.jsonl` to choose the JSONL path. Request and response headers are written immediately, body records when streams finish or abort, binary bodies are omitted, and captured bodies are limited to 1 MiB on the wire and after decompression. Filtering is partial: JSONL captures may contain credentials, tokens, cookies, or other sensitive data. Treat them as private run artifacts.
+
 ---
 
 <h2 id="how-it-works">🔧 How It Works</h2>
@@ -193,6 +210,9 @@ droid-cua run tests/login.dcua --platform ios --avd "iPhone 16"
 # List running desktop applications and their window IDs (macOS or Windows)
 droid-cua --target desktop --list-desktop-apps
 
+# Control the full primary display across applications
+droid-cua --target desktop --instructions tests/launcher.dcua
+
 # Control one already-running desktop application window
 droid-cua --target desktop --desktop-app com.apple.TextEdit --desktop-window 123 --instructions tests/editor.dcua
 
@@ -276,8 +296,8 @@ Supported CLI options include:
 - `--list-devices`
 - `--browser`
 - `--target` (`mobile`, `web`, or `desktop`)
-- `--desktop-app` (desktop application ID or exact name)
-- `--desktop-window` (required when the application has multiple visible windows)
+- `--desktop-app` (limit desktop control to an application ID or exact name; omit for full desktop)
+- `--desktop-window` (select a window within `--desktop-app`)
 - `--desktop-input` (`physical` by default; use `accessibility` for cursor-free input)
 - `--list-desktop-apps`
 - `--instructions`
@@ -294,6 +314,8 @@ Supported CLI options include:
 - `--execution-prompt-file`
 - `--secrets`
 - `--record`
+- `--network-proxy` (local mobile sessions only; starts idle forwarding at `127.0.0.1:8765`)
+- `--capture-network` (local mobile and Chrome/Edge test runs; writes a JSONL log for each test, using a proxy only for mobile)
 - `--report`
 - `--report-metadata <path>`
 - `--no-cloud-report`
@@ -304,7 +326,15 @@ Supported CLI options include:
 `OPENAI_API_KEY`. Saved direct OpenAI selections of GPT-5.6 Terra, GPT-5.6 Luna,
 or GPT-5.4 move to GPT-6 Sol, GPT-6 Luna, or GPT-6 Sol, respectively.
 
-Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default: Droid shows a system notice for three seconds before taking control, brings the selected window to the foreground, then sends physical pointer and keyboard input at the model's coordinates. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. When the run ends, Droid leaves the target window in front and shows another system notice that control has finished, including when the run fails or is stopped. CLI runs also leave the target in front. Use `--desktop-input accessibility` for cursor-free CLI input instead. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. Windows targets must run at the same integrity level as Droid; a non-elevated Droid process cannot inject input into an elevated application. If Windows returns an all-black window capture, Droid brings the target back to the foreground and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The crop is resized only after extraction to preserve window detail. The fallback logs a warning and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target to the foreground yourself when requested.
+Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use physical pointer and keyboard input by default. Droid shows a system notice for three seconds before taking control and a completion notice when the run finishes, fails, or is stopped. The delay is fixed; use Stop in Droid once the run starts. Avoid using the mouse or keyboard during the run. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. On Windows, Droid must run at the same integrity level as the target application. System notices and automatic minimization apply only to runs started in the desktop app.
+
+CLI runs with `--target desktop` control the full primary display by default, with no application flag needed. In the desktop app, choose **Full desktop**, the first option in the Application dropdown. This mode captures and controls the primary display across applications, including apps launched during the test; no target application needs to be running when connecting. Keep all test windows on the primary display. Full desktop requires physical input. Droid minimizes itself after the three-second notice and remains minimized when the run ends; open Droid to review the result or use Stop. Existing saved application selections are preserved.
+
+Use `--desktop-app <id-or-exact-name>` or select an application in Droid to limit control to that application's windows. Application mode brings the selected window forward and leaves it there when the run ends. Add `--desktop-window <id>` to select a particular window, or `--desktop-input accessibility` for cursor-free CLI input; both options require `--desktop-app`. Before each run and screenshot, Droid refreshes window selection within the original application so it can follow a recreated login window. If replacement is ambiguous, select a window again. If a captured window disappears or moves before input, Droid requests a fresh screenshot instead of transferring its coordinates to a replacement. Focus errors report the activation reason; they do not establish an elevation problem.
+
+If Windows returns an all-black window capture, Droid brings the target forward and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The fallback logs a warning, crops before resizing to preserve detail, and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target forward yourself when requested. Full desktop reports capture failures directly, without this window capture fallback.
+
+Desktop **Test design** uses the same notices once you submit your instructions and exploration starts. Full desktop minimizes Droid at that point. Droid sends the completion notice when the generated test is ready, exploration is stopped, or recovery ends in an error. It does not automatically bring itself forward; open Droid whenever you need to give guidance, use Stop, or review the generated test. Opening Test design alone does not take desktop control. When a controller rejects an action, Droid skips the remaining actions in that batch and asks the model to continue from a fresh screenshot with the failure reason. Rejected actions are not counted as completed.
 
 The system notices are shown only for desktop control runs started in the desktop app. On macOS, they may not appear in the unsigned Electron dev app; verify them in a signed packaged build.
 

package/build/index.js

@@ -3,7 +3,7 @@ import minimist from "minimist";
 import dotenv from "dotenv";
 import path from "path";
 import { mkdir } from "fs/promises";
-import { connectToDevice, disconnect as disconnectDevice, getDeviceInfo } from "./src/device/connection.js";
+import { connectToDevice, disconnect as disconnectDevice, getDeviceInfo, getCurrentPlatform as getConnectedDevicePlatform } from "./src/device/connection.js";
 import { connectToBrowser, disconnect as disconnectBrowser, getBrowserInfo, } from "./src/browser/connection.js";
 import { setBrowserBackendId } from "./src/browser/factory.js";
 import { normalizeBrowserInfoForSession } from "./src/browser/normalize.js";
@@ -45,9 +45,14 @@ import { ARTIFACT_READY_STATUSES } from "./src/device/cloud/loadmill-cloud/run-s
 import { formatDuration } from "./src/core/report/run-log-model.js";
 import { createBrowserSessionTeardown, withLoadmillCloudSession } from "./src/cli/loadmill-cloud-session.js";
 import { createDesktopController, listDesktopApplications } from "./src/desktop/computer-use.js";
+import { createRunNetworkCapture } from "./src/network/run-network-capture.js";
+import { loadCertificateAuthority } from "./src/network/certificate-authority.js";
+import { ensureCliAndroidCertificate, inspectNetworkDevice } from "./src/network/certificate-install.js";
 dotenv.config();
+let networkCapture = null;
 process.once("SIGTERM", async () => {
     process.exitCode = 143;
+    await networkCapture?.shutdown().catch(() => { });
     // Registering a SIGTERM listener cancels Node's default terminate, so exit
     // explicitly; await the /stop because a bare fire-and-forget fetch may not
     // dispatch before exit. Capped so a slow server can't hang the exit.
@@ -386,6 +391,36 @@ async function main() {
     if (!isHeadlessInstructionsRun && args["test-data"] != null) {
         throw new Error("--test-data requires `droid-cua run <path>` or --instructions headless execution.");
     }
+    if (args["network-proxy"] != null && args["network-proxy"] !== true) {
+        throw new Error("--network-proxy is a flag and does not take a value.");
+    }
+    if (args["network-proxy"] === true && (isBrowserTarget || isDesktopTarget || isCloudTarget)) {
+        throw new Error("--network-proxy is available only for local mobile sessions.");
+    }
+    if (args["capture-network"] != null && args["capture-network"] !== true) {
+        throw new Error("--capture-network is a flag and does not take a value.");
+    }
+    if (args["capture-network"] === true && (!isHeadlessInstructionsRun || isDesktopTarget || isCloudTarget)) {
+        throw new Error("--capture-network requires a local mobile or browser test run.");
+    }
+    const captureNetwork = args["capture-network"] === true;
+    if (captureNetwork && !isBrowserTarget)
+        await loadCertificateAuthority();
+    networkCapture = args["network-proxy"] === true || captureNetwork ? createRunNetworkCapture() : null;
+    if (captureNetwork) {
+        process.once("SIGINT", async () => {
+            await networkCapture?.shutdown().catch(() => { });
+            process.exit(130);
+        });
+    }
+    if (networkCapture && !isBrowserTarget) {
+        try {
+            await networkCapture.startProxy();
+        }
+        catch (error) {
+            throw new Error(`Droid network proxy could not start: ${error.message}`, { cause: error });
+        }
+    }
     let preloadedHeadlessConfig = null;
     let preloadedHeadlessTarget = null;
     let headlessInstructionRuns = [];
@@ -520,7 +555,7 @@ async function main() {
     }
     else if (isBrowserTarget) {
         targetId = await connectToBrowser({
-            url: startupUrl,
+            url: captureNetwork ? undefined : startupUrl,
             headless: true,
             browserFamily: targetConfig.browserFamily,
             sessionMode: targetConfig.sessionMode
@@ -534,6 +569,7 @@ async function main() {
     }
     else if (isDesktopTarget) {
         desktopController = await createDesktopController({
+            scope: targetConfig.desktopScope,
             app: targetConfig.desktopApp,
             windowId: targetConfig.desktopWindow,
             inputMode: targetConfig.desktopInput,
@@ -542,8 +578,8 @@ async function main() {
         });
         targetId = desktopController.deviceId;
         deviceInfo = desktopController.deviceInfo;
-        console.log(`Attached to desktop window: ${deviceInfo.device_name}`);
-        console.log(`Model sees window: ${deviceInfo.scaled_width}x${deviceInfo.scaled_height}`);
+        console.log(`Attached to desktop target: ${deviceInfo.device_name}`);
+        console.log(`Model sees desktop target: ${deviceInfo.scaled_width}x${deviceInfo.scaled_height}`);
         console.log(`Desktop input: ${deviceInfo.desktopInputMode === "physical" ? "physical OS pointer" : "cursor-free accessibility"}`);
         process.once("SIGINT", async () => {
             console.log("\nStopping desktop run...");
@@ -766,6 +802,20 @@ async function main() {
     }
     else {
         targetId = await connectToDevice(avdName, platform);
+        if (captureNetwork) {
+            const connectedPlatform = getConnectedDevicePlatform();
+            if (connectedPlatform === "android") {
+                if (!/^emulator-\d+$/.test(targetId))
+                    throw new Error("--capture-network supports Android Emulators, not physical Android devices.");
+                const ca = await loadCertificateAuthority();
+                const trust = await ensureCliAndroidCertificate(targetId, ca);
+                if (trust.changed)
+                    console.log(`Reapplied Droid's network certificate on ${targetId}.`);
+            }
+            else {
+                await inspectNetworkDevice({ platform: "ios", deviceId: targetId });
+            }
+        }
         deviceInfo = await getDeviceInfo(targetId);
         console.log(`Using real resolution: ${deviceInfo.device_width}x${deviceInfo.device_height}`);
         console.log(`Model sees resolution: ${deviceInfo.scaled_width}x${deviceInfo.scaled_height}`);
@@ -836,12 +886,15 @@ async function main() {
                 desktopController,
                 isCloudTarget,
                 runTarget: { isBrowserTarget, isDesktopTarget, isCloudTarget },
-                report: reportConfig
+                report: reportConfig,
+                networkCapture: captureNetwork ? networkCapture : null,
+                initialBrowserUrl: captureNetwork && isBrowserTarget ? startupUrl : null
             });
             exitCode = result.exitCode;
             succeeded = true;
         }
         finally {
+            await networkCapture?.shutdown();
             await droidReportUploadManager.drain();
             if (lambdaTestSessionConnected) {
                 console.log("Ending LambdaTest session...");
@@ -910,12 +963,18 @@ async function main() {
         reviseTestScript: runtimeContext.openAi.reviseTestScript,
     });
     // Otherwise, start interactive Ink shell
-    await startInkShell(session, engine, {
-        appContextPath: appContextPath ? path.resolve(appContextPath) : null,
-        runtimeContext,
-    });
+    try {
+        await startInkShell(session, engine, {
+            appContextPath: appContextPath ? path.resolve(appContextPath) : null,
+            runtimeContext,
+        });
+    }
+    finally {
+        await networkCapture?.shutdown();
+    }
 }
-main().catch((error) => {
+main().catch(async (error) => {
+    await networkCapture?.shutdown().catch(() => { });
     const message = error instanceof Error ? error.message : String(error);
     console.error(`\nTest failed: ${message}`);
     process.exit(1);

package/build/src/browser/connection.js

@@ -35,6 +35,11 @@ export async function closeCurrentTab(browserSessionId) {
     return currentBackend.closeCurrentTab(browserSessionId);
 }
 export { getCurrentPlatform };
+export function getNetworkCaptureContext(browserSessionId) {
+    if (!currentBackend?.getNetworkCaptureContext)
+        throw new Error("Network capture supports local Chrome and Edge sessions only.");
+    return currentBackend.getNetworkCaptureContext(browserSessionId);
+}
 export async function disconnect() {
     if (currentBackend?.disconnect) {
         await currentBackend.disconnect();

package/build/src/browser/factory.js

@@ -16,6 +16,7 @@ function buildPlaywrightBackend() {
         closeCurrentTab: playwrightConnection.closeCurrentTab,
         handleModelAction: playwrightActions.handleModelAction,
         disconnect: playwrightConnection.disconnect,
+        getNetworkCaptureContext: playwrightConnection.getNetworkCaptureContext,
     };
 }
 function buildWebDriverBackend() {

package/build/src/browser/playwright/connection.js

@@ -373,6 +373,9 @@ export async function connectToBrowser(targetOrOptions) {
     });
     return sessionId;
 }
+export function getNetworkCaptureContext(browserSessionId) {
+    return getActiveBrowserSession(browserSessionId).context;
+}
 export async function getBrowserInfo(browserSessionId) {
     const session = getActiveBrowserSession(browserSessionId);
     const activePage = getActivePage(session);

package/build/src/cli/headless-runner.js

@@ -1,6 +1,8 @@
 import path from "path";
 import { mkdir, writeFile } from "fs/promises";
-import { getCurrentPlatform as getBrowserCurrentPlatform, getScreenshotAsBase64 as getBrowserScreenshotAsBase64, } from "../browser/connection.js";
+import { getNetworkCaptureContext, navigateToUrl as navigateBrowserToUrl, getBrowserInfo, getCurrentPlatform as getBrowserCurrentPlatform, getScreenshotAsBase64 as getBrowserScreenshotAsBase64, } from "../browser/connection.js";
+import { normalizeBrowserInfoForSession } from "../browser/normalize.js";
+import { updateBrowserSessionInfo } from "../core/tool-handlers/browser-session.js";
 import { handleModelAction as handleBrowserModelAction } from "../browser/actions.js";
 import { ExecutionEngine } from "../core/execution-engine.js";
 import { buildExecutionModePrompt } from "../core/prompts.js";
@@ -17,6 +19,7 @@ import { clearLoadmillReportMetadata } from "./loadmill-report-metadata.js";
 import { formatHeadlessTestSourceLabel } from "./headless-test-source-label.js";
 import { buildCliRunReportModel, createCapturingAddOutput, createCliAggregateRunReport, createCliRunReport, renderCliCloudReport, writeCliRunReportArtifact, } from "./headless-report.js";
 import { createRefuseReconnectConnectToBrowser } from "../desktop-shared/connection-predicates.js";
+import { describeNetworkCapture, hasNetworkCaptureIssue } from "../network/run-network-capture.js";
 function sanitizePathSegment(value) {
     return String(value || "test")
         .replace(/\.dcua$/i, "")
@@ -29,6 +32,15 @@ function createHeadlessRunId(index = 0) {
     const suffix = index > 0 ? `-${index + 1}` : "";
     return `run-${Date.now()}${suffix}`;
 }
+function networkLogPath(runId, report) {
+    if (!report?.enabled)
+        return path.resolve("logs", `network-${runId}.jsonl`);
+    if (!report.path)
+        return path.resolve(`network-${runId}.jsonl`);
+    const resolved = path.resolve(report.path);
+    const directory = /\.html?$/i.test(report.path) ? path.dirname(resolved) : resolved;
+    return path.join(directory, `network-${runId}.jsonl`);
+}
 function logPendingScreenshotUploads(uploadManager, uploadSession, subject) {
     console.log('Verifying no pending upload exists');
     const pendingCount = uploadManager.getPendingCount?.(uploadSession) ?? 0;
@@ -167,8 +179,27 @@ export function buildHeadlessEngineOptions({ runtimeContext, headlessConfig, exe
     }
     return engineOptions;
 }
-async function runHeadlessTest({ runId, testPath, testName, instructions, session, deviceInfo, headlessConfig, appContext, testDataContext = "", runtimeContext, headlessDebug, recordScreenshots, screenshotDir, isBrowserTarget, isDesktopTarget, desktopController, isCloudTarget, runTarget = null, secrets = null, report = null, isBatchRun = false, reportToLoadmill = true, runConfiguration = {}, cloudUploadSession = null, uploadManager = droidReportUploadManager, }) {
+async function runHeadlessTest({ runId, testPath, testName, instructions, session, deviceInfo, headlessConfig, appContext, testDataContext = "", runtimeContext, headlessDebug, recordScreenshots, screenshotDir, isBrowserTarget, isDesktopTarget, desktopController, isCloudTarget, runTarget = null, secrets = null, report = null, isBatchRun = false, reportToLoadmill = true, runConfiguration = {}, cloudUploadSession = null, uploadManager = droidReportUploadManager, networkCapture = null, initialBrowserUrl = null, }) {
     let executionMode = null;
+    let captureStarted = false;
+    const finishCapture = async () => {
+        if (!captureStarted)
+            return;
+        captureStarted = false;
+        try {
+            const result = await networkCapture.stopCapture();
+            if (result) {
+                const line = `${describeNetworkCapture(result)} Network log: ${result.path}`;
+                if (hasNetworkCaptureIssue(result))
+                    console.warn(`Warning: ${line}`);
+                else
+                    console.log(line);
+            }
+        }
+        catch (error) {
+            console.warn(`Warning: network capture could not finish: ${error.message}`);
+        }
+    };
     let loadmillSessionUser = null;
     let runUsageTotals = setDroidCuaRunUsageModel(createDroidCuaRunUsageTotals(), runtimeContext.config.cuaModel);
     const buildSegmentUsageProperties = () => ({
@@ -254,6 +285,23 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
         });
         const engine = new ExecutionEngine(session, engineOptions);
         executionMode = new ExecutionMode(session, engine, instructions, true);
+        if (networkCapture) {
+            try {
+                const browserContext = isBrowserTarget ? getNetworkCaptureContext(session.deviceId) : null;
+                const capturePath = await networkCapture.startCapture(networkLogPath(runId, report), { browserContext });
+                console.log(`Network log: ${capturePath}`);
+                captureStarted = true;
+                engine.networkQuery = networkCapture.query;
+                engine.toolCapabilities.queryNetwork = true;
+            }
+            catch (error) {
+                throw new Error(`Network capture setup failed: ${error.message}`, { cause: error });
+            }
+        }
+        if (initialBrowserUrl) {
+            await navigateBrowserToUrl(session.deviceId, initialBrowserUrl);
+            updateBrowserSessionInfo(session, normalizeBrowserInfoForSession(await getBrowserInfo(session.deviceId)));
+        }
         const result = await executionMode.execute({
             runId,
             addOutput,
@@ -264,6 +312,7 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
             // Inline screenshots as data URLs so the HTML report is self-contained.
             ...(reportEnabled ? { includeScreenshotDataUrl: true } : {}),
         });
+        await finishCapture();
         const stats = executionMode.stats || {};
         const durationMs = stats.startTime ? Math.max(0, Date.now() - stats.startTime) : 0;
         await headlessDebug.endExecutionSession(runId, {
@@ -363,6 +412,7 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
         };
     }
     catch (error) {
+        await finishCapture();
         const message = error instanceof Error ? error.message : "Failed to start execution.";
         const stats = executionMode?.stats || {};
         const durationMs = stats.startTime ? Math.max(0, Date.now() - stats.startTime) : 0;
@@ -468,7 +518,7 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
         throw error;
     }
 }
-export async function runHeadlessInstructionRuns({ runs, instructionsTarget, session, deviceInfo, headlessConfig, appContext, runtimeContext, headlessDebug, secrets = null, recordScreenshots, screenshotDir, isBrowserTarget, isDesktopTarget = false, desktopController = null, isCloudTarget, runTarget = null, report = null, droidVersion = getDroidVersion(), runHeadlessTestImpl = runHeadlessTest, uploadManager = droidReportUploadManager, }) {
+export async function runHeadlessInstructionRuns({ runs, instructionsTarget, session, deviceInfo, headlessConfig, appContext, runtimeContext, headlessDebug, secrets = null, recordScreenshots, screenshotDir, isBrowserTarget, isDesktopTarget = false, desktopController = null, isCloudTarget, runTarget = null, report = null, droidVersion = getDroidVersion(), runHeadlessTestImpl = runHeadlessTest, uploadManager = droidReportUploadManager, networkCapture = null, initialBrowserUrl = null, }) {
     await clearLoadmillReportMetadata(report?.metadataPath);
     const resolvedRunTarget = runTarget ?? { isBrowserTarget, isDesktopTarget, isCloudTarget };
     const runConfiguration = buildDroidRunConfiguration({
@@ -546,6 +596,8 @@ export async function runHeadlessInstructionRuns({ runs, instructionsTarget, ses
                 runConfiguration,
                 cloudUploadSession,
                 uploadManager,
+                networkCapture,
+                initialBrowserUrl: index === 0 ? initialBrowserUrl : null,
             });
             aggregateStats.durationMs += result.durationMs ?? 0;
             aggregateStats.instructionsTotal += result.instructionsTotal ?? 0;

package/build/src/cli/headless-target.js

@@ -184,6 +184,9 @@ export function shouldWaitForLoadmillCloudCompletion(targetConfig) {
 const MOBILE_ONLY_CLOUD_SOURCES = new Set(["lambdatest", "aws-device-farm"]);
 export function resolveHeadlessTarget(args = {}) {
     const target = normalizeTarget(args.target);
+    if (target !== "desktop" && (args["desktop-app"] != null || args["desktop-window"] != null || args["desktop-input"] != null || args["list-desktop-apps"] != null)) {
+        throw new Error("--desktop-app, --desktop-window, --desktop-input, and --list-desktop-apps can only be used with --target desktop.");
+    }
     const targetGiven = hasNonEmptyString(args.target);
     const avdName = hasNonEmptyString(args.avd) ? args.avd.trim() : null;
     const platform = normalizeMobilePlatform(args.platform, "--platform");
@@ -264,18 +267,25 @@ export function resolveHeadlessTarget(args = {}) {
         }
         const listDesktopApps = args["list-desktop-apps"] != null;
         const desktopApp = hasNonEmptyString(args["desktop-app"]) ? args["desktop-app"].trim() : null;
+        if (args["desktop-app"] != null && !desktopApp) {
+            throw new Error("--desktop-app requires an application ID or exact name.");
+        }
         const rawWindow = readNonEmptyCliValue(args["desktop-window"]);
         const desktopWindow = rawWindow == null ? null : Number(rawWindow);
         const desktopInput = hasNonEmptyString(args["desktop-input"]) ? args["desktop-input"].trim().toLowerCase() : "physical";
-        if (!listDesktopApps && !desktopApp) {
-            throw new Error("--desktop-app is required with --target desktop.");
+        if (args["desktop-window"] != null && !desktopApp) {
+            throw new Error("--desktop-window requires --desktop-app.");
         }
-        if (rawWindow != null && (!Number.isInteger(desktopWindow) || desktopWindow < 0)) {
+        if (args["desktop-window"] != null && (rawWindow == null || !Number.isInteger(desktopWindow) || desktopWindow < 0)) {
             throw new Error("--desktop-window must be a non-negative integer window ID.");
         }
-        if (desktopInput !== "accessibility" && desktopInput !== "physical") {
+        if ((args["desktop-input"] != null && !hasNonEmptyString(args["desktop-input"])) ||
+            (desktopInput !== "accessibility" && desktopInput !== "physical")) {
             throw new Error('--desktop-input must be "accessibility" or "physical".');
         }
+        if (desktopInput === "accessibility" && !desktopApp) {
+            throw new Error("--desktop-input accessibility requires --desktop-app. Full desktop uses physical input only.");
+        }
         return {
             target,
             sessionMode: null,
@@ -290,14 +300,12 @@ export function resolveHeadlessTarget(args = {}) {
             artifacts: null,
             listDevices: false,
             listDesktopApps,
+            desktopScope: desktopApp ? "window" : "desktop",
             desktopApp,
             desktopWindow,
             desktopInput,
         };
     }
-    if (args["desktop-app"] != null || args["desktop-window"] != null || args["desktop-input"] != null || args["list-desktop-apps"] != null) {
-        throw new Error("--desktop-app, --desktop-window, --desktop-input, and --list-desktop-apps can only be used with --target desktop.");
-    }
     if (args["session-mode"] != null) {
         throw new Error("--session-mode can only be used with --target web.");
     }

package/build/src/commands/help.js

@@ -11,19 +11,21 @@ import { CUA_MODELS } from '../core/cua-models.js';
  */
 export async function handleHelp(args, session, context) {
     const addOutput = context?.addOutput || ((item) => console.log(item.text || item));
-    addOutput({ type: 'system', text: 'droid-cua - AI-powered mobile and web testing CLI' });
+    addOutput({ type: 'system', text: 'droid-cua - AI-powered mobile, web, and desktop testing CLI' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Usage:' });
     addOutput({ type: 'info', text: '  droid-cua --avd <device-name> [options]' });
     addOutput({ type: 'info', text: '  droid-cua run <file-or-directory...> [options]' });
     addOutput({ type: 'info', text: '  droid-cua --target web --instructions <file> [options]' });
+    addOutput({ type: 'info', text: '  droid-cua --target desktop --instructions <file> [options]' });
     addOutput({ type: 'info', text: '  droid-cua --device-source lambdatest --instructions <file> [options]' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Options:' });
     addOutput({ type: 'info', text: '  --target <kind>        Target kind: mobile (default), web, or desktop' });
-    addOutput({ type: 'info', text: '  --desktop-app <id>     Running desktop app ID or exact name' });
-    addOutput({ type: 'info', text: '  --desktop-window <id>  Window ID when the app has multiple windows' });
-    addOutput({ type: 'info', text: '  --desktop-input <mode> Physical pointer (default) or cursor-free accessibility' });
+    addOutput({ type: 'info', text: '                        Desktop controls the primary display by default' });
+    addOutput({ type: 'info', text: '  --desktop-app <id>     Limit desktop control to a running app ID or exact name' });
+    addOutput({ type: 'info', text: '  --desktop-window <id>  Select a window within --desktop-app' });
+    addOutput({ type: 'info', text: '  --desktop-input <mode> Physical pointer (default); accessibility requires --desktop-app' });
     addOutput({ type: 'info', text: '  --list-desktop-apps    List running desktop apps and windows' });
     addOutput({ type: 'info', text: '  --avd <name>           Device name (Android device ID/serial or iOS Simulator)' });
     addOutput({ type: 'info', text: '  --platform <platform>  Force platform: android or ios' });
@@ -53,6 +55,8 @@ export async function handleHelp(args, session, context) {
     addOutput({ type: 'info', text: '  --record               Record screenshots during execution' });
     addOutput({ type: 'info', text: '  --explain              Explain non-obvious decisions and completion evidence' });
     addOutput({ type: 'info', text: '  --report [path]        Save a self-contained HTML report locally' });
+    addOutput({ type: 'info', text: '  --network-proxy        Forward local mobile traffic through 127.0.0.1:8765 without capture' });
+    addOutput({ type: 'info', text: '  --capture-network      Capture local mobile or browser test traffic to one JSONL log per test' });
     addOutput({ type: 'info', text: '  --no-cloud-report      Skip uploading the HTML report and screenshots to Loadmill' });
     addOutput({ type: 'info', text: '  --report-metadata <path> Write uploaded Loadmill report metadata as JSON' });
     addOutput({ type: 'info', text: '  --debug                Enable structured JSONL debug artifacts' });
@@ -76,6 +80,7 @@ export async function handleHelp(args, session, context) {
     addOutput({ type: 'info', text: '  iOS:     Uses Appium + XCUITest for iOS Simulator automation' });
     addOutput({ type: 'info', text: '  LambdaTest: Uses real cloud devices for headless mobile runs' });
     addOutput({ type: 'info', text: '  Web:     Uses Playwright with installed Google Chrome or Microsoft Edge for browser execution' });
+    addOutput({ type: 'info', text: '  Desktop: Controls the primary display or a selected application on macOS and Windows' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Platform Detection:' });
     addOutput({ type: 'info', text: '  - Use --platform flag to force a specific platform' });
@@ -98,6 +103,7 @@ export async function handleHelp(args, session, context) {
     addOutput({ type: 'info', text: '  droid-cua --instructions tests/login.dcua --context app/context.md' });
     addOutput({ type: 'info', text: '  droid-cua --instructions tests --test-data tests/test-data.md --scenario "Run every checkout test with each buyer account"' });
     addOutput({ type: 'info', text: '  droid-cua --target web --instructions tests/search.dcua' });
+    addOutput({ type: 'info', text: '  droid-cua --target desktop --instructions tests/launcher.dcua' });
     addOutput({ type: 'info', text: '  droid-cua --device-source lambdatest --platform android --device-name "Galaxy S24" --os-version 14 --app app.apk --instructions tests/login.dcua' });
     addOutput({ type: 'info', text: '  /create login-test                      (design a new test)' });
     addOutput({ type: 'info', text: '  /list                                   (see all tests)' });

package/build/src/core/execution-engine.js

@@ -1,3 +1,4 @@
+import { answerNetworkQuestion, handleQueryNetworkToolCall, NETWORK_QUERY_FAILURE_MESSAGE } from "./tool-handlers/query-network.js";
 import { randomUUID } from "node:crypto";
 import { getScreenshotAsBase64, getCurrentPlatform } from "../device/connection.js";
 import { closeCurrentTab as closeCurrentBrowserTab, getBrowserInfo, navigateToUrl as navigateBrowserToUrl } from "../browser/connection.js";
@@ -246,6 +247,8 @@ export class ExecutionEngine {
         this.handleModelAction = options.handleModelAction || handleModelAction;
         this.closeKeyboard = options.closeKeyboard || closeKeyboard;
         this.sendCUARequest = options.sendCUARequest || sendCUARequest;
+        this.networkQuery = options.networkQuery || null;
+        this.answerNetworkQuestion = options.answerNetworkQuestion || answerNetworkQuestion;
         this.reviseTestScript = options.reviseTestScript || null;
         this.getCurrentPlatform = options.getCurrentPlatform || getCurrentPlatform;
         this.getBrowserInfo = options.getBrowserInfo || getBrowserInfo;
@@ -323,6 +326,8 @@ export class ExecutionEngine {
         switch (toolName) {
             case "preview_click":
                 return await handlePreviewClick(handlerArgs);
+            case "query_network":
+                return await handleQueryNetworkToolCall(handlerArgs);
             case "assert":
                 return await handleAssertionToolCall(handlerArgs);
             case "navigate":
@@ -408,7 +413,7 @@ export class ExecutionEngine {
             // The Desktop recorder returns one from disk; for runs without a recorder (e.g. a
             // plain CLI `--report`) render the same marker in-memory from the raw screenshot.
             const annotatedDataUrl = await this.resolveAnnotatedDataUrl(annotatedScreenshot, currentScreenshot, action, context);
-            await this.handleModelAction(this.session.deviceId, action, this.session.deviceInfo.scale, {
+            const actionResult = await this.handleModelAction(this.session.deviceId, action, this.session.deviceInfo.scale, {
                 ...context,
                 shouldStop,
                 stepId: stepContext?.stepId,
@@ -430,6 +435,13 @@ export class ExecutionEngine {
                 ...(annotatedScreenshot?.screenshotPath ? { screenshotPath: annotatedScreenshot.screenshotPath } : {}),
                 ...(context?.includeScreenshotDataUrl && annotatedDataUrl ? { screenshotDataUrl: annotatedDataUrl } : {})
             });
+            if (actionResult?.success === false) {
+                if (shouldStop())
+                    return { stopped: true, sawExplicitScreenshotAction };
+                const actionError = `${action.type} failed: ${actionResult.error || "The action could not be completed."}`;
+                this.session.addToTranscript?.(`[Action Error] ${actionError}`);
+                return { refreshModelState: true, actionError, sawExplicitScreenshotAction };
+            }
             if (trackAction) {
                 trackAction(action);
             }
@@ -599,6 +611,7 @@ export class ExecutionEngine {
             ...extra
         });
         let refreshBeforeNextComputerAction = false;
+        let consecutiveNetworkQueryFailures = 0;
         let observationOnlyTurnsSinceLastAction = 0;
         let observationOnlyRecoverySent = false;
         let pendingObservationOnlyRecoveryMessage = null;
@@ -745,6 +758,15 @@ export class ExecutionEngine {
                     summary.error = toolResult.error;
                     break;
                 }
+                if (actionableItem.name === "query_network") {
+                    consecutiveNetworkQueryFailures = toolResult.queryFailed ? consecutiveNetworkQueryFailures + 1 : 0;
+                    if (consecutiveNetworkQueryFailures >= 3) {
+                        summary.failed = true;
+                        summary.error = NETWORK_QUERY_FAILURE_MESSAGE;
+                        addOutput({ type: "error", text: summary.error, eventType: "error", ...eventMeta() });
+                        break;
+                    }
+                }
                 if (toolResult.kind === "retry_instruction") {
                     summary.retryInstruction = true;
                     newResponseId = null;
@@ -823,6 +845,9 @@ export class ExecutionEngine {
             }
             if (actionBatchResult.refreshModelState) {
                 const followUp = await this.buildActionFollowUp(call_id, pendingSafetyChecks, true, addOutput, eventMeta, context, stepContext);
+                if (actionBatchResult.actionError) {
+                    followUp.input.push({ role: "user", content: `${actionBatchResult.actionError} The remaining actions were skipped. Use the fresh screenshot to decide what to do next.` });
+                }
                 currentScreenshot = followUp.currentScreenshot ?? currentScreenshot;
                 response = await this.requestNextTurn(followUp.input, newResponseId, context, stepContext);
                 newResponseId = response.id;