@loadmill/droid-cua 2.40.0 → 2.41.0

package/README.md

@@ -193,6 +193,9 @@ droid-cua run tests/login.dcua --platform ios --avd "iPhone 16"
 # List running desktop applications and their window IDs (macOS or Windows)
 droid-cua --target desktop --list-desktop-apps
 
+# Control the full primary display across applications
+droid-cua --target desktop --instructions tests/launcher.dcua
+
 # Control one already-running desktop application window
 droid-cua --target desktop --desktop-app com.apple.TextEdit --desktop-window 123 --instructions tests/editor.dcua
 
@@ -276,8 +279,8 @@ Supported CLI options include:
 - `--list-devices`
 - `--browser`
 - `--target` (`mobile`, `web`, or `desktop`)
-- `--desktop-app` (desktop application ID or exact name)
-- `--desktop-window` (required when the application has multiple visible windows)
+- `--desktop-app` (limit desktop control to an application ID or exact name; omit for full desktop)
+- `--desktop-window` (select a window within `--desktop-app`)
 - `--desktop-input` (`physical` by default; use `accessibility` for cursor-free input)
 - `--list-desktop-apps`
 - `--instructions`
@@ -304,7 +307,15 @@ Supported CLI options include:
 `OPENAI_API_KEY`. Saved direct OpenAI selections of GPT-5.6 Terra, GPT-5.6 Luna,
 or GPT-5.4 move to GPT-6 Sol, GPT-6 Luna, or GPT-6 Sol, respectively.
 
-Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default: Droid shows a system notice for three seconds before taking control, brings the selected window to the foreground, then sends physical pointer and keyboard input at the model's coordinates. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. When the run ends, Droid leaves the target window in front and shows another system notice that control has finished, including when the run fails or is stopped. CLI runs also leave the target in front. Use `--desktop-input accessibility` for cursor-free CLI input instead. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. Windows targets must run at the same integrity level as Droid; a non-elevated Droid process cannot inject input into an elevated application. If Windows returns an all-black window capture, Droid brings the target back to the foreground and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The crop is resized only after extraction to preserve window detail. The fallback logs a warning and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target to the foreground yourself when requested.
+Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use physical pointer and keyboard input by default. Droid shows a system notice for three seconds before taking control and a completion notice when the run finishes, fails, or is stopped. The delay is fixed; use Stop in Droid once the run starts. Avoid using the mouse or keyboard during the run. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. On Windows, Droid must run at the same integrity level as the target application. System notices and automatic minimization apply only to runs started in the desktop app.
+
+CLI runs with `--target desktop` control the full primary display by default, with no application flag needed. In the desktop app, choose **Full desktop**, the first option in the Application dropdown. This mode captures and controls the primary display across applications, including apps launched during the test; no target application needs to be running when connecting. Keep all test windows on the primary display. Full desktop requires physical input. Droid minimizes itself after the three-second notice and remains minimized when the run ends; open Droid to review the result or use Stop. Existing saved application selections are preserved.
+
+Use `--desktop-app <id-or-exact-name>` or select an application in Droid to limit control to that application's windows. Application mode brings the selected window forward and leaves it there when the run ends. Add `--desktop-window <id>` to select a particular window, or `--desktop-input accessibility` for cursor-free CLI input; both options require `--desktop-app`. Before each run and screenshot, Droid refreshes window selection within the original application so it can follow a recreated login window. If replacement is ambiguous, select a window again. If a captured window disappears or moves before input, Droid requests a fresh screenshot instead of transferring its coordinates to a replacement. Focus errors report the activation reason; they do not establish an elevation problem.
+
+If Windows returns an all-black window capture, Droid brings the target forward and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The fallback logs a warning, crops before resizing to preserve detail, and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target forward yourself when requested. Full desktop reports capture failures directly, without this window capture fallback.
+
+Desktop **Test design** uses the same notices once you submit your instructions and exploration starts. Full desktop minimizes Droid at that point. Droid sends the completion notice when the generated test is ready, exploration is stopped, or recovery ends in an error. It does not automatically bring itself forward; open Droid whenever you need to give guidance, use Stop, or review the generated test. Opening Test design alone does not take desktop control. When a controller rejects an action, Droid skips the remaining actions in that batch and asks the model to continue from a fresh screenshot with the failure reason. Rejected actions are not counted as completed.
 
 The system notices are shown only for desktop control runs started in the desktop app. On macOS, they may not appear in the unsigned Electron dev app; verify them in a signed packaged build.
 

package/build/index.js

@@ -534,6 +534,7 @@ async function main() {
     }
     else if (isDesktopTarget) {
         desktopController = await createDesktopController({
+            scope: targetConfig.desktopScope,
             app: targetConfig.desktopApp,
             windowId: targetConfig.desktopWindow,
             inputMode: targetConfig.desktopInput,
@@ -542,8 +543,8 @@ async function main() {
         });
         targetId = desktopController.deviceId;
         deviceInfo = desktopController.deviceInfo;
-        console.log(`Attached to desktop window: ${deviceInfo.device_name}`);
-        console.log(`Model sees window: ${deviceInfo.scaled_width}x${deviceInfo.scaled_height}`);
+        console.log(`Attached to desktop target: ${deviceInfo.device_name}`);
+        console.log(`Model sees desktop target: ${deviceInfo.scaled_width}x${deviceInfo.scaled_height}`);
         console.log(`Desktop input: ${deviceInfo.desktopInputMode === "physical" ? "physical OS pointer" : "cursor-free accessibility"}`);
         process.once("SIGINT", async () => {
             console.log("\nStopping desktop run...");

package/build/src/cli/headless-target.js

@@ -184,6 +184,9 @@ export function shouldWaitForLoadmillCloudCompletion(targetConfig) {
 const MOBILE_ONLY_CLOUD_SOURCES = new Set(["lambdatest", "aws-device-farm"]);
 export function resolveHeadlessTarget(args = {}) {
     const target = normalizeTarget(args.target);
+    if (target !== "desktop" && (args["desktop-app"] != null || args["desktop-window"] != null || args["desktop-input"] != null || args["list-desktop-apps"] != null)) {
+        throw new Error("--desktop-app, --desktop-window, --desktop-input, and --list-desktop-apps can only be used with --target desktop.");
+    }
     const targetGiven = hasNonEmptyString(args.target);
     const avdName = hasNonEmptyString(args.avd) ? args.avd.trim() : null;
     const platform = normalizeMobilePlatform(args.platform, "--platform");
@@ -264,18 +267,25 @@ export function resolveHeadlessTarget(args = {}) {
         }
         const listDesktopApps = args["list-desktop-apps"] != null;
         const desktopApp = hasNonEmptyString(args["desktop-app"]) ? args["desktop-app"].trim() : null;
+        if (args["desktop-app"] != null && !desktopApp) {
+            throw new Error("--desktop-app requires an application ID or exact name.");
+        }
         const rawWindow = readNonEmptyCliValue(args["desktop-window"]);
         const desktopWindow = rawWindow == null ? null : Number(rawWindow);
         const desktopInput = hasNonEmptyString(args["desktop-input"]) ? args["desktop-input"].trim().toLowerCase() : "physical";
-        if (!listDesktopApps && !desktopApp) {
-            throw new Error("--desktop-app is required with --target desktop.");
+        if (args["desktop-window"] != null && !desktopApp) {
+            throw new Error("--desktop-window requires --desktop-app.");
         }
-        if (rawWindow != null && (!Number.isInteger(desktopWindow) || desktopWindow < 0)) {
+        if (args["desktop-window"] != null && (rawWindow == null || !Number.isInteger(desktopWindow) || desktopWindow < 0)) {
             throw new Error("--desktop-window must be a non-negative integer window ID.");
         }
-        if (desktopInput !== "accessibility" && desktopInput !== "physical") {
+        if ((args["desktop-input"] != null && !hasNonEmptyString(args["desktop-input"])) ||
+            (desktopInput !== "accessibility" && desktopInput !== "physical")) {
             throw new Error('--desktop-input must be "accessibility" or "physical".');
         }
+        if (desktopInput === "accessibility" && !desktopApp) {
+            throw new Error("--desktop-input accessibility requires --desktop-app. Full desktop uses physical input only.");
+        }
         return {
             target,
             sessionMode: null,
@@ -290,14 +300,12 @@ export function resolveHeadlessTarget(args = {}) {
             artifacts: null,
             listDevices: false,
             listDesktopApps,
+            desktopScope: desktopApp ? "window" : "desktop",
             desktopApp,
             desktopWindow,
             desktopInput,
         };
     }
-    if (args["desktop-app"] != null || args["desktop-window"] != null || args["desktop-input"] != null || args["list-desktop-apps"] != null) {
-        throw new Error("--desktop-app, --desktop-window, --desktop-input, and --list-desktop-apps can only be used with --target desktop.");
-    }
     if (args["session-mode"] != null) {
         throw new Error("--session-mode can only be used with --target web.");
     }

package/build/src/commands/help.js

@@ -11,19 +11,21 @@ import { CUA_MODELS } from '../core/cua-models.js';
  */
 export async function handleHelp(args, session, context) {
     const addOutput = context?.addOutput || ((item) => console.log(item.text || item));
-    addOutput({ type: 'system', text: 'droid-cua - AI-powered mobile and web testing CLI' });
+    addOutput({ type: 'system', text: 'droid-cua - AI-powered mobile, web, and desktop testing CLI' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Usage:' });
     addOutput({ type: 'info', text: '  droid-cua --avd <device-name> [options]' });
     addOutput({ type: 'info', text: '  droid-cua run <file-or-directory...> [options]' });
     addOutput({ type: 'info', text: '  droid-cua --target web --instructions <file> [options]' });
+    addOutput({ type: 'info', text: '  droid-cua --target desktop --instructions <file> [options]' });
     addOutput({ type: 'info', text: '  droid-cua --device-source lambdatest --instructions <file> [options]' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Options:' });
     addOutput({ type: 'info', text: '  --target <kind>        Target kind: mobile (default), web, or desktop' });
-    addOutput({ type: 'info', text: '  --desktop-app <id>     Running desktop app ID or exact name' });
-    addOutput({ type: 'info', text: '  --desktop-window <id>  Window ID when the app has multiple windows' });
-    addOutput({ type: 'info', text: '  --desktop-input <mode> Physical pointer (default) or cursor-free accessibility' });
+    addOutput({ type: 'info', text: '                        Desktop controls the primary display by default' });
+    addOutput({ type: 'info', text: '  --desktop-app <id>     Limit desktop control to a running app ID or exact name' });
+    addOutput({ type: 'info', text: '  --desktop-window <id>  Select a window within --desktop-app' });
+    addOutput({ type: 'info', text: '  --desktop-input <mode> Physical pointer (default); accessibility requires --desktop-app' });
     addOutput({ type: 'info', text: '  --list-desktop-apps    List running desktop apps and windows' });
     addOutput({ type: 'info', text: '  --avd <name>           Device name (Android device ID/serial or iOS Simulator)' });
     addOutput({ type: 'info', text: '  --platform <platform>  Force platform: android or ios' });
@@ -76,6 +78,7 @@ export async function handleHelp(args, session, context) {
     addOutput({ type: 'info', text: '  iOS:     Uses Appium + XCUITest for iOS Simulator automation' });
     addOutput({ type: 'info', text: '  LambdaTest: Uses real cloud devices for headless mobile runs' });
     addOutput({ type: 'info', text: '  Web:     Uses Playwright with installed Google Chrome or Microsoft Edge for browser execution' });
+    addOutput({ type: 'info', text: '  Desktop: Controls the primary display or a selected application on macOS and Windows' });
     addOutput({ type: 'info', text: '' });
     addOutput({ type: 'info', text: 'Platform Detection:' });
     addOutput({ type: 'info', text: '  - Use --platform flag to force a specific platform' });
@@ -98,6 +101,7 @@ export async function handleHelp(args, session, context) {
     addOutput({ type: 'info', text: '  droid-cua --instructions tests/login.dcua --context app/context.md' });
     addOutput({ type: 'info', text: '  droid-cua --instructions tests --test-data tests/test-data.md --scenario "Run every checkout test with each buyer account"' });
     addOutput({ type: 'info', text: '  droid-cua --target web --instructions tests/search.dcua' });
+    addOutput({ type: 'info', text: '  droid-cua --target desktop --instructions tests/launcher.dcua' });
     addOutput({ type: 'info', text: '  droid-cua --device-source lambdatest --platform android --device-name "Galaxy S24" --os-version 14 --app app.apk --instructions tests/login.dcua' });
     addOutput({ type: 'info', text: '  /create login-test                      (design a new test)' });
     addOutput({ type: 'info', text: '  /list                                   (see all tests)' });

package/build/src/core/execution-engine.js

@@ -408,7 +408,7 @@ export class ExecutionEngine {
             // The Desktop recorder returns one from disk; for runs without a recorder (e.g. a
             // plain CLI `--report`) render the same marker in-memory from the raw screenshot.
             const annotatedDataUrl = await this.resolveAnnotatedDataUrl(annotatedScreenshot, currentScreenshot, action, context);
-            await this.handleModelAction(this.session.deviceId, action, this.session.deviceInfo.scale, {
+            const actionResult = await this.handleModelAction(this.session.deviceId, action, this.session.deviceInfo.scale, {
                 ...context,
                 shouldStop,
                 stepId: stepContext?.stepId,
@@ -430,6 +430,13 @@ export class ExecutionEngine {
                 ...(annotatedScreenshot?.screenshotPath ? { screenshotPath: annotatedScreenshot.screenshotPath } : {}),
                 ...(context?.includeScreenshotDataUrl && annotatedDataUrl ? { screenshotDataUrl: annotatedDataUrl } : {})
             });
+            if (actionResult?.success === false) {
+                if (shouldStop())
+                    return { stopped: true, sawExplicitScreenshotAction };
+                const actionError = `${action.type} failed: ${actionResult.error || "The action could not be completed."}`;
+                this.session.addToTranscript?.(`[Action Error] ${actionError}`);
+                return { refreshModelState: true, actionError, sawExplicitScreenshotAction };
+            }
             if (trackAction) {
                 trackAction(action);
             }
@@ -823,6 +830,9 @@ export class ExecutionEngine {
             }
             if (actionBatchResult.refreshModelState) {
                 const followUp = await this.buildActionFollowUp(call_id, pendingSafetyChecks, true, addOutput, eventMeta, context, stepContext);
+                if (actionBatchResult.actionError) {
+                    followUp.input.push({ role: "user", content: `${actionBatchResult.actionError} The remaining actions were skipped. Use the fresh screenshot to decide what to do next.` });
+                }
                 currentScreenshot = followUp.currentScreenshot ?? currentScreenshot;
                 response = await this.requestNextTurn(followUp.input, newResponseId, context, stepContext);
                 newResponseId = response.id;

package/build/src/desktop/computer-use.js

@@ -3,6 +3,7 @@ import { buildTypeActionPayload, buildTypeActionText, } from "../device/sensitiv
 import { getConfiguredStepDelayMs } from "../utils/step-delay.js";
 import { getWindowsVisibleBounds } from "./windows-visible-bounds.js";
 import sharp from "sharp";
+import { capturePrimaryDisplay } from "./primary-display.js";
 const SCREENSHOT_WIDTH = 1440;
 const ACCESSIBILITY_HIT_TOLERANCE = 12;
 const MACOS_CAPTURE_INSETS = { left: 50, right: 50, top: 38, bottom: 62 };
@@ -104,7 +105,7 @@ function formatPermissionError(error, platform) {
     if (platform === "darwin" && /permission|not authorized|screencapture|screen recording|cgwindow|axerror/i.test(message)) {
         return `${message} Grant Screen & System Audio Recording and Accessibility access to Droid (or your terminal for CLI runs) in System Settings > Privacy & Security, then restart it.`;
     }
-    if (platform === "win32" && /access denied|elevat|integrity level|could not focus/i.test(message)) {
+    if (platform === "win32" && /access denied|elevat|integrity level/i.test(message)) {
         return `${message} Run Droid in an interactive desktop session and at the same integrity level as the target application; elevated Windows applications cannot be controlled by a non-elevated Droid process.`;
     }
     return message;
@@ -236,22 +237,26 @@ function normalizeShortcut(keys, platform) {
     return keys.map((key) => aliases[String(key).toUpperCase()] || String(key).toLowerCase()).join("+");
 }
 /**
- * @param {{ app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", focusOnConnect?: boolean, client?: any, accessibility?: any, visibleBoundsReader?: typeof getWindowsVisibleBounds }} options
+ * @param {{ scope?: "window" | "desktop", app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", focusOnConnect?: boolean, client?: any, accessibility?: any, visibleBoundsReader?: typeof getWindowsVisibleBounds, displayCapture?: typeof capturePrimaryDisplay }} options
  */
-export async function createDesktopController({ app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", focusOnConnect = false, client: injectedClient = null, accessibility: injectedAccessibility = null, visibleBoundsReader = getWindowsVisibleBounds, } = {}) {
+export async function createDesktopController({ scope = "window", app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", focusOnConnect = false, client: injectedClient = null, accessibility: injectedAccessibility = null, visibleBoundsReader = getWindowsVisibleBounds, displayCapture = capturePrimaryDisplay, } = {}) {
     if (!SUPPORTED_PLATFORMS.has(platform)) {
         throw new Error("Desktop control is currently supported only on macOS and Windows.");
     }
-    if (!requestedApp)
+    if (scope !== "window" && scope !== "desktop")
+        throw new Error('Desktop scope must be "window" or "desktop".');
+    if (scope === "desktop" && (requestedApp || requestedWindowId != null))
+        throw new Error("Full desktop cannot select an application or window.");
+    if (scope === "desktop" && inputMode !== "physical")
+        throw new Error("Full desktop requires physical input.");
+    if (scope === "window" && !requestedApp)
         throw new Error("--desktop-app is required with --target desktop.");
     if (!INPUT_MODES.has(inputMode))
         throw new Error('Desktop input mode must be "accessibility" or "physical".');
     const client = injectedClient || await createLibraryClient();
     try {
-        const apps = (await discoverApplications(client)).filter((app) => typeof app.bundleId === "string" && app.bundleId && !excludePids.includes(app.pid));
-        const application = chooseApplication(apps, requestedApp);
-        const windows = readWindows(await client.listWindows(application.bundleId));
-        let targetWindow = chooseWindow(windows, requestedWindowId == null ? null : Number(requestedWindowId));
+        const application = scope === "window" ? chooseApplication((await discoverApplications(client)).filter((app) => typeof app.bundleId === "string" && app.bundleId && !excludePids.includes(app.pid)), requestedApp) : null;
+        let targetWindow = application ? chooseWindow(readWindows(await client.listWindows(application.bundleId)), requestedWindowId == null ? null : Number(requestedWindowId)) : null;
         let capture = null;
         let deviceInfo = null;
         let native = injectedAccessibility;
@@ -260,19 +265,38 @@ export async function createDesktopController({ app: requestedApp, windowId: req
         async function getWindow() {
             return parseJsonResult(await client.getWindow(targetWindow.windowId), "Window lookup");
         }
-        async function focus() {
+        async function refreshWindow() {
+            const windows = readWindows(await client.listWindows(application.bundleId))
+                .filter((window) => window.isOnScreen && window.bounds?.width > 0 && window.bounds?.height > 0);
+            const selected = windows.find((window) => window.isFocused)
+                || windows.find((window) => window.windowId === targetWindow.windowId)
+                || chooseWindow(windows, null);
+            if (selected.windowId !== targetWindow.windowId)
+                capture = null;
+            targetWindow = selected;
+            if (deviceInfo) {
+                deviceInfo.desktopWindowId = selected.windowId;
+                deviceInfo.device_name = `${application.displayName} — ${selected.title || "Untitled"}`;
+            }
+        }
+        async function activateWindow() {
             const activation = parseJsonResult(await client.activateWindow(targetWindow.windowId, 2000), "Window activation");
-            const focused = await getWindow();
-            if (!activation.activated || !focused.isFocused) {
-                throw new Error("Droid could not focus the selected desktop window, so no input was sent.");
+            if (!activation.activated || !(await getWindow()).isFocused) {
+                throw new Error(`Droid could not focus the selected desktop window, so no input was sent.${activation.reason ? ` Reason: ${activation.reason}.` : ""}`);
             }
         }
+        async function focus() {
+            if (scope === "desktop")
+                return;
+            await refreshWindow();
+            await activateWindow();
+        }
         async function captureFromDesktop(window) {
             if (!window.isFocused) {
                 if (inputMode !== "physical") {
                     throw new Error("Windows desktop capture requires the target window to be visible and focused. Bring it to the foreground and retry.");
                 }
-                await focus();
+                await activateWindow();
                 window = await getWindow();
             }
             native ||= await loadNativeDriver();
@@ -293,29 +317,38 @@ export async function createDesktopController({ app: requestedApp, windowId: req
         }
         async function screenshot({ allowDesktopFallback = true } = {}) {
             try {
-                const currentWindows = readWindows(await client.listWindows(application.bundleId));
-                const focusedWindow = currentWindows.find((candidate) => candidate.isFocused && candidate.isOnScreen);
-                if (focusedWindow)
-                    targetWindow = focusedWindow;
-                const window = await getWindow();
-                if (!window.isOnScreen)
-                    throw new Error("The selected desktop window is no longer visible.");
-                const result = await client.screenshot({
-                    width: SCREENSHOT_WIDTH,
-                    quality: 0,
-                    target_window_id: targetWindow.windowId,
-                });
-                assertResult(result, "Desktop screenshot");
-                let image = { ...readScreenshot(result), bounds: { ...window.bounds }, actionBounds: { ...window.bounds } };
-                if (allowDesktopFallback && platform === "win32" && await isBlackScreenshot(image.data)) {
-                    image = await captureFromDesktop(window);
+                let image;
+                if (scope === "desktop") {
+                    native ||= await loadNativeDriver();
+                    const screen = await displayCapture(native, platform);
+                    image = { ...screen, actionBounds: { ...screen.bounds } };
+                }
+                else {
+                    await refreshWindow();
+                    const window = await getWindow();
+                    if (!window.isOnScreen)
+                        throw new Error("The selected desktop window is no longer visible.");
+                    const result = await client.screenshot({
+                        width: SCREENSHOT_WIDTH,
+                        quality: 0,
+                        target_window_id: targetWindow.windowId,
+                    });
+                    assertResult(result, "Desktop screenshot");
+                    image = { ...readScreenshot(result), bounds: { ...window.bounds }, actionBounds: { ...window.bounds } };
+                    if (allowDesktopFallback && platform === "win32" && await isBlackScreenshot(image.data)) {
+                        image = await captureFromDesktop(window);
+                    }
                 }
                 capture = {
                     bounds: image.bounds,
                     actionBounds: image.actionBounds,
                     width: image.width,
                     height: image.height,
-                    contentBounds: getScreenshotContentBounds(image.width, image.height, image.bounds, platform),
+                    windowId: targetWindow?.windowId ?? null,
+                    displayId: image.displayId,
+                    contentBounds: scope === "desktop"
+                        ? { x: 0, y: 0, width: image.width, height: image.height }
+                        : getScreenshotContentBounds(image.width, image.height, image.bounds, platform),
                 };
                 if (deviceInfo) {
                     deviceInfo.device_width = image.actionBounds.width;
@@ -323,7 +356,7 @@ export async function createDesktopController({ app: requestedApp, windowId: req
                     deviceInfo.scaled_width = image.width;
                     deviceInfo.scaled_height = image.height;
                     deviceInfo.scale = image.width / image.actionBounds.width;
-                    deviceInfo.desktopWindowId = targetWindow.windowId;
+                    deviceInfo.desktopWindowId = targetWindow?.windowId ?? null;
                 }
                 return image.data;
             }
@@ -334,25 +367,77 @@ export async function createDesktopController({ app: requestedApp, windowId: req
         function scaledPoint(x, y) {
             if (!capture)
                 throw new Error("Capture the desktop window before executing an action.");
+            if (scope === "desktop" && (!Number.isFinite(x) || !Number.isFinite(y) || x < 0 || y < 0 || x >= capture.width || y >= capture.height)) {
+                throw new Error("The action is outside the primary display.");
+            }
             const content = capture.contentBounds;
             const bounds = capture.actionBounds;
-            return [
+            const point = [
                 Math.round(bounds.x + ((x - content.x) * bounds.width) / content.width),
                 Math.round(bounds.y + ((y - content.y) * bounds.height) / content.height),
             ];
+            // Retina rounding at the last image pixel must stay inside the display.
+            return scope === "desktop" ? [Math.min(point[0], bounds.width - 1), Math.min(point[1], bounds.height - 1)] : point;
         }
         async function prepareAction(requireFocus) {
             if (!capture)
-                throw new Error("Capture the desktop window before executing an action.");
-            const before = await getWindow();
-            if (!sameBounds(before.bounds, capture.bounds)) {
+                throw new Error("A fresh desktop screenshot is required before acting.");
+            if (scope === "desktop") {
+                const display = native.getDisplaySize(capture.displayId);
+                if (display.width !== capture.bounds.width || display.height !== capture.bounds.height) {
+                    throw new Error("The display resized after the screenshot. A fresh screenshot is required before acting.");
+                }
+                return;
+            }
+            let window;
+            try {
+                window = parseJsonResult(await client.getWindow(capture.windowId), "Window lookup");
+            }
+            catch {
+                throw new Error("The captured desktop window is no longer available. A fresh screenshot is required before acting.");
+            }
+            if (!window.isOnScreen || window.bundleId !== application.bundleId || capture.windowId !== targetWindow.windowId) {
+                throw new Error("The captured desktop window is no longer available. A fresh screenshot is required before acting.");
+            }
+            if (!sameBounds(window.bounds, capture.bounds)) {
                 throw new Error("The desktop window moved or resized after the screenshot. A fresh screenshot is required before acting.");
             }
-            if (!requireFocus)
-                return;
-            await focus();
+            if (requireFocus) {
+                await activateWindow();
+                if (!sameBounds((await getWindow()).bounds, capture.bounds)) {
+                    throw new Error("The desktop window moved or resized during focus. A fresh screenshot is required before acting.");
+                }
+            }
         }
-        const targetOptions = () => ({ targetWindowId: targetWindow.windowId, focusStrategy: "strict" });
+        const targetOptions = () => scope === "desktop"
+            ? { focusStrategy: "none" }
+            : { targetWindowId: targetWindow.windowId, focusStrategy: "strict" };
+        // Native pointer calls avoid the library's first-monitor coordinate validation.
+        // Keyboard calls retain its text/clipboard handling, with app focus disabled.
+        const physical = scope === "window" ? client : {
+            click: (x, y) => native.mouseClick(x, y, "left", 1),
+            doubleClick: (x, y) => native.mouseClick(x, y, "left", 2),
+            moveMouse: (x, y) => native.mouseMove(x, y),
+            drag: async (to, from) => {
+                native.mouseMove(from[0], from[1]);
+                await new Promise((resolve) => setTimeout(resolve, 50));
+                native.mouseButton("press", from[0], from[1]);
+                try {
+                    await new Promise((resolve) => setTimeout(resolve, 50));
+                    native.mouseDrag(to[0], to[1]);
+                    await new Promise((resolve) => setTimeout(resolve, 50));
+                }
+                finally {
+                    native.mouseButton("release", to[0], to[1]);
+                }
+            },
+            scroll: (x, y, direction, amount) => {
+                native.mouseMove(x, y);
+                native.mouseScroll(direction === "down" ? amount : direction === "up" ? -amount : 0, direction === "right" ? amount : direction === "left" ? -amount : 0);
+            },
+            type: client.type,
+            key: client.key,
+        };
         async function performCursorFreeAction(actionType, x, y, direction = null) {
             const tree = parseJsonResult(await client.getUiTree(targetWindow.windowId, 20), "Accessibility tree");
             const selected = chooseAccessibilityElement(tree, x, y, actionType, direction);
@@ -385,8 +470,8 @@ export async function createDesktopController({ app: requestedApp, windowId: req
                         }
                         else {
                             const result = action.type === "click"
-                                ? await client.click(x, y, application.bundleId, targetOptions())
-                                : await client.doubleClick(x, y, application.bundleId, targetOptions());
+                                ? await physical.click(x, y, application?.bundleId, targetOptions())
+                                : await physical.doubleClick(x, y, application?.bundleId, targetOptions());
                             assertResult(result, "Desktop click");
                         }
                         break;
@@ -396,7 +481,7 @@ export async function createDesktopController({ app: requestedApp, windowId: req
                         addOutput({ type: "action", text: `Moving mouse to (${x}, ${y})`, ...meta({ x, y, unit: "px" }) });
                         if (inputMode === "accessibility")
                             throw new Error(cursorFreeError(action.type));
-                        assertResult(await client.moveMouse(x, y, application.bundleId, targetOptions()), "Desktop mouse move");
+                        assertResult(await physical.moveMouse(x, y, application?.bundleId, targetOptions()), "Desktop mouse move");
                         break;
                     }
                     case "drag": {
@@ -408,7 +493,7 @@ export async function createDesktopController({ app: requestedApp, windowId: req
                         addOutput({ type: "action", text: `Dragging from (${from[0]}, ${from[1]}) to (${to[0]}, ${to[1]})`, ...meta({ pathStart: { x: from[0], y: from[1] }, pathEnd: { x: to[0], y: to[1] }, unit: "px" }) });
                         if (inputMode === "accessibility")
                             throw new Error(cursorFreeError(action.type));
-                        assertResult(await client.drag(to, from, application.bundleId, targetOptions()), "Desktop drag");
+                        assertResult(await physical.drag(to, from, application?.bundleId, targetOptions()), "Desktop drag");
                         break;
                     }
                     case "scroll": {
@@ -423,20 +508,20 @@ export async function createDesktopController({ app: requestedApp, windowId: req
                             await performCursorFreeAction(action.type, x, y, direction);
                         }
                         else {
-                            assertResult(await client.scroll(x, y, direction, amount, application.bundleId, targetOptions()), "Desktop scroll");
+                            assertResult(await physical.scroll(x, y, direction, amount, application?.bundleId, targetOptions()), "Desktop scroll");
                         }
                         break;
                     }
                     case "type": {
                         const text = typeof action.text === "string" ? action.text : "";
                         addOutput({ type: "action", text: buildTypeActionText(action, text), ...meta(buildTypeActionPayload(action, text)) });
-                        assertResult(await client.type(text, application.bundleId, targetOptions()), "Desktop typing");
+                        assertResult(await physical.type(text, application?.bundleId, targetOptions()), "Desktop typing");
                         break;
                     }
                     case "keypress": {
                         const shortcut = normalizeShortcut(action.keys, platform);
                         addOutput({ type: "action", text: `Pressing key: ${shortcut}`, ...meta({ keys: action.keys, shortcut }) });
-                        assertResult(await client.key(shortcut, application.bundleId, targetOptions()), "Desktop keypress");
+                        assertResult(await physical.key(shortcut, application?.bundleId, targetOptions()), "Desktop keypress");
                         break;
                     }
                     case "wait": {
@@ -463,20 +548,21 @@ export async function createDesktopController({ app: requestedApp, windowId: req
         deviceInfo = {
             platform: "desktop",
             source: "local",
-            device_name: `${application.displayName} — ${targetWindow.title || "Untitled"}`,
+            device_name: application ? `${application.displayName} — ${targetWindow.title || "Untitled"}` : "Full desktop",
             device_width: capture.actionBounds.width,
             device_height: capture.actionBounds.height,
             scaled_width: capture.width,
             scaled_height: capture.height,
             scale: capture.width / capture.actionBounds.width,
-            desktopAppId: application.bundleId,
-            desktopWindowId: targetWindow.windowId,
+            desktopScope: scope,
+            desktopAppId: application?.bundleId ?? null,
+            desktopWindowId: targetWindow?.windowId ?? null,
             desktopInputMode: inputMode,
         };
         return {
             application,
-            window: targetWindow,
-            deviceId: `desktop:${targetWindow.windowId}`,
+            get window() { return targetWindow; },
+            deviceId: scope === "desktop" ? "desktop:primary" : `desktop:${targetWindow.windowId}`,
             deviceInfo,
             focus,
             getScreenshotAsBase64: () => screenshot(),

package/build/src/desktop/primary-display.js

@@ -0,0 +1,74 @@
+import { execFile } from "node:child_process";
+import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { promisify } from "node:util";
+import sharp from "sharp";
+const runFile = promisify(execFile);
+const WINDOWS_CAPTURE = `
+$ErrorActionPreference = 'Stop'
+Add-Type -AssemblyName System.Drawing
+Add-Type -TypeDefinition @'
+using System;
+using System.Runtime.InteropServices;
+public static class PrimaryDisplay {
+  [StructLayout(LayoutKind.Sequential)] public struct Point { public int X; public int Y; }
+  [DllImport("user32.dll")] public static extern bool SetProcessDPIAware();
+  [DllImport("user32.dll")] public static extern int GetSystemMetrics(int index);
+  [DllImport("user32.dll")] public static extern IntPtr MonitorFromPoint(Point point, uint flags);
+}
+'@
+[void][PrimaryDisplay]::SetProcessDPIAware()
+$width = [PrimaryDisplay]::GetSystemMetrics(0)
+$height = [PrimaryDisplay]::GetSystemMetrics(1)
+$bitmap = [System.Drawing.Bitmap]::new($width, $height)
+$graphics = [System.Drawing.Graphics]::FromImage($bitmap)
+try {
+  $graphics.CopyFromScreen(0, 0, 0, 0, $bitmap.Size)
+  $bitmap.Save('CAPTURE_PATH', [System.Drawing.Imaging.ImageFormat]::Png)
+  # The native driver's display lookup accepts a uint32 monitor handle.
+  [PrimaryDisplay]::MonitorFromPoint([PrimaryDisplay+Point]::new(), 1).ToInt64() -band 0xffffffff
+} finally {
+  $graphics.Dispose()
+  $bitmap.Dispose()
+}
+`;
+/** Capture only the primary display, independently of the library's cached window target. */
+export async function capturePrimaryDisplay(native, platform = process.platform, run = runFile) {
+    const directory = await mkdtemp(join(tmpdir(), "droid-desktop-"));
+    const path = join(directory, "screen.png");
+    try {
+        let displayId;
+        if (platform === "darwin") {
+            await run("/usr/sbin/screencapture", ["-x", "-m", "-t", "png", path], { timeout: 10000 });
+        }
+        else if (platform === "win32") {
+            const script = WINDOWS_CAPTURE.replace("CAPTURE_PATH", path.replaceAll("'", "''"));
+            const { stdout } = await run("powershell.exe", ["-NoProfile", "-NonInteractive", "-Command", script], {
+                windowsHide: true,
+                timeout: 10000,
+            });
+            displayId = Number(stdout.trim());
+            if (!Number.isSafeInteger(displayId) || displayId <= 0)
+                throw new Error("Primary display lookup failed.");
+        }
+        else {
+            throw new Error("Desktop control is currently supported only on macOS and Windows.");
+        }
+        // Use the native driver's coordinate units, which can differ from screenshot pixels.
+        const display = native.getDisplaySize(displayId);
+        const { data, info } = await sharp(await readFile(path))
+            .resize({ width: 1440, withoutEnlargement: true })
+            .png().toBuffer({ resolveWithObject: true });
+        return {
+            data: data.toString("base64"),
+            width: info.width,
+            height: info.height,
+            bounds: { x: 0, y: 0, width: display.width, height: display.height },
+            displayId: displayId ?? display.displayId,
+        };
+    }
+    finally {
+        await rm(directory, { recursive: true, force: true });
+    }
+}

package/build/src/prompts/base.js

@@ -46,6 +46,8 @@ function describeControlledTarget(deviceInfo = {}) {
         return browserName ? `a browser session (${browserName})` : "a browser session";
     }
     if (platform === "desktop") {
+        if (deviceInfo.desktopScope === "desktop")
+            return "the full primary desktop display";
         return deviceName ? `a desktop application (${deviceName})` : "a desktop application";
     }
     if (platform === "ios") {
@@ -255,11 +257,12 @@ function buildMobileBaseSystemPrompt(deviceInfo, options = {}) {
   `;
 }
 function buildDesktopBaseSystemPrompt(deviceInfo, options = {}) {
+    const fullDesktop = deviceInfo.desktopScope === "desktop";
     let browserPrompt = buildBrowserBaseSystemPrompt(deviceInfo, options)
-        .replace("Follow the user's instructions to interact with the browser.", "Follow the user's instructions to interact with the selected application.")
-        .replace("visible browser window", "selected application window")
+        .replace("Follow the user's instructions to interact with the browser.", fullDesktop ? "Follow the user's instructions to interact with applications on the primary display." : "Follow the user's instructions to interact with the selected application.")
+        .replace("visible browser window", fullDesktop ? "primary desktop display" : "selected application window")
         .replace("CRITICAL - Desktop Browser Input Constraints:", "CRITICAL - Desktop Input Constraints:")
-        .replace("This is a desktop browser environment.", "This is a desktop application environment.");
+        .replace("This is a desktop browser environment.", fullDesktop ? "This is a full desktop environment." : "This is a desktop application environment.");
     if (deviceInfo.desktopInputMode !== "physical") {
         browserPrompt = browserPrompt
             .replace("Available actions: click, scroll, type, keypress, wait, screenshot.", "Available actions: click, scroll, wait, screenshot.")
@@ -274,10 +277,10 @@ function buildDesktopBaseSystemPrompt(deviceInfo, options = {}) {
         ? "Droid may use the physical OS pointer for this run"
         : "Cursor-free mode never focuses the application: use only clicks and scrolling on controls with matching accessibility actions; never emit type, keypress, move, hover, or drag";
     return `${withoutBrowserNotes}  Desktop-Specific Notes:
-  - Droid is controlling only the selected application window, not the whole display
+  - ${fullDesktop ? "Droid is controlling the whole primary display across applications. Applications launched during the test are available once visible." : "Droid is controlling only the selected application window, not the whole display"}
   - ${inputNote}
-  - Do not use shortcuts that switch applications, virtual desktops, or operating-system context
-  - Dialogs owned by the selected application may be controlled when they become its active window
+  - ${fullDesktop ? "Click visible controls to focus the intended application before typing. Application switching is allowed when needed for the task; remain on the primary display and current virtual desktop." : "Do not use shortcuts that switch applications, virtual desktops, or operating-system context"}
+  - ${fullDesktop ? "Windows may open or close during the run. Use the latest screenshot to choose the next action." : "Dialogs owned by the selected application may be controlled when they become its active window"}
   `;
 }
 function buildBaseSystemPrompt(deviceInfo, options = {}) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@loadmill/droid-cua",
-  "version": "2.40.0",
+  "version": "2.41.0",
   "description": "AI-powered Android testing agent using OpenAI's computer-use model and ADB",
   "main": "build/index.js",
   "type": "module",

package/skill/droid-cua/SKILL.md

@@ -2,7 +2,7 @@
 name: droid-cua
 description: Operate and improve Droid CUA `.dcua` test projects. Use when the user explicitly names Droid CUA or @loadmill/droid-cua, or asks to author, configure, run, diagnose, heal, stabilize, or speed up `.dcua` tests with the Droid CUA CLI or desktop app. Do not use for general web browsing, website development, browser automation, generic mobile work, or non-Droid test tasks.
 metadata:
-  version: "1.0.8"
+  version: "1.0.10"
 ---
 
 # Droid CUA

package/skill/droid-cua/references/running-and-healing.md

@@ -20,10 +20,14 @@ All Loadmill models require `LOADMILL_API_TOKEN`. Smart and Pulse now route thro
 
 Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
 
-For macOS and Windows desktop tests, the desktop app shows a system notice for three seconds before Droid takes control. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. Droid brings the selected target window to the foreground before the first capture. When the run ends, the target stays in front and a second system notice says Droid has finished controlling the desktop. Open Droid to review the result. Some Windows apps return an all-black window capture; Droid brings the target back to the foreground before cropping the visible desktop to its visible frame, excluding the invisible Windows resize border. The fallback logs an all-black capture warning and crops before resizing to preserve detail. A visible-frame lookup warning means Droid is using the outer window bounds, which may include invisible resize borders. Keep the target visible and unobstructed on the primary display. In accessibility input mode, bring the target to the foreground yourself if the fallback requests it; Droid does not focus it for screenshots. If a screenshot shows another app or black off-screen edges, check whether the target could be focused and fully displayed before changing the `.dcua` instructions.
+For macOS and Windows desktop tests in application mode, the desktop app shows a system notice for three seconds before Droid takes control. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. Droid brings the selected target window to the foreground before the first capture. When the run ends, the target stays in front and a second system notice says Droid has finished controlling the desktop. Open Droid to review the result. Some Windows apps return an all-black window capture; Droid brings the target back to the foreground before cropping the visible desktop to its visible frame, excluding the invisible Windows resize border. The fallback logs an all-black capture warning and crops before resizing to preserve detail. A visible-frame lookup warning means Droid is using the outer window bounds, which may include invisible resize borders. Keep the target visible and unobstructed on the primary display. In accessibility input mode, bring the target to the foreground yourself if the fallback requests it; Droid does not focus it for screenshots. If a screenshot shows another app or black off-screen edges, check whether the target could be focused and fully displayed before changing the `.dcua` instructions.
 
 When downstream automation needs an exact Loadmill handoff, add `--report-metadata reports/<test-name>.json`. Read it only after Droid exits. Droid clears the destination at run start and writes the saved run ID plus the last uploaded screenshot object name only after cloud report publication succeeds, including when a failed test's report was published. Do not infer the run or screenshot by searching recent Loadmill runs when this file was requested but absent.
 
+For launcher-to-app or other cross-application workflows on macOS and Windows, choose **Full desktop** in the Application dropdown, or use `droid-cua --target desktop --instructions tests/launcher.dcua --debug`. CLI desktop runs use full desktop by default when no `--desktop-app` is supplied. Full desktop controls only the primary display and requires physical input. Keep all test windows on the primary display. The target application can be launched during the test; it does not need to exist when connecting. Add `--desktop-app <id-or-exact-name>` to limit control to one running application. `--desktop-window` and `--desktop-input accessibility` require `--desktop-app`. Use `--target desktop --list-desktop-apps` to discover applications and windows without connecting. In the desktop app's full desktop mode, Droid minimizes itself after the three-second notice and remains minimized after completion; reopen Droid to review results or use Stop. CLI runs do not minimize Droid or show system notices. Click the intended application before typing. Application mode refreshes window selection before focusing or capturing, so recreated login windows can be followed. If replacement is ambiguous, select a window again. If a window disappears or moves after capture, get a fresh screenshot before acting. A generic focus failure is not proof of an elevation problem; inspect the activation reason first.
+
+Desktop Test design shows the startup notice after the user submits instructions, before exploration captures or controls the screen. Full desktop minimizes Droid then; opening the design screen alone does not minimize it. Completion, Stop, and an unrecoverable exploration error send the finish notice without bringing Droid forward. The user can open Droid to give guidance or review the generated test. When an action is rejected, the remaining batch is skipped and the model receives a fresh screenshot and the failure reason; the rejected action is not counted as completed.
+
 For desktop runs, treat the test result and dashboard report as separate outcomes. A run can appear in Droid Runs while its report is still uploading. A `report.upload.skipped` entry in the app's `desktop-events.jsonl` explains when sharing was disabled or authentication was unavailable. For failures, use the run warning and `report.upload.failed` entry, which includes the upload stage and HTTP status when available. The absence of a failure entry does not prove that a still-unavailable report was published; it may still be uploading. Direct the user to the local report in Droid when sharing did not complete. Do not infer the cause from an unclickable dashboard row alone.
 
 ```sh