@loadmill/droid-cua 2.39.0 → 2.40.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -158,8 +158,13 @@ droid-cua skill install codex
158
158
  droid-cua skill install codex --local
159
159
  ```
160
160
 
161
- Use `droid-cua skill status <agent>` to inspect an installation and
162
- `droid-cua skill uninstall <agent>` to remove it. Installed skill
161
+ Use `droid-cua skill status <agent>` to see the installed and bundled skill
162
+ versions and installation path; add `--local` to inspect a project installation.
163
+ The skill version is stored in `SKILL.md` under `metadata.version`, independently
164
+ of the CLI package version. Older copies without version metadata show `unknown`.
165
+ The `modified` status means the files differ from the bundled copy, which can
166
+ indicate an older release or local edits even when version labels match.
167
+ Use `droid-cua skill uninstall <agent>` to remove an installation. Installed skill
163
168
  files are never overwritten or removed after local modification unless `--force`
164
169
  is supplied. After upgrading the npm package, update an installed skill explicitly:
165
170
 
@@ -239,7 +244,7 @@ backward compatibility.
239
244
  Example headless config:
240
245
  ```json
241
246
  {
242
- "cuaModel": "gpt-5.4",
247
+ "cuaModel": "gpt-6-sol",
243
248
  "promptCustomizations": {
244
249
  "basePromptInstructions": "",
245
250
  "designModeInstructions": "",
@@ -295,10 +300,13 @@ Supported CLI options include:
295
300
  - `--debug`
296
301
 
297
302
  `--cua-model` accepts `loadmill-smart`, `loadmill-pulse`, `loadmill-beacon`,
298
- `gpt-5.6-terra`, `gpt-5.6-luna`, and `gpt-5.4`. OpenAI models use direct OpenAI
299
- mode and require `OPENAI_API_KEY`.
303
+ `gpt-6-sol`, and `gpt-6-luna`. OpenAI models use direct OpenAI mode and require
304
+ `OPENAI_API_KEY`. Saved direct OpenAI selections of GPT-5.6 Terra, GPT-5.6 Luna,
305
+ or GPT-5.4 move to GPT-6 Sol, GPT-6 Luna, or GPT-6 Sol, respectively.
300
306
 
301
- Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default: they capture the selected window and send physical pointer and keyboard input at the model's coordinates. Use `--desktop-input accessibility` for cursor-free CLI input instead. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. Windows targets must run at the same integrity level as Droid; a non-elevated Droid process cannot inject input into an elevated application.
307
+ Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default: Droid shows a system notice for three seconds before taking control, brings the selected window to the foreground, then sends physical pointer and keyboard input at the model's coordinates. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. When the run ends, Droid leaves the target window in front and shows another system notice that control has finished, including when the run fails or is stopped. CLI runs also leave the target in front. Use `--desktop-input accessibility` for cursor-free CLI input instead. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. Windows targets must run at the same integrity level as Droid; a non-elevated Droid process cannot inject input into an elevated application. If Windows returns an all-black window capture, Droid brings the target back to the foreground and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The crop is resized only after extraction to preserve window detail. The fallback logs a warning and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target to the foreground yourself when requested.
308
+
309
+ The system notices are shown only for desktop control runs started in the desktop app. On macOS, they may not appear in the unsigned Electron dev app; verify them in a signed packaged build.
302
310
 
303
311
  Config and precedence rules:
304
312
  - Use `--config <file>` to supply prompt-affecting settings for headless runs.
package/build/index.js CHANGED
@@ -537,6 +537,7 @@ async function main() {
537
537
  app: targetConfig.desktopApp,
538
538
  windowId: targetConfig.desktopWindow,
539
539
  inputMode: targetConfig.desktopInput,
540
+ focusOnConnect: true,
540
541
  excludePids: [process.pid],
541
542
  });
542
543
  targetId = desktopController.deviceId;
@@ -2,7 +2,7 @@ import path from "path";
2
2
  import { readFile } from "fs/promises";
3
3
  import { DEFAULT_CONTEXT_OPTIMIZATION_THRESHOLD, MAX_CONTEXT_OPTIMIZATION_THRESHOLD, MIN_CONTEXT_OPTIMIZATION_THRESHOLD, } from "../core/context-optimization.js";
4
4
  import { AIM_ASSIST_MODES, DEFAULT_AIM_ASSIST_MODE } from "../core/aim-assist.js";
5
- import { CUA_MODELS, DEFAULT_OPENAI_CUA_MODEL, isOpenAiCuaModel, isValidCuaModel, resolveCuaModel } from "../core/cua-models.js";
5
+ import { CUA_MODELS, DEFAULT_OPENAI_CUA_MODEL, isOpenAiCuaModel, isValidCuaModel, resolveCuaModel, upgradeLegacyCuaModel } from "../core/cua-models.js";
6
6
  import { LLM_PROVIDER_MODES, isValidLlmProviderMode, resolveLlmProviderMode } from "../core/runtime-config.js";
7
7
  import { parseSecretsOverrideString } from "../core/secrets.js";
8
8
  function createEmptyPromptCustomizations() {
@@ -21,10 +21,11 @@ function assertNonEmptyString(value, label) {
21
21
  }
22
22
  }
23
23
  function validateCuaModel(value, label) {
24
- if (typeof value !== "string" || !isValidCuaModel(value)) {
24
+ const upgraded = typeof value === "string" ? upgradeLegacyCuaModel(value) : value;
25
+ if (!isValidCuaModel(upgraded)) {
25
26
  throw new Error(`${label} must be one of: ${CUA_MODELS.join(", ")}.`);
26
27
  }
27
- return value;
28
+ return upgraded;
28
29
  }
29
30
  function validateAimAssistMode(value, label) {
30
31
  if (typeof value !== "string" || !AIM_ASSIST_MODES.includes(value)) {
@@ -44,8 +45,8 @@ function readEnvDefaultCuaModel() {
44
45
  // OpenAI model, stay on the OpenAI default rather than crossing over to the
45
46
  // global (Loadmill) default.
46
47
  if (typeof envValue === "string" && envValue.trim()) {
47
- const trimmed = envValue.trim();
48
- return isOpenAiCuaModel(trimmed) ? trimmed : DEFAULT_OPENAI_CUA_MODEL;
48
+ const upgraded = upgradeLegacyCuaModel(envValue.trim());
49
+ return isOpenAiCuaModel(upgraded) ? upgraded : DEFAULT_OPENAI_CUA_MODEL;
49
50
  }
50
51
  return resolveCuaModel(envValue);
51
52
  }
@@ -12,7 +12,7 @@ const SUBCOMMANDS = new Set(["install", "status", "uninstall"]);
12
12
  const ALLOWED_OPTIONS = new Set(["_", "local", "force", "h", "help"]);
13
13
  const COMMAND_DESCRIPTIONS = Object.freeze({
14
14
  install: "Install Droid CUA skill for a coding agent",
15
- status: "Show Droid CUA skill status for a coding agent",
15
+ status: "Show installation status and installed/bundled skill versions",
16
16
  uninstall: "Uninstall Droid CUA skill for a coding agent",
17
17
  });
18
18
  function skillCommandHelp(subcommand) {
@@ -53,7 +53,7 @@ function skillRootHelp() {
53
53
  "",
54
54
  "Commands:",
55
55
  " install Install the skill",
56
- " status Show installation status",
56
+ " status Show installation status and skill versions",
57
57
  " uninstall Uninstall the skill",
58
58
  ].join("\n");
59
59
  }
@@ -103,6 +103,18 @@ async function directoryMatches(sourcePath, destinationPath) {
103
103
  ]);
104
104
  return JSON.stringify(sourceSnapshot) === JSON.stringify(destinationSnapshot);
105
105
  }
106
+ // Read the bundled skill's block-style metadata.version scalar without a YAML dependency.
107
+ async function readSkillVersion(skillPath) {
108
+ if (await pathType(skillPath) !== "directory")
109
+ return null;
110
+ if (await pathType(path.join(skillPath, "SKILL.md")) !== "file")
111
+ return null;
112
+ const markdown = await readFile(path.join(skillPath, "SKILL.md"), "utf8");
113
+ const frontmatter = markdown.replace(/\r\n/g, "\n").match(/^---\n([\s\S]*?)\n---(?:\n|$)/)?.[1];
114
+ const metadata = frontmatter?.match(/^metadata:[ \t]*\n((?:[ \t]+[^\n]*(?:\n|$))*)/m)?.[1];
115
+ const value = metadata?.match(/^ version:[ \t]*(?:"([^"\n]+)"|'([^'\n]+)'|([^\s#]+))[ \t]*(?:#.*)?$/m);
116
+ return value ? (value[1] ?? value[2] ?? value[3]) : null;
117
+ }
106
118
  async function getInstallationState(sourcePath, destinationPath) {
107
119
  if (await pathType(destinationPath) === "missing")
108
120
  return "not-installed";
@@ -214,7 +226,12 @@ export async function runSkillCommand({ args, cwd = process.cwd(), home = os.hom
214
226
  }
215
227
  for (const destination of states) {
216
228
  if (command.subcommand === "status") {
217
- output.log(`${destination.displayName}: ${stateLabel(destination.state)} at ${destination.path}`);
229
+ const [installedVersion, bundledVersion] = await Promise.all([
230
+ readSkillVersion(destination.path),
231
+ readSkillVersion(bundledSkillPath),
232
+ ]);
233
+ const installedLabel = destination.state === "not-installed" ? "not installed" : installedVersion ?? "unknown";
234
+ output.log(`${destination.displayName}: ${stateLabel(destination.state)} at ${destination.path}\nInstalled skill version: ${installedLabel}\nBundled skill version: ${bundledVersion ?? "unknown"}`);
218
235
  }
219
236
  else if (command.subcommand === "install") {
220
237
  if (destination.state === "installed") {
@@ -1,4 +1,4 @@
1
- export const DEFAULT_OPENAI_CUA_MODEL = "gpt-5.4";
1
+ export const DEFAULT_OPENAI_CUA_MODEL = "gpt-6-sol";
2
2
  export const DEFAULT_LOADMILL_CUA_MODEL = "loadmill-smart";
3
3
  export const DEFAULT_CUA_MODEL = DEFAULT_LOADMILL_CUA_MODEL;
4
4
  export const CUA_CONVERSATION_POLICIES = Object.freeze({
@@ -28,27 +28,28 @@ export const CUA_MODEL_OPTIONS = Object.freeze([
28
28
  label: "Loadmill - Beacon",
29
29
  }),
30
30
  Object.freeze({
31
- id: "gpt-5.6-terra",
31
+ id: "gpt-6-sol",
32
32
  providerMode: "openai",
33
33
  providerLabel: "OpenAI",
34
- name: "GPT-5.6 Terra",
35
- label: "OpenAI GPT-5.6 Terra",
34
+ name: "GPT-6 Sol",
35
+ label: "OpenAI GPT-6 Sol",
36
36
  }),
37
37
  Object.freeze({
38
- id: "gpt-5.6-luna",
38
+ id: "gpt-6-luna",
39
39
  providerMode: "openai",
40
40
  providerLabel: "OpenAI",
41
- name: "GPT-5.6 Luna",
42
- label: "OpenAI GPT-5.6 Luna",
43
- }),
44
- Object.freeze({
45
- id: "gpt-5.4",
46
- providerMode: "openai",
47
- providerLabel: "OpenAI",
48
- name: "GPT-5.4",
49
- label: "OpenAI GPT-5.4",
41
+ name: "GPT-6 Luna",
42
+ label: "OpenAI GPT-6 Luna",
50
43
  }),
51
44
  ]);
45
+ const LEGACY_OPENAI_CUA_MODELS = Object.freeze({
46
+ "gpt-5.6-terra": "gpt-6-sol",
47
+ "gpt-5.6-luna": "gpt-6-luna",
48
+ "gpt-5.4": "gpt-6-sol",
49
+ });
50
+ export function upgradeLegacyCuaModel(value) {
51
+ return LEGACY_OPENAI_CUA_MODELS[value] ?? value;
52
+ }
52
53
  export const CUA_MODELS = Object.freeze(CUA_MODEL_OPTIONS.map((option) => option.id));
53
54
  export const OPENAI_CUA_MODELS = Object.freeze(CUA_MODEL_OPTIONS
54
55
  .filter((option) => option.providerMode === "openai")
@@ -70,7 +71,8 @@ export function isLoadmillCuaModel(value) {
70
71
  return LOADMILL_CUA_MODEL_SET.has(value);
71
72
  }
72
73
  export function resolveCuaModel(value) {
73
- return isValidCuaModel(value) ? value : DEFAULT_CUA_MODEL;
74
+ const upgraded = upgradeLegacyCuaModel(value);
75
+ return isValidCuaModel(upgraded) ? upgraded : DEFAULT_CUA_MODEL;
74
76
  }
75
77
  export function resolveCuaModelProviderMode(value) {
76
78
  const normalized = resolveCuaModel(value);
@@ -3,29 +3,26 @@ const OPENAI_GPT_5_4_PRICING = Object.freeze({
3
3
  inputPerMillionUsd: 2.5,
4
4
  });
5
5
  const OPENAI_GPT_5_6_TERRA_PRICING = Object.freeze({
6
- inputPerMillionUsd: 2.5,
6
+ inputPerMillionUsd: 2,
7
7
  });
8
8
  const OPENAI_GPT_5_6_LUNA_PRICING = Object.freeze({
9
- inputPerMillionUsd: 1,
9
+ inputPerMillionUsd: 0.2,
10
+ });
11
+ const OPENAI_GPT_6_SOL_PRICING = Object.freeze({
12
+ inputPerMillionUsd: 2,
13
+ });
14
+ const OPENAI_GPT_6_LUNA_PRICING = Object.freeze({
15
+ inputPerMillionUsd: 0.1,
10
16
  });
11
17
  const MODEL_PRICING = Object.freeze({
12
- "gpt-5.6-terra": Object.freeze({
13
- ...OPENAI_GPT_5_6_TERRA_PRICING,
14
- }),
15
- "gpt-5.6-luna": Object.freeze({
16
- ...OPENAI_GPT_5_6_LUNA_PRICING,
17
- }),
18
- "gpt-5.4": Object.freeze({
19
- ...OPENAI_GPT_5_4_PRICING,
20
- }),
21
- // Loadmill Smart currently routes to GPT-5.6 Terra behind the Loadmill LLM gateway.
22
- // Keep this explicit so pricing can be updated here if the backing model changes.
23
- "loadmill-smart": Object.freeze({
24
- ...OPENAI_GPT_5_6_TERRA_PRICING,
25
- }),
26
- "loadmill-pulse": Object.freeze({
27
- ...OPENAI_GPT_5_6_LUNA_PRICING,
28
- }),
18
+ "gpt-6-sol": OPENAI_GPT_6_SOL_PRICING,
19
+ "gpt-6-luna": OPENAI_GPT_6_LUNA_PRICING,
20
+ "gpt-5.6-terra": OPENAI_GPT_5_6_TERRA_PRICING,
21
+ "gpt-5.6-luna": OPENAI_GPT_5_6_LUNA_PRICING,
22
+ "gpt-5.4": OPENAI_GPT_5_4_PRICING,
23
+ // Loadmill Smart and Pulse route to GPT-6 Sol and Luna, respectively.
24
+ "loadmill-smart": OPENAI_GPT_6_SOL_PRICING,
25
+ "loadmill-pulse": OPENAI_GPT_6_LUNA_PRICING,
29
26
  });
30
27
  function finiteNumber(value) {
31
28
  return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
@@ -41,6 +38,12 @@ export function normalizeDroidCuaCostModel(model) {
41
38
  if (typeof model !== "string") {
42
39
  return null;
43
40
  }
41
+ if (model === "gpt-6-sol" || model.startsWith("gpt-6-sol-")) {
42
+ return "gpt-6-sol";
43
+ }
44
+ if (model === "gpt-6-luna" || model.startsWith("gpt-6-luna-")) {
45
+ return "gpt-6-luna";
46
+ }
44
47
  if (model === "gpt-5.6-terra" || model.startsWith("gpt-5.6-terra-")) {
45
48
  return "gpt-5.6-terra";
46
49
  }
@@ -1,6 +1,8 @@
1
1
  import { buildActionEventMeta } from "../device/action-event-meta.js";
2
2
  import { buildTypeActionPayload, buildTypeActionText, } from "../device/sensitive-action.js";
3
3
  import { getConfiguredStepDelayMs } from "../utils/step-delay.js";
4
+ import { getWindowsVisibleBounds } from "./windows-visible-bounds.js";
5
+ import sharp from "sharp";
4
6
  const SCREENSHOT_WIDTH = 1440;
5
7
  const ACCESSIBILITY_HIT_TOLERANCE = 12;
6
8
  const MACOS_CAPTURE_INSETS = { left: 50, right: 50, top: 38, bottom: 62 };
@@ -26,6 +28,57 @@ function assertResult(result, label) {
26
28
  throw new Error(result.content?.find((item) => item.type === "text")?.text || `${label} failed.`);
27
29
  }
28
30
  }
31
+ function readScreenshot(result) {
32
+ const image = result.content?.find((item) => item.type === "image");
33
+ const sizeText = result.content?.find((item) => item.type === "text")?.text || "";
34
+ const size = /^(\d+)x(\d+)$/.exec(sizeText.trim());
35
+ if (!image?.data || !size)
36
+ throw new Error("Desktop screenshot returned no image or dimensions.");
37
+ return { data: image.data, width: Number(size[1]), height: Number(size[2]) };
38
+ }
39
+ async function isBlackScreenshot(data) {
40
+ const { channels } = await sharp(Buffer.from(data, "base64")).stats();
41
+ return channels.slice(0, 3).every((channel) => channel.max === 0);
42
+ }
43
+ async function captureWindowFromDesktop(native, bounds) {
44
+ const display = native.getDisplaySize();
45
+ // Read full-resolution pixels; resizing the desktop first loses window detail.
46
+ const screen = native.takeScreenshot(undefined, undefined, 0, undefined, undefined);
47
+ if (!screen.base64)
48
+ throw new Error("Desktop screen capture returned no image.");
49
+ const left = Math.round(bounds.x * screen.width / display.width);
50
+ const top = Math.round(bounds.y * screen.height / display.height);
51
+ const width = Math.round(bounds.width * screen.width / display.width);
52
+ const height = Math.round(bounds.height * screen.height / display.height);
53
+ const visibleLeft = Math.max(0, left);
54
+ const visibleTop = Math.max(0, top);
55
+ const visibleRight = Math.min(screen.width, left + width);
56
+ const visibleBottom = Math.min(screen.height, top + height);
57
+ if (visibleRight <= visibleLeft || visibleBottom <= visibleTop) {
58
+ throw new Error("The window must overlap the primary display for Windows screen capture.");
59
+ }
60
+ let image = sharp(Buffer.from(screen.base64, "base64")).extract({
61
+ left: visibleLeft,
62
+ top: visibleTop,
63
+ width: visibleRight - visibleLeft,
64
+ height: visibleBottom - visibleTop,
65
+ });
66
+ if (visibleLeft !== left || visibleTop !== top || visibleRight !== left + width || visibleBottom !== top + height) {
67
+ image = image.extend({
68
+ left: visibleLeft - left,
69
+ top: visibleTop - top,
70
+ right: left + width - visibleRight,
71
+ bottom: top + height - visibleBottom,
72
+ background: "black",
73
+ });
74
+ }
75
+ // Materialize the crop and padding before resize (Sharp can reorder operations).
76
+ const cropped = await image.png().toBuffer();
77
+ const { data, info } = await sharp(cropped)
78
+ .resize({ width: SCREENSHOT_WIDTH, withoutEnlargement: true })
79
+ .png().toBuffer({ resolveWithObject: true });
80
+ return { data: data.toString("base64"), width: info.width, height: info.height };
81
+ }
29
82
  function sameBounds(left, right) {
30
83
  return left && right && left.x === right.x && left.y === right.y && left.width === right.width && left.height === right.height;
31
84
  }
@@ -63,7 +116,7 @@ async function createLibraryClient() {
63
116
  ]);
64
117
  return connectInProcess(createComputerUseServer({ profile: "ax", vision: true }));
65
118
  }
66
- async function loadAccessibilityDriver() {
119
+ async function loadNativeDriver() {
67
120
  const { loadNative } = await import("@zavora-ai/computer-use-mcp/host-native");
68
121
  return loadNative();
69
122
  }
@@ -183,9 +236,9 @@ function normalizeShortcut(keys, platform) {
183
236
  return keys.map((key) => aliases[String(key).toUpperCase()] || String(key).toLowerCase()).join("+");
184
237
  }
185
238
  /**
186
- * @param {{ app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", client?: any, accessibility?: any }} options
239
+ * @param {{ app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", focusOnConnect?: boolean, client?: any, accessibility?: any, visibleBoundsReader?: typeof getWindowsVisibleBounds }} options
187
240
  */
188
- export async function createDesktopController({ app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", client: injectedClient = null, accessibility: injectedAccessibility = null, } = {}) {
241
+ export async function createDesktopController({ app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", focusOnConnect = false, client: injectedClient = null, accessibility: injectedAccessibility = null, visibleBoundsReader = getWindowsVisibleBounds, } = {}) {
189
242
  if (!SUPPORTED_PLATFORMS.has(platform)) {
190
243
  throw new Error("Desktop control is currently supported only on macOS and Windows.");
191
244
  }
@@ -201,59 +254,91 @@ export async function createDesktopController({ app: requestedApp, windowId: req
201
254
  let targetWindow = chooseWindow(windows, requestedWindowId == null ? null : Number(requestedWindowId));
202
255
  let capture = null;
203
256
  let deviceInfo = null;
204
- let accessibility = injectedAccessibility;
257
+ let native = injectedAccessibility;
258
+ let lastVisibleBounds = null;
259
+ let fallbackLogged = false;
205
260
  async function getWindow() {
206
261
  return parseJsonResult(await client.getWindow(targetWindow.windowId), "Window lookup");
207
262
  }
208
- async function screenshot() {
209
- let window;
210
- let result;
263
+ async function focus() {
264
+ const activation = parseJsonResult(await client.activateWindow(targetWindow.windowId, 2000), "Window activation");
265
+ const focused = await getWindow();
266
+ if (!activation.activated || !focused.isFocused) {
267
+ throw new Error("Droid could not focus the selected desktop window, so no input was sent.");
268
+ }
269
+ }
270
+ async function captureFromDesktop(window) {
271
+ if (!window.isFocused) {
272
+ if (inputMode !== "physical") {
273
+ throw new Error("Windows desktop capture requires the target window to be visible and focused. Bring it to the foreground and retry.");
274
+ }
275
+ await focus();
276
+ window = await getWindow();
277
+ }
278
+ native ||= await loadNativeDriver();
279
+ if (lastVisibleBounds?.windowId !== targetWindow.windowId || !sameBounds(lastVisibleBounds.outer, window.bounds)) {
280
+ lastVisibleBounds = {
281
+ windowId: targetWindow.windowId,
282
+ outer: { ...window.bounds },
283
+ visible: await visibleBoundsReader(targetWindow.windowId, window.bounds),
284
+ };
285
+ }
286
+ const actionBounds = lastVisibleBounds.visible;
287
+ const image = await captureWindowFromDesktop(native, actionBounds);
288
+ if (!fallbackLogged) {
289
+ console.warn("Windows window capture returned all-black pixels; using a visible-desktop crop. Keep the target unobstructed on the primary display.");
290
+ fallbackLogged = true;
291
+ }
292
+ return { ...image, bounds: { ...window.bounds }, actionBounds: { ...actionBounds } };
293
+ }
294
+ async function screenshot({ allowDesktopFallback = true } = {}) {
211
295
  try {
212
296
  const currentWindows = readWindows(await client.listWindows(application.bundleId));
213
297
  const focusedWindow = currentWindows.find((candidate) => candidate.isFocused && candidate.isOnScreen);
214
298
  if (focusedWindow)
215
299
  targetWindow = focusedWindow;
216
- window = await getWindow();
300
+ const window = await getWindow();
217
301
  if (!window.isOnScreen)
218
302
  throw new Error("The selected desktop window is no longer visible.");
219
- result = await client.screenshot({
303
+ const result = await client.screenshot({
220
304
  width: SCREENSHOT_WIDTH,
221
305
  quality: 0,
222
306
  target_window_id: targetWindow.windowId,
223
307
  });
224
308
  assertResult(result, "Desktop screenshot");
309
+ let image = { ...readScreenshot(result), bounds: { ...window.bounds }, actionBounds: { ...window.bounds } };
310
+ if (allowDesktopFallback && platform === "win32" && await isBlackScreenshot(image.data)) {
311
+ image = await captureFromDesktop(window);
312
+ }
313
+ capture = {
314
+ bounds: image.bounds,
315
+ actionBounds: image.actionBounds,
316
+ width: image.width,
317
+ height: image.height,
318
+ contentBounds: getScreenshotContentBounds(image.width, image.height, image.bounds, platform),
319
+ };
320
+ if (deviceInfo) {
321
+ deviceInfo.device_width = image.actionBounds.width;
322
+ deviceInfo.device_height = image.actionBounds.height;
323
+ deviceInfo.scaled_width = image.width;
324
+ deviceInfo.scaled_height = image.height;
325
+ deviceInfo.scale = image.width / image.actionBounds.width;
326
+ deviceInfo.desktopWindowId = targetWindow.windowId;
327
+ }
328
+ return image.data;
225
329
  }
226
330
  catch (error) {
227
331
  throw new Error(formatPermissionError(error, platform));
228
332
  }
229
- const image = result.content?.find((item) => item.type === "image");
230
- const sizeText = result.content?.find((item) => item.type === "text")?.text || "";
231
- const size = /^(\d+)x(\d+)$/.exec(sizeText.trim());
232
- if (!image?.data || !size)
233
- throw new Error("Desktop screenshot returned no image or dimensions.");
234
- capture = {
235
- bounds: { ...window.bounds },
236
- width: Number(size[1]),
237
- height: Number(size[2]),
238
- };
239
- capture.contentBounds = getScreenshotContentBounds(capture.width, capture.height, window.bounds, platform);
240
- if (deviceInfo) {
241
- deviceInfo.device_width = window.bounds.width;
242
- deviceInfo.device_height = window.bounds.height;
243
- deviceInfo.scaled_width = capture.width;
244
- deviceInfo.scaled_height = capture.height;
245
- deviceInfo.scale = capture.width / window.bounds.width;
246
- deviceInfo.desktopWindowId = targetWindow.windowId;
247
- }
248
- return image.data;
249
333
  }
250
334
  function scaledPoint(x, y) {
251
335
  if (!capture)
252
336
  throw new Error("Capture the desktop window before executing an action.");
253
337
  const content = capture.contentBounds;
338
+ const bounds = capture.actionBounds;
254
339
  return [
255
- Math.round(capture.bounds.x + ((x - content.x) * capture.bounds.width) / content.width),
256
- Math.round(capture.bounds.y + ((y - content.y) * capture.bounds.height) / content.height),
340
+ Math.round(bounds.x + ((x - content.x) * bounds.width) / content.width),
341
+ Math.round(bounds.y + ((y - content.y) * bounds.height) / content.height),
257
342
  ];
258
343
  }
259
344
  async function prepareAction(requireFocus) {
@@ -265,11 +350,7 @@ export async function createDesktopController({ app: requestedApp, windowId: req
265
350
  }
266
351
  if (!requireFocus)
267
352
  return;
268
- const activation = parseJsonResult(await client.activateWindow(targetWindow.windowId, 2000), "Window activation");
269
- const focused = await getWindow();
270
- if (!activation.activated || !focused.isFocused) {
271
- throw new Error("Droid could not focus the selected desktop window, so no input was sent.");
272
- }
353
+ await focus();
273
354
  }
274
355
  const targetOptions = () => ({ targetWindowId: targetWindow.windowId, focusStrategy: "strict" });
275
356
  async function performCursorFreeAction(actionType, x, y, direction = null) {
@@ -277,8 +358,8 @@ export async function createDesktopController({ app: requestedApp, windowId: req
277
358
  const selected = chooseAccessibilityElement(tree, x, y, actionType, direction);
278
359
  if (!selected)
279
360
  throw new Error(cursorFreeError(actionType));
280
- accessibility ||= await loadAccessibilityDriver();
281
- const result = accessibility.performAction(targetWindow.windowId, selected.element.role, selected.element.label, selected.action);
361
+ native ||= await loadNativeDriver();
362
+ const result = native.performAction(targetWindow.windowId, selected.element.role, selected.element.label, selected.action);
282
363
  if (!result?.performed)
283
364
  throw new Error(`${cursorFreeError(actionType)}${result?.reason ? ` Reason: ${result.reason}.` : ""}`);
284
365
  }
@@ -375,16 +456,19 @@ export async function createDesktopController({ app: requestedApp, windowId: req
375
456
  return { success: false, error: message };
376
457
  }
377
458
  }
378
- await screenshot();
459
+ if (focusOnConnect)
460
+ await focus();
461
+ // Without focus, connection only probes the window; the desktop may be covered.
462
+ await screenshot({ allowDesktopFallback: focusOnConnect });
379
463
  deviceInfo = {
380
464
  platform: "desktop",
381
465
  source: "local",
382
466
  device_name: `${application.displayName} — ${targetWindow.title || "Untitled"}`,
383
- device_width: targetWindow.bounds.width,
384
- device_height: targetWindow.bounds.height,
467
+ device_width: capture.actionBounds.width,
468
+ device_height: capture.actionBounds.height,
385
469
  scaled_width: capture.width,
386
470
  scaled_height: capture.height,
387
- scale: capture.width / targetWindow.bounds.width,
471
+ scale: capture.width / capture.actionBounds.width,
388
472
  desktopAppId: application.bundleId,
389
473
  desktopWindowId: targetWindow.windowId,
390
474
  desktopInputMode: inputMode,
@@ -394,7 +478,8 @@ export async function createDesktopController({ app: requestedApp, windowId: req
394
478
  window: targetWindow,
395
479
  deviceId: `desktop:${targetWindow.windowId}`,
396
480
  deviceInfo,
397
- getScreenshotAsBase64: screenshot,
481
+ focus,
482
+ getScreenshotAsBase64: () => screenshot(),
398
483
  handleModelAction: handleAction,
399
484
  close: () => client.close(),
400
485
  };
@@ -0,0 +1,55 @@
1
+ import { execFile } from "node:child_process";
2
+ import { promisify } from "node:util";
3
+ const runFile = promisify(execFile);
4
+ const RECT_SCRIPT = `
5
+ Add-Type -TypeDefinition @'
6
+ using System;
7
+ using System.Runtime.InteropServices;
8
+ public static class WindowFrame {
9
+ public const int DWMWA_EXTENDED_FRAME_BOUNDS = 9;
10
+ [StructLayout(LayoutKind.Sequential)] public struct Rect { public int Left; public int Top; public int Right; public int Bottom; }
11
+ [DllImport("user32.dll")] public static extern bool GetWindowRect(IntPtr handle, out Rect rect);
12
+ [DllImport("dwmapi.dll")] public static extern int DwmGetWindowAttribute(IntPtr handle, int attribute, out Rect rect, int size);
13
+ }
14
+ '@
15
+ $handle = [IntPtr]::new([long]WINDOW_ID)
16
+ $outer = [WindowFrame+Rect]::new()
17
+ $visible = [WindowFrame+Rect]::new()
18
+ if (-not [WindowFrame]::GetWindowRect($handle, [ref]$outer)) { exit 1 }
19
+ $result = [WindowFrame]::DwmGetWindowAttribute($handle, [WindowFrame]::DWMWA_EXTENDED_FRAME_BOUNDS, [ref]$visible, 16)
20
+ if ($result -ne 0) { exit 1 }
21
+ @($outer.Left, $outer.Top, $outer.Right, $outer.Bottom, $visible.Left, $visible.Top, $visible.Right, $visible.Bottom) -join ','
22
+ `;
23
+ export async function getWindowsVisibleBounds(windowId, bounds) {
24
+ if (process.platform !== "win32")
25
+ return bounds;
26
+ if (!Number.isSafeInteger(windowId))
27
+ return bounds;
28
+ try {
29
+ // GetWindowRect includes invisible resize borders; DWM reports the pixels actually drawn.
30
+ const script = RECT_SCRIPT.replace("WINDOW_ID", String(windowId));
31
+ const { stdout } = await runFile("powershell.exe", ["-NoProfile", "-NonInteractive", "-Command", script], {
32
+ windowsHide: true,
33
+ timeout: 5000,
34
+ });
35
+ const [outerLeft, outerTop, outerRight, outerBottom, left, top, right, bottom] = stdout.trim().split(",").map(Number);
36
+ const scaleX = bounds.width / (outerRight - outerLeft);
37
+ const scaleY = bounds.height / (outerBottom - outerTop);
38
+ const insetLeft = Math.round((left - outerLeft) * scaleX);
39
+ const insetTop = Math.round((top - outerTop) * scaleY);
40
+ const insetRight = Math.round((outerRight - right) * scaleX);
41
+ const insetBottom = Math.round((outerBottom - bottom) * scaleY);
42
+ if (![insetLeft, insetTop, insetRight, insetBottom].every((value) => Number.isFinite(value) && value >= 0)) {
43
+ throw new Error("Invalid window frame insets.");
44
+ }
45
+ const width = bounds.width - insetLeft - insetRight;
46
+ const height = bounds.height - insetTop - insetBottom;
47
+ if (width <= 0 || height <= 0)
48
+ throw new Error("Invalid visible window dimensions.");
49
+ return { x: bounds.x + insetLeft, y: bounds.y + insetTop, width, height };
50
+ }
51
+ catch {
52
+ console.warn("Windows visible-frame lookup failed; using outer window bounds, which may include invisible resize borders.");
53
+ return bounds;
54
+ }
55
+ }
@@ -1,8 +1,10 @@
1
1
  import { createHash, randomUUID } from "node:crypto";
2
+ import { findDeepestErrorCode } from "../../utils/error-details.js";
2
3
  export const MAX_SCREENSHOTS_PER_REPORT = 1000;
3
4
  export const MAX_REPORT_OBJECT_BYTES = 10 * 1024 * 1024;
4
- function warn(session, message) {
5
+ function warn(session, message, phase) {
5
6
  session.failed = true;
7
+ session.onFailure?.({ uploadId: session.uploadId, phase, objectName: null, httpStatus: null, errorName: "ValidationError", errorCode: null });
6
8
  session.onWarning?.(message);
7
9
  }
8
10
  function getHttpStatus(error) {
@@ -16,6 +18,20 @@ function warnAboutUploadFailure(session) {
16
18
  const status = session.uploadFailureStatus ? ` (HTTP ${session.uploadFailureStatus})` : "";
17
19
  session.onWarning?.(`Cloud report upload failed${status}; further uploads skipped.`);
18
20
  }
21
+ function recordFailure(session, phase, objectName, error) {
22
+ if (session.uploadFailed)
23
+ return;
24
+ session.uploadFailed = true;
25
+ session.uploadFailureStatus = getHttpStatus(error);
26
+ session.onFailure?.({
27
+ uploadId: session.uploadId,
28
+ phase,
29
+ objectName,
30
+ httpStatus: session.uploadFailureStatus,
31
+ errorName: error instanceof Error ? error.name : typeof error,
32
+ errorCode: findDeepestErrorCode(error),
33
+ });
34
+ }
19
35
  function decodeScreenshot(dataUrl) {
20
36
  const match = /^data:image\/png;base64,([a-zA-Z0-9+/=\s]+)$/.exec(dataUrl);
21
37
  return match ? Buffer.from(match[1], "base64") : null;
@@ -46,13 +62,15 @@ export class DroidReportUploadManager {
46
62
  * associateDroidRunReport: (runId: string, payload: object) => Promise<unknown>,
47
63
  * },
48
64
  * onWarning?: ((message: string) => void) | null,
65
+ * onFailure?: ((details: { uploadId: string, phase: string, objectName: string | null, httpStatus: string | null, errorName: string, errorCode: string | null }) => void) | null,
49
66
  * }} options
50
67
  */
51
- startSession({ loadmillClient, onWarning = null } = {}) {
68
+ startSession({ loadmillClient, onWarning = null, onFailure = null } = {}) {
52
69
  const session = {
53
70
  uploadId: randomUUID(),
54
71
  loadmillClient,
55
72
  onWarning,
73
+ onFailure,
56
74
  sequence: 0,
57
75
  screenshotCount: 0,
58
76
  lastScreenshotObjectName: null,
@@ -79,15 +97,15 @@ export class DroidReportUploadManager {
79
97
  }
80
98
  const body = decodeScreenshot(dataUrl);
81
99
  if (!body) {
82
- warn(session, "Failed to upload Droid report screenshot: unsupported image data.");
100
+ warn(session, "Failed to upload Droid report screenshot: unsupported image data.", "validate_screenshot");
83
101
  return null;
84
102
  }
85
103
  if (body.length > MAX_REPORT_OBJECT_BYTES) {
86
- warn(session, `Droid report screenshot exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`);
104
+ warn(session, `Droid report screenshot exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`, "validate_screenshot");
87
105
  return null;
88
106
  }
89
107
  if (session.screenshotCount >= MAX_SCREENSHOTS_PER_REPORT) {
90
- warn(session, `Droid report exceeds ${MAX_SCREENSHOTS_PER_REPORT} screenshots.`);
108
+ warn(session, `Droid report exceeds ${MAX_SCREENSHOTS_PER_REPORT} screenshots.`, "validate_screenshot");
91
109
  return null;
92
110
  }
93
111
  const name = `screenshots/${String(++session.sequence).padStart(4, "0")}.png`;
@@ -107,7 +125,16 @@ export class DroidReportUploadManager {
107
125
  session.finishing = true;
108
126
  session.finishPromise = this.#finish(session, { savedRunId, html }).catch((error) => {
109
127
  const status = getHttpStatus(error);
110
- warn(session, `Failed to publish Droid run report${status ? ` (HTTP ${status})` : ""}.`);
128
+ session.onFailure?.({
129
+ uploadId: session.uploadId,
130
+ phase: "associate_report",
131
+ objectName: null,
132
+ httpStatus: status,
133
+ errorName: error instanceof Error ? error.name : typeof error,
134
+ errorCode: findDeepestErrorCode(error),
135
+ });
136
+ session.failed = true;
137
+ session.onWarning?.(`Failed to publish Droid run report${status ? ` (HTTP ${status})` : ""}.`);
111
138
  return { published: false, reason: "publish_failed" };
112
139
  }).finally(() => {
113
140
  this.sessions.delete(session);
@@ -125,7 +152,7 @@ export class DroidReportUploadManager {
125
152
  }
126
153
  const body = Buffer.from(html, "utf8");
127
154
  if (body.length > MAX_REPORT_OBJECT_BYTES) {
128
- warn(session, `Droid HTML report exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`);
155
+ warn(session, `Droid HTML report exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`, "validate_report");
129
156
  return { published: false, reason: "too_large" };
130
157
  }
131
158
  const htmlUpload = await this.#enqueue(session, {
@@ -198,6 +225,7 @@ export class DroidReportUploadManager {
198
225
  }
199
226
  async #runTask(task) {
200
227
  let result = { uploaded: false, key: null };
228
+ let phase = "request_policy";
201
229
  try {
202
230
  if (task.session.uploadFailed) {
203
231
  return;
@@ -207,13 +235,13 @@ export class DroidReportUploadManager {
207
235
  if (!uploadPolicy?.key) {
208
236
  throw new Error("Loadmill did not return an upload policy.");
209
237
  }
238
+ phase = "upload_object";
210
239
  await uploadObject(uploadPolicy, task.object.body, task.object.name.split("/").at(-1), task.object.contentType, this.fetchImpl);
211
240
  result = { uploaded: true, key: uploadPolicy.key };
212
241
  }
213
242
  catch (error) {
214
243
  task.session.failed = true;
215
- task.session.uploadFailed = true;
216
- task.session.uploadFailureStatus ??= getHttpStatus(error);
244
+ recordFailure(task.session, phase, task.object.name, error);
217
245
  }
218
246
  finally {
219
247
  task.session.pending.delete(task.promise);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@loadmill/droid-cua",
3
- "version": "2.39.0",
3
+ "version": "2.40.0",
4
4
  "description": "AI-powered Android testing agent using OpenAI's computer-use model and ADB",
5
5
  "main": "build/index.js",
6
6
  "type": "module",
@@ -1,6 +1,8 @@
1
1
  ---
2
2
  name: droid-cua
3
3
  description: Operate and improve Droid CUA `.dcua` test projects. Use when the user explicitly names Droid CUA or @loadmill/droid-cua, or asks to author, configure, run, diagnose, heal, stabilize, or speed up `.dcua` tests with the Droid CUA CLI or desktop app. Do not use for general web browsing, website development, browser automation, generic mobile work, or non-Droid test tasks.
4
+ metadata:
5
+ version: "1.0.8"
4
6
  ---
5
7
 
6
8
  # Droid CUA
@@ -12,7 +14,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
12
14
  1. Inspect the active test project before creating files. Find existing `.dcua` tests, `context.md`, `test-data.md`, `.secrets`, `.env`, and `ci/droid-cua.json` without reading secret values. Do not search other projects or the user's home directory for credentials.
13
15
  2. Determine the requested platform and target from the task or project. Ask the user only when the choice is unclear.
14
16
  3. Check authentication and environment readiness before the first run. Read [project-and-auth.md](references/project-and-auth.md) for project boundaries and token handling. Read [onboarding-and-troubleshooting.md](references/onboarding-and-troubleshooting.md) when setup, dependency, device, simulator, browser, or connection work is needed.
15
- 4. Author or update the smallest useful test. Read [authoring-tests.md](references/authoring-tests.md) before changing `.dcua`, context, test data, secrets, or config.
17
+ 4. Identify the intended coverage, starting state, scenario inputs, and cleanup; author or update the smallest useful test that preserves them. Read [authoring-tests.md](references/authoring-tests.md) before changing `.dcua`, context, test data, secrets, or config.
16
18
  5. Before execution, present the proposed tests and target to the user, resolve any questions, and get their confirmation. Read [running-and-healing.md](references/running-and-healing.md) for the required pre-run check.
17
19
  6. Run the confirmed, narrowest relevant test and monitor it to completion. Keep the user oriented during the run by surfacing meaningful live screenshots, not just terminal output.
18
20
  7. For a new, changed, flaky, slow, or explicitly optimized test, continue with the evidence-driven author-run-improve loop in [iterative-improvement.md](references/iterative-improvement.md). A passing run is a baseline, not proof of readiness.
@@ -27,7 +29,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
27
29
  - Get approval before installing dependencies, running `sudo`, accepting licenses, or changing system configuration.
28
30
  - During setup, do not create or edit user tests. Rerun only the failed readiness check, then resume the original task.
29
31
  - During healing, edit only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never modify application code under the guise of healing a test.
30
- - Never weaken or remove an assertion merely to make a failing run pass.
32
+ - Never weaken required coverage or remove an assertion merely to make a failing run pass. Agent-added, out-of-scope assertions may be removed only with a coverage-based explanation; see [authoring-tests.md](references/authoring-tests.md).
31
33
  - Keep one canonical context file and one canonical Droid CUA config file per test project. Reuse and update those shared files; never create test-specific context or config copies for a new or improved test.
32
34
  - When improving an existing test, keep its complete flow in one independently runnable `.dcua` file. Do not split it into ordered or state-dependent tests as an optimization; Droid CUA does not currently guarantee that separate tests will run consecutively or share state.
33
35
  - Before rerunning a long test, apply the smallest evidence-supported batch that addresses its known bottlenecks. The batch may include multiple `.dcua`, context, test-data, or config improvements when each has a clear rationale and preserves coverage. Record the complete change set; isolate or revert individual changes only if the combined result regresses or remains ambiguous.
@@ -39,7 +41,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
39
41
 
40
42
  - Prefer an existing project convention over creating a parallel structure.
41
43
  - Prefer headless runs for agent work because they are observable and return deterministic exit codes.
42
- - Write tests as concise, human-readable user journeys. Prefer one business-level instruction per user goal over one instruction per tap, menu, dialog, or other UI gesture; add lower-level guidance only when run evidence shows it is necessary.
44
+ - Preserve the user's intended coverage. Write one readable micro-goal per step: a small user task with a recognizable completion point. During improvement, simplify before adding detail, and retain only detail that is necessary and supported by evidence. Do not optimize for fewer lines or compress several goals into one instruction.
43
45
  - Use explicit, user-observable outcomes in tests rather than implementation details or coordinates.
44
46
  - Diagnose the first divergence between the instruction and observed behavior before changing anything.
45
47
  - Compare runs on the same target and app build, and clearly identify every test, context, data, model, or config change included in the optimization batch.
@@ -4,32 +4,91 @@
4
4
 
5
5
  Find nearby `.dcua` tests and project files before writing. Match their directory layout, naming, vocabulary, starting state, and config conventions. Do not open `.env` or `.secrets`.
6
6
 
7
+ ## Establish intended coverage
8
+
9
+ Before drafting or healing, identify the requested behavior, starting state, scenario inputs, business outcome, and cleanup from the user request and existing test. Keep this brief in the working plan; no separate specification file is required. Distinguish user-required checks from incidental checks introduced during earlier agent iterations. If an existing assertion's purpose is unclear, preserve it until its scope is resolved.
10
+
11
+ Preserve scenario choices during simplification: a full-day booking and a three-hour booking exercise different behavior. Adding UI cancellation also changes coverage; do it when requested, not automatically for every creation test.
12
+
13
+ Prefer predictable setup and data over conditional recovery. Use known cleanup and simple relative dates when appropriate for the project. A date such as next Monday assumes a working day and available account state; establish those assumptions rather than inventing a search for the earliest conflict-free date. Restarting the app is a project-specific starting condition, not a universal requirement. Include only setup needed by this scenario and supported by the request, project knowledge, or observed behavior. Do not import dates, reset flows, or other prerequisites from a nearby example just because they made that example runnable. If an unknown prerequisite matters, identify the gap rather than silently inventing a mandatory step.
14
+
7
15
  ## Write `.dcua` instructions
8
16
 
9
- - Put one clear user goal or assertion on each non-empty line. One instruction does not need to equal one UI action.
10
- - Write concise, human-readable business or journey instructions that describe what the user wants to accomplish. Prefer `Select the desired date` or `Log out` over separate instructions for opening a picker or menu, clicking controls, handling an expected confirmation, and submitting.
11
- - Do not prescribe coordinates, selectors, implementation details, model strategy, or every tap and keystroke. These micro-steps couple the test to incidental UI structure, add model rounds, and make harmless UI changes more likely to break it.
12
- - Break a business-level instruction into smaller actions only when run evidence shows that the agent cannot complete it reliably or the intermediate state is itself behavior the test must verify.
13
- - State the starting condition when it is not guaranteed by the environment.
14
- - Use explicit assertions for important outcomes. Prefer stable visible facts such as a screen title, confirmation message, changed value, or navigation result.
15
- - Use exact visible labels when naming controls or outcomes.
17
+ - Put one readable micro-goal on each non-empty line: a small user task with a recognizable completion point, such as choosing a room, selecting dates, or submitting a request. One goal can require several UI actions.
18
+ - Keep useful task boundaries. Do not compress login, navigation, data entry, and verification into a single instruction to reduce the line count.
19
+ - Specify the scenario choices precisely while leaving ordinary UI mechanics to Droid. Avoid coordinates, selectors, icon appearance, and instructions for every tap or keystroke.
20
+ - Use visible labels when they identify a meaningful choice or resolve ambiguity. Short navigation hints are welcome when useful, such as `Custom duration (scroll right in the duration options)`; do not require exact layout or incidental copy.
21
+ - Add lower-level guidance only when evidence shows it is necessary to complete the goal reliably, or the intermediate behavior is explicitly under test. Prefer clarifying the goal over appending a transcript of a failed run.
16
22
  - Keep each new test focused on one behavior. Separate unrelated flows only when each resulting test is fully independent and does not rely on execution order or state created by another test.
17
- - When updating or optimizing an existing test, preserve its complete flow in one `.dcua` file. Do not split it into sequential or dependent tests unless the user explicitly requests that structure and the project has a verified orchestration mechanism that guarantees ordering and shared state.
18
- - Use `//` comments sparingly for human context; comments are not executed.
19
- - Never include secret values in the test.
23
+ - When updating or optimizing an existing test, preserve its complete intended flow in one independently runnable `.dcua` file. Do not split it into sequential or dependent tests without explicit user direction and verified orchestration.
24
+ - Use `//` comments sparingly for human context; comments are not executed. Never include secret values.
25
+
26
+ ## Choose assertions by coverage
27
+
28
+ Ask: **If this check failed while the requested business outcome remained correct, would that be a defect this scenario is intended to catch?** If not, it generally should not be a standalone assertion here. Droid must observe intermediate states to perform actions; those observations do not all need to become test assertions.
29
+
30
+ Verify meaningful outcomes and identify the created or changed object by the relevant scenario inputs. Checking that the request for the selected room, dates, and duration is confirmed prevents an unrelated approved request from satisfying the test. Do not automatically assert every dialog closure, available option, summary label, or absence of a warning when the final outcome already covers the intended behavior.
31
+
32
+ Preserve explicit user requirements and genuine product failures. Remove an agent-added assertion only when it is demonstrably outside intended coverage or redundant with a retained check; explain that coverage-based reason, not merely that it failed or cost time. An intermediate control or exact message still needs an assertion when that behavior is the requested subject of the test.
33
+
34
+ ## Example: readable booking journeys
35
+
36
+ These examples use a fictional room-booking app, generic test accounts, and invented flow and control names. They do not describe a customer's product. Assume the project maps the test account to configured secret keys and the named cleanup flow makes the sample room available. This hypothetical user explicitly wants a fresh app start and UI cancellation coverage. Adapt those choices to the actual project; neither is a universal requirement.
37
+
38
+ Full-day booking:
39
+
40
+ ```text
41
+ Run the Loadmill "Reset sample room bookings" flow.
42
+ Open the booking app; if already open, fully close it and reopen it.
43
+ Log in as the test member, logging out first if another user is signed in.
44
+ Go to Home.
45
+ Start a room booking.
46
+ Choose "Sample room".
47
+ Select next Monday–Tuesday as full days.
48
+ Submit the booking.
49
+ Go to My bookings.
50
+ Verify the new booking for Sample room and the selected dates is confirmed for 2 full days.
51
+ Open that booking and cancel it through its menu.
52
+ Verify that booking no longer appears in My bookings.
53
+ ```
54
+
55
+ Custom-duration booking:
56
+
57
+ ```text
58
+ Run the Loadmill "Reset sample room bookings" flow.
59
+ Open the booking app; if already open, fully close it and reopen it.
60
+ Log in as the test member, logging out first if another user is signed in.
61
+ Go to Home.
62
+ Start a room booking.
63
+ Choose "Sample room".
64
+ Select next Monday at 09:00.
65
+ Choose "Custom duration" (scroll right in the duration options) and set 3 hours.
66
+ Submit the booking.
67
+ Go to My bookings.
68
+ Verify the new booking for Sample room on the selected date at 09:00 is confirmed for 3 hours.
69
+ Open that booking and cancel it through its menu.
70
+ Verify that booking no longer appears in My bookings.
71
+ ```
72
+
73
+ The final assertions identify the requested result and cleanup. Neither test checks unrelated duration options or how the calendar collapses. The short scrolling hint helps find a relevant choice without scripting the entire interaction.
74
+
75
+ If a run leaves the calendar selection unconfirmed, first clarify `Select next Monday–Tuesday as full days` to `Select next Monday–Tuesday as full days and confirm the date selection.` Avoid accumulating instructions such as “tap Apply now even if the total updated” plus assertions that the grid and button disappear. Verify that the clearer goal resolves the observed failure; retain a specific control hint only if it is still needed. Do not hide an unresolved Droid or product defect in test prose.
76
+
77
+ ## Example: account preferences
20
78
 
21
- Example:
79
+ This fictional settings app illustrates the same approach without booking dates, cleanup flows, or app restarts. The requested behavior is saving a notification preference and retaining it when settings are reopened; assume the test account initially uses Daily notifications.
22
80
 
23
81
  ```text
24
- Sign in using the USER_EMAIL and USER_PASSWORD secrets.
25
- Assert that the Home screen is visible and shows the signed-in account.
26
- Select December 15 as the requested date.
27
- Assert that December 15 is shown as the requested date.
28
- Log out.
29
- Assert that the sign-in screen is visible.
82
+ Log in to the settings app as the test member.
83
+ Open Notification settings.
84
+ Change the email digest frequency to Weekly.
85
+ Save the preferences.
86
+ Leave Notification settings and reopen it.
87
+ Verify the email digest frequency is still Weekly.
88
+ Restore the email digest frequency to Daily and save.
30
89
  ```
31
90
 
32
- Keep meaningful outcomes explicit even when actions are high-level. For example, follow `Log out` with an assertion that the sign-in screen is visible when logout behavior is under test.
91
+ The persistence check protects the requested behavior. Checking a toast, button color, or menu animation would add unrelated requirements. Restore the known starting value for repeatability; do not invent cleanup when the scenario or environment does not need it.
33
92
 
34
93
  Use the secret key names that exist in `.secrets`; do not inspect their values. If key discovery is necessary, filter the file so only keys reach output, for example `sed -n 's/^\([^#][^=]*\)=.*/\1/p' .secrets`. If a required key is absent, ask the user to add it without sharing its value.
35
94
 
@@ -10,7 +10,7 @@ One confirmation may cover that iterative workflow. Continue without asking agai
10
10
 
11
11
  ## 2. Establish a reproducible baseline
12
12
 
13
- Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
13
+ Record the intended coverage using [authoring-tests.md](authoring-tests.md) before editing, including which assertions are user requirements. Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
14
14
 
15
15
  - App build and initial account/data state.
16
16
  - Device, OS, device source, or browser session mode.
@@ -49,18 +49,18 @@ Stop healing when evidence indicates an application regression. Preserve the fai
49
49
  For long-running tests, fix all well-supported bottlenecks that can be addressed safely before paying for another full run. Use this order to build the batch unless evidence points elsewhere:
50
50
 
51
51
  1. **`.dcua` wording and scope**
52
- - Express one human-readable business goal per instruction, not one click, menu, field, dialog, or submit action per line.
52
+ - Express one readable micro-goal per instruction. Choosing a room, selecting dates, and submitting a booking are useful separate goals; opening each menu or tapping each control usually is not.
53
53
  - Collapse consecutive micro-steps into the user outcome they jointly accomplish, such as `Select the desired date` or `Log out`, while keeping meaningful assertions explicit.
54
54
  - Use exact visible labels only when they disambiguate the goal or identify an outcome the test must verify.
55
- - Add the missing starting condition or assertion at the first ambiguous transition.
55
+ - Resolve an ambiguous transition by checking the starting state, data, and wording first. Add an assertion only when it protects intended coverage; observing progress does not require a new test assertion.
56
56
  - Remove scenario steps that do not protect the behavior under test.
57
- - Add lower-level actions only when the evidence shows the business-level instruction is unreliable or an intermediate state is part of the coverage.
57
+ - Rewrite the unclear goal before adding lower-level actions. Add a short hint only when the evidence shows it is still necessary or an intermediate state is part of the coverage; do not append emphatic commands and UI-state checks from each failed run.
58
58
  - Keep the existing scenario in one independently runnable `.dcua` file. Splitting a long test into ordered or state-dependent tests is not an available optimization because separate execution order and shared state are not guaranteed.
59
- - Never remove or weaken a meaningful assertion to obtain a pass.
59
+ - Preserve required assertions and scenario choices. Remove agent-added, out-of-scope or redundant assertions only with the coverage-based explanation required by the authoring guide; preserve checks whose purpose is unresolved.
60
60
 
61
61
  2. **`context.md`**
62
62
  - Update the project's existing canonical context file; never create a test-specific context file for the optimization.
63
- - Add durable product navigation, terminology, stable interruptions, and narrowly observed recovery behavior.
63
+ - Add durable product navigation, terminology, and stable interruptions. Do not move accumulated scenario recovery instructions into shared context to make the test appear simpler.
64
64
  - Move reusable mechanics out of the test, but keep scenario-specific values and expectations in the test.
65
65
  - Remove redundant, speculative, or conflicting instructions. Do not restate Droid's base behavior or encode a workaround for a Droid/product bug as product knowledge.
66
66
 
@@ -114,11 +114,14 @@ For every included change, record the evidence, rationale, expected effect, and
114
114
 
115
115
  ## 6. Compare and iterate
116
116
 
117
+ After each edit, reread the whole test against its intended coverage. Remove superseded hints and redundant checks, keep one micro-goal per instruction, and ensure setup or data problems have not become speculative recovery branches. Keep diagnostic history in the report, not the test or shared context. Validate the simplified version in the next representative run.
118
+
117
119
  After each rerun, compare:
118
120
 
119
121
  - Did the first divergence disappear for the expected reason?
120
122
  - Did the hotspot's duration, rounds, actions, previews, retries, or recoveries improve?
121
123
  - Did another step regress or did coverage change?
124
+ - Did the test become harder to read or more dependent on incidental UI details? Explain any retained complexity by coverage or execution evidence; fewer lines alone are not an improvement.
122
125
  - Did the combined change set improve the intended hotspots without obscuring a regression?
123
126
 
124
127
  Keep the batch when the evidence supports the overall result. If it regresses or remains ambiguous, use the timeline to isolate, revise, or revert only the suspect subset instead of automatically rerunning every change separately. Continue while the evidence supports another useful test, context, data, or configuration improvement. Stop and report the strongest remaining explanation only when a genuine blocker remains or no evidence-supported improvement is left to try.
@@ -129,6 +132,7 @@ After the optimized test passes, decide whether the accumulated evidence is stro
129
132
 
130
133
  Call the test ship-ready only when:
131
134
 
135
+ - The final test still expresses the intended coverage as readable micro-goals, with no unnecessary assertions, stale debugging instructions, or speculative recovery branches.
132
136
  - All intended instructions and meaningful assertions pass in the final representative evidence.
133
137
  - No unexplained recovery, repeated-action loop, or recurring high-risk hotspot remains.
134
138
  - Timing and action counts are reasonably consistent, with transient provider retries identified separately.
@@ -4,6 +4,12 @@ Follow the Desktop Device Setup Wizard sequence: choose a platform, run checks,
4
4
 
5
5
  Do not create or modify user test files merely to perform setup. Get approval before installing software, changing system configuration, running `sudo`, or accepting a license. After remediation, rerun only the failed check.
6
6
 
7
+ ## Identify the installed skill
8
+
9
+ When diagnosing skill behavior, report the installed skill version and bundled skill version from `droid-cua skill status <agent>` (`cursor`, `claude-code`, or `codex`). Use `--local` for a project installation. The skill's own version is `metadata.version` in `SKILL.md`; it is independent of the CLI package version. If an older CLI does not display versions, read this field from the active skill copy instead.
10
+
11
+ `unknown` means version metadata is unavailable, as with older copies. `modified` means the installed files differ from the bundled skill; it can indicate an older release or local edits, including edits that left the version unchanged. Report that state alongside the version. Updating the CLI does not update existing skill copies; explain any mismatch before using `droid-cua skill install <agent> --force`, which overwrites local changes.
12
+
7
13
  ## Droid CUA CLI compatibility
8
14
 
9
15
  If the CLI rejects a model, option, command, or capability that this skill explicitly documents, treat an outdated global `@loadmill/droid-cua` package as a likely setup issue. Do not suggest an update for an ordinary test, target, provider, or application failure.
@@ -40,4 +40,4 @@ Offer exactly these two choices, then wait for the user's selection:
40
40
 
41
41
  Use a secret-safe mechanism that does not print or expose the value in commands, logs, or responses. Verify only that the key is present. Never copy a token from another file, a shell profile, another project, an agent skill/configuration, or any other location. Do not inspect inherited environment variables to discover a token. When the active project's `.env` contains the key, run `droid-cua` normally from that project; the CLI loads that file automatically.
42
42
 
43
- Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with one of the supported OpenAI CUA models: `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Check only the active project's `.env` for `OPENAI_API_KEY`; do not use one found elsewhere. If it is missing, guide the user to create and configure their own key in the same project using the same secret-safe process. Do not silently switch providers to bypass missing Loadmill authentication.
43
+ Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Saved direct GPT-5.6 Terra and GPT-5.4 selections migrate to GPT-6 Sol; saved GPT-5.6 Luna selections migrate to GPT-6 Luna. Check only the active project's `.env` for `OPENAI_API_KEY`; do not use one found elsewhere. If it is missing, guide the user to create and configure their own key in the same project using the same secret-safe process. Do not silently switch providers to bypass missing Loadmill authentication.
@@ -16,12 +16,16 @@ Preserve an existing provider/model choice in `ci/droid-cua.json`; otherwise use
16
16
  --llm-provider loadmill --cua-model loadmill-smart
17
17
  ```
18
18
 
19
- All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with one of `gpt-5.6-terra`, `gpt-5.6-luna`, or `gpt-5.4`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
19
+ All Loadmill models require `LOADMILL_API_TOKEN`. Smart and Pulse now route through the Loadmill gateway to GPT-6 Sol and GPT-6 Luna. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
20
20
 
21
21
  Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
22
22
 
23
+ For macOS and Windows desktop tests, the desktop app shows a system notice for three seconds before Droid takes control. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. Droid brings the selected target window to the foreground before the first capture. When the run ends, the target stays in front and a second system notice says Droid has finished controlling the desktop. Open Droid to review the result. Some Windows apps return an all-black window capture; Droid brings the target back to the foreground before cropping the visible desktop to its visible frame, excluding the invisible Windows resize border. The fallback logs an all-black capture warning and crops before resizing to preserve detail. A visible-frame lookup warning means Droid is using the outer window bounds, which may include invisible resize borders. Keep the target visible and unobstructed on the primary display. In accessibility input mode, bring the target to the foreground yourself if the fallback requests it; Droid does not focus it for screenshots. If a screenshot shows another app or black off-screen edges, check whether the target could be focused and fully displayed before changing the `.dcua` instructions.
24
+
23
25
  When downstream automation needs an exact Loadmill handoff, add `--report-metadata reports/<test-name>.json`. Read it only after Droid exits. Droid clears the destination at run start and writes the saved run ID plus the last uploaded screenshot object name only after cloud report publication succeeds, including when a failed test's report was published. Do not infer the run or screenshot by searching recent Loadmill runs when this file was requested but absent.
24
26
 
27
+ For desktop runs, treat the test result and dashboard report as separate outcomes. A run can appear in Droid Runs while its report is still uploading. A `report.upload.skipped` entry in the app's `desktop-events.jsonl` explains when sharing was disabled or authentication was unavailable. For failures, use the run warning and `report.upload.failed` entry, which includes the upload stage and HTTP status when available. The absence of a failure entry does not prove that a still-unavailable report was published; it may still be uploading. Direct the user to the local report in Droid when sharing did not complete. Do not infer the cause from an unclickable dashboard row alone.
28
+
25
29
  ```sh
26
30
  # Attached Android device or running emulator
27
31
  droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
@@ -42,7 +46,7 @@ droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pu
42
46
  droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --device-source aws-device-farm --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --debug --report reports/example.html
43
47
 
44
48
  # Direct OpenAI alternative
45
- droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-5.4 --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
49
+ droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-6-sol --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
46
50
 
47
51
  # Selected tests from different folders
48
52
  droid-cua run tests/auth/login.dcua tests/checkout/purchase.dcua admin/refund.dcua --avd adb:<serial> --config ci/droid-cua.json --debug --report reports/selected-tests.html
@@ -133,6 +137,6 @@ Keep this retrospective concise and lead with the result and improvement. Do not
133
137
 
134
138
  ## Guarded healing
135
139
 
136
- Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete coverage, loosen the expected outcome, or add broad waits simply to obtain green output.
140
+ Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete required coverage, loosen the expected outcome, or add broad waits simply to obtain green output. Use the coverage rules in [authoring-tests.md](authoring-tests.md) to distinguish required assertions from agent-added, out-of-scope or redundant checks; retain assertions whose purpose is unclear. A failing business outcome remains a failure.
137
141
 
138
- Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.
142
+ Simplify unclear goals and check setup/data before adding UI detail. Replace obsolete guidance rather than accumulating commands from each failed run, and review the whole test for readability after editing. Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.