@loadmill/droid-cua 2.39.0 → 2.40.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -6
- package/build/index.js +1 -0
- package/build/src/cli/headless-execution-config.js +6 -5
- package/build/src/cli/skill-command.js +20 -3
- package/build/src/core/cua-models.js +17 -15
- package/build/src/core/droid-cua-run-usage.js +22 -19
- package/build/src/desktop/computer-use.js +128 -43
- package/build/src/desktop/windows-visible-bounds.js +55 -0
- package/build/src/integrations/loadmill/report-upload-manager.js +37 -9
- package/package.json +1 -1
- package/skill/droid-cua/SKILL.md +5 -3
- package/skill/droid-cua/references/authoring-tests.md +77 -18
- package/skill/droid-cua/references/iterative-improvement.md +10 -6
- package/skill/droid-cua/references/onboarding-and-troubleshooting.md +6 -0
- package/skill/droid-cua/references/project-and-auth.md +1 -1
- package/skill/droid-cua/references/running-and-healing.md +8 -4
package/README.md
CHANGED
|
@@ -158,8 +158,13 @@ droid-cua skill install codex
|
|
|
158
158
|
droid-cua skill install codex --local
|
|
159
159
|
```
|
|
160
160
|
|
|
161
|
-
Use `droid-cua skill status <agent>` to
|
|
162
|
-
|
|
161
|
+
Use `droid-cua skill status <agent>` to see the installed and bundled skill
|
|
162
|
+
versions and installation path; add `--local` to inspect a project installation.
|
|
163
|
+
The skill version is stored in `SKILL.md` under `metadata.version`, independently
|
|
164
|
+
of the CLI package version. Older copies without version metadata show `unknown`.
|
|
165
|
+
The `modified` status means the files differ from the bundled copy, which can
|
|
166
|
+
indicate an older release or local edits even when version labels match.
|
|
167
|
+
Use `droid-cua skill uninstall <agent>` to remove an installation. Installed skill
|
|
163
168
|
files are never overwritten or removed after local modification unless `--force`
|
|
164
169
|
is supplied. After upgrading the npm package, update an installed skill explicitly:
|
|
165
170
|
|
|
@@ -239,7 +244,7 @@ backward compatibility.
|
|
|
239
244
|
Example headless config:
|
|
240
245
|
```json
|
|
241
246
|
{
|
|
242
|
-
"cuaModel": "gpt-
|
|
247
|
+
"cuaModel": "gpt-6-sol",
|
|
243
248
|
"promptCustomizations": {
|
|
244
249
|
"basePromptInstructions": "",
|
|
245
250
|
"designModeInstructions": "",
|
|
@@ -295,10 +300,13 @@ Supported CLI options include:
|
|
|
295
300
|
- `--debug`
|
|
296
301
|
|
|
297
302
|
`--cua-model` accepts `loadmill-smart`, `loadmill-pulse`, `loadmill-beacon`,
|
|
298
|
-
`gpt-
|
|
299
|
-
|
|
303
|
+
`gpt-6-sol`, and `gpt-6-luna`. OpenAI models use direct OpenAI mode and require
|
|
304
|
+
`OPENAI_API_KEY`. Saved direct OpenAI selections of GPT-5.6 Terra, GPT-5.6 Luna,
|
|
305
|
+
or GPT-5.4 move to GPT-6 Sol, GPT-6 Luna, or GPT-6 Sol, respectively.
|
|
300
306
|
|
|
301
|
-
Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default:
|
|
307
|
+
Desktop control supports macOS and Windows. In the desktop app, enable **Experimental desktop control** at the bottom of Settings to show Desktop in the platform picker. Desktop runs use visual input by default: Droid shows a system notice for three seconds before taking control, brings the selected window to the foreground, then sends physical pointer and keyboard input at the model's coordinates. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. When the run ends, Droid leaves the target window in front and shows another system notice that control has finished, including when the run fails or is stopped. CLI runs also leave the target in front. Use `--desktop-input accessibility` for cursor-free CLI input instead. macOS requires Screen & System Audio Recording and Accessibility permissions for Droid or the terminal running the CLI. Windows targets must run at the same integrity level as Droid; a non-elevated Droid process cannot inject input into an elevated application. If Windows returns an all-black window capture, Droid brings the target back to the foreground and crops the visible desktop to the window's visible frame, excluding its invisible resize border. The crop is resized only after extraction to preserve window detail. The fallback logs a warning and requires the target to remain unobstructed on the primary display; off-screen portions appear black. With accessibility input, bring the target to the foreground yourself when requested.
|
|
308
|
+
|
|
309
|
+
The system notices are shown only for desktop control runs started in the desktop app. On macOS, they may not appear in the unsigned Electron dev app; verify them in a signed packaged build.
|
|
302
310
|
|
|
303
311
|
Config and precedence rules:
|
|
304
312
|
- Use `--config <file>` to supply prompt-affecting settings for headless runs.
|
package/build/index.js
CHANGED
|
@@ -2,7 +2,7 @@ import path from "path";
|
|
|
2
2
|
import { readFile } from "fs/promises";
|
|
3
3
|
import { DEFAULT_CONTEXT_OPTIMIZATION_THRESHOLD, MAX_CONTEXT_OPTIMIZATION_THRESHOLD, MIN_CONTEXT_OPTIMIZATION_THRESHOLD, } from "../core/context-optimization.js";
|
|
4
4
|
import { AIM_ASSIST_MODES, DEFAULT_AIM_ASSIST_MODE } from "../core/aim-assist.js";
|
|
5
|
-
import { CUA_MODELS, DEFAULT_OPENAI_CUA_MODEL, isOpenAiCuaModel, isValidCuaModel, resolveCuaModel } from "../core/cua-models.js";
|
|
5
|
+
import { CUA_MODELS, DEFAULT_OPENAI_CUA_MODEL, isOpenAiCuaModel, isValidCuaModel, resolveCuaModel, upgradeLegacyCuaModel } from "../core/cua-models.js";
|
|
6
6
|
import { LLM_PROVIDER_MODES, isValidLlmProviderMode, resolveLlmProviderMode } from "../core/runtime-config.js";
|
|
7
7
|
import { parseSecretsOverrideString } from "../core/secrets.js";
|
|
8
8
|
function createEmptyPromptCustomizations() {
|
|
@@ -21,10 +21,11 @@ function assertNonEmptyString(value, label) {
|
|
|
21
21
|
}
|
|
22
22
|
}
|
|
23
23
|
function validateCuaModel(value, label) {
|
|
24
|
-
|
|
24
|
+
const upgraded = typeof value === "string" ? upgradeLegacyCuaModel(value) : value;
|
|
25
|
+
if (!isValidCuaModel(upgraded)) {
|
|
25
26
|
throw new Error(`${label} must be one of: ${CUA_MODELS.join(", ")}.`);
|
|
26
27
|
}
|
|
27
|
-
return
|
|
28
|
+
return upgraded;
|
|
28
29
|
}
|
|
29
30
|
function validateAimAssistMode(value, label) {
|
|
30
31
|
if (typeof value !== "string" || !AIM_ASSIST_MODES.includes(value)) {
|
|
@@ -44,8 +45,8 @@ function readEnvDefaultCuaModel() {
|
|
|
44
45
|
// OpenAI model, stay on the OpenAI default rather than crossing over to the
|
|
45
46
|
// global (Loadmill) default.
|
|
46
47
|
if (typeof envValue === "string" && envValue.trim()) {
|
|
47
|
-
const
|
|
48
|
-
return isOpenAiCuaModel(
|
|
48
|
+
const upgraded = upgradeLegacyCuaModel(envValue.trim());
|
|
49
|
+
return isOpenAiCuaModel(upgraded) ? upgraded : DEFAULT_OPENAI_CUA_MODEL;
|
|
49
50
|
}
|
|
50
51
|
return resolveCuaModel(envValue);
|
|
51
52
|
}
|
|
@@ -12,7 +12,7 @@ const SUBCOMMANDS = new Set(["install", "status", "uninstall"]);
|
|
|
12
12
|
const ALLOWED_OPTIONS = new Set(["_", "local", "force", "h", "help"]);
|
|
13
13
|
const COMMAND_DESCRIPTIONS = Object.freeze({
|
|
14
14
|
install: "Install Droid CUA skill for a coding agent",
|
|
15
|
-
status: "Show
|
|
15
|
+
status: "Show installation status and installed/bundled skill versions",
|
|
16
16
|
uninstall: "Uninstall Droid CUA skill for a coding agent",
|
|
17
17
|
});
|
|
18
18
|
function skillCommandHelp(subcommand) {
|
|
@@ -53,7 +53,7 @@ function skillRootHelp() {
|
|
|
53
53
|
"",
|
|
54
54
|
"Commands:",
|
|
55
55
|
" install Install the skill",
|
|
56
|
-
" status Show installation status",
|
|
56
|
+
" status Show installation status and skill versions",
|
|
57
57
|
" uninstall Uninstall the skill",
|
|
58
58
|
].join("\n");
|
|
59
59
|
}
|
|
@@ -103,6 +103,18 @@ async function directoryMatches(sourcePath, destinationPath) {
|
|
|
103
103
|
]);
|
|
104
104
|
return JSON.stringify(sourceSnapshot) === JSON.stringify(destinationSnapshot);
|
|
105
105
|
}
|
|
106
|
+
// Read the bundled skill's block-style metadata.version scalar without a YAML dependency.
|
|
107
|
+
async function readSkillVersion(skillPath) {
|
|
108
|
+
if (await pathType(skillPath) !== "directory")
|
|
109
|
+
return null;
|
|
110
|
+
if (await pathType(path.join(skillPath, "SKILL.md")) !== "file")
|
|
111
|
+
return null;
|
|
112
|
+
const markdown = await readFile(path.join(skillPath, "SKILL.md"), "utf8");
|
|
113
|
+
const frontmatter = markdown.replace(/\r\n/g, "\n").match(/^---\n([\s\S]*?)\n---(?:\n|$)/)?.[1];
|
|
114
|
+
const metadata = frontmatter?.match(/^metadata:[ \t]*\n((?:[ \t]+[^\n]*(?:\n|$))*)/m)?.[1];
|
|
115
|
+
const value = metadata?.match(/^ version:[ \t]*(?:"([^"\n]+)"|'([^'\n]+)'|([^\s#]+))[ \t]*(?:#.*)?$/m);
|
|
116
|
+
return value ? (value[1] ?? value[2] ?? value[3]) : null;
|
|
117
|
+
}
|
|
106
118
|
async function getInstallationState(sourcePath, destinationPath) {
|
|
107
119
|
if (await pathType(destinationPath) === "missing")
|
|
108
120
|
return "not-installed";
|
|
@@ -214,7 +226,12 @@ export async function runSkillCommand({ args, cwd = process.cwd(), home = os.hom
|
|
|
214
226
|
}
|
|
215
227
|
for (const destination of states) {
|
|
216
228
|
if (command.subcommand === "status") {
|
|
217
|
-
|
|
229
|
+
const [installedVersion, bundledVersion] = await Promise.all([
|
|
230
|
+
readSkillVersion(destination.path),
|
|
231
|
+
readSkillVersion(bundledSkillPath),
|
|
232
|
+
]);
|
|
233
|
+
const installedLabel = destination.state === "not-installed" ? "not installed" : installedVersion ?? "unknown";
|
|
234
|
+
output.log(`${destination.displayName}: ${stateLabel(destination.state)} at ${destination.path}\nInstalled skill version: ${installedLabel}\nBundled skill version: ${bundledVersion ?? "unknown"}`);
|
|
218
235
|
}
|
|
219
236
|
else if (command.subcommand === "install") {
|
|
220
237
|
if (destination.state === "installed") {
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export const DEFAULT_OPENAI_CUA_MODEL = "gpt-
|
|
1
|
+
export const DEFAULT_OPENAI_CUA_MODEL = "gpt-6-sol";
|
|
2
2
|
export const DEFAULT_LOADMILL_CUA_MODEL = "loadmill-smart";
|
|
3
3
|
export const DEFAULT_CUA_MODEL = DEFAULT_LOADMILL_CUA_MODEL;
|
|
4
4
|
export const CUA_CONVERSATION_POLICIES = Object.freeze({
|
|
@@ -28,27 +28,28 @@ export const CUA_MODEL_OPTIONS = Object.freeze([
|
|
|
28
28
|
label: "Loadmill - Beacon",
|
|
29
29
|
}),
|
|
30
30
|
Object.freeze({
|
|
31
|
-
id: "gpt-
|
|
31
|
+
id: "gpt-6-sol",
|
|
32
32
|
providerMode: "openai",
|
|
33
33
|
providerLabel: "OpenAI",
|
|
34
|
-
name: "GPT-
|
|
35
|
-
label: "OpenAI GPT-
|
|
34
|
+
name: "GPT-6 Sol",
|
|
35
|
+
label: "OpenAI GPT-6 Sol",
|
|
36
36
|
}),
|
|
37
37
|
Object.freeze({
|
|
38
|
-
id: "gpt-
|
|
38
|
+
id: "gpt-6-luna",
|
|
39
39
|
providerMode: "openai",
|
|
40
40
|
providerLabel: "OpenAI",
|
|
41
|
-
name: "GPT-
|
|
42
|
-
label: "OpenAI GPT-
|
|
43
|
-
}),
|
|
44
|
-
Object.freeze({
|
|
45
|
-
id: "gpt-5.4",
|
|
46
|
-
providerMode: "openai",
|
|
47
|
-
providerLabel: "OpenAI",
|
|
48
|
-
name: "GPT-5.4",
|
|
49
|
-
label: "OpenAI GPT-5.4",
|
|
41
|
+
name: "GPT-6 Luna",
|
|
42
|
+
label: "OpenAI GPT-6 Luna",
|
|
50
43
|
}),
|
|
51
44
|
]);
|
|
45
|
+
const LEGACY_OPENAI_CUA_MODELS = Object.freeze({
|
|
46
|
+
"gpt-5.6-terra": "gpt-6-sol",
|
|
47
|
+
"gpt-5.6-luna": "gpt-6-luna",
|
|
48
|
+
"gpt-5.4": "gpt-6-sol",
|
|
49
|
+
});
|
|
50
|
+
export function upgradeLegacyCuaModel(value) {
|
|
51
|
+
return LEGACY_OPENAI_CUA_MODELS[value] ?? value;
|
|
52
|
+
}
|
|
52
53
|
export const CUA_MODELS = Object.freeze(CUA_MODEL_OPTIONS.map((option) => option.id));
|
|
53
54
|
export const OPENAI_CUA_MODELS = Object.freeze(CUA_MODEL_OPTIONS
|
|
54
55
|
.filter((option) => option.providerMode === "openai")
|
|
@@ -70,7 +71,8 @@ export function isLoadmillCuaModel(value) {
|
|
|
70
71
|
return LOADMILL_CUA_MODEL_SET.has(value);
|
|
71
72
|
}
|
|
72
73
|
export function resolveCuaModel(value) {
|
|
73
|
-
|
|
74
|
+
const upgraded = upgradeLegacyCuaModel(value);
|
|
75
|
+
return isValidCuaModel(upgraded) ? upgraded : DEFAULT_CUA_MODEL;
|
|
74
76
|
}
|
|
75
77
|
export function resolveCuaModelProviderMode(value) {
|
|
76
78
|
const normalized = resolveCuaModel(value);
|
|
@@ -3,29 +3,26 @@ const OPENAI_GPT_5_4_PRICING = Object.freeze({
|
|
|
3
3
|
inputPerMillionUsd: 2.5,
|
|
4
4
|
});
|
|
5
5
|
const OPENAI_GPT_5_6_TERRA_PRICING = Object.freeze({
|
|
6
|
-
inputPerMillionUsd: 2
|
|
6
|
+
inputPerMillionUsd: 2,
|
|
7
7
|
});
|
|
8
8
|
const OPENAI_GPT_5_6_LUNA_PRICING = Object.freeze({
|
|
9
|
-
inputPerMillionUsd:
|
|
9
|
+
inputPerMillionUsd: 0.2,
|
|
10
|
+
});
|
|
11
|
+
const OPENAI_GPT_6_SOL_PRICING = Object.freeze({
|
|
12
|
+
inputPerMillionUsd: 2,
|
|
13
|
+
});
|
|
14
|
+
const OPENAI_GPT_6_LUNA_PRICING = Object.freeze({
|
|
15
|
+
inputPerMillionUsd: 0.1,
|
|
10
16
|
});
|
|
11
17
|
const MODEL_PRICING = Object.freeze({
|
|
12
|
-
"gpt-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
"gpt-5.6-luna":
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
"
|
|
19
|
-
|
|
20
|
-
}),
|
|
21
|
-
// Loadmill Smart currently routes to GPT-5.6 Terra behind the Loadmill LLM gateway.
|
|
22
|
-
// Keep this explicit so pricing can be updated here if the backing model changes.
|
|
23
|
-
"loadmill-smart": Object.freeze({
|
|
24
|
-
...OPENAI_GPT_5_6_TERRA_PRICING,
|
|
25
|
-
}),
|
|
26
|
-
"loadmill-pulse": Object.freeze({
|
|
27
|
-
...OPENAI_GPT_5_6_LUNA_PRICING,
|
|
28
|
-
}),
|
|
18
|
+
"gpt-6-sol": OPENAI_GPT_6_SOL_PRICING,
|
|
19
|
+
"gpt-6-luna": OPENAI_GPT_6_LUNA_PRICING,
|
|
20
|
+
"gpt-5.6-terra": OPENAI_GPT_5_6_TERRA_PRICING,
|
|
21
|
+
"gpt-5.6-luna": OPENAI_GPT_5_6_LUNA_PRICING,
|
|
22
|
+
"gpt-5.4": OPENAI_GPT_5_4_PRICING,
|
|
23
|
+
// Loadmill Smart and Pulse route to GPT-6 Sol and Luna, respectively.
|
|
24
|
+
"loadmill-smart": OPENAI_GPT_6_SOL_PRICING,
|
|
25
|
+
"loadmill-pulse": OPENAI_GPT_6_LUNA_PRICING,
|
|
29
26
|
});
|
|
30
27
|
function finiteNumber(value) {
|
|
31
28
|
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : 0;
|
|
@@ -41,6 +38,12 @@ export function normalizeDroidCuaCostModel(model) {
|
|
|
41
38
|
if (typeof model !== "string") {
|
|
42
39
|
return null;
|
|
43
40
|
}
|
|
41
|
+
if (model === "gpt-6-sol" || model.startsWith("gpt-6-sol-")) {
|
|
42
|
+
return "gpt-6-sol";
|
|
43
|
+
}
|
|
44
|
+
if (model === "gpt-6-luna" || model.startsWith("gpt-6-luna-")) {
|
|
45
|
+
return "gpt-6-luna";
|
|
46
|
+
}
|
|
44
47
|
if (model === "gpt-5.6-terra" || model.startsWith("gpt-5.6-terra-")) {
|
|
45
48
|
return "gpt-5.6-terra";
|
|
46
49
|
}
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
import { buildActionEventMeta } from "../device/action-event-meta.js";
|
|
2
2
|
import { buildTypeActionPayload, buildTypeActionText, } from "../device/sensitive-action.js";
|
|
3
3
|
import { getConfiguredStepDelayMs } from "../utils/step-delay.js";
|
|
4
|
+
import { getWindowsVisibleBounds } from "./windows-visible-bounds.js";
|
|
5
|
+
import sharp from "sharp";
|
|
4
6
|
const SCREENSHOT_WIDTH = 1440;
|
|
5
7
|
const ACCESSIBILITY_HIT_TOLERANCE = 12;
|
|
6
8
|
const MACOS_CAPTURE_INSETS = { left: 50, right: 50, top: 38, bottom: 62 };
|
|
@@ -26,6 +28,57 @@ function assertResult(result, label) {
|
|
|
26
28
|
throw new Error(result.content?.find((item) => item.type === "text")?.text || `${label} failed.`);
|
|
27
29
|
}
|
|
28
30
|
}
|
|
31
|
+
function readScreenshot(result) {
|
|
32
|
+
const image = result.content?.find((item) => item.type === "image");
|
|
33
|
+
const sizeText = result.content?.find((item) => item.type === "text")?.text || "";
|
|
34
|
+
const size = /^(\d+)x(\d+)$/.exec(sizeText.trim());
|
|
35
|
+
if (!image?.data || !size)
|
|
36
|
+
throw new Error("Desktop screenshot returned no image or dimensions.");
|
|
37
|
+
return { data: image.data, width: Number(size[1]), height: Number(size[2]) };
|
|
38
|
+
}
|
|
39
|
+
async function isBlackScreenshot(data) {
|
|
40
|
+
const { channels } = await sharp(Buffer.from(data, "base64")).stats();
|
|
41
|
+
return channels.slice(0, 3).every((channel) => channel.max === 0);
|
|
42
|
+
}
|
|
43
|
+
async function captureWindowFromDesktop(native, bounds) {
|
|
44
|
+
const display = native.getDisplaySize();
|
|
45
|
+
// Read full-resolution pixels; resizing the desktop first loses window detail.
|
|
46
|
+
const screen = native.takeScreenshot(undefined, undefined, 0, undefined, undefined);
|
|
47
|
+
if (!screen.base64)
|
|
48
|
+
throw new Error("Desktop screen capture returned no image.");
|
|
49
|
+
const left = Math.round(bounds.x * screen.width / display.width);
|
|
50
|
+
const top = Math.round(bounds.y * screen.height / display.height);
|
|
51
|
+
const width = Math.round(bounds.width * screen.width / display.width);
|
|
52
|
+
const height = Math.round(bounds.height * screen.height / display.height);
|
|
53
|
+
const visibleLeft = Math.max(0, left);
|
|
54
|
+
const visibleTop = Math.max(0, top);
|
|
55
|
+
const visibleRight = Math.min(screen.width, left + width);
|
|
56
|
+
const visibleBottom = Math.min(screen.height, top + height);
|
|
57
|
+
if (visibleRight <= visibleLeft || visibleBottom <= visibleTop) {
|
|
58
|
+
throw new Error("The window must overlap the primary display for Windows screen capture.");
|
|
59
|
+
}
|
|
60
|
+
let image = sharp(Buffer.from(screen.base64, "base64")).extract({
|
|
61
|
+
left: visibleLeft,
|
|
62
|
+
top: visibleTop,
|
|
63
|
+
width: visibleRight - visibleLeft,
|
|
64
|
+
height: visibleBottom - visibleTop,
|
|
65
|
+
});
|
|
66
|
+
if (visibleLeft !== left || visibleTop !== top || visibleRight !== left + width || visibleBottom !== top + height) {
|
|
67
|
+
image = image.extend({
|
|
68
|
+
left: visibleLeft - left,
|
|
69
|
+
top: visibleTop - top,
|
|
70
|
+
right: left + width - visibleRight,
|
|
71
|
+
bottom: top + height - visibleBottom,
|
|
72
|
+
background: "black",
|
|
73
|
+
});
|
|
74
|
+
}
|
|
75
|
+
// Materialize the crop and padding before resize (Sharp can reorder operations).
|
|
76
|
+
const cropped = await image.png().toBuffer();
|
|
77
|
+
const { data, info } = await sharp(cropped)
|
|
78
|
+
.resize({ width: SCREENSHOT_WIDTH, withoutEnlargement: true })
|
|
79
|
+
.png().toBuffer({ resolveWithObject: true });
|
|
80
|
+
return { data: data.toString("base64"), width: info.width, height: info.height };
|
|
81
|
+
}
|
|
29
82
|
function sameBounds(left, right) {
|
|
30
83
|
return left && right && left.x === right.x && left.y === right.y && left.width === right.width && left.height === right.height;
|
|
31
84
|
}
|
|
@@ -63,7 +116,7 @@ async function createLibraryClient() {
|
|
|
63
116
|
]);
|
|
64
117
|
return connectInProcess(createComputerUseServer({ profile: "ax", vision: true }));
|
|
65
118
|
}
|
|
66
|
-
async function
|
|
119
|
+
async function loadNativeDriver() {
|
|
67
120
|
const { loadNative } = await import("@zavora-ai/computer-use-mcp/host-native");
|
|
68
121
|
return loadNative();
|
|
69
122
|
}
|
|
@@ -183,9 +236,9 @@ function normalizeShortcut(keys, platform) {
|
|
|
183
236
|
return keys.map((key) => aliases[String(key).toUpperCase()] || String(key).toLowerCase()).join("+");
|
|
184
237
|
}
|
|
185
238
|
/**
|
|
186
|
-
* @param {{ app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", client?: any, accessibility?: any }} options
|
|
239
|
+
* @param {{ app?: string, windowId?: number | null, excludePids?: number[], platform?: string, inputMode?: "accessibility" | "physical", focusOnConnect?: boolean, client?: any, accessibility?: any, visibleBoundsReader?: typeof getWindowsVisibleBounds }} options
|
|
187
240
|
*/
|
|
188
|
-
export async function createDesktopController({ app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", client: injectedClient = null, accessibility: injectedAccessibility = null, } = {}) {
|
|
241
|
+
export async function createDesktopController({ app: requestedApp, windowId: requestedWindowId = null, excludePids = [], platform = process.platform, inputMode = "accessibility", focusOnConnect = false, client: injectedClient = null, accessibility: injectedAccessibility = null, visibleBoundsReader = getWindowsVisibleBounds, } = {}) {
|
|
189
242
|
if (!SUPPORTED_PLATFORMS.has(platform)) {
|
|
190
243
|
throw new Error("Desktop control is currently supported only on macOS and Windows.");
|
|
191
244
|
}
|
|
@@ -201,59 +254,91 @@ export async function createDesktopController({ app: requestedApp, windowId: req
|
|
|
201
254
|
let targetWindow = chooseWindow(windows, requestedWindowId == null ? null : Number(requestedWindowId));
|
|
202
255
|
let capture = null;
|
|
203
256
|
let deviceInfo = null;
|
|
204
|
-
let
|
|
257
|
+
let native = injectedAccessibility;
|
|
258
|
+
let lastVisibleBounds = null;
|
|
259
|
+
let fallbackLogged = false;
|
|
205
260
|
async function getWindow() {
|
|
206
261
|
return parseJsonResult(await client.getWindow(targetWindow.windowId), "Window lookup");
|
|
207
262
|
}
|
|
208
|
-
async function
|
|
209
|
-
|
|
210
|
-
|
|
263
|
+
async function focus() {
|
|
264
|
+
const activation = parseJsonResult(await client.activateWindow(targetWindow.windowId, 2000), "Window activation");
|
|
265
|
+
const focused = await getWindow();
|
|
266
|
+
if (!activation.activated || !focused.isFocused) {
|
|
267
|
+
throw new Error("Droid could not focus the selected desktop window, so no input was sent.");
|
|
268
|
+
}
|
|
269
|
+
}
|
|
270
|
+
async function captureFromDesktop(window) {
|
|
271
|
+
if (!window.isFocused) {
|
|
272
|
+
if (inputMode !== "physical") {
|
|
273
|
+
throw new Error("Windows desktop capture requires the target window to be visible and focused. Bring it to the foreground and retry.");
|
|
274
|
+
}
|
|
275
|
+
await focus();
|
|
276
|
+
window = await getWindow();
|
|
277
|
+
}
|
|
278
|
+
native ||= await loadNativeDriver();
|
|
279
|
+
if (lastVisibleBounds?.windowId !== targetWindow.windowId || !sameBounds(lastVisibleBounds.outer, window.bounds)) {
|
|
280
|
+
lastVisibleBounds = {
|
|
281
|
+
windowId: targetWindow.windowId,
|
|
282
|
+
outer: { ...window.bounds },
|
|
283
|
+
visible: await visibleBoundsReader(targetWindow.windowId, window.bounds),
|
|
284
|
+
};
|
|
285
|
+
}
|
|
286
|
+
const actionBounds = lastVisibleBounds.visible;
|
|
287
|
+
const image = await captureWindowFromDesktop(native, actionBounds);
|
|
288
|
+
if (!fallbackLogged) {
|
|
289
|
+
console.warn("Windows window capture returned all-black pixels; using a visible-desktop crop. Keep the target unobstructed on the primary display.");
|
|
290
|
+
fallbackLogged = true;
|
|
291
|
+
}
|
|
292
|
+
return { ...image, bounds: { ...window.bounds }, actionBounds: { ...actionBounds } };
|
|
293
|
+
}
|
|
294
|
+
async function screenshot({ allowDesktopFallback = true } = {}) {
|
|
211
295
|
try {
|
|
212
296
|
const currentWindows = readWindows(await client.listWindows(application.bundleId));
|
|
213
297
|
const focusedWindow = currentWindows.find((candidate) => candidate.isFocused && candidate.isOnScreen);
|
|
214
298
|
if (focusedWindow)
|
|
215
299
|
targetWindow = focusedWindow;
|
|
216
|
-
window = await getWindow();
|
|
300
|
+
const window = await getWindow();
|
|
217
301
|
if (!window.isOnScreen)
|
|
218
302
|
throw new Error("The selected desktop window is no longer visible.");
|
|
219
|
-
result = await client.screenshot({
|
|
303
|
+
const result = await client.screenshot({
|
|
220
304
|
width: SCREENSHOT_WIDTH,
|
|
221
305
|
quality: 0,
|
|
222
306
|
target_window_id: targetWindow.windowId,
|
|
223
307
|
});
|
|
224
308
|
assertResult(result, "Desktop screenshot");
|
|
309
|
+
let image = { ...readScreenshot(result), bounds: { ...window.bounds }, actionBounds: { ...window.bounds } };
|
|
310
|
+
if (allowDesktopFallback && platform === "win32" && await isBlackScreenshot(image.data)) {
|
|
311
|
+
image = await captureFromDesktop(window);
|
|
312
|
+
}
|
|
313
|
+
capture = {
|
|
314
|
+
bounds: image.bounds,
|
|
315
|
+
actionBounds: image.actionBounds,
|
|
316
|
+
width: image.width,
|
|
317
|
+
height: image.height,
|
|
318
|
+
contentBounds: getScreenshotContentBounds(image.width, image.height, image.bounds, platform),
|
|
319
|
+
};
|
|
320
|
+
if (deviceInfo) {
|
|
321
|
+
deviceInfo.device_width = image.actionBounds.width;
|
|
322
|
+
deviceInfo.device_height = image.actionBounds.height;
|
|
323
|
+
deviceInfo.scaled_width = image.width;
|
|
324
|
+
deviceInfo.scaled_height = image.height;
|
|
325
|
+
deviceInfo.scale = image.width / image.actionBounds.width;
|
|
326
|
+
deviceInfo.desktopWindowId = targetWindow.windowId;
|
|
327
|
+
}
|
|
328
|
+
return image.data;
|
|
225
329
|
}
|
|
226
330
|
catch (error) {
|
|
227
331
|
throw new Error(formatPermissionError(error, platform));
|
|
228
332
|
}
|
|
229
|
-
const image = result.content?.find((item) => item.type === "image");
|
|
230
|
-
const sizeText = result.content?.find((item) => item.type === "text")?.text || "";
|
|
231
|
-
const size = /^(\d+)x(\d+)$/.exec(sizeText.trim());
|
|
232
|
-
if (!image?.data || !size)
|
|
233
|
-
throw new Error("Desktop screenshot returned no image or dimensions.");
|
|
234
|
-
capture = {
|
|
235
|
-
bounds: { ...window.bounds },
|
|
236
|
-
width: Number(size[1]),
|
|
237
|
-
height: Number(size[2]),
|
|
238
|
-
};
|
|
239
|
-
capture.contentBounds = getScreenshotContentBounds(capture.width, capture.height, window.bounds, platform);
|
|
240
|
-
if (deviceInfo) {
|
|
241
|
-
deviceInfo.device_width = window.bounds.width;
|
|
242
|
-
deviceInfo.device_height = window.bounds.height;
|
|
243
|
-
deviceInfo.scaled_width = capture.width;
|
|
244
|
-
deviceInfo.scaled_height = capture.height;
|
|
245
|
-
deviceInfo.scale = capture.width / window.bounds.width;
|
|
246
|
-
deviceInfo.desktopWindowId = targetWindow.windowId;
|
|
247
|
-
}
|
|
248
|
-
return image.data;
|
|
249
333
|
}
|
|
250
334
|
function scaledPoint(x, y) {
|
|
251
335
|
if (!capture)
|
|
252
336
|
throw new Error("Capture the desktop window before executing an action.");
|
|
253
337
|
const content = capture.contentBounds;
|
|
338
|
+
const bounds = capture.actionBounds;
|
|
254
339
|
return [
|
|
255
|
-
Math.round(
|
|
256
|
-
Math.round(
|
|
340
|
+
Math.round(bounds.x + ((x - content.x) * bounds.width) / content.width),
|
|
341
|
+
Math.round(bounds.y + ((y - content.y) * bounds.height) / content.height),
|
|
257
342
|
];
|
|
258
343
|
}
|
|
259
344
|
async function prepareAction(requireFocus) {
|
|
@@ -265,11 +350,7 @@ export async function createDesktopController({ app: requestedApp, windowId: req
|
|
|
265
350
|
}
|
|
266
351
|
if (!requireFocus)
|
|
267
352
|
return;
|
|
268
|
-
|
|
269
|
-
const focused = await getWindow();
|
|
270
|
-
if (!activation.activated || !focused.isFocused) {
|
|
271
|
-
throw new Error("Droid could not focus the selected desktop window, so no input was sent.");
|
|
272
|
-
}
|
|
353
|
+
await focus();
|
|
273
354
|
}
|
|
274
355
|
const targetOptions = () => ({ targetWindowId: targetWindow.windowId, focusStrategy: "strict" });
|
|
275
356
|
async function performCursorFreeAction(actionType, x, y, direction = null) {
|
|
@@ -277,8 +358,8 @@ export async function createDesktopController({ app: requestedApp, windowId: req
|
|
|
277
358
|
const selected = chooseAccessibilityElement(tree, x, y, actionType, direction);
|
|
278
359
|
if (!selected)
|
|
279
360
|
throw new Error(cursorFreeError(actionType));
|
|
280
|
-
|
|
281
|
-
const result =
|
|
361
|
+
native ||= await loadNativeDriver();
|
|
362
|
+
const result = native.performAction(targetWindow.windowId, selected.element.role, selected.element.label, selected.action);
|
|
282
363
|
if (!result?.performed)
|
|
283
364
|
throw new Error(`${cursorFreeError(actionType)}${result?.reason ? ` Reason: ${result.reason}.` : ""}`);
|
|
284
365
|
}
|
|
@@ -375,16 +456,19 @@ export async function createDesktopController({ app: requestedApp, windowId: req
|
|
|
375
456
|
return { success: false, error: message };
|
|
376
457
|
}
|
|
377
458
|
}
|
|
378
|
-
|
|
459
|
+
if (focusOnConnect)
|
|
460
|
+
await focus();
|
|
461
|
+
// Without focus, connection only probes the window; the desktop may be covered.
|
|
462
|
+
await screenshot({ allowDesktopFallback: focusOnConnect });
|
|
379
463
|
deviceInfo = {
|
|
380
464
|
platform: "desktop",
|
|
381
465
|
source: "local",
|
|
382
466
|
device_name: `${application.displayName} — ${targetWindow.title || "Untitled"}`,
|
|
383
|
-
device_width:
|
|
384
|
-
device_height:
|
|
467
|
+
device_width: capture.actionBounds.width,
|
|
468
|
+
device_height: capture.actionBounds.height,
|
|
385
469
|
scaled_width: capture.width,
|
|
386
470
|
scaled_height: capture.height,
|
|
387
|
-
scale: capture.width /
|
|
471
|
+
scale: capture.width / capture.actionBounds.width,
|
|
388
472
|
desktopAppId: application.bundleId,
|
|
389
473
|
desktopWindowId: targetWindow.windowId,
|
|
390
474
|
desktopInputMode: inputMode,
|
|
@@ -394,7 +478,8 @@ export async function createDesktopController({ app: requestedApp, windowId: req
|
|
|
394
478
|
window: targetWindow,
|
|
395
479
|
deviceId: `desktop:${targetWindow.windowId}`,
|
|
396
480
|
deviceInfo,
|
|
397
|
-
|
|
481
|
+
focus,
|
|
482
|
+
getScreenshotAsBase64: () => screenshot(),
|
|
398
483
|
handleModelAction: handleAction,
|
|
399
484
|
close: () => client.close(),
|
|
400
485
|
};
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
import { execFile } from "node:child_process";
|
|
2
|
+
import { promisify } from "node:util";
|
|
3
|
+
const runFile = promisify(execFile);
|
|
4
|
+
const RECT_SCRIPT = `
|
|
5
|
+
Add-Type -TypeDefinition @'
|
|
6
|
+
using System;
|
|
7
|
+
using System.Runtime.InteropServices;
|
|
8
|
+
public static class WindowFrame {
|
|
9
|
+
public const int DWMWA_EXTENDED_FRAME_BOUNDS = 9;
|
|
10
|
+
[StructLayout(LayoutKind.Sequential)] public struct Rect { public int Left; public int Top; public int Right; public int Bottom; }
|
|
11
|
+
[DllImport("user32.dll")] public static extern bool GetWindowRect(IntPtr handle, out Rect rect);
|
|
12
|
+
[DllImport("dwmapi.dll")] public static extern int DwmGetWindowAttribute(IntPtr handle, int attribute, out Rect rect, int size);
|
|
13
|
+
}
|
|
14
|
+
'@
|
|
15
|
+
$handle = [IntPtr]::new([long]WINDOW_ID)
|
|
16
|
+
$outer = [WindowFrame+Rect]::new()
|
|
17
|
+
$visible = [WindowFrame+Rect]::new()
|
|
18
|
+
if (-not [WindowFrame]::GetWindowRect($handle, [ref]$outer)) { exit 1 }
|
|
19
|
+
$result = [WindowFrame]::DwmGetWindowAttribute($handle, [WindowFrame]::DWMWA_EXTENDED_FRAME_BOUNDS, [ref]$visible, 16)
|
|
20
|
+
if ($result -ne 0) { exit 1 }
|
|
21
|
+
@($outer.Left, $outer.Top, $outer.Right, $outer.Bottom, $visible.Left, $visible.Top, $visible.Right, $visible.Bottom) -join ','
|
|
22
|
+
`;
|
|
23
|
+
export async function getWindowsVisibleBounds(windowId, bounds) {
|
|
24
|
+
if (process.platform !== "win32")
|
|
25
|
+
return bounds;
|
|
26
|
+
if (!Number.isSafeInteger(windowId))
|
|
27
|
+
return bounds;
|
|
28
|
+
try {
|
|
29
|
+
// GetWindowRect includes invisible resize borders; DWM reports the pixels actually drawn.
|
|
30
|
+
const script = RECT_SCRIPT.replace("WINDOW_ID", String(windowId));
|
|
31
|
+
const { stdout } = await runFile("powershell.exe", ["-NoProfile", "-NonInteractive", "-Command", script], {
|
|
32
|
+
windowsHide: true,
|
|
33
|
+
timeout: 5000,
|
|
34
|
+
});
|
|
35
|
+
const [outerLeft, outerTop, outerRight, outerBottom, left, top, right, bottom] = stdout.trim().split(",").map(Number);
|
|
36
|
+
const scaleX = bounds.width / (outerRight - outerLeft);
|
|
37
|
+
const scaleY = bounds.height / (outerBottom - outerTop);
|
|
38
|
+
const insetLeft = Math.round((left - outerLeft) * scaleX);
|
|
39
|
+
const insetTop = Math.round((top - outerTop) * scaleY);
|
|
40
|
+
const insetRight = Math.round((outerRight - right) * scaleX);
|
|
41
|
+
const insetBottom = Math.round((outerBottom - bottom) * scaleY);
|
|
42
|
+
if (![insetLeft, insetTop, insetRight, insetBottom].every((value) => Number.isFinite(value) && value >= 0)) {
|
|
43
|
+
throw new Error("Invalid window frame insets.");
|
|
44
|
+
}
|
|
45
|
+
const width = bounds.width - insetLeft - insetRight;
|
|
46
|
+
const height = bounds.height - insetTop - insetBottom;
|
|
47
|
+
if (width <= 0 || height <= 0)
|
|
48
|
+
throw new Error("Invalid visible window dimensions.");
|
|
49
|
+
return { x: bounds.x + insetLeft, y: bounds.y + insetTop, width, height };
|
|
50
|
+
}
|
|
51
|
+
catch {
|
|
52
|
+
console.warn("Windows visible-frame lookup failed; using outer window bounds, which may include invisible resize borders.");
|
|
53
|
+
return bounds;
|
|
54
|
+
}
|
|
55
|
+
}
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
import { createHash, randomUUID } from "node:crypto";
|
|
2
|
+
import { findDeepestErrorCode } from "../../utils/error-details.js";
|
|
2
3
|
export const MAX_SCREENSHOTS_PER_REPORT = 1000;
|
|
3
4
|
export const MAX_REPORT_OBJECT_BYTES = 10 * 1024 * 1024;
|
|
4
|
-
function warn(session, message) {
|
|
5
|
+
function warn(session, message, phase) {
|
|
5
6
|
session.failed = true;
|
|
7
|
+
session.onFailure?.({ uploadId: session.uploadId, phase, objectName: null, httpStatus: null, errorName: "ValidationError", errorCode: null });
|
|
6
8
|
session.onWarning?.(message);
|
|
7
9
|
}
|
|
8
10
|
function getHttpStatus(error) {
|
|
@@ -16,6 +18,20 @@ function warnAboutUploadFailure(session) {
|
|
|
16
18
|
const status = session.uploadFailureStatus ? ` (HTTP ${session.uploadFailureStatus})` : "";
|
|
17
19
|
session.onWarning?.(`Cloud report upload failed${status}; further uploads skipped.`);
|
|
18
20
|
}
|
|
21
|
+
function recordFailure(session, phase, objectName, error) {
|
|
22
|
+
if (session.uploadFailed)
|
|
23
|
+
return;
|
|
24
|
+
session.uploadFailed = true;
|
|
25
|
+
session.uploadFailureStatus = getHttpStatus(error);
|
|
26
|
+
session.onFailure?.({
|
|
27
|
+
uploadId: session.uploadId,
|
|
28
|
+
phase,
|
|
29
|
+
objectName,
|
|
30
|
+
httpStatus: session.uploadFailureStatus,
|
|
31
|
+
errorName: error instanceof Error ? error.name : typeof error,
|
|
32
|
+
errorCode: findDeepestErrorCode(error),
|
|
33
|
+
});
|
|
34
|
+
}
|
|
19
35
|
function decodeScreenshot(dataUrl) {
|
|
20
36
|
const match = /^data:image\/png;base64,([a-zA-Z0-9+/=\s]+)$/.exec(dataUrl);
|
|
21
37
|
return match ? Buffer.from(match[1], "base64") : null;
|
|
@@ -46,13 +62,15 @@ export class DroidReportUploadManager {
|
|
|
46
62
|
* associateDroidRunReport: (runId: string, payload: object) => Promise<unknown>,
|
|
47
63
|
* },
|
|
48
64
|
* onWarning?: ((message: string) => void) | null,
|
|
65
|
+
* onFailure?: ((details: { uploadId: string, phase: string, objectName: string | null, httpStatus: string | null, errorName: string, errorCode: string | null }) => void) | null,
|
|
49
66
|
* }} options
|
|
50
67
|
*/
|
|
51
|
-
startSession({ loadmillClient, onWarning = null } = {}) {
|
|
68
|
+
startSession({ loadmillClient, onWarning = null, onFailure = null } = {}) {
|
|
52
69
|
const session = {
|
|
53
70
|
uploadId: randomUUID(),
|
|
54
71
|
loadmillClient,
|
|
55
72
|
onWarning,
|
|
73
|
+
onFailure,
|
|
56
74
|
sequence: 0,
|
|
57
75
|
screenshotCount: 0,
|
|
58
76
|
lastScreenshotObjectName: null,
|
|
@@ -79,15 +97,15 @@ export class DroidReportUploadManager {
|
|
|
79
97
|
}
|
|
80
98
|
const body = decodeScreenshot(dataUrl);
|
|
81
99
|
if (!body) {
|
|
82
|
-
warn(session, "Failed to upload Droid report screenshot: unsupported image data.");
|
|
100
|
+
warn(session, "Failed to upload Droid report screenshot: unsupported image data.", "validate_screenshot");
|
|
83
101
|
return null;
|
|
84
102
|
}
|
|
85
103
|
if (body.length > MAX_REPORT_OBJECT_BYTES) {
|
|
86
|
-
warn(session, `Droid report screenshot exceeds ${MAX_REPORT_OBJECT_BYTES} bytes
|
|
104
|
+
warn(session, `Droid report screenshot exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`, "validate_screenshot");
|
|
87
105
|
return null;
|
|
88
106
|
}
|
|
89
107
|
if (session.screenshotCount >= MAX_SCREENSHOTS_PER_REPORT) {
|
|
90
|
-
warn(session, `Droid report exceeds ${MAX_SCREENSHOTS_PER_REPORT} screenshots
|
|
108
|
+
warn(session, `Droid report exceeds ${MAX_SCREENSHOTS_PER_REPORT} screenshots.`, "validate_screenshot");
|
|
91
109
|
return null;
|
|
92
110
|
}
|
|
93
111
|
const name = `screenshots/${String(++session.sequence).padStart(4, "0")}.png`;
|
|
@@ -107,7 +125,16 @@ export class DroidReportUploadManager {
|
|
|
107
125
|
session.finishing = true;
|
|
108
126
|
session.finishPromise = this.#finish(session, { savedRunId, html }).catch((error) => {
|
|
109
127
|
const status = getHttpStatus(error);
|
|
110
|
-
|
|
128
|
+
session.onFailure?.({
|
|
129
|
+
uploadId: session.uploadId,
|
|
130
|
+
phase: "associate_report",
|
|
131
|
+
objectName: null,
|
|
132
|
+
httpStatus: status,
|
|
133
|
+
errorName: error instanceof Error ? error.name : typeof error,
|
|
134
|
+
errorCode: findDeepestErrorCode(error),
|
|
135
|
+
});
|
|
136
|
+
session.failed = true;
|
|
137
|
+
session.onWarning?.(`Failed to publish Droid run report${status ? ` (HTTP ${status})` : ""}.`);
|
|
111
138
|
return { published: false, reason: "publish_failed" };
|
|
112
139
|
}).finally(() => {
|
|
113
140
|
this.sessions.delete(session);
|
|
@@ -125,7 +152,7 @@ export class DroidReportUploadManager {
|
|
|
125
152
|
}
|
|
126
153
|
const body = Buffer.from(html, "utf8");
|
|
127
154
|
if (body.length > MAX_REPORT_OBJECT_BYTES) {
|
|
128
|
-
warn(session, `Droid HTML report exceeds ${MAX_REPORT_OBJECT_BYTES} bytes
|
|
155
|
+
warn(session, `Droid HTML report exceeds ${MAX_REPORT_OBJECT_BYTES} bytes.`, "validate_report");
|
|
129
156
|
return { published: false, reason: "too_large" };
|
|
130
157
|
}
|
|
131
158
|
const htmlUpload = await this.#enqueue(session, {
|
|
@@ -198,6 +225,7 @@ export class DroidReportUploadManager {
|
|
|
198
225
|
}
|
|
199
226
|
async #runTask(task) {
|
|
200
227
|
let result = { uploaded: false, key: null };
|
|
228
|
+
let phase = "request_policy";
|
|
201
229
|
try {
|
|
202
230
|
if (task.session.uploadFailed) {
|
|
203
231
|
return;
|
|
@@ -207,13 +235,13 @@ export class DroidReportUploadManager {
|
|
|
207
235
|
if (!uploadPolicy?.key) {
|
|
208
236
|
throw new Error("Loadmill did not return an upload policy.");
|
|
209
237
|
}
|
|
238
|
+
phase = "upload_object";
|
|
210
239
|
await uploadObject(uploadPolicy, task.object.body, task.object.name.split("/").at(-1), task.object.contentType, this.fetchImpl);
|
|
211
240
|
result = { uploaded: true, key: uploadPolicy.key };
|
|
212
241
|
}
|
|
213
242
|
catch (error) {
|
|
214
243
|
task.session.failed = true;
|
|
215
|
-
task.session.
|
|
216
|
-
task.session.uploadFailureStatus ??= getHttpStatus(error);
|
|
244
|
+
recordFailure(task.session, phase, task.object.name, error);
|
|
217
245
|
}
|
|
218
246
|
finally {
|
|
219
247
|
task.session.pending.delete(task.promise);
|
package/package.json
CHANGED
package/skill/droid-cua/SKILL.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: droid-cua
|
|
3
3
|
description: Operate and improve Droid CUA `.dcua` test projects. Use when the user explicitly names Droid CUA or @loadmill/droid-cua, or asks to author, configure, run, diagnose, heal, stabilize, or speed up `.dcua` tests with the Droid CUA CLI or desktop app. Do not use for general web browsing, website development, browser automation, generic mobile work, or non-Droid test tasks.
|
|
4
|
+
metadata:
|
|
5
|
+
version: "1.0.8"
|
|
4
6
|
---
|
|
5
7
|
|
|
6
8
|
# Droid CUA
|
|
@@ -12,7 +14,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
|
|
|
12
14
|
1. Inspect the active test project before creating files. Find existing `.dcua` tests, `context.md`, `test-data.md`, `.secrets`, `.env`, and `ci/droid-cua.json` without reading secret values. Do not search other projects or the user's home directory for credentials.
|
|
13
15
|
2. Determine the requested platform and target from the task or project. Ask the user only when the choice is unclear.
|
|
14
16
|
3. Check authentication and environment readiness before the first run. Read [project-and-auth.md](references/project-and-auth.md) for project boundaries and token handling. Read [onboarding-and-troubleshooting.md](references/onboarding-and-troubleshooting.md) when setup, dependency, device, simulator, browser, or connection work is needed.
|
|
15
|
-
4.
|
|
17
|
+
4. Identify the intended coverage, starting state, scenario inputs, and cleanup; author or update the smallest useful test that preserves them. Read [authoring-tests.md](references/authoring-tests.md) before changing `.dcua`, context, test data, secrets, or config.
|
|
16
18
|
5. Before execution, present the proposed tests and target to the user, resolve any questions, and get their confirmation. Read [running-and-healing.md](references/running-and-healing.md) for the required pre-run check.
|
|
17
19
|
6. Run the confirmed, narrowest relevant test and monitor it to completion. Keep the user oriented during the run by surfacing meaningful live screenshots, not just terminal output.
|
|
18
20
|
7. For a new, changed, flaky, slow, or explicitly optimized test, continue with the evidence-driven author-run-improve loop in [iterative-improvement.md](references/iterative-improvement.md). A passing run is a baseline, not proof of readiness.
|
|
@@ -27,7 +29,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
|
|
|
27
29
|
- Get approval before installing dependencies, running `sudo`, accepting licenses, or changing system configuration.
|
|
28
30
|
- During setup, do not create or edit user tests. Rerun only the failed readiness check, then resume the original task.
|
|
29
31
|
- During healing, edit only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never modify application code under the guise of healing a test.
|
|
30
|
-
- Never weaken or remove an assertion merely to make a failing run pass.
|
|
32
|
+
- Never weaken required coverage or remove an assertion merely to make a failing run pass. Agent-added, out-of-scope assertions may be removed only with a coverage-based explanation; see [authoring-tests.md](references/authoring-tests.md).
|
|
31
33
|
- Keep one canonical context file and one canonical Droid CUA config file per test project. Reuse and update those shared files; never create test-specific context or config copies for a new or improved test.
|
|
32
34
|
- When improving an existing test, keep its complete flow in one independently runnable `.dcua` file. Do not split it into ordered or state-dependent tests as an optimization; Droid CUA does not currently guarantee that separate tests will run consecutively or share state.
|
|
33
35
|
- Before rerunning a long test, apply the smallest evidence-supported batch that addresses its known bottlenecks. The batch may include multiple `.dcua`, context, test-data, or config improvements when each has a clear rationale and preserves coverage. Record the complete change set; isolate or revert individual changes only if the combined result regresses or remains ambiguous.
|
|
@@ -39,7 +41,7 @@ Use the existing `droid-cua` CLI and repository files. Do not invent commands, f
|
|
|
39
41
|
|
|
40
42
|
- Prefer an existing project convention over creating a parallel structure.
|
|
41
43
|
- Prefer headless runs for agent work because they are observable and return deterministic exit codes.
|
|
42
|
-
-
|
|
44
|
+
- Preserve the user's intended coverage. Write one readable micro-goal per step: a small user task with a recognizable completion point. During improvement, simplify before adding detail, and retain only detail that is necessary and supported by evidence. Do not optimize for fewer lines or compress several goals into one instruction.
|
|
43
45
|
- Use explicit, user-observable outcomes in tests rather than implementation details or coordinates.
|
|
44
46
|
- Diagnose the first divergence between the instruction and observed behavior before changing anything.
|
|
45
47
|
- Compare runs on the same target and app build, and clearly identify every test, context, data, model, or config change included in the optimization batch.
|
|
@@ -4,32 +4,91 @@
|
|
|
4
4
|
|
|
5
5
|
Find nearby `.dcua` tests and project files before writing. Match their directory layout, naming, vocabulary, starting state, and config conventions. Do not open `.env` or `.secrets`.
|
|
6
6
|
|
|
7
|
+
## Establish intended coverage
|
|
8
|
+
|
|
9
|
+
Before drafting or healing, identify the requested behavior, starting state, scenario inputs, business outcome, and cleanup from the user request and existing test. Keep this brief in the working plan; no separate specification file is required. Distinguish user-required checks from incidental checks introduced during earlier agent iterations. If an existing assertion's purpose is unclear, preserve it until its scope is resolved.
|
|
10
|
+
|
|
11
|
+
Preserve scenario choices during simplification: a full-day booking and a three-hour booking exercise different behavior. Adding UI cancellation also changes coverage; do it when requested, not automatically for every creation test.
|
|
12
|
+
|
|
13
|
+
Prefer predictable setup and data over conditional recovery. Use known cleanup and simple relative dates when appropriate for the project. A date such as next Monday assumes a working day and available account state; establish those assumptions rather than inventing a search for the earliest conflict-free date. Restarting the app is a project-specific starting condition, not a universal requirement. Include only setup needed by this scenario and supported by the request, project knowledge, or observed behavior. Do not import dates, reset flows, or other prerequisites from a nearby example just because they made that example runnable. If an unknown prerequisite matters, identify the gap rather than silently inventing a mandatory step.
|
|
14
|
+
|
|
7
15
|
## Write `.dcua` instructions
|
|
8
16
|
|
|
9
|
-
- Put one
|
|
10
|
-
-
|
|
11
|
-
-
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
- Use explicit assertions for important outcomes. Prefer stable visible facts such as a screen title, confirmation message, changed value, or navigation result.
|
|
15
|
-
- Use exact visible labels when naming controls or outcomes.
|
|
17
|
+
- Put one readable micro-goal on each non-empty line: a small user task with a recognizable completion point, such as choosing a room, selecting dates, or submitting a request. One goal can require several UI actions.
|
|
18
|
+
- Keep useful task boundaries. Do not compress login, navigation, data entry, and verification into a single instruction to reduce the line count.
|
|
19
|
+
- Specify the scenario choices precisely while leaving ordinary UI mechanics to Droid. Avoid coordinates, selectors, icon appearance, and instructions for every tap or keystroke.
|
|
20
|
+
- Use visible labels when they identify a meaningful choice or resolve ambiguity. Short navigation hints are welcome when useful, such as `Custom duration (scroll right in the duration options)`; do not require exact layout or incidental copy.
|
|
21
|
+
- Add lower-level guidance only when evidence shows it is necessary to complete the goal reliably, or the intermediate behavior is explicitly under test. Prefer clarifying the goal over appending a transcript of a failed run.
|
|
16
22
|
- Keep each new test focused on one behavior. Separate unrelated flows only when each resulting test is fully independent and does not rely on execution order or state created by another test.
|
|
17
|
-
- When updating or optimizing an existing test, preserve its complete flow in one `.dcua` file. Do not split it into sequential or dependent tests
|
|
18
|
-
- Use `//` comments sparingly for human context; comments are not executed.
|
|
19
|
-
|
|
23
|
+
- When updating or optimizing an existing test, preserve its complete intended flow in one independently runnable `.dcua` file. Do not split it into sequential or dependent tests without explicit user direction and verified orchestration.
|
|
24
|
+
- Use `//` comments sparingly for human context; comments are not executed. Never include secret values.
|
|
25
|
+
|
|
26
|
+
## Choose assertions by coverage
|
|
27
|
+
|
|
28
|
+
Ask: **If this check failed while the requested business outcome remained correct, would that be a defect this scenario is intended to catch?** If not, it generally should not be a standalone assertion here. Droid must observe intermediate states to perform actions; those observations do not all need to become test assertions.
|
|
29
|
+
|
|
30
|
+
Verify meaningful outcomes and identify the created or changed object by the relevant scenario inputs. Checking that the request for the selected room, dates, and duration is confirmed prevents an unrelated approved request from satisfying the test. Do not automatically assert every dialog closure, available option, summary label, or absence of a warning when the final outcome already covers the intended behavior.
|
|
31
|
+
|
|
32
|
+
Preserve explicit user requirements and genuine product failures. Remove an agent-added assertion only when it is demonstrably outside intended coverage or redundant with a retained check; explain that coverage-based reason, not merely that it failed or cost time. An intermediate control or exact message still needs an assertion when that behavior is the requested subject of the test.
|
|
33
|
+
|
|
34
|
+
## Example: readable booking journeys
|
|
35
|
+
|
|
36
|
+
These examples use a fictional room-booking app, generic test accounts, and invented flow and control names. They do not describe a customer's product. Assume the project maps the test account to configured secret keys and the named cleanup flow makes the sample room available. This hypothetical user explicitly wants a fresh app start and UI cancellation coverage. Adapt those choices to the actual project; neither is a universal requirement.
|
|
37
|
+
|
|
38
|
+
Full-day booking:
|
|
39
|
+
|
|
40
|
+
```text
|
|
41
|
+
Run the Loadmill "Reset sample room bookings" flow.
|
|
42
|
+
Open the booking app; if already open, fully close it and reopen it.
|
|
43
|
+
Log in as the test member, logging out first if another user is signed in.
|
|
44
|
+
Go to Home.
|
|
45
|
+
Start a room booking.
|
|
46
|
+
Choose "Sample room".
|
|
47
|
+
Select next Monday–Tuesday as full days.
|
|
48
|
+
Submit the booking.
|
|
49
|
+
Go to My bookings.
|
|
50
|
+
Verify the new booking for Sample room and the selected dates is confirmed for 2 full days.
|
|
51
|
+
Open that booking and cancel it through its menu.
|
|
52
|
+
Verify that booking no longer appears in My bookings.
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Custom-duration booking:
|
|
56
|
+
|
|
57
|
+
```text
|
|
58
|
+
Run the Loadmill "Reset sample room bookings" flow.
|
|
59
|
+
Open the booking app; if already open, fully close it and reopen it.
|
|
60
|
+
Log in as the test member, logging out first if another user is signed in.
|
|
61
|
+
Go to Home.
|
|
62
|
+
Start a room booking.
|
|
63
|
+
Choose "Sample room".
|
|
64
|
+
Select next Monday at 09:00.
|
|
65
|
+
Choose "Custom duration" (scroll right in the duration options) and set 3 hours.
|
|
66
|
+
Submit the booking.
|
|
67
|
+
Go to My bookings.
|
|
68
|
+
Verify the new booking for Sample room on the selected date at 09:00 is confirmed for 3 hours.
|
|
69
|
+
Open that booking and cancel it through its menu.
|
|
70
|
+
Verify that booking no longer appears in My bookings.
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
The final assertions identify the requested result and cleanup. Neither test checks unrelated duration options or how the calendar collapses. The short scrolling hint helps find a relevant choice without scripting the entire interaction.
|
|
74
|
+
|
|
75
|
+
If a run leaves the calendar selection unconfirmed, first clarify `Select next Monday–Tuesday as full days` to `Select next Monday–Tuesday as full days and confirm the date selection.` Avoid accumulating instructions such as “tap Apply now even if the total updated” plus assertions that the grid and button disappear. Verify that the clearer goal resolves the observed failure; retain a specific control hint only if it is still needed. Do not hide an unresolved Droid or product defect in test prose.
|
|
76
|
+
|
|
77
|
+
## Example: account preferences
|
|
20
78
|
|
|
21
|
-
|
|
79
|
+
This fictional settings app illustrates the same approach without booking dates, cleanup flows, or app restarts. The requested behavior is saving a notification preference and retaining it when settings are reopened; assume the test account initially uses Daily notifications.
|
|
22
80
|
|
|
23
81
|
```text
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
82
|
+
Log in to the settings app as the test member.
|
|
83
|
+
Open Notification settings.
|
|
84
|
+
Change the email digest frequency to Weekly.
|
|
85
|
+
Save the preferences.
|
|
86
|
+
Leave Notification settings and reopen it.
|
|
87
|
+
Verify the email digest frequency is still Weekly.
|
|
88
|
+
Restore the email digest frequency to Daily and save.
|
|
30
89
|
```
|
|
31
90
|
|
|
32
|
-
|
|
91
|
+
The persistence check protects the requested behavior. Checking a toast, button color, or menu animation would add unrelated requirements. Restore the known starting value for repeatability; do not invent cleanup when the scenario or environment does not need it.
|
|
33
92
|
|
|
34
93
|
Use the secret key names that exist in `.secrets`; do not inspect their values. If key discovery is necessary, filter the file so only keys reach output, for example `sed -n 's/^\([^#][^=]*\)=.*/\1/p' .secrets`. If a required key is absent, ask the user to add it without sharing its value.
|
|
35
94
|
|
|
@@ -10,7 +10,7 @@ One confirmation may cover that iterative workflow. Continue without asking agai
|
|
|
10
10
|
|
|
11
11
|
## 2. Establish a reproducible baseline
|
|
12
12
|
|
|
13
|
-
Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
|
|
13
|
+
Record the intended coverage using [authoring-tests.md](authoring-tests.md) before editing, including which assertions are user requirements. Run the narrowest test with its existing project model and config. Keep these fixed for the baseline, then change any of them only as part of a documented, evidence-supported optimization batch:
|
|
14
14
|
|
|
15
15
|
- App build and initial account/data state.
|
|
16
16
|
- Device, OS, device source, or browser session mode.
|
|
@@ -49,18 +49,18 @@ Stop healing when evidence indicates an application regression. Preserve the fai
|
|
|
49
49
|
For long-running tests, fix all well-supported bottlenecks that can be addressed safely before paying for another full run. Use this order to build the batch unless evidence points elsewhere:
|
|
50
50
|
|
|
51
51
|
1. **`.dcua` wording and scope**
|
|
52
|
-
- Express one
|
|
52
|
+
- Express one readable micro-goal per instruction. Choosing a room, selecting dates, and submitting a booking are useful separate goals; opening each menu or tapping each control usually is not.
|
|
53
53
|
- Collapse consecutive micro-steps into the user outcome they jointly accomplish, such as `Select the desired date` or `Log out`, while keeping meaningful assertions explicit.
|
|
54
54
|
- Use exact visible labels only when they disambiguate the goal or identify an outcome the test must verify.
|
|
55
|
-
-
|
|
55
|
+
- Resolve an ambiguous transition by checking the starting state, data, and wording first. Add an assertion only when it protects intended coverage; observing progress does not require a new test assertion.
|
|
56
56
|
- Remove scenario steps that do not protect the behavior under test.
|
|
57
|
-
-
|
|
57
|
+
- Rewrite the unclear goal before adding lower-level actions. Add a short hint only when the evidence shows it is still necessary or an intermediate state is part of the coverage; do not append emphatic commands and UI-state checks from each failed run.
|
|
58
58
|
- Keep the existing scenario in one independently runnable `.dcua` file. Splitting a long test into ordered or state-dependent tests is not an available optimization because separate execution order and shared state are not guaranteed.
|
|
59
|
-
-
|
|
59
|
+
- Preserve required assertions and scenario choices. Remove agent-added, out-of-scope or redundant assertions only with the coverage-based explanation required by the authoring guide; preserve checks whose purpose is unresolved.
|
|
60
60
|
|
|
61
61
|
2. **`context.md`**
|
|
62
62
|
- Update the project's existing canonical context file; never create a test-specific context file for the optimization.
|
|
63
|
-
- Add durable product navigation, terminology, stable interruptions
|
|
63
|
+
- Add durable product navigation, terminology, and stable interruptions. Do not move accumulated scenario recovery instructions into shared context to make the test appear simpler.
|
|
64
64
|
- Move reusable mechanics out of the test, but keep scenario-specific values and expectations in the test.
|
|
65
65
|
- Remove redundant, speculative, or conflicting instructions. Do not restate Droid's base behavior or encode a workaround for a Droid/product bug as product knowledge.
|
|
66
66
|
|
|
@@ -114,11 +114,14 @@ For every included change, record the evidence, rationale, expected effect, and
|
|
|
114
114
|
|
|
115
115
|
## 6. Compare and iterate
|
|
116
116
|
|
|
117
|
+
After each edit, reread the whole test against its intended coverage. Remove superseded hints and redundant checks, keep one micro-goal per instruction, and ensure setup or data problems have not become speculative recovery branches. Keep diagnostic history in the report, not the test or shared context. Validate the simplified version in the next representative run.
|
|
118
|
+
|
|
117
119
|
After each rerun, compare:
|
|
118
120
|
|
|
119
121
|
- Did the first divergence disappear for the expected reason?
|
|
120
122
|
- Did the hotspot's duration, rounds, actions, previews, retries, or recoveries improve?
|
|
121
123
|
- Did another step regress or did coverage change?
|
|
124
|
+
- Did the test become harder to read or more dependent on incidental UI details? Explain any retained complexity by coverage or execution evidence; fewer lines alone are not an improvement.
|
|
122
125
|
- Did the combined change set improve the intended hotspots without obscuring a regression?
|
|
123
126
|
|
|
124
127
|
Keep the batch when the evidence supports the overall result. If it regresses or remains ambiguous, use the timeline to isolate, revise, or revert only the suspect subset instead of automatically rerunning every change separately. Continue while the evidence supports another useful test, context, data, or configuration improvement. Stop and report the strongest remaining explanation only when a genuine blocker remains or no evidence-supported improvement is left to try.
|
|
@@ -129,6 +132,7 @@ After the optimized test passes, decide whether the accumulated evidence is stro
|
|
|
129
132
|
|
|
130
133
|
Call the test ship-ready only when:
|
|
131
134
|
|
|
135
|
+
- The final test still expresses the intended coverage as readable micro-goals, with no unnecessary assertions, stale debugging instructions, or speculative recovery branches.
|
|
132
136
|
- All intended instructions and meaningful assertions pass in the final representative evidence.
|
|
133
137
|
- No unexplained recovery, repeated-action loop, or recurring high-risk hotspot remains.
|
|
134
138
|
- Timing and action counts are reasonably consistent, with transient provider retries identified separately.
|
|
@@ -4,6 +4,12 @@ Follow the Desktop Device Setup Wizard sequence: choose a platform, run checks,
|
|
|
4
4
|
|
|
5
5
|
Do not create or modify user test files merely to perform setup. Get approval before installing software, changing system configuration, running `sudo`, or accepting a license. After remediation, rerun only the failed check.
|
|
6
6
|
|
|
7
|
+
## Identify the installed skill
|
|
8
|
+
|
|
9
|
+
When diagnosing skill behavior, report the installed skill version and bundled skill version from `droid-cua skill status <agent>` (`cursor`, `claude-code`, or `codex`). Use `--local` for a project installation. The skill's own version is `metadata.version` in `SKILL.md`; it is independent of the CLI package version. If an older CLI does not display versions, read this field from the active skill copy instead.
|
|
10
|
+
|
|
11
|
+
`unknown` means version metadata is unavailable, as with older copies. `modified` means the installed files differ from the bundled skill; it can indicate an older release or local edits, including edits that left the version unchanged. Report that state alongside the version. Updating the CLI does not update existing skill copies; explain any mismatch before using `droid-cua skill install <agent> --force`, which overwrites local changes.
|
|
12
|
+
|
|
7
13
|
## Droid CUA CLI compatibility
|
|
8
14
|
|
|
9
15
|
If the CLI rejects a model, option, command, or capability that this skill explicitly documents, treat an outdated global `@loadmill/droid-cua` package as a likely setup issue. Do not suggest an update for an ordinary test, target, provider, or application failure.
|
|
@@ -40,4 +40,4 @@ Offer exactly these two choices, then wait for the user's selection:
|
|
|
40
40
|
|
|
41
41
|
Use a secret-safe mechanism that does not print or expose the value in commands, logs, or responses. Verify only that the key is present. Never copy a token from another file, a shell profile, another project, an agent skill/configuration, or any other location. Do not inspect inherited environment variables to discover a token. When the active project's `.env` contains the key, run `droid-cua` normally from that project; the CLI loads that file automatically.
|
|
42
42
|
|
|
43
|
-
Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with
|
|
43
|
+
Only when the user explicitly selects direct OpenAI access, use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Saved direct GPT-5.6 Terra and GPT-5.4 selections migrate to GPT-6 Sol; saved GPT-5.6 Luna selections migrate to GPT-6 Luna. Check only the active project's `.env` for `OPENAI_API_KEY`; do not use one found elsewhere. If it is missing, guide the user to create and configure their own key in the same project using the same secret-safe process. Do not silently switch providers to bypass missing Loadmill authentication.
|
|
@@ -16,12 +16,16 @@ Preserve an existing provider/model choice in `ci/droid-cua.json`; otherwise use
|
|
|
16
16
|
--llm-provider loadmill --cua-model loadmill-smart
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with
|
|
19
|
+
All Loadmill models require `LOADMILL_API_TOKEN`. Smart and Pulse now route through the Loadmill gateway to GPT-6 Sol and GPT-6 Luna. Do not switch from Pulse to Smart merely because a test failed or was slow; first identify a model-related stability issue, then keep Smart only when comparison evidence shows an improvement. Do not switch between Smart, Pulse, and Beacon silently; preserve the project's model or explain the capability, speed, and experimental tradeoffs and follow the user's choice. If the user chooses direct OpenAI access instead, require `OPENAI_API_KEY` and use `--llm-provider openai` with `gpt-6-sol` or `gpt-6-luna`. Do not mix a provider with another provider's model, silently switch providers, or override an existing project choice without explaining it.
|
|
20
20
|
|
|
21
21
|
Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
|
|
22
22
|
|
|
23
|
+
For macOS and Windows desktop tests, the desktop app shows a system notice for three seconds before Droid takes control. The three-second delay is fixed; once the test starts, use Stop in Droid to end it. Avoid using the mouse or keyboard during the run. Droid brings the selected target window to the foreground before the first capture. When the run ends, the target stays in front and a second system notice says Droid has finished controlling the desktop. Open Droid to review the result. Some Windows apps return an all-black window capture; Droid brings the target back to the foreground before cropping the visible desktop to its visible frame, excluding the invisible Windows resize border. The fallback logs an all-black capture warning and crops before resizing to preserve detail. A visible-frame lookup warning means Droid is using the outer window bounds, which may include invisible resize borders. Keep the target visible and unobstructed on the primary display. In accessibility input mode, bring the target to the foreground yourself if the fallback requests it; Droid does not focus it for screenshots. If a screenshot shows another app or black off-screen edges, check whether the target could be focused and fully displayed before changing the `.dcua` instructions.
|
|
24
|
+
|
|
23
25
|
When downstream automation needs an exact Loadmill handoff, add `--report-metadata reports/<test-name>.json`. Read it only after Droid exits. Droid clears the destination at run start and writes the saved run ID plus the last uploaded screenshot object name only after cloud report publication succeeds, including when a failed test's report was published. Do not infer the run or screenshot by searching recent Loadmill runs when this file was requested but absent.
|
|
24
26
|
|
|
27
|
+
For desktop runs, treat the test result and dashboard report as separate outcomes. A run can appear in Droid Runs while its report is still uploading. A `report.upload.skipped` entry in the app's `desktop-events.jsonl` explains when sharing was disabled or authentication was unavailable. For failures, use the run warning and `report.upload.failed` entry, which includes the upload stage and HTTP status when available. The absence of a failure entry does not prove that a still-unavailable report was published; it may still be uploading. Direct the user to the local report in Droid when sharing did not complete. Do not infer the cause from an unclickable dashboard row alone.
|
|
28
|
+
|
|
25
29
|
```sh
|
|
26
30
|
# Attached Android device or running emulator
|
|
27
31
|
droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
|
|
@@ -42,7 +46,7 @@ droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pu
|
|
|
42
46
|
droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --device-source aws-device-farm --platform android --device-name "<name>" --os-version "<version>" --app <app.apk> --debug --report reports/example.html
|
|
43
47
|
|
|
44
48
|
# Direct OpenAI alternative
|
|
45
|
-
droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-
|
|
49
|
+
droid-cua run tests/example.dcua --llm-provider openai --cua-model gpt-6-sol --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
|
|
46
50
|
|
|
47
51
|
# Selected tests from different folders
|
|
48
52
|
droid-cua run tests/auth/login.dcua tests/checkout/purchase.dcua admin/refund.dcua --avd adb:<serial> --config ci/droid-cua.json --debug --report reports/selected-tests.html
|
|
@@ -133,6 +137,6 @@ Keep this retrospective concise and lead with the result and improvement. Do not
|
|
|
133
137
|
|
|
134
138
|
## Guarded healing
|
|
135
139
|
|
|
136
|
-
Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete coverage, loosen the expected outcome, or add broad waits simply to obtain green output.
|
|
140
|
+
Before editing, state the classification and evidence. Change only `.dcua`, `context.md`, `test-data.md`, or Droid CUA config. Never edit app code, delete required coverage, loosen the expected outcome, or add broad waits simply to obtain green output. Use the coverage rules in [authoring-tests.md](authoring-tests.md) to distinguish required assertions from agent-added, out-of-scope or redundant checks; retain assertions whose purpose is unclear. A failing business outcome remains a failure.
|
|
137
141
|
|
|
138
|
-
Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.
|
|
142
|
+
Simplify unclear goals and check setup/data before adding UI detail. Replace obsolete guidance rather than accumulating commands from each failed run, and review the whole test for readability after editing. Apply the smallest evidence-supported batch, explain why each change corrects the test rather than hides a product defect, and rerun only the affected test. Continue analyzing, editing, and rerunning until the final representative evidence supports readiness or a genuine blocker remains. Do not stop based on a fixed rerun count. Stop when the evidence indicates an application regression, an environment or product dependency prevents progress, or no evidence-supported test, context, data, or configuration improvement remains; report the original and final evidence, changes attempted, and recommended owner/action.
|