@loadmill/droid-cua 2.37.0 → 2.39.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -1
- package/build/index.js +10 -0
- package/build/src/cli/headless-runner.js +34 -4
- package/build/src/cli/loadmill-report-metadata.js +7 -0
- package/build/src/commands/help.js +1 -0
- package/build/src/core/report/render-html-report.js +27 -1
- package/build/src/device/android/connection.js +13 -4
- package/build/src/device/cloud/actions.js +10 -1
- package/build/src/device/cloud/cloud-appium-client.js +6 -0
- package/build/src/device/interface.js +1 -1
- package/build/src/device/ios/actions.js +10 -1
- package/build/src/device/ios/appium-client.js +6 -0
- package/build/src/device/mobile-keypress.js +13 -0
- package/build/src/device/openai.js +2 -2
- package/build/src/integrations/loadmill/report-upload-manager.js +8 -1
- package/build/src/integrations/loadmill/run-reporter.js +5 -1
- package/build/src/prompts/base.js +1 -1
- package/build/src/prompts/execution.js +1 -1
- package/package.json +1 -1
- package/skill/droid-cua/references/running-and-healing.md +3 -0
package/README.md
CHANGED
|
@@ -290,6 +290,7 @@ Supported CLI options include:
|
|
|
290
290
|
- `--secrets`
|
|
291
291
|
- `--record`
|
|
292
292
|
- `--report`
|
|
293
|
+
- `--report-metadata <path>`
|
|
293
294
|
- `--no-cloud-report`
|
|
294
295
|
- `--debug`
|
|
295
296
|
|
|
@@ -325,8 +326,11 @@ HTML report (`--report`):
|
|
|
325
326
|
- With no value it saves `<test>--<date>--<time>.html` (e.g. `login--2026-06-03--16-09-20.html`) in the current directory; pass `--report <file>` to pick a filename or `--report <dir>` to pick a folder.
|
|
326
327
|
- A multi-path selection writes the individual test reports plus one aggregate `Selected tests` report with a child result and timeline for every selected test. When `--report <file.html>` is used, that exact path is used for the aggregate report.
|
|
327
328
|
- In the desktop app the same report is available from a run's **Share → Download HTML report** button.
|
|
328
|
-
-
|
|
329
|
+
- Generated HTML reports opt out of Cloudflare email obfuscation so ordinary email values remain visible in hosted reports. Secret-typing values are omitted from typing logs, but may still appear in screenshots.
|
|
330
|
+
- Loadmill flow entries link **Test started** to the flow's test-suite run results, including in reports generated by CI.
|
|
331
|
+
- Runs with captured screenshots also expose **Share → Generate video**. The desktop app renders portrait mobile runs as square replays and landscape mobile, web, and desktop runs as 16:9 replays. Landscape Android and iOS screenshots keep a device frame sized to the captured screen instead of browser chrome. Rendering happens locally, then the app prompts for a save location using `<test>--<date>--<time>--test-replay.mp4` as the suggested filename. While rendering, a compact **Background tasks** card shows progress for **Generating test replay**; it can be minimized to a task-count card or canceled without opening the save dialog.
|
|
329
332
|
- Authenticated headless runs upload their HTML report and screenshots to Loadmill by default. The Loadmill run record also includes the resolved Droid version, model, provider, Agent Pace, Aim Assist, Context Optimization, and target environment. Use `--no-cloud-report` to disable artifact upload; the Droid run record is still reported.
|
|
333
|
+
- Use `--report-metadata <path>` when automation needs the exact published Loadmill run ID and last uploaded screenshot object name. Droid clears that destination when the invocation starts and writes it only after report publication succeeds, including for published failure reports.
|
|
330
334
|
|
|
331
335
|
```sh
|
|
332
336
|
droid-cua run tests/login.dcua --avd adb:emulator-5554 --report ./reports
|
package/build/index.js
CHANGED
|
@@ -22,6 +22,7 @@ import { resolveHeadlessExecutionConfig } from "./src/cli/headless-execution-con
|
|
|
22
22
|
import { artifactKindsForPool, resolveHeadlessTarget, scopeArtifactKindsToPool, shouldWaitForLoadmillCloudCompletion, } from "./src/cli/headless-target.js";
|
|
23
23
|
import { createHeadlessDebugArtifacts } from "./src/cli/headless-debug.js";
|
|
24
24
|
import { loadHeadlessInstructions } from "./src/cli/headless-instructions.js";
|
|
25
|
+
import { clearLoadmillReportMetadata } from "./src/cli/loadmill-report-metadata.js";
|
|
25
26
|
import { assertRunCommandPosition, getHeadlessInstructionSelectors, resolveHeadlessInstructionSelection, } from "./src/cli/headless-selection.js";
|
|
26
27
|
import { runHeadlessInstructionRuns } from "./src/cli/headless-runner.js";
|
|
27
28
|
import { buildLoadmillCloudDeviceConnectSummaryLines } from "./src/cli/loadmill-cloud-device-connect-summary.js";
|
|
@@ -72,6 +73,11 @@ if (args._[0] === "skill") {
|
|
|
72
73
|
process.exit(1);
|
|
73
74
|
}
|
|
74
75
|
}
|
|
76
|
+
const reportMetadataPath = typeof args["report-metadata"] === "string"
|
|
77
|
+
&& args["report-metadata"].trim().length > 0
|
|
78
|
+
? args["report-metadata"].trim()
|
|
79
|
+
: null;
|
|
80
|
+
await clearLoadmillReportMetadata(reportMetadataPath);
|
|
75
81
|
const targetConfig = resolveHeadlessTarget(args);
|
|
76
82
|
let avdName = targetConfig.avdName;
|
|
77
83
|
let platform = targetConfig.platform || null; // 'ios' or 'android'
|
|
@@ -98,6 +104,10 @@ const reportConfig = {
|
|
|
98
104
|
enabled: args.report === true || (typeof args.report === "string" && args.report.trim().length > 0),
|
|
99
105
|
path: typeof args.report === "string" && args.report.trim().length > 0 ? args.report.trim() : null,
|
|
100
106
|
cloudEnabled: args["no-cloud-report"] !== true,
|
|
107
|
+
metadataPath: reportMetadataPath,
|
|
108
|
+
lloydJobId: typeof process.env.LLOYD_JOB_ID === "string" && process.env.LLOYD_JOB_ID.trim().length > 0
|
|
109
|
+
? process.env.LLOYD_JOB_ID.trim()
|
|
110
|
+
: null,
|
|
101
111
|
};
|
|
102
112
|
const screenshotDir = path.join("droid-cua-recording-" + Date.now());
|
|
103
113
|
if (recordScreenshots)
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import path from "path";
|
|
2
|
-
import { mkdir } from "fs/promises";
|
|
2
|
+
import { mkdir, writeFile } from "fs/promises";
|
|
3
3
|
import { getCurrentPlatform as getBrowserCurrentPlatform, getScreenshotAsBase64 as getBrowserScreenshotAsBase64, } from "../browser/connection.js";
|
|
4
4
|
import { handleModelAction as handleBrowserModelAction } from "../browser/actions.js";
|
|
5
5
|
import { ExecutionEngine } from "../core/execution-engine.js";
|
|
@@ -13,6 +13,7 @@ import { reportDroidCuaCiTestRunCompleted } from "../integrations/segment/droid-
|
|
|
13
13
|
import { printCliOutput } from "../utils/console-output.js";
|
|
14
14
|
import { emitDesktopDebug } from "../utils/desktop-debug.js";
|
|
15
15
|
import { loadHeadlessInstructions } from "./headless-instructions.js";
|
|
16
|
+
import { clearLoadmillReportMetadata } from "./loadmill-report-metadata.js";
|
|
16
17
|
import { formatHeadlessTestSourceLabel } from "./headless-test-source-label.js";
|
|
17
18
|
import { buildCliRunReportModel, createCapturingAddOutput, createCliAggregateRunReport, createCliRunReport, renderCliCloudReport, writeCliRunReportArtifact, } from "./headless-report.js";
|
|
18
19
|
import { createRefuseReconnectConnectToBrowser } from "../desktop-shared/connection-predicates.js";
|
|
@@ -36,6 +37,24 @@ function logPendingScreenshotUploads(uploadManager, uploadSession, subject) {
|
|
|
36
37
|
console.log(`${subject} finished. Waiting for ${pendingCount} screenshot ${uploads} to finish...`);
|
|
37
38
|
}
|
|
38
39
|
}
|
|
40
|
+
function loadmillReportMetadata(savedRunId, publication) {
|
|
41
|
+
if (!savedRunId || !publication?.published) {
|
|
42
|
+
return null;
|
|
43
|
+
}
|
|
44
|
+
return {
|
|
45
|
+
runId: savedRunId,
|
|
46
|
+
...(publication.lastScreenshotObjectName
|
|
47
|
+
? { lastScreenshotObjectName: publication.lastScreenshotObjectName }
|
|
48
|
+
: {}),
|
|
49
|
+
};
|
|
50
|
+
}
|
|
51
|
+
async function writeLoadmillReportMetadata(filePath, metadata) {
|
|
52
|
+
if (!filePath || !metadata) {
|
|
53
|
+
return;
|
|
54
|
+
}
|
|
55
|
+
await mkdir(path.dirname(path.resolve(filePath)), { recursive: true });
|
|
56
|
+
await writeFile(filePath, `${JSON.stringify(metadata, null, 2)}\n`, { mode: 0o600 });
|
|
57
|
+
}
|
|
39
58
|
function getRecordScreenshotDirForTest(baseScreenshotDir, testName, isBatchRun) {
|
|
40
59
|
if (!baseScreenshotDir || !isBatchRun) {
|
|
41
60
|
return baseScreenshotDir;
|
|
@@ -320,9 +339,10 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
|
|
|
320
339
|
const cloudModel = cloudReportEnabled
|
|
321
340
|
? buildCliRunReportModel({ ...reportInput, events: cloudEvents })
|
|
322
341
|
: null;
|
|
342
|
+
let reportPublication = null;
|
|
323
343
|
if (cloudModel && reportToLoadmill && reportedRun?.savedRunId) {
|
|
324
344
|
logPendingScreenshotUploads(uploadManager, cloudUploadSession, "Test");
|
|
325
|
-
await uploadManager.finish(cloudUploadSession, {
|
|
345
|
+
reportPublication = await uploadManager.finish(cloudUploadSession, {
|
|
326
346
|
savedRunId: reportedRun.savedRunId,
|
|
327
347
|
html: renderCliCloudReport(cloudModel, reportedRun.savedRunId),
|
|
328
348
|
});
|
|
@@ -339,6 +359,7 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
|
|
|
339
359
|
retries: stats.retryCount ?? 0,
|
|
340
360
|
localReportModel: localReport?.model ?? null,
|
|
341
361
|
reportModel: cloudModel,
|
|
362
|
+
loadmillReport: loadmillReportMetadata(reportedRun?.savedRunId, reportPublication),
|
|
342
363
|
};
|
|
343
364
|
}
|
|
344
365
|
catch (error) {
|
|
@@ -418,9 +439,10 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
|
|
|
418
439
|
const cloudModel = cloudReportEnabled
|
|
419
440
|
? buildCliRunReportModel({ ...failedReportInput, events: cloudEvents })
|
|
420
441
|
: null;
|
|
442
|
+
let reportPublication = null;
|
|
421
443
|
if (cloudModel && reportToLoadmill && reportedRun?.savedRunId) {
|
|
422
444
|
logPendingScreenshotUploads(uploadManager, cloudUploadSession, "Test");
|
|
423
|
-
await uploadManager.finish(cloudUploadSession, {
|
|
445
|
+
reportPublication = await uploadManager.finish(cloudUploadSession, {
|
|
424
446
|
savedRunId: reportedRun.savedRunId,
|
|
425
447
|
html: renderCliCloudReport(cloudModel, reportedRun.savedRunId),
|
|
426
448
|
});
|
|
@@ -439,17 +461,21 @@ async function runHeadlessTest({ runId, testPath, testName, instructions, sessio
|
|
|
439
461
|
retries: stats.retryCount ?? 0,
|
|
440
462
|
localReportModel: localReport?.model ?? null,
|
|
441
463
|
reportModel: cloudModel,
|
|
464
|
+
loadmillReport: loadmillReportMetadata(reportedRun?.savedRunId, reportPublication),
|
|
442
465
|
};
|
|
443
466
|
}
|
|
467
|
+
await writeLoadmillReportMetadata(report?.metadataPath, loadmillReportMetadata(reportedRun?.savedRunId, reportPublication));
|
|
444
468
|
throw error;
|
|
445
469
|
}
|
|
446
470
|
}
|
|
447
471
|
export async function runHeadlessInstructionRuns({ runs, instructionsTarget, session, deviceInfo, headlessConfig, appContext, runtimeContext, headlessDebug, secrets = null, recordScreenshots, screenshotDir, isBrowserTarget, isDesktopTarget = false, desktopController = null, isCloudTarget, runTarget = null, report = null, droidVersion = getDroidVersion(), runHeadlessTestImpl = runHeadlessTest, uploadManager = droidReportUploadManager, }) {
|
|
472
|
+
await clearLoadmillReportMetadata(report?.metadataPath);
|
|
448
473
|
const resolvedRunTarget = runTarget ?? { isBrowserTarget, isDesktopTarget, isCloudTarget };
|
|
449
474
|
const runConfiguration = buildDroidRunConfiguration({
|
|
450
475
|
droidVersion,
|
|
451
476
|
runtimeConfig: runtimeContext.config,
|
|
452
477
|
aimAssistMode: headlessConfig.aimAssistMode,
|
|
478
|
+
lloydJobId: report?.lloydJobId,
|
|
453
479
|
});
|
|
454
480
|
const isBatchRun = ["directory", "selection", "scenario"].includes(instructionsTarget?.type);
|
|
455
481
|
const cloudReportEnabled = report?.cloudEnabled !== false &&
|
|
@@ -470,6 +496,7 @@ export async function runHeadlessInstructionRuns({ runs, instructionsTarget, ses
|
|
|
470
496
|
console.log(`\nRunning ${runs.length} tests from: ${instructionsTarget.path}\n`);
|
|
471
497
|
}
|
|
472
498
|
let failures = 0;
|
|
499
|
+
let reportMetadata = null;
|
|
473
500
|
const batchStartedAt = Date.now();
|
|
474
501
|
const aggregateStats = {
|
|
475
502
|
durationMs: 0,
|
|
@@ -530,6 +557,7 @@ export async function runHeadlessInstructionRuns({ runs, instructionsTarget, ses
|
|
|
530
557
|
if (result.reportModel) {
|
|
531
558
|
aggregateReportModels.push(result.reportModel);
|
|
532
559
|
}
|
|
560
|
+
reportMetadata = result.loadmillReport ?? reportMetadata;
|
|
533
561
|
if (result.localReportModel) {
|
|
534
562
|
aggregateLocalReportModels.push(result.localReportModel);
|
|
535
563
|
}
|
|
@@ -642,12 +670,14 @@ export async function runHeadlessInstructionRuns({ runs, instructionsTarget, ses
|
|
|
642
670
|
childModels: aggregateReportModels,
|
|
643
671
|
});
|
|
644
672
|
logPendingScreenshotUploads(uploadManager, cloudUploadSession, "Run");
|
|
645
|
-
await uploadManager.finish(cloudUploadSession, {
|
|
673
|
+
const publication = await uploadManager.finish(cloudUploadSession, {
|
|
646
674
|
savedRunId: reportedRun.savedRunId,
|
|
647
675
|
html: renderCliCloudReport(aggregateReport.model, reportedRun.savedRunId),
|
|
648
676
|
});
|
|
677
|
+
reportMetadata = loadmillReportMetadata(reportedRun.savedRunId, publication);
|
|
649
678
|
}
|
|
650
679
|
}
|
|
680
|
+
await writeLoadmillReportMetadata(report?.metadataPath, reportMetadata);
|
|
651
681
|
return {
|
|
652
682
|
exitCode: failures === 0 ? 0 : 1,
|
|
653
683
|
failures
|
|
@@ -54,6 +54,7 @@ export async function handleHelp(args, session, context) {
|
|
|
54
54
|
addOutput({ type: 'info', text: ' --explain Explain non-obvious decisions and completion evidence' });
|
|
55
55
|
addOutput({ type: 'info', text: ' --report [path] Save a self-contained HTML report locally' });
|
|
56
56
|
addOutput({ type: 'info', text: ' --no-cloud-report Skip uploading the HTML report and screenshots to Loadmill' });
|
|
57
|
+
addOutput({ type: 'info', text: ' --report-metadata <path> Write uploaded Loadmill report metadata as JSON' });
|
|
57
58
|
addOutput({ type: 'info', text: ' --debug Enable structured JSONL debug artifacts' });
|
|
58
59
|
addOutput({ type: 'info', text: '' });
|
|
59
60
|
addOutput({ type: 'info', text: 'Interactive commands:' });
|
|
@@ -255,12 +255,36 @@ function renderCloudCard(events) {
|
|
|
255
255
|
}
|
|
256
256
|
function renderLoadmillCard(events) {
|
|
257
257
|
const lines = events.map((event) => event.text.replace(/^\[Loadmill\]\s*/, "").trim());
|
|
258
|
+
const runId = events.map((event) => event.payload?.loadmillSuiteRunId)
|
|
259
|
+
.find((value) => typeof value === "string" && value.trim()) ??
|
|
260
|
+
lines.map((line) => line.match(/Test started \(ID:\s*([^)]+)\)/i)?.[1]?.trim() ||
|
|
261
|
+
line.match(/Run ID:\s*(.+)$/i)?.[1]?.trim()).find(Boolean);
|
|
262
|
+
const baseUrl = events.map((event) => event.payload?.loadmillBaseUrl)
|
|
263
|
+
.find((value) => typeof value === "string" && value.trim()) ?? "https://app.loadmill.com";
|
|
264
|
+
let runUrl = null;
|
|
265
|
+
if (runId) {
|
|
266
|
+
try {
|
|
267
|
+
const url = new URL(`${baseUrl.trim().replace(/\/$/, "")}/app/api-tests/test-suite-runs/${encodeURIComponent(runId.trim())}`);
|
|
268
|
+
if (url.protocol === "https:" || url.protocol === "http:")
|
|
269
|
+
runUrl = url.href;
|
|
270
|
+
}
|
|
271
|
+
catch {
|
|
272
|
+
// Keep the original log text when the recorded base URL is invalid.
|
|
273
|
+
}
|
|
274
|
+
}
|
|
275
|
+
const renderedLines = lines.map((line) => {
|
|
276
|
+
const started = line.match(/^Test started \(ID:\s*([^)]+)\)\.(.*)$/i);
|
|
277
|
+
const text = started && runUrl
|
|
278
|
+
? `<a href="${escapeHtml(runUrl)}" target="_blank" rel="noreferrer">Test started</a>${renderLinkedText(started[2])}`
|
|
279
|
+
: renderLinkedText(line);
|
|
280
|
+
return `<div>${text}</div>`;
|
|
281
|
+
});
|
|
258
282
|
const failed = events.some((event) => /\[Loadmill\]\s*FAILED|^Error:/i.test(event.text));
|
|
259
283
|
const passed = !failed && events.some((event) => /\bpassed\b/i.test(event.text));
|
|
260
284
|
const status = failed ? "failed" : passed ? "passed" : "running";
|
|
261
285
|
return `<details class="dc-special dc-loadmill"${passed ? "" : " open"}>
|
|
262
286
|
<summary><span class="dc-special-title">${ICONS.loadmill}Loadmill Flow</span><span class="dc-pill dc-pill-${status}">${status}</span></summary>
|
|
263
|
-
<div class="dc-special-body">${
|
|
287
|
+
<div class="dc-special-body">${renderedLines.join("")}</div>
|
|
264
288
|
</details>`;
|
|
265
289
|
}
|
|
266
290
|
function renderTimeline(events, shots, warnings = []) {
|
|
@@ -464,10 +488,12 @@ ${STYLES}
|
|
|
464
488
|
</style>
|
|
465
489
|
</head>
|
|
466
490
|
<body>
|
|
491
|
+
<!--email_off-->
|
|
467
492
|
<div class="dc-wrap" id="dc-top">
|
|
468
493
|
<div class="dc-header"><div class="dc-title">Run Details: ${escapeHtml(heading)}</div><span class="dc-pill dc-pill-${escapeHtml(model.status)}">${escapeHtml(runStatusLabel(model.status))}</span></div>
|
|
469
494
|
${body}
|
|
470
495
|
</div>${lightboxes ? `\n${lightboxes}` : ""}${script ? `\n${script}` : ""}
|
|
496
|
+
<!--/email_off-->
|
|
471
497
|
</body>
|
|
472
498
|
</html>`;
|
|
473
499
|
}
|
|
@@ -5,6 +5,18 @@ import { emitDesktopDebug } from "../../utils/desktop-debug.js";
|
|
|
5
5
|
import { execAdb, execEmulator, spawnAdb, spawnEmulator } from "./tools.js";
|
|
6
6
|
const AVD_READY_TIMEOUT_MS = 180000;
|
|
7
7
|
const AVD_READY_TIMEOUT_SECONDS = AVD_READY_TIMEOUT_MS / 1000;
|
|
8
|
+
const TARGET_SCALED_SHORT_EDGE = 400;
|
|
9
|
+
export function calculateScaledScreenDimensions(width, height) {
|
|
10
|
+
const shortestEdge = Math.min(width, height);
|
|
11
|
+
const scale = shortestEdge > TARGET_SCALED_SHORT_EDGE
|
|
12
|
+
? TARGET_SCALED_SHORT_EDGE / shortestEdge
|
|
13
|
+
: 1.0;
|
|
14
|
+
return {
|
|
15
|
+
scaledWidth: Math.round(width * scale),
|
|
16
|
+
scaledHeight: Math.round(height * scale),
|
|
17
|
+
scale,
|
|
18
|
+
};
|
|
19
|
+
}
|
|
8
20
|
async function listConnectedDevices() {
|
|
9
21
|
const { stdout } = await execAdb(["devices"]);
|
|
10
22
|
return stdout
|
|
@@ -238,10 +250,7 @@ export async function getDeviceInfo(deviceId) {
|
|
|
238
250
|
process.exit(1);
|
|
239
251
|
}
|
|
240
252
|
const [_, width, height] = match.map(Number);
|
|
241
|
-
const
|
|
242
|
-
const scale = width > targetWidth ? targetWidth / width : 1.0;
|
|
243
|
-
const scaledWidth = Math.round(width * scale);
|
|
244
|
-
const scaledHeight = Math.round(height * scale);
|
|
253
|
+
const { scaledWidth, scaledHeight, scale } = calculateScaledScreenDimensions(width, height);
|
|
245
254
|
const [{ stdout: manufacturerRaw }, { stdout: modelRaw }, { stdout: deviceNameRaw }] = await Promise.all([
|
|
246
255
|
execAdb(["-s", deviceId, "shell", "getprop", "ro.product.manufacturer"]),
|
|
247
256
|
execAdb(["-s", deviceId, "shell", "getprop", "ro.product.model"]),
|
|
@@ -3,7 +3,7 @@ import { emitDesktopDebug, truncateForDebug } from "../../utils/desktop-debug.js
|
|
|
3
3
|
import { getConfiguredStepDelayMs } from "../../utils/step-delay.js";
|
|
4
4
|
import { buildActionEventMeta } from "../action-event-meta.js";
|
|
5
5
|
import { actionForErrorLog, buildActionErrorOutput, buildDebugTextField, buildSensitiveDebugFields, buildTypeActionPayload, buildTypeActionText, isSensitiveAction, } from "../sensitive-action.js";
|
|
6
|
-
import { buildMobileKeypressPayload, getCloudAndroidTransport, getIosTransport, normalizeMobileKeypress } from "../mobile-keypress.js";
|
|
6
|
+
import { buildIosAppSwitcherGesture, buildMobileKeypressPayload, getCloudAndroidTransport, getIosTransport, normalizeMobileKeypress } from "../mobile-keypress.js";
|
|
7
7
|
import { getActiveSession, getDevicePixelRatio } from "./connection.js";
|
|
8
8
|
import { resolveScrollGesture } from "../scroll-gesture.js";
|
|
9
9
|
import { closeAppiumKeyboard, closeAppiumKeyboardWithTap } from "../keyboard.js";
|
|
@@ -149,6 +149,15 @@ export async function handleModelAction(deviceId, action, scale = 1.0, context =
|
|
|
149
149
|
});
|
|
150
150
|
await session.client.pressIosButton(session.sessionId, transport.transportValue);
|
|
151
151
|
}
|
|
152
|
+
else if (transport.transportType === "ios-gesture") {
|
|
153
|
+
const gesture = buildIosAppSwitcherGesture(await session.client.getWindowRect(session.sessionId));
|
|
154
|
+
addOutput({
|
|
155
|
+
type: "action",
|
|
156
|
+
text: normalized.logText,
|
|
157
|
+
...meta(buildMobileKeypressPayload(normalized, transport, { gesture }))
|
|
158
|
+
});
|
|
159
|
+
await session.client.dragFromToForDuration(session.sessionId, gesture);
|
|
160
|
+
}
|
|
152
161
|
else {
|
|
153
162
|
addOutput({
|
|
154
163
|
type: "action",
|
|
@@ -175,6 +175,12 @@ export class CloudAppiumClient {
|
|
|
175
175
|
]
|
|
176
176
|
});
|
|
177
177
|
}
|
|
178
|
+
async dragFromToForDuration(sessionId, gesture) {
|
|
179
|
+
await this.request("POST", `/session/${sessionId}/execute/sync`, {
|
|
180
|
+
script: "mobile: dragFromToForDuration",
|
|
181
|
+
args: [gesture]
|
|
182
|
+
});
|
|
183
|
+
}
|
|
178
184
|
async pressIosButton(sessionId, buttonName) {
|
|
179
185
|
await this.request("POST", `/session/${sessionId}/execute/sync`, {
|
|
180
186
|
script: "mobile: pressButton",
|
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
*
|
|
32
32
|
* getDeviceInfo(deviceId: string): Promise<DeviceInfo>
|
|
33
33
|
* - Gets screen dimensions and calculates scale factor
|
|
34
|
-
* -
|
|
34
|
+
* - Local Android screenshots target a 400px shortest edge
|
|
35
35
|
*
|
|
36
36
|
* getScreenshotAsBase64(deviceId: string, deviceInfo: DeviceInfo): Promise<string>
|
|
37
37
|
* - Captures current screen state
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
*/
|
|
6
6
|
import * as appium from "./appium-client.js";
|
|
7
7
|
import { buildActionEventMeta } from "../action-event-meta.js";
|
|
8
|
-
import { buildMobileKeypressPayload, getIosTransport, normalizeMobileKeypress } from "../mobile-keypress.js";
|
|
8
|
+
import { buildIosAppSwitcherGesture, buildMobileKeypressPayload, getIosTransport, normalizeMobileKeypress } from "../mobile-keypress.js";
|
|
9
9
|
import { actionForErrorLog, buildActionErrorOutput, buildDebugTextField, buildSensitiveDebugFields, buildTypeActionPayload, buildTypeActionText, isSensitiveAction, } from "../sensitive-action.js";
|
|
10
10
|
import { getActiveSession, getDevicePixelRatio } from "./connection.js";
|
|
11
11
|
import { logger } from "../../utils/logger.js";
|
|
@@ -151,6 +151,15 @@ export async function handleModelAction(simulatorId, action, scale = 1.0, contex
|
|
|
151
151
|
});
|
|
152
152
|
await appium.pressButton(session.sessionId, transport.transportValue);
|
|
153
153
|
}
|
|
154
|
+
else if (transport.transportType === "ios-gesture") {
|
|
155
|
+
const gesture = buildIosAppSwitcherGesture(await appium.getWindowSize(session.sessionId));
|
|
156
|
+
addOutput({
|
|
157
|
+
type: "action",
|
|
158
|
+
text: normalized.logText,
|
|
159
|
+
...meta(buildMobileKeypressPayload(normalized, transport, { gesture }))
|
|
160
|
+
});
|
|
161
|
+
await appium.dragFromToForDuration(session.sessionId, gesture);
|
|
162
|
+
}
|
|
154
163
|
else {
|
|
155
164
|
addOutput({
|
|
156
165
|
type: "action",
|
|
@@ -206,6 +206,12 @@ export async function drag(sessionId, startX, startY, endX, endY, duration = 500
|
|
|
206
206
|
],
|
|
207
207
|
});
|
|
208
208
|
}
|
|
209
|
+
export async function dragFromToForDuration(sessionId, gesture) {
|
|
210
|
+
await appiumRequest("POST", `/session/${sessionId}/execute/sync`, {
|
|
211
|
+
script: "mobile: dragFromToForDuration",
|
|
212
|
+
args: [gesture],
|
|
213
|
+
});
|
|
214
|
+
}
|
|
209
215
|
/**
|
|
210
216
|
* Press a device button (home, volumeUp, volumeDown)
|
|
211
217
|
* @param {string} sessionId
|
|
@@ -15,6 +15,7 @@ export function normalizeMobileKeypress(platform, keys = []) {
|
|
|
15
15
|
ESC: { semanticKey: "IGNORE_ESC", logText: "Ignoring Escape on iOS" },
|
|
16
16
|
ESCAPE: { semanticKey: "IGNORE_ESC", logText: "Ignoring Escape on iOS" },
|
|
17
17
|
HOME: { semanticKey: "HOME", logText: "Pressing Home" },
|
|
18
|
+
APP_SWITCH: { semanticKey: "APP_SWITCH", logText: "Opening App Switcher" },
|
|
18
19
|
ENTER: { semanticKey: "ENTER", logText: "Pressing Enter" },
|
|
19
20
|
RETURN: { semanticKey: "ENTER", logText: "Pressing Enter" },
|
|
20
21
|
BACKSPACE: { semanticKey: "BACKSPACE", logText: "Pressing Backspace" },
|
|
@@ -98,6 +99,8 @@ export function getIosTransport(normalized) {
|
|
|
98
99
|
return { transportType: "noop", transportValue: null, ignored: true };
|
|
99
100
|
case "HOME":
|
|
100
101
|
return { transportType: "ios-button", transportValue: "home" };
|
|
102
|
+
case "APP_SWITCH":
|
|
103
|
+
return { transportType: "ios-gesture", transportValue: "mobile: dragFromToForDuration" };
|
|
101
104
|
case "ENTER":
|
|
102
105
|
return { transportType: "text", transportValue: "\n" };
|
|
103
106
|
case "BACKSPACE":
|
|
@@ -108,6 +111,16 @@ export function getIosTransport(normalized) {
|
|
|
108
111
|
throw new Error(`Unsupported iOS semantic key: ${normalized.semanticKey}`);
|
|
109
112
|
}
|
|
110
113
|
}
|
|
114
|
+
export function buildIosAppSwitcherGesture({ width, height }) {
|
|
115
|
+
const centerX = Math.round(width / 2);
|
|
116
|
+
return {
|
|
117
|
+
duration: 1,
|
|
118
|
+
fromX: centerX,
|
|
119
|
+
fromY: Math.round(height) - 5,
|
|
120
|
+
toX: centerX,
|
|
121
|
+
toY: Math.round(height / 2)
|
|
122
|
+
};
|
|
123
|
+
}
|
|
111
124
|
export function buildMobileKeypressPayload(normalized, transport, extras = {}) {
|
|
112
125
|
return {
|
|
113
126
|
keys: [normalized.requestedKey],
|
|
@@ -130,12 +130,12 @@ const iosCloseKeyboardTool = {
|
|
|
130
130
|
const loadmillTool = {
|
|
131
131
|
type: "function",
|
|
132
132
|
name: "loadmill",
|
|
133
|
-
description: "
|
|
133
|
+
description: "Run a saved Loadmill API flow. Use this tool only when the current instruction contains the word 'Loadmill' (case-insensitive) or starts with the explicit 'loadmill:' directive. Never call it based only on generic words such as 'run', 'execute', 'flow', 'test', or 'script'. Without an explicit Loadmill marker, treat the instruction as a normal UI or browser test step.",
|
|
134
134
|
strict: true,
|
|
135
135
|
parameters: {
|
|
136
136
|
type: "object",
|
|
137
137
|
properties: {
|
|
138
|
-
command: { type: "string" },
|
|
138
|
+
command: { type: "string", description: "The explicitly requested Loadmill flow name or intent and any parameters." },
|
|
139
139
|
},
|
|
140
140
|
required: ["command"],
|
|
141
141
|
additionalProperties: false,
|
|
@@ -55,6 +55,7 @@ export class DroidReportUploadManager {
|
|
|
55
55
|
onWarning,
|
|
56
56
|
sequence: 0,
|
|
57
57
|
screenshotCount: 0,
|
|
58
|
+
lastScreenshotObjectName: null,
|
|
58
59
|
deduplication: new Map(),
|
|
59
60
|
pending: new Set(),
|
|
60
61
|
failed: false,
|
|
@@ -73,6 +74,7 @@ export class DroidReportUploadManager {
|
|
|
73
74
|
const hash = createHash("sha256").update(dataUrl).digest("hex");
|
|
74
75
|
const existing = session.deduplication.get(hash);
|
|
75
76
|
if (existing) {
|
|
77
|
+
session.lastScreenshotObjectName = existing;
|
|
76
78
|
return existing;
|
|
77
79
|
}
|
|
78
80
|
const body = decodeScreenshot(dataUrl);
|
|
@@ -90,6 +92,7 @@ export class DroidReportUploadManager {
|
|
|
90
92
|
}
|
|
91
93
|
const name = `screenshots/${String(++session.sequence).padStart(4, "0")}.png`;
|
|
92
94
|
session.screenshotCount += 1;
|
|
95
|
+
session.lastScreenshotObjectName = name;
|
|
93
96
|
session.deduplication.set(hash, name);
|
|
94
97
|
this.#enqueue(session, { name, body, contentType: "image/png" });
|
|
95
98
|
return name;
|
|
@@ -139,7 +142,11 @@ export class DroidReportUploadManager {
|
|
|
139
142
|
key,
|
|
140
143
|
screenshotCount: session.screenshotCount,
|
|
141
144
|
});
|
|
142
|
-
return {
|
|
145
|
+
return {
|
|
146
|
+
published: true,
|
|
147
|
+
key,
|
|
148
|
+
lastScreenshotObjectName: session.lastScreenshotObjectName,
|
|
149
|
+
};
|
|
143
150
|
}
|
|
144
151
|
async waitForSession(session) {
|
|
145
152
|
while (session?.pending?.size) {
|
|
@@ -61,6 +61,7 @@ export function normalizeDroidRunStatus({ status = null, outcome = null, success
|
|
|
61
61
|
* contextOptimizationThreshold?: number | null,
|
|
62
62
|
* } | null,
|
|
63
63
|
* aimAssistMode?: string | null,
|
|
64
|
+
* lloydJobId?: string | null,
|
|
64
65
|
* }} input
|
|
65
66
|
* @returns {{
|
|
66
67
|
* droidVersion: string | null,
|
|
@@ -68,17 +69,19 @@ export function normalizeDroidRunStatus({ status = null, outcome = null, success
|
|
|
68
69
|
* llmProviderMode: string | null | undefined,
|
|
69
70
|
* stepDelayMs: number | null | undefined,
|
|
70
71
|
* aimAssistMode: string | null,
|
|
72
|
+
* lloydJobId?: string,
|
|
71
73
|
* contextOptimizationEnabled: boolean | null | undefined,
|
|
72
74
|
* contextOptimizationThreshold: number | null | undefined,
|
|
73
75
|
* }}
|
|
74
76
|
*/
|
|
75
|
-
export function buildDroidRunConfiguration({ droidVersion = null, runtimeConfig = null, aimAssistMode = null, } = {}) {
|
|
77
|
+
export function buildDroidRunConfiguration({ droidVersion = null, runtimeConfig = null, aimAssistMode = null, lloydJobId = null, } = {}) {
|
|
76
78
|
return {
|
|
77
79
|
droidVersion,
|
|
78
80
|
model: runtimeConfig?.cuaModel,
|
|
79
81
|
llmProviderMode: runtimeConfig?.llmProviderMode,
|
|
80
82
|
stepDelayMs: runtimeConfig?.stepDelayMs,
|
|
81
83
|
aimAssistMode,
|
|
84
|
+
...(lloydJobId ? { lloydJobId } : {}),
|
|
82
85
|
contextOptimizationEnabled: runtimeConfig?.contextOptimizationEnabled,
|
|
83
86
|
contextOptimizationThreshold: runtimeConfig?.contextOptimizationThreshold,
|
|
84
87
|
};
|
|
@@ -108,6 +111,7 @@ export function buildDroidRunReportPayload(input = {}) {
|
|
|
108
111
|
assertionsFailed: nonNegativeIntegerOrNull(input.assertionsFailed),
|
|
109
112
|
retries: nonNegativeIntegerOrNull(input.retries),
|
|
110
113
|
error: trimToNull(input.error),
|
|
114
|
+
lloydJobId: trimToNull(input.lloydJobId),
|
|
111
115
|
};
|
|
112
116
|
for (const key of Object.keys(payload)) {
|
|
113
117
|
if (payload[key] == null) {
|
|
@@ -248,7 +248,7 @@ function buildMobileBaseSystemPrompt(deviceInfo, options = {}) {
|
|
|
248
248
|
|
|
249
249
|
Mobile-Specific Notes:
|
|
250
250
|
- HOME key returns to the home screen
|
|
251
|
-
-
|
|
251
|
+
- APP_SWITCH opens Recent Apps on Android and the App Switcher on iOS
|
|
252
252
|
- On Android, ESC key maps to Back
|
|
253
253
|
- On iOS, ESC has no effect; use visible on-screen controls instead
|
|
254
254
|
- Never use CTRL, CMD, ALT, OPTION, or SHIFT in a keypress action
|
|
@@ -83,7 +83,7 @@ ${browserTarget ? "" : ` - Every \`type\` action must end its computer-action
|
|
|
83
83
|
- Otherwise, including when only a slim hardware-keyboard toolbar or suggestions are visible, continue without \`close_keyboard\`
|
|
84
84
|
- Apply the same dismissal decision after \`type_sensitive_text\`, \`type_text_slowly\`, and \`type_secret_slowly\`
|
|
85
85
|
`}
|
|
86
|
-
- Use the \`loadmill\` tool
|
|
86
|
+
- Use the \`loadmill\` tool only when the current instruction contains the word "Loadmill" (case-insensitive) or starts with the explicit \`loadmill:\` directive. Pass the requested flow name/intent and parameters as the command; the tool will search Loadmill, select the best matching flow, run it, and wait for completion. Never infer Loadmill intent from generic words such as "run", "execute", "flow", "test", or "script". Without an explicit Loadmill marker, handle the instruction as a normal UI or browser test step. Do not handle an explicitly requested Loadmill flow by typing in the app UI.
|
|
87
87
|
3. Execute the required actions
|
|
88
88
|
4. ${browserTarget ? "Before tapping a mutating action, dismiss the keyboard if it is open and not needed" : "Use `close_keyboard` before any later interaction when the post-type dismissal rules require it"}
|
|
89
89
|
5. After a mutating action, inspect the resulting screen for success cues such as cleared fields, reset buttons, changed status, refreshed content, or navigation
|
package/package.json
CHANGED
|
@@ -20,6 +20,8 @@ All Loadmill models require `LOADMILL_API_TOKEN`. Do not switch from Pulse to Sm
|
|
|
20
20
|
|
|
21
21
|
Include `--report reports/<test-name>.html` for normal test runs unless the user opts out. Prefer an explicit filename over a directory so the result is easy to locate. Add `--debug` for every agent-observed run: it writes the screenshots needed for live progress updates as well as diagnosis.
|
|
22
22
|
|
|
23
|
+
When downstream automation needs an exact Loadmill handoff, add `--report-metadata reports/<test-name>.json`. Read it only after Droid exits. Droid clears the destination at run start and writes the saved run ID plus the last uploaded screenshot object name only after cloud report publication succeeds, including when a failed test's report was published. Do not infer the run or screenshot by searching recent Loadmill runs when this file was requested but absent.
|
|
24
|
+
|
|
23
25
|
```sh
|
|
24
26
|
# Attached Android device or running emulator
|
|
25
27
|
droid-cua run tests/example.dcua --llm-provider loadmill --cua-model loadmill-pulse --avd adb:<serial> --context tests/context.md --debug --report reports/example.html
|
|
@@ -105,6 +107,7 @@ When the command completes, show the final safe screenshot in addition to the re
|
|
|
105
107
|
- Exit `130`: interrupted run.
|
|
106
108
|
- `--debug`: writes structured execution JSONL and adjacent screenshots under `logs/`.
|
|
107
109
|
- `--report`: writes a self-contained HTML report. Prefer an explicit `.html` file path such as `reports/example.html` so the evidence location is unambiguous.
|
|
110
|
+
- `--report-metadata`: writes the exact published Loadmill run and last screenshot handoff for downstream automation. Treat an absent file as no successful publication for the current invocation.
|
|
108
111
|
|
|
109
112
|
After the run, verify that the report exists and resolve its absolute path. Open it for the user when the agent environment supports local-file opening and policy permits; otherwise provide a clickable local-file link or the absolute path so the user can open it directly.
|
|
110
113
|
|