verikun 0.16.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +49 -24
- package/dist/agent/remote.js +5 -1
- package/dist/args.js +1 -0
- package/dist/cli.js +110 -12
- package/dist/drivers/adb.js +27 -6
- package/dist/drivers/ios.js +5 -0
- package/dist/report.js +68 -9
- package/dist/run.js +167 -8
- package/dist/server.js +44 -1
- package/dist/suite.js +179 -49
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -104,7 +104,7 @@ vk screenshot # -> ./.verikun/screen.png
|
|
|
104
104
|
| Command | Description |
|
|
105
105
|
|---|---|
|
|
106
106
|
| `ai <file> [--model m] [--max-cost-usd n] [--timeout dur] [--cost-override in/out] [--effort e] [--package pkg] [--app-build id] [--server url] [--show-plan] [--recompile] [--json]` | Run a plain-English test: compile it to a deterministic plan once, replay it model-free, and self-heal failures via the model. Needs `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` (per model), or no key with `--model codex-cli` / `cursor-cli` (a logged-in `codex` / `cursor-agent` CLI). See [AI](#ai--natural-language-tests). |
|
|
107
|
-
| `suite <dir> [--app <id>] [--name n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
|
|
107
|
+
| `suite <dir> [--app <id>] [--name n] [--retries n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
|
|
108
108
|
|
|
109
109
|
### Remote
|
|
110
110
|
| Command | Description |
|
|
@@ -122,7 +122,7 @@ vk screenshot # -> ./.verikun/screen.png
|
|
|
122
122
|
|---|---|
|
|
123
123
|
| `run start [name] [--force]` | Begin a named run. One auto-starts on the first action if you don't. |
|
|
124
124
|
| `run status` | Show the active run and its recorded steps. |
|
|
125
|
-
| `run archive [name]` | Write JUnit + HTML report to `./.verikun/runs/<id>/`; exits non-zero if any step failed. |
|
|
125
|
+
| `run archive [name] [--no-logs]` | Write JUnit + HTML report to `./.verikun/runs/<id>/`; exits non-zero if any step failed. Captures `artifacts/logcat.txt` by default. |
|
|
126
126
|
| `run clear` | Discard the active run without a report. |
|
|
127
127
|
|
|
128
128
|
## Test runs & reports
|
|
@@ -130,10 +130,11 @@ vk screenshot # -> ./.verikun/screen.png
|
|
|
130
130
|
Actions are recorded into a **test run** — one auto-starts on the first action
|
|
131
131
|
(set `VERIKUN_NO_RUN=1` to disable). Every command becomes a step with its
|
|
132
132
|
timing, the selector + identifier it resolved through, and pass/fail; a failing
|
|
133
|
-
step also captures a screenshot **and** the UI hierarchy of the page.
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
the report
|
|
133
|
+
step also captures a screenshot **and** the UI hierarchy of the page. Device
|
|
134
|
+
logs are captured automatically at archive time into `artifacts/logcat.txt`
|
|
135
|
+
(session-scoped when possible). You can still run `vk log <package>` mid-run to
|
|
136
|
+
pull a snapshot into a step for the report; `--no-logs` / `VERIKUN_NO_LOGS`
|
|
137
|
+
skips the archive dump on green runs (failures still capture).
|
|
137
138
|
|
|
138
139
|
`vk run archive` finalizes the run into `./.verikun/runs/<id>/`:
|
|
139
140
|
|
|
@@ -141,8 +142,11 @@ the report alongside the failure.
|
|
|
141
142
|
for failed assertions, `<error>` for environment errors, and the resolved
|
|
142
143
|
identifier in `<system-out>`. Drops straight into CI.
|
|
143
144
|
- **`report.html`** — a self-contained report: every step, the identifiers used,
|
|
144
|
-
any screenshots taken, the screenshot + hierarchy of any failed page,
|
|
145
|
-
device
|
|
145
|
+
any screenshots taken, the screenshot + hierarchy of any failed page, a link to
|
|
146
|
+
the full device log, an app-scoped log accordion when the run launched an app,
|
|
147
|
+
and any per-step logs from `vk log`.
|
|
148
|
+
- **`artifacts/logcat.txt`** — full device log for the run window (default).
|
|
149
|
+
- **`artifacts/logcat-app.txt`** — app-scoped log (when a package/bundle was launched).
|
|
146
150
|
- **`run.json`** — the raw recording.
|
|
147
151
|
|
|
148
152
|
`vk run archive` exits non-zero when the run contained failures, so the same
|
|
@@ -287,12 +291,29 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER" # remote devi
|
|
|
287
291
|
- **Each test is a full `vk ai` run** — plan cache, self-healing, cost budget, and
|
|
288
292
|
its own archived JUnit + HTML report under `./.verikun/runs/<id>/`. A test that
|
|
289
293
|
fails (or errors) doesn't stop the suite; the rest still run.
|
|
290
|
-
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
294
|
+
- **`--retries N` recovers from flakes.** A failed test is re-run up to N times
|
|
295
|
+
(default `0` — opt-in, so CI cost/time stay predictable). If a later attempt
|
|
296
|
+
passes, the suite exits `0` and the flake is a **warning**, not a hard failure.
|
|
297
|
+
Failed attempt archives stay linked from the suite overview (`attempts` on the
|
|
298
|
+
test row + a `warnings` list on the manifest), so flakiness remains visible.
|
|
299
|
+
Cost and duration sum across attempts.
|
|
300
|
+
- **What earns a retry:** anything that might come out differently — a flaky
|
|
301
|
+
selector, a wedged app, and **a broken environment**, including a `vk server`
|
|
302
|
+
connection dropping mid-suite. The bias is intentional: an attempt costs one test,
|
|
303
|
+
giving up costs the whole suite plus a human rerunning it. Environment retries wait
|
|
304
|
+
a little longer each time (an outage that survives the health probe usually needs
|
|
305
|
+
seconds, not milliseconds) and each one lands in `warnings`, so riding out a wobble
|
|
306
|
+
is never silent. Exactly two failures are never retried, because a rerun cannot
|
|
307
|
+
change them: a **budget abort** (each attempt gets its own ceiling, so it would just
|
|
308
|
+
re-abort having spent twice) and a **usage error** (exit `2` — an unreadable test
|
|
309
|
+
file, a payload the server refuses).
|
|
310
|
+
- **But a broken *environment* does stop it, once the attempts are gone.** If a test
|
|
311
|
+
dies from an environment error (exit 3 — tool gone, device unplugged, server
|
|
312
|
+
unreachable), the toolchain is re-probed; only if it is *still* broken **and** no
|
|
313
|
+
retries remain does the suite abort. That re-probe matters: a transient
|
|
314
|
+
`uiautomator` dump failure also exits 3, and shouldn't vaporize a 20-test run.
|
|
315
|
+
Continuing on a genuinely dead box just produces one identical red row per remaining
|
|
316
|
+
test — noise that reads exactly like a mass regression.
|
|
296
317
|
- **The suite writes an overview** to `./.verikun/suites/<id>/`:
|
|
297
318
|
- **`index.json`** — a stable, `schemaVersion`ed manifest: per-test pass/fail,
|
|
298
319
|
steps, model repairs, cost, duration, and the run id, plus suite totals. This
|
|
@@ -300,17 +321,21 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER" # remote devi
|
|
|
300
321
|
it (see the [CI recipe](#ci-recipe)) instead of verikun growing upload plugins.
|
|
301
322
|
On an abort it also carries `aborted: {reason, notRun}`; the not-run tests get
|
|
302
323
|
**no rows and no place in `totals`**, so `passed + failed === tests` still holds
|
|
303
|
-
and nothing downstream mistakes a skipped test for a regression.
|
|
324
|
+
and nothing downstream mistakes a skipped test for a regression. Retried flakes
|
|
325
|
+
add `flaky` / `attempts` on the test row and suite-level `warnings` (additive;
|
|
326
|
+
`schemaVersion` stays `1`).
|
|
304
327
|
- **`index.html`** — a summary page linking every test's `report.html`, with a
|
|
305
|
-
banner naming the not-run tests when the suite aborted
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
328
|
+
banner naming the not-run tests when the suite aborted, and a warnings banner
|
|
329
|
+
when a flake recovered on retry (prior failed attempts stay linked).
|
|
330
|
+
- **Exit code is the CI gate:** `0` all green (including flakes that recovered with
|
|
331
|
+
`--retries`) · `1` a test failed · `2` bad/empty directory · `3` environment (the
|
|
332
|
+
provider or the device toolchain is unavailable, or the box broke mid-run). The
|
|
333
|
+
`1`-vs-`3` split is the point: `1` is a regression to investigate, `3` is a
|
|
334
|
+
machine to fix. All `ai` flags (`--model`, `--max-cost-usd`, `--timeout`, …)
|
|
335
|
+
apply to every test; both the provider (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`,
|
|
336
|
+
or the `codex` / `cursor-agent` CLI for `--model codex-cli` / `cursor-cli`)
|
|
337
|
+
**and** the device toolchain (`adb` / `idb` + a resolvable device) are checked up
|
|
338
|
+
front, before anything is compiled.
|
|
314
339
|
|
|
315
340
|
## Remote devices — `vk server`
|
|
316
341
|
|
package/dist/agent/remote.js
CHANGED
|
@@ -112,7 +112,7 @@ function createRemoteBackend(opts, health) {
|
|
|
112
112
|
const execRaw = async (req, record) => {
|
|
113
113
|
const res = await t.postJson('/v1/exec', req, EXEC_TIMEOUT_MS);
|
|
114
114
|
if (record && res.step)
|
|
115
|
-
opts.onStep?.(res.step, decodeArtifacts(res.artifacts));
|
|
115
|
+
opts.onStep?.(res.step, decodeArtifacts(res.artifacts), res.logStart);
|
|
116
116
|
return { code: res.code, error: res.error ? (0, rpc_1.rebuildError)(res.error) : undefined };
|
|
117
117
|
};
|
|
118
118
|
return {
|
|
@@ -121,6 +121,10 @@ function createRemoteBackend(opts, health) {
|
|
|
121
121
|
const res = await t.postJson('/v1/elements', {}, ELEMENTS_TIMEOUT_MS);
|
|
122
122
|
return res.elements;
|
|
123
123
|
},
|
|
124
|
+
async getLogs(logOpts = {}) {
|
|
125
|
+
const res = await t.postJson('/v1/logs', logOpts, ELEMENTS_TIMEOUT_MS);
|
|
126
|
+
return res.logs ?? '';
|
|
127
|
+
},
|
|
124
128
|
async install(appPath) {
|
|
125
129
|
// v1 remote installs are single-file uploads; the extension is the only thing
|
|
126
130
|
// the client tells the server about the artifact (never a path).
|
package/dist/args.js
CHANGED
package/dist/cli.js
CHANGED
|
@@ -975,16 +975,47 @@ function cmdRun(positionals, flags, platform, device) {
|
|
|
975
975
|
case 'archive':
|
|
976
976
|
case 'finish':
|
|
977
977
|
case 'save': {
|
|
978
|
-
const
|
|
978
|
+
const noLogs = (0, args_1.flagBool)(flags, 'no-logs');
|
|
979
|
+
// Best-effort: archive-time log capture needs a device. Prefer the run's
|
|
980
|
+
// bound serial/platform so a multi-device host hits the right one. A
|
|
981
|
+
// missing/broken toolchain must not prevent sealing the report.
|
|
982
|
+
let fetchLogs;
|
|
983
|
+
const active = run_1.Recorder.status();
|
|
984
|
+
const hasFailures = !!active?.steps.some((s) => s.status !== 'passed');
|
|
985
|
+
if ((0, run_1.wantsArchiveLogs)(hasFailures, noLogs)) {
|
|
986
|
+
try {
|
|
987
|
+
const plat = active?.platform === 'ios' || active?.platform === 'android'
|
|
988
|
+
? active.platform
|
|
989
|
+
: platform;
|
|
990
|
+
const driver = (0, drivers_1.getDriver)(plat, active?.device || device);
|
|
991
|
+
fetchLogs = (opts) => driver.getLogs(opts);
|
|
992
|
+
}
|
|
993
|
+
catch (e) {
|
|
994
|
+
(0, output_1.err)(`[verikun] archive log capture unavailable (${e.message})`);
|
|
995
|
+
}
|
|
996
|
+
}
|
|
997
|
+
const { dir, xmlPath, htmlPath, state } = run_1.Recorder.archive(positionals[1], { noLogs, fetchLogs });
|
|
979
998
|
const { passed, failed } = tally(state.steps);
|
|
980
999
|
if (asJson) {
|
|
981
|
-
(0, output_1.json)({
|
|
1000
|
+
(0, output_1.json)({
|
|
1001
|
+
archived: dir,
|
|
1002
|
+
report: htmlPath,
|
|
1003
|
+
junit: xmlPath,
|
|
1004
|
+
steps: state.steps.length,
|
|
1005
|
+
passed,
|
|
1006
|
+
failed,
|
|
1007
|
+
...(state.logFile ? { logFile: state.logFile } : {}),
|
|
1008
|
+
});
|
|
982
1009
|
}
|
|
983
1010
|
else {
|
|
984
1011
|
(0, output_1.out)(dir); // primary result: the archived run directory
|
|
985
1012
|
(0, output_1.err)(`archived '${state.name}': ${state.steps.length} step(s), ${passed} passed, ${failed} failed/error`);
|
|
986
1013
|
(0, output_1.err)(` JUnit: ${xmlPath}`);
|
|
987
1014
|
(0, output_1.err)(` HTML: ${htmlPath}`);
|
|
1015
|
+
if (state.logFile)
|
|
1016
|
+
(0, output_1.err)(` Logs: ${(0, node_path_1.join)(dir, state.logFile)}`);
|
|
1017
|
+
if (state.appLogFile)
|
|
1018
|
+
(0, output_1.err)(` App: ${(0, node_path_1.join)(dir, state.appLogFile)}`);
|
|
988
1019
|
}
|
|
989
1020
|
// Exit non-zero when the run contained failures, so CI can gate on it.
|
|
990
1021
|
return failed > 0 ? 1 : 0;
|
|
@@ -1280,6 +1311,7 @@ async function resolveBackend(platform, device, flags) {
|
|
|
1280
1311
|
backend: {
|
|
1281
1312
|
exec: (command, positionals, f) => executeOutcome(command, positionals, f, driver),
|
|
1282
1313
|
getElements: () => driver.getElements(),
|
|
1314
|
+
getLogs: (opts) => driver.getLogs(opts),
|
|
1283
1315
|
install: (appPath) => driver.install(appPath),
|
|
1284
1316
|
reset: (appId) => {
|
|
1285
1317
|
assertSafeAppId(appId);
|
|
@@ -1300,8 +1332,9 @@ async function resolveBackend(platform, device, flags) {
|
|
|
1300
1332
|
url: server,
|
|
1301
1333
|
authKey: (0, args_1.flagStr)(flags, 'auth-key') || process.env.VERIKUN_SERVER_AUTH_KEY || undefined,
|
|
1302
1334
|
// Each remote step is spliced into the local active run so the archived report
|
|
1303
|
-
// is identical to a local run's.
|
|
1304
|
-
|
|
1335
|
+
// is identical to a local run's. logStart travels from the server's device clock
|
|
1336
|
+
// so archive-time / vk log scoping works without a local driver.
|
|
1337
|
+
onStep: (step, artifacts, logStart) => run_1.Recorder.appendForeignStep(step, artifacts, { ...runCtx, logStart }),
|
|
1305
1338
|
};
|
|
1306
1339
|
const health = await (0, remote_1.pingServer)(opts); // fails fast (exit 3) on a bad URL or key
|
|
1307
1340
|
runCtx = { platform: health.platform, device: health.serial };
|
|
@@ -1325,6 +1358,45 @@ async function resolveBackend(platform, device, flags) {
|
|
|
1325
1358
|
remote: { url: server, version: health.version },
|
|
1326
1359
|
};
|
|
1327
1360
|
}
|
|
1361
|
+
/**
|
|
1362
|
+
* Best-effort archive-time device-log capture via an ExecBackend (local driver or
|
|
1363
|
+
* remote `/v1/logs`). Writes `artifacts/logcat.txt` onto the active run so a later
|
|
1364
|
+
* `Recorder.archive()` finds `logFile` already set. Never throws — a gone device
|
|
1365
|
+
* must not prevent sealing the report.
|
|
1366
|
+
*/
|
|
1367
|
+
async function prefetchArchiveLogs(backend, noLogs = false) {
|
|
1368
|
+
const state = run_1.Recorder.status();
|
|
1369
|
+
if (!state)
|
|
1370
|
+
return;
|
|
1371
|
+
const hasFailures = state.steps.some((s) => s.status !== 'passed');
|
|
1372
|
+
if (!(0, run_1.wantsArchiveLogs)(hasFailures, noLogs))
|
|
1373
|
+
return;
|
|
1374
|
+
if (!backend.getLogs)
|
|
1375
|
+
return;
|
|
1376
|
+
// Skip only when both artifacts are already attached (e.g. a prior prefetch).
|
|
1377
|
+
if (state.logFile && state.appLogFile)
|
|
1378
|
+
return;
|
|
1379
|
+
const window = (0, run_1.archiveLogWindow)(state);
|
|
1380
|
+
try {
|
|
1381
|
+
const full = state.logFile ? undefined : await backend.getLogs(window);
|
|
1382
|
+
const appId = (0, run_1.inferRunAppId)(state);
|
|
1383
|
+
let app;
|
|
1384
|
+
if (appId && !state.appLogFile) {
|
|
1385
|
+
try {
|
|
1386
|
+
app = await backend.getLogs({ ...window, appId, scopedOnly: true });
|
|
1387
|
+
}
|
|
1388
|
+
catch (e) {
|
|
1389
|
+
(0, output_1.err)(`[verikun] could not capture archive app logs (${e.message})`);
|
|
1390
|
+
}
|
|
1391
|
+
}
|
|
1392
|
+
if (full !== undefined || (app !== undefined && app !== '')) {
|
|
1393
|
+
run_1.Recorder.attachArchiveLogs(full, app);
|
|
1394
|
+
}
|
|
1395
|
+
}
|
|
1396
|
+
catch (e) {
|
|
1397
|
+
(0, output_1.err)(`[verikun] could not capture archive device logs (${e.message})`);
|
|
1398
|
+
}
|
|
1399
|
+
}
|
|
1328
1400
|
/**
|
|
1329
1401
|
* Run one natural-language test through a backend and return DATA — no stdout
|
|
1330
1402
|
* writes (stdout stays the caller's one result; progress streams to stderr).
|
|
@@ -1365,10 +1437,14 @@ async function runAiTest(file, opts, backend, platform, device) {
|
|
|
1365
1437
|
if (existing && existing.steps.length > 0) {
|
|
1366
1438
|
// Seal the pre-existing run into the archive instead of letting start(force=true)
|
|
1367
1439
|
// discard it — a manual in-progress run should never be silently lost.
|
|
1440
|
+
await prefetchArchiveLogs(backend);
|
|
1368
1441
|
const sealed = run_1.Recorder.archive();
|
|
1369
1442
|
(0, output_1.err)(`[ai] archived the active run ('${existing.name}', ${existing.steps.length} step(s)) → ${sealed.dir}`);
|
|
1370
1443
|
}
|
|
1371
1444
|
const started = run_1.Recorder.start(`ai: ${(0, node_path_1.basename)(file)}`, platform, device, true);
|
|
1445
|
+
// Prefer --package when the prose never launches by id (rare but possible).
|
|
1446
|
+
if (opts.pkg)
|
|
1447
|
+
run_1.Recorder.annotateRun({ appId: opts.pkg });
|
|
1372
1448
|
// Suppress per-step `out()` so stdout stays the one final result; progress -> stderr.
|
|
1373
1449
|
const prevQuiet = (0, output_1.setOutputQuiet)(true);
|
|
1374
1450
|
let result;
|
|
@@ -1395,6 +1471,7 @@ async function runAiTest(file, opts, backend, platform, device) {
|
|
|
1395
1471
|
// to roll over. Then let the error map to an exit code as usual.
|
|
1396
1472
|
run_1.Recorder.annotateRun({ ai: { ok: false, cost: cost.summaryLine(), modelRepairs: 0, improvements: [] } });
|
|
1397
1473
|
try {
|
|
1474
|
+
await prefetchArchiveLogs(backend);
|
|
1398
1475
|
run_1.Recorder.archive();
|
|
1399
1476
|
}
|
|
1400
1477
|
catch (sealErr) {
|
|
@@ -1422,6 +1499,7 @@ async function runAiTest(file, opts, backend, platform, device) {
|
|
|
1422
1499
|
run_1.Recorder.annotateRun({
|
|
1423
1500
|
ai: { ok: result.ok, cost: costLine, modelRepairs: result.modelRepairs, improvements: result.improvements },
|
|
1424
1501
|
});
|
|
1502
|
+
await prefetchArchiveLogs(backend);
|
|
1425
1503
|
const { dir, xmlPath, htmlPath, state } = run_1.Recorder.archive();
|
|
1426
1504
|
const status = result.ok
|
|
1427
1505
|
? 'PASS'
|
|
@@ -1537,7 +1615,7 @@ async function cmdInstall(positionals, flags) {
|
|
|
1537
1615
|
async function cmdSuiteEntry(positionals, flags) {
|
|
1538
1616
|
const dirArg = positionals[0];
|
|
1539
1617
|
if (!dirArg)
|
|
1540
|
-
throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--json]', 2);
|
|
1618
|
+
throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--retries n] [--json]', 2);
|
|
1541
1619
|
const opts = parseAiOptions(flags);
|
|
1542
1620
|
// Pre-flight the provider BEFORE touching any device/server: every test needs it
|
|
1543
1621
|
// to compile (on a cache miss) or to repair at runtime.
|
|
@@ -1708,11 +1786,23 @@ async function executeForServer(command, positionals, flags, driver, platform) {
|
|
|
1708
1786
|
catch {
|
|
1709
1787
|
/* surfaced by the command handler below */
|
|
1710
1788
|
}
|
|
1789
|
+
// Sample the device clock up front so the caller's run can set logStart (the
|
|
1790
|
+
// ephemeral recorder never persists RunState). Best-effort — empty/unavailable
|
|
1791
|
+
// just means archive / vk log fall back to last-N.
|
|
1792
|
+
let logStart;
|
|
1793
|
+
try {
|
|
1794
|
+
const t = driver.deviceTime();
|
|
1795
|
+
if (t)
|
|
1796
|
+
logStart = t;
|
|
1797
|
+
}
|
|
1798
|
+
catch {
|
|
1799
|
+
/* device clock unavailable */
|
|
1800
|
+
}
|
|
1711
1801
|
const recorder = run_1.Recorder.beginEphemeralStep(command, positionals, flags, platform, serial);
|
|
1712
1802
|
const ctx = { driver, platform, device: serial, positionals, flags, record: recorder };
|
|
1713
1803
|
const outcome = await runRecorded(command, ctx, recorder, driver);
|
|
1714
1804
|
const { step, artifacts } = recorder.takeEphemeral();
|
|
1715
|
-
return { ...outcome, step, artifacts };
|
|
1805
|
+
return { ...outcome, step, artifacts, ...(logStart ? { logStart } : {}) };
|
|
1716
1806
|
}
|
|
1717
1807
|
/**
|
|
1718
1808
|
* Run one already-parsed command for the CLI / `batch`: dispatch meta-commands,
|
|
@@ -1849,14 +1939,19 @@ AI (run a natural-language test — compile once, replay model-free, self-heal)
|
|
|
1849
1939
|
codex-cli | cursor-cli.
|
|
1850
1940
|
|
|
1851
1941
|
SUITE (run a directory of natural-language tests as one gated suite)
|
|
1852
|
-
suite <dir> [--app <id>] [--name n] [--
|
|
1942
|
+
suite <dir> [--app <id>] [--name n] [--retries n] [--json]
|
|
1943
|
+
(+ all \`ai\` flags, incl. --server)
|
|
1853
1944
|
Run every *.md in <dir> (lexicographic order —
|
|
1854
1945
|
prefix 01-, 02- to sequence; README.md skipped)
|
|
1855
1946
|
through \`vk ai\`. With --app, app data is reset
|
|
1856
|
-
between tests (iOS: force-stop).
|
|
1857
|
-
|
|
1858
|
-
|
|
1859
|
-
|
|
1947
|
+
between tests (iOS: force-stop). --retries N
|
|
1948
|
+
re-runs a failed test up to N times; a later
|
|
1949
|
+
pass recovers the suite (exit 0) and surfaces a
|
|
1950
|
+
warning, keeping failed-attempt evidence in the
|
|
1951
|
+
report. Writes a suite overview to
|
|
1952
|
+
./.verikun/suites/<id>/{index.json, index.html}
|
|
1953
|
+
linking each test's report. Exits 1 if any test
|
|
1954
|
+
failed — the CI gate.
|
|
1860
1955
|
|
|
1861
1956
|
SERVER (expose a locally-connected device to remote verikun clients)
|
|
1862
1957
|
server [--bind addr] [--port n] [--auth-key k] [--allow-install]
|
|
@@ -1879,7 +1974,10 @@ ENVIRONMENT
|
|
|
1879
1974
|
TEST RUNS (actions are recorded; a run auto-starts on first action)
|
|
1880
1975
|
run start [name] [--force] Begin a named run (else one starts implicitly)
|
|
1881
1976
|
run status Show the active run, its device/session, and steps
|
|
1882
|
-
run archive [name]
|
|
1977
|
+
run archive [name] [--no-logs] Write JUnit + HTML report, move to ./.verikun/runs/<id>/
|
|
1978
|
+
Captures artifacts/logcat.txt by default (session-scoped);
|
|
1979
|
+
--no-logs / VERIKUN_NO_LOGS skips on green runs (failures
|
|
1980
|
+
still capture). Capture is best-effort and never blocks archive.
|
|
1883
1981
|
run clear Discard the active run with no report
|
|
1884
1982
|
An implicit run auto-closes (archives) and rolls over on a device change, a
|
|
1885
1983
|
VERIKUN_SESSION change, or VERIKUN_RUN_IDLE_MIN minutes idle (default 30; 0 off).
|
package/dist/drivers/adb.js
CHANGED
|
@@ -356,15 +356,36 @@ class AdbDriver {
|
|
|
356
356
|
args.push('-t', String(n));
|
|
357
357
|
}
|
|
358
358
|
if (opts.appId) {
|
|
359
|
-
//
|
|
360
|
-
//
|
|
361
|
-
//
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
359
|
+
// Prefer --uid: it survives process death/restart (crash traces stay under the
|
|
360
|
+
// package's uid). Fall back to --pid for a live process when uid isn't known.
|
|
361
|
+
// When neither works: vk log falls through to system-wide so a FATAL EXCEPTION
|
|
362
|
+
// isn't missed; archive accordion passes scopedOnly to keep the dump empty.
|
|
363
|
+
const uid = this.packageUid(opts.appId);
|
|
364
|
+
if (uid) {
|
|
365
|
+
args.push(`--uid=${uid}`);
|
|
366
|
+
}
|
|
367
|
+
else {
|
|
368
|
+
const pid = this.shell(['pidof', opts.appId]).trim().split(/\s+/)[0];
|
|
369
|
+
if (pid)
|
|
370
|
+
args.push(`--pid=${pid}`);
|
|
371
|
+
else if (opts.scopedOnly)
|
|
372
|
+
return '';
|
|
373
|
+
}
|
|
365
374
|
}
|
|
366
375
|
return this.shell(args, 15000);
|
|
367
376
|
}
|
|
377
|
+
/** Android userId for an installed package, or '' if unknown. Used to scope
|
|
378
|
+
* logcat across process restarts (unlike pidof, which only sees a live process). */
|
|
379
|
+
packageUid(appId) {
|
|
380
|
+
try {
|
|
381
|
+
const out = this.shell(['dumpsys', 'package', appId], 10000);
|
|
382
|
+
const m = /\buserId=(\d+)\b/.exec(out);
|
|
383
|
+
return m?.[1] ?? '';
|
|
384
|
+
}
|
|
385
|
+
catch {
|
|
386
|
+
return '';
|
|
387
|
+
}
|
|
388
|
+
}
|
|
368
389
|
deviceTime() {
|
|
369
390
|
// logcat's default timestamp is MM-DD HH:MM:SS.mmm in the device's LOCAL time.
|
|
370
391
|
// Sample it from the device clock with a space-free format (so no device-shell
|
package/dist/drivers/ios.js
CHANGED
|
@@ -433,6 +433,11 @@ class IdbDriver {
|
|
|
433
433
|
const proc = opts.appId.split('.').pop() || opts.appId;
|
|
434
434
|
args.push('--predicate', `process CONTAINS "${proc}"`);
|
|
435
435
|
}
|
|
436
|
+
else if (opts.scopedOnly) {
|
|
437
|
+
// scopedOnly without an appId can't mean anything on iOS — empty rather than
|
|
438
|
+
// a full dump (mirrors Android's "couldn't scope" behaviour).
|
|
439
|
+
return '';
|
|
440
|
+
}
|
|
436
441
|
const out = (0, exec_1.runText)(XCRUN, args, { timeout: 20000 }).stdout;
|
|
437
442
|
if (opts.since)
|
|
438
443
|
return out; // the whole session window
|
package/dist/report.js
CHANGED
|
@@ -95,14 +95,20 @@ function toJUnitXml(run) {
|
|
|
95
95
|
.join('\n');
|
|
96
96
|
const suiteAttrs = `name="${xmlAttr(run.name)}" tests="${c.tests}" failures="${c.failures}" ` +
|
|
97
97
|
`errors="${c.errors}" time="${suiteTime}" timestamp="${xmlAttr(run.startedAt)}"`;
|
|
98
|
+
const suiteExtras = [];
|
|
99
|
+
if (run.logFile)
|
|
100
|
+
suiteExtras.push(`device log: ${run.logFile}`);
|
|
101
|
+
if (run.ai) {
|
|
102
|
+
suiteExtras.push('vk ai: ' +
|
|
103
|
+
run.ai.cost +
|
|
104
|
+
(run.ai.improvements.length ? '\nSuggested improvements:\n' + run.ai.improvements.join('\n') : ''));
|
|
105
|
+
}
|
|
98
106
|
return (`<?xml version="1.0" encoding="UTF-8"?>\n` +
|
|
99
107
|
`<testsuites name="verikun" tests="${c.tests}" failures="${c.failures}" errors="${c.errors}" time="${suiteTime}">\n` +
|
|
100
108
|
`<testsuite ${suiteAttrs}>\n` +
|
|
101
109
|
`${cases}\n` +
|
|
102
|
-
(
|
|
103
|
-
? ` <system-out>${xmlText('
|
|
104
|
-
run.ai.cost +
|
|
105
|
-
(run.ai.improvements.length ? '\nSuggested improvements:\n' + run.ai.improvements.join('\n') : ''))}</system-out>\n`
|
|
110
|
+
(suiteExtras.length
|
|
111
|
+
? ` <system-out>${xmlText(suiteExtras.join('\n'))}</system-out>\n`
|
|
106
112
|
: '') +
|
|
107
113
|
`</testsuite>\n</testsuites>\n`);
|
|
108
114
|
}
|
|
@@ -118,6 +124,7 @@ const STYLE = `
|
|
|
118
124
|
.summary { display:flex; gap:8px; flex-wrap:wrap; align-items:center; margin-bottom: 20px; }
|
|
119
125
|
.chip { font-weight:600; font-size:13px; padding:4px 10px; border-radius:999px; color:#fff; }
|
|
120
126
|
.chip.pass{background:var(--pass)} .chip.fail{background:var(--fail)} .chip.err{background:var(--err)}
|
|
127
|
+
.chip.warn{background:var(--err)}
|
|
121
128
|
.chip.muted{ background:#eaeef2; color:var(--muted); }
|
|
122
129
|
ol.steps { list-style:none; margin:0; padding:0; }
|
|
123
130
|
li.step { background:#fff; border:1px solid var(--line); border-left-width:4px; border-radius:8px; margin-bottom:10px; padding:12px 14px; }
|
|
@@ -135,6 +142,10 @@ const STYLE = `
|
|
|
135
142
|
.msg.fail { color:var(--fail); }
|
|
136
143
|
img.shot { display:block; margin-top:10px; max-width:300px; max-height:520px; border:1px solid var(--line); border-radius:6px; }
|
|
137
144
|
details { margin-top:8px; }
|
|
145
|
+
details.run-log { margin-top:20px; background:#fff; border:1px solid var(--line); border-radius:8px; padding:10px 14px; }
|
|
146
|
+
details.run-log > summary { font-weight:600; color:#1f2328; }
|
|
147
|
+
details.run-log > summary a { font-weight:400; }
|
|
148
|
+
details.run-log pre { max-height:480px; }
|
|
138
149
|
summary { cursor:pointer; color:var(--muted); font-size:13px; }
|
|
139
150
|
pre { background:#0d1117; color:#e6edf3; padding:12px; border-radius:6px; overflow:auto; font-size:12px; line-height:1.45; max-height:360px; }
|
|
140
151
|
.aibox { background:#fff; border:1px solid var(--line); border-radius:8px; padding:12px 14px; margin-bottom:20px; font-size:13px; }
|
|
@@ -206,19 +217,40 @@ const SUITE_STYLE = `
|
|
|
206
217
|
table.tests td.num { text-align:right; font-variant-numeric:tabular-nums; white-space:nowrap; }
|
|
207
218
|
table.tests a { color:inherit; }
|
|
208
219
|
.fail-reason { color:var(--fail); font-size:12px; margin-top:2px; }
|
|
220
|
+
.flake-note { color:var(--err); font-size:12px; margin-top:2px; }
|
|
221
|
+
.attempts { margin-top:4px; font-size:12px; color:var(--muted); }
|
|
222
|
+
.attempts a { color:var(--fail); }
|
|
209
223
|
.aborted { background:#fff4e5; border:1px solid #f0b429; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
|
|
210
224
|
.aborted strong { color:#8a5300; }
|
|
211
225
|
.aborted ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
|
|
226
|
+
.warnings { background:#fff8c5; border:1px solid #d4a72c; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
|
|
227
|
+
.warnings strong { color:#7d4e00; }
|
|
228
|
+
.warnings ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
|
|
212
229
|
`;
|
|
230
|
+
function suiteAttemptLinks(attempts, linkBase) {
|
|
231
|
+
const links = attempts
|
|
232
|
+
.map((a, i) => {
|
|
233
|
+
const label = `attempt ${i + 1}`;
|
|
234
|
+
if (!a.id)
|
|
235
|
+
return htmlEsc(label);
|
|
236
|
+
return `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(a.id)}/report.html`)}">${htmlEsc(label)}</a>`;
|
|
237
|
+
})
|
|
238
|
+
.join(', ');
|
|
239
|
+
return `<div class="attempts">prior failed: ${links}</div>`;
|
|
240
|
+
}
|
|
213
241
|
function suiteTestRow(t, linkBase) {
|
|
214
242
|
// A test that errored before its run started (id '') has no report to link.
|
|
215
243
|
const label = t.id
|
|
216
244
|
? `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(t.id)}/report.html`)}">${htmlEsc(t.name)}</a>`
|
|
217
245
|
: htmlEsc(t.name);
|
|
218
246
|
const failure = t.failure ? `<div class="fail-reason">${htmlEsc(t.failure)}</div>` : '';
|
|
247
|
+
const flake = t.flaky ? `<div class="flake-note">passed on retry (flake)</div>` : '';
|
|
248
|
+
const prior = t.attempts?.length ? suiteAttemptLinks(t.attempts, linkBase) : '';
|
|
249
|
+
const status = t.flaky ? 'FLAKY' : t.ok ? 'PASS' : 'FAIL';
|
|
250
|
+
const statusClass = t.ok ? 'passed' : 'failed';
|
|
219
251
|
return ` <tr>
|
|
220
|
-
<td><span class="st ${
|
|
221
|
-
<td>${label}${failure}</td>
|
|
252
|
+
<td><span class="st ${statusClass}">${status}</span></td>
|
|
253
|
+
<td>${label}${flake}${failure}${prior}</td>
|
|
222
254
|
<td class="num">${t.passedSteps}/${t.steps}${t.failedSteps ? ` (${t.failedSteps} failed)` : ''}</td>
|
|
223
255
|
<td class="num">${t.modelRepairs || ''}</td>
|
|
224
256
|
<td class="num">$${t.costUsd.toFixed(4)}</td>
|
|
@@ -233,9 +265,11 @@ function suiteTestRow(t, linkBase) {
|
|
|
233
265
|
function toSuiteHtml(suite, opts = {}) {
|
|
234
266
|
const linkBase = opts.linkBase ?? '../../';
|
|
235
267
|
const t = suite.totals;
|
|
268
|
+
const flaky = suite.tests.filter((x) => x.flaky).length;
|
|
236
269
|
const chips = [
|
|
237
270
|
`<span class="chip pass">${t.passed} passed</span>`,
|
|
238
271
|
t.failed ? `<span class="chip fail">${t.failed} failed</span>` : '',
|
|
272
|
+
flaky ? `<span class="chip warn">${flaky} flaky</span>` : '',
|
|
239
273
|
suite.aborted ? `<span class="chip fail">ABORTED</span>` : '',
|
|
240
274
|
`<span class="chip muted">${t.tests} tests · ${t.steps} steps · ${fmtDuration(t.durationMs)} · $${t.costUsd.toFixed(4)}</span>`,
|
|
241
275
|
]
|
|
@@ -250,6 +284,13 @@ function toSuiteHtml(suite, opts = {}) {
|
|
|
250
284
|
${suite.aborted.notRun.length
|
|
251
285
|
? ` <ul>${suite.aborted.notRun.map((f) => `<li>${htmlEsc(f)} — not run</li>`).join('')}</ul>\n`
|
|
252
286
|
: ''} </div>
|
|
287
|
+
`
|
|
288
|
+
: '';
|
|
289
|
+
const warningsBanner = suite.warnings?.length
|
|
290
|
+
? ` <div class="warnings">
|
|
291
|
+
<strong>Warnings</strong>
|
|
292
|
+
<ul>${suite.warnings.map((w) => `<li>${htmlEsc(w)}</li>`).join('')}</ul>
|
|
293
|
+
</div>
|
|
253
294
|
`
|
|
254
295
|
: '';
|
|
255
296
|
const metaBits = [
|
|
@@ -274,7 +315,7 @@ ${suite.aborted.notRun.length
|
|
|
274
315
|
<div class="summary">
|
|
275
316
|
${chips}
|
|
276
317
|
</div>
|
|
277
|
-
${abortedBanner} <table class="tests">
|
|
318
|
+
${abortedBanner}${warningsBanner} <table class="tests">
|
|
278
319
|
<thead><tr><th></th><th>Test</th><th>Steps</th><th>Repairs</th><th>Cost</th><th>Duration</th></tr></thead>
|
|
279
320
|
<tbody>
|
|
280
321
|
${suite.tests.map((x) => suiteTestRow(x, linkBase)).join('\n')}
|
|
@@ -285,7 +326,12 @@ ${suite.tests.map((x) => suiteTestRow(x, linkBase)).join('\n')}
|
|
|
285
326
|
</html>
|
|
286
327
|
`;
|
|
287
328
|
}
|
|
288
|
-
|
|
329
|
+
/**
|
|
330
|
+
* @param opts.appLog app-scoped logcat body for the bottom accordion (kept out
|
|
331
|
+
* of RunState / run.json — pass the file contents when writing
|
|
332
|
+
* report.html). The full device dump stays a meta-row file link.
|
|
333
|
+
*/
|
|
334
|
+
function toHtml(run, opts = {}) {
|
|
289
335
|
const c = counts(run);
|
|
290
336
|
const chips = [
|
|
291
337
|
`<span class="chip pass">${c.passed} passed</span>`,
|
|
@@ -301,7 +347,20 @@ function toHtml(run) {
|
|
|
301
347
|
`started ${htmlEsc(run.startedAt)}`,
|
|
302
348
|
run.finishedAt ? `finished ${htmlEsc(run.finishedAt)}` : '',
|
|
303
349
|
run.implicit ? 'implicit run' : '',
|
|
350
|
+
run.logFile ? `<a href="${htmlEsc(run.logFile)}">device log</a>` : '',
|
|
304
351
|
].filter(Boolean);
|
|
352
|
+
// Bottom accordion: app-scoped dump only. The noisy full device log stays a
|
|
353
|
+
// download via the meta link — same shape on every framework (package/uid scope).
|
|
354
|
+
const appLabel = run.appId ? ` for ${htmlEsc(run.appId)}` : '';
|
|
355
|
+
const appFileLink = run.appLogFile
|
|
356
|
+
? ` (<a href="${htmlEsc(run.appLogFile)}">${htmlEsc(run.appLogFile)}</a>)`
|
|
357
|
+
: '';
|
|
358
|
+
const appLogPanel = opts.appLog !== undefined && opts.appLog !== ''
|
|
359
|
+
? `\n <details class="run-log">
|
|
360
|
+
<summary>App log${appLabel}${appFileLink}</summary>
|
|
361
|
+
<pre>${htmlEsc(opts.appLog)}</pre>
|
|
362
|
+
</details>`
|
|
363
|
+
: '';
|
|
305
364
|
return `<!doctype html>
|
|
306
365
|
<html lang="en">
|
|
307
366
|
<head>
|
|
@@ -320,7 +379,7 @@ function toHtml(run) {
|
|
|
320
379
|
${run.ai ? aiPanelHtml(run.ai) : ''}
|
|
321
380
|
<ol class="steps">
|
|
322
381
|
${run.steps.map(stepHtml).join('\n ')}
|
|
323
|
-
</ol
|
|
382
|
+
</ol>${appLogPanel}
|
|
324
383
|
</div>
|
|
325
384
|
</body>
|
|
326
385
|
</html>
|
package/dist/run.js
CHANGED
|
@@ -1,10 +1,14 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
-
exports.Recorder = void 0;
|
|
3
|
+
exports.Recorder = exports.RUN_APP_LOG_ARTIFACT = exports.RUN_LOG_ARTIFACT = void 0;
|
|
4
4
|
exports.isRecordable = isRecordable;
|
|
5
|
+
exports.wantsArchiveLogs = wantsArchiveLogs;
|
|
6
|
+
exports.archiveLogWindow = archiveLogWindow;
|
|
7
|
+
exports.inferRunAppId = inferRunAppId;
|
|
5
8
|
exports.runId = runId;
|
|
6
9
|
exports.uniqueDir = uniqueDir;
|
|
7
10
|
exports.rolloverReason = rolloverReason;
|
|
11
|
+
exports.rolloverLogsSameDevice = rolloverLogsSameDevice;
|
|
8
12
|
exports.stepName = stepName;
|
|
9
13
|
const node_fs_1 = require("node:fs");
|
|
10
14
|
const node_path_1 = require("node:path");
|
|
@@ -32,6 +36,58 @@ function isRecordable(command) {
|
|
|
32
36
|
}
|
|
33
37
|
const HIERARCHY_CAP = 24000; // chars of failure hierarchy kept inline in run.json
|
|
34
38
|
const LOG_CAP = 50000; // chars of device logs kept inline in run.json (tail-kept)
|
|
39
|
+
/** Max chars written to the archive-time `artifacts/logcat.txt` (tail-kept). Larger
|
|
40
|
+
* than the per-step LOG_CAP because the file is not inlined into run.json. */
|
|
41
|
+
const LOG_FILE_CAP = 512_000;
|
|
42
|
+
/** When no session `logStart` marker exists, archive captures this many trailing lines. */
|
|
43
|
+
const ARCHIVE_LOG_LINES = 5000;
|
|
44
|
+
exports.RUN_LOG_ARTIFACT = 'artifacts/logcat.txt';
|
|
45
|
+
exports.RUN_APP_LOG_ARTIFACT = 'artifacts/logcat-app.txt';
|
|
46
|
+
const APP_LIFECYCLE = new Set(['launch', 'open', 'stop', 'clear']);
|
|
47
|
+
/** Whether archive-time device-log capture should run. Default on; `--no-logs` /
|
|
48
|
+
* `VERIKUN_NO_LOGS` opts out of *green* runs only — a failed run always captures
|
|
49
|
+
* (best-effort), because that is when the log is load-bearing. Exported for tests. */
|
|
50
|
+
function wantsArchiveLogs(hasFailures, noLogsFlag = false) {
|
|
51
|
+
if (hasFailures)
|
|
52
|
+
return true;
|
|
53
|
+
if (noLogsFlag)
|
|
54
|
+
return false;
|
|
55
|
+
if (process.env.VERIKUN_NO_LOGS)
|
|
56
|
+
return false;
|
|
57
|
+
return true;
|
|
58
|
+
}
|
|
59
|
+
/** Window passed to `getLogs` at archive time: session-scoped when `logStart` is
|
|
60
|
+
* known, otherwise a bounded trailing dump. Exported for tests. */
|
|
61
|
+
function archiveLogWindow(state) {
|
|
62
|
+
if (state.logStart)
|
|
63
|
+
return { since: state.logStart };
|
|
64
|
+
return { lines: ARCHIVE_LOG_LINES };
|
|
65
|
+
}
|
|
66
|
+
/** Package / bundle id the run exercised, if we can tell. Prefer an explicit
|
|
67
|
+
* `state.appId` (set on launch/open/stop/clear); otherwise recover from those
|
|
68
|
+
* steps' names. Exported for tests. */
|
|
69
|
+
function inferRunAppId(state) {
|
|
70
|
+
if (state.appId && /^[A-Za-z0-9._-]+$/.test(state.appId))
|
|
71
|
+
return state.appId;
|
|
72
|
+
for (let i = state.steps.length - 1; i >= 0; i--) {
|
|
73
|
+
const s = state.steps[i];
|
|
74
|
+
if (!APP_LIFECYCLE.has(s.command))
|
|
75
|
+
continue;
|
|
76
|
+
// stepName is `launch com.foo` / `clear com.foo` (no flags in the name).
|
|
77
|
+
const m = /^(?:launch|open|stop|clear)\s+([A-Za-z0-9._-]+)\s*$/.exec(s.name);
|
|
78
|
+
if (m)
|
|
79
|
+
return m[1];
|
|
80
|
+
}
|
|
81
|
+
return undefined;
|
|
82
|
+
}
|
|
83
|
+
function truncateLogFile(text) {
|
|
84
|
+
if (text.length <= LOG_FILE_CAP)
|
|
85
|
+
return text;
|
|
86
|
+
return '…(truncated)\n' + text.slice(-LOG_FILE_CAP);
|
|
87
|
+
}
|
|
88
|
+
function runHasFailures(state) {
|
|
89
|
+
return state.steps.some((s) => s.status !== 'passed');
|
|
90
|
+
}
|
|
35
91
|
// --- paths & persistence --------------------------------------------------
|
|
36
92
|
const activeDir = () => (0, node_path_1.join)((0, output_1.artifactDir)(), 'run');
|
|
37
93
|
const archiveBase = () => (0, node_path_1.join)((0, output_1.artifactDir)(), 'runs');
|
|
@@ -123,6 +179,14 @@ function rolloverReason(state, serial, session) {
|
|
|
123
179
|
return `idle for ${fmtAge(ageMs(state))} (>${idle}m)`;
|
|
124
180
|
return null;
|
|
125
181
|
}
|
|
182
|
+
/** Whether the incoming step's driver is safe to use for archive-time logs of the
|
|
183
|
+
* run being sealed. On a device-change rollover the driver is already pointed at
|
|
184
|
+
* the *new* serial — pulling logcat from it would attribute the wrong device's
|
|
185
|
+
* output to the old run. Idle/session rollover on the same device is fine.
|
|
186
|
+
* Exported for tests. */
|
|
187
|
+
function rolloverLogsSameDevice(runDevice, newSerial) {
|
|
188
|
+
return !runDevice || !newSerial || runDevice === newSerial;
|
|
189
|
+
}
|
|
126
190
|
function stepName(command, positionals, flags) {
|
|
127
191
|
const p = positionals;
|
|
128
192
|
const at = typeof flags['at'] === 'string' ? flags['at'] : undefined;
|
|
@@ -187,7 +251,13 @@ class Recorder {
|
|
|
187
251
|
const reason = rolloverReason(state, serial, session);
|
|
188
252
|
if (reason) {
|
|
189
253
|
try {
|
|
190
|
-
|
|
254
|
+
// Only reuse this step's driver when it still targets the run's device.
|
|
255
|
+
// A device-change rollover must not write the new device's logcat into
|
|
256
|
+
// the old run's archive (vk run archive binds a driver to state.device).
|
|
257
|
+
const fetchLogs = driver && rolloverLogsSameDevice(state.device, serial)
|
|
258
|
+
? (opts) => driver.getLogs(opts)
|
|
259
|
+
: undefined;
|
|
260
|
+
const dest = Recorder.seal(state, dir, { fetchLogs });
|
|
191
261
|
(0, output_1.err)(`[verikun] previous run '${state.name}' (${state.steps.length} step(s)) auto-closed → ${dest} (${reason}); starting a fresh run`);
|
|
192
262
|
state = null;
|
|
193
263
|
rolledOver = true;
|
|
@@ -220,6 +290,11 @@ class Recorder {
|
|
|
220
290
|
if (!state.session && session)
|
|
221
291
|
state.session = session;
|
|
222
292
|
}
|
|
293
|
+
// Remember the app under test from lifecycle commands so archive can scope
|
|
294
|
+
// the accordion log without requiring a separate flag.
|
|
295
|
+
if (APP_LIFECYCLE.has(command) && positionals[0] && /^[A-Za-z0-9._-]+$/.test(positionals[0])) {
|
|
296
|
+
state.appId = positionals[0];
|
|
297
|
+
}
|
|
223
298
|
// Anchor the log window at the session's first step (covers both implicit
|
|
224
299
|
// creation and an explicit `vk run start`, which records no marker itself).
|
|
225
300
|
// Best-effort — a missing marker just means `vk log` falls back to last-N.
|
|
@@ -386,18 +461,87 @@ class Recorder {
|
|
|
386
461
|
this.state.updatedAt = nowIso();
|
|
387
462
|
saveState(this.dir, this.state);
|
|
388
463
|
}
|
|
389
|
-
/** Finalize a run: write reports next to it, then
|
|
390
|
-
|
|
464
|
+
/** Finalize a run: optionally capture device logs, write reports next to it, then
|
|
465
|
+
* move it into ./.verikun/runs/<id>/. Log capture is best-effort — a device that
|
|
466
|
+
* is gone (often why the run failed) must never prevent sealing the report. */
|
|
467
|
+
static seal(state, dir, opts = {}) {
|
|
468
|
+
Recorder.maybeCaptureArchiveLogs(state, dir, opts);
|
|
391
469
|
state.finishedAt = nowIso();
|
|
392
470
|
state.updatedAt = nowIso();
|
|
393
471
|
saveState(dir, state);
|
|
394
472
|
(0, node_fs_1.writeFileSync)((0, node_path_1.join)(dir, 'report.xml'), (0, report_1.toJUnitXml)(state));
|
|
395
|
-
|
|
473
|
+
// Accordion embeds the *app*-scoped dump (readable); the full device dump stays
|
|
474
|
+
// a file link in the meta row. Missing file is fine — capture is best-effort.
|
|
475
|
+
let appLog;
|
|
476
|
+
if (state.appLogFile) {
|
|
477
|
+
try {
|
|
478
|
+
appLog = (0, node_fs_1.readFileSync)((0, node_path_1.join)(dir, state.appLogFile), 'utf8');
|
|
479
|
+
}
|
|
480
|
+
catch {
|
|
481
|
+
/* report still writes */
|
|
482
|
+
}
|
|
483
|
+
}
|
|
484
|
+
(0, node_fs_1.writeFileSync)((0, node_path_1.join)(dir, 'report.html'), (0, report_1.toHtml)(state, { appLog }));
|
|
396
485
|
(0, node_fs_1.mkdirSync)(archiveBase(), { recursive: true });
|
|
397
486
|
const dest = uniqueDir((0, node_path_1.join)(archiveBase(), state.id));
|
|
398
487
|
(0, node_fs_1.renameSync)(dir, dest);
|
|
399
488
|
return dest;
|
|
400
489
|
}
|
|
490
|
+
/** Write archive-time log artifacts on the ACTIVE run without sealing. Used by
|
|
491
|
+
* remote (`--server`) callers that fetch asynchronously before `archive()`.
|
|
492
|
+
* Pass `full` / `app` only for the pieces you have — omitted sides are left alone. */
|
|
493
|
+
static attachArchiveLogs(full, app) {
|
|
494
|
+
const dir = activeDir();
|
|
495
|
+
const state = loadState(dir);
|
|
496
|
+
if (!state)
|
|
497
|
+
return;
|
|
498
|
+
if (full !== undefined) {
|
|
499
|
+
Recorder.writeArchiveLog(state, dir, full, exports.RUN_LOG_ARTIFACT, 'logFile');
|
|
500
|
+
}
|
|
501
|
+
if (app !== undefined && app !== '') {
|
|
502
|
+
if (!state.appId)
|
|
503
|
+
state.appId = inferRunAppId(state);
|
|
504
|
+
Recorder.writeArchiveLog(state, dir, app, exports.RUN_APP_LOG_ARTIFACT, 'appLogFile');
|
|
505
|
+
}
|
|
506
|
+
saveState(dir, state);
|
|
507
|
+
}
|
|
508
|
+
static writeArchiveLog(state, dir, text, rel, field) {
|
|
509
|
+
const body = truncateLogFile(text);
|
|
510
|
+
(0, node_fs_1.mkdirSync)((0, node_path_1.join)(dir, 'artifacts'), { recursive: true });
|
|
511
|
+
(0, node_fs_1.writeFileSync)((0, node_path_1.join)(dir, rel), body);
|
|
512
|
+
state[field] = rel;
|
|
513
|
+
}
|
|
514
|
+
static maybeCaptureArchiveLogs(state, dir, opts) {
|
|
515
|
+
if (!wantsArchiveLogs(runHasFailures(state), opts.noLogs))
|
|
516
|
+
return;
|
|
517
|
+
if (!opts.fetchLogs)
|
|
518
|
+
return;
|
|
519
|
+
const window = archiveLogWindow(state);
|
|
520
|
+
if (!state.logFile) {
|
|
521
|
+
try {
|
|
522
|
+
const text = opts.fetchLogs(window);
|
|
523
|
+
if (text !== undefined && text !== null) {
|
|
524
|
+
Recorder.writeArchiveLog(state, dir, text, exports.RUN_LOG_ARTIFACT, 'logFile');
|
|
525
|
+
}
|
|
526
|
+
}
|
|
527
|
+
catch (e) {
|
|
528
|
+
(0, output_1.err)(`[verikun] could not capture archive device logs (${e.message})`);
|
|
529
|
+
}
|
|
530
|
+
}
|
|
531
|
+
const appId = inferRunAppId(state);
|
|
532
|
+
if (appId)
|
|
533
|
+
state.appId = appId;
|
|
534
|
+
if (appId && !state.appLogFile) {
|
|
535
|
+
try {
|
|
536
|
+
const text = opts.fetchLogs({ ...window, appId, scopedOnly: true });
|
|
537
|
+
if (text)
|
|
538
|
+
Recorder.writeArchiveLog(state, dir, text, exports.RUN_APP_LOG_ARTIFACT, 'appLogFile');
|
|
539
|
+
}
|
|
540
|
+
catch (e) {
|
|
541
|
+
(0, output_1.err)(`[verikun] could not capture archive app logs (${e.message})`);
|
|
542
|
+
}
|
|
543
|
+
}
|
|
544
|
+
}
|
|
401
545
|
/** One-line context summary for `vk run status`. */
|
|
402
546
|
static contextLine(state) {
|
|
403
547
|
const bits = [];
|
|
@@ -428,6 +572,8 @@ class Recorder {
|
|
|
428
572
|
* a local one. Re-indexes the step, writes its artifact buffers under the new
|
|
429
573
|
* index, and rewrites the step's artifact references to match. Creates an
|
|
430
574
|
* implicit run first if none is active (parity with beginStep's auto-start).
|
|
575
|
+
* `ctx.logStart` (device-clock marker from the server) is recorded on the first
|
|
576
|
+
* splice so archive-time / `vk log` scoping works the same as a local run.
|
|
431
577
|
*/
|
|
432
578
|
static appendForeignStep(step, artifacts = {}, ctx = {}) {
|
|
433
579
|
if (process.env.VERIKUN_NO_RUN)
|
|
@@ -448,6 +594,17 @@ class Recorder {
|
|
|
448
594
|
};
|
|
449
595
|
(0, output_1.err)('[verikun] recording test run (implicit) — archive: `vk run archive` · discard: `vk run clear`');
|
|
450
596
|
}
|
|
597
|
+
// Anchor the log window from the server's device clock (beginEphemeralStep
|
|
598
|
+
// never persists state, so the marker has to travel on the wire).
|
|
599
|
+
if (!state.logStart && ctx.logStart)
|
|
600
|
+
state.logStart = ctx.logStart;
|
|
601
|
+
// Mirror local beginStep: remember the app from lifecycle steps so archive
|
|
602
|
+
// can scope the accordion without a local launch having set state.appId.
|
|
603
|
+
if (APP_LIFECYCLE.has(step.command)) {
|
|
604
|
+
const m = /^(?:launch|open|stop|clear)\s+([A-Za-z0-9._-]+)\s*$/.exec(step.name);
|
|
605
|
+
if (m)
|
|
606
|
+
state.appId = m[1];
|
|
607
|
+
}
|
|
451
608
|
const index = state.steps.length;
|
|
452
609
|
const spliced = { ...step, index };
|
|
453
610
|
for (const [rel, buf] of Object.entries(artifacts)) {
|
|
@@ -521,8 +678,10 @@ class Recorder {
|
|
|
521
678
|
(0, node_fs_1.rmSync)(dir, { recursive: true, force: true });
|
|
522
679
|
return existing;
|
|
523
680
|
}
|
|
524
|
-
/** Write JUnit + HTML reports and move the run into ./.verikun/runs/<id>/.
|
|
525
|
-
|
|
681
|
+
/** Write JUnit + HTML reports and move the run into ./.verikun/runs/<id>/.
|
|
682
|
+
* By default captures a bounded device-log tail into `artifacts/logcat.txt`
|
|
683
|
+
* (see wantsArchiveLogs / ArchiveLogOpts). */
|
|
684
|
+
static archive(name, opts = {}) {
|
|
526
685
|
const dir = activeDir();
|
|
527
686
|
if (!(0, node_fs_1.existsSync)(statePath(dir))) {
|
|
528
687
|
throw new errors_1.CliError('No active test run to archive. Run an action first, or `vk run start`.', 1);
|
|
@@ -532,7 +691,7 @@ class Recorder {
|
|
|
532
691
|
throw new errors_1.CliError('Active run state is unreadable (.verikun/run/run.json is corrupt).', 3);
|
|
533
692
|
if (name)
|
|
534
693
|
state.name = name;
|
|
535
|
-
const dest = Recorder.seal(state, dir);
|
|
694
|
+
const dest = Recorder.seal(state, dir, opts);
|
|
536
695
|
return { dir: dest, xmlPath: (0, node_path_1.join)(dest, 'report.xml'), htmlPath: (0, node_path_1.join)(dest, 'report.html'), state };
|
|
537
696
|
}
|
|
538
697
|
}
|
package/dist/server.js
CHANGED
|
@@ -11,6 +11,7 @@
|
|
|
11
11
|
// through the SAME validateNode gate that guards `vk ai` model repairs, so only
|
|
12
12
|
// KNOWN_COMMANDS action verbs execute — never `ui`/`log`, never a shell. The
|
|
13
13
|
// server's driver is fixed at startup; client flags can never repoint the device.
|
|
14
|
+
// Archive-time log capture uses the dedicated /v1/logs endpoint instead.
|
|
14
15
|
// - /v1/install is a privileged management verb: auth PLUS --allow-install, body
|
|
15
16
|
// streamed to a server-generated temp path (the client supplies only an
|
|
16
17
|
// allowlisted extension — never a path), optional sha256 verification.
|
|
@@ -154,13 +155,14 @@ function buildServer(config) {
|
|
|
154
155
|
if (node.type !== 'command')
|
|
155
156
|
throw new HttpError(400, 'rejected: not a command leaf');
|
|
156
157
|
const t0 = Date.now();
|
|
157
|
-
const { code, error, step, artifacts } = await (0, cli_1.executeForServer)(node.command, node.positionals, (0, ir_1.leafToFlags)(node), config.driver, config.platform);
|
|
158
|
+
const { code, error, step, artifacts, logStart } = await (0, cli_1.executeForServer)(node.command, node.positionals, (0, ir_1.leafToFlags)(node), config.driver, config.platform);
|
|
158
159
|
(0, output_1.err)(`[server] exec ${node.command} ${node.positionals.join(' ')} → exit ${code} (${Date.now() - t0}ms)`);
|
|
159
160
|
const payload = {
|
|
160
161
|
code,
|
|
161
162
|
...(error ? { error: (0, rpc_1.describeError)(error) } : {}),
|
|
162
163
|
...(step ? { step } : {}),
|
|
163
164
|
...(artifacts && Object.keys(artifacts).length ? { artifacts: encodeArtifacts(artifacts) } : {}),
|
|
165
|
+
...(logStart ? { logStart } : {}),
|
|
164
166
|
};
|
|
165
167
|
sendJson(res, 200, payload);
|
|
166
168
|
}
|
|
@@ -169,6 +171,45 @@ function buildServer(config) {
|
|
|
169
171
|
const elements = config.driver.getElements(); // CliError(3) on dump failure → 500 below
|
|
170
172
|
sendJson(res, 200, { elements });
|
|
171
173
|
}
|
|
174
|
+
async function handleLogs(req, res) {
|
|
175
|
+
const body = await readBody(req, EXEC_BODY_CAP);
|
|
176
|
+
let parsed = {};
|
|
177
|
+
if (body.length) {
|
|
178
|
+
try {
|
|
179
|
+
parsed = JSON.parse(body.toString('utf8'));
|
|
180
|
+
}
|
|
181
|
+
catch {
|
|
182
|
+
throw new HttpError(400, 'invalid JSON body');
|
|
183
|
+
}
|
|
184
|
+
}
|
|
185
|
+
// Mirror the driver's --since charset gate so a remote caller cannot inject
|
|
186
|
+
// into the device shell via a crafted marker (see AdbDriver.getLogs).
|
|
187
|
+
if (parsed.since !== undefined && parsed.since !== null) {
|
|
188
|
+
if (typeof parsed.since !== 'string' || !/^[0-9 :.\-]+$/.test(parsed.since)) {
|
|
189
|
+
throw new HttpError(400, `invalid since: only a logcat timestamp (digits, space, '-', ':', '.') is allowed`);
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
const lines = parsed.lines === undefined || parsed.lines === null
|
|
193
|
+
? undefined
|
|
194
|
+
: typeof parsed.lines === 'number' && Number.isFinite(parsed.lines) && parsed.lines > 0
|
|
195
|
+
? Math.floor(parsed.lines)
|
|
196
|
+
: undefined;
|
|
197
|
+
const appId = parsed.appId === undefined || parsed.appId === null
|
|
198
|
+
? undefined
|
|
199
|
+
: typeof parsed.appId === 'string' && /^[A-Za-z0-9_.-]+$/.test(parsed.appId)
|
|
200
|
+
? parsed.appId
|
|
201
|
+
: (() => {
|
|
202
|
+
throw new HttpError(400, `invalid appId '${String(parsed.appId)}'`);
|
|
203
|
+
})();
|
|
204
|
+
const logs = config.driver.getLogs({
|
|
205
|
+
...(lines !== undefined ? { lines } : {}),
|
|
206
|
+
...(parsed.since ? { since: parsed.since } : {}),
|
|
207
|
+
...(appId ? { appId } : {}),
|
|
208
|
+
...(parsed.scopedOnly ? { scopedOnly: true } : {}),
|
|
209
|
+
});
|
|
210
|
+
const payload = { logs };
|
|
211
|
+
sendJson(res, 200, payload);
|
|
212
|
+
}
|
|
172
213
|
async function handleInstall(req, res) {
|
|
173
214
|
const ext = String(req.headers['x-verikun-ext'] ?? '').toLowerCase();
|
|
174
215
|
if (ext !== 'apk' && ext !== 'ipa') {
|
|
@@ -256,6 +297,8 @@ function buildServer(config) {
|
|
|
256
297
|
return deviceEndpoint(() => handleExec(req, res));
|
|
257
298
|
if (req.method === 'POST' && path === '/v1/elements')
|
|
258
299
|
return deviceEndpoint(() => handleElements(req, res));
|
|
300
|
+
if (req.method === 'POST' && path === '/v1/logs')
|
|
301
|
+
return deviceEndpoint(() => handleLogs(req, res));
|
|
259
302
|
if (req.method === 'POST' && path === '/v1/install') {
|
|
260
303
|
if (!config.allowInstall) {
|
|
261
304
|
throw new HttpError(403, 'install is disabled on this server (start it with --allow-install)', 3);
|
package/dist/suite.js
CHANGED
|
@@ -13,6 +13,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
13
13
|
exports.sortTestFiles = sortTestFiles;
|
|
14
14
|
exports.listTestFiles = listTestFiles;
|
|
15
15
|
exports.toSuiteResult = toSuiteResult;
|
|
16
|
+
exports.toSuiteAttempt = toSuiteAttempt;
|
|
17
|
+
exports.mergeSuiteAttempts = mergeSuiteAttempts;
|
|
16
18
|
exports.cmdSuite = cmdSuite;
|
|
17
19
|
const node_fs_1 = require("node:fs");
|
|
18
20
|
const node_path_1 = require("node:path");
|
|
@@ -100,6 +102,64 @@ function toSuiteResult(file, r, durationMs) {
|
|
|
100
102
|
...(r.ok ? {} : { failure: failure ?? 'failed' }),
|
|
101
103
|
};
|
|
102
104
|
}
|
|
105
|
+
/** Compact one attempt for the `attempts` evidence array (pure). */
|
|
106
|
+
function toSuiteAttempt(r) {
|
|
107
|
+
return {
|
|
108
|
+
id: r.id,
|
|
109
|
+
ok: r.ok,
|
|
110
|
+
durationMs: r.durationMs,
|
|
111
|
+
costUsd: r.costUsd,
|
|
112
|
+
...(r.failure ? { failure: r.failure } : {}),
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
/**
|
|
116
|
+
* Merge a sequence of attempt rows into the final suite row: primary `id` is the last
|
|
117
|
+
* attempt (winning green, or last red), cost/duration/repairs sum across attempts, and
|
|
118
|
+
* prior attempts are retained as flake evidence.
|
|
119
|
+
*/
|
|
120
|
+
function mergeSuiteAttempts(attempts) {
|
|
121
|
+
if (attempts.length === 0)
|
|
122
|
+
throw new Error('mergeSuiteAttempts: empty');
|
|
123
|
+
const last = attempts[attempts.length - 1];
|
|
124
|
+
if (attempts.length === 1)
|
|
125
|
+
return last;
|
|
126
|
+
const round = (n) => Number(n.toFixed(4));
|
|
127
|
+
const prior = attempts.slice(0, -1).map(toSuiteAttempt);
|
|
128
|
+
const flaky = last.ok && prior.some((a) => !a.ok);
|
|
129
|
+
return {
|
|
130
|
+
...last,
|
|
131
|
+
durationMs: attempts.reduce((a, t) => a + t.durationMs, 0),
|
|
132
|
+
costUsd: round(attempts.reduce((a, t) => a + t.costUsd, 0)),
|
|
133
|
+
modelRepairs: attempts.reduce((a, t) => a + t.modelRepairs, 0),
|
|
134
|
+
attempts: prior,
|
|
135
|
+
...(flaky ? { flaky: true } : {}),
|
|
136
|
+
};
|
|
137
|
+
}
|
|
138
|
+
// What --retries will and won't spend an attempt on. The bias is deliberate and
|
|
139
|
+
// asymmetric: a retry costs one test, while giving up costs the whole suite plus a
|
|
140
|
+
// human rerunning it. So the rule is *retry unless a rerun provably cannot change the
|
|
141
|
+
// outcome* — the two predicates below are the only "provably" cases, everything else
|
|
142
|
+
// (flaky selector, wedged app, a wobbling network to `vk server`) earns another go.
|
|
143
|
+
/** Budget aborts won't heal on retry: each attempt gets its own cost ceiling, so a
|
|
144
|
+
* rerun just re-aborts at the same place having spent the money twice. */
|
|
145
|
+
function isRetryable(r) {
|
|
146
|
+
return !r.ok && !r.abortedForBudget;
|
|
147
|
+
}
|
|
148
|
+
/** A thrown USAGE error (exit 2) is the one throw a rerun cannot change — an unreadable
|
|
149
|
+
* test file, a payload the server refuses, a flag it doesn't understand. Everything
|
|
150
|
+
* else, including every environment error, is retried while attempts remain. */
|
|
151
|
+
function isRetryableThrow(e) {
|
|
152
|
+
return !(e instanceof errors_1.CliError && e.exitCode === 2);
|
|
153
|
+
}
|
|
154
|
+
function parseRetries(flags) {
|
|
155
|
+
const n = (0, args_1.flagNum)(flags, 'retries');
|
|
156
|
+
if (n === undefined)
|
|
157
|
+
return 0;
|
|
158
|
+
if (!Number.isInteger(n) || n < 0) {
|
|
159
|
+
throw new errors_1.CliError(`--retries must be a non-negative integer, got '${n}'`, 2);
|
|
160
|
+
}
|
|
161
|
+
return n;
|
|
162
|
+
}
|
|
103
163
|
async function cmdSuite(dirArg, flags, deps) {
|
|
104
164
|
const dir = (0, node_path_1.resolve)(process.cwd(), dirArg);
|
|
105
165
|
if (!(0, node_fs_1.existsSync)(dir) || !(0, node_fs_1.statSync)(dir).isDirectory()) {
|
|
@@ -109,69 +169,133 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
109
169
|
if (files.length === 0) {
|
|
110
170
|
throw new errors_1.CliError(`suite: no test files (*.md) in '${dirArg}'`, 2);
|
|
111
171
|
}
|
|
172
|
+
const retries = parseRetries(flags);
|
|
112
173
|
const suiteId = (0, run_1.runId)();
|
|
113
174
|
const name = (0, args_1.flagStr)(flags, 'name') || (0, node_path_1.basename)(dir);
|
|
114
175
|
const startedAt = new Date().toISOString();
|
|
115
|
-
(0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})`);
|
|
176
|
+
(0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})${retries > 0 ? ` · up to ${retries} retry(ies) on failure` : ''}`);
|
|
116
177
|
const results = [];
|
|
178
|
+
const warnings = [];
|
|
117
179
|
let aborted;
|
|
180
|
+
async function resetApp(label) {
|
|
181
|
+
// Returns the abort reason when the suite should stop (confirmed env break during reset).
|
|
182
|
+
if (!deps.reset)
|
|
183
|
+
return undefined;
|
|
184
|
+
try {
|
|
185
|
+
await deps.reset();
|
|
186
|
+
(0, output_1.err)(`[suite] app state reset${label}`);
|
|
187
|
+
return undefined;
|
|
188
|
+
}
|
|
189
|
+
catch (e) {
|
|
190
|
+
// A reset that failed because the BOX is broken means nothing after it is
|
|
191
|
+
// trustworthy — but only if a re-probe agrees. Otherwise surface and continue:
|
|
192
|
+
// a flaky reset should not zero out the whole suite, and the test itself will
|
|
193
|
+
// fail loudly if the stale state actually matters.
|
|
194
|
+
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
195
|
+
if (broken)
|
|
196
|
+
return broken;
|
|
197
|
+
(0, output_1.err)(`[suite] reset failed (${e.message}) — continuing`);
|
|
198
|
+
return undefined;
|
|
199
|
+
}
|
|
200
|
+
}
|
|
201
|
+
/** A confirmed env break with attempts left: say so, pause, and let the loop retry.
|
|
202
|
+
* The pause matters — the failures this rides out (a server restart, a wifi drop, a
|
|
203
|
+
* USB re-enumeration) clear in seconds, and retrying into the same dead socket
|
|
204
|
+
* immediately would burn every attempt inside the outage. */
|
|
205
|
+
async function noteEnvRetry(file, attempt, reason) {
|
|
206
|
+
const warn = `${file}: environment error on attempt ${attempt + 1} (${reason}) — retried`;
|
|
207
|
+
warnings.push(warn);
|
|
208
|
+
(0, output_1.err)(`[suite] WARN ${warn}`);
|
|
209
|
+
await sleep((deps.probeRetryMs ?? PROBE_RETRY_MS) * (attempt + 1));
|
|
210
|
+
}
|
|
118
211
|
for (let i = 0; i < files.length && !aborted; i++) {
|
|
119
212
|
const file = files[i];
|
|
120
213
|
(0, output_1.err)(`[suite] ── (${i + 1}/${files.length}) ${file} ──`);
|
|
121
|
-
|
|
214
|
+
const attemptRows = [];
|
|
215
|
+
for (let attempt = 0; attempt <= retries; attempt++) {
|
|
216
|
+
// The last attempt is where a retryable failure becomes the verdict: a confirmed
|
|
217
|
+
// env break aborts the suite, anything else stands as this test's failed row.
|
|
218
|
+
const lastAttempt = attempt === retries;
|
|
219
|
+
if (attempt > 0)
|
|
220
|
+
(0, output_1.err)(`[suite] retry ${attempt}/${retries} for ${file}`);
|
|
221
|
+
// Re-isolate before EVERY attempt — between tests and between retries alike.
|
|
222
|
+
const resetBreak = await resetApp(attempt > 0 ? ' (retry)' : '');
|
|
223
|
+
if (resetBreak) {
|
|
224
|
+
if (!lastAttempt) {
|
|
225
|
+
await noteEnvRetry(file, attempt, `reset failed: ${resetBreak}`);
|
|
226
|
+
continue;
|
|
227
|
+
}
|
|
228
|
+
// With no attempt row this test never ran, so notRun starts at the CURRENT file.
|
|
229
|
+
aborted = {
|
|
230
|
+
reason: `reset failed: ${resetBreak}`,
|
|
231
|
+
notRun: files.slice(attemptRows.length ? i + 1 : i),
|
|
232
|
+
};
|
|
233
|
+
break;
|
|
234
|
+
}
|
|
235
|
+
const t0 = Date.now();
|
|
122
236
|
try {
|
|
123
|
-
await deps.
|
|
124
|
-
(
|
|
237
|
+
const r = await deps.runTest((0, node_path_1.join)(dir, file));
|
|
238
|
+
attemptRows.push(toSuiteResult(file, r, Date.now() - t0));
|
|
239
|
+
if (r.abortedForEnv) {
|
|
240
|
+
const broken = await stillBroken(deps);
|
|
241
|
+
if (broken) {
|
|
242
|
+
if (!lastAttempt) {
|
|
243
|
+
// Even a CONFIRMED break is worth an attempt: the probe window is a couple
|
|
244
|
+
// of seconds, which a server restart outlives — and aborting costs the run.
|
|
245
|
+
await noteEnvRetry(file, attempt, broken);
|
|
246
|
+
continue;
|
|
247
|
+
}
|
|
248
|
+
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
249
|
+
break;
|
|
250
|
+
}
|
|
251
|
+
// Transient env blip: retryable like any other failure.
|
|
252
|
+
}
|
|
253
|
+
if (r.ok || !isRetryable(r) || lastAttempt)
|
|
254
|
+
break;
|
|
125
255
|
}
|
|
126
256
|
catch (e) {
|
|
127
|
-
// A
|
|
128
|
-
//
|
|
129
|
-
//
|
|
130
|
-
|
|
257
|
+
// A test that THREW (device gone, server unreachable, bad file) still becomes a
|
|
258
|
+
// failed row — one broken test must not vaporize the suite report for the tests
|
|
259
|
+
// that already ran. Out of attempts, a confirmed env break stops the suite.
|
|
260
|
+
const msg = e instanceof Error ? e.message : String(e);
|
|
261
|
+
(0, output_1.err)(`[suite] ${file} errored: ${msg}`);
|
|
262
|
+
attemptRows.push({
|
|
263
|
+
id: '',
|
|
264
|
+
file,
|
|
265
|
+
name: (0, node_path_1.basename)(file, (0, node_path_1.extname)(file)),
|
|
266
|
+
ok: false,
|
|
267
|
+
durationMs: Date.now() - t0,
|
|
268
|
+
costUsd: 0,
|
|
269
|
+
steps: 0,
|
|
270
|
+
passedSteps: 0,
|
|
271
|
+
failedSteps: 0,
|
|
272
|
+
modelRepairs: 0,
|
|
273
|
+
failure: msg.split('\n')[0],
|
|
274
|
+
});
|
|
131
275
|
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
132
|
-
if (
|
|
133
|
-
|
|
134
|
-
|
|
276
|
+
if (lastAttempt) {
|
|
277
|
+
if (broken)
|
|
278
|
+
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
135
279
|
break;
|
|
136
280
|
}
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
}
|
|
140
|
-
const t0 = Date.now();
|
|
141
|
-
try {
|
|
142
|
-
const r = await deps.runTest((0, node_path_1.join)(dir, file));
|
|
143
|
-
results.push(toSuiteResult(file, r, Date.now() - t0));
|
|
144
|
-
// The test itself reported an environment abort (exit 3 mid-plan). Same rule:
|
|
145
|
-
// fatal only if the box is still broken. This test HAS a row and a real report,
|
|
146
|
-
// so notRun starts after it.
|
|
147
|
-
if (r.abortedForEnv) {
|
|
148
|
-
const broken = await stillBroken(deps);
|
|
281
|
+
if (!isRetryableThrow(e))
|
|
282
|
+
break;
|
|
149
283
|
if (broken)
|
|
150
|
-
|
|
284
|
+
await noteEnvRetry(file, attempt, broken);
|
|
151
285
|
}
|
|
152
286
|
}
|
|
153
|
-
|
|
154
|
-
//
|
|
155
|
-
//
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
costUsd: 0,
|
|
166
|
-
steps: 0,
|
|
167
|
-
passedSteps: 0,
|
|
168
|
-
failedSteps: 0,
|
|
169
|
-
modelRepairs: 0,
|
|
170
|
-
failure: msg.split('\n')[0],
|
|
171
|
-
});
|
|
172
|
-
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
173
|
-
if (broken)
|
|
174
|
-
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
287
|
+
if (attemptRows.length === 0) {
|
|
288
|
+
// Every attempt was blocked by a failing reset, so the test never ran and gets no
|
|
289
|
+
// row — `aborted.notRun` (set above) already names it. Nothing to merge.
|
|
290
|
+
break;
|
|
291
|
+
}
|
|
292
|
+
const merged = mergeSuiteAttempts(attemptRows);
|
|
293
|
+
results.push(merged);
|
|
294
|
+
if (merged.flaky) {
|
|
295
|
+
const n = merged.attempts?.length ?? 0;
|
|
296
|
+
const warn = `${file} passed on retry after ${n} failed attempt${n === 1 ? '' : 's'}`;
|
|
297
|
+
warnings.push(warn);
|
|
298
|
+
(0, output_1.err)(`[suite] WARN ${warn}`);
|
|
175
299
|
}
|
|
176
300
|
}
|
|
177
301
|
if (aborted) {
|
|
@@ -189,6 +313,7 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
189
313
|
totals: (0, report_1.suiteTotals)(results),
|
|
190
314
|
tests: results,
|
|
191
315
|
...(aborted ? { aborted } : {}),
|
|
316
|
+
...(warnings.length ? { warnings } : {}),
|
|
192
317
|
};
|
|
193
318
|
// .verikun/suites/<id>/ sits beside .verikun/runs/<id>/, so index.html reaches a
|
|
194
319
|
// test report at ../../runs/<id>/report.html — the linkBase below.
|
|
@@ -198,8 +323,12 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
198
323
|
(0, node_fs_1.writeFileSync)((0, node_path_1.join)(outDir, 'index.html'), (0, report_1.toSuiteHtml)(suite, { linkBase: '../../' }));
|
|
199
324
|
const t = suite.totals;
|
|
200
325
|
(0, output_1.err)(`[suite] ${t.passed}/${t.tests} passed · ${t.steps} steps · $${t.costUsd.toFixed(4)} · ${(t.durationMs / 1000).toFixed(1)}s`);
|
|
201
|
-
for (const r of results)
|
|
202
|
-
|
|
326
|
+
for (const r of results) {
|
|
327
|
+
const tag = r.flaky ? 'FLAKY' : r.ok ? 'PASS' : 'FAIL';
|
|
328
|
+
(0, output_1.err)(` ${tag} ${r.file}${r.failure ? ` — ${r.failure}` : r.flaky ? ' — passed on retry' : ''}`);
|
|
329
|
+
}
|
|
330
|
+
if (warnings.length)
|
|
331
|
+
(0, output_1.err)(`[suite] ${warnings.length} warning(s)`);
|
|
203
332
|
(0, output_1.err)(`[suite] overview: ${(0, node_path_1.join)(outDir, 'index.html')}`);
|
|
204
333
|
if ((0, args_1.flagBool)(flags, 'json'))
|
|
205
334
|
(0, output_1.json)(suite);
|
|
@@ -207,6 +336,7 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
207
336
|
(0, output_1.out)(outDir); // primary machine result: the suite directory
|
|
208
337
|
// The CI gate: any failed test fails the invocation (mirrors `vk run archive`). An
|
|
209
338
|
// environment abort exits 3 instead, so CI can tell "the runner is broken" from "the
|
|
210
|
-
// app regressed" — the whole point of stopping early.
|
|
339
|
+
// app regressed" — the whole point of stopping early. A flake that recovered is ok
|
|
340
|
+
// (exit 0) with a warning — that is the whole point of --retries.
|
|
211
341
|
return aborted ? 3 : t.failed > 0 ? 1 : 0;
|
|
212
342
|
}
|
package/dist/version.js
CHANGED
|
@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
3
3
|
exports.VERSION = void 0;
|
|
4
4
|
// GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
|
|
5
5
|
// (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
|
|
6
|
-
exports.VERSION = '0.
|
|
6
|
+
exports.VERSION = '0.18.0';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "verikun",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.18.0",
|
|
4
4
|
"description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"android",
|