verikun 0.15.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +79 -22
- package/dist/agent/engine.js +8 -0
- package/dist/agent/grammar.js +17 -0
- package/dist/args.js +13 -0
- package/dist/cli.js +65 -17
- package/dist/report.js +34 -3
- package/dist/suite.js +179 -49
- package/dist/ui/ios-parse.js +19 -0
- package/dist/ui/selector.js +81 -18
- package/dist/ui/state-support.js +41 -0
- package/dist/version.js +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -104,7 +104,7 @@ vk screenshot # -> ./.verikun/screen.png
|
|
|
104
104
|
| Command | Description |
|
|
105
105
|
|---|---|
|
|
106
106
|
| `ai <file> [--model m] [--max-cost-usd n] [--timeout dur] [--cost-override in/out] [--effort e] [--package pkg] [--app-build id] [--server url] [--show-plan] [--recompile] [--json]` | Run a plain-English test: compile it to a deterministic plan once, replay it model-free, and self-heal failures via the model. Needs `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` (per model), or no key with `--model codex-cli` / `cursor-cli` (a logged-in `codex` / `cursor-agent` CLI). See [AI](#ai--natural-language-tests). |
|
|
107
|
-
| `suite <dir> [--app <id>] [--name n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
|
|
107
|
+
| `suite <dir> [--app <id>] [--name n] [--retries n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
|
|
108
108
|
|
|
109
109
|
### Remote
|
|
110
110
|
| Command | Description |
|
|
@@ -287,12 +287,29 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER" # remote devi
|
|
|
287
287
|
- **Each test is a full `vk ai` run** — plan cache, self-healing, cost budget, and
|
|
288
288
|
its own archived JUnit + HTML report under `./.verikun/runs/<id>/`. A test that
|
|
289
289
|
fails (or errors) doesn't stop the suite; the rest still run.
|
|
290
|
-
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
290
|
+
- **`--retries N` recovers from flakes.** A failed test is re-run up to N times
|
|
291
|
+
(default `0` — opt-in, so CI cost/time stay predictable). If a later attempt
|
|
292
|
+
passes, the suite exits `0` and the flake is a **warning**, not a hard failure.
|
|
293
|
+
Failed attempt archives stay linked from the suite overview (`attempts` on the
|
|
294
|
+
test row + a `warnings` list on the manifest), so flakiness remains visible.
|
|
295
|
+
Cost and duration sum across attempts.
|
|
296
|
+
- **What earns a retry:** anything that might come out differently — a flaky
|
|
297
|
+
selector, a wedged app, and **a broken environment**, including a `vk server`
|
|
298
|
+
connection dropping mid-suite. The bias is intentional: an attempt costs one test,
|
|
299
|
+
giving up costs the whole suite plus a human rerunning it. Environment retries wait
|
|
300
|
+
a little longer each time (an outage that survives the health probe usually needs
|
|
301
|
+
seconds, not milliseconds) and each one lands in `warnings`, so riding out a wobble
|
|
302
|
+
is never silent. Exactly two failures are never retried, because a rerun cannot
|
|
303
|
+
change them: a **budget abort** (each attempt gets its own ceiling, so it would just
|
|
304
|
+
re-abort having spent twice) and a **usage error** (exit `2` — an unreadable test
|
|
305
|
+
file, a payload the server refuses).
|
|
306
|
+
- **But a broken *environment* does stop it, once the attempts are gone.** If a test
|
|
307
|
+
dies from an environment error (exit 3 — tool gone, device unplugged, server
|
|
308
|
+
unreachable), the toolchain is re-probed; only if it is *still* broken **and** no
|
|
309
|
+
retries remain does the suite abort. That re-probe matters: a transient
|
|
310
|
+
`uiautomator` dump failure also exits 3, and shouldn't vaporize a 20-test run.
|
|
311
|
+
Continuing on a genuinely dead box just produces one identical red row per remaining
|
|
312
|
+
test — noise that reads exactly like a mass regression.
|
|
296
313
|
- **The suite writes an overview** to `./.verikun/suites/<id>/`:
|
|
297
314
|
- **`index.json`** — a stable, `schemaVersion`ed manifest: per-test pass/fail,
|
|
298
315
|
steps, model repairs, cost, duration, and the run id, plus suite totals. This
|
|
@@ -300,17 +317,21 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER" # remote devi
|
|
|
300
317
|
it (see the [CI recipe](#ci-recipe)) instead of verikun growing upload plugins.
|
|
301
318
|
On an abort it also carries `aborted: {reason, notRun}`; the not-run tests get
|
|
302
319
|
**no rows and no place in `totals`**, so `passed + failed === tests` still holds
|
|
303
|
-
and nothing downstream mistakes a skipped test for a regression.
|
|
320
|
+
and nothing downstream mistakes a skipped test for a regression. Retried flakes
|
|
321
|
+
add `flaky` / `attempts` on the test row and suite-level `warnings` (additive;
|
|
322
|
+
`schemaVersion` stays `1`).
|
|
304
323
|
- **`index.html`** — a summary page linking every test's `report.html`, with a
|
|
305
|
-
banner naming the not-run tests when the suite aborted
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
324
|
+
banner naming the not-run tests when the suite aborted, and a warnings banner
|
|
325
|
+
when a flake recovered on retry (prior failed attempts stay linked).
|
|
326
|
+
- **Exit code is the CI gate:** `0` all green (including flakes that recovered with
|
|
327
|
+
`--retries`) · `1` a test failed · `2` bad/empty directory · `3` environment (the
|
|
328
|
+
provider or the device toolchain is unavailable, or the box broke mid-run). The
|
|
329
|
+
`1`-vs-`3` split is the point: `1` is a regression to investigate, `3` is a
|
|
330
|
+
machine to fix. All `ai` flags (`--model`, `--max-cost-usd`, `--timeout`, …)
|
|
331
|
+
apply to every test; both the provider (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`,
|
|
332
|
+
or the `codex` / `cursor-agent` CLI for `--model codex-cli` / `cursor-cli`)
|
|
333
|
+
**and** the device toolchain (`adb` / `idb` + a resolvable device) are checked up
|
|
334
|
+
front, before anything is compiled.
|
|
314
335
|
|
|
315
336
|
## Remote devices — `vk server`
|
|
316
337
|
|
|
@@ -390,15 +411,51 @@ class:Button simplified type ("Button") or full class ("android.widget.Button
|
|
|
390
411
|
```
|
|
391
412
|
|
|
392
413
|
Modifiers: `--contains` makes text/desc matches substring-based; `--index N`
|
|
393
|
-
selects the Nth match (0-based) when a selector intentionally matches several
|
|
394
|
-
`--enabled` matches only a control that is **actionable right now** — use it for a
|
|
395
|
-
Submit/Check button the app disables until a form is valid, since such a button is
|
|
396
|
-
present long before it is usable and tapping presence taps a dead control (with
|
|
397
|
-
auto-wait this reads as "wait until it is pressable").
|
|
414
|
+
selects the Nth match (0-based) when a selector intentionally matches several.
|
|
398
415
|
If a selector for an action matches more than one element and no `--index` is
|
|
399
416
|
given, the command fails with exit code 2 and lists the candidates — it never
|
|
400
417
|
taps a guess.
|
|
401
418
|
|
|
419
|
+
### State modifiers
|
|
420
|
+
|
|
421
|
+
A selector can also require an element's a11y **state**, in both polarities:
|
|
422
|
+
|
|
423
|
+
| Modifier | Matches | Negative form |
|
|
424
|
+
|---|---|---|
|
|
425
|
+
| `--enabled` | actionable right now | `--not-enabled` |
|
|
426
|
+
| `--selected` | the current option of a segmented control / tab bar / mode picker | `--not-selected` |
|
|
427
|
+
| `--checked` | a ticked checkbox / switch / radio | `--not-checked` |
|
|
428
|
+
| `--focused` | the element holding input focus | `--not-focused` |
|
|
429
|
+
|
|
430
|
+
Unset means *don't care*; these never narrow a selector you didn't ask them to.
|
|
431
|
+
|
|
432
|
+
Reach for `--enabled` on a Submit/Check button the app disables until a form is
|
|
433
|
+
valid: such a button is present long before it is usable, so tapping presence taps
|
|
434
|
+
a dead control (with auto-wait this reads as "wait until it is pressable").
|
|
435
|
+
|
|
436
|
+
The **negative** forms are what make a toggle drivable. A segmented control whose
|
|
437
|
+
options share one handler *flips* on any tap, so "tap the option I want" lands on
|
|
438
|
+
the other one whenever it was already chosen — exit 0, nothing to notice, and the
|
|
439
|
+
run exercises the wrong mode. Guard it instead:
|
|
440
|
+
|
|
441
|
+
```sh
|
|
442
|
+
vk find "@mode_video --not-selected" --no-wait && vk tap @mode_video
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
A modifier can be written as a flag **or appended to the selector string**, as
|
|
446
|
+
above. The string form exists because a `vk ai` control node (`if-present`,
|
|
447
|
+
`when`, `repeat`, `while-present`, `read`) holds a bare selector with nowhere to
|
|
448
|
+
put a flag — and a guard is exactly where the toggle case needs one:
|
|
449
|
+
|
|
450
|
+
```
|
|
451
|
+
if-present "id:mode_video --not-selected" { tap id:mode_video }
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
**`--selected` and `--focused` are Android-only.** `idb` reports no such
|
|
455
|
+
attribute for iOS — not merely unset, the key does not exist in its output — so
|
|
456
|
+
using them with `--ios` exits **3** rather than silently matching nothing.
|
|
457
|
+
`--enabled` and `--checked` work on both.
|
|
458
|
+
|
|
402
459
|
### Auto scroll-into-view
|
|
403
460
|
|
|
404
461
|
An element can be in the hierarchy without being reachable at the point a tap
|
package/dist/agent/engine.js
CHANGED
|
@@ -4,6 +4,7 @@ exports.DEFAULT_GUARD_SETTLE_MS = exports.DEFAULT_RUN_TIMEOUT_MS = void 0;
|
|
|
4
4
|
exports.runPlan = runPlan;
|
|
5
5
|
const node_crypto_1 = require("node:crypto");
|
|
6
6
|
const selector_1 = require("../ui/selector");
|
|
7
|
+
const state_support_1 = require("../ui/state-support");
|
|
7
8
|
const errors_1 = require("../errors");
|
|
8
9
|
const ir_1 = require("./ir");
|
|
9
10
|
/** An outcome is environment-flavoured if it carries an exit-3 CliError, or simply
|
|
@@ -191,6 +192,11 @@ async function runPlan(plan, deps) {
|
|
|
191
192
|
deps.log(`[ai] guard selector '${selector}' did not parse (${e.message}) — treating as not present`);
|
|
192
193
|
return false;
|
|
193
194
|
}
|
|
195
|
+
// Outside the catch on purpose. A guard pinning state this platform cannot report
|
|
196
|
+
// would match nothing forever, so "not present" is a lie that silently skips the
|
|
197
|
+
// body — the plan is unrunnable HERE and must say so, not quietly do nothing.
|
|
198
|
+
if (deps.platform)
|
|
199
|
+
(0, state_support_1.assertStateSupported)(sel, deps.platform);
|
|
194
200
|
const deadline = Date.now() + Math.max(0, settleMs);
|
|
195
201
|
// A non-zero window must buy at least one SECOND look, independent of the clock.
|
|
196
202
|
// Measured on emulator-5554: one uiautomator dump costs ~2.4s, which already exceeds
|
|
@@ -368,6 +374,8 @@ async function runPlan(plan, deps) {
|
|
|
368
374
|
catch (e) {
|
|
369
375
|
return { status: 'fail', where, reason: `read selector '${selector}' did not parse: ${e.message}` };
|
|
370
376
|
}
|
|
377
|
+
if (deps.platform)
|
|
378
|
+
(0, state_support_1.assertStateSupported)(sel, deps.platform);
|
|
371
379
|
// Same patience as a conditional guard: the value may not have rendered yet.
|
|
372
380
|
const deadline = Date.now() + Math.max(0, guardSettleMs);
|
|
373
381
|
let looks = 0;
|
package/dist/agent/grammar.js
CHANGED
|
@@ -112,12 +112,29 @@ SELECTORS (the engine auto-heals case/whitespace/partial, so prefer stable ident
|
|
|
112
112
|
class:Button type or class
|
|
113
113
|
"Sign in" bare string == text:Sign in
|
|
114
114
|
|
|
115
|
+
A selector may also pin ELEMENT STATE, in both polarities:
|
|
116
|
+
--enabled / --not-enabled actionable right now
|
|
117
|
+
--selected / --not-selected current option of a segmented control / tab bar / mode picker
|
|
118
|
+
--checked / --not-checked checkbox / switch / radio state
|
|
119
|
+
--focused / --not-focused holds input focus
|
|
120
|
+
On a command leaf write it as a flag. On a CONTROL NODE append it to the selector string —
|
|
121
|
+
that is the only place one can go, and it is what makes a state-conditional guard possible:
|
|
122
|
+
{ "type":"if-present", "selector":"id:mode_video --not-selected",
|
|
123
|
+
"body":[ { "type":"command","command":"tap","positionals":["id:mode_video"],"flags":[] } ] }
|
|
124
|
+
|
|
115
125
|
RULES:
|
|
116
126
|
- --enabled on a tap makes it match only a control that is ACTIONABLE right now, and (with
|
|
117
127
|
auto-wait) wait until it becomes so. Use it for any button that the app disables until
|
|
118
128
|
something else is done — a Check/Submit/Continue that only lights up once an answer is
|
|
119
129
|
selected or a form is valid. Without it the step taps a dead control, does nothing, and
|
|
120
130
|
the failure surfaces later as a confusing timeout on the NEXT step.
|
|
131
|
+
- A picker or toggle whose options share ONE handler FLIPS on any tap, so an unconditional
|
|
132
|
+
"tap the option you want" lands on the option you did NOT want whenever it was already
|
|
133
|
+
chosen — and its starting state is usually content-driven, so you cannot know it now.
|
|
134
|
+
Guard it: if-present "id:<option> --not-selected" { tap id:<option> }. The guard makes an
|
|
135
|
+
already-correct state a no-op instead of a flip. Unguarded, the flow completes either way
|
|
136
|
+
and the test PASSES having exercised the opposite mode — a false green, worse than a fail.
|
|
137
|
+
Same shape for a checkbox that toggles: guard with --not-checked / --checked.
|
|
121
138
|
- assert is for VERIFICATION only and is terminal — never use it as a step you expect to
|
|
122
139
|
fail. Put genuinely-optional UI behind if-present.
|
|
123
140
|
- tap/text SCROLL THEIR TARGET INTO VIEW automatically, so "scroll down to X and tap it"
|
package/dist/args.js
CHANGED
|
@@ -44,6 +44,19 @@ const BOOLEAN = new Set([
|
|
|
44
44
|
'no-restart',
|
|
45
45
|
'allow-install',
|
|
46
46
|
'allow-unsafe-anonymous',
|
|
47
|
+
// Selector state modifiers (STATE_ATTRS in ui/selector.ts) and their negations.
|
|
48
|
+
// `enabled` was missing here until 0.15.0, and the omission was not cosmetic: a
|
|
49
|
+
// non-BOOLEAN flag swallows the next token, so `vk tap --enabled @submit` bound
|
|
50
|
+
// the SELECTOR as the flag's value and died with "Missing selector". Only the
|
|
51
|
+
// trailing form worked. Any new modifier must be listed here.
|
|
52
|
+
'enabled',
|
|
53
|
+
'selected',
|
|
54
|
+
'checked',
|
|
55
|
+
'focused',
|
|
56
|
+
'not-enabled',
|
|
57
|
+
'not-selected',
|
|
58
|
+
'not-checked',
|
|
59
|
+
'not-focused',
|
|
47
60
|
]);
|
|
48
61
|
function parseArgs(argv) {
|
|
49
62
|
const positionals = [];
|
package/dist/cli.js
CHANGED
|
@@ -35,6 +35,7 @@ var __importStar = (this && this.__importStar) || (function () {
|
|
|
35
35
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
36
|
exports.platformFromFlags = platformFromFlags;
|
|
37
37
|
exports.deviceFromFlags = deviceFromFlags;
|
|
38
|
+
exports.stateFromFlags = stateFromFlags;
|
|
38
39
|
exports.parsePoint = parsePoint;
|
|
39
40
|
exports.healNote = healNote;
|
|
40
41
|
exports.parseDuration = parseDuration;
|
|
@@ -58,6 +59,7 @@ const errors_1 = require("./errors");
|
|
|
58
59
|
const exec_1 = require("./exec");
|
|
59
60
|
const drivers_1 = require("./drivers");
|
|
60
61
|
const selector_1 = require("./ui/selector");
|
|
62
|
+
const state_support_1 = require("./ui/state-support");
|
|
61
63
|
const format_1 = require("./ui/format");
|
|
62
64
|
const viewport_1 = require("./ui/viewport");
|
|
63
65
|
const output_1 = require("./output");
|
|
@@ -93,15 +95,42 @@ function deviceFromFlags(flags, platform) {
|
|
|
93
95
|
(platform === 'android' ? process.env.ANDROID_SERIAL : undefined) ||
|
|
94
96
|
undefined);
|
|
95
97
|
}
|
|
96
|
-
|
|
98
|
+
/**
|
|
99
|
+
* Read the `--enabled` / `--not-enabled` / `--selected` / … pairs off the flags.
|
|
100
|
+
*
|
|
101
|
+
* An ABSENT flag must stay `undefined`, never `false`: these modifiers are tri-state
|
|
102
|
+
* ("must be" / "must not be" / "don't care"), so passing `flagBool()` straight through —
|
|
103
|
+
* which is what this replaced — would quietly turn every selector on every command into
|
|
104
|
+
* "must be disabled, unselected, unchecked and unfocused".
|
|
105
|
+
*/
|
|
106
|
+
function stateFromFlags(flags) {
|
|
107
|
+
const state = {};
|
|
108
|
+
for (const attr of selector_1.STATE_ATTRS) {
|
|
109
|
+
const yes = (0, args_1.flagBool)(flags, attr);
|
|
110
|
+
const no = (0, args_1.flagBool)(flags, `not-${attr}`);
|
|
111
|
+
if (yes && no)
|
|
112
|
+
throw new errors_1.CliError(`Cannot combine --${attr} with --not-${attr}.`, 2);
|
|
113
|
+
if (yes)
|
|
114
|
+
state[attr] = true;
|
|
115
|
+
else if (no)
|
|
116
|
+
state[attr] = false;
|
|
117
|
+
}
|
|
118
|
+
return state;
|
|
119
|
+
}
|
|
120
|
+
function buildSelector(ctx, raw) {
|
|
97
121
|
if (!raw) {
|
|
98
122
|
throw new errors_1.CliError('Missing selector. e.g. `@login_button`, `text:Login`, `desc:Submit`.', 2);
|
|
99
123
|
}
|
|
100
|
-
|
|
101
|
-
contains: (0, args_1.flagBool)(flags, 'contains'),
|
|
102
|
-
index: (0, args_1.flagNum)(flags, 'index'),
|
|
103
|
-
|
|
124
|
+
const sel = (0, selector_1.parseSelector)(raw, {
|
|
125
|
+
contains: (0, args_1.flagBool)(ctx.flags, 'contains'),
|
|
126
|
+
index: (0, args_1.flagNum)(ctx.flags, 'index'),
|
|
127
|
+
...stateFromFlags(ctx.flags),
|
|
104
128
|
});
|
|
129
|
+
// Every command's selector — and so every `vk ai` leaf, which reaches these handlers
|
|
130
|
+
// through executeOutcome — funnels through here, which is why the platform check lives
|
|
131
|
+
// at this seam rather than in the (platform-free) selector layer.
|
|
132
|
+
(0, state_support_1.assertStateSupported)(sel, ctx.platform);
|
|
133
|
+
return sel;
|
|
105
134
|
}
|
|
106
135
|
function parsePoint(s) {
|
|
107
136
|
const m = /^(-?\d+)\s*,\s*(-?\d+)$/.exec(s.trim());
|
|
@@ -458,7 +487,7 @@ function cmdUi(ctx) {
|
|
|
458
487
|
return 0;
|
|
459
488
|
}
|
|
460
489
|
async function cmdFind(ctx) {
|
|
461
|
-
const sel = buildSelector(ctx.positionals[0]
|
|
490
|
+
const sel = buildSelector(ctx, ctx.positionals[0]);
|
|
462
491
|
const { matches, tier } = await matchWaiting(ctx, sel, { all: (0, args_1.flagBool)(ctx.flags, 'all') });
|
|
463
492
|
if ((0, args_1.flagBool)(ctx.flags, 'json'))
|
|
464
493
|
(0, output_1.json)(matches.map(format_1.toJsonShape));
|
|
@@ -511,7 +540,7 @@ async function cmdTap(ctx) {
|
|
|
511
540
|
ctx.record?.note({ element: target, message: `tapped by index [${idx}]` });
|
|
512
541
|
}
|
|
513
542
|
else {
|
|
514
|
-
const sel = buildSelector(
|
|
543
|
+
const sel = buildSelector(ctx, raw);
|
|
515
544
|
({ element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel, {
|
|
516
545
|
all: (0, args_1.flagBool)(ctx.flags, 'all'),
|
|
517
546
|
}));
|
|
@@ -530,7 +559,7 @@ async function cmdText(ctx) {
|
|
|
530
559
|
if (ctx.positionals.length < 2) {
|
|
531
560
|
throw new errors_1.CliError('Usage: verikun text <selector> <text...> (use -- before text starting with "-")', 2);
|
|
532
561
|
}
|
|
533
|
-
const sel = buildSelector(ctx.positionals[0]
|
|
562
|
+
const sel = buildSelector(ctx, ctx.positionals[0]);
|
|
534
563
|
const value = ctx.positionals.slice(1).join(' ');
|
|
535
564
|
const { element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel);
|
|
536
565
|
ctx.record?.note({
|
|
@@ -606,7 +635,9 @@ async function cmdSwipe(ctx) {
|
|
|
606
635
|
let waitedMs = 0;
|
|
607
636
|
const on = (0, args_1.flagStr)(ctx.flags, 'on');
|
|
608
637
|
if (on) {
|
|
609
|
-
|
|
638
|
+
// Through buildSelector, not parseSelector: --on used to see only --contains, so
|
|
639
|
+
// `swipe --on X --enabled` silently ignored the modifier it was given.
|
|
640
|
+
const onSel = buildSelector(ctx, on);
|
|
610
641
|
const { element, waitedMs: w } = await resolveOneWaiting(ctx, onSel);
|
|
611
642
|
waitedMs = w;
|
|
612
643
|
ctx.record?.note({ selector: onSel, element });
|
|
@@ -762,7 +793,7 @@ function cmdLog(ctx) {
|
|
|
762
793
|
return 0;
|
|
763
794
|
}
|
|
764
795
|
async function cmdWait(ctx) {
|
|
765
|
-
const sel = buildSelector(ctx.positionals[0]
|
|
796
|
+
const sel = buildSelector(ctx, ctx.positionals[0]);
|
|
766
797
|
const gone = (0, args_1.flagBool)(ctx.flags, 'gone');
|
|
767
798
|
const timeout = (0, args_1.flagNum)(ctx.flags, 'timeout') ?? 10000;
|
|
768
799
|
const interval = (0, args_1.flagNum)(ctx.flags, 'interval') ?? 400;
|
|
@@ -822,7 +853,7 @@ function evalAssert(els, sel, flags) {
|
|
|
822
853
|
return { pass, reason, matches };
|
|
823
854
|
}
|
|
824
855
|
async function cmdAssert(ctx) {
|
|
825
|
-
const sel = buildSelector(ctx.positionals[0]
|
|
856
|
+
const sel = buildSelector(ctx, ctx.positionals[0]);
|
|
826
857
|
// Auto-wait subsumes the common "wait then assert": poll until the assertion
|
|
827
858
|
// passes or the window elapses. `--gone` therefore waits for disappearance.
|
|
828
859
|
const deadline = Date.now() + waitWindowMs(ctx.flags);
|
|
@@ -1353,6 +1384,9 @@ async function runAiTest(file, opts, backend, platform, device) {
|
|
|
1353
1384
|
guardSettleMs: guardSettleMs(),
|
|
1354
1385
|
runId: started.id,
|
|
1355
1386
|
deadline,
|
|
1387
|
+
// The RESOLVED platform — for --server that is the server's, which supersedes
|
|
1388
|
+
// the client's --platform (leaves are gated server-side; guards run here).
|
|
1389
|
+
platform,
|
|
1356
1390
|
});
|
|
1357
1391
|
}
|
|
1358
1392
|
catch (e) {
|
|
@@ -1503,7 +1537,7 @@ async function cmdInstall(positionals, flags) {
|
|
|
1503
1537
|
async function cmdSuiteEntry(positionals, flags) {
|
|
1504
1538
|
const dirArg = positionals[0];
|
|
1505
1539
|
if (!dirArg)
|
|
1506
|
-
throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--json]', 2);
|
|
1540
|
+
throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--retries n] [--json]', 2);
|
|
1507
1541
|
const opts = parseAiOptions(flags);
|
|
1508
1542
|
// Pre-flight the provider BEFORE touching any device/server: every test needs it
|
|
1509
1543
|
// to compile (on a cache miss) or to repair at runtime.
|
|
@@ -1815,14 +1849,19 @@ AI (run a natural-language test — compile once, replay model-free, self-heal)
|
|
|
1815
1849
|
codex-cli | cursor-cli.
|
|
1816
1850
|
|
|
1817
1851
|
SUITE (run a directory of natural-language tests as one gated suite)
|
|
1818
|
-
suite <dir> [--app <id>] [--name n] [--
|
|
1852
|
+
suite <dir> [--app <id>] [--name n] [--retries n] [--json]
|
|
1853
|
+
(+ all \`ai\` flags, incl. --server)
|
|
1819
1854
|
Run every *.md in <dir> (lexicographic order —
|
|
1820
1855
|
prefix 01-, 02- to sequence; README.md skipped)
|
|
1821
1856
|
through \`vk ai\`. With --app, app data is reset
|
|
1822
|
-
between tests (iOS: force-stop).
|
|
1823
|
-
|
|
1824
|
-
|
|
1825
|
-
|
|
1857
|
+
between tests (iOS: force-stop). --retries N
|
|
1858
|
+
re-runs a failed test up to N times; a later
|
|
1859
|
+
pass recovers the suite (exit 0) and surfaces a
|
|
1860
|
+
warning, keeping failed-attempt evidence in the
|
|
1861
|
+
report. Writes a suite overview to
|
|
1862
|
+
./.verikun/suites/<id>/{index.json, index.html}
|
|
1863
|
+
linking each test's report. Exits 1 if any test
|
|
1864
|
+
failed — the CI gate.
|
|
1826
1865
|
|
|
1827
1866
|
SERVER (expose a locally-connected device to remote verikun clients)
|
|
1828
1867
|
server [--bind addr] [--port n] [--auth-key k] [--allow-install]
|
|
@@ -1859,6 +1898,15 @@ SELECTORS
|
|
|
1859
1898
|
class:Button type or full class name
|
|
1860
1899
|
"Sign in" bare string == text:"Sign in"
|
|
1861
1900
|
Modifiers: --contains (substring), --index N (pick Nth match)
|
|
1901
|
+
State: --enabled / --selected / --checked / --focused, each with a --not-
|
|
1902
|
+
form (--not-selected). Unset = don't care. Use the negative to guard
|
|
1903
|
+
a toggle: tapping a picker whose options share a handler FLIPS it, so
|
|
1904
|
+
an unconditional tap lands on the wrong mode and still exits 0.
|
|
1905
|
+
May be written as a flag or appended to the selector string
|
|
1906
|
+
("@mode_video --not-selected") — the latter is how a \`vk ai\`
|
|
1907
|
+
if-present/when/repeat guard carries one.
|
|
1908
|
+
--selected and --focused are Android-only (idb reports neither);
|
|
1909
|
+
on iOS they exit 3 rather than matching nothing.
|
|
1862
1910
|
|
|
1863
1911
|
AUTO-WAIT (selector lookups retry until they resolve)
|
|
1864
1912
|
Selector commands (tap, text, find, assert, swipe --on) re-poll the screen for
|
package/dist/report.js
CHANGED
|
@@ -118,6 +118,7 @@ const STYLE = `
|
|
|
118
118
|
.summary { display:flex; gap:8px; flex-wrap:wrap; align-items:center; margin-bottom: 20px; }
|
|
119
119
|
.chip { font-weight:600; font-size:13px; padding:4px 10px; border-radius:999px; color:#fff; }
|
|
120
120
|
.chip.pass{background:var(--pass)} .chip.fail{background:var(--fail)} .chip.err{background:var(--err)}
|
|
121
|
+
.chip.warn{background:var(--err)}
|
|
121
122
|
.chip.muted{ background:#eaeef2; color:var(--muted); }
|
|
122
123
|
ol.steps { list-style:none; margin:0; padding:0; }
|
|
123
124
|
li.step { background:#fff; border:1px solid var(--line); border-left-width:4px; border-radius:8px; margin-bottom:10px; padding:12px 14px; }
|
|
@@ -206,19 +207,40 @@ const SUITE_STYLE = `
|
|
|
206
207
|
table.tests td.num { text-align:right; font-variant-numeric:tabular-nums; white-space:nowrap; }
|
|
207
208
|
table.tests a { color:inherit; }
|
|
208
209
|
.fail-reason { color:var(--fail); font-size:12px; margin-top:2px; }
|
|
210
|
+
.flake-note { color:var(--err); font-size:12px; margin-top:2px; }
|
|
211
|
+
.attempts { margin-top:4px; font-size:12px; color:var(--muted); }
|
|
212
|
+
.attempts a { color:var(--fail); }
|
|
209
213
|
.aborted { background:#fff4e5; border:1px solid #f0b429; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
|
|
210
214
|
.aborted strong { color:#8a5300; }
|
|
211
215
|
.aborted ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
|
|
216
|
+
.warnings { background:#fff8c5; border:1px solid #d4a72c; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
|
|
217
|
+
.warnings strong { color:#7d4e00; }
|
|
218
|
+
.warnings ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
|
|
212
219
|
`;
|
|
220
|
+
function suiteAttemptLinks(attempts, linkBase) {
|
|
221
|
+
const links = attempts
|
|
222
|
+
.map((a, i) => {
|
|
223
|
+
const label = `attempt ${i + 1}`;
|
|
224
|
+
if (!a.id)
|
|
225
|
+
return htmlEsc(label);
|
|
226
|
+
return `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(a.id)}/report.html`)}">${htmlEsc(label)}</a>`;
|
|
227
|
+
})
|
|
228
|
+
.join(', ');
|
|
229
|
+
return `<div class="attempts">prior failed: ${links}</div>`;
|
|
230
|
+
}
|
|
213
231
|
function suiteTestRow(t, linkBase) {
|
|
214
232
|
// A test that errored before its run started (id '') has no report to link.
|
|
215
233
|
const label = t.id
|
|
216
234
|
? `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(t.id)}/report.html`)}">${htmlEsc(t.name)}</a>`
|
|
217
235
|
: htmlEsc(t.name);
|
|
218
236
|
const failure = t.failure ? `<div class="fail-reason">${htmlEsc(t.failure)}</div>` : '';
|
|
237
|
+
const flake = t.flaky ? `<div class="flake-note">passed on retry (flake)</div>` : '';
|
|
238
|
+
const prior = t.attempts?.length ? suiteAttemptLinks(t.attempts, linkBase) : '';
|
|
239
|
+
const status = t.flaky ? 'FLAKY' : t.ok ? 'PASS' : 'FAIL';
|
|
240
|
+
const statusClass = t.ok ? 'passed' : 'failed';
|
|
219
241
|
return ` <tr>
|
|
220
|
-
<td><span class="st ${
|
|
221
|
-
<td>${label}${failure}</td>
|
|
242
|
+
<td><span class="st ${statusClass}">${status}</span></td>
|
|
243
|
+
<td>${label}${flake}${failure}${prior}</td>
|
|
222
244
|
<td class="num">${t.passedSteps}/${t.steps}${t.failedSteps ? ` (${t.failedSteps} failed)` : ''}</td>
|
|
223
245
|
<td class="num">${t.modelRepairs || ''}</td>
|
|
224
246
|
<td class="num">$${t.costUsd.toFixed(4)}</td>
|
|
@@ -233,9 +255,11 @@ function suiteTestRow(t, linkBase) {
|
|
|
233
255
|
function toSuiteHtml(suite, opts = {}) {
|
|
234
256
|
const linkBase = opts.linkBase ?? '../../';
|
|
235
257
|
const t = suite.totals;
|
|
258
|
+
const flaky = suite.tests.filter((x) => x.flaky).length;
|
|
236
259
|
const chips = [
|
|
237
260
|
`<span class="chip pass">${t.passed} passed</span>`,
|
|
238
261
|
t.failed ? `<span class="chip fail">${t.failed} failed</span>` : '',
|
|
262
|
+
flaky ? `<span class="chip warn">${flaky} flaky</span>` : '',
|
|
239
263
|
suite.aborted ? `<span class="chip fail">ABORTED</span>` : '',
|
|
240
264
|
`<span class="chip muted">${t.tests} tests · ${t.steps} steps · ${fmtDuration(t.durationMs)} · $${t.costUsd.toFixed(4)}</span>`,
|
|
241
265
|
]
|
|
@@ -250,6 +274,13 @@ function toSuiteHtml(suite, opts = {}) {
|
|
|
250
274
|
${suite.aborted.notRun.length
|
|
251
275
|
? ` <ul>${suite.aborted.notRun.map((f) => `<li>${htmlEsc(f)} — not run</li>`).join('')}</ul>\n`
|
|
252
276
|
: ''} </div>
|
|
277
|
+
`
|
|
278
|
+
: '';
|
|
279
|
+
const warningsBanner = suite.warnings?.length
|
|
280
|
+
? ` <div class="warnings">
|
|
281
|
+
<strong>Warnings</strong>
|
|
282
|
+
<ul>${suite.warnings.map((w) => `<li>${htmlEsc(w)}</li>`).join('')}</ul>
|
|
283
|
+
</div>
|
|
253
284
|
`
|
|
254
285
|
: '';
|
|
255
286
|
const metaBits = [
|
|
@@ -274,7 +305,7 @@ ${suite.aborted.notRun.length
|
|
|
274
305
|
<div class="summary">
|
|
275
306
|
${chips}
|
|
276
307
|
</div>
|
|
277
|
-
${abortedBanner} <table class="tests">
|
|
308
|
+
${abortedBanner}${warningsBanner} <table class="tests">
|
|
278
309
|
<thead><tr><th></th><th>Test</th><th>Steps</th><th>Repairs</th><th>Cost</th><th>Duration</th></tr></thead>
|
|
279
310
|
<tbody>
|
|
280
311
|
${suite.tests.map((x) => suiteTestRow(x, linkBase)).join('\n')}
|
package/dist/suite.js
CHANGED
|
@@ -13,6 +13,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
13
13
|
exports.sortTestFiles = sortTestFiles;
|
|
14
14
|
exports.listTestFiles = listTestFiles;
|
|
15
15
|
exports.toSuiteResult = toSuiteResult;
|
|
16
|
+
exports.toSuiteAttempt = toSuiteAttempt;
|
|
17
|
+
exports.mergeSuiteAttempts = mergeSuiteAttempts;
|
|
16
18
|
exports.cmdSuite = cmdSuite;
|
|
17
19
|
const node_fs_1 = require("node:fs");
|
|
18
20
|
const node_path_1 = require("node:path");
|
|
@@ -100,6 +102,64 @@ function toSuiteResult(file, r, durationMs) {
|
|
|
100
102
|
...(r.ok ? {} : { failure: failure ?? 'failed' }),
|
|
101
103
|
};
|
|
102
104
|
}
|
|
105
|
+
/** Compact one attempt for the `attempts` evidence array (pure). */
|
|
106
|
+
function toSuiteAttempt(r) {
|
|
107
|
+
return {
|
|
108
|
+
id: r.id,
|
|
109
|
+
ok: r.ok,
|
|
110
|
+
durationMs: r.durationMs,
|
|
111
|
+
costUsd: r.costUsd,
|
|
112
|
+
...(r.failure ? { failure: r.failure } : {}),
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
/**
|
|
116
|
+
* Merge a sequence of attempt rows into the final suite row: primary `id` is the last
|
|
117
|
+
* attempt (winning green, or last red), cost/duration/repairs sum across attempts, and
|
|
118
|
+
* prior attempts are retained as flake evidence.
|
|
119
|
+
*/
|
|
120
|
+
function mergeSuiteAttempts(attempts) {
|
|
121
|
+
if (attempts.length === 0)
|
|
122
|
+
throw new Error('mergeSuiteAttempts: empty');
|
|
123
|
+
const last = attempts[attempts.length - 1];
|
|
124
|
+
if (attempts.length === 1)
|
|
125
|
+
return last;
|
|
126
|
+
const round = (n) => Number(n.toFixed(4));
|
|
127
|
+
const prior = attempts.slice(0, -1).map(toSuiteAttempt);
|
|
128
|
+
const flaky = last.ok && prior.some((a) => !a.ok);
|
|
129
|
+
return {
|
|
130
|
+
...last,
|
|
131
|
+
durationMs: attempts.reduce((a, t) => a + t.durationMs, 0),
|
|
132
|
+
costUsd: round(attempts.reduce((a, t) => a + t.costUsd, 0)),
|
|
133
|
+
modelRepairs: attempts.reduce((a, t) => a + t.modelRepairs, 0),
|
|
134
|
+
attempts: prior,
|
|
135
|
+
...(flaky ? { flaky: true } : {}),
|
|
136
|
+
};
|
|
137
|
+
}
|
|
138
|
+
// What --retries will and won't spend an attempt on. The bias is deliberate and
|
|
139
|
+
// asymmetric: a retry costs one test, while giving up costs the whole suite plus a
|
|
140
|
+
// human rerunning it. So the rule is *retry unless a rerun provably cannot change the
|
|
141
|
+
// outcome* — the two predicates below are the only "provably" cases, everything else
|
|
142
|
+
// (flaky selector, wedged app, a wobbling network to `vk server`) earns another go.
|
|
143
|
+
/** Budget aborts won't heal on retry: each attempt gets its own cost ceiling, so a
|
|
144
|
+
* rerun just re-aborts at the same place having spent the money twice. */
|
|
145
|
+
function isRetryable(r) {
|
|
146
|
+
return !r.ok && !r.abortedForBudget;
|
|
147
|
+
}
|
|
148
|
+
/** A thrown USAGE error (exit 2) is the one throw a rerun cannot change — an unreadable
|
|
149
|
+
* test file, a payload the server refuses, a flag it doesn't understand. Everything
|
|
150
|
+
* else, including every environment error, is retried while attempts remain. */
|
|
151
|
+
function isRetryableThrow(e) {
|
|
152
|
+
return !(e instanceof errors_1.CliError && e.exitCode === 2);
|
|
153
|
+
}
|
|
154
|
+
function parseRetries(flags) {
|
|
155
|
+
const n = (0, args_1.flagNum)(flags, 'retries');
|
|
156
|
+
if (n === undefined)
|
|
157
|
+
return 0;
|
|
158
|
+
if (!Number.isInteger(n) || n < 0) {
|
|
159
|
+
throw new errors_1.CliError(`--retries must be a non-negative integer, got '${n}'`, 2);
|
|
160
|
+
}
|
|
161
|
+
return n;
|
|
162
|
+
}
|
|
103
163
|
async function cmdSuite(dirArg, flags, deps) {
|
|
104
164
|
const dir = (0, node_path_1.resolve)(process.cwd(), dirArg);
|
|
105
165
|
if (!(0, node_fs_1.existsSync)(dir) || !(0, node_fs_1.statSync)(dir).isDirectory()) {
|
|
@@ -109,69 +169,133 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
109
169
|
if (files.length === 0) {
|
|
110
170
|
throw new errors_1.CliError(`suite: no test files (*.md) in '${dirArg}'`, 2);
|
|
111
171
|
}
|
|
172
|
+
const retries = parseRetries(flags);
|
|
112
173
|
const suiteId = (0, run_1.runId)();
|
|
113
174
|
const name = (0, args_1.flagStr)(flags, 'name') || (0, node_path_1.basename)(dir);
|
|
114
175
|
const startedAt = new Date().toISOString();
|
|
115
|
-
(0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})`);
|
|
176
|
+
(0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})${retries > 0 ? ` · up to ${retries} retry(ies) on failure` : ''}`);
|
|
116
177
|
const results = [];
|
|
178
|
+
const warnings = [];
|
|
117
179
|
let aborted;
|
|
180
|
+
async function resetApp(label) {
|
|
181
|
+
// Returns the abort reason when the suite should stop (confirmed env break during reset).
|
|
182
|
+
if (!deps.reset)
|
|
183
|
+
return undefined;
|
|
184
|
+
try {
|
|
185
|
+
await deps.reset();
|
|
186
|
+
(0, output_1.err)(`[suite] app state reset${label}`);
|
|
187
|
+
return undefined;
|
|
188
|
+
}
|
|
189
|
+
catch (e) {
|
|
190
|
+
// A reset that failed because the BOX is broken means nothing after it is
|
|
191
|
+
// trustworthy — but only if a re-probe agrees. Otherwise surface and continue:
|
|
192
|
+
// a flaky reset should not zero out the whole suite, and the test itself will
|
|
193
|
+
// fail loudly if the stale state actually matters.
|
|
194
|
+
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
195
|
+
if (broken)
|
|
196
|
+
return broken;
|
|
197
|
+
(0, output_1.err)(`[suite] reset failed (${e.message}) — continuing`);
|
|
198
|
+
return undefined;
|
|
199
|
+
}
|
|
200
|
+
}
|
|
201
|
+
/** A confirmed env break with attempts left: say so, pause, and let the loop retry.
|
|
202
|
+
* The pause matters — the failures this rides out (a server restart, a wifi drop, a
|
|
203
|
+
* USB re-enumeration) clear in seconds, and retrying into the same dead socket
|
|
204
|
+
* immediately would burn every attempt inside the outage. */
|
|
205
|
+
async function noteEnvRetry(file, attempt, reason) {
|
|
206
|
+
const warn = `${file}: environment error on attempt ${attempt + 1} (${reason}) — retried`;
|
|
207
|
+
warnings.push(warn);
|
|
208
|
+
(0, output_1.err)(`[suite] WARN ${warn}`);
|
|
209
|
+
await sleep((deps.probeRetryMs ?? PROBE_RETRY_MS) * (attempt + 1));
|
|
210
|
+
}
|
|
118
211
|
for (let i = 0; i < files.length && !aborted; i++) {
|
|
119
212
|
const file = files[i];
|
|
120
213
|
(0, output_1.err)(`[suite] ── (${i + 1}/${files.length}) ${file} ──`);
|
|
121
|
-
|
|
214
|
+
const attemptRows = [];
|
|
215
|
+
for (let attempt = 0; attempt <= retries; attempt++) {
|
|
216
|
+
// The last attempt is where a retryable failure becomes the verdict: a confirmed
|
|
217
|
+
// env break aborts the suite, anything else stands as this test's failed row.
|
|
218
|
+
const lastAttempt = attempt === retries;
|
|
219
|
+
if (attempt > 0)
|
|
220
|
+
(0, output_1.err)(`[suite] retry ${attempt}/${retries} for ${file}`);
|
|
221
|
+
// Re-isolate before EVERY attempt — between tests and between retries alike.
|
|
222
|
+
const resetBreak = await resetApp(attempt > 0 ? ' (retry)' : '');
|
|
223
|
+
if (resetBreak) {
|
|
224
|
+
if (!lastAttempt) {
|
|
225
|
+
await noteEnvRetry(file, attempt, `reset failed: ${resetBreak}`);
|
|
226
|
+
continue;
|
|
227
|
+
}
|
|
228
|
+
// With no attempt row this test never ran, so notRun starts at the CURRENT file.
|
|
229
|
+
aborted = {
|
|
230
|
+
reason: `reset failed: ${resetBreak}`,
|
|
231
|
+
notRun: files.slice(attemptRows.length ? i + 1 : i),
|
|
232
|
+
};
|
|
233
|
+
break;
|
|
234
|
+
}
|
|
235
|
+
const t0 = Date.now();
|
|
122
236
|
try {
|
|
123
|
-
await deps.
|
|
124
|
-
(
|
|
237
|
+
const r = await deps.runTest((0, node_path_1.join)(dir, file));
|
|
238
|
+
attemptRows.push(toSuiteResult(file, r, Date.now() - t0));
|
|
239
|
+
if (r.abortedForEnv) {
|
|
240
|
+
const broken = await stillBroken(deps);
|
|
241
|
+
if (broken) {
|
|
242
|
+
if (!lastAttempt) {
|
|
243
|
+
// Even a CONFIRMED break is worth an attempt: the probe window is a couple
|
|
244
|
+
// of seconds, which a server restart outlives — and aborting costs the run.
|
|
245
|
+
await noteEnvRetry(file, attempt, broken);
|
|
246
|
+
continue;
|
|
247
|
+
}
|
|
248
|
+
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
249
|
+
break;
|
|
250
|
+
}
|
|
251
|
+
// Transient env blip: retryable like any other failure.
|
|
252
|
+
}
|
|
253
|
+
if (r.ok || !isRetryable(r) || lastAttempt)
|
|
254
|
+
break;
|
|
125
255
|
}
|
|
126
256
|
catch (e) {
|
|
127
|
-
// A
|
|
128
|
-
//
|
|
129
|
-
//
|
|
130
|
-
|
|
257
|
+
// A test that THREW (device gone, server unreachable, bad file) still becomes a
|
|
258
|
+
// failed row — one broken test must not vaporize the suite report for the tests
|
|
259
|
+
// that already ran. Out of attempts, a confirmed env break stops the suite.
|
|
260
|
+
const msg = e instanceof Error ? e.message : String(e);
|
|
261
|
+
(0, output_1.err)(`[suite] ${file} errored: ${msg}`);
|
|
262
|
+
attemptRows.push({
|
|
263
|
+
id: '',
|
|
264
|
+
file,
|
|
265
|
+
name: (0, node_path_1.basename)(file, (0, node_path_1.extname)(file)),
|
|
266
|
+
ok: false,
|
|
267
|
+
durationMs: Date.now() - t0,
|
|
268
|
+
costUsd: 0,
|
|
269
|
+
steps: 0,
|
|
270
|
+
passedSteps: 0,
|
|
271
|
+
failedSteps: 0,
|
|
272
|
+
modelRepairs: 0,
|
|
273
|
+
failure: msg.split('\n')[0],
|
|
274
|
+
});
|
|
131
275
|
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
132
|
-
if (
|
|
133
|
-
|
|
134
|
-
|
|
276
|
+
if (lastAttempt) {
|
|
277
|
+
if (broken)
|
|
278
|
+
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
135
279
|
break;
|
|
136
280
|
}
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
}
|
|
140
|
-
const t0 = Date.now();
|
|
141
|
-
try {
|
|
142
|
-
const r = await deps.runTest((0, node_path_1.join)(dir, file));
|
|
143
|
-
results.push(toSuiteResult(file, r, Date.now() - t0));
|
|
144
|
-
// The test itself reported an environment abort (exit 3 mid-plan). Same rule:
|
|
145
|
-
// fatal only if the box is still broken. This test HAS a row and a real report,
|
|
146
|
-
// so notRun starts after it.
|
|
147
|
-
if (r.abortedForEnv) {
|
|
148
|
-
const broken = await stillBroken(deps);
|
|
281
|
+
if (!isRetryableThrow(e))
|
|
282
|
+
break;
|
|
149
283
|
if (broken)
|
|
150
|
-
|
|
284
|
+
await noteEnvRetry(file, attempt, broken);
|
|
151
285
|
}
|
|
152
286
|
}
|
|
153
|
-
|
|
154
|
-
//
|
|
155
|
-
//
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
costUsd: 0,
|
|
166
|
-
steps: 0,
|
|
167
|
-
passedSteps: 0,
|
|
168
|
-
failedSteps: 0,
|
|
169
|
-
modelRepairs: 0,
|
|
170
|
-
failure: msg.split('\n')[0],
|
|
171
|
-
});
|
|
172
|
-
const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
|
|
173
|
-
if (broken)
|
|
174
|
-
aborted = { reason: broken, notRun: files.slice(i + 1) };
|
|
287
|
+
if (attemptRows.length === 0) {
|
|
288
|
+
// Every attempt was blocked by a failing reset, so the test never ran and gets no
|
|
289
|
+
// row — `aborted.notRun` (set above) already names it. Nothing to merge.
|
|
290
|
+
break;
|
|
291
|
+
}
|
|
292
|
+
const merged = mergeSuiteAttempts(attemptRows);
|
|
293
|
+
results.push(merged);
|
|
294
|
+
if (merged.flaky) {
|
|
295
|
+
const n = merged.attempts?.length ?? 0;
|
|
296
|
+
const warn = `${file} passed on retry after ${n} failed attempt${n === 1 ? '' : 's'}`;
|
|
297
|
+
warnings.push(warn);
|
|
298
|
+
(0, output_1.err)(`[suite] WARN ${warn}`);
|
|
175
299
|
}
|
|
176
300
|
}
|
|
177
301
|
if (aborted) {
|
|
@@ -189,6 +313,7 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
189
313
|
totals: (0, report_1.suiteTotals)(results),
|
|
190
314
|
tests: results,
|
|
191
315
|
...(aborted ? { aborted } : {}),
|
|
316
|
+
...(warnings.length ? { warnings } : {}),
|
|
192
317
|
};
|
|
193
318
|
// .verikun/suites/<id>/ sits beside .verikun/runs/<id>/, so index.html reaches a
|
|
194
319
|
// test report at ../../runs/<id>/report.html — the linkBase below.
|
|
@@ -198,8 +323,12 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
198
323
|
(0, node_fs_1.writeFileSync)((0, node_path_1.join)(outDir, 'index.html'), (0, report_1.toSuiteHtml)(suite, { linkBase: '../../' }));
|
|
199
324
|
const t = suite.totals;
|
|
200
325
|
(0, output_1.err)(`[suite] ${t.passed}/${t.tests} passed · ${t.steps} steps · $${t.costUsd.toFixed(4)} · ${(t.durationMs / 1000).toFixed(1)}s`);
|
|
201
|
-
for (const r of results)
|
|
202
|
-
|
|
326
|
+
for (const r of results) {
|
|
327
|
+
const tag = r.flaky ? 'FLAKY' : r.ok ? 'PASS' : 'FAIL';
|
|
328
|
+
(0, output_1.err)(` ${tag} ${r.file}${r.failure ? ` — ${r.failure}` : r.flaky ? ' — passed on retry' : ''}`);
|
|
329
|
+
}
|
|
330
|
+
if (warnings.length)
|
|
331
|
+
(0, output_1.err)(`[suite] ${warnings.length} warning(s)`);
|
|
203
332
|
(0, output_1.err)(`[suite] overview: ${(0, node_path_1.join)(outDir, 'index.html')}`);
|
|
204
333
|
if ((0, args_1.flagBool)(flags, 'json'))
|
|
205
334
|
(0, output_1.json)(suite);
|
|
@@ -207,6 +336,7 @@ async function cmdSuite(dirArg, flags, deps) {
|
|
|
207
336
|
(0, output_1.out)(outDir); // primary machine result: the suite directory
|
|
208
337
|
// The CI gate: any failed test fails the invocation (mirrors `vk run archive`). An
|
|
209
338
|
// environment abort exits 3 instead, so CI can tell "the runner is broken" from "the
|
|
210
|
-
// app regressed" — the whole point of stopping early.
|
|
339
|
+
// app regressed" — the whole point of stopping early. A flake that recovered is ok
|
|
340
|
+
// (exit 0) with a warning — that is the whole point of --retries.
|
|
211
341
|
return aborted ? 3 : t.failed > 0 ? 1 : 0;
|
|
212
342
|
}
|
package/dist/ui/ios-parse.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.IOS_UNREPORTED_STATE = void 0;
|
|
3
4
|
exports.isInteresting = isInteresting;
|
|
4
5
|
exports.parseIosHierarchy = parseIosHierarchy;
|
|
5
6
|
const viewport_1 = require("./viewport");
|
|
@@ -13,6 +14,23 @@ const TAPPABLE_TYPES = new Set([
|
|
|
13
14
|
const SCROLLABLE_TYPES = new Set(['ScrollView', 'Table', 'TableView', 'CollectionView', 'WebView']);
|
|
14
15
|
const CHECKABLE_TYPES = new Set(['Switch', 'Toggle', 'CheckBox', 'RadioButton']);
|
|
15
16
|
const TEXT_INPUT_TYPES = new Set(['TextField', 'SecureTextField', 'SearchField', 'TextView']);
|
|
17
|
+
/**
|
|
18
|
+
* State attributes idb does not report, so `Element` can only ever say `false`.
|
|
19
|
+
*
|
|
20
|
+
* MEASURED against the Flutter fixture's `@vk_state` screen on an iPhone 17 Pro simulator
|
|
21
|
+
* (iOS 26.5): a selected and an unselected segment came back byte-identical apart from
|
|
22
|
+
* label and frame, and a focused text field was indistinguishable from an unfocused one.
|
|
23
|
+
* The key is not merely unset — `idb ui describe-all` has no such key in its schema at all
|
|
24
|
+
* (it emits AXFrame, AXLabel, AXUniqueId, AXValue, content_required, custom_actions,
|
|
25
|
+
* enabled, frame, help, role, role_description, subrole, title, type, and nothing else),
|
|
26
|
+
* so no app can supply it and there is nothing to derive it from.
|
|
27
|
+
*
|
|
28
|
+
* Exported so `--selected` / `--focused` can be REJECTED on iOS rather than silently
|
|
29
|
+
* matching nothing — a filter that can never match is exactly the false-green failure the
|
|
30
|
+
* modifier exists to prevent. Keep this list next to the hard-coded `false`s below; if idb
|
|
31
|
+
* ever starts reporting one, delete it from here in the same change that parses it.
|
|
32
|
+
*/
|
|
33
|
+
exports.IOS_UNREPORTED_STATE = ['selected', 'focused'];
|
|
16
34
|
function str(v) {
|
|
17
35
|
return typeof v === 'string' ? v : v == null ? '' : String(v);
|
|
18
36
|
}
|
|
@@ -81,6 +99,7 @@ function buildElement(raw) {
|
|
|
81
99
|
checkable,
|
|
82
100
|
checked: checkable && isTrue(raw.AXValue),
|
|
83
101
|
focusable: false,
|
|
102
|
+
// `focused` and `selected` are not derivable — see IOS_UNREPORTED_STATE above.
|
|
84
103
|
focused: false,
|
|
85
104
|
scrollable: SCROLLABLE_TYPES.has(type),
|
|
86
105
|
enabled: raw.enabled === undefined ? true : isTrue(raw.enabled),
|
package/dist/ui/selector.js
CHANGED
|
@@ -1,19 +1,68 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.STATE_ATTRS = void 0;
|
|
3
4
|
exports.parseSelector = parseSelector;
|
|
4
5
|
exports.matchElements = matchElements;
|
|
5
6
|
exports.resolveOne = resolveOne;
|
|
6
7
|
const errors_1 = require("../errors");
|
|
7
8
|
const format_1 = require("./format");
|
|
9
|
+
/**
|
|
10
|
+
* The element state a selector can require. Each is TRI-STATE on a Selector:
|
|
11
|
+
* unset = don't care, `true` = must be, `false` = must NOT be.
|
|
12
|
+
*
|
|
13
|
+
* The negative half is not symmetry for its own sake. A segmented control whose
|
|
14
|
+
* options share one handler *flips* on any tap, so "tap it unless it is already
|
|
15
|
+
* selected" is the only safe way to land on a known option — and a plan that
|
|
16
|
+
* cannot say that taps blind, completes either way, and passes having exercised
|
|
17
|
+
* the opposite mode.
|
|
18
|
+
*/
|
|
19
|
+
exports.STATE_ATTRS = ['enabled', 'selected', 'checked', 'focused'];
|
|
20
|
+
/** Trailing ` --selected` / ` --not-checked` / … on a selector STRING. */
|
|
21
|
+
const STATE_MODIFIER = new RegExp(`\\s+--(not-)?(${exports.STATE_ATTRS.join('|')})\\s*$`, 'i');
|
|
22
|
+
/**
|
|
23
|
+
* Peel state modifiers off the END of a selector string.
|
|
24
|
+
*
|
|
25
|
+
* Flags normally arrive as flags, but a control node's selector has nowhere to put
|
|
26
|
+
* one: `if-present` / `when` / `repeat` / `while-present` / `read` all hold a bare
|
|
27
|
+
* `selector: string` (and `swipe --on` is a flag value). Guards are exactly where
|
|
28
|
+
* "tap it only if it is not already selected" needs to be expressed, so the string
|
|
29
|
+
* itself carries them and every caller converges on this one parser:
|
|
30
|
+
*
|
|
31
|
+
* if-present "id:mode_video --not-selected" { tap id:mode_video }
|
|
32
|
+
*
|
|
33
|
+
* Only these modifiers, only at the end, and only after whitespace — so `text:--selected`
|
|
34
|
+
* and `text:a --selected b` are still plain values. A `text:` value that genuinely ENDS
|
|
35
|
+
* in " --selected" would be misread; use `--contains` on a shorter substring if you ever
|
|
36
|
+
* meet one. Note the engine interpolates `{{ctx.…}}` before parsing, so a value captured
|
|
37
|
+
* by `read` could in principle end in a modifier — end-anchoring plus the required space
|
|
38
|
+
* is what keeps that from being a practical concern.
|
|
39
|
+
*/
|
|
40
|
+
function splitStateModifiers(raw) {
|
|
41
|
+
const state = {};
|
|
42
|
+
let rest = raw;
|
|
43
|
+
for (;;) {
|
|
44
|
+
const m = STATE_MODIFIER.exec(rest);
|
|
45
|
+
if (!m)
|
|
46
|
+
return { rest, state };
|
|
47
|
+
const attr = m[2].toLowerCase();
|
|
48
|
+
const want = !m[1];
|
|
49
|
+
if (state[attr] !== undefined && state[attr] !== want) {
|
|
50
|
+
throw new errors_1.CliError(`Selector '${raw}' asks for both --${attr} and --not-${attr}.`, 2);
|
|
51
|
+
}
|
|
52
|
+
state[attr] = want;
|
|
53
|
+
rest = rest.slice(0, m.index);
|
|
54
|
+
}
|
|
55
|
+
}
|
|
8
56
|
function parseSelector(raw, opts = {}) {
|
|
57
|
+
const { rest, state } = splitStateModifiers(raw);
|
|
9
58
|
let kind = 'text';
|
|
10
|
-
let value =
|
|
11
|
-
if (
|
|
59
|
+
let value = rest;
|
|
60
|
+
if (rest.startsWith('@')) {
|
|
12
61
|
kind = 'id';
|
|
13
|
-
value =
|
|
62
|
+
value = rest.slice(1);
|
|
14
63
|
}
|
|
15
64
|
else {
|
|
16
|
-
const m = /^(id|text|desc|class):([\s\S]*)$/.exec(
|
|
65
|
+
const m = /^(id|text|desc|class):([\s\S]*)$/.exec(rest);
|
|
17
66
|
if (m) {
|
|
18
67
|
kind = m[1];
|
|
19
68
|
value = m[2];
|
|
@@ -21,20 +70,35 @@ function parseSelector(raw, opts = {}) {
|
|
|
21
70
|
}
|
|
22
71
|
if (!value)
|
|
23
72
|
throw new errors_1.CliError(`Empty selector value in '${raw}'`, 2);
|
|
24
|
-
|
|
73
|
+
const sel = { kind, value, contains: !!opts.contains, index: opts.index, raw };
|
|
74
|
+
for (const attr of exports.STATE_ATTRS) {
|
|
75
|
+
// An explicit flag beats one embedded in the string; absent means absent, not false.
|
|
76
|
+
const want = opts[attr] !== undefined ? opts[attr] : state[attr];
|
|
77
|
+
if (want !== undefined)
|
|
78
|
+
sel[attr] = want;
|
|
79
|
+
}
|
|
80
|
+
return sel;
|
|
25
81
|
}
|
|
26
|
-
/**
|
|
82
|
+
/** Keep only elements whose state matches every attribute the selector pins.
|
|
27
83
|
*
|
|
28
|
-
*
|
|
29
|
-
*
|
|
30
|
-
*
|
|
31
|
-
*
|
|
32
|
-
*
|
|
33
|
-
*
|
|
34
|
-
*
|
|
35
|
-
*
|
|
36
|
-
* app drift and sends the
|
|
37
|
-
|
|
84
|
+
* Each predicate is exactly the one a11y attribute and nothing else — `--enabled` is
|
|
85
|
+
* `enabled`, matching what Maestro's `enabled: true` means. An earlier version also
|
|
86
|
+
* required `clickable || longClickable`, reasoning that a disabled Button might report
|
|
87
|
+
* clickable=false. That was speculation and it was wrong in the direction that hurts:
|
|
88
|
+
* plenty of legitimate tap targets are CONTAINERS whose own clickable flag is false (the
|
|
89
|
+
* tappable child is inside), so the extra conjunct filtered out real elements and turned
|
|
90
|
+
* `--enabled` into a source of phantom "not found" misses — which then burned model
|
|
91
|
+
* repairs. Prefer under-filtering here: a tap on a present-but-odd element fails loudly,
|
|
92
|
+
* whereas a selector that silently matches nothing looks like app drift and sends the
|
|
93
|
+
* heal loop chasing it. Same rule for any attribute added to STATE_ATTRS — do not
|
|
94
|
+
* strengthen a predicate with a conjunct the platform reports unreliably (`--not-checked`
|
|
95
|
+
* deliberately does NOT also require `checkable`). */
|
|
96
|
+
function filterByState(elements, sel) {
|
|
97
|
+
const pinned = exports.STATE_ATTRS.filter((attr) => sel[attr] !== undefined);
|
|
98
|
+
if (pinned.length === 0)
|
|
99
|
+
return elements;
|
|
100
|
+
return elements.filter((el) => pinned.every((attr) => el[attr] === sel[attr]));
|
|
101
|
+
}
|
|
38
102
|
const norm = (s) => s.trim().toLowerCase();
|
|
39
103
|
const strip = (s) => s.toLowerCase().replace(/[^a-z0-9]+/g, '');
|
|
40
104
|
/** The ordered match tiers for a selector. First tier with a hit wins. */
|
|
@@ -91,8 +155,7 @@ function tiers(sel) {
|
|
|
91
155
|
function matchElements(elements, sel) {
|
|
92
156
|
// Applied BEFORE the tier ladder, not after: filtering the candidate pool keeps a
|
|
93
157
|
// disabled exact match from shadowing an enabled partial one.
|
|
94
|
-
|
|
95
|
-
elements = elements.filter(isActionable);
|
|
158
|
+
elements = filterByState(elements, sel);
|
|
96
159
|
for (const { tier, test } of tiers(sel)) {
|
|
97
160
|
const found = elements.filter(test);
|
|
98
161
|
if (found.length === 0)
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.unsupportedStateAttrs = unsupportedStateAttrs;
|
|
4
|
+
exports.assertStateSupported = assertStateSupported;
|
|
5
|
+
const errors_1 = require("../errors");
|
|
6
|
+
const selector_1 = require("./selector");
|
|
7
|
+
const ios_parse_1 = require("./ios-parse");
|
|
8
|
+
// Which state attributes a platform's accessibility backend can actually answer for.
|
|
9
|
+
//
|
|
10
|
+
// A state modifier narrows the candidate pool, so one the platform never populates does
|
|
11
|
+
// not fail — it matches NOTHING, quietly, after burning the full auto-wait window, and
|
|
12
|
+
// then reports "No element matched selector", which is a claim about the screen that
|
|
13
|
+
// isn't true. That is the same false-signal failure mode `--selected` was added to fix,
|
|
14
|
+
// so it is refused here instead: an unsupported modifier is an environment error (exit
|
|
15
|
+
// 3), the way `clearApp` and `currentApp` already refuse on iOS rather than pretending.
|
|
16
|
+
//
|
|
17
|
+
// This is a capability table, not device I/O — the per-platform truth lives with the
|
|
18
|
+
// parser that hard-codes the value (see IOS_UNREPORTED_STATE), so the two cannot drift.
|
|
19
|
+
const UNREPORTED = {
|
|
20
|
+
android: [], // uiautomator dumps all four as real node attributes
|
|
21
|
+
ios: ios_parse_1.IOS_UNREPORTED_STATE,
|
|
22
|
+
};
|
|
23
|
+
/** State attributes this selector pins that the platform cannot report. */
|
|
24
|
+
function unsupportedStateAttrs(sel, platform) {
|
|
25
|
+
const blind = UNREPORTED[platform];
|
|
26
|
+
if (blind.length === 0)
|
|
27
|
+
return [];
|
|
28
|
+
return selector_1.STATE_ATTRS.filter((attr) => sel[attr] !== undefined && blind.includes(attr));
|
|
29
|
+
}
|
|
30
|
+
/** Throw unless every state modifier on `sel` means something on `platform`. */
|
|
31
|
+
function assertStateSupported(sel, platform) {
|
|
32
|
+
const bad = unsupportedStateAttrs(sel, platform);
|
|
33
|
+
if (bad.length === 0)
|
|
34
|
+
return;
|
|
35
|
+
// Name both polarities: --not-selected is refused for the same reason as --selected,
|
|
36
|
+
// and echoing only the attribute reads as though the wrong flag was typed.
|
|
37
|
+
const named = bad.map((a) => `--${a}/--not-${a}`).join(' and ');
|
|
38
|
+
throw new errors_1.CliError(`${named} cannot be used on ${platform}: its accessibility backend does not report ` +
|
|
39
|
+
`${bad.join(' or ')}, so the selector could only ever match nothing. ` +
|
|
40
|
+
`Match on ${platform === 'ios' ? '@id, text or --enabled/--checked' : 'another attribute'} instead.`, 3);
|
|
41
|
+
}
|
package/dist/version.js
CHANGED
|
@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
3
3
|
exports.VERSION = void 0;
|
|
4
4
|
// GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
|
|
5
5
|
// (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
|
|
6
|
-
exports.VERSION = '0.
|
|
6
|
+
exports.VERSION = '0.17.0';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "verikun",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.17.0",
|
|
4
4
|
"description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"android",
|