verikun 0.13.0 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -399,6 +399,63 @@ If a selector for an action matches more than one element and no `--index` is
399
399
  given, the command fails with exit code 2 and lists the candidates — it never
400
400
  taps a guess.
401
401
 
402
+ ### Auto scroll-into-view
403
+
404
+ An element can be in the hierarchy without being reachable at the point a tap
405
+ would land: scrolled past the edge of its list, or with a sticky bar drawn across
406
+ its middle. Pressing its coordinates then hits whatever is actually there — and
407
+ the step reports success, so the run continues from the wrong place.
408
+
409
+ So **actions scroll; inspection does not**. `tap` and `text` bring their target
410
+ into the clear first — into its scroll container, and out from under anything
411
+ drawn over it — then act, adding `(scrolled into view: N swipes)` to the
412
+ confirmation. "Scroll down to X and tap it" is therefore just `vk tap X`; you
413
+ rarely need an explicit `swipe`.
414
+
415
+ `ui`, `find` and `assert` never scroll and never hide anything: an element with no
416
+ pixel on screen is listed as usual, marked `offscreen`. Where an element cannot be
417
+ reached at all, the action **fails with exit 1** rather than pressing blind
418
+ coordinates. `--no-scroll` opts out of the scrolling.
419
+
420
+ > Note what this cannot see: a control covered by something the accessibility tree
421
+ > does not contain (a decorative container with no label or id) is invisible to any
422
+ > tool reading that tree. Scrolling the target clear of screen edges is what
423
+ > avoids most of these; verikun warns on stderr when it presses an element it
424
+ > believes is covered.
425
+
426
+ ### Which selector to reach for: `@id` first, `text:` second, `desc:` never
427
+
428
+ Not all four selector kinds travel equally well. If a flow has to run on both
429
+ Android and iOS, this ordering matters:
430
+
431
+ | selector | Android | iOS | portable? |
432
+ |---|---|---|---|
433
+ | `@id` | `resource-id` | `AXUniqueId` | **yes — always prefer this** |
434
+ | `text:` | visible text, falling back to `content-desc` | `AXLabel` / `title` / `AXValue` | yes |
435
+ | `desc:` | `content-desc` | `accessibilityHint` only | **no — Android in practice** |
436
+ | `class:` | widget class | element role | no — see below |
437
+
438
+ Two traps worth knowing:
439
+
440
+ - **`desc:` does not fall back.** `text:` falls back to `desc` when no text
441
+ matches, so a `text:Submit` selector finds an element carrying only an
442
+ accessibility label. The reverse is not true — `desc:Submit` will never match
443
+ visible text. On iOS an accessibility label arrives as `text`, so a `desc:`
444
+ selector written against Android silently stops matching there.
445
+ - **`class:` is mostly useless on a cross-platform UI toolkit.** Flutter text
446
+ inputs report as `android.widget.EditText` / `TextField`, but almost everything
447
+ else is `android.view.View` — so `class:Button` cannot match a Flutter button
448
+ regardless of what the widget is.
449
+
450
+ There is a further, sharper reason to prefer `@id`: it is the only selector that
451
+ is not text, so it survives **localisation**. A flow pinned with `text:` breaks
452
+ the moment the device is in a different language.
453
+
454
+ For a Flutter app, `@id` comes from `Semantics(identifier:)`; `Semantics(label:)`
455
+ gives you `desc` on Android but `text` on iOS. A worked example, with the
456
+ cross-platform gotchas measured on real hardware, is in
457
+ [`example/flutter-app`](example/flutter-app/).
458
+
402
459
  ## Auto-wait
403
460
 
404
461
  A UI rarely settles the instant the previous action returns. So selector
@@ -577,6 +634,20 @@ vk tap @tap_to_continue_label_id
577
634
  **Cost:** $0.45 · **Wall time:** ~4 min · **Model:** Claude Sonnet 4.6 with
578
635
  prompt-cache hits (1 M cache-read tokens kept cost low on a long conversation).
579
636
 
637
+ ## Feedback — help improve verikun
638
+
639
+ verikun improves from the rough edges people hit while driving it. When verikun *itself* is
640
+ the friction — a step that heals on every cached replay (an unstable compiled selector,
641
+ often a label-only control with no resource-id), a repair "give-up", or a gotcha in its own
642
+ operation — that's worth an issue at
643
+ [github.com/ddikman/verikun/issues](https://github.com/ddikman/verikun/issues).
644
+
645
+ Driving verikun with an AI agent + the [skill](.claude/skills/verikun/SKILL.md)? It hands
646
+ off to the **`suggest-verikun-improvement`** skill, which drafts a short, TL;DR-first
647
+ suggestion, **reviews it with you before anything is submitted**, and **redacts every
648
+ app-under-test specific** (package, on-screen text, selector values, test prose, logs) so no
649
+ client code or logic can leak.
650
+
580
651
  ## Build from source
581
652
 
582
653
  For local development, or to run an unreleased version, build from a clone:
@@ -20,7 +20,7 @@ Each step is one of three node types:
20
20
  rerun starts FRESH (--clear also wipes data → fresh-install;
21
21
  --no-restart skips the force-stop, just bringing it forward)
22
22
  stop <package> — force-stop the app
23
- tap <selector> — tap the element a selector resolves to
23
+ tap <selector> — tap the element a selector resolves to (scrolls it into view first)
24
24
  text <selector> <value...> — focus a field and type value (--clear to clear first, --enter to submit)
25
25
  type <value...> — type into the already-focused field
26
26
  key <name> | back | home | enter
@@ -35,9 +35,9 @@ Each step is one of three node types:
35
35
  keep a flow from breaking when an extra screen sometimes appears.
36
36
 
37
37
  3. REPEAT — { "type":"repeat", "selector":<sel>, "cap":<n>, "body":[<nodes>] }
38
- Repeat body UNTIL the selector appears, up to cap iterations. Use for "scroll until X
39
- is visible", or "keep answering until the results screen". Always set a sane cap (e.g.
40
- 10). The engine also stops early if the screen stops changing. A repeat that finishes
38
+ Repeat body UNTIL the selector appears, up to cap iterations. Use for "keep answering
39
+ until the results screen" NOT for scrolling to something that is already in the
40
+ hierarchy, since tap scrolls to its own target. Always set a sane cap (e.g. 10). The engine also stops early if the screen stops changing. A repeat that finishes
41
41
  without its selector ever appearing FAILS the test — it did not do its job.
42
42
 
43
43
  4. WHEN — { "type":"when", "branches":[{ "selector":<sel>, "body":[<nodes>] }, ...],
@@ -120,6 +120,11 @@ RULES:
120
120
  the failure surfaces later as a confusing timeout on the NEXT step.
121
121
  - assert is for VERIFICATION only and is terminal — never use it as a step you expect to
122
122
  fail. Put genuinely-optional UI behind if-present.
123
+ - tap/text SCROLL THEIR TARGET INTO VIEW automatically, so "scroll down to X and tap it"
124
+ is just \`tap X\`. Do NOT wrap a tap in a repeat-until-visible loop to reach something
125
+ below the fold — that is now redundant. Emit an explicit swipe only when the SCROLLING
126
+ ITSELF is what the test asks for ("scroll the feed three times"), or to reveal content
127
+ that is not in the hierarchy until it is built (an infinite/lazy list).
123
128
  - Prefer resource-id / accessibility selectors over visible text where possible.
124
129
  - Translate the test literally and minimally: do not invent ACTION steps (tap/text/swipe/key/assert)
125
130
  the prose does not imply. The ONE exception is screenshot — insert screenshot steps liberally as
@@ -149,4 +154,7 @@ Emit ONLY an object matching the schema:
149
154
  { "decision":"repair", "step": { "type":"command","command","positionals":[...],"flags":[{"name","value"}] } }
150
155
  { "decision":"give_up", "reason": "<why no element on this screen matches the intent>" }
151
156
  Prefer a stable selector (resource-id / accessibility label) visible in the hierarchy.
152
- Do not invent elements that are not in the hierarchy.`;
157
+ Do not invent elements that are not in the hierarchy.
158
+ An element tagged \`offscreen\` is in the tree but scrolled out of view; tap/text scroll
159
+ to their target on their own, so the fact a step failed on one means scrolling could not
160
+ reach it — pick a different element only if one genuinely serves the same purpose.`;
package/dist/args.js CHANGED
@@ -35,6 +35,7 @@ const BOOLEAN = new Set([
35
35
  'android',
36
36
  'fix',
37
37
  'no-wait',
38
+ 'no-scroll',
38
39
  'full',
39
40
  'more',
40
41
  'show-plan',
package/dist/cli.js CHANGED
@@ -40,6 +40,7 @@ exports.healNote = healNote;
40
40
  exports.parseDuration = parseDuration;
41
41
  exports.waitWindowMs = waitWindowMs;
42
42
  exports.waitNote = waitNote;
43
+ exports.scrollNote = scrollNote;
43
44
  exports.formatDeviceTable = formatDeviceTable;
44
45
  exports.guardSettleMs = guardSettleMs;
45
46
  exports.confineToCwd = confineToCwd;
@@ -58,6 +59,7 @@ const exec_1 = require("./exec");
58
59
  const drivers_1 = require("./drivers");
59
60
  const selector_1 = require("./ui/selector");
60
61
  const format_1 = require("./ui/format");
62
+ const viewport_1 = require("./ui/viewport");
61
63
  const output_1 = require("./output");
62
64
  const run_1 = require("./run");
63
65
  const image_1 = require("./image");
@@ -173,7 +175,10 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
173
175
  const els = ctx.driver.getElements(opts);
174
176
  if ((0, selector_1.matchElements)(els, sel).matches.length >= 1) {
175
177
  const { element, tier } = (0, selector_1.resolveOne)(els, sel); // 1 → resolved; >1 → throws ambiguity
176
- return { element, tier, waitedMs: Date.now() - start };
178
+ // The snapshot rides along: scroll-into-view needs the scrollable containers
179
+ // from the SAME dump the element came from, and re-capturing to find them
180
+ // would both cost a round-trip and risk describing a screen that moved on.
181
+ return { element, tier, waitedMs: Date.now() - start, elements: els };
177
182
  }
178
183
  if (Date.now() >= deadline) {
179
184
  const waited = windowMs > 0 ? ` after ${(windowMs / 1000).toFixed(1)}s` : '';
@@ -182,6 +187,146 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
182
187
  await sleep(pollStep(ctx.flags, deadline));
183
188
  }
184
189
  }
190
+ // --- Auto scroll-into-view --------------------------------------------------
191
+ // An element's centre is not always a point that reaches it, and a tap on the
192
+ // wrong point still reported success — so the run carried on from the wrong place
193
+ // and failed several steps later on an unrelated symptom (issue #42). Actions
194
+ // therefore bring their target into the clear first, the same contract as
195
+ // Playwright's scrollIntoViewIfNeeded.
196
+ //
197
+ // MEASURED, and it is not only the obvious case: Android's dumper already drops
198
+ // nodes it considers invisible and clips the rest to the display, so the usual
199
+ // shape is an element that IS on screen — a row cut off by its list, or one with a
200
+ // sticky bar drawn across its middle. Both are handled by asking the question
201
+ // against `clipRegion()` (screen ∩ scroll container) and `isOccluded()` (what is
202
+ // painted after it), not against the screen alone.
203
+ //
204
+ // Load-bearing split, mirroring auto-wait: ACTIONS scroll, INSPECTION does not.
205
+ // `ui`/`find`/`assert` report an element exactly as it is (tagged `offscreen` when
206
+ // it has no pixel on screen) — hiding it would turn a wrong tap into a mysterious
207
+ // miss — while `tap`/`text` refuse to press a point that would hit something else.
208
+ //
209
+ // Scrolling only ever happens when the alternative is a wrong tap, so a target
210
+ // already in the clear costs nothing: no extra dump, no swipe.
211
+ const SCROLL_INTO_VIEW_MAX = 10;
212
+ /** Let the scroll settle before re-dumping — a mid-fling hierarchy reads as a stall. */
213
+ const SCROLL_SETTLE_MS = 500;
214
+ /** Movement below this is measurement noise, not progress. */
215
+ const NO_PROGRESS_PX = 8;
216
+ /** Consecutive non-moving swipes before we accept the list will not go further. */
217
+ const NO_PROGRESS_STRIKES = 2;
218
+ /** A short note appended to action output when the target had to be scrolled to. */
219
+ function scrollNote(swipes) {
220
+ return swipes > 0 ? ` (scrolled into view: ${swipes} swipe${swipes === 1 ? '' : 's'})` : '';
221
+ }
222
+ /** Swipe until `target` sits fully inside its clip region, or we run out of room. */
223
+ async function scrollIntoView(ctx, sel, target, elements, screen, opts = {}) {
224
+ let current = target;
225
+ let snapshot = elements;
226
+ let swipes = 0;
227
+ let stalled = 0;
228
+ while (swipes < SCROLL_INTO_VIEW_MAX && stalled < NO_PROGRESS_STRIKES) {
229
+ // Re-derived every iteration: scrolling can change which container holds the
230
+ // element, and a stale clip would aim the next swipe at the wrong box.
231
+ const clip = (0, viewport_1.clipRegion)(snapshot, current, screen);
232
+ const axis = Math.abs(current.center.y - (clip.y1 + clip.y2) / 2) >= Math.abs(current.center.x - (clip.x1 + clip.x2) / 2)
233
+ ? 'y'
234
+ : 'x';
235
+ // A covered element is scrolled to the MIDDLE of its container even though it is
236
+ // technically in view: a sticky bar overlaps the edges of a list, and moving the
237
+ // target away from them is the one reliable way to get a touch through to it.
238
+ const centre = (0, viewport_1.isOccluded)(snapshot, current, (0, viewport_1.tapPoint)(current, screen));
239
+ const plan = (0, viewport_1.scrollPlan)(current.bounds, (0, viewport_1.scrollSurface)(snapshot, current, screen, axis), clip, { centre });
240
+ if (!plan)
241
+ break; // in view, or no swipe big enough to be worth making
242
+ ctx.driver.swipe(plan.from.x, plan.from.y, plan.to.x, plan.to.y, (0, viewport_1.swipeDurationMs)(plan.distance));
243
+ swipes++;
244
+ await sleep(SCROLL_SETTLE_MS);
245
+ const before = current.bounds;
246
+ snapshot = ctx.driver.getElements(opts);
247
+ // An empty tree is a bad read, not a screen (the device returns partial dumps
248
+ // mid-transition) — retry rather than conclude the element is gone.
249
+ if (snapshot.length === 0)
250
+ continue;
251
+ if ((0, selector_1.matchElements)(snapshot, sel).matches.length === 0) {
252
+ // Android drops a node that scrolls out of view, so overshooting LOSES the
253
+ // target rather than leaving it visibly off-position. Give the last swipe back
254
+ // (half of it, to land between the two) and look once more before giving up.
255
+ const mid = { x: Math.round((plan.from.x + plan.to.x) / 2), y: Math.round((plan.from.y + plan.to.y) / 2) };
256
+ ctx.driver.swipe(plan.to.x, plan.to.y, mid.x, mid.y, (0, viewport_1.swipeDurationMs)(plan.distance / 2));
257
+ await sleep(SCROLL_SETTLE_MS);
258
+ snapshot = ctx.driver.getElements(opts);
259
+ if ((0, selector_1.matchElements)(snapshot, sel).matches.length === 0) {
260
+ throw new errors_1.SelectorNotFoundError(`'${sel.raw}' left the hierarchy while being scrolled into view (after ${swipes} swipe(s)) — ` +
261
+ 'a lazy list may have recycled it. Run `verikun ui` to inspect the current screen.');
262
+ }
263
+ }
264
+ current = (0, selector_1.resolveOne)(snapshot, sel).element; // >1 → ambiguity, exit 2, as everywhere else
265
+ const moved = Math.abs(plan.axis === 'y' ? current.bounds.y1 - before.y1 : current.bounds.x1 - before.x1);
266
+ stalled = moved < NO_PROGRESS_PX ? stalled + 1 : 0;
267
+ }
268
+ return { element: current, swipes, elements: snapshot };
269
+ }
270
+ /** The screen as a rectangle, or null on a device whose size could not be read. */
271
+ function screenOf(ctx) {
272
+ const vp = ctx.driver.viewport();
273
+ return vp ? (0, viewport_1.screenRect)(vp) : null;
274
+ }
275
+ /** The point to press for `el`, preferring one nothing else is drawn over. Falls back
276
+ * to the visible centre — an ordering-based guess must never block a real tap. */
277
+ function pressPoint(els, el, screen) {
278
+ if (!screen)
279
+ return el.center;
280
+ return (0, viewport_1.reachablePoint)(els, el, screen) ?? (0, viewport_1.tapPoint)(el, screen);
281
+ }
282
+ /** Say so when the element we are about to press is only partly on screen, or is
283
+ * covered by something drawn over it — both mean the touch may not reach it. */
284
+ function reachWarning(els, el, screen) {
285
+ if (!screen)
286
+ return;
287
+ if (!(0, viewport_1.isFullyVisible)(el.bounds, screen)) {
288
+ (0, output_1.err)(`(only ${Math.round((0, viewport_1.visibleFraction)(el.bounds, screen) * 100)}% of ${(0, format_1.formatInline)(el)} is on screen)`);
289
+ }
290
+ if (!(0, viewport_1.reachablePoint)(els, el, screen)) {
291
+ (0, output_1.err)(`(${(0, format_1.formatInline)(el)} is covered by another element — the tap may land on whatever is on top)`);
292
+ }
293
+ }
294
+ /**
295
+ * Resolve a selector to something that can actually be pressed: wait for it, scroll it
296
+ * into view when it is not fully inside its scroll container, and fail loudly rather
297
+ * than tap blind.
298
+ *
299
+ * When the screen size is unknown this is exactly the old behaviour — resolve and
300
+ * press the element's centre.
301
+ */
302
+ async function resolveTappable(ctx, sel, opts = {}) {
303
+ const { element, tier, waitedMs, elements } = await resolveOneWaiting(ctx, sel, opts);
304
+ const screen = screenOf(ctx);
305
+ const clip = screen ? (0, viewport_1.clipRegion)(elements, element, screen) : null;
306
+ // Scroll when the element is cut off by its container, and also when something is
307
+ // drawn over the point we would press — but only if it HAS a container to scroll
308
+ // (clip !== screen). Swiping the whole screen at a covered toolbar button would be
309
+ // a random gesture, and the point-picking fallback below handles that case.
310
+ const covered = !!screen && !!clip && clip !== screen && (0, viewport_1.isOccluded)(elements, element, (0, viewport_1.tapPoint)(element, screen));
311
+ if (!screen || !clip || ((0, viewport_1.isFullyVisible)(element.bounds, clip) && !covered)) {
312
+ reachWarning(elements, element, screen);
313
+ return { element, tier, waitedMs, swipes: 0, point: pressPoint(elements, element, screen) };
314
+ }
315
+ const scrolled = (0, args_1.flagBool)(ctx.flags, 'no-scroll')
316
+ ? { element, swipes: 0, elements }
317
+ : await scrollIntoView(ctx, sel, element, elements, screen, opts);
318
+ if ((0, viewport_1.isOffscreen)(scrolled.element.bounds, screen)) {
319
+ const why = (0, args_1.flagBool)(ctx.flags, 'no-scroll')
320
+ ? '--no-scroll is set'
321
+ : scrolled.swipes === 0
322
+ ? 'no scrollable container could move it'
323
+ : `${scrolled.swipes} swipe(s) did not bring it into view`;
324
+ throw new errors_1.SelectorNotFoundError(`'${sel.raw}' is in the screen's element tree but scrolled out of view (${why}), so tapping it ` +
325
+ 'would press whatever is at those coordinates instead. Run `verikun ui` to inspect the current screen.');
326
+ }
327
+ reachWarning(scrolled.elements, scrolled.element, screen);
328
+ return { ...scrolled, tier, waitedMs, point: pressPoint(scrolled.elements, scrolled.element, screen) };
329
+ }
185
330
  // ---------------------------------------------------------------------------
186
331
  // Commands
187
332
  // ---------------------------------------------------------------------------
@@ -346,22 +491,39 @@ async function cmdTap(ctx) {
346
491
  let target;
347
492
  let tier = null;
348
493
  let waitedMs = 0;
494
+ let swipes = 0;
495
+ let point;
349
496
  if (isBareIndex) {
350
497
  const els = ctx.driver.getElements({ all: (0, args_1.flagBool)(ctx.flags, 'all') });
351
498
  const idx = Number(raw);
352
499
  const found = els.find((e) => e.index === idx);
353
500
  if (!found)
354
501
  throw new errors_1.CliError(`No element with index [${idx}] on the current screen. Run \`verikun ui\`.`, 1);
502
+ // An index names a row of one specific dump, and scrolling renumbers every row —
503
+ // so this path cannot scroll. Refuse instead of pressing coordinates off-screen.
504
+ if (found.offscreen) {
505
+ throw new errors_1.CliError(`Element [${idx}] ${(0, format_1.formatInline)(found)} is scrolled out of view. Tap it by selector ` +
506
+ `(e.g. \`verikun tap @${found.idShort || 'id'}\`) so verikun can scroll it into view first.`, 1);
507
+ }
355
508
  target = found;
509
+ reachWarning(els, target, screenOf(ctx));
510
+ point = pressPoint(els, target, screenOf(ctx));
356
511
  ctx.record?.note({ element: target, message: `tapped by index [${idx}]` });
357
512
  }
358
513
  else {
359
514
  const sel = buildSelector(raw, ctx.flags);
360
- ({ element: target, tier, waitedMs } = await resolveOneWaiting(ctx, sel, { all: (0, args_1.flagBool)(ctx.flags, 'all') }));
361
- ctx.record?.note({ selector: sel, tier, element: target });
515
+ ({ element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel, {
516
+ all: (0, args_1.flagBool)(ctx.flags, 'all'),
517
+ }));
518
+ ctx.record?.note({
519
+ selector: sel,
520
+ tier,
521
+ element: target,
522
+ message: swipes > 0 ? `scrolled into view (${swipes} swipe(s)) and tapped` : undefined,
523
+ });
362
524
  }
363
- ctx.driver.tap(target.center.x, target.center.y);
364
- (0, output_1.out)(`tapped ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}`);
525
+ ctx.driver.tap(point.x, point.y);
526
+ (0, output_1.out)(`tapped ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}${scrollNote(swipes)}`);
365
527
  return 0;
366
528
  }
367
529
  async function cmdText(ctx) {
@@ -370,14 +532,14 @@ async function cmdText(ctx) {
370
532
  }
371
533
  const sel = buildSelector(ctx.positionals[0], ctx.flags);
372
534
  const value = ctx.positionals.slice(1).join(' ');
373
- const { element: target, tier, waitedMs } = await resolveOneWaiting(ctx, sel);
535
+ const { element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel);
374
536
  ctx.record?.note({
375
537
  selector: sel,
376
538
  tier,
377
539
  element: target,
378
540
  message: target.password ? 'typed «redacted»' : `typed ${JSON.stringify(value)}`,
379
541
  });
380
- ctx.driver.tap(target.center.x, target.center.y);
542
+ ctx.driver.tap(point.x, point.y);
381
543
  // Wait for field to be focused after tap
382
544
  await sleep(100);
383
545
  if ((0, args_1.flagBool)(ctx.flags, 'clear') && target.text) {
@@ -394,7 +556,7 @@ async function cmdText(ctx) {
394
556
  ctx.driver.inputText(value);
395
557
  if ((0, args_1.flagBool)(ctx.flags, 'enter'))
396
558
  ctx.driver.pressKey('enter');
397
- (0, output_1.out)(`typed ${JSON.stringify(value)} into ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}`);
559
+ (0, output_1.out)(`typed ${JSON.stringify(value)} into ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}${scrollNote(swipes)}`);
398
560
  return 0;
399
561
  }
400
562
  function cmdType(ctx) {
@@ -454,33 +616,11 @@ async function cmdSwipe(ctx) {
454
616
  const { width, height } = ctx.driver.screenSize();
455
617
  region = { x1: 0, y1: 0, x2: width, y2: height };
456
618
  }
457
- const cx = Math.floor((region.x1 + region.x2) / 2);
458
- const cy = Math.floor((region.y1 + region.y2) / 2);
459
- const frac = Math.min(Math.max((0, args_1.flagNum)(ctx.flags, 'distance') ?? 0.6, 0.1), 0.95);
460
- const dx = Math.floor(((region.x2 - region.x1) * frac) / 2);
461
- const dy = Math.floor(((region.y2 - region.y1) * frac) / 2);
462
- let a;
463
- let b;
464
- switch (dir) {
465
- case 'up':
466
- a = { x: cx, y: cy + dy };
467
- b = { x: cx, y: cy - dy };
468
- break;
469
- case 'down':
470
- a = { x: cx, y: cy - dy };
471
- b = { x: cx, y: cy + dy };
472
- break;
473
- case 'left':
474
- a = { x: cx + dx, y: cy };
475
- b = { x: cx - dx, y: cy };
476
- break;
477
- case 'right':
478
- a = { x: cx - dx, y: cy };
479
- b = { x: cx + dx, y: cy };
480
- break;
481
- default:
482
- throw new errors_1.CliError(`Unknown direction '${dir}' (use up|down|left|right)`, 2);
619
+ if (dir !== 'up' && dir !== 'down' && dir !== 'left' && dir !== 'right') {
620
+ throw new errors_1.CliError(`Unknown direction '${dir}' (use up|down|left|right)`, 2);
483
621
  }
622
+ const frac = Math.min(Math.max((0, args_1.flagNum)(ctx.flags, 'distance') ?? viewport_1.DEFAULT_SWIPE_FRACTION, 0.1), 0.95);
623
+ const { from: a, to: b } = (0, viewport_1.swipeVector)(region, dir, frac);
484
624
  ctx.driver.swipe(a.x, a.y, b.x, b.y, duration);
485
625
  ctx.record?.note({ message: `swiped ${dir}${on ? ` on ${on}` : ''}` });
486
626
  (0, output_1.out)(`swiped ${dir}${waitNote(waitedMs)}`);
@@ -1176,6 +1316,7 @@ async function runAiTest(file, opts, backend, platform, device) {
1176
1316
  (0, output_1.err)(`[ai] cost ceiling $${opts.maxCostUsd} reached during compile (${cost.summaryLine()}) — not running`);
1177
1317
  return {
1178
1318
  ok: false,
1319
+ cached,
1179
1320
  costUsd: Number(cost.usd().toFixed(4)),
1180
1321
  costLine: cost.summaryLine(),
1181
1322
  modelRepairs: 0,
@@ -1267,6 +1408,7 @@ async function runAiTest(file, opts, backend, platform, device) {
1267
1408
  (0, output_1.err)(`[ai] estimated total cost: $${cost.usd().toFixed(4)}`);
1268
1409
  return {
1269
1410
  ok: result.ok,
1411
+ cached,
1270
1412
  costUsd: Number(cost.usd().toFixed(4)),
1271
1413
  costLine,
1272
1414
  modelRepairs: result.modelRepairs,
@@ -1308,6 +1450,7 @@ async function cmdAi(positionals, flags) {
1308
1450
  if ((0, args_1.flagBool)(flags, 'json')) {
1309
1451
  (0, output_1.json)({
1310
1452
  ok: result.ok,
1453
+ cached: result.cached,
1311
1454
  model: opts.model,
1312
1455
  cost: result.costLine,
1313
1456
  costUsd: result.costUsd,
@@ -1727,6 +1870,15 @@ AUTO-WAIT (selector lookups retry until they resolve)
1727
1870
  Ambiguity is never waited on (the elements are already there). The \`wait\`
1728
1871
  command stays for explicit polling, including --gone, with --timeout/--interval.
1729
1872
 
1873
+ AUTO-SCROLL (actions bring their target into view)
1874
+ \`tap\` / \`text\` first scroll their target into the clear — inside its scroll
1875
+ container, and out from under anything drawn over it (a sticky bar) — then act,
1876
+ so "scroll down then tap X" is just \`tap X\`. \`ui\` / \`find\` / \`assert\` never
1877
+ scroll and hide nothing: an element with no pixel on screen is listed as usual,
1878
+ marked \`offscreen\`. One that cannot be reached fails with exit 1 rather than
1879
+ being tapped at coordinates that would hit something else.
1880
+ --no-scroll act where the element is; do not scroll to it
1881
+
1730
1882
  GLOBAL FLAGS
1731
1883
  -d, --device <serial> target a specific device (or VERIKUN_DEVICE / ANDROID_SERIAL)
1732
1884
  -p, --platform <android|ios> (default: android; --ios / --android shortcuts)
@@ -6,6 +6,7 @@ exports.escapeText = escapeText;
6
6
  const errors_1 = require("../errors");
7
7
  const exec_1 = require("../exec");
8
8
  const android_parse_1 = require("../ui/android-parse");
9
+ const viewport_1 = require("../ui/viewport");
9
10
  const ADB = process.env.ADB || 'adb';
10
11
  const ADB_HINT = 'install the Android platform-tools (`brew install --cask android-platform-tools`), or point ADB at the binary';
11
12
  /** Is `adb` present and runnable? Shared by `vk doctor` and AdbDriver.preflight() so
@@ -96,6 +97,11 @@ class AdbDriver {
96
97
  platform = 'android';
97
98
  requested;
98
99
  cachedSerial;
100
+ /** null = asked and failed. Cached either way: getElements runs on a ~300ms poll
101
+ * during auto-wait, and a broken `wm size` must not cost a round-trip every time. */
102
+ cachedScreen;
103
+ /** Rotation of the most recent dump — see viewport(). */
104
+ lastRotation;
99
105
  constructor(serial) {
100
106
  this.requested = serial;
101
107
  }
@@ -178,7 +184,28 @@ class AdbDriver {
178
184
  return (0, exec_1.runText)(ADB, this.withSerial(['shell', ...args]), { timeout }).stdout;
179
185
  }
180
186
  getElements(opts = {}) {
181
- return (0, android_parse_1.parseHierarchy)(this.dumpXml(), opts);
187
+ const xml = this.dumpXml();
188
+ // Read off the dump we already have: free, and it refreshes every capture, so a
189
+ // device rotated mid-run is handled without re-asking for the screen size.
190
+ this.lastRotation = (0, android_parse_1.parseRotation)(xml);
191
+ return (0, android_parse_1.parseHierarchy)(xml, { ...opts, screen: this.screenOrNull() ?? undefined });
192
+ }
193
+ viewport() {
194
+ const screen = this.screenOrNull();
195
+ return screen ? (0, viewport_1.viewportFor)(screen, this.lastRotation) : null;
196
+ }
197
+ /** screenSize() memoized, failure included — the parser uses it to mark elements
198
+ * scrolled out of view, and "we could not tell" must degrade to "all visible". */
199
+ screenOrNull() {
200
+ if (this.cachedScreen === undefined) {
201
+ try {
202
+ this.cachedScreen = this.screenSize();
203
+ }
204
+ catch {
205
+ this.cachedScreen = null;
206
+ }
207
+ }
208
+ return this.cachedScreen;
182
209
  }
183
210
  dumpXml() {
184
211
  let lastErr = '';
@@ -10,6 +10,7 @@ const node_fs_1 = require("node:fs");
10
10
  const errors_1 = require("../errors");
11
11
  const exec_1 = require("../exec");
12
12
  const ios_parse_1 = require("../ui/ios-parse");
13
+ const viewport_1 = require("../ui/viewport");
13
14
  // iOS driver. `xcrun simctl` / `devicectl` cover device discovery and — on a
14
15
  // simulator — screenshots, app lifecycle, and logs (no extra install needed).
15
16
  // Everything interactive (UI hierarchy, tap, type, swipe, keys, screen size) and
@@ -147,6 +148,9 @@ class IdbDriver {
147
148
  requested;
148
149
  cachedSerial;
149
150
  cachedIsSim;
151
+ /** null = asked and failed. Memoized because screenSize()'s fallback path runs a
152
+ * SECOND full hierarchy dump, which auto-wait would otherwise pay every poll. */
153
+ cachedScreen;
150
154
  constructor(device) {
151
155
  // 'booted' is a simctl-only alias idb can't address, so treat it as "auto-resolve".
152
156
  this.requested = device && device !== 'booted' ? device : undefined;
@@ -250,7 +254,28 @@ class IdbDriver {
250
254
  getElements(opts = {}) {
251
255
  // `idb ui describe-all` prints the accessibility tree as JSON (array or NDJSON);
252
256
  // parseIosHierarchy handles either.
253
- return (0, ios_parse_1.parseIosHierarchy)(this.idbText(['ui', 'describe-all'], { timeout: 15000 }), opts);
257
+ return (0, ios_parse_1.parseIosHierarchy)(this.idbText(['ui', 'describe-all'], { timeout: 15000 }), {
258
+ ...opts,
259
+ screen: this.screenOrNull() ?? undefined,
260
+ });
261
+ }
262
+ viewport() {
263
+ const screen = this.screenOrNull();
264
+ // No orientation signal from idb, so viewportFor uses the max(w,h) square: exact
265
+ // on the vertical axis lists scroll on, permissive across it.
266
+ return screen ? (0, viewport_1.viewportFor)(screen) : null;
267
+ }
268
+ /** screenSize() memoized, failure included — see AdbDriver.screenOrNull. */
269
+ screenOrNull() {
270
+ if (this.cachedScreen === undefined) {
271
+ try {
272
+ this.cachedScreen = this.screenSize();
273
+ }
274
+ catch {
275
+ this.cachedScreen = null;
276
+ }
277
+ }
278
+ return this.cachedScreen;
254
279
  }
255
280
  screenshot() {
256
281
  const tmp = (0, node_path_1.join)((0, node_os_1.tmpdir)(), `verikun-ios-${process.pid}.png`);
@@ -1,7 +1,9 @@
1
1
  "use strict";
2
2
  Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.isInteresting = isInteresting;
4
+ exports.parseRotation = parseRotation;
4
5
  exports.parseHierarchy = parseHierarchy;
6
+ const viewport_1 = require("./viewport");
5
7
  // Parses uiautomator XML (from `adb shell uiautomator dump`) into normalized
6
8
  // Element[]. We hand-roll a tiny tag scanner rather than pulling in an XML
7
9
  // dependency: uiautomator output is regular and entity-escaped, so a
@@ -99,6 +101,18 @@ function isInteresting(el) {
99
101
  return true;
100
102
  return false;
101
103
  }
104
+ /**
105
+ * The dump's `<hierarchy rotation="N">` (Surface.ROTATION_*), or undefined if absent.
106
+ *
107
+ * `adb shell wm size` always reports the device's NATURAL orientation, so this is the
108
+ * only thing in the dump that says whether the coordinates are portrait or landscape.
109
+ * Exported so the driver can read it off the same XML it just fetched, rather than
110
+ * paying another round-trip to ask the device which way up it is.
111
+ */
112
+ function parseRotation(xml) {
113
+ const m = /<hierarchy\b[^>]*\brotation="(\d+)"/.exec(xml);
114
+ return m ? Number(m[1]) : undefined;
115
+ }
102
116
  function parseHierarchy(xml, opts = {}) {
103
117
  const all = [];
104
118
  const n = xml.length;
@@ -142,8 +156,13 @@ function parseHierarchy(xml, opts = {}) {
142
156
  }
143
157
  }
144
158
  const result = opts.all ? all : all.filter(isInteresting);
159
+ // No screen size (the driver could not read one) → nothing is marked, which is
160
+ // exactly the behaviour before viewport awareness existed.
161
+ const vp = opts.screen ? (0, viewport_1.screenRect)((0, viewport_1.viewportFor)(opts.screen, parseRotation(xml))) : undefined;
145
162
  result.forEach((el, idx) => {
146
163
  el.index = idx;
164
+ if (vp && (0, viewport_1.isOffscreen)(el.bounds, vp))
165
+ el.offscreen = true;
147
166
  });
148
167
  return result;
149
168
  }
package/dist/ui/format.js CHANGED
@@ -34,6 +34,10 @@ function formatInline(el) {
34
34
  flags.push('selected');
35
35
  if (!el.enabled)
36
36
  flags.push('disabled');
37
+ // Shown rather than hidden: the element IS there, just scrolled away — and an
38
+ // action will scroll it into view. Hiding it would only make the miss mysterious.
39
+ if (el.offscreen)
40
+ flags.push('offscreen');
37
41
  if (flags.length)
38
42
  parts.push(flags.join(','));
39
43
  return parts.join(' ');
@@ -67,5 +71,6 @@ function toJsonShape(el) {
67
71
  password: el.password || undefined,
68
72
  enabled: el.enabled,
69
73
  selected: el.selected || undefined,
74
+ offscreen: el.offscreen || undefined,
70
75
  };
71
76
  }
@@ -2,6 +2,7 @@
2
2
  Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.isInteresting = isInteresting;
4
4
  exports.parseIosHierarchy = parseIosHierarchy;
5
+ const viewport_1 = require("./viewport");
5
6
  // XCUIElementType names (idb `type`) that are meaningful tap targets — used to
6
7
  // derive `clickable`, which iOS accessibility does not expose as a flag.
7
8
  const TAPPABLE_TYPES = new Set([
@@ -111,8 +112,13 @@ function isInteresting(el) {
111
112
  function parseIosHierarchy(jsonText, opts = {}) {
112
113
  const all = parseIdbJson(jsonText).map(buildElement);
113
114
  const result = opts.all ? all : all.filter(isInteresting);
115
+ // idb reports no orientation, so viewportFor falls back to the max(w,h) square —
116
+ // exact vertically (the axis lists scroll on), deliberately permissive across.
117
+ const vp = opts.screen ? (0, viewport_1.screenRect)((0, viewport_1.viewportFor)(opts.screen)) : undefined;
114
118
  result.forEach((el, idx) => {
115
119
  el.index = idx;
120
+ if (vp && (0, viewport_1.isOffscreen)(el.bounds, vp))
121
+ el.offscreen = true;
116
122
  });
117
123
  return result;
118
124
  }
@@ -0,0 +1,267 @@
1
+ "use strict";
2
+ Object.defineProperty(exports, "__esModule", { value: true });
3
+ exports.DEFAULT_SWIPE_FRACTION = void 0;
4
+ exports.viewportFor = viewportFor;
5
+ exports.screenRect = screenRect;
6
+ exports.isOffscreen = isOffscreen;
7
+ exports.visibleRect = visibleRect;
8
+ exports.isFullyVisible = isFullyVisible;
9
+ exports.visibleFraction = visibleFraction;
10
+ exports.tapPoint = tapPoint;
11
+ exports.clipRegion = clipRegion;
12
+ exports.swipeVector = swipeVector;
13
+ exports.scrollSurface = scrollSurface;
14
+ exports.reachablePoint = reachablePoint;
15
+ exports.isOccluded = isOccluded;
16
+ exports.swipeDurationMs = swipeDurationMs;
17
+ exports.scrollPlan = scrollPlan;
18
+ /** How much of a region a directional swipe drags across, as a fraction of its span. */
19
+ exports.DEFAULT_SWIPE_FRACTION = 0.6;
20
+ /** Below this the gesture reads as a tap rather than a drag, so scrolling by less
21
+ * than this is refused — pressing the wrong control is exactly what we are fixing. */
22
+ const MIN_DRAG_PX = 40;
23
+ /** Bounds within a pixel of the viewport edge count as inside: uiautomator rounds,
24
+ * and a 1px overhang is not a reason to swipe a screen. */
25
+ const EDGE_TOLERANCE_PX = 1;
26
+ /**
27
+ * The screen rectangle a dump's coordinates live in.
28
+ *
29
+ * `rotation` is Android's `<hierarchy rotation="N">` (Surface.ROTATION_*): `adb shell
30
+ * wm size` always reports the NATURAL orientation, so on a landscape device (1/3) the
31
+ * axes must be swapped — without that, everything past the portrait width looks
32
+ * off-screen and every tap on a landscape app would try to scroll.
33
+ *
34
+ * `undefined` means no orientation signal (iOS: `idb describe` has none). Rather than
35
+ * guess, use the max(w,h) square: it can never be SMALLER than the true viewport, so
36
+ * it can only under-detect off-screen elements — i.e. degrade to today's behaviour —
37
+ * and never claim a visible element is out of view.
38
+ */
39
+ function viewportFor(screen, rotation) {
40
+ const w = Math.max(0, Math.round(screen.width));
41
+ const h = Math.max(0, Math.round(screen.height));
42
+ if (rotation === undefined) {
43
+ const square = Math.max(w, h);
44
+ return { width: square, height: square };
45
+ }
46
+ return rotation === 1 || rotation === 3 ? { width: h, height: w } : { width: w, height: h };
47
+ }
48
+ /** The screen as a rectangle — the outermost clip region. */
49
+ function screenRect(vp) {
50
+ return { x1: 0, y1: 0, x2: vp.width, y2: vp.height };
51
+ }
52
+ /** No overlap with the region at all — there is no honest point to press. */
53
+ function isOffscreen(b, region) {
54
+ return b.x2 <= region.x1 || b.y2 <= region.y1 || b.x1 >= region.x2 || b.y1 >= region.y2;
55
+ }
56
+ /** The part of `b` inside the region (empty rect when there is no overlap). */
57
+ function visibleRect(b, region) {
58
+ return {
59
+ x1: Math.max(b.x1, region.x1),
60
+ y1: Math.max(b.y1, region.y1),
61
+ x2: Math.min(b.x2, region.x2),
62
+ y2: Math.min(b.y2, region.y2),
63
+ };
64
+ }
65
+ /** Entirely within the region — the state "scroll into view" aims for. */
66
+ function isFullyVisible(b, region) {
67
+ return (b.x1 >= region.x1 - EDGE_TOLERANCE_PX &&
68
+ b.y1 >= region.y1 - EDGE_TOLERANCE_PX &&
69
+ b.x2 <= region.x2 + EDGE_TOLERANCE_PX &&
70
+ b.y2 <= region.y2 + EDGE_TOLERANCE_PX);
71
+ }
72
+ const area = (b) => Math.max(0, b.x2 - b.x1) * Math.max(0, b.y2 - b.y1);
73
+ /** 0..1 — how much of the element the region shows. 1 when fully visible (or degenerate). */
74
+ function visibleFraction(b, region) {
75
+ const total = area(b);
76
+ if (total <= 0)
77
+ return isOffscreen(b, region) ? 0 : 1;
78
+ return area(visibleRect(b, region)) / total;
79
+ }
80
+ /**
81
+ * Where a tap on this element should land.
82
+ *
83
+ * The centre of the VISIBLE part, not of the raw bounds: an element straddling an
84
+ * edge has a geometric centre that can sit outside it, and pressing that is the blind
85
+ * tap this module exists to prevent. Identical to `el.center` whenever the element is
86
+ * fully visible (or no region is known), so the ordinary path is unchanged.
87
+ */
88
+ function tapPoint(el, region) {
89
+ if (!region || isOffscreen(el.bounds, region) || isFullyVisible(el.bounds, region))
90
+ return el.center;
91
+ const v = visibleRect(el.bounds, region);
92
+ return { x: Math.floor((v.x1 + v.x2) / 2), y: Math.floor((v.y1 + v.y2) / 2) };
93
+ }
94
+ /**
95
+ * The region an element must be inside to be reliably tappable: the screen, further
96
+ * clipped to its nearest SCROLLABLE ancestor.
97
+ *
98
+ * This is what catches the case the platform hides from us. Android reports a row
99
+ * scrolled past the fold with bounds already clipped to the display, so nothing about
100
+ * the row itself says it is cut off — but it still overflows the list it lives in,
101
+ * and that is visible in the tree. Elements outside any scroll container (a toolbar,
102
+ * a sticky bar) get the plain screen, so nothing pins them into a smaller box.
103
+ *
104
+ * The hierarchy is pre-order, so an element's ancestors are the nodes before it whose
105
+ * depth keeps decreasing — walking back is exact, not a containment guess.
106
+ */
107
+ function clipRegion(els, target, screen) {
108
+ const at = els.indexOf(target);
109
+ let depth = target.depth;
110
+ for (let i = at - 1; i >= 0; i--) {
111
+ const el = els[i];
112
+ if (el.depth >= depth)
113
+ continue; // not an ancestor: a sibling or its subtree
114
+ depth = el.depth;
115
+ if (el.scrollable)
116
+ return visibleRect(el.bounds, screen);
117
+ }
118
+ return screen;
119
+ }
120
+ /** The two points of a directional swipe across `region`, `frac` of its span. */
121
+ function swipeVector(region, dir, frac) {
122
+ const cx = Math.floor((region.x1 + region.x2) / 2);
123
+ const cy = Math.floor((region.y1 + region.y2) / 2);
124
+ const dx = Math.floor(((region.x2 - region.x1) * frac) / 2);
125
+ const dy = Math.floor(((region.y2 - region.y1) * frac) / 2);
126
+ switch (dir) {
127
+ case 'up':
128
+ return { from: { x: cx, y: cy + dy }, to: { x: cx, y: cy - dy } };
129
+ case 'down':
130
+ return { from: { x: cx, y: cy - dy }, to: { x: cx, y: cy + dy } };
131
+ case 'left':
132
+ return { from: { x: cx + dx, y: cy }, to: { x: cx - dx, y: cy } };
133
+ case 'right':
134
+ return { from: { x: cx - dx, y: cy }, to: { x: cx + dx, y: cy } };
135
+ }
136
+ }
137
+ /**
138
+ * The region to swipe within to move `target`: its own scroll container when it has
139
+ * one, else the largest scrollable that spans it, else the whole screen.
140
+ *
141
+ * The ancestor is exact (see clipRegion) and therefore preferred. The span-based
142
+ * fallback picks by extent ALONG THE SCROLL AXIS, so a page-level vertical list wins
143
+ * over a horizontal carousel nested inside it. Falling back to the screen matters: a
144
+ * framework that never sets `scrollable` still gets a working full-screen swipe.
145
+ */
146
+ function scrollSurface(els, target, screen, axis) {
147
+ const ancestor = clipRegion(els, target, screen);
148
+ if (ancestor !== screen)
149
+ return ancestor;
150
+ const spans = (b) => (axis === 'y' ? b.y2 - b.y1 : b.x2 - b.x1);
151
+ const crossCentre = axis === 'y' ? target.center.x : target.center.y;
152
+ const covers = (b) => axis === 'y' ? b.x1 <= crossCentre && b.x2 >= crossCentre : b.y1 <= crossCentre && b.y2 >= crossCentre;
153
+ let best = null;
154
+ for (const el of els) {
155
+ if (!el.scrollable || el === target)
156
+ continue;
157
+ if (isOffscreen(el.bounds, screen))
158
+ continue;
159
+ const visible = visibleRect(el.bounds, screen);
160
+ if (spans(visible) < MIN_DRAG_PX || !covers(el.bounds))
161
+ continue;
162
+ if (!best || spans(visible) > spans(best))
163
+ best = visible;
164
+ }
165
+ return best ?? screen;
166
+ }
167
+ const contains = (b, p) => p.x >= b.x1 && p.x < b.x2 && p.y >= b.y1 && p.y < b.y2;
168
+ /**
169
+ * Where a tap on this element can actually reach it: the centre of its visible part,
170
+ * or the nearest point to that centre which nothing else is drawn over.
171
+ *
172
+ * MEASURED, and the other half of the bug in #42: a row can be entirely inside the
173
+ * screen AND inside its list, yet have a sticky bottom bar drawn across its lower
174
+ * half — so pressing its centre presses the bar, and the step reports success having
175
+ * tapped a completely different control. There is no occlusion flag in any dump; what
176
+ * there IS, is order. The hierarchy is pre-order, later siblings paint over earlier
177
+ * ones, so an element listed AFTER the target which covers the point is on top of it.
178
+ *
179
+ * Ancestors are excluded for free (they precede the target); descendants are the
180
+ * contiguous run after it with greater depth, and they ARE the target for tapping
181
+ * purposes. Returns null when every candidate point is covered — the caller warns
182
+ * rather than failing, since this is an ordering heuristic and a wrong refusal would
183
+ * be worse than the tap it prevents.
184
+ */
185
+ function reachablePoint(els, target, region) {
186
+ const visible = visibleRect(target.bounds, region);
187
+ if (visible.x2 <= visible.x1 || visible.y2 <= visible.y1)
188
+ return null;
189
+ // Centre first, then inward-inset points: any interior point of a control taps it,
190
+ // and staying inset keeps us off a neighbour's edge.
191
+ const xs = [0.5, 0.5, 0.5, 0.25, 0.75, 0.25, 0.75];
192
+ const ys = [0.5, 0.25, 0.75, 0.5, 0.5, 0.25, 0.75];
193
+ for (let i = 0; i < xs.length; i++) {
194
+ const p = {
195
+ x: Math.floor(visible.x1 + (visible.x2 - visible.x1) * xs[i]),
196
+ y: Math.floor(visible.y1 + (visible.y2 - visible.y1) * ys[i]),
197
+ };
198
+ if (contains(visible, p) && !isOccluded(els, target, p))
199
+ return p;
200
+ }
201
+ return null;
202
+ }
203
+ /** Is `point` covered by an element painted after `target` (and not part of it)? */
204
+ function isOccluded(els, target, point) {
205
+ const at = els.indexOf(target);
206
+ if (at < 0)
207
+ return false;
208
+ let i = at + 1;
209
+ while (i < els.length && els[i].depth > target.depth)
210
+ i++; // the target's own subtree is the target
211
+ for (; i < els.length; i++)
212
+ if (contains(els[i].bounds, point))
213
+ return true;
214
+ return false;
215
+ }
216
+ /**
217
+ * How long to spend dragging `distance` pixels.
218
+ *
219
+ * MEASURED on emulator-5554 (API 34), and the reason this is not a constant: the same
220
+ * 1118px swipe delivered over 400ms took the app off the screen entirely — the fling
221
+ * velocity is read as another gesture — while over 1500ms it scrolled cleanly and
222
+ * stopped where it was put. Pacing by distance keeps a scroll a DRAG: the list lands
223
+ * where we aimed, so the next measurement is of the screen we asked for.
224
+ */
225
+ function swipeDurationMs(distance) {
226
+ const PX_PER_MS = 0.75;
227
+ return Math.min(2000, Math.max(300, Math.round(Math.abs(distance) / PX_PER_MS)));
228
+ }
229
+ /**
230
+ * The swipe that brings `target` into view within `surface`, or null when no useful
231
+ * one exists (already visible, or the move would be too small to be a drag).
232
+ *
233
+ * Aims to CENTRE the target rather than merely nudge it past the edge: an element
234
+ * resting against a screen edge is the case most likely to sit under a sticky app
235
+ * bar, which no hierarchy dump can tell us about. Centring costs nothing extra —
236
+ * the swipe is capped by the surface either way — and lands well clear of both.
237
+ */
238
+ function scrollPlan(target, surface, clip, opts = {}) {
239
+ if (!opts.centre && isFullyVisible(target, clip))
240
+ return null;
241
+ // Distance the CONTENT must move for the target to end up centred. Positive means
242
+ // the target is too far down/right, so the content has to move up/left.
243
+ const needY = Math.round((target.y1 + target.y2) / 2 - (clip.y1 + clip.y2) / 2);
244
+ const needX = Math.round((target.x1 + target.x2) / 2 - (clip.x1 + clip.x2) / 2);
245
+ const fitsY = target.y2 - target.y1 <= clip.y2 - clip.y1;
246
+ const fitsX = target.x2 - target.x1 <= clip.x2 - clip.x1;
247
+ // Only chase an axis the element is actually clipped on: a full-width row is
248
+ // "outside the region in x" by its own size, which no sideways swipe fixes.
249
+ // `centre` overrides that — the caller wants the element moved to the middle even
250
+ // though it is already inside the region (something is drawn over it there).
251
+ const wantY = fitsY && (opts.centre || target.y1 < clip.y1 || target.y2 > clip.y2) ? Math.abs(needY) : 0;
252
+ const wantX = fitsX && (opts.centre || target.x1 < clip.x1 || target.x2 > clip.x2) ? Math.abs(needX) : 0;
253
+ if (wantY === 0 && wantX === 0)
254
+ return null;
255
+ const axis = wantY >= wantX ? 'y' : 'x';
256
+ const need = axis === 'y' ? needY : needX;
257
+ const region = visibleRect(surface, clip);
258
+ const span = axis === 'y' ? region.y2 - region.y1 : region.x2 - region.x1;
259
+ if (span < MIN_DRAG_PX * 2)
260
+ return null;
261
+ const drag = Math.min(Math.abs(need), Math.floor(span * exports.DEFAULT_SWIPE_FRACTION));
262
+ if (drag < MIN_DRAG_PX)
263
+ return null;
264
+ // `up` scrolls the page DOWN (content moves up), revealing what was below.
265
+ const direction = axis === 'y' ? (need > 0 ? 'up' : 'down') : need > 0 ? 'left' : 'right';
266
+ return { ...swipeVector(region, direction, drag / span), axis, direction, distance: drag };
267
+ }
package/dist/version.js CHANGED
@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.VERSION = void 0;
4
4
  // GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
5
5
  // (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
6
- exports.VERSION = '0.13.0';
6
+ exports.VERSION = '0.15.0';
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "verikun",
3
- "version": "0.13.0",
3
+ "version": "0.15.0",
4
4
  "description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
5
5
  "keywords": [
6
6
  "android",
@@ -37,6 +37,9 @@
37
37
  "test": "tsc -p tsconfig.test.json && node --test --test-reporter=spec .test-build/tests/*.test.js",
38
38
  "test:ci": "mkdir -p test-results && tsc -p tsconfig.test.json && node --test --test-reporter=spec --test-reporter-destination=stdout --test-reporter=./scripts/github-test-summary.mjs --test-reporter-destination=test-results/summary.md .test-build/tests/*.test.js",
39
39
  "test:watch": "tsc -p tsconfig.test.json && node --test --watch --test-reporter=spec .test-build/tests/*.test.js",
40
+ "test:e2e": "npm run build && tsc -p tsconfig.test.json && node --test --test-reporter=spec .test-build/tests/e2e/*.test.js",
41
+ "flutter-app:apk": "cd example/flutter-app && fvm flutter build apk --debug",
42
+ "flutter-app:ios": "cd example/flutter-app && fvm flutter build ios --simulator --debug",
40
43
  "prepare": "npm run build"
41
44
  },
42
45
  "engines": {