verikun 0.14.0 → 0.15.0

package/README.md

@@ -399,6 +399,63 @@ If a selector for an action matches more than one element and no `--index` is
 given, the command fails with exit code 2 and lists the candidates — it never
 taps a guess.
 
+### Auto scroll-into-view
+
+An element can be in the hierarchy without being reachable at the point a tap
+would land: scrolled past the edge of its list, or with a sticky bar drawn across
+its middle. Pressing its coordinates then hits whatever is actually there — and
+the step reports success, so the run continues from the wrong place.
+
+So **actions scroll; inspection does not**. `tap` and `text` bring their target
+into the clear first — into its scroll container, and out from under anything
+drawn over it — then act, adding `(scrolled into view: N swipes)` to the
+confirmation. "Scroll down to X and tap it" is therefore just `vk tap X`; you
+rarely need an explicit `swipe`.
+
+`ui`, `find` and `assert` never scroll and never hide anything: an element with no
+pixel on screen is listed as usual, marked `offscreen`. Where an element cannot be
+reached at all, the action **fails with exit 1** rather than pressing blind
+coordinates. `--no-scroll` opts out of the scrolling.
+
+> Note what this cannot see: a control covered by something the accessibility tree
+> does not contain (a decorative container with no label or id) is invisible to any
+> tool reading that tree. Scrolling the target clear of screen edges is what
+> avoids most of these; verikun warns on stderr when it presses an element it
+> believes is covered.
+
+### Which selector to reach for: `@id` first, `text:` second, `desc:` never
+
+Not all four selector kinds travel equally well. If a flow has to run on both
+Android and iOS, this ordering matters:
+
+| selector | Android | iOS | portable? |
+|---|---|---|---|
+| `@id` | `resource-id` | `AXUniqueId` | **yes — always prefer this** |
+| `text:` | visible text, falling back to `content-desc` | `AXLabel` / `title` / `AXValue` | yes |
+| `desc:` | `content-desc` | `accessibilityHint` only | **no — Android in practice** |
+| `class:` | widget class | element role | no — see below |
+
+Two traps worth knowing:
+
+- **`desc:` does not fall back.** `text:` falls back to `desc` when no text
+  matches, so a `text:Submit` selector finds an element carrying only an
+  accessibility label. The reverse is not true — `desc:Submit` will never match
+  visible text. On iOS an accessibility label arrives as `text`, so a `desc:`
+  selector written against Android silently stops matching there.
+- **`class:` is mostly useless on a cross-platform UI toolkit.** Flutter text
+  inputs report as `android.widget.EditText` / `TextField`, but almost everything
+  else is `android.view.View` — so `class:Button` cannot match a Flutter button
+  regardless of what the widget is.
+
+There is a further, sharper reason to prefer `@id`: it is the only selector that
+is not text, so it survives **localisation**. A flow pinned with `text:` breaks
+the moment the device is in a different language.
+
+For a Flutter app, `@id` comes from `Semantics(identifier:)`; `Semantics(label:)`
+gives you `desc` on Android but `text` on iOS. A worked example, with the
+cross-platform gotchas measured on real hardware, is in
+[`example/flutter-app`](example/flutter-app/).
+
 ## Auto-wait
 
 A UI rarely settles the instant the previous action returns. So selector

package/dist/agent/grammar.js

@@ -20,7 +20,7 @@ Each step is one of three node types:
                                   rerun starts FRESH (--clear also wipes data → fresh-install;
                                   --no-restart skips the force-stop, just bringing it forward)
      stop <package>               — force-stop the app
-     tap <selector>               — tap the element a selector resolves to
+     tap <selector>               — tap the element a selector resolves to (scrolls it into view first)
      text <selector> <value...>   — focus a field and type value (--clear to clear first, --enter to submit)
      type <value...>              — type into the already-focused field
      key <name> | back | home | enter
@@ -35,9 +35,9 @@ Each step is one of three node types:
    keep a flow from breaking when an extra screen sometimes appears.
 
 3. REPEAT — { "type":"repeat", "selector":<sel>, "cap":<n>, "body":[<nodes>] }
-   Repeat body UNTIL the selector appears, up to cap iterations. Use for "scroll until X
-   is visible", or "keep answering until the results screen". Always set a sane cap (e.g.
-   10). The engine also stops early if the screen stops changing. A repeat that finishes
+   Repeat body UNTIL the selector appears, up to cap iterations. Use for "keep answering
+   until the results screen" — NOT for scrolling to something that is already in the
+   hierarchy, since tap scrolls to its own target. Always set a sane cap (e.g. 10). The engine also stops early if the screen stops changing. A repeat that finishes
    without its selector ever appearing FAILS the test — it did not do its job.
 
 4. WHEN — { "type":"when", "branches":[{ "selector":<sel>, "body":[<nodes>] }, ...],
@@ -120,6 +120,11 @@ RULES:
   the failure surfaces later as a confusing timeout on the NEXT step.
 - assert is for VERIFICATION only and is terminal — never use it as a step you expect to
   fail. Put genuinely-optional UI behind if-present.
+- tap/text SCROLL THEIR TARGET INTO VIEW automatically, so "scroll down to X and tap it"
+  is just \`tap X\`. Do NOT wrap a tap in a repeat-until-visible loop to reach something
+  below the fold — that is now redundant. Emit an explicit swipe only when the SCROLLING
+  ITSELF is what the test asks for ("scroll the feed three times"), or to reveal content
+  that is not in the hierarchy until it is built (an infinite/lazy list).
 - Prefer resource-id / accessibility selectors over visible text where possible.
 - Translate the test literally and minimally: do not invent ACTION steps (tap/text/swipe/key/assert)
   the prose does not imply. The ONE exception is screenshot — insert screenshot steps liberally as
@@ -149,4 +154,7 @@ Emit ONLY an object matching the schema:
   { "decision":"repair", "step": { "type":"command","command","positionals":[...],"flags":[{"name","value"}] } }
   { "decision":"give_up", "reason": "<why no element on this screen matches the intent>" }
 Prefer a stable selector (resource-id / accessibility label) visible in the hierarchy.
-Do not invent elements that are not in the hierarchy.`;
+Do not invent elements that are not in the hierarchy.
+An element tagged \`offscreen\` is in the tree but scrolled out of view; tap/text scroll
+to their target on their own, so the fact a step failed on one means scrolling could not
+reach it — pick a different element only if one genuinely serves the same purpose.`;

package/dist/args.js

@@ -35,6 +35,7 @@ const BOOLEAN = new Set([
     'android',
     'fix',
     'no-wait',
+    'no-scroll',
     'full',
     'more',
     'show-plan',

package/dist/cli.js

@@ -40,6 +40,7 @@ exports.healNote = healNote;
 exports.parseDuration = parseDuration;
 exports.waitWindowMs = waitWindowMs;
 exports.waitNote = waitNote;
+exports.scrollNote = scrollNote;
 exports.formatDeviceTable = formatDeviceTable;
 exports.guardSettleMs = guardSettleMs;
 exports.confineToCwd = confineToCwd;
@@ -58,6 +59,7 @@ const exec_1 = require("./exec");
 const drivers_1 = require("./drivers");
 const selector_1 = require("./ui/selector");
 const format_1 = require("./ui/format");
+const viewport_1 = require("./ui/viewport");
 const output_1 = require("./output");
 const run_1 = require("./run");
 const image_1 = require("./image");
@@ -173,7 +175,10 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
         const els = ctx.driver.getElements(opts);
         if ((0, selector_1.matchElements)(els, sel).matches.length >= 1) {
             const { element, tier } = (0, selector_1.resolveOne)(els, sel); // 1 → resolved; >1 → throws ambiguity
-            return { element, tier, waitedMs: Date.now() - start };
+            // The snapshot rides along: scroll-into-view needs the scrollable containers
+            // from the SAME dump the element came from, and re-capturing to find them
+            // would both cost a round-trip and risk describing a screen that moved on.
+            return { element, tier, waitedMs: Date.now() - start, elements: els };
         }
         if (Date.now() >= deadline) {
             const waited = windowMs > 0 ? ` after ${(windowMs / 1000).toFixed(1)}s` : '';
@@ -182,6 +187,146 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
         await sleep(pollStep(ctx.flags, deadline));
     }
 }
+// --- Auto scroll-into-view --------------------------------------------------
+// An element's centre is not always a point that reaches it, and a tap on the
+// wrong point still reported success — so the run carried on from the wrong place
+// and failed several steps later on an unrelated symptom (issue #42). Actions
+// therefore bring their target into the clear first, the same contract as
+// Playwright's scrollIntoViewIfNeeded.
+//
+// MEASURED, and it is not only the obvious case: Android's dumper already drops
+// nodes it considers invisible and clips the rest to the display, so the usual
+// shape is an element that IS on screen — a row cut off by its list, or one with a
+// sticky bar drawn across its middle. Both are handled by asking the question
+// against `clipRegion()` (screen ∩ scroll container) and `isOccluded()` (what is
+// painted after it), not against the screen alone.
+//
+// Load-bearing split, mirroring auto-wait: ACTIONS scroll, INSPECTION does not.
+// `ui`/`find`/`assert` report an element exactly as it is (tagged `offscreen` when
+// it has no pixel on screen) — hiding it would turn a wrong tap into a mysterious
+// miss — while `tap`/`text` refuse to press a point that would hit something else.
+//
+// Scrolling only ever happens when the alternative is a wrong tap, so a target
+// already in the clear costs nothing: no extra dump, no swipe.
+const SCROLL_INTO_VIEW_MAX = 10;
+/** Let the scroll settle before re-dumping — a mid-fling hierarchy reads as a stall. */
+const SCROLL_SETTLE_MS = 500;
+/** Movement below this is measurement noise, not progress. */
+const NO_PROGRESS_PX = 8;
+/** Consecutive non-moving swipes before we accept the list will not go further. */
+const NO_PROGRESS_STRIKES = 2;
+/** A short note appended to action output when the target had to be scrolled to. */
+function scrollNote(swipes) {
+    return swipes > 0 ? ` (scrolled into view: ${swipes} swipe${swipes === 1 ? '' : 's'})` : '';
+}
+/** Swipe until `target` sits fully inside its clip region, or we run out of room. */
+async function scrollIntoView(ctx, sel, target, elements, screen, opts = {}) {
+    let current = target;
+    let snapshot = elements;
+    let swipes = 0;
+    let stalled = 0;
+    while (swipes < SCROLL_INTO_VIEW_MAX && stalled < NO_PROGRESS_STRIKES) {
+        // Re-derived every iteration: scrolling can change which container holds the
+        // element, and a stale clip would aim the next swipe at the wrong box.
+        const clip = (0, viewport_1.clipRegion)(snapshot, current, screen);
+        const axis = Math.abs(current.center.y - (clip.y1 + clip.y2) / 2) >= Math.abs(current.center.x - (clip.x1 + clip.x2) / 2)
+            ? 'y'
+            : 'x';
+        // A covered element is scrolled to the MIDDLE of its container even though it is
+        // technically in view: a sticky bar overlaps the edges of a list, and moving the
+        // target away from them is the one reliable way to get a touch through to it.
+        const centre = (0, viewport_1.isOccluded)(snapshot, current, (0, viewport_1.tapPoint)(current, screen));
+        const plan = (0, viewport_1.scrollPlan)(current.bounds, (0, viewport_1.scrollSurface)(snapshot, current, screen, axis), clip, { centre });
+        if (!plan)
+            break; // in view, or no swipe big enough to be worth making
+        ctx.driver.swipe(plan.from.x, plan.from.y, plan.to.x, plan.to.y, (0, viewport_1.swipeDurationMs)(plan.distance));
+        swipes++;
+        await sleep(SCROLL_SETTLE_MS);
+        const before = current.bounds;
+        snapshot = ctx.driver.getElements(opts);
+        // An empty tree is a bad read, not a screen (the device returns partial dumps
+        // mid-transition) — retry rather than conclude the element is gone.
+        if (snapshot.length === 0)
+            continue;
+        if ((0, selector_1.matchElements)(snapshot, sel).matches.length === 0) {
+            // Android drops a node that scrolls out of view, so overshooting LOSES the
+            // target rather than leaving it visibly off-position. Give the last swipe back
+            // (half of it, to land between the two) and look once more before giving up.
+            const mid = { x: Math.round((plan.from.x + plan.to.x) / 2), y: Math.round((plan.from.y + plan.to.y) / 2) };
+            ctx.driver.swipe(plan.to.x, plan.to.y, mid.x, mid.y, (0, viewport_1.swipeDurationMs)(plan.distance / 2));
+            await sleep(SCROLL_SETTLE_MS);
+            snapshot = ctx.driver.getElements(opts);
+            if ((0, selector_1.matchElements)(snapshot, sel).matches.length === 0) {
+                throw new errors_1.SelectorNotFoundError(`'${sel.raw}' left the hierarchy while being scrolled into view (after ${swipes} swipe(s)) — ` +
+                    'a lazy list may have recycled it. Run `verikun ui` to inspect the current screen.');
+            }
+        }
+        current = (0, selector_1.resolveOne)(snapshot, sel).element; // >1 → ambiguity, exit 2, as everywhere else
+        const moved = Math.abs(plan.axis === 'y' ? current.bounds.y1 - before.y1 : current.bounds.x1 - before.x1);
+        stalled = moved < NO_PROGRESS_PX ? stalled + 1 : 0;
+    }
+    return { element: current, swipes, elements: snapshot };
+}
+/** The screen as a rectangle, or null on a device whose size could not be read. */
+function screenOf(ctx) {
+    const vp = ctx.driver.viewport();
+    return vp ? (0, viewport_1.screenRect)(vp) : null;
+}
+/** The point to press for `el`, preferring one nothing else is drawn over. Falls back
+ *  to the visible centre — an ordering-based guess must never block a real tap. */
+function pressPoint(els, el, screen) {
+    if (!screen)
+        return el.center;
+    return (0, viewport_1.reachablePoint)(els, el, screen) ?? (0, viewport_1.tapPoint)(el, screen);
+}
+/** Say so when the element we are about to press is only partly on screen, or is
+ *  covered by something drawn over it — both mean the touch may not reach it. */
+function reachWarning(els, el, screen) {
+    if (!screen)
+        return;
+    if (!(0, viewport_1.isFullyVisible)(el.bounds, screen)) {
+        (0, output_1.err)(`(only ${Math.round((0, viewport_1.visibleFraction)(el.bounds, screen) * 100)}% of ${(0, format_1.formatInline)(el)} is on screen)`);
+    }
+    if (!(0, viewport_1.reachablePoint)(els, el, screen)) {
+        (0, output_1.err)(`(${(0, format_1.formatInline)(el)} is covered by another element — the tap may land on whatever is on top)`);
+    }
+}
+/**
+ * Resolve a selector to something that can actually be pressed: wait for it, scroll it
+ * into view when it is not fully inside its scroll container, and fail loudly rather
+ * than tap blind.
+ *
+ * When the screen size is unknown this is exactly the old behaviour — resolve and
+ * press the element's centre.
+ */
+async function resolveTappable(ctx, sel, opts = {}) {
+    const { element, tier, waitedMs, elements } = await resolveOneWaiting(ctx, sel, opts);
+    const screen = screenOf(ctx);
+    const clip = screen ? (0, viewport_1.clipRegion)(elements, element, screen) : null;
+    // Scroll when the element is cut off by its container, and also when something is
+    // drawn over the point we would press — but only if it HAS a container to scroll
+    // (clip !== screen). Swiping the whole screen at a covered toolbar button would be
+    // a random gesture, and the point-picking fallback below handles that case.
+    const covered = !!screen && !!clip && clip !== screen && (0, viewport_1.isOccluded)(elements, element, (0, viewport_1.tapPoint)(element, screen));
+    if (!screen || !clip || ((0, viewport_1.isFullyVisible)(element.bounds, clip) && !covered)) {
+        reachWarning(elements, element, screen);
+        return { element, tier, waitedMs, swipes: 0, point: pressPoint(elements, element, screen) };
+    }
+    const scrolled = (0, args_1.flagBool)(ctx.flags, 'no-scroll')
+        ? { element, swipes: 0, elements }
+        : await scrollIntoView(ctx, sel, element, elements, screen, opts);
+    if ((0, viewport_1.isOffscreen)(scrolled.element.bounds, screen)) {
+        const why = (0, args_1.flagBool)(ctx.flags, 'no-scroll')
+            ? '--no-scroll is set'
+            : scrolled.swipes === 0
+                ? 'no scrollable container could move it'
+                : `${scrolled.swipes} swipe(s) did not bring it into view`;
+        throw new errors_1.SelectorNotFoundError(`'${sel.raw}' is in the screen's element tree but scrolled out of view (${why}), so tapping it ` +
+            'would press whatever is at those coordinates instead. Run `verikun ui` to inspect the current screen.');
+    }
+    reachWarning(scrolled.elements, scrolled.element, screen);
+    return { ...scrolled, tier, waitedMs, point: pressPoint(scrolled.elements, scrolled.element, screen) };
+}
 // ---------------------------------------------------------------------------
 // Commands
 // ---------------------------------------------------------------------------
@@ -346,22 +491,39 @@ async function cmdTap(ctx) {
     let target;
     let tier = null;
     let waitedMs = 0;
+    let swipes = 0;
+    let point;
     if (isBareIndex) {
         const els = ctx.driver.getElements({ all: (0, args_1.flagBool)(ctx.flags, 'all') });
         const idx = Number(raw);
         const found = els.find((e) => e.index === idx);
         if (!found)
             throw new errors_1.CliError(`No element with index [${idx}] on the current screen. Run \`verikun ui\`.`, 1);
+        // An index names a row of one specific dump, and scrolling renumbers every row —
+        // so this path cannot scroll. Refuse instead of pressing coordinates off-screen.
+        if (found.offscreen) {
+            throw new errors_1.CliError(`Element [${idx}] ${(0, format_1.formatInline)(found)} is scrolled out of view. Tap it by selector ` +
+                `(e.g. \`verikun tap @${found.idShort || 'id'}\`) so verikun can scroll it into view first.`, 1);
+        }
         target = found;
+        reachWarning(els, target, screenOf(ctx));
+        point = pressPoint(els, target, screenOf(ctx));
         ctx.record?.note({ element: target, message: `tapped by index [${idx}]` });
     }
     else {
         const sel = buildSelector(raw, ctx.flags);
-        ({ element: target, tier, waitedMs } = await resolveOneWaiting(ctx, sel, { all: (0, args_1.flagBool)(ctx.flags, 'all') }));
-        ctx.record?.note({ selector: sel, tier, element: target });
+        ({ element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel, {
+            all: (0, args_1.flagBool)(ctx.flags, 'all'),
+        }));
+        ctx.record?.note({
+            selector: sel,
+            tier,
+            element: target,
+            message: swipes > 0 ? `scrolled into view (${swipes} swipe(s)) and tapped` : undefined,
+        });
     }
-    ctx.driver.tap(target.center.x, target.center.y);
-    (0, output_1.out)(`tapped ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}`);
+    ctx.driver.tap(point.x, point.y);
+    (0, output_1.out)(`tapped ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}${scrollNote(swipes)}`);
     return 0;
 }
 async function cmdText(ctx) {
@@ -370,14 +532,14 @@ async function cmdText(ctx) {
     }
     const sel = buildSelector(ctx.positionals[0], ctx.flags);
     const value = ctx.positionals.slice(1).join(' ');
-    const { element: target, tier, waitedMs } = await resolveOneWaiting(ctx, sel);
+    const { element: target, tier, waitedMs, swipes, point } = await resolveTappable(ctx, sel);
     ctx.record?.note({
         selector: sel,
         tier,
         element: target,
         message: target.password ? 'typed «redacted»' : `typed ${JSON.stringify(value)}`,
     });
-    ctx.driver.tap(target.center.x, target.center.y);
+    ctx.driver.tap(point.x, point.y);
     // Wait for field to be focused after tap
     await sleep(100);
     if ((0, args_1.flagBool)(ctx.flags, 'clear') && target.text) {
@@ -394,7 +556,7 @@ async function cmdText(ctx) {
     ctx.driver.inputText(value);
     if ((0, args_1.flagBool)(ctx.flags, 'enter'))
         ctx.driver.pressKey('enter');
-    (0, output_1.out)(`typed ${JSON.stringify(value)} into ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}`);
+    (0, output_1.out)(`typed ${JSON.stringify(value)} into ${(0, format_1.formatInline)(target)}${healNote(tier)}${waitNote(waitedMs)}${scrollNote(swipes)}`);
     return 0;
 }
 function cmdType(ctx) {
@@ -454,33 +616,11 @@ async function cmdSwipe(ctx) {
         const { width, height } = ctx.driver.screenSize();
         region = { x1: 0, y1: 0, x2: width, y2: height };
     }
-    const cx = Math.floor((region.x1 + region.x2) / 2);
-    const cy = Math.floor((region.y1 + region.y2) / 2);
-    const frac = Math.min(Math.max((0, args_1.flagNum)(ctx.flags, 'distance') ?? 0.6, 0.1), 0.95);
-    const dx = Math.floor(((region.x2 - region.x1) * frac) / 2);
-    const dy = Math.floor(((region.y2 - region.y1) * frac) / 2);
-    let a;
-    let b;
-    switch (dir) {
-        case 'up':
-            a = { x: cx, y: cy + dy };
-            b = { x: cx, y: cy - dy };
-            break;
-        case 'down':
-            a = { x: cx, y: cy - dy };
-            b = { x: cx, y: cy + dy };
-            break;
-        case 'left':
-            a = { x: cx + dx, y: cy };
-            b = { x: cx - dx, y: cy };
-            break;
-        case 'right':
-            a = { x: cx - dx, y: cy };
-            b = { x: cx + dx, y: cy };
-            break;
-        default:
-            throw new errors_1.CliError(`Unknown direction '${dir}' (use up|down|left|right)`, 2);
+    if (dir !== 'up' && dir !== 'down' && dir !== 'left' && dir !== 'right') {
+        throw new errors_1.CliError(`Unknown direction '${dir}' (use up|down|left|right)`, 2);
     }
+    const frac = Math.min(Math.max((0, args_1.flagNum)(ctx.flags, 'distance') ?? viewport_1.DEFAULT_SWIPE_FRACTION, 0.1), 0.95);
+    const { from: a, to: b } = (0, viewport_1.swipeVector)(region, dir, frac);
     ctx.driver.swipe(a.x, a.y, b.x, b.y, duration);
     ctx.record?.note({ message: `swiped ${dir}${on ? ` on ${on}` : ''}` });
     (0, output_1.out)(`swiped ${dir}${waitNote(waitedMs)}`);
@@ -1730,6 +1870,15 @@ AUTO-WAIT (selector lookups retry until they resolve)
   Ambiguity is never waited on (the elements are already there). The \`wait\`
   command stays for explicit polling, including --gone, with --timeout/--interval.
 
+AUTO-SCROLL (actions bring their target into view)
+  \`tap\` / \`text\` first scroll their target into the clear — inside its scroll
+  container, and out from under anything drawn over it (a sticky bar) — then act,
+  so "scroll down then tap X" is just \`tap X\`. \`ui\` / \`find\` / \`assert\` never
+  scroll and hide nothing: an element with no pixel on screen is listed as usual,
+  marked \`offscreen\`. One that cannot be reached fails with exit 1 rather than
+  being tapped at coordinates that would hit something else.
+  --no-scroll    act where the element is; do not scroll to it
+
 GLOBAL FLAGS
   -d, --device <serial>   target a specific device (or VERIKUN_DEVICE / ANDROID_SERIAL)
   -p, --platform <android|ios>   (default: android;  --ios / --android shortcuts)

package/dist/drivers/adb.js

@@ -6,6 +6,7 @@ exports.escapeText = escapeText;
 const errors_1 = require("../errors");
 const exec_1 = require("../exec");
 const android_parse_1 = require("../ui/android-parse");
+const viewport_1 = require("../ui/viewport");
 const ADB = process.env.ADB || 'adb';
 const ADB_HINT = 'install the Android platform-tools (`brew install --cask android-platform-tools`), or point ADB at the binary';
 /** Is `adb` present and runnable? Shared by `vk doctor` and AdbDriver.preflight() so
@@ -96,6 +97,11 @@ class AdbDriver {
     platform = 'android';
     requested;
     cachedSerial;
+    /** null = asked and failed. Cached either way: getElements runs on a ~300ms poll
+     *  during auto-wait, and a broken `wm size` must not cost a round-trip every time. */
+    cachedScreen;
+    /** Rotation of the most recent dump — see viewport(). */
+    lastRotation;
     constructor(serial) {
         this.requested = serial;
     }
@@ -178,7 +184,28 @@ class AdbDriver {
         return (0, exec_1.runText)(ADB, this.withSerial(['shell', ...args]), { timeout }).stdout;
     }
     getElements(opts = {}) {
-        return (0, android_parse_1.parseHierarchy)(this.dumpXml(), opts);
+        const xml = this.dumpXml();
+        // Read off the dump we already have: free, and it refreshes every capture, so a
+        // device rotated mid-run is handled without re-asking for the screen size.
+        this.lastRotation = (0, android_parse_1.parseRotation)(xml);
+        return (0, android_parse_1.parseHierarchy)(xml, { ...opts, screen: this.screenOrNull() ?? undefined });
+    }
+    viewport() {
+        const screen = this.screenOrNull();
+        return screen ? (0, viewport_1.viewportFor)(screen, this.lastRotation) : null;
+    }
+    /** screenSize() memoized, failure included — the parser uses it to mark elements
+     *  scrolled out of view, and "we could not tell" must degrade to "all visible". */
+    screenOrNull() {
+        if (this.cachedScreen === undefined) {
+            try {
+                this.cachedScreen = this.screenSize();
+            }
+            catch {
+                this.cachedScreen = null;
+            }
+        }
+        return this.cachedScreen;
     }
     dumpXml() {
         let lastErr = '';

package/dist/drivers/ios.js

@@ -10,6 +10,7 @@ const node_fs_1 = require("node:fs");
 const errors_1 = require("../errors");
 const exec_1 = require("../exec");
 const ios_parse_1 = require("../ui/ios-parse");
+const viewport_1 = require("../ui/viewport");
 // iOS driver. `xcrun simctl` / `devicectl` cover device discovery and — on a
 // simulator — screenshots, app lifecycle, and logs (no extra install needed).
 // Everything interactive (UI hierarchy, tap, type, swipe, keys, screen size) and
@@ -147,6 +148,9 @@ class IdbDriver {
     requested;
     cachedSerial;
     cachedIsSim;
+    /** null = asked and failed. Memoized because screenSize()'s fallback path runs a
+     *  SECOND full hierarchy dump, which auto-wait would otherwise pay every poll. */
+    cachedScreen;
     constructor(device) {
         // 'booted' is a simctl-only alias idb can't address, so treat it as "auto-resolve".
         this.requested = device && device !== 'booted' ? device : undefined;
@@ -250,7 +254,28 @@ class IdbDriver {
     getElements(opts = {}) {
         // `idb ui describe-all` prints the accessibility tree as JSON (array or NDJSON);
         // parseIosHierarchy handles either.
-        return (0, ios_parse_1.parseIosHierarchy)(this.idbText(['ui', 'describe-all'], { timeout: 15000 }), opts);
+        return (0, ios_parse_1.parseIosHierarchy)(this.idbText(['ui', 'describe-all'], { timeout: 15000 }), {
+            ...opts,
+            screen: this.screenOrNull() ?? undefined,
+        });
+    }
+    viewport() {
+        const screen = this.screenOrNull();
+        // No orientation signal from idb, so viewportFor uses the max(w,h) square: exact
+        // on the vertical axis lists scroll on, permissive across it.
+        return screen ? (0, viewport_1.viewportFor)(screen) : null;
+    }
+    /** screenSize() memoized, failure included — see AdbDriver.screenOrNull. */
+    screenOrNull() {
+        if (this.cachedScreen === undefined) {
+            try {
+                this.cachedScreen = this.screenSize();
+            }
+            catch {
+                this.cachedScreen = null;
+            }
+        }
+        return this.cachedScreen;
     }
     screenshot() {
         const tmp = (0, node_path_1.join)((0, node_os_1.tmpdir)(), `verikun-ios-${process.pid}.png`);

package/dist/ui/android-parse.js

@@ -1,7 +1,9 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.isInteresting = isInteresting;
+exports.parseRotation = parseRotation;
 exports.parseHierarchy = parseHierarchy;
+const viewport_1 = require("./viewport");
 // Parses uiautomator XML (from `adb shell uiautomator dump`) into normalized
 // Element[]. We hand-roll a tiny tag scanner rather than pulling in an XML
 // dependency: uiautomator output is regular and entity-escaped, so a
@@ -99,6 +101,18 @@ function isInteresting(el) {
         return true;
     return false;
 }
+/**
+ * The dump's `<hierarchy rotation="N">` (Surface.ROTATION_*), or undefined if absent.
+ *
+ * `adb shell wm size` always reports the device's NATURAL orientation, so this is the
+ * only thing in the dump that says whether the coordinates are portrait or landscape.
+ * Exported so the driver can read it off the same XML it just fetched, rather than
+ * paying another round-trip to ask the device which way up it is.
+ */
+function parseRotation(xml) {
+    const m = /<hierarchy\b[^>]*\brotation="(\d+)"/.exec(xml);
+    return m ? Number(m[1]) : undefined;
+}
 function parseHierarchy(xml, opts = {}) {
     const all = [];
     const n = xml.length;
@@ -142,8 +156,13 @@ function parseHierarchy(xml, opts = {}) {
         }
     }
     const result = opts.all ? all : all.filter(isInteresting);
+    // No screen size (the driver could not read one) → nothing is marked, which is
+    // exactly the behaviour before viewport awareness existed.
+    const vp = opts.screen ? (0, viewport_1.screenRect)((0, viewport_1.viewportFor)(opts.screen, parseRotation(xml))) : undefined;
     result.forEach((el, idx) => {
         el.index = idx;
+        if (vp && (0, viewport_1.isOffscreen)(el.bounds, vp))
+            el.offscreen = true;
     });
     return result;
 }

package/dist/ui/format.js

@@ -34,6 +34,10 @@ function formatInline(el) {
         flags.push('selected');
     if (!el.enabled)
         flags.push('disabled');
+    // Shown rather than hidden: the element IS there, just scrolled away — and an
+    // action will scroll it into view. Hiding it would only make the miss mysterious.
+    if (el.offscreen)
+        flags.push('offscreen');
     if (flags.length)
         parts.push(flags.join(','));
     return parts.join(' ');
@@ -67,5 +71,6 @@ function toJsonShape(el) {
         password: el.password || undefined,
         enabled: el.enabled,
         selected: el.selected || undefined,
+        offscreen: el.offscreen || undefined,
     };
 }

package/dist/ui/ios-parse.js

@@ -2,6 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.isInteresting = isInteresting;
 exports.parseIosHierarchy = parseIosHierarchy;
+const viewport_1 = require("./viewport");
 // XCUIElementType names (idb `type`) that are meaningful tap targets — used to
 // derive `clickable`, which iOS accessibility does not expose as a flag.
 const TAPPABLE_TYPES = new Set([
@@ -111,8 +112,13 @@ function isInteresting(el) {
 function parseIosHierarchy(jsonText, opts = {}) {
     const all = parseIdbJson(jsonText).map(buildElement);
     const result = opts.all ? all : all.filter(isInteresting);
+    // idb reports no orientation, so viewportFor falls back to the max(w,h) square —
+    // exact vertically (the axis lists scroll on), deliberately permissive across.
+    const vp = opts.screen ? (0, viewport_1.screenRect)((0, viewport_1.viewportFor)(opts.screen)) : undefined;
     result.forEach((el, idx) => {
         el.index = idx;
+        if (vp && (0, viewport_1.isOffscreen)(el.bounds, vp))
+            el.offscreen = true;
     });
     return result;
 }

package/dist/ui/viewport.js

@@ -0,0 +1,267 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DEFAULT_SWIPE_FRACTION = void 0;
+exports.viewportFor = viewportFor;
+exports.screenRect = screenRect;
+exports.isOffscreen = isOffscreen;
+exports.visibleRect = visibleRect;
+exports.isFullyVisible = isFullyVisible;
+exports.visibleFraction = visibleFraction;
+exports.tapPoint = tapPoint;
+exports.clipRegion = clipRegion;
+exports.swipeVector = swipeVector;
+exports.scrollSurface = scrollSurface;
+exports.reachablePoint = reachablePoint;
+exports.isOccluded = isOccluded;
+exports.swipeDurationMs = swipeDurationMs;
+exports.scrollPlan = scrollPlan;
+/** How much of a region a directional swipe drags across, as a fraction of its span. */
+exports.DEFAULT_SWIPE_FRACTION = 0.6;
+/** Below this the gesture reads as a tap rather than a drag, so scrolling by less
+ *  than this is refused — pressing the wrong control is exactly what we are fixing. */
+const MIN_DRAG_PX = 40;
+/** Bounds within a pixel of the viewport edge count as inside: uiautomator rounds,
+ *  and a 1px overhang is not a reason to swipe a screen. */
+const EDGE_TOLERANCE_PX = 1;
+/**
+ * The screen rectangle a dump's coordinates live in.
+ *
+ * `rotation` is Android's `<hierarchy rotation="N">` (Surface.ROTATION_*): `adb shell
+ * wm size` always reports the NATURAL orientation, so on a landscape device (1/3) the
+ * axes must be swapped — without that, everything past the portrait width looks
+ * off-screen and every tap on a landscape app would try to scroll.
+ *
+ * `undefined` means no orientation signal (iOS: `idb describe` has none). Rather than
+ * guess, use the max(w,h) square: it can never be SMALLER than the true viewport, so
+ * it can only under-detect off-screen elements — i.e. degrade to today's behaviour —
+ * and never claim a visible element is out of view.
+ */
+function viewportFor(screen, rotation) {
+    const w = Math.max(0, Math.round(screen.width));
+    const h = Math.max(0, Math.round(screen.height));
+    if (rotation === undefined) {
+        const square = Math.max(w, h);
+        return { width: square, height: square };
+    }
+    return rotation === 1 || rotation === 3 ? { width: h, height: w } : { width: w, height: h };
+}
+/** The screen as a rectangle — the outermost clip region. */
+function screenRect(vp) {
+    return { x1: 0, y1: 0, x2: vp.width, y2: vp.height };
+}
+/** No overlap with the region at all — there is no honest point to press. */
+function isOffscreen(b, region) {
+    return b.x2 <= region.x1 || b.y2 <= region.y1 || b.x1 >= region.x2 || b.y1 >= region.y2;
+}
+/** The part of `b` inside the region (empty rect when there is no overlap). */
+function visibleRect(b, region) {
+    return {
+        x1: Math.max(b.x1, region.x1),
+        y1: Math.max(b.y1, region.y1),
+        x2: Math.min(b.x2, region.x2),
+        y2: Math.min(b.y2, region.y2),
+    };
+}
+/** Entirely within the region — the state "scroll into view" aims for. */
+function isFullyVisible(b, region) {
+    return (b.x1 >= region.x1 - EDGE_TOLERANCE_PX &&
+        b.y1 >= region.y1 - EDGE_TOLERANCE_PX &&
+        b.x2 <= region.x2 + EDGE_TOLERANCE_PX &&
+        b.y2 <= region.y2 + EDGE_TOLERANCE_PX);
+}
+const area = (b) => Math.max(0, b.x2 - b.x1) * Math.max(0, b.y2 - b.y1);
+/** 0..1 — how much of the element the region shows. 1 when fully visible (or degenerate). */
+function visibleFraction(b, region) {
+    const total = area(b);
+    if (total <= 0)
+        return isOffscreen(b, region) ? 0 : 1;
+    return area(visibleRect(b, region)) / total;
+}
+/**
+ * Where a tap on this element should land.
+ *
+ * The centre of the VISIBLE part, not of the raw bounds: an element straddling an
+ * edge has a geometric centre that can sit outside it, and pressing that is the blind
+ * tap this module exists to prevent. Identical to `el.center` whenever the element is
+ * fully visible (or no region is known), so the ordinary path is unchanged.
+ */
+function tapPoint(el, region) {
+    if (!region || isOffscreen(el.bounds, region) || isFullyVisible(el.bounds, region))
+        return el.center;
+    const v = visibleRect(el.bounds, region);
+    return { x: Math.floor((v.x1 + v.x2) / 2), y: Math.floor((v.y1 + v.y2) / 2) };
+}
+/**
+ * The region an element must be inside to be reliably tappable: the screen, further
+ * clipped to its nearest SCROLLABLE ancestor.
+ *
+ * This is what catches the case the platform hides from us. Android reports a row
+ * scrolled past the fold with bounds already clipped to the display, so nothing about
+ * the row itself says it is cut off — but it still overflows the list it lives in,
+ * and that is visible in the tree. Elements outside any scroll container (a toolbar,
+ * a sticky bar) get the plain screen, so nothing pins them into a smaller box.
+ *
+ * The hierarchy is pre-order, so an element's ancestors are the nodes before it whose
+ * depth keeps decreasing — walking back is exact, not a containment guess.
+ */
+function clipRegion(els, target, screen) {
+    const at = els.indexOf(target);
+    let depth = target.depth;
+    for (let i = at - 1; i >= 0; i--) {
+        const el = els[i];
+        if (el.depth >= depth)
+            continue; // not an ancestor: a sibling or its subtree
+        depth = el.depth;
+        if (el.scrollable)
+            return visibleRect(el.bounds, screen);
+    }
+    return screen;
+}
+/** The two points of a directional swipe across `region`, `frac` of its span. */
+function swipeVector(region, dir, frac) {
+    const cx = Math.floor((region.x1 + region.x2) / 2);
+    const cy = Math.floor((region.y1 + region.y2) / 2);
+    const dx = Math.floor(((region.x2 - region.x1) * frac) / 2);
+    const dy = Math.floor(((region.y2 - region.y1) * frac) / 2);
+    switch (dir) {
+        case 'up':
+            return { from: { x: cx, y: cy + dy }, to: { x: cx, y: cy - dy } };
+        case 'down':
+            return { from: { x: cx, y: cy - dy }, to: { x: cx, y: cy + dy } };
+        case 'left':
+            return { from: { x: cx + dx, y: cy }, to: { x: cx - dx, y: cy } };
+        case 'right':
+            return { from: { x: cx - dx, y: cy }, to: { x: cx + dx, y: cy } };
+    }
+}
+/**
+ * The region to swipe within to move `target`: its own scroll container when it has
+ * one, else the largest scrollable that spans it, else the whole screen.
+ *
+ * The ancestor is exact (see clipRegion) and therefore preferred. The span-based
+ * fallback picks by extent ALONG THE SCROLL AXIS, so a page-level vertical list wins
+ * over a horizontal carousel nested inside it. Falling back to the screen matters: a
+ * framework that never sets `scrollable` still gets a working full-screen swipe.
+ */
+function scrollSurface(els, target, screen, axis) {
+    const ancestor = clipRegion(els, target, screen);
+    if (ancestor !== screen)
+        return ancestor;
+    const spans = (b) => (axis === 'y' ? b.y2 - b.y1 : b.x2 - b.x1);
+    const crossCentre = axis === 'y' ? target.center.x : target.center.y;
+    const covers = (b) => axis === 'y' ? b.x1 <= crossCentre && b.x2 >= crossCentre : b.y1 <= crossCentre && b.y2 >= crossCentre;
+    let best = null;
+    for (const el of els) {
+        if (!el.scrollable || el === target)
+            continue;
+        if (isOffscreen(el.bounds, screen))
+            continue;
+        const visible = visibleRect(el.bounds, screen);
+        if (spans(visible) < MIN_DRAG_PX || !covers(el.bounds))
+            continue;
+        if (!best || spans(visible) > spans(best))
+            best = visible;
+    }
+    return best ?? screen;
+}
+const contains = (b, p) => p.x >= b.x1 && p.x < b.x2 && p.y >= b.y1 && p.y < b.y2;
+/**
+ * Where a tap on this element can actually reach it: the centre of its visible part,
+ * or the nearest point to that centre which nothing else is drawn over.
+ *
+ * MEASURED, and the other half of the bug in #42: a row can be entirely inside the
+ * screen AND inside its list, yet have a sticky bottom bar drawn across its lower
+ * half — so pressing its centre presses the bar, and the step reports success having
+ * tapped a completely different control. There is no occlusion flag in any dump; what
+ * there IS, is order. The hierarchy is pre-order, later siblings paint over earlier
+ * ones, so an element listed AFTER the target which covers the point is on top of it.
+ *
+ * Ancestors are excluded for free (they precede the target); descendants are the
+ * contiguous run after it with greater depth, and they ARE the target for tapping
+ * purposes. Returns null when every candidate point is covered — the caller warns
+ * rather than failing, since this is an ordering heuristic and a wrong refusal would
+ * be worse than the tap it prevents.
+ */
+function reachablePoint(els, target, region) {
+    const visible = visibleRect(target.bounds, region);
+    if (visible.x2 <= visible.x1 || visible.y2 <= visible.y1)
+        return null;
+    // Centre first, then inward-inset points: any interior point of a control taps it,
+    // and staying inset keeps us off a neighbour's edge.
+    const xs = [0.5, 0.5, 0.5, 0.25, 0.75, 0.25, 0.75];
+    const ys = [0.5, 0.25, 0.75, 0.5, 0.5, 0.25, 0.75];
+    for (let i = 0; i < xs.length; i++) {
+        const p = {
+            x: Math.floor(visible.x1 + (visible.x2 - visible.x1) * xs[i]),
+            y: Math.floor(visible.y1 + (visible.y2 - visible.y1) * ys[i]),
+        };
+        if (contains(visible, p) && !isOccluded(els, target, p))
+            return p;
+    }
+    return null;
+}
+/** Is `point` covered by an element painted after `target` (and not part of it)? */
+function isOccluded(els, target, point) {
+    const at = els.indexOf(target);
+    if (at < 0)
+        return false;
+    let i = at + 1;
+    while (i < els.length && els[i].depth > target.depth)
+        i++; // the target's own subtree is the target
+    for (; i < els.length; i++)
+        if (contains(els[i].bounds, point))
+            return true;
+    return false;
+}
+/**
+ * How long to spend dragging `distance` pixels.
+ *
+ * MEASURED on emulator-5554 (API 34), and the reason this is not a constant: the same
+ * 1118px swipe delivered over 400ms took the app off the screen entirely — the fling
+ * velocity is read as another gesture — while over 1500ms it scrolled cleanly and
+ * stopped where it was put. Pacing by distance keeps a scroll a DRAG: the list lands
+ * where we aimed, so the next measurement is of the screen we asked for.
+ */
+function swipeDurationMs(distance) {
+    const PX_PER_MS = 0.75;
+    return Math.min(2000, Math.max(300, Math.round(Math.abs(distance) / PX_PER_MS)));
+}
+/**
+ * The swipe that brings `target` into view within `surface`, or null when no useful
+ * one exists (already visible, or the move would be too small to be a drag).
+ *
+ * Aims to CENTRE the target rather than merely nudge it past the edge: an element
+ * resting against a screen edge is the case most likely to sit under a sticky app
+ * bar, which no hierarchy dump can tell us about. Centring costs nothing extra —
+ * the swipe is capped by the surface either way — and lands well clear of both.
+ */
+function scrollPlan(target, surface, clip, opts = {}) {
+    if (!opts.centre && isFullyVisible(target, clip))
+        return null;
+    // Distance the CONTENT must move for the target to end up centred. Positive means
+    // the target is too far down/right, so the content has to move up/left.
+    const needY = Math.round((target.y1 + target.y2) / 2 - (clip.y1 + clip.y2) / 2);
+    const needX = Math.round((target.x1 + target.x2) / 2 - (clip.x1 + clip.x2) / 2);
+    const fitsY = target.y2 - target.y1 <= clip.y2 - clip.y1;
+    const fitsX = target.x2 - target.x1 <= clip.x2 - clip.x1;
+    // Only chase an axis the element is actually clipped on: a full-width row is
+    // "outside the region in x" by its own size, which no sideways swipe fixes.
+    // `centre` overrides that — the caller wants the element moved to the middle even
+    // though it is already inside the region (something is drawn over it there).
+    const wantY = fitsY && (opts.centre || target.y1 < clip.y1 || target.y2 > clip.y2) ? Math.abs(needY) : 0;
+    const wantX = fitsX && (opts.centre || target.x1 < clip.x1 || target.x2 > clip.x2) ? Math.abs(needX) : 0;
+    if (wantY === 0 && wantX === 0)
+        return null;
+    const axis = wantY >= wantX ? 'y' : 'x';
+    const need = axis === 'y' ? needY : needX;
+    const region = visibleRect(surface, clip);
+    const span = axis === 'y' ? region.y2 - region.y1 : region.x2 - region.x1;
+    if (span < MIN_DRAG_PX * 2)
+        return null;
+    const drag = Math.min(Math.abs(need), Math.floor(span * exports.DEFAULT_SWIPE_FRACTION));
+    if (drag < MIN_DRAG_PX)
+        return null;
+    // `up` scrolls the page DOWN (content moves up), revealing what was below.
+    const direction = axis === 'y' ? (need > 0 ? 'up' : 'down') : need > 0 ? 'left' : 'right';
+    return { ...swipeVector(region, direction, drag / span), axis, direction, distance: drag };
+}

package/dist/version.js

@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.VERSION = void 0;
 // GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
 // (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
-exports.VERSION = '0.14.0';
+exports.VERSION = '0.15.0';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "verikun",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
   "keywords": [
     "android",
@@ -37,6 +37,9 @@
     "test": "tsc -p tsconfig.test.json && node --test --test-reporter=spec .test-build/tests/*.test.js",
     "test:ci": "mkdir -p test-results && tsc -p tsconfig.test.json && node --test --test-reporter=spec --test-reporter-destination=stdout --test-reporter=./scripts/github-test-summary.mjs --test-reporter-destination=test-results/summary.md .test-build/tests/*.test.js",
     "test:watch": "tsc -p tsconfig.test.json && node --test --watch --test-reporter=spec .test-build/tests/*.test.js",
+    "test:e2e": "npm run build && tsc -p tsconfig.test.json && node --test --test-reporter=spec .test-build/tests/e2e/*.test.js",
+    "flutter-app:apk": "cd example/flutter-app && fvm flutter build apk --debug",
+    "flutter-app:ios": "cd example/flutter-app && fvm flutter build ios --simulator --debug",
     "prepare": "npm run build"
   },
   "engines": {