simframe 0.14.2 → 0.15.0

package/README.md

@@ -323,6 +323,7 @@ boundary hands it frames and nothing above it knows what a simulator is.
 | Clipboard, and `paste` into a field | yes | the emulator's gRPC `setClipboard`, over `node:http2`, no dependency, then `KEYCODE_PASTE` to deliver it |
 | List/resolve devices, launch, terminate, open a URL, permissions | yes | `adb`, with the permission state read back off the device |
 | Accessibility tree | **not available (OCR + CV only)** | `uiautomator dump` costs **2,012 ms** a read, against 45 ms for the iOS tree. See [`docs/DEFERRED.md`](docs/DEFERRED.md) |
+| A launch confirmed to have reached the front | **not available (the launch is not checked)** | iOS compares the pid `simctl launch` printed against the pid the device reports as frontmost, in **2–5 ms**. Nothing here reports either; `am start` fronts synchronously, which is why it has not bitten — but that is not a check. See [`docs/DEFERRED.md`](docs/DEFERRED.md) |
 
 ```bash
 # an emulator is found the same way a simulator is
@@ -402,7 +403,7 @@ steer the model is a tool surface the model uses wrong.
 | `sim_flow_run` | Replay a flow that verified end to end. |
 | `sim_find` | Resolve an intent to one control, without acting on it. |
 | `sim_tap` · `sim_type_into` · `sim_scroll_to` · `sim_wait_for` · `sim_assert` | Single actions, for when you genuinely only have one step. Each is one `sim_do` step underneath. |
-| `sim_launch` · `sim_open_url` · `sim_permission` | Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert. |
+| `sim_launch` · `sim_open_url` · `sim_permission` | Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert. A launch is **confirmed to have reached the front**, by comparing the pid `simctl` started against the pid the device reports as frontmost — so *"the process started"* is no longer reported as *"the app is on screen"*. |
 | `sim_wait` | Waits for the screen to change *and then* settle. |
 | `sim_look` | **The only tool that returns an image**, capped at 1024 px. For layout, colour, spacing — questions text cannot answer. |
 | `sim_recall` · `sim_strip` | Look backwards: a text timeline of what happened, or recent frames tiled into one image. |
@@ -994,11 +995,22 @@ said a word — the exact failure shape, found by the thing built to catch it.
   dramatically between visits will simply be rebuilt.
 - It speeds up *confirming* a fix, not *locating* one. A bug living in a memo
   comparator or a stale closure is not visible in any frame.
-- A switch is tapped at the centre of its frame, and a switch's frame is the
-  whole row — so the tap lands on the label and the control, which sits at the
-  trailing end, does not move. Use `@x,y` on the control for now. Filed with
-  the measurement in `docs/DEFERRED.md`; it is a role-specific tap point, not a
-  patch at one call site.
+- A switch is tapped at its **activation point** when the app publishes one —
+  UIKit's `accessibilityActivationPoint`, which is what a switch answers with,
+  and tapping it flipped a real switch 3 of 3 times where the frame centre
+  managed 0 of 3. Only **4 of 75** elements on a measured screen publish one,
+  though, so where the app says nothing the tap still goes to the centre of the
+  frame — and a switch's frame is the whole row, so it lands on the label and
+  the control at the trailing end does not move. `@x,y` remains the escape
+  hatch there. Never a guessed offset: nil means the app did not answer.
+- A launch is confirmed to have fronted **on iOS only**. It compares the pid
+  `simctl launch` printed against the pid the device reports as frontmost, in
+  2–5 ms. Android reports neither, so a launch there is not checked — `am start`
+  fronts synchronously, which is why it has not bitten, but that is not a check
+  and `doctor` says so. Note that a launch which starts a process without
+  bringing it forward now **fails** rather than returning success: a flow that
+  used to pass through such a launch and then assert on the previous app's
+  screen will start failing, correctly.
 - The simulator's display pipeline stops rendering under rapid app relaunch —
   about six cycles, reproducibly — and every frame comes back black while
   `simctl` itself reports success. simframe now says so instead of reading a

package/native/simframed/Sources/PrivateAPI/AccessibilityBridge.swift

@@ -62,6 +62,24 @@ public struct AXTree: Sendable {
     }
 }
 
+/// The application the device is showing right now.
+///
+/// Deliberately not a bundle id: nothing on this path publishes one, and a
+/// display name that merely *looks* like an identifier would invite exactly the
+/// string comparison this type exists to avoid.
+public struct FrontmostApp: Sendable {
+    /// The guest pid, or nil when the frontmost application object would not
+    /// answer for it. Nil is "cannot say", never "not that app".
+    public let pid: Int32?
+    /// The application element's own title, for reporting.
+    public let title: String?
+
+    public init(pid: Int32?, title: String?) {
+        self.pid = pid
+        self.title = title
+    }
+}
+
 public enum AccessibilityError: Error, CustomStringConvertible {
     case unavailable(String)
     case noFrontmostApplication
@@ -219,11 +237,54 @@ public final class AccessibilityBridge {
         }
     }
 
-    /// The pid of the app currently frontmost, or nil when the bridge cannot say.
-    public func frontmostPid() -> Int32? {
-        guard let app = frontmostApplication() else { return nil }
-        guard app.responds(to: NSSelectorFromString("pid")) else { return nil }
-        return (app.value(forKey: "pid") as? NSNumber)?.int32Value
+    /// Who is in front, as the device itself reports it.
+    ///
+    /// This exists because `launch` could not tell "the app was already in
+    /// front" from "the app never came forward": both return ok and both leave
+    /// the screen unchanged. Screen change cannot settle it either way —
+    /// relaunching an app that is already frontmost legitimately lands on the
+    /// same screen — so the discriminator has to be identity.
+    ///
+    /// `pid` is the number to compare against, because it is the same one
+    /// `simctl launch` prints, so the caller tests equality rather than
+    /// matching a display name against a bundle id. `title` is for the human
+    /// reading the verdict and is a display name, not an identifier.
+    public func frontmostApp() throws -> FrontmostApp {
+        guard let app = frontmostApplication() else { throw AccessibilityError.noFrontmostApplication }
+        let pid = app.responds(to: NSSelectorFromString("pid"))
+            ? (app.value(forKey: "pid") as? NSNumber)?.int32Value
+            : nil
+        // Probing several names rather than trusting one, because a bare pid is
+        // not a diagnosis. A runner held the front at pid 7797 through nine
+        // consecutive failed launches and the whole question was whether that
+        // was SpringBoard or a lock screen — unanswerable from a number.
+        var title: String?
+        if let root = elementClass
+            .perform(NSSelectorFromString("platformElementWithTranslationObject:"), with: app)?
+            .takeUnretainedValue() as? NSObject {
+            // Reported RAW, generic answers included, and the judgement about
+            // them is made a layer up. Measured: in Settings this reads
+            // "Settings"; press home and the SAME pid is still frontmost while
+            // the answer degrades to the bare word "application". That
+            // degradation is a signal — an app frontmost by pid that has
+            // stopped naming itself is the shape item 171 is about — so
+            // swallowing it here would throw away the interesting half. This
+            // file's job is to say what the translator said.
+            for name in ["AXTitle", "AXDescription"] {
+                if let found = string(attribute(root, name)) { title = found; break }
+            }
+        }
+        // The translation object itself, if the element would not say. Keys, not
+        // selectors, because these are properties on a private class and a key
+        // that is absent is caught by `responds(to:)` rather than by an
+        // Objective-C exception Swift cannot catch.
+        if title == nil {
+            for key in ["bundleId", "bundleIdentifier", "displayName", "processName"]
+            where app.responds(to: NSSelectorFromString(key)) {
+                if let found = string(app.value(forKey: key)) { title = found; break }
+            }
+        }
+        return FrontmostApp(pid: pid, title: title)
     }
 
     private func frontmostApplication() -> NSObject? {

package/native/simframed/Sources/PrivateAPI/CoreSimulatorPlatform.swift

@@ -275,6 +275,10 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
         try bridge().tree()
     }
 
+    public func frontmostApp() throws -> FrontmostApp {
+        try bridge().frontmostApp()
+    }
+
     /// Serialised, because building a bridge installs a delegate on a
     /// **process-global** translator that holds it weakly.
     ///

package/native/simframed/Sources/PrivateAPI/PrivateAPI.swift

@@ -171,6 +171,13 @@ public protocol SimulatorPlatform: AnyObject {
     /// An app still launching has no tree yet, and this reports that as it is —
     /// one node, no children — rather than retrying until it looks populated.
     func accessibilityTree() throws -> AXTree
+    /// Which application the device is showing, when the platform can say.
+    ///
+    /// Separate from `accessibilityTree()` even though the same bridge answers
+    /// both, because the caller that needs it — "did the app I launched come
+    /// forward?" — must not pay for a whole tree walk to ask a one-field
+    /// question, and must be able to ask it repeatedly while waiting.
+    func frontmostApp() throws -> FrontmostApp
 
     // MARK: App lifecycle. These are simctl, not private API — no HID needed.
 

package/native/simframed/Sources/PrivateAPI/StubPlatform.swift

@@ -14,6 +14,8 @@ public final class StubPlatform: SimulatorPlatform {
     /// What `accessibilityTree()` should answer. Empty by default, which is
     /// what a device with no app in the foreground genuinely looks like.
     public var stubTree: AXTree = AXTree(nodes: [])
+    /// What `frontmostApp()` should answer.
+    public var stubFrontmost = FrontmostApp(pid: 1234, title: "Stub")
 
     public init(width: Int = 1206, height: Int = 2622, tint: UInt8 = 0) {
         self.width = width
@@ -130,4 +132,9 @@ extension StubPlatform {
         recorded.append("accessibilityTree()")
         return stubTree
     }
+
+    public func frontmostApp() throws -> FrontmostApp {
+        recorded.append("frontmostApp()")
+        return stubFrontmost
+    }
 }

package/native/simframed/Sources/simframed/main.swift

@@ -235,6 +235,16 @@ case "run":
                                    "scale": device.scale],
                         "engine": "simframed",
                     ])
+                case "frontmost":
+                    // The one-field question, on its own action, because the
+                    // caller asks it in a loop while waiting for a launch to
+                    // land. Routing it through "ui" would pay for a tree walk
+                    // and an OCR pass per poll.
+                    let front = try platform.frontmostApp()
+                    var out: [String: Any] = [:]
+                    if let pid = front.pid { out["pid"] = Int(pid) }
+                    if let title = front.title { out["title"] = title }
+                    return done(out)
                 case "ui":
                     // The accessibility tree and OCR read the same instant of
                     // the screen and neither needs the other, so they run

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "simframe",
-  "version": "0.14.2",
+  "version": "0.15.0",
   "mcpName": "io.github.lvlrSajjad/simframe",
   "description": "Always-warm iOS Simulator and Android emulator frames: agents read the screen in ~20ms instead of waiting on screenshots. MCP server + CLI.",
   "keywords": [

package/scripts/check-published.mjs

@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+/**
+ * Does the version on npm contain the code you think it does?
+ *
+ * **DEFERRED 149.** `0.14.1` was tagged at one commit and a change landed
+ * immediately after it, so `npm install simframe@0.14.1` shipped the previous
+ * `centerOf()`. An external tester was asked to evaluate that change, diffed
+ * the package against the checkout themselves, and wrote: *"I came within one
+ * command of filing this report against the wrong binary."*
+ *
+ * Nothing in the pipeline compared what was tagged against what was being asked
+ * for, and the release workflow cannot: it checks that the tag, `package.json`
+ * and `server.json` agree, which they did. The gap is between the tag and
+ * whatever arrived afterwards.
+ *
+ * So this is the check to run **before handing someone a version to test**, and
+ * before cutting the next one. It fetches the published tarball and compares
+ * every shipped source file against the working tree.
+ *
+ * It deliberately compares against the WORKING TREE rather than a tag: the
+ * question being asked is "is what I am about to ask someone to install the code
+ * I am looking at", and a tag cannot answer that.
+ *
+ * Usage:
+ *   node scripts/check-published.mjs            # the version in package.json
+ *   node scripts/check-published.mjs 0.14.1     # any published version
+ *   node scripts/check-published.mjs --latest   # whatever npm serves as latest
+ *
+ * Exit 0 when every shipped file matches, 1 when any differs, 2 when the
+ * version is not published or npm could not be reached — which is a different
+ * answer and must not read as "it matches".
+ */
+import { execFileSync } from 'node:child_process';
+import { createHash } from 'node:crypto';
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+
+const md5 = (buf) => createHash('md5').update(buf).digest('hex');
+const here = path.resolve(path.dirname(new URL(import.meta.url).pathname), '..');
+
+const arg = process.argv.slice(2).find((a) => !a.startsWith('-'));
+const wantLatest = process.argv.includes('--latest');
+const local = JSON.parse(fs.readFileSync(path.join(here, 'package.json'), 'utf8'));
+
+let version = arg ?? local.version;
+if (wantLatest) {
+  try {
+    version = execFileSync('npm', ['view', local.name, 'version'], { encoding: 'utf8' }).trim();
+  } catch (err) {
+    console.error(`could not ask npm for the latest ${local.name}: ${err.message}`);
+    process.exit(2);
+  }
+}
+
+console.log(`comparing ${local.name}@${version} on npm against this working tree`);
+
+const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'simframe-published-'));
+let tarball;
+try {
+  tarball = execFileSync('npm', ['pack', `${local.name}@${version}`, '--silent', '--pack-destination', tmp], {
+    encoding: 'utf8',
+  }).trim().split('\n').pop();
+} catch (err) {
+  // Not published, unpublished, or no network. None of those is a match.
+  console.error(`could not fetch ${local.name}@${version} from npm.`);
+  console.error('That is not the same answer as "it differs" — nothing was compared.');
+  console.error(String(err.stderr || err.message).trim().split('\n').slice(-3).join('\n'));
+  process.exit(2);
+}
+
+execFileSync('tar', ['-xzf', path.join(tmp, tarball), '-C', tmp]);
+const root = path.join(tmp, 'package');
+
+/** Every file the package ships, relative to the package root. */
+const shipped = [];
+const walk = (dir) => {
+  for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
+    const full = path.join(dir, e.name);
+    if (e.isDirectory()) walk(full);
+    else shipped.push(path.relative(root, full));
+  }
+};
+walk(root);
+
+// Only compare what this repo is the source of. `package.json` is rewritten by
+// npm on publish (it adds `_id`, `dist`, and normalises fields), so byte
+// equality there is not the question and would be a permanent false alarm.
+const comparable = shipped.filter((f) => /^(src|scripts|native)\//.test(f) && !f.includes('/.build/'));
+
+const same = [];
+const differ = [];
+const missing = [];
+for (const rel of comparable) {
+  const mine = path.join(here, rel);
+  if (!fs.existsSync(mine)) { missing.push(rel); continue; }
+  if (md5(fs.readFileSync(mine)) === md5(fs.readFileSync(path.join(root, rel)))) same.push(rel);
+  else differ.push(rel);
+}
+
+const publishedVersion = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8')).version;
+console.log(`  published version: ${publishedVersion}`);
+console.log(`  files compared:    ${comparable.length}`);
+console.log(`  identical:         ${same.length}`);
+
+if (missing.length) {
+  console.log(`  shipped but absent here: ${missing.length}`);
+  for (const f of missing) console.log(`    ${f}`);
+}
+
+if (!differ.length && !missing.length) {
+  console.log(`\nok — ${local.name}@${version} is the code in this tree`);
+  process.exit(0);
+}
+
+console.error(`\nFAIL ${differ.length} shipped file(s) differ from this tree:`);
+for (const f of differ) console.error(`    ${f}`);
+console.error('\nIf you are about to ask someone to test a change, they would be testing');
+console.error('the published bytes above, not what you are reading. Publish first, or point');
+console.error('them at the checkout and say so.');
+process.exit(1);

package/scripts/ci-device-guard.mjs

@@ -23,15 +23,8 @@
 // becomes visible instead of arguable.
 import { spawn } from 'node:child_process';
 import fs from 'node:fs';
+import { deviceCause } from './device-state.mjs';
 
-/** Conditions that are the simulator, not the code. Each seen in a real run. */
-const DEVICE_STATE = [
-  [/NSPOSIXErrorDomain.*code=?\s*60|Operation timed out/i, 'simctl stopped answering (NSPOSIXErrorDomain 60)'],
-  [/did not produce a frame|produced no frame in \d+s/i, 'the daemon is up and the display renders nothing'],
-  [/Timeout waiting for screen surfaces|display surface is not answering|display surface could not be read/i, 'the display surface is wedged'],
-  [/no frames buffered|capture is wedged/i, 'capture stopped'],
-  [/the second app never launched|could not be dispatched/i, 'an app would not launch'],
-];
 
 const udid = process.argv[2];
 const sep = process.argv.indexOf('--');
@@ -56,8 +49,6 @@ const summary = (line) => {
   if (f) { try { fs.appendFileSync(f, `${line}\n`); } catch { /* summaries are a nicety */ } }
 };
 
-const deviceCause = (text) => DEVICE_STATE.find(([re]) => re.test(text))?.[1] ?? null;
-
 const first = await run(cmd);
 if (first.code === 0) process.exit(0);
 

package/scripts/classify-stray.mjs

@@ -0,0 +1,63 @@
+/**
+ * Why does a fingerprint reading not resemble its own screen?
+ *
+ * Its own module, with no side effects, for one reason: this logic has failed
+ * at RUNTIME twice — once reaching for a variable local to another function,
+ * once on declaration order — while being correct both times. `eval-fingerprint.mjs`
+ * runs the whole eval on import, so nothing could test it there, and the only
+ * thing that ever exercised it was a hosted runner at the end of a
+ * fifteen-minute job, in the middle of a report. That is the most expensive
+ * place in this project to find a typo.
+ *
+ * Three causes, and they have different remedies — which is the whole reason to
+ * tell them apart rather than print one sentence about all three:
+ *
+ * - **collided** — neither reading carries a chrome label, so both are
+ *   structure with no name and the fingerprint has nothing left to separate two
+ *   list screens. That is this harness's own subject, and a real finding.
+ * - **wrongScreen** — the tokens are identical to another screen AND this
+ *   screen reads differently in its other rounds, so it is demonstrably
+ *   distinguishable and the tour was simply somewhere else. A tap that missed.
+ * - **underRead** — the reading stayed under the token floor and the screen
+ *   cannot tell itself apart in any round, so we never looked long enough. Ours
+ *   to fix, and nothing about the tour or the fingerprint.
+ *
+ * The middle case is the correction that prompted this. Sparseness alone used
+ * to claim `underRead`, and a wrong turn onto a screen that *legitimately*
+ * reads sparse — the Settings root, at 4 tokens on a runner — is flagged sparse
+ * too. So a genuine tour failure was reported as our instrument's fault, and
+ * the harness's original and correct message had been silenced by an
+ * "improvement".
+ */
+import * as fingerprint from '../src/fingerprint.js';
+
+/**
+ * @param {object} o
+ * @param {object} o.reading           the stray
+ * @param {object|null} o.match        the other screen's reading it most resembles
+ * @param {number} o.bestOther         how much it resembles that one, 0..1
+ * @param {object[]} o.siblings        every reading of the stray's own screen
+ * @param {boolean} o.wasSparse        did it stay under the token floor after retries
+ * @param {string[]} [o.named]         chrome-labelled tokens in the stray
+ * @param {string[]} [o.matchNamed]    chrome-labelled tokens in the match
+ */
+export function classifyStray({
+  reading, match, bestOther, siblings, wasSparse, named = [], matchNamed = [],
+}) {
+  // Does any other round of this same screen read differently from the screen
+  // we collided with? If so this screen CAN be told apart, and a round that
+  // matched the other one exactly was somewhere else.
+  const distinguishable = (siblings ?? [])
+    .some((o) => o !== reading && fingerprint.similarity(o.tokens, match?.tokens ?? []) < 0.99);
+  const identical = bestOther >= 0.99;
+  // Checked first and exclusively: a reading with no names at all cannot be
+  // said to have gone anywhere, because there is nothing in it that would have
+  // named a destination.
+  const collided = identical && named.length === 0 && matchNamed.length === 0;
+  return {
+    collided,
+    wrongScreen: !collided && identical && distinguishable,
+    underRead: !collided && Boolean(wasSparse) && !distinguishable,
+    distinguishable,
+  };
+}

package/scripts/device-state.mjs

@@ -0,0 +1,62 @@
+// Is a failed CI step the device's fault or the check's?
+//
+// Its own module, and the reason is the same one that moved `classifyStray` out
+// of `eval-fingerprint.mjs`: `ci-device-guard.mjs` reads `process.argv` and
+// `process.exit(2)`s at import, so nothing could ever test this table there —
+// and the only thing that exercised it was a hosted runner, at the end of a
+// fifteen-minute job, in the middle of a report. Two runtime bugs in this
+// project came from logic that was correct and had never executed.
+//
+// Every entry here has been seen in a real run. Adding one from imagination is
+// how a guard starts reviving genuine failures into passes.
+
+/** Conditions that are the simulator, not the code. Each seen in a real run. */
+export const DEVICE_STATE = [
+  [/NSPOSIXErrorDomain.*code=?\s*60|Operation timed out/i, 'simctl stopped answering (NSPOSIXErrorDomain 60)'],
+  [/did not produce a frame|produced no frame in \d+s/i, 'the daemon is up and the display renders nothing'],
+  [/Timeout waiting for screen surfaces|display surface is not answering|display surface could not be read/i, 'the display surface is wedged'],
+  [/no frames buffered|capture is wedged/i, 'capture stopped'],
+  [/the second app never launched|could not be dispatched/i, 'an app would not launch'],
+  // A launched app that never comes to the front, seen as the tour waiting for
+  // one of its landmarks on a screen that is showing a clock and nothing else.
+  //
+  // Measured on a runner: `ok launch — launched com.apple.Preferences
+  // (relaunched)` followed by `waited 8000ms for General: "General" is not on
+  // this screen. Visible: 10:50, .?o (the screen has not moved for 6181ms)`.
+  // Two labels, one of them a clock, on a still screen — the device is not
+  // presenting the app, and the guard called that a check failing on its
+  // merits and declined to revive.
+  //
+  // Deliberately narrow. It requires the wait to have failed AND the screen to
+  // have been still AND almost nothing readable: a tour that genuinely asks for
+  // the wrong label has a screen full of other labels, and must keep failing
+  // rather than being retried into a pass.
+  [
+    /never arrived[\s\S]*?Visible:[^\n]{0,24}\(the screen has not moved for \d+ms/i,
+    'a launched app never came to the front (the screen shows a clock and nothing else)',
+  ],
+  // The same condition, now said outright by the step that suffered it instead
+  // of inferred from the shape of the screen afterwards. Item 169 gave `launch`
+  // a pid to compare, so a launch that starts a process and never fronts it
+  // reports itself; this signature fires on the cause rather than on a
+  // consequence that had to be recognised by "two labels, one a clock".
+  //
+  // It cannot be triggered by a tour asking for the wrong label — only a failed
+  // launch emits this sentence — so it needs none of the narrowing above.
+  [
+    /never came to the front within \d+ms/i,
+    'a launched app never came to the front (the launch said so itself, by pid)',
+  ],
+  // Seen on the v0.14.3 bench run: `could not launch com.apple.Preferences:
+  // The system shell (SpringBoard:36454) probably crashed.` The guest's window
+  // server going down is the device, not the check, and nothing here matched it.
+  [
+    /system shell \(SpringBoard[^)]*\) probably crashed/i,
+    "the guest's SpringBoard crashed, so nothing can be fronted",
+  ],
+];
+
+/** The condition this output shows, or null when the check failed on its merits. */
+export function deviceCause(text) {
+  return DEVICE_STATE.find(([re]) => re.test(String(text ?? '')))?.[1] ?? null;
+}