simframe 0.5.0 → 0.6.0

package/skills/simframe/SKILL.md

@@ -0,0 +1,173 @@
+---
+name: simframe
+description: Drive and inspect the iOS Simulator with eyes, hands and memory. Use for any task that involves running, testing, navigating or verifying an iOS app on a simulator — "does this screen look right", "tap through the signup flow", "why is this button not working", "is the list loading". Reads screens as text rather than screenshots, batches whole flows into one command, and verifies each step against what it did last time.
+---
+
+# simframe
+
+A background daemon keeps the simulator's framebuffer warm, reads the screen
+through the accessibility tree and on-device OCR, and remembers which action
+leads from which screen to which. So the three things that make simulator work
+expensive — waiting for screenshots, spending tokens on images, and re-deriving
+the same screen every time — are already paid for.
+
+## Read the screen as text, not as an image
+
+```bash
+simframe ui
+```
+
+```
+iPhone 17 Pro · 402x874pt · screen a1b2c3d4 "Inbox" (known, 3 known exits)
+nav-bar:
+  #1 button    24,64      Back
+  #2 text      201,64     Inbox
+content:
+  #3 cell      201,140    Weekly digest
+  #4 cell      201,196    Payment received
+tab-bar:
+  #5 text      62,835     Inbox
+  #6 text      201,835    Settings
+```
+
+That is the whole screen: region, a number, type, tap point in points, label.
+Measured against the same screen as an image: **~460 tokens of text versus
+~1,600 for a correctly-handled image**, and 10–40× worse than that if the MCP
+image path degrades to base64-as-text. The text also says what is *tappable*
+and where, which an image does not.
+
+The numbers are selectors. Whatever `ui` calls `#3`, you can tap as `#3`.
+
+**Reach for an image only when the text genuinely cannot answer the question:**
+visual layout, colour, spacing, an animation, or something neither the
+accessibility tree nor OCR can see. Then `simframe frame --out=/tmp/s.png`, or
+`sim_look` over MCP.
+
+## Run the whole flow in one command
+
+One command, not one per tap. Each step waits for the screen to settle against
+a baseline captured *before* it, so steps cannot race the UI.
+
+```bash
+cat > /tmp/flow.json <<'JSON'
+[{"tap": "Inbox tab"},
+ {"assert": {"value": "Weekly digest", "is": "visible"}},
+ {"tap": "#3"},
+ {"type": {"into": "Reply", "text": "on it"}},
+ {"scrollTo": "Send"},
+ {"tap": "Send"},
+ {"waitFor": {"value": "Sent", "timeoutMs": 5000}}]
+JSON
+simframe do /tmp/flow.json
+```
+
+Steps stop at the first failure and say which step and why. Measured: **a
+10-step flow is one command, ~5 seconds, ~460 tokens, zero images.**
+
+Steps — every place a control is named accepts a selector:
+
+| Act | Check |
+| --- | --- |
+| `{"tap": "Save"}` · add `"index"` if a label is ambiguous | `{"assert": {"value": "Saved", "is": "visible"}}` |
+| `{"type": {"into": "Name", "text": "..."}}` | `is`: `visible` · `gone` · `enabled` · `disabled` · `value` (with `equals`) |
+| `{"paste": {"into": "Notes", "text": "long text"}}` | `{"waitFor": {"value": "Saved", "timeoutMs": 5000}}` |
+| `{"scroll": "down"}` · `{"scrollTo": "Delete account"}` | `{"settle": {"stableMs": 600}}` |
+| `{"swipe": {"from": [x,y], "to": [x,y]}}` | `{"pause": 300}` |
+| `{"button": "HOME"}` | |
+| `{"launch": {"value": "com.example.app", "relaunch": true, "args": ["-uiTest","1"]}}` | |
+| `{"openUrl": "myapp://path"}` | |
+| `{"permission": {"value": "photos", "grant": "grant", "bundleId": "com.example.app"}}` | |
+
+## Selectors
+
+| | |
+| --- | --- |
+| `#3` | the number `simframe ui` gave it. Cheapest, and unambiguous. |
+| `"Save"` · `the Assets tab` · `back` | resolved by intent — verbs, typos, synonyms, icon-only controls by their common name |
+| `@120,400` | raw point coordinates. Last resort; it cannot tell you it missed. |
+
+A ref is valid only while that screen is showing. Use one on a different screen
+and it refuses rather than tapping whatever now sits at those coordinates.
+
+## Every step is verified, and the verdict means something
+
+simframe records which action led from which screen to which, so it can check
+each step against what that action did here last time.
+
+| Verdict | What it means | What to do |
+| --- | --- | --- |
+| `ok` | landed where this action has landed before | nothing |
+| `unverified` | this action has not been taken on this screen before | nothing — it is learning. Run the flow again and it becomes `ok`. |
+| `no-visible-change` | the screen is stable and nothing moved | the tap may have missed, or its effect may be invisible (a checkbox, a button state). Check with `simframe ui`, not by waiting longer. |
+| `unexpected-screen` | it went somewhere it has not gone before from here | the flow **stops here**. Read the map it returns: either the app changed, or the tap hit the wrong thing. |
+
+A first run through a new part of an app is mostly `unverified`, and a
+transition-kind mismatch is reported inside `ok` rather than failing — that
+classifier is noisy and a verdict that cries wolf teaches you to ignore
+verdicts.
+
+## Navigate by memory
+
+Once simframe has been somewhere, getting back is a search over remembered
+transitions — no reasoning, no images.
+
+```bash
+simframe screens                 # what it knows, and how many exits each has
+simframe goto "Settings"         # plan a route and walk it, verifying each step
+simframe do /tmp/flow.json --save=checkout   # save it if every step verified
+simframe flow run checkout       # replay it
+```
+
+`goto` refuses rather than guesses. Unknown screen, a name that fits two
+screens equally, no remembered path — each is reported, with what it does know.
+A wrong route is worse than no route, because it taps things.
+
+## It refuses rather than guesses
+
+When two controls answer a query equally well, simframe lists them and asks
+instead of picking. That is deliberate: a wrong tap can *do something* and
+leave you believing it did the right thing. Pass `index`, or use a `#ref`.
+
+## Everything speaks JSON
+
+`--json` is on every command, so nothing has to be parsed out of prose:
+
+```bash
+simframe ui --json | jq '.elements[] | select(.type=="button") | .label'
+simframe do /tmp/flow.json --json | jq '.results[] | select(.ok==false)'
+```
+
+## The cheap-to-expensive order
+
+1. `simframe state` — has anything changed at all? Cheapest thing there is.
+2. `simframe ui` — what is on screen and what can I tap? Text.
+3. `simframe do` — act, in a batch, with asserts inside the batch.
+4. `simframe frame` / `sim_look` — pixels. Only for a question about pixels.
+
+## When something is wrong with simframe itself
+
+```bash
+simframe doctor           # capture engine, input driver, a11y, OCR — each honestly
+simframe doctor --strict  # any degraded layer is a non-zero exit
+```
+
+simframe falls back when it must — the simctl capture loop instead of the
+daemon, idb instead of the in-process input and accessibility paths — but it
+never falls back quietly. If
+`doctor` says a layer is degraded, believe it: the numbers above assume the
+daemon.
+
+## Other commands
+
+```bash
+simframe start [device]     # capture starts on first use anyway
+simframe devices            # booted simulators
+simframe recall             # what happened in the last ~60s, as text
+simframe strip              # recent frames tiled into one image, for an animation
+simframe find "the save button"   # resolve an intent without acting on it
+simframe wait --mode=settle       # block until the screen stops reacting
+```
+
+`recall` matters more than it looks: if you look up and the screen is already
+different, it tells you what happened and when, instead of you re-running the
+action to find out.

package/src/actions.js

@@ -6,14 +6,25 @@ import * as api from './index.js';
 import * as graph from './graph.js';
 import * as input from './input.js';
 import * as intent from './intent.js';
-import { launchApp, openUrl, setPasteboard, terminateApp } from './simctl.js';
+import { launchApp, openUrl, setPasteboard, setPermission, terminateApp } from './simctl.js';
 
 const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
 const MAX_PAUSE_MS = 5000;
+/** How long a text field needs after being tapped before it holds the keyboard focus. */
+const FOCUS_SETTLE_MS = 150;
+const POLL_MS = 250;
+/** A list that has not produced the target in this many screens does not contain it. */
+const MAX_SCROLLS = 20;
 
+/**
+ * Steps that change the device. Only these get a settle wait and a verified
+ * edge in the graph — asserting something is on screen does not move it.
+ * `scrollTo` is here because it scrolls; `permission` because a granted
+ * permission can change what the app shows.
+ */
 const ACTION_STEPS = new Set([
-  'tap', 'tapAt', 'type', 'paste', 'swipe', 'scroll', 'button', 'key',
-  'launch', 'terminate', 'openUrl', 'confirm', 'chooseAny',
+  'tap', 'tapAt', 'type', 'paste', 'swipe', 'scroll', 'scrollTo', 'button', 'key',
+  'launch', 'terminate', 'openUrl', 'confirm', 'chooseAny', 'permission',
 ]);
 
 /** Accept both `{tap: "Save"}` shorthand and `{action: "tap", target: "Save"}`. */
@@ -33,6 +44,34 @@ export function normalizeStep(raw) {
   return step;
 }
 
+/**
+ * Does this verdict mean the flow went somewhere nobody intended?
+ *
+ * Only an unexpected *screen* does. `unexpected-transition` is not a verdict at
+ * all any more — a noisy classifier disagreeing about whether a tab switch was
+ * a push or a pop is not a reason to call a correct navigation wrong, and a
+ * verdict that cries wolf trains you to ignore verdicts.
+ */
+export function wrongTurnFrom(verification) {
+  return verification?.verdict === 'unexpected-screen';
+}
+
+/**
+ * What a halted step does to the run as a whole.
+ *
+ * Both halves matter, and only one of them used to happen: the step is marked
+ * failed, AND so is the run. Without the second, `ok` meant no more than
+ * "nothing threw", so a flow stopped dead at step 0 by a wrong turn reported
+ * "flow completed" with no error — the exact shape of failure the verdict
+ * exists to make loud.
+ */
+export function haltDecision({ verification, stopOnUnexpected = true, continueOnError = false } = {}) {
+  if (!wrongTurnFrom(verification) || !stopOnUnexpected || continueOnError) {
+    return { halt: false, failRun: false, error: null };
+  }
+  return { halt: true, failRun: true, error: `${verification.verdict}: ${verification.detail}` };
+}
+
 export async function runScript(
   deviceQuery,
   {
@@ -49,6 +88,9 @@ export async function runScript(
     // Stop when a verified step lands somewhere it should not have. A flow
     // continuing past a wrong turn taps controls on a screen nobody intended.
     stopOnUnexpected = true,
+    // Rebuild the HID session and retry once when a hardware button provably
+    // did nothing. Off only for a caller deliberately testing that path.
+    recoverInput = true,
     options,
   } = {},
 ) {
@@ -74,6 +116,14 @@ export async function runScript(
   const frames = [];
   let failed = false;
   let carriedScreen = null;
+  // The last reading of where we ended up, confirmed or not. The compact map
+  // the caller returns to Claude is rendered from this, so describing the end
+  // state costs nothing beyond the verification pass the flow already ran.
+  let endScreen = null;
+  // At most one recovery per run. Pressing home while already on the springboard
+  // moves nothing and is not a failure, so an unbounded retry would rebuild the
+  // session and press again on every such step for no reason.
+  let inputRecovered = false;
 
   for (const [i, raw] of steps.entries()) {
     const step = normalizeStep(raw);
@@ -93,9 +143,9 @@ export async function runScript(
     // What this action did last time it was taken here, if ever.
     const prediction = verify && beforeScreen?.hash ? graph.predict(udid, beforeScreen, step) : null;
     try {
-      const detail = await runStep(deviceQuery, udid, step, { screen, options, frames });
-      let settled = null;
-      if (autoSettle && ACTION_STEPS.has(step.action)) {
+      let detail = await runStep(deviceQuery, udid, step, { screen, options, frames });
+      const settleFor = async () => {
+        if (!autoSettle || !ACTION_STEPS.has(step.action)) return null;
         const w = await api.waitFor(deviceQuery, {
           mode: 'settle',
           since: before,
@@ -103,13 +153,37 @@ export async function runScript(
           timeoutMs: step.timeoutMs ?? timeoutMs,
           options,
         });
-        settled = {
+        return {
           ok: w.satisfied,
           waitedMs: w.waitedMs,
           sawChange: w.sawChange,
           stalled: Boolean(w.stalled),
           noVisibleChange: Boolean(w.noVisibleChange),
         };
+      };
+      let settled = await settleFor();
+
+      // A hardware button that moved nothing did not arrive.
+      //
+      // Input is the one path with no feedback, so a dispatched Indigo message
+      // reports success whether or not the device acted on it — measured, a
+      // long-running daemon returned `press in 66ms` with the screen frozen,
+      // and the same press worked on a fresh daemon. The frames are the only
+      // witness, and by here we have them.
+      //
+      // Only buttons, and only on no visible change. Home and lock always move
+      // the screen, so nothing moving is unambiguous; a tap that changes
+      // nothing is ordinary, and retrying one could act twice. Retrying an
+      // action that provably did nothing is not a repeat — it is the first
+      // attempt that counts.
+      if (recoverInput && !inputRecovered && step.action === 'button' && settled?.noVisibleChange) {
+        inputRecovered = true;
+        const reset = await input.resetSession(udid);
+        if (reset) {
+          detail += ' [input was not being delivered; HID session reset and retried]';
+          await runStep(deviceQuery, udid, step, { screen, options, frames });
+          settled = await settleFor();
+        }
       }
       // Verify against what was predicted, and remember what actually
       // happened. Without this a step that moved the screen the wrong way
@@ -130,17 +204,14 @@ export async function runScript(
         // independent readings, which is the thing `settled` was standing in
         // for. Requiring both meant a screen that settled slowly recorded
         // nothing at all.
+        endScreen = afterScreen;
         if (afterScreen.confirmed && afterScreen.hash) {
           graph.record(udid, { from: beforeScreen, action: step, to: afterScreen, kind });
           carriedScreen = afterScreen;
         }
       }
 
-      // Only an unexpected *screen* stops a flow. `unexpected-transition` is no
-      // longer a verdict at all — a noisy classifier disagreeing about whether
-      // a tab switch was a push or a pop is not a reason to call a correct
-      // navigation wrong.
-      const wrongTurn = verification?.verdict === 'unexpected-screen';
+      const wrongTurn = wrongTurnFrom(verification);
       const note = settled?.noVisibleChange ? ' [no visible change]' : '';
       results.push({
         index: i,
@@ -151,9 +222,11 @@ export async function runScript(
         detail: `${detail}${note}${wrongTurn ? ` [${verification.verdict}: ${verification.detail}]` : ''}`,
         settled,
       });
-      if (wrongTurn && stopOnUnexpected && !continueOnError) {
+      const halt = haltDecision({ verification, stopOnUnexpected, continueOnError });
+      if (halt.halt) {
         results[results.length - 1].ok = false;
-        results[results.length - 1].error = `${verification.verdict}: ${verification.detail}`;
+        results[results.length - 1].error = halt.error;
+        failed = halt.failRun;
         break;
       }
     } catch (err) {
@@ -168,6 +241,7 @@ export async function runScript(
     // Returned so a run that verified end to end can be handed straight to
     // navigate.saveFlow without the caller reassembling what it just ran.
     steps,
+    endScreen,
     results,
     ok: !failed,
     totalMs: Date.now() - startedAt,
@@ -203,10 +277,14 @@ async function runStep(deviceQuery, udid, step, ctx) {
     }
     case 'type': {
       if (step.into) {
-        const { node } = await input.tapLabel(udid, step.into, { index: step.index });
-        await sleep(150);
+        // locate, not tapLabel: tapLabel asks the accessibility tree directly,
+        // so a field that only OCR can see was untypeable, and a selector
+        // (`#4`, `@x,y`) meant nothing here.
+        const found = await api.locate(deviceQuery, step.into, { index: step.index, refresh: step.refresh });
+        await input.tapPoint(udid, found.target.x, found.target.y);
+        await sleep(FOCUS_SETTLE_MS);
         await input.typeText(udid, step.text ?? step.value);
-        return `typed into "${node.label ?? step.into}"`;
+        return `typed into "${found.target.label}" at ${found.target.x},${found.target.y}`;
       }
       await input.typeText(udid, step.text ?? step.value);
       return 'typed text';
@@ -245,9 +323,15 @@ async function runStep(deviceQuery, udid, step, ctx) {
     case 'key':
       await input.pressKey(udid, step.value ?? step.code);
       return `pressed key ${step.value ?? step.code}`;
-    case 'launch':
-      await launchApp(udid, step.value ?? step.bundleId);
-      return `launched ${step.value ?? step.bundleId}`;
+    case 'launch': {
+      const bundleId = step.value ?? step.bundleId;
+      await launchApp(udid, bundleId, {
+        args: step.args ?? [],
+        env: step.env ?? {},
+        terminateFirst: step.relaunch === true,
+      });
+      return `launched ${bundleId}${step.relaunch ? ' (relaunched)' : ''}`;
+    }
     case 'terminate':
       await terminateApp(udid, step.value ?? step.bundleId);
       return `terminated ${step.value ?? step.bundleId}`;
@@ -305,6 +389,91 @@ async function runStep(deviceQuery, udid, step, ctx) {
       }
       throw new Error(`"${target}" is still on screen`);
     }
+    // Bring something into view. A control that scrolled off the bottom of a
+    // list is not missing, and "not on this screen" is the wrong answer to give
+    // about it.
+    case 'scrollTo': {
+      const query = step.value ?? step.target ?? step.label;
+      const dir = String(step.direction ?? 'down').toLowerCase();
+      const max = Math.min(MAX_SCROLLS, step.maxScrolls ?? 6);
+      for (let i = 0; i <= max; i += 1) {
+        try {
+          const found = await api.locate(deviceQuery, query, { index: step.index, refresh: i > 0 });
+          return `"${found.target.label}" is in view at ${found.target.x},${found.target.y}` +
+            (i ? ` after ${i} scroll${i === 1 ? '' : 's'}` : ' already');
+        } catch (err) {
+          if (i === max) throw new Error(`scrolled ${dir} ${max}x without finding ${query}: ${err.message}`);
+        }
+        await runStep(deviceQuery, udid, { action: 'scroll', value: dir }, ctx);
+        await api.waitFor(deviceQuery, { mode: 'stable', stableMs: 250, timeoutMs: 2500, options: ctx.options });
+      }
+      throw new Error(`could not bring ${query} into view`);
+    }
+
+    // Wait for a selector rather than a label, so it works on screens the
+    // accessibility tree never described.
+    case 'waitFor': {
+      const query = step.value ?? step.target ?? step.text;
+      const limit = Date.now() + (step.timeoutMs ?? 8000);
+      let lastError = 'never appeared';
+      for (let attempt = 0; ; attempt += 1) {
+        try {
+          const found = await api.locate(deviceQuery, query, { index: step.index, refresh: attempt > 0 });
+          return `"${found.target.label}" appeared at ${found.target.x},${found.target.y}`;
+        } catch (err) {
+          lastError = err.message;
+        }
+        if (Date.now() >= limit) break;
+        await sleep(POLL_MS);
+      }
+      throw new Error(`waited ${step.timeoutMs ?? 8000}ms for ${query}: ${lastError}`);
+    }
+
+    // One assert step for every condition, because `assertText` could only ask
+    // one question and the interesting ones are about state: is Save enabled
+    // yet, does the field hold what was typed into it.
+    case 'assert': {
+      const query = step.value ?? step.target ?? step.text;
+      const want = String(step.is ?? (step.gone ? 'gone' : 'visible')).toLowerCase();
+      let found = null;
+      try {
+        found = await api.locate(deviceQuery, query, { index: step.index, refresh: step.refresh });
+      } catch (err) {
+        if (want === 'gone') return `${query} is gone`;
+        throw new Error(`${query}: ${err.message}`);
+      }
+      const t = found.target;
+      switch (want) {
+        case 'visible':
+          return `${query} is on screen at ${t.x},${t.y}`;
+        case 'gone':
+          throw new Error(`${query} is still on screen at ${t.x},${t.y}`);
+        case 'enabled':
+          if (t.enabled === false) throw new Error(`"${t.label}" is disabled`);
+          return `"${t.label}" is enabled`;
+        case 'disabled':
+          if (t.enabled !== false) throw new Error(`"${t.label}" is not disabled`);
+          return `"${t.label}" is disabled`;
+        case 'value': {
+          const expected = String(step.equals ?? step.text ?? '');
+          const actual = [t.label, t.value, ...(t.aliases ?? [])].filter(Boolean).join(' ');
+          if (!actual.toLowerCase().includes(expected.toLowerCase())) {
+            throw new Error(`expected "${expected}" but read "${actual}"`);
+          }
+          return `"${expected}" is what ${query} reads`;
+        }
+        default:
+          throw new Error(`unknown assert condition "${want}" — visible, gone, enabled, disabled or value`);
+      }
+    }
+
+    // Answering a system permission alert is not a test of the app. Setting the
+    // permission is.
+    case 'permission': {
+      const service = step.value ?? step.service;
+      return await setPermission(udid, step.grant ?? step.action ?? 'grant', service, step.bundleId);
+    }
+
     case 'look': {
       const frame = await api.getFrame(deviceQuery, { detail: step.detail ?? 'normal', options: ctx.options });
       ctx.frames.push({ label: step.label ?? `step frame`, png: frame.png });