simframe 0.18.0 → 0.19.1

package/README.md

@@ -3,152 +3,38 @@
 [![ci](https://github.com/lvlrSajjad/simframe/actions/workflows/ci.yml/badge.svg)](https://github.com/lvlrSajjad/simframe/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/simframe.svg)](https://www.npmjs.com/package/simframe)
 [![license](https://img.shields.io/npm/l/simframe.svg)](./LICENSE)
+[![Glama score](https://glama.ai/mcp/servers/lvlrSajjad/simframe/badges/score.svg)](https://glama.ai/mcp/servers/lvlrSajjad/simframe)
 
-**Eyes, hands and memory for an agent driving the iOS Simulator or an Android emulator.**
+**Eyes, hands and memory for a coding agent driving the iOS Simulator or an
+Android emulator.** The agent reads the screen as text with tap points, runs a
+whole flow in one call, and remembers screens it has seen — so it stops paying
+a screenshot, a model round trip and a re-read for every single step.
 
-[Website](https://lvlrsajjad.github.io/simframe/) · [npm](https://www.npmjs.com/package/simframe)
+![simframe driving Settings on an iPhone 17 Pro simulator: one call, four verified steps, no screenshots, and the screen returned as text with tap points](https://raw.githubusercontent.com/lvlrSajjad/simframe/main/docs/simframe-demo.gif)
 
-An agent driving a simulator is slow for three reasons, and only the first
-one is obvious:
+## Install in ten seconds
 
-1. **Every look is a wait.** `simctl io screenshot` costs ~130 ms of blocking
-   latency, paid again on every glance — and paid twice whenever the agent
-   captures mid-animation and has to look again.
-2. **Every step is a round trip.** Tap, screenshot, reason, tap, screenshot. A
-   twelve-step flow costs twelve model turns, and the model turns cost far more
-   than the milliseconds.
-3. **Nothing is remembered.** The same screen gets re-read and re-reasoned about
-   every single time it appears.
+You need a Mac with Xcode (you have one if you have a simulator) and Node 18+.
+Nothing else — no idb, no Appium, no Python.
 
-And there is a fourth that is pure waste: **an image is the most expensive way
-to ask what is on screen.** A screenshot costs ~1,600 tokens when it is handled
-as a native image block and 15,000–25,000 when it is not, and it does not tell
-you what is tappable or where — you have to measure that by eye.
-
-simframe attacks all four: a background loop keeps the newest frame warm, whole
-flows run in one call, screens the agent has seen before are answered from
-memory, and every answer is text with tap points in it. Nothing returns an
-image unless you ask for one.
-
-### It is also a study case, and the numbers are the point
-
-Almost everything here was decided by a measurement rather than by an argument,
-and several of those measurements **reversed a decision that had already
-shipped**. They are written down in full, with what we expected beforehand, what
-it cost to find out, and the mistakes made getting there — because a benchmark
-that only records the winner teaches nothing.
-
-| | |
-| --- | --- |
-| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | the record: N, median, p95, machine, every round |
-| [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) | what we believed *before* measuring, and the fourteen times it was wrong |
-| [`docs/DEFERRED.md`](docs/DEFERRED.md) | every known defect, open or closed, with the evidence |
-| [`docs/ARTICLE.md`](docs/ARTICLE.md) | the argument the whole thing adds up to |
-
-The headline results, all free to reuse:
-
-- **A warm transition graph removes ~27% of an agent's model round trips** —
-  isolated with a three-pass field experiment, and corroborated to three points
-  by an independent estimate. Operator knowledge is worth twice that, and no
-  tool here carries it yet. ([§14](docs/EXPERIMENTS.md))
-- **A bigger local judge is not a better one.** `qwen3:14b` scored *lower* than
-  `qwen3:8b` — 82% against 91% — while being 79% larger and 62% slower. A
-  one-line threshold on a number already computed beat all of them at **95% and
-  zero latency**.
-- **Every interactive element came from the accessibility tree — 72 of 72 — and
-  83% of them have no text at all.** Screen *recognition*, though, did not need
-  the tree once. The ladder splits there: semantics need a tree, identity needs
-  pixels.
-- **Capture is ~31× faster than a screenshot** for the same work, and the
-  capture primitive itself about a thousand times faster.
-- **Three independent reporters converged on one thing**: a refusal is cheap and
-  a confident wrong answer is expensive. Every serious bug they found was a
-  component reporting more certainty than it had.
-
-## What changed, measured
-
-Same four-tab navigation flow, on a real production app:
-
-| | Before | With simframe |
-| --- | --- | --- |
-| Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
-
-| "Did anything change?" | a full image | **~2 ms**, text only |
-| Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
-| A 4-step flow, verified | 4+ model round trips | **1 call**, 3.6 s |
-| Same flow, 3rd run | no improvement — every run is the first | **3.7 s, 4/4 from memory, 4/4 verified** |
-| A 10-step flow | 10 turns, 10 images (~16,000 tokens at best) | **1 turn, 0 images, ~1,650 characters** |
-| Reading a screen | an image: ~1,600 tokens, no tap points | **~330 tokens** of text, with tap points |
-
-Every figure above is the cost inside a live process — the MCP server, or the
-daemon answering a socket — which is how an agent actually uses simframe. A
-one-shot `simframe` command from a shell pays about 200 ms of Node startup on
-top, and a frame sitting on an idle screen can be older than 20 ms because the
-capture loop throttles when nothing moves. `~20 ms` is the read, not the
-process.
-
-The four-tab tour, three times back to back from a cleared memory:
-
-| Pass | Wall clock | Steps verified | Controls from memory |
-| --- | --- | --- | --- |
-| 1 | 10.2 s | 0/4 — nothing is known yet | **4/4** |
-| 2 | **3.6 s** | **4/4** | **4/4** |
-| 3 | **3.7 s** | **4/4** | **4/4** |
-
-Every step is checked against what the same action did last time, and the run
-records its own preconditions — which input path, which daemon, whether the
-daemon was replaced mid-run — so a regression shows up in the measurement rather
-than hiding inside it. Earlier versions of this table quoted 7.4 s → 3.3 s with
-verification switched off; those numbers were measured while input was silently
-falling back to a slower path and the capture daemon was being replaced by every
-command, so they measured two bugs rather than the tool.
-
-## Install
+**Claude Code** — as a plugin, which brings the MCP server and the
+[skill](skills/simframe/SKILL.md) that teaches the protocol in one install:
 
 ```bash
-npm install -g simframe
-simframe doctor
+claude plugin marketplace add lvlrSajjad/simframe
+claude plugin install simframe@simframe
 ```
 
-**Whichever of those two commands you run first** builds a small Swift daemon
-from source — including `doctor`, which is why a cold `doctor` takes around 15
-seconds and every later one takes two. It needs the Xcode command line tools,
-which you already have if you have a simulator. Without them simframe falls
-back to the original `simctl` loop and says so.
-
-If more than one simulator is booted, name the one you mean — `--device=<udid>`,
-or `export SIMFRAME_DEVICE=<udid>` once per shell. simframe refuses to choose
-for you, because the first booted device is nobody's idea of "yours" and the
-command that would act on it is a tap.
-
-`doctor` checks each capability separately and tells you what you have:
-
-```
-ok   xcrun              xcrun version 72.
-ok   sips               available
-ok   input driver      simframed: Indigo HID
-ok   accessibility tree simframed: AXPTranslator, host-side
-ok   on-device OCR      available
-ok   booted simulator   iPhone 17 Pro (iOS 26.5)
-ok   capture            frame #888 322x700 in 2ms (age 538ms)
-ok   sensor mode        full — accessibility and OCR fused on every read (~164ms)
-ok   local supervisor   none — not requested (SIMFRAME_SUPERVISOR is unset)
-ok   local planner      none — not requested (SIMFRAME_PLANNER is unset)
-```
-
-The last three are experiments and `none` is their normal answer. See
-[Local tiers, off by default](#local-tiers-off-by-default).
-
-### Claude Code
+or the MCP server alone:
 
 ```bash
 claude mcp add --scope user simframe -- npx -y simframe mcp
 ```
 
-`--scope user` makes it available in every session; without it the server is
-registered only for the directory you ran the command in.
+Pick one, not both: two installs mean two servers driving the same device. If
+you added it with `claude mcp add` before, `claude mcp remove simframe` first.
 
-### Any other MCP client
+**Claude Desktop** — Settings → Developer → Edit Config, then add:
 
 ```json
 {
@@ -158,271 +44,52 @@ registered only for the directory you ran the command in.
 }
 ```
 
-## Recovering without a round trip
-
-The measured cost of driving an app is not perception — warm, an
-accessibility-only read is 85 ms and a fused read 142 ms. It is **round trips**:
-in one instrumented run, 75% of the wall time was the agent thinking and the
-call boundary, not simframe working. So the tools that matter most are the ones
-that let a batch survive a problem instead of handing it back.
-
-**Fallback selectors.** `{"tap": "Save", "or": ["Done", "Confirm"]}` — tried
-locally in order, only an exhausted list reaching the model. Eligible after a
-selector that did not *resolve* and nothing else, because retrying from a screen
-you did not expect to be on is a second guess. A destructive-looking label is
-refused as a substitute even if you list it.
-
-**Steps that may legitimately have nothing to act on.** `{"tap": "Not Now",
-"optional": true}` is skipped when nothing matches and runs normally when
-something does. It exists because the opposite cost real time: a batch that
-included a dismissal for a first-launch sheet lost **six correct steps** on the
-next run because the sheet *did not appear*. A nag screen, a permission prompt,
-a "What's New" or a cold-start splash otherwise makes a flow unbatchable, which
-is the expensive outcome — a call per step instead of a call per flow. Only a
-selector that resolved to **nothing** is absorbed: a target that is on screen
-twice is ambiguous, not absent, and must still verify. Skipped steps are
-reported, because "the sheet was gone" and "the sheet was dismissed" are
-different facts.
-
-**`{"seek": "change username", "budget": 6}`** opens containers, checks, and
-comes back, depth first, inside a hard budget. It **acts** — opening a door
-changes state — and it refuses to open anything that commits, abandons or
-answers. It does not tap the target; it leaves you on the screen where the target
-resolves.
-
-**`{"sweep": "all", "fill": {…}}`** reads a long screen a viewport at a time and
-fills each field while it is on screen. A form taller than the screen is only
-knowable in pieces — the tree publishes what is rendered — and one scroll gesture
-travels a non-deterministic distance, so finding a field and scrolling back to it
-does not work. Sweeping does: on a real web form it filled every field in one
-call. It detects both ends by measuring how far the *content* moved, ignoring
-fixed chrome, which is the only reliable signal available since nothing reports a
-scroll offset.
-
-**`worked here before:`** puts the graph's own vocabulary in the map, most-used
-first, rather than reporting a count. When the remembered controls are *not* on
-the screen it says so instead, because that means two screens share one
-fingerprint — and confident advice on a misidentified screen is how a remembered
-label ends up pointing at a submit button.
-
-## Local tiers, off by default
-
-Two on-device model experiments, both `none` unless asked for, both degrading to
-the existing matcher-then-model ladder, and CI runs with both off. They ship no
-weights: Apple's Foundation Models framework has nothing to download, which is
-the whole reason it clears this project's non-goal on shipping model weights.
-
-| flag | what it does |
-|---|---|
-| `--sensor=ax-first` | read the accessibility tree alone (~50 ms) and pay for OCR only when a resolve fails |
-| `--planner=apple` | order the containers `seek` opens; it cannot choose an action |
-| `--supervisor=apple` | when a step fails, answer `wait`, `retry` or `stop` — nothing else — before the failure reaches the model |
-
-All three are also per-call arguments on every MCP tool, because an MCP server's
-environment is fixed when it spawns and comparing two modes inside one session
-was otherwise impossible.
-
-**What is measured and what is not.** The ranker: 5 of 6 top-1 on hand-written
-cases, median 564 ms warm, and on a real exploration it went to the right region
-in two steps where reading order wandered into version strings. `ax-first` made
-no measurable difference to how an agent drove a real app, with one small
-regression and one small win.
-
-**The supervisor, on 22 labelled failures from a seeded React Native app** —
-and the number to judge it by is round trips, not accuracy:
-
-| | |
-|---|---|
-| handled locally, no round trip | **9 of 22** |
-| escalated to Claude | 13 of 22 |
-| a model round trip, measured in the field | 10–16 s |
-
-So roughly 90–145 s saved on that population. Accuracy was 77 / 82 / 86% across
-three runs of the *same* questions — it is **not deterministic**, and every
-earlier single-run figure in this project carried that spread without reporting
-it. All of its errors were `wait` where `stop` was right, which is the cheap
-direction: a wrong `wait` costs a settle and a re-run, and `stop` already means
-"hand back to Claude with the unattempted steps" rather than "give up".
-
-**Still a bench, not the field.** These are fixtures we designed, on a testbed we
-built, labelled by the person who then scored them. Numbers and conditions are in
-[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md); the judgements, including a phase
-cancelled by its own measurement, are in [`docs/DECISIONS.md`](docs/DECISIONS.md).
-
-**And the measurements that changed our minds** are in
-[`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md), with what we expected beforehand
-written down beside each — including the capacity comparison the supervisor
-design was assumed to make unnecessary. Asked the same 22 situations, three
-times each:
-
-| arm | accuracy | median | deterministic |
-|---|---|---|---|
-| always the commonest answer | 55% | — | — |
-| **`stillMs > 3000ms`, no model at all** | **95%** | **0 ms** | yes |
-| Apple Foundation Models (~3B, on-device) | 77 / 82 / 86% | ~640 ms | **no** |
-| `qwen3:8b` via Ollama (4-bit, 5.2 GB) | **91%** | 919 ms | yes |
-| `qwen3:14b` via Ollama (4-bit, 9.3 GB) | 82% | 1,489 ms | yes |
-
-Three results, and the third is the one that matters. The larger model scored
-*lower* than the smaller one. A free threshold on a number the daemon already
-computes beat all three, on a population it was never fitted to. And **the
-accuracy ranking inverts the safety ranking**: every arm errs in one direction
-only, Apple's errors are all `wait` where `stop` was right and both Qwen arms'
-are all `stop` where `wait` was right — and a wrong `stop` abandons a plan that
-would have worked, while a wrong `wait` costs a settle. A comparison reporting
-only the percentages would have recommended the wrong model.
-
-The Ollama arm is an **experiment, not a recommendation**: off unless named,
-no weights shipped, no dependency added, and every arm reads the same briefing
-out of `native/supervise.swift` so no arm is answering a different question.
-
-**Two further results, both negative, both worth more than the table.** A
-*cascade* — free rule first, model where it is unsure, Claude after that — was
-worse at every abstention band we tried (95% for the rule alone; 91%, 86%, 82%
-as more was handed to the model). And giving the model a fourth word, `abstain`,
-cost the Apple arm about a third of its accuracy (77/82/86% → 45/50/55%) **while
-it never used the word once**. One added paragraph, nothing else changed. So the
-fourth word ships off, and "add an abstain token" became "find a judge that will
-use one".
-
-The supervisor's whole vocabulary is three words on purpose. It cannot invent a
-step, skip one, substitute a target or continue past an unexpected screen — not
-because a threshold forbids it but because those are not answers it can give.
-That constraint replaced an earlier version of the same idea that was given
-latitude over *what* to open and pressed a button labelled "YES, THIS FIXED MY
-PROBLEM" in a live app.
-
-## Capabilities are independent
-
-Each layer works without the ones above it, and `doctor` tells you which you
-have. **Observation needs nothing but Xcode.**
-
-| Capability | Needs | Without it |
-| --- | --- | --- |
-| Watch the screen, wait, recall | nothing extra | — |
-| Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
-| Tap, type, swipe | nothing extra | simframe observes but cannot touch |
-| Accessibility tree | nothing extra | OCR alone still yields labels and coordinates |
-
-`simframe doctor` names which engine is carrying each capability, per device.
-**Nothing beyond Xcode is required.** Capture, input, text recognition and the
-accessibility tree all run in-process, in one daemon.
-
-[`idb`](https://fbidb.io) is still accepted as a fallback for input and for the
-tree, for a machine where the daemon cannot run — and `SIMFRAME_AX_DRIVER=idb`
-forces the tree back onto it, which is the escape hatch if an Xcode upgrade
-breaks the host-side path.
-
-```bash
-# optional fallback, not a requirement
-brew tap facebook/fb && brew install idb-companion && pipx install fb-idb
-```
-
-## Android
-
-simframe's second backend drives an Android emulator with the same commands, the
-same screen map and the same memory as a simulator. Everything above the
-platform boundary — the frame store, settle, the structural fingerprint, the
-screen map, refs and the transition graph — runs on it unmodified, because the
-boundary hands it frames and nothing above it knows what a simulator is.
+**Cursor** — [![Install in Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en/install-mcp?name=simframe&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInNpbWZyYW1lIiwibWNwIl19)
+or add the same `simframe` block to `~/.cursor/mcp.json`.
 
-| Capability | Android | How |
-| --- | --- | --- |
-| Watch the screen, wait, recall | yes | frames at **41 ms** through the emulator console, host-side — no adb in the capture path |
-| Read labels + coordinates from pixels | yes | the same Vision OCR + CV, off the same PNG |
-| Screen map, refs, screen memory, the graph | yes | unchanged above the boundary |
-| Tap, type, swipe, keys | yes | the console's `event mouse` as a real down/move/up, `event text` for characters, `input keyevent` for keys |
-| Clipboard, and `paste` into a field | yes | the emulator's gRPC `setClipboard`, over `node:http2`, no dependency, then `KEYCODE_PASTE` to deliver it |
-| List/resolve devices, launch, terminate, open a URL, permissions | yes | `adb`, with the permission state read back off the device |
-| Accessibility tree | **not available (OCR + CV only)** | `uiautomator dump` costs **2,012 ms** a read, against 45 ms for the iOS tree. See [`docs/DEFERRED.md`](docs/DEFERRED.md) |
-| A launch confirmed to have reached the front | **not available (the launch is not checked)** | iOS compares the pid `simctl launch` printed against the pid the device reports as frontmost, in **2–5 ms**. Nothing here reports either; `am start` fronts synchronously, which is why it has not bitten — but that is not a check. See [`docs/DEFERRED.md`](docs/DEFERRED.md) |
+**Codex, Windsurf, Zed, VS Code, anything else that speaks MCP** — the same
+JSON block, wherever that client keeps its `mcpServers`.
 
-```bash
-# an emulator is found the same way a simulator is
-simframe devices          # ● Small_Phone_API_36  Android 16 (API 36)  emulator-5554
-simframe ui --device=emulator-5554
-simframe do --device=emulator-5554 flow.json
-```
+Then boot a simulator and ask the agent for something: *"open my app and tap
+through the signup flow"*, *"is the list on the Orders tab loading?"*, *"why
+does the Save button do nothing?"*. The first call builds a small Swift daemon
+from source — about 15 seconds, once.
 
-A host with a booted simulator **and** a booted emulator has no default, and
-simframe will not pick one for you: preferring iOS because it came first would
-tap a simulator while you were driving an emulator, and acting on the wrong
-device is worse than refusing. So a command with no device names both and stops.
-`--device` answers it per command; `SIMFRAME_DEVICE` answers it per shell:
+The CLI is the same thing without an agent in the loop, and the cheapest way
+to check the install:
 
 ```bash
-export SIMFRAME_DEVICE=emulator-5554
-simframe ui               # the emulator, without saying so every time
+npm install -g simframe
+simframe doctor          # every layer, and which engine carries it
+simframe ui              # the current screen as a numbered text map
 ```
 
-The tree is a deliberate omission, not an oversight. Making it fast needs a
-resident instrumentation APK on the device — the shape uiautomator2, Maestro and
-Appium all converged on — and that would be simframe's first runtime artifact
-installed onto your device. The perception ladder was built so a missing tier
-degrades rather than fails, and this is exactly that case: OCR and CV yield
-labels and coordinates on Android today, and a tap by label works without a tree
-at all. `simframe doctor` reports the tier as `optional` with that number, so
-the gap is visible rather than silent, and the criteria for revisiting it are in
-`docs/DEFERRED.md` under **Phase 8b**.
-
-**What the missing tree costs, measured rather than hand-waved.** Screen
-*identity* is weaker on Android than on iOS, and specifically so. Tokens per
-screen, and where they come from:
-
-| Screen | Tokens | Regions | Roles | Chrome labels |
-| --- | --- | --- | --- | --- |
-| launcher | **1** | nav-bar 1 | text 1 | 0 |
-| Settings root | 9 | content 9 | text 9 | 0 |
-| example.com in Chrome | 6 | content 3, nav-bar 3 | text 6 | 0 |
-
-Every token has role `text`, because without a tree nothing infers a button from
-a rectangle reliably enough to say so, and no screen here carries a chrome label
-at all. So on Android a screen is recognised by the geometry of its text, which
-is thinner and noisier than the iOS mix of roles, chrome labels and geometry.
-Flows still work; screen *memory* is doing more guessing, and that is the honest
-cost of the tier being absent.
+If more than one simulator is booted, name the one you mean with
+`--device=<udid>` or `export SIMFRAME_DEVICE=<udid>`; simframe refuses to guess,
+because the command that would act on the wrong one is a tap.
 
-It is also why the obvious fix for the iOS drift — dropping content-region text
-out of identity, which would be a strict improvement there — is not available:
-it would leave Settings' root with zero tokens, and zero tokens is no identity
-at all. See [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
+## What it changes
 
-That last column read **3** before this measurement changed it. Chrome's address
-bar is chrome by every structural test there is, so the screen's identity
-contained `"== example.com"` — a URL, meaning the same browser on a different
-page was a different screen and every learned route through it broke on
-navigation — plus `":"` and `"+"`, which are OCR reading punctuation off icons.
-A chrome label now has to be a name: two letters at minimum, and not an
-address.
-
-The emulator's own gRPC surface was checked for anything tree-shaped and has
-nothing: 43 RPCs for sensors, input, screenshots and VM state, and no notion of
-a view. That question is settled, not open. The same surface is what carries the
-clipboard.
-
-## The tools
+An agent driving a simulator is slow for three reasons, and only the first is
+obvious. Every look is a wait (`simctl io screenshot` costs ~130 ms, paid on
+every glance). Every step is a model round trip. And nothing is remembered, so
+the same screen is re-read and re-reasoned about each time it appears. There
+is a fourth that is pure waste: an image is the most expensive way to ask what
+is on screen, and it still does not say what is tappable.
 
-Read first, act in batches, and look at pixels only when the question is about
-pixels. Every tool description says so, because a tool surface that does not
-steer the model is a tool surface the model uses wrong.
+| | Without simframe | With simframe |
+| --- | --- | --- |
+| Look at the screen | ~130–400 ms screenshot, blocking | **~20 ms**, already captured |
+| "Did anything change?" | a full image | **~2 ms**, text only |
+| Reading a screen | an image, ~1,600 tokens, no tap points | **~330 tokens** of text, with tap points |
+| A 10-step flow | 10 turns, 10 images | **1 turn, 0 images** |
+| Same flow, again | every run is the first | replayed from memory, every step verified, **zero model calls** |
 
-| Tool | What it does |
-| --- | --- |
-| `sim_ui` | **Start here.** The screen as a numbered text map: region, type, label, state, tap point, source. A tenth the cost of a screenshot and strictly more useful. |
-| `sim_do` | **The main tool.** A whole flow in one call — tap, type, scroll, wait, assert — each step settling before the next and verified against what it did last time. |
-| `sim_state` | The cheapest question there is: has anything changed **since your last look**, and which regions moved. |
-| `sim_goto` | Walk to a screen simframe has been to before, planning the route through remembered transitions. |
-| `sim_flow_run` | Replay a saved flow — **zero model calls**, which is the only path to human wall clock. A first traversal saves as *provisional*; one replay in which every step passed confirms it. A run with a contradicted step, a failed step, or one that never reached its last step is refused and says which. |
-| `sim_find` | Resolve an intent to one control, without acting on it. |
-| `sim_tap` · `sim_type_into` · `sim_scroll_to` · `sim_wait_for` · `sim_assert` | Single actions, for when you genuinely only have one step. Each is one `sim_do` step underneath. |
-| `sim_launch` · `sim_open_url` · `sim_permission` | Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert. A launch is **confirmed to have reached the front**, by comparing the pid `simctl` started against the pid the device reports as frontmost — so *"the process started"* is no longer reported as *"the app is on screen"*. |
-| `sim_wait` | Waits for the screen to change *and then* settle. |
-| `sim_look` | **The only tool that returns an image**, capped at 1024 px. For layout, colour, spacing — questions text cannot answer. |
-| `sim_recall` · `sim_strip` | Look backwards: a text timeline of what happened, or recent frames tiled into one image. |
-| `sim_storage` | **What the app believes**, as opposed to what it drew: its `UserDefaults` and, for React Native, its `AsyncStorage`. Reads the data container off disk, so it answers on a device that is **not running**. |
-| `sim_capture` · `sim_devices` | Manage capture loops; list simulators. |
+Measured on an iPhone 17 Pro simulator, iOS 26.5, Apple Silicon, against a real
+production app; the full tables with N, median and p95 are in
+[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
 
-### What the screen looks like as text
+This is what the agent gets back instead of a picture:
 
 ```
 iPhone 17 Pro · 402x874pt · screen a1b2c3d4 "Inbox" (known, 3 known exits)
@@ -437,745 +104,109 @@ content:
 tab-bar:
   #6 text      62,835     Inbox
   #7 text      201,835    Settings
+next: settled; nothing ambiguous — chain the next steps in one sim_do without looking again.
 ```
 
-Region first, because "Inbox" the title and "Inbox" the tab differ only by where
-they are. A tap point, because that is what an action needs. And a number, which
-is a selector: whatever this calls `#3`, the next call can tap as `#3` without
-describing it. A ref is valid only while that screen is showing — used on a
-different screen it refuses rather than tapping whatever now sits there.
-
-`= something` is what the control *contains*, from the accessibility tree, and
-`~ something` is what OCR read off the pixels. Both are printed, and where they
-disagree that is the point: one is authoritative and the other is what is
-actually on screen, and a field mid-edit can legitimately differ. A row with no
-`=` is a control that reports no value, not an empty one.
-
-When the elements were recalled from screen memory rather than looked at just
-now, the header says so and how long ago — `elements recalled from 41s ago —
-pass refresh for what is there now`. Identity is cached on purpose, because a
-list with new rows is the same screen; contents are exactly what changes without
-the screen changing, so the age is worth seeing.
-
-Three ways to name a control, anywhere one is named:
-
-| | |
-| --- | --- |
-| `"Save"` · `the Assets tab` · `back` | **start here** — resolved by intent: verbs, typos, synonyms, icon-only controls by their common name |
-| `#3` | the number the map gave it. Cheap and exact, but only within the round trip that numbered it |
-| `@120,400` | raw point coordinates. Last resort: it cannot tell you it missed. |
-
-The order is deliberate and it used to be the other way round. Four peer rounds
-reported that intent resolution worked every time while refs renumbered
-underneath them, so a table that led with `#3` and called it "unambiguous" was
-recommending the more brittle of the two.
-
-## Baselines: the thing to understand
-
-Every change question is really "changed **since when**?" — and the answer is
-almost never "since the previous frame". A UI transition is over in about 700 ms,
-so comparing consecutive frames tells a caller that polls every few seconds
-"nothing changed", even though the screen is completely different from when it
-last looked.
-
-So simframe compares against **the last frame you observed**. Over MCP that is
-automatic. From the CLI, capture a baseline before you act:
-
-```bash
-H=$(simframe mark)
-# ...tap, launch, navigate...
-simframe wait --since=$H     # change, then settle
-simframe state --since=$H    # what moved, as text
-```
-
-The same applies to waiting. `--mode=settle` (the default) waits for a change and
-*then* for stillness, because a bare "wait until stable" called in the moment
-before an animation starts will correctly, and uselessly, return immediately.
-
-## Screen memory
-
-An accessibility tree is a promise apps do not always keep. In testing against a
-real production app, its custom tab bar published **no children at all**, its
-icon buttons carried unreadable private-use glyphs, and its React Native text
-inputs were **absent from the tree entirely** — the controls used most were
-exactly the ones that could not be tapped by name.
-
-So simframe reads the screen two ways and remembers the result:
-
-- **Accessibility** gives real hit targets, types and enabled state.
-- **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
-  label a person can actually see, with coordinates.
-- The merge is keyed by a **structural fingerprint**, so the next visit is a
-  file read. What that fingerprint is, and why it is not a pixel hash, is below.
-
-```
-first visit to a screen   ~1000 ms   read tree + OCR, store the map
-every visit after that       ~1 ms   look it up
-```
-
-OCR is also more accurate than measuring by eye. On one tab bar the first tab
-centre sat at x=62, not the x=40 an even five-way split predicts — a silent
-mis-tap on every attempt.
-
-Two details that matter:
-
-- **Containers do not absorb their contents.** A tab bar encloses all five tab
-  labels but is not any of them, so the merge only combines an element with text
-  of comparable size.
-- **Ambiguity is reported, not guessed.** A word that is both a screen title and
-  a tab returns an error listing both with coordinates, because silently tapping
-  the title looks exactly like nothing happening.
+Region, type, label, contents, a tap point, and a number the next call can use
+as a selector. `= value` is what the accessibility tree says the control
+contains; `~ value` is what OCR read off the pixels. Where they disagree, that
+is the point.
 
-### Two hashes, because there are two questions
-
-"Did this move?" and "is this the same screen?" look like one question and are
-not. simframe answers them separately, and getting that wrong was the single
-most expensive mistake in its development.
-
-**Change and settle** are questions about pixels, so a pixel hash answers them.
-The frame hash changes whenever any pixel group changes — a clock digit, one new
-row — which is exactly right for "did anything happen?" and useless as a key for
-"have I been here before?". For change detection there is a layout hash: status
-bar cropped, difference hash over a 12×24 grid.
-
-A mean-threshold hash was tried first and was actively dangerous: low-contrast
-screens collapsed onto identical values, so unrelated screens matched at distance
-0 and taps landed on the wrong control. The difference hash fixed that.
-
-**Identity is not a question about pixels**, and this is the part that took three
-attempts. Content *is* pixels: a list whose rows changed drifts as far as a
-different screen does. Measured on a real app, same-screen revisits reached 62
-bits against a different-screen floor of 74 — overlapping, with no threshold
-available to choose. An earlier calibration had suggested a comfortable margin
-(0–4 against 77–113), but it was measured on screens whose content happened to be
-stable and did not survive contact with a real list.
-
-So identity is **structural**. The fingerprint is built from element roles,
-frames quantised to a 24 px grid, the region each element sits in, and repeated
-siblings bucketed as "one" or "many" rather than counted. Deliberately included:
-the labels of chrome elements only — nav title, tab labels, toolbar buttons —
-because two list screens with identical structure are told apart by their title
-and nothing else. Deliberately excluded: all content text and values, the status
-bar, and the keyboard region when a keyboard is up.
-
-It does not depend on the accessibility tree. Fingerprinting from OCR boxes
-alone, with the tree discarded entirely, still separates screens — different
-screens ceiling 0.35 against the same threshold.
+## The tools
 
-| | Jaccard similarity |
+| Tool | What it does |
 | --- | --- |
-| Same screen, revisited | 0.41–1.00 |
-| **Different screens** | **0.00–0.31** |
-
-The threshold sits in that gap, but the gap is narrower than anyone would want,
-and one screen causes it: a screen whose sections load from different sources has
-more than one genuine settled structure, and two structures of one screen are as
-far apart as two different screens.
-
-No threshold can express that, so a screen may hold **several** accepted
-fingerprints instead. A new one is admitted only when a known edge lands
-somewhere its target does not recognise — the edge is the evidence that it is the
-same place — and only if no other stored screen claims that reading. Identity
-stays exact rather than being loosened, and the count is capped, so a
-non-deterministic action shows up as a node collecting variants rather than as
-screens silently merging.
-
-It earns its place on real apps: an app reconnecting to its bundler put an alert
-over one screen, and that screen gained a variant instead of a duplicate
-appearing.
-
-Failing to recognise a screen you have seen is harmless — it rebuilds the map and
-taps correctly. Matching the *wrong* screen taps the wrong control. The threshold
-is set to err toward the first.
-
-## Navigating by memory
-
-Once simframe knows which screens exist and which action leads from one to the
-next, getting somewhere is a search over known edges rather than a question for a
-model:
+| `sim_ui` | **Start here.** The screen as the numbered text map above. |
+| `sim_do` | **The main tool.** A whole flow in one call — tap, type, scroll, wait, assert — each step settling before the next and verified against what it did last time. Fallback selectors, optional steps, and a bounded `seek` let a batch survive a surprise instead of handing it back. |
+| `sim_state` | Has anything changed since your last look, and which regions moved. |
+| `sim_goto` · `sim_flow_run` | Walk to a screen simframe has been to before; replay a saved flow with **zero model calls**. |
+| `sim_find` · `sim_tap` · `sim_type_into` · `sim_scroll_to` · `sim_wait_for` · `sim_assert` | Single actions, for when you genuinely only have one step. |
+| `sim_launch` · `sim_open_url` · `sim_permission` | Launch (confirmed to have reached the front), open a deep link, grant a privacy permission without tapping the alert. |
+| `sim_look` | **The only tool that returns an image**, capped at 1024 px — for layout, colour and spacing. |
+| `sim_recall` · `sim_strip` · `sim_storage` | What happened in the last minute, as text or as a contact sheet; what the app *saved* (`UserDefaults`, `AsyncStorage`), even on a device that is shut down. |
+| `sim_wait` · `sim_capture` · `sim_devices` | Wait for change then settle; manage capture; list simulators and emulators. |
+
+Three ways to name a control, anywhere one is named: `"Save"` (resolved by
+intent — verbs, typos, synonyms, icon-only controls by their common name),
+`#3` (the number the map gave it, valid for that screen only), or `@120,400`
+(raw points, last resort). A label that matches the destructive vocabulary —
+Delete, Pay, Send, Sign out — is never substituted for or guessed at.
+
+A [Claude Code skill](skills/simframe/SKILL.md) ships in the package and
+teaches the CLI path, which is cheaper still: a few lines of output instead of
+a tool result, and no MCP schema in context until a tool is used.
 
-```bash
-simframe screens              # what this device has learned
-simframe goto invoices        # walk there, verifying every step
-```
-
-And when the screen and the behaviour disagree, the question is usually not
-about the screen at all:
-
-```bash
-simframe storage                          # apps with a data container
-simframe storage com.example.myapp        # what that app saved
-```
-
-`sim_ui` says what is drawn; `sim_storage` says what the app believes. It reads
-the data container straight off the host filesystem, which means it works on a
-device that is **shut down** — `simctl` cannot do this at all, on any of its own
-paths, once a device stops running.
-
-
-Measured on a four-tab tour, `goto` plans and walks three-step routes with every
-step verified and no model call. It fails rather than guesses: an unknown
-destination, a query matching two screens equally, or no path of known edges all
-report themselves instead of tapping hopefully.
-
-Flows work the same way and refuse to save if any step went unverified —
-replaying a recording of something that may not have worked just reproduces the
-doubt.
-
-```bash
-simframe flow save checkout ./checkout.json
-simframe flow run checkout
-```
-
-### What the memory is worth, isolated
-
-Three agent sessions drove the same task family on the same production app. The
-third existed only to hold a variable still: pass 2 beat pass 1 by twenty-one
-model round trips, but it had a warm graph *and* an operator who had already
-driven the app once, so the gap could belong to either.
-
-| pass | operator | graph | round trips |
-| --- | --- | --- | --- |
-| 1 | fresh | cold | **33** |
-| 3 | fresh | **warm** | **24** |
-| 2 | experienced | warm | **12** |
-
-**The graph is worth ~27%** of the round trips. Operator knowledge is worth the
-rest — half of everything left after the graph had taken its share. A second
-reporter, unable to isolate the graph, derived **~23%** from escalation rate per
-step without knowing that number.
-
-One app, one operator per pass, and round trips are not seconds. What it settles
-is the direction: the memory is real, it is the smaller half, and the larger half
-is what an operator learns and no tool here carries between sessions yet. Full
-working in [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) §14.
-
-## Does this work on *your* app?
-
-Nothing in simframe is written for a particular app. What varies between apps is
-how much of the accessibility tree exists, and simframe is built to degrade
-rather than fail:
-
-- **Good tree** → tap by label, batch aggressively, everything just works.
-- **Partial tree** (custom tab bars, icon buttons) → OCR fills the gaps; you tap
-  by the visible text instead.
-- **No tree at all** → OCR alone still yields labels and coordinates.
+## Android
 
-Run `simframe ui` on any screen to see exactly what simframe can see, with each
-target marked `ax` or `ocr`. If something you can read is not listed, that is a
-bug worth reporting.
+The second backend drives an Android emulator with the same commands, the same
+screen map and the same memory. Frames arrive at ~41 ms through the emulator
+console; tap, swipe, text, keys and the clipboard work; `simframe devices`
+lists emulators next to simulators. What Android does not have is an
+accessibility tree, so its screens are read by OCR and CV alone — and
+`simframe doctor` says so rather than pretending. Details, costs and the
+reasoning are in [the guide](docs/GUIDE.md#android).
 
-A control the app **declares but never names** — an icon-only overflow menu, a
-back chevron — is listed with its coordinates and tappable by `#ref` rather than
-by name, and the map counts how many such controls are on the screen. A view the
-app never declared accessible at all is invisible to any accessibility tree, ours
-included, and the same line says so: a count of zero on a screen that has one
-would be the more expensive answer.
+## How it works, briefly
 
-**What a settle can and cannot see.** Stillness is decided from a mean over a
-grid of the frame, and a small animation does not move a mean. Measured on a
-screen built to never settle — a spinner, frames every 77–95 ms — the mean
-difference was **0.00196** against a 0.004 threshold while one cell moved by
-**0.0275**, and `stableForMs` reported **79 seconds** of stillness. So a settle
-can return satisfied while part of the screen is still moving, and it now says
-so: `settled after 63ms (a 13x13 region is still animating)`. A settle that
-times out names where the movement is instead. Refusing to settle on any
-animation would be right for a spinner and wrong for a blinking cursor, so the
-disagreement is reported rather than resolved by a guessed threshold.
+One Swift daemon per device reads the simulator's framebuffer straight off its
+IOSurface when the display reports damage, runs Apple's Vision OCR on the same
+surface, reads the accessibility tree host-side, and sends taps as real
+down/move/up sequences over the HID channel — no PNG, no file, no spawned
+process. Frames land in `~/.simframe/<udid>/`, input goes over a `0600` Unix
+socket, and the MCP server and CLI are thin clients over both.
 
-Two honest caveats. OCR reads **text**, so a purely graphical icon with no label
-is invisible to both paths — the tree still gives you its coordinates, and
-`#ref` still taps it. And the confirm-button vocabulary (`APPLY`, `OK`, `SAVE`,
-`DONE`…) is English; a localised UI needs those words extended.
+Screen memory keys a merged element map by a structural fingerprint, so a
+screen seen before is a file read. A transition graph records which action led
+from which screen to which, so `sim_goto` plans a route and a saved flow
+replays with no model in the loop.
 
-## Measured
+Every layer degrades rather than fails, and never quietly: `simframe doctor`
+reports each one and `--strict` turns any downgrade into a non-zero exit. The
+private frameworks the daemon links are documented, with evidence, in
+[`docs/PRIVATE_API.md`](docs/PRIVATE_API.md).
 
-iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
+## Read on
 
 | | |
 | --- | --- |
-| Frame capture, whole pipeline | **6.6 ms** |
-| Frame grab alone | **0.13 ms** |
-| The `simctl` + `sips` path it replaces | ~210 ms |
-| Warm frame read (`sim_look`) | ~20 ms |
-| State check (`sim_state`) | ~2 ms |
-| Input round trip (`ping`) | **0 ms** |
-| Tap (70 ms hold / 10 ms hold) | 76 ms / 13 ms |
-| Text recognition, in-process | **~174 ms** |
-| Text recognition, via PNG + helper (fallback) | ~555 ms |
-| Accessibility tree read, in-process | **~45 ms** |
-| Accessibility tree read, via idb (fallback) | ~203 ms |
-| Screen map: first visit / remembered | ~305 ms / **~1 ms** |
-| CPU | 1.1 % idle · 3.1 % active |
-| Frame memory | ~60 s of screen, ~2.7 MB |
-
-Reproduce all of it with `npm run bench`, which prints the same table against
-your machine. Full detail, including the measurement traps, is in
-[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
-
-### Wall clock per step — where the time really goes
-
-Every number above is microscopic next to the one that decides how fast this
-feels, and it took two field reports to see it. Per-step wall clock is
-
-```
-(model round trip + simframe work × n) / n        for n steps in one call
-```
-
-**Compare like with like, which this table used not to.** A human tester's
-1.95 s per step is the *whole* loop — look at the screen, decide what to do,
-do it. Any row that does not include a decision is not comparable to it.
-
-| | perceives | decides | acts | per step |
-| --- | --- | --- | --- | --- |
-| **a human tester, measured** | yes | yes | yes | **1.95 s** |
-| **simframe, batch of 2 — the recorded median** | yes | yes | yes | **~11.7 s** |
-| simframe, batch of 4 | yes | yes | yes | ~6.7 s |
-| simframe, one model call per step | yes | yes | yes | ~21.7 s |
-| — simframe's mechanical half alone | yes | **no** | yes | ~1.7 s |
-| — a saved flow replayed | yes | **no** | yes | 1.98 s |
-
-**The only like-for-like comparison is 1.95 s against ~11.7 s: about six times
-a human**, and about eleven times when a failure forces one model call per step.
-
-The two indented rows are the ones this README used to lead with, and both are
-category errors when set against 1.95 s. The 1.7 s is simframe with the thinking
-taken out — the thinking is the model round trip, which is most of the clock.
-And a replay *decides nothing*: it is a recording being played back, so its fair
-counterpart is a human repeating a flow they have memorised, who would be well
-under 1.95 s. Replay against a human working something out for the first time is
-a rehearsal measured against a first attempt.
-
-What the two rows do establish, and it is the finding that reordered this
-project: **there is nothing left to win inside the engine.** 1.7 s is small
-beside a 20 s round trip, so making perception or input faster buys single-digit
-percentages. The only variable that matters is `n` — how many steps one decision
-covers. Every improvement here has come from raising it, not from faster code.
-
-### One more term: the launch is a fixed cost
-
-Measured on the benchmark suite with **no model in the loop at all**:
-
-| route | steps | agent | per step | human | per step |
-| --- | --- | --- | --- | --- | --- |
-| contacts-kate-bell | 2 | 12419 ms | **6.2 s** | 4300 ms | 2.15 s |
-| settings-larger-text | 4 | 15281 ms | 3.8 s | 7799 ms | 1.95 s |
-
-2.9× a human on the short route with nothing thinking, which does not fit
-`1.7s × n`. A cold app launch is a **one-off cost amortised over the route**:
-
-```
-per step = (model round trip + launch cost + ~1.7s × n) / n
-```
-
-That reconciles a 7-step replay at 1.82 s/step with a 2-step route at 6.2 s/step
-— one launch spread over 7 steps or over 2. **The `~1.7 s` figure is warm taps
-inside a batch**, and short routes are materially worse than the table above
-implies on its own.
-
-`step_ratio` was **1** throughout: when a run completed it took exactly the
-minimum number of steps.
-A field report put the split at **34% simframe, 60% agent round trips** over
-462 s of wall clock — the tester's *"30+ seconds between each step"* was
-accurate and was not simframe. So there is nothing left to win inside the
-engine, and the only variable is `n`: `simframe hpi` reports `steps_per_call`
-for exactly that reason.
-
-Which is why **every hard-fail that drops a caller back to single-stepping is a
-latency bug**. Between two field reports on the same flow, one run took 33 tool
-calls and the next took **16**, for 28 executed steps and no screenshots at
-all — the difference being defects fixed, not code made faster.
-
-## How it works
-
-```
-┌─── simframed — one Swift daemon per simulator ──────────────────────────────┐
-│                                                                             │
-│   display damage callback ──► read IOSurface ──► scale ──► hash             │
-│        the screen tells us          0.13 ms      6.6 ms total               │
-│                                          │                                  │
-│   Vision OCR reads the same surface ─────┤   no PNG, no file, no spawn      │
-│                                          │                                  │
-│   Indigo HID ◄── control socket ◄────────┤   0600, one JSON object per line │
-│   taps, swipes, text                     │                                  │
-│                                          ▼                                  │
-│                        ~/.simframe/<udid>/  frames · state.json · meta.json │
-└─────────────────────────────────────────────────────────────────────────────┘
-                                     │  a rename is atomic
-┌────────────────────────────────────▼────────────────────────────────────────┐
-│   MCP server / CLI: stat + read, or one socket round trip for input.        │
-│   Screen memory: layout hash ──► label → point, built once per screen.      │
-└─────────────────────────────────────────────────────────────────────────────┘
-```
-
-The daemon links CoreSimulator and SimulatorKit, which are private frameworks
-with no documentation and no stability promise. Everything it calls is recorded
-in [`docs/PRIVATE_API.md`](docs/PRIVATE_API.md) with the evidence behind it, so
-an Xcode upgrade that moves something is a bounded fix rather than an
-archaeology project. If a layer breaks, simframe degrades to the layer below
-and `doctor` says which.
-
-Run `simframe start --engine=screenshot` to use the one-frame-at-a-time loop
-instead — it is also the only capture engine on Android, where it reaches the
-emulator console rather than any simulator tool. `--engine=simctl` is still
-accepted as the name that loop used to have.
-
-- **Files are the IPC for reads.** The daemon renames completed frames into
-  place; readers just read them. A rename is atomic, so a reader can never see a
-  half-written frame. Input is the one thing that needs a reply, and it goes
-  over a `0600` Unix socket — the file system is the whole permission model.
-- **Almost no dependencies.** The only runtime npm dependency is the MCP SDK.
-  The daemon is Swift built from source against frameworks already on the
-  machine.
-- **The screen says when it changed.** The capture loop is driven by the
-  display's damage callback rather than a timer, so an idle screen costs
-  nothing and a moving one is picked up at once.
-- **It backs off when nothing happens.** 4 fps while the screen moves, 1.5 fps
-  once still, snapping back instantly on change.
-- **One writer per device.** Ownership lives in `meta.json`; `stop` refuses to
-  kill a loop another client is using unless forced.
-- **A wedged capture loop never looks like a calm screen.** Every answer carries
-  a liveness check, and `wait` fails loudly rather than quietly timing out.
-- **An action with no visible effect is reported, not waited out.** Selecting a
-  radio button moves ~0.1 % of the screen — below the change threshold — which
-  used to burn the full timeout. Now the step returns in ~3 s marked
-  `[no visible change]`, so you know to check rather than wait.
-- **And a gesture aimed at a coordinate says what it landed on.** A tap or swipe
-  that changed nothing prints the element covering its start point, because a
-  gesture goes to whatever is on top there:
-
-  ```
-  swiped 83,141 -> 83,800 [no visible change]
-    [the swipe start point 83,141 is inside "Settings" (nav-bar)]
-  ```
-
-  Reported from the field as fifteen minutes lost to six identical swipes that
-  a support banner was swallowing — a banner that was *in the element list the
-  same call printed*. Nothing said "there is something at y≈753 and you started
-  at y=750".
-- **A control half off the edge is offered, clamped, and says so.** A filter
-  chip with 29 pt of itself visible used to be dropped — the off-screen test
-  reads the *centre* — and OCR's reading of that same sliver, a box labelled
-  `Flc`, was printed in its place. So the map did not merely omit a control; it
-  offered a meaningless name for it at an ordinary-looking coordinate. Now:
-
-  ```
-  #6 button 387,191  Flowering  (partly off-screen — the coordinate is the middle of the visible part)
-  ```
-
-  `scrollTo` is deliberately unchanged: a clipped element is still "not in
-  view", so it still scrolls rather than calling a sliver good enough.
-
-## CLI
-
-The CLI is the low-token path, and it is a first-class one: `--json` is on every
-command, so nothing has to be parsed out of prose.
-
-```bash
-simframe ui                    # the numbered screen map — start here
-simframe ui --json | jq '.elements[] | select(.type=="button") | .label'
-simframe do flow.json          # a whole flow, verified, then the end-state map
-simframe do flow.json --save=checkout   # save it if every step verified
-simframe flow run checkout     # replay it
-simframe tap "#3"              # or "Save", or "@120,400"
-simframe find "the save button"   # resolve an intent without acting
-simframe screens               # screens this device has learned
-simframe goto invoices         # walk to a known screen over known steps
-simframe state --since=$H      # what changed, as text
-simframe mark                  # hash of the current frame, for --since
-simframe wait --since=$H       # change, then settle
-simframe recall                # what happened in the last minute, as text
-simframe recall --ago=15000    # the frame from 15s ago
-simframe frame --out=now.png   # newest frame, native resolution, to a file
-simframe strip --count=6       # contact sheet, for an animation
-simframe doctor --strict       # any degraded layer is a non-zero exit
-simframe escalations           # why simframe still needs a model, by reason
-simframe escalations --session # ...this agent only, not every agent on the device
-simframe supervisions          # local supervisor rulings, and what came of each
-simframe hpi                   # speed and accuracy against a human baseline
-simframe baseline record settings-larger-text --runs=5   # record the human
-simframe input reset           # rebuild the HID session, without restarting anything
-simframe diagnose              # what this device is doing right now, and which failure it is
-simframe revive                # power-cycle a wedged device: stop, shutdown, boot, start, reset input
-simframe start / status / stop [--force] / devices
-simframe ui --device=emulator-5554      # or export SIMFRAME_DEVICE once
-```
-
-`--device` takes `--device=X` and `--device X` alike. It used to take only the
-first: the space form set the flag to `true` and then resolved a device named
-"true", which is a poor answer to a flag `doctor`'s own advice tells you to
-type.
-
-### When the simulator stops rendering
-
-A simulator driven hard for several minutes can stop rendering: capture fails,
-`simctl screenshot` fails too, and the daemon's own recoveries — re-resolving the
-display port, then rebinding the device — do not help. It reports the state
-rather than acting on it, because a capture loop that rebooted the device it was
-watching would be a tool reaching for the mains:
-
-```
-capture is wedged and both recoveries are spent (2 port re-resolves, 2 device
-rebinds, no frame since). This needs the device restarted —
-`simframe revive --device=<udid>`. Backing off until a frame arrives.
-```
-
-**`simframe diagnose` says which failure it is**, which `doctor` cannot: `doctor`
-answers "can this machine capture", and this answers "what is this device doing
-right now". It reads only what discriminates — frame sequence, age and
-stillness, element counts split by sensor, how many of them fuse, and who holds
-the front by pid — and returns one of:
-
-| verdict | what it means |
-| --- | --- |
-| `capture-down` | no frames at all, carrying the daemon's own sentence |
-| `nothing-readable` | frames arriving, neither sensor finds a single element |
-| `stale-frame` | both sensors full, almost nothing fuses — the framebuffer is behind the tree, so **an image from this device is not safe to trust** |
-| `not-presenting` | an app holds the front by pid and the display shows almost nothing |
-| `healthy` | — |
-
-`not-presenting` deliberately does **not** say whether that is a lock screen, a
-dead surface or a crashed system shell. It is not knowable from here, and
-guessing is how a regex ended up standing where a measurement belongs.
-
-The `stale-frame` threshold is measured rather than chosen: element fusion on
-five healthy screens ran 0.667–0.929, so the threshold sits at 0.1 — 6.7× below
-the observed floor rather than inside the metric's own noise. Numbers in
-[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
-
-`simframe revive` is that restart, in the order that matters — stop the daemon,
-shut the device down, boot it and *wait for the boot to finish*, start capture,
-rebuild the HID session — and it ends by checking frames are flowing again
-rather than by reporting that the steps ran. It is a command and not a
-behaviour: the decision stays yours.
-
-### Reading what the local supervisor decided
-
-With `SIMFRAME_SUPERVISOR=apple`, every consultation is written to
-`~/.simframe/<udid>/supervisions.jsonl` with what the executor observed
-afterwards, which is what makes a ruling scoreable rather than merely recorded.
-`simframe supervisions` reads it:
-
-```
-14 supervisor rulings on iPhone 17 Pro
-
-  stop -> stopped                    7
-  wait -> recovered                  5
-  wait -> still_failed               2
-
-sourced: model 14
-median latency: 1440ms
-edges the graph had timed: 0/14 — 14 ruling(s) are on edges with no p95
-```
-
-That last line is the honest one: a step that failed is usually a step that has
-never succeeded on that edge, so the graph has no timing to compare against.
-
-### Keeping a session cheap
-
-The expensive part of driving a simulator with an agent is not the tapping, it
-is the thinking between taps — observe, think, tap, observe, think. Measured
-over one real session against a third-party app: **62 tool calls for 179 steps**,
-and 48 of those calls were three steps or fewer. A twelve-step flow arrived as
-five calls, and every boundary between them was a think.
-
-Three things move that number, and simframe does the first two for you:
-
-- **Batch.** `sim_do` runs a whole flow in one call, with an assert after each
-  step that matters. The asserts are what make it safe not to look in between:
-  a step that lands somewhere unplanned halts the flow instead of letting the
-  next four run against the wrong screen.
-- **A `next:` line on every action result** — from the CLI and the MCP server
-  alike — computed locally from what the daemon already knows — whether the screen settled, whether the graph
-  recognises it, how many elements it has, whether any labels repeat. When it
-  says *nothing ambiguous — chain the next steps in one sim_do without looking
-  again*, that is the tool telling the agent it does not need to think.
-- **A trailing map that was re-read, not recalled.** An action pays one
-  perception pass — a few hundred milliseconds, locally — so the map it returns
-  is the screen as it is now. The alternative was an agent spending a whole turn
-  on `ui --refresh` because it could not trust the one it was given.
-- **One goal per session.** Sessions get slower with every turn. A flow that
-  runs as one call adds one exchange to the context instead of twelve.
-
-And one thing to know about waiting: `settle` asks whether the screen stopped
-moving, and a screen waiting on a network call has stopped moving. For anything
-that arrives over the network, assert on the content you expect —
-`{"waitFor": {"value": "Kate Bell"}}` — rather than on stillness. The map says
-`STILL LOADING` when the classifier can see a load in flight, but only you know
-what "arrived" means.
-
-And the expensive habit worth naming: in that session, **28 of 62 calls returned
-a screenshot** — about a third of its entire token cost — because the text map
-could not report what a text field contained. It can now, so check the map
-before reaching for pixels: a row carries the element's contents (`= Fryer 3`)
-and its state (`disabled`), and the flow's own verdict already said whether the
-action worked.
-
-### The Claude Code skill
-
-[`skills/simframe/SKILL.md`](skills/simframe/SKILL.md) teaches the CLI path
-directly: the cheap-to-expensive order, the selector grammar, what each verdict
-means and what to do about it. It ships with the package, so an installed copy
-has it.
-
-```bash
-mkdir -p ~/.claude/skills
-ln -s "$(npm root -g)/simframe/skills/simframe" ~/.claude/skills/simframe
-```
-
-A skill and an MCP server are not redundant. The MCP server is discoverable —
-it appears in the tool list without anybody setting it up. The skill is cheaper:
-Claude reads 3–5 lines of CLI output instead of a tool result, and none of the
-MCP schema is in context until a tool is actually used. Ship both, use whichever
-the client makes easy.
-
-## Degrading is allowed. Degrading quietly is not
-
-simframe is built to degrade rather than fail: no Swift toolchain still gives
-you frames through `simctl`, no accessibility tree still gives you OCR. That
-policy is right, and it nearly sank the tool twice — because a downgrade looked
-exactly like everything working.
-
-Once, one file was missing from the published package, so `Package.swift`
-declared a test target with no directory, SwiftPM reported overlapping sources,
-and **every install silently fell back to the slow engine**. Another time OCR
-shipped disabled the same way. Both passed the tests. Both printed nothing. The
-bug was never the missing file; it was the silence.
-
-So every downgrade now announces itself:
-
-- `simframe start` prints the engine it chose, and if it is the slow one, why —
-  build error, missing sources, or "reason unrecorded" if the daemon was started
-  by an earlier process.
-- `simframe doctor` marks a degraded layer `WARN`, not `ok`, and summarises what
-  is degraded and what that costs.
-- A dependency that is simply not installed is `--`, not `WARN`. The distinction
-  is deliberate: `WARN` means this machine could be doing better and silently is
-  not, which is the failure worth shouting about. An optional fallback missing on
-  a fresh machine has not degraded from anything, and `--strict` ignores it.
-- A device whose capture has **wedged** says so: `capture: stalled — the display
-  surface has been unreadable for 62s; 3 re-attaches did not help; only
-  restarting the device is known to cure it`. This is a different thing from a
-  still screen, and it used to look identical, because a damage-driven engine
-  produces no frames for either. An agent told "nothing changed" keeps tapping;
-  one told the simulator is wedged stops. simframe reports it and does not
-  restart your device.
-- `--strict`, or `SIMFRAME_STRICT=1`, turns any downgrade into a non-zero exit.
-  CI runs strict, so a release cannot ship in the state that shipped twice.
-
-```
-$ simframe doctor
-ok   capture engine        simframed
-WARN input driver          idb — the daemon's control socket is not up
-WARN accessibility tree    idb — the host-side translator did not load
-```
-
-Two checks enforce it. A packaging check derives the required file list from the
-build's own inputs — a hand-written list is what rotted last time — and runs in
-seconds without a simulator. An integration job installs the packed tarball on a
-real simulator and asserts `capture.engine`, `input.driver` and `ocr.available`
-are all the good values, under `--strict`.
+| [`docs/GUIDE.md`](docs/GUIDE.md) | the full guide: recovering without a round trip, screen memory and the two hashes, navigating by memory, the CLI, diagnosing a wedged simulator, keeping a session cheap, the roadmap |
+| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | every number, with N, median, p95, machine and the measurement traps |
+| [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) | what we believed *before* measuring, and the fourteen times it was wrong |
+| [`docs/DECISIONS.md`](docs/DECISIONS.md) | the judgements that changed the plan, including a phase cancelled by its own measurement |
+| [`docs/DEFERRED.md`](docs/DEFERRED.md) | every known defect, open or closed, with the evidence |
+| [`docs/ARTICLE.md`](docs/ARTICLE.md) | *Agents shouldn't blink* — the argument the whole thing adds up to |
 
-That last check found a real bug the day it was written: a daemon shutting down
-unlinked the control socket unconditionally, so restarting deleted the *new*
-daemon's socket. Capture kept working, input quietly dropped to idb, and nothing
-said a word — the exact failure shape, found by the thing built to catch it.
+Almost everything here was decided by a measurement rather than an argument,
+and several measurements reversed a decision that had already shipped. Two
+headline results: a warm transition graph removes about 27% of an agent's
+model round trips, and a one-line threshold on a number the daemon already
+computes beat every local model tried as a step supervisor — 95% against
+77–91%, at zero latency.
 
 ## Limitations
 
-- Simulators and Android emulators only. Neither the framebuffer nor `simctl`
-  nor the emulator console can reach a physical device.
-- Android has no accessibility tree, so its screen identity rests on the
-  geometry of OCR'd text: thinner and noisier than iOS's. See
-  [Android](#android) above.
-- The daemon depends on private frameworks. They are stable enough to build on —
-  capture and accessibility survived the iOS 26 transition — but an Xcode
-  upgrade can move a symbol. `doctor` reports each layer separately so a break
-  is visible rather than mysterious, and `--engine=screenshot` still works.
-- Hardware buttons: only `home` is implemented. The other Indigo codes are
-  unverified, and a wrong one can crash `backboardd` or lock the device, so they
-  return an error rather than a guess.
-- Typing sends key positions, which iOS maps through the device's active
-  keyboard layout. Text that must be exact goes through the pasteboard, which
-  `sim_do` does by default.
-- Region maps need a baseline inside the ~90 s history window. Older baselines
-  still get a reliable changed / did-not-change, without a map of what moved.
-- Screen memory assumes a screen's layout is stable. A screen that reflows
-  dramatically between visits will simply be rebuilt.
-- It speeds up *confirming* a fix, not *locating* one. A bug living in a memo
-  comparator or a stale closure is not visible in any frame.
-- A switch is tapped at its **activation point** when the app publishes one —
-  UIKit's `accessibilityActivationPoint`, which is what a switch answers with,
-  and tapping it flipped a real switch 3 of 3 times where the frame centre
-  managed 0 of 3. Only **4 of 75** elements on a measured screen publish one,
-  though, so where the app says nothing the tap still goes to the centre of the
-  frame — and a switch's frame is the whole row, so it lands on the label and
-  the control at the trailing end does not move. `@x,y` remains the escape
-  hatch there. Never a guessed offset: nil means the app did not answer.
-- A launch is confirmed to have fronted **on iOS only**. It compares the pid
-  `simctl launch` printed against the pid the device reports as frontmost, in
-  2–5 ms. Android reports neither, so a launch there is not checked — `am start`
-  fronts synchronously, which is why it has not bitten, but that is not a check
-  and `doctor` says so. Note that a launch which starts a process without
-  bringing it forward now **fails** rather than returning success: a flow that
-  used to pass through such a launch and then assert on the previous app's
-  screen will start failing, correctly.
-- The simulator's display pipeline stops rendering under rapid app relaunch —
-  about six cycles, reproducibly — and every frame comes back black while
-  `simctl` itself reports success. simframe now says so instead of reading a
-  black screen as a calm one, but it cannot fix it: restarting the device is
-  the cure that always works, and it usually recovers on its own.
-
-## What was decided, and what was not built
-
-[`docs/DECISIONS.md`](docs/DECISIONS.md) is the register of judgements that
-changed the plan: a phase cancelled by its own measurement, two features built
-and reverted for cause, and the premises that turned out to be false. It is
-short on purpose — the numbers live in
-[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) and the open work in
-[`docs/DEFERRED.md`](docs/DEFERRED.md).
-
-The most useful entry is a **no-go**: Phase 17 proposed a small on-device model
-to choose the next element, and measuring the prize before the model showed the
-matcher already resolves 37 of 40 real decisions. By the time a step reaches
-simframe the decision has already been made — the goal names the option,
-because the agent chose it and then asked for it by name.
-
-## Roadmap
-
-- **Extend the confirm vocabulary beyond English.**
-- **Region bands from clustering**, replacing the positional bands. They have
-  produced three bugs in three phases, and on Android they put a URL bar in the
-  nav bar and its URL into the screen's identity.
-- **Phase 8b, conditionally:** an instrumentation APK for the Android
-  accessibility tree, with the criteria for doing it stated in
-  `docs/DEFERRED.md` rather than left to enthusiasm.
+- Simulators and emulators only; nothing here can reach a physical device.
+- The daemon links private frameworks. They survived the iOS 26 transition,
+  but an Xcode upgrade can move a symbol; `doctor` names the broken layer and
+  `--engine=screenshot` still works.
+- Android has no accessibility tree, so screen identity there rests on OCR
+  geometry alone: thinner and noisier than iOS.
+- Hardware buttons: only `home`. The other codes are unverified and a wrong
+  one can crash the device, so they return an error rather than a guess.
+- The confirm vocabulary (`OK`, `Save`, `Done`…) is English.
+- It speeds up *confirming* a fix, not *locating* one. A bug in a stale
+  closure is not visible in any frame.
+
+The full list, with the measurements behind each item, is in
+[the guide](docs/GUIDE.md#limitations).
 
 ## Releasing
 
-`npm version` runs a `version` hook that rewrites `server.json` to match and
-stages it, so one command covers both files:
-
-```bash
-npm version minor          # bumps package.json + server.json, commits, tags
-git push --follow-tags
-```
-
-Before that hook existed, `server.json` had to be hand-edited between two
-commands, and the release that forgot failed at the workflow's own agreement
-check — which is the one thing that check is for.
-
-The `release` workflow verifies tag/`package.json`/`server.json` agree, validates
-`server.json` against the live registry, and publishes to npm and the MCP
-Registry. It holds **no secrets** — both halves authenticate with the workflow's
-GitHub OIDC identity.
-
-That needs one setup step on npmjs.com, not in this repo: the package must have a
-Trusted Publisher pointing at this repository and `release.yml` (Package →
-Settings → Trusted Publisher → GitHub Actions). Without it npm has nothing to
-trust and fails with `ENEEDAUTH`. npm is ending token publishing in January 2027,
-and the tokens that work in CI need 2FA bypass, which npm's own UI warns against —
-so OIDC is the durable path, not merely the tidier one.
+`npm version` is the only way to bump a version — its hook keeps `server.json`
+in step — and `git push --follow-tags` runs a workflow that publishes to npm
+and the official MCP Registry with no secrets. Why it is built that way, and
+the release that once vanished for a week, is in
+[the guide](docs/GUIDE.md#releasing).
 
 ## License