simframe 0.1.0 → 0.4.1

package/README.md

@@ -4,59 +4,41 @@
 [![npm](https://img.shields.io/npm/v/simframe.svg)](https://www.npmjs.com/package/simframe)
 [![license](https://img.shields.io/npm/l/simframe.svg)](./LICENSE)
 
-**Always-warm iOS Simulator frames for coding agents.**
+**Eyes, hands and memory for an agent driving the iOS Simulator.**
 
 [Website](https://lvlrsajjad.github.io/simframe/) · [npm](https://www.npmjs.com/package/simframe)
 
-An agent that drives the iOS Simulator spends most of its time waiting on
-screenshots. Every "let me check the screen" is a fresh `simctl io screenshot`:
-a process spawn, a framebuffer grab, a file write, an image encode. On this
-machine that is ~130 ms of pure blocking latency, paid again on every look — and
-the agent pays it *twice* whenever it screenshots too early, sees a
-mid-animation frame, and has to look again.
+An agent driving the iOS Simulator is slow for three reasons, and only the first
+one is obvious:
 
-simframe removes the wait from the request path. A tiny background loop keeps
-the newest frame of your simulator permanently warm on disk, so when the agent
-asks what's on screen it gets an answer in **~20 ms** instead of ~130 ms — and
-can ask "did anything change?" for **~2 ms and no image at all**.
+1. **Every look is a wait.** `simctl io screenshot` costs ~130 ms of blocking
+   latency, paid again on every glance — and paid twice whenever the agent
+   captures mid-animation and has to look again.
+2. **Every step is a round trip.** Tap, screenshot, reason, tap, screenshot. A
+   twelve-step flow costs twelve model turns, and the model turns cost far more
+   than the milliseconds.
+3. **Nothing is remembered.** The same screen gets re-read and re-reasoned about
+   every single time it appears.
 
-```
-                    without simframe                  with simframe
-agent asks    ──►  spawn simctl ──► grab ──► encode ──► image      ~130-400 ms
-agent asks    ──►  read the frame that is already there ──► image   ~20 ms
-agent polls   ──►  read 300 bytes of JSON ──► text                  ~2 ms
-```
+simframe attacks all three: a background loop keeps the newest frame warm, whole
+flows run in one call, and screens the agent has seen before are answered from
+memory.
 
-## Why this makes an agent faster
+## What changed, measured
 
-Speed is not only latency. It is also *how many* round trips a question takes
-and how many tokens each one costs.
+Same four-tab navigation flow, on a real production app:
 
-| Question the agent has | Before | With simframe |
+| | Before | With simframe |
 | --- | --- | --- |
-| "What's on screen?" | screenshot, ~130-400 ms, full image every time | `sim_look`, ~20 ms, warm frame |
-| "Has it finished loading yet?" | screenshot in a loop, an image per attempt | `sim_state`, ~2 ms, **text only** |
-| "Did my tap do anything?" | screenshot, compare by eye | `sim_state` — a stable screen hash plus an ASCII map of which regions moved |
-| "Wait for the animation to end" | sleep, screenshot, hope, repeat | `sim_wait` — blocks until the screen actually settles, then returns the frame |
-| "What did that transition look like?" | 5 screenshots, 5 round trips, 5 images | `sim_strip` — the last N buffered frames tiled into **one** image, looking backwards in time |
-
-The change map is the part that pays for itself. A screen hash and a
-4×8 movement grid cost a couple of hundred bytes, so an agent can poll freely
-and only spend image tokens when there is genuinely something new to look at:
+| Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
+| "Did anything change?" | a full image | **~2 ms**, text only |
+| A 5-step flow | 5+ model round trips | **1 call**, ~7 s |
+| Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
+| Same flow, 3rd run | no improvement — every run is the first | **3304 ms, 4/4 from memory** |
 
-```
-$ simframe state
-iPhone 17 Pro  frame #3  age 124ms  322x700
-hash 007cfefefefefefefefefefefefefe00  diff 0.15453  stable 0ms
-@@@@
-###*
-+*#*
-:.#*
-..#*
-..#*
-..#*
-@@@@      ← a screen sliding in from the right, mid-transition
-```
+The same four-tab tour, run three times back to back: **7370 ms → 5160 ms →
+3304 ms**, with 1, then 3, then 4 of the four controls resolved from memory and
+no mis-taps. What is left is mostly the app's own animation and data load.
 
 ## Install
 
@@ -65,171 +47,271 @@ npm install -g simframe
 simframe doctor
 ```
 
-`doctor` verifies Xcode's command line tools, `sips`, a booted simulator, and
-an actual round-trip capture.
+`doctor` checks each capability separately and tells you what you have:
+
+```
+ok   xcrun              xcrun version 72.
+ok   sips               available
+ok   input driver (idb) companion built Sep 1 2026
+ok   on-device OCR      available
+ok   booted simulator   iPhone 17 Pro (iOS 26.5)
+ok   capture            frame #888 322x700 in 2ms (age 538ms)
+```
 
 ### Claude Code
 
 ```bash
-claude mcp add simframe -- npx -y simframe mcp
+claude mcp add --scope user simframe -- npx -y simframe mcp
 ```
 
+`--scope user` makes it available in every session; without it the server is
+registered only for the directory you ran the command in.
+
 ### Any other MCP client
 
 ```json
 {
   "mcpServers": {
-    "simframe": {
-      "command": "npx",
-      "args": ["-y", "simframe", "mcp"]
-    }
+    "simframe": { "command": "npx", "args": ["-y", "simframe", "mcp"] }
   }
 }
 ```
 
-Nothing else to set up. Capture starts on the first tool call, targets the
-booted simulator, and stops itself 15 minutes after the last request.
+## Capabilities are independent
 
-## MCP tools
+Each layer works without the ones above it, and `doctor` tells you which you
+have. **Observation needs nothing but Xcode.**
 
-| Tool | What it does |
-| --- | --- |
-| `sim_look` | The newest buffered frame as an image, no capture wait. `detail`: `low` (~420 px) / `normal` (~700 px, default) / `high` (~1100 px) / `full` (native). |
-| `sim_state` | Text only: screen hash, how long the screen has been still, change since the previous frame, and the region movement map. |
-| `sim_wait` | Blocks until the screen settles (`mode: "stable"`) or moves away from what it shows now (`mode: "change"`), then returns the frame. |
-| `sim_strip` | The last N buffered frames tiled into one image, oldest first, with millisecond offsets. |
-| `sim_capture` | `status` / `start` / `stop` for the background loops. Rarely needed. |
-| `sim_devices` | Booted simulators simframe can capture. |
+| Capability | Needs | Without it |
+| --- | --- | --- |
+| Watch the screen, wait, recall | nothing extra | — |
+| Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
+| Tap, type, swipe | [`idb`](https://fbidb.io) | simframe observes but cannot touch |
 
-Every tool takes an optional `device` (UDID or a substring of the name) and
-defaults to the booted simulator.
+```bash
+# input, optional
+brew tap facebook/fb && brew install idb-companion && pipx install fb-idb
+```
 
-## CLI
+Homebrew may ask you to trust the tap first; that is a deliberate prompt for a
+human, and the narrow form is `brew trust --formula facebook/fb/idb-companion`.
 
-The same capabilities without an agent, useful for debugging and scripts:
+## The tools
+
+| Tool | What it does |
+| --- | --- |
+| `sim_look` | Newest frame as an image, no capture wait. |
+| `sim_state` | Text only: screen hash, what changed **since your last look**, region movement map. |
+| `sim_wait` | Waits for the screen to change *and then* settle. |
+| `sim_do` | A whole flow in one call — tap, type, scroll, assert — each step settling before the next. |
+| `sim_ui` | The screen as labels + tap coordinates, from accessibility **and** OCR. |
+| `sim_recall` | Look backwards: a timeline of what happened, or the frame from N seconds ago. |
+| `sim_strip` | Recent frames tiled into one image. |
+| `sim_capture` / `sim_devices` | Manage capture loops; list simulators. |
+
+## Baselines: the thing to understand
+
+Every change question is really "changed **since when**?" — and the answer is
+almost never "since the previous frame". A UI transition is over in about 700 ms,
+so comparing consecutive frames tells a caller that polls every few seconds
+"nothing changed", even though the screen is completely different from when it
+last looked.
+
+So simframe compares against **the last frame you observed**. Over MCP that is
+automatic. From the CLI, capture a baseline before you act:
 
 ```bash
-simframe start                 # start the capture loop
-simframe state                 # metadata + change map
-simframe frame --out=now.png   # newest frame, --detail=low|normal|high|full
-simframe wait --stable-ms=700  # block until the screen settles
-simframe wait --change         # block until the screen changes
-simframe strip --count=6       # contact sheet of recent frames
-simframe status                # what is running, and how fresh
-simframe stop --all
-simframe devices --all
-simframe doctor
+H=$(simframe mark)
+# ...tap, launch, navigate...
+simframe wait --since=$H     # change, then settle
+simframe state --since=$H    # what moved, as text
 ```
 
-## How it works
+The same applies to waiting. `--mode=settle` (the default) waits for a change and
+*then* for stillness, because a bare "wait until stable" called in the moment
+before an animation starts will correctly, and uselessly, return immediately.
+
+## Screen memory
+
+An accessibility tree is a promise apps do not always keep. In testing against a
+real production app, its custom tab bar published **no children at all**, its
+icon buttons carried unreadable private-use glyphs, and its React Native text
+inputs were **absent from the tree entirely** — the controls used most were
+exactly the ones that could not be tapped by name.
+
+So simframe reads the screen two ways and remembers the result:
+
+- **Accessibility** gives real hit targets, types and enabled state.
+- **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
+  label a person can actually see, with coordinates.
+- The merge is keyed by a **layout hash**, so the next visit is a file read.
 
 ```
-┌──────────────────────────── background, one per simulator ───┐
-│  xcrun simctl io screenshot  ──►  sips -Z  ──►  decode PNG   │
-│         ~130 ms                    ~30 ms        ~6 ms       │
-│                          │                                   │
-│              rename into ~/.simframe/<udid>/                  │
-│              latest.png · ring/<seq>.png · state.json         │
-└───────────────────────────────────────────────────────────────┘
-                           │  a rename is atomic
-┌──────────────────────────▼────────────────────────────────────┐
-│  MCP server / CLI: stat + read. No simctl in the request path.│
-└───────────────────────────────────────────────────────────────┘
+first visit to a screen   ~1000 ms   read tree + OCR, store the map
+every visit after that       ~1 ms   look it up
 ```
 
-A few decisions worth knowing about:
-
-- **Files are the IPC.** The loop renames completed frames into place and
-  readers just read them. A rename is atomic, so a reader can never see a
-  half-written frame, and there is no socket, port or protocol to get wrong.
-- **Zero image dependencies.** Resizing uses `sips`, which ships with macOS.
-  PNG encode/decode and all frame comparison are a few hundred lines of plain
-  JavaScript over `node:zlib`. The only runtime dependency is the MCP SDK.
-- **It backs off when nothing is happening.** 4 fps while the screen is moving,
-  1.5 fps once it has been still for 2.5 s, snapping back instantly on change.
-  Measured on an M-series Mac: **1.1 % CPU idle, 3.1 % active.**
-- **Comparison is done on a small grayscale grid,** which is why "did anything
-  change?" costs microseconds. The screen hash is a 128-bit mean-threshold
-  hash: the same screen always produces the same hash, even though the JPEG and
-  PNG bytes coming out of `simctl` are not stable frame to frame.
-- **One writer per device.** Ownership is recorded in `meta.json`; a second loop
-  refuses to start, and a loop that has been superseded retires itself. Two
-  loops would otherwise overwrite and prune each other's frames.
-- **Capture is independent of the Simulator window.** `simctl` reads the
-  framebuffer, so frames keep flowing while the window is hidden, behind other
-  windows, or on another Space.
+OCR is also more accurate than measuring by eye. On one tab bar the first tab
+centre sat at x=62, not the x=40 an even five-way split predicts — a silent
+mis-tap on every attempt.
+
+Two details that matter:
+
+- **Containers do not absorb their contents.** A tab bar encloses all five tab
+  labels but is not any of them, so the merge only combines an element with text
+  of comparable size.
+- **Ambiguity is reported, not guessed.** A word that is both a screen title and
+  a tab returns an error listing both with coordinates, because silently tapping
+  the title looks exactly like nothing happening.
+
+### Why a layout hash, not a frame hash
+
+The frame hash changes whenever any pixel group changes — a clock digit, one new
+row of data — which makes it useless as a key for "have I seen this screen
+before?". The layout hash crops the status bar and takes a difference hash over a
+12×24 grid.
+
+A mean-threshold hash was tried first and was actively dangerous: low-contrast
+app screens collapsed onto identical values, so unrelated screens matched at
+distance 0 and taps landed on the wrong control. Measured on a real app:
+
+| | Hamming distance |
+| --- | --- |
+| Same screen, revisited while settled | **0–4** |
+| Same screen, but loading vs loaded | 48–98 |
+| **Different screens** | **77–113** |
+
+Only one of those errors is dangerous. Matching the *wrong* screen would tap the
+wrong control, and that needs two different screens to land within 12 bits of
+each other — the closest pair ever measured was 77. Failing to recognise a screen
+you have seen is harmless: it rebuilds the map, costs ~600 ms, and taps correctly.
+So the tolerance is deliberately far below the collision floor rather than tuned
+to maximise hits.
+
+That middle row is worth knowing about: a screen mid-load genuinely does not look
+like the same screen loaded, so the first visit after a cold launch usually
+rebuilds. Hit rates climb as an app warms up, which is exactly what the three-pass
+numbers above show.
+
+## Does this work on *your* app?
+
+Nothing in simframe is written for a particular app. What varies between apps is
+how much of the accessibility tree exists, and simframe is built to degrade
+rather than fail:
+
+- **Good tree** → tap by label, batch aggressively, everything just works.
+- **Partial tree** (custom tab bars, icon buttons) → OCR fills the gaps; you tap
+  by the visible text instead.
+- **No tree at all** → OCR alone still yields labels and coordinates.
+
+Run `simframe ui` on any screen to see exactly what simframe can see, with each
+target marked `ax` or `ocr`. If something you can read is not listed, that is a
+bug worth reporting.
+
+Two honest caveats. OCR reads **text**, so a purely graphical icon with no label
+is invisible to both paths — use `sim_ui` to get its coordinates from the tree,
+or tap by position. And the confirm-button vocabulary (`APPLY`, `OK`, `SAVE`,
+`DONE`…) is English; a localised UI needs those words extended.
 
 ## Measured
 
-iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings:
+iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
 
 | | |
 | --- | --- |
 | Warm frame read (`sim_look`) | ~20 ms |
 | State check (`sim_state`) | ~2 ms |
 | Contact sheet (`sim_strip`, 5 frames) | ~30 ms |
-| Cold start (first frame after boot) | ~400 ms, once |
-| Frame age when read | ≤ ~250 ms active, ≤ ~670 ms idle |
-| Raw `simctl io screenshot` for comparison | ~130 ms, on every single look |
-| CPU | 1.1 % idle, 3.1 % active |
-| Disk | ~2 MB per device (24-frame ring) |
+| Accessibility tree read | ~570 ms |
+| On-device OCR of a full frame | ~290 ms |
+| Screen map: first visit / remembered | ~600 ms / **~1 ms** |
+| Raw `simctl io screenshot`, for comparison | ~130 ms, every look |
+| Cold start, first frame | ~400 ms, once |
+| CPU | 1.1 % idle · 3.1 % active |
+| Frame memory | ~60 s of screen, ~2.7 MB |
 
-Image sizes are chosen for token cost as much as legibility: at `detail: normal`
-a frame is ~322×700, roughly a third of the pixels — and so roughly a third of
-the image tokens — of a native-resolution screenshot, while the status bar stays
-readable. `sim_state` sends no image at all.
+## How it works
 
-## Requirements
+```
+┌──────────────────── background, one loop per simulator ─────────────────────┐
+│   simctl screenshot ──► sips ──► decode ──► hash + diff ──► rename into      │
+│      ~130 ms            ~30 ms    ~6 ms                  ~/.simframe/<udid>/ │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                     │  a rename is atomic
+┌────────────────────────────────────▼────────────────────────────────────────┐
+│   MCP server / CLI: stat + read. No simctl anywhere in the request path.     │
+│   Screen memory: layout hash ──► label → point, built once per screen.       │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
 
-- macOS with Xcode command line tools (`xcrun simctl`)
-- Node.js ≥ 18.17
-- A booted iOS Simulator
+- **Files are the IPC.** The loop renames completed frames into place; readers
+  just read them. A rename is atomic, so a reader can never see a half-written
+  frame, and there is no socket or protocol to get wrong.
+- **Almost no dependencies.** Resizing uses `sips`; PNG codec, hashing and frame
+  comparison are plain JavaScript over `node:zlib`. The only runtime dependency
+  is the MCP SDK. OCR is a ~60-line Swift file compiled on first use.
+- **It backs off when nothing happens.** 4 fps while the screen moves, 1.5 fps
+  once still, snapping back instantly on change.
+- **One writer per device.** Ownership lives in `meta.json`; `stop` refuses to
+  kill a loop another client is using unless forced.
+- **A wedged capture loop never looks like a calm screen.** Every answer carries
+  a liveness check, and `wait` fails loudly rather than quietly timing out.
+- **An action with no visible effect is reported, not waited out.** Selecting a
+  radio button moves ~0.1 % of the screen — below the change threshold — which
+  used to burn the full timeout. Now the step returns in ~3 s marked
+  `[no visible change]`, so you know to check rather than wait.
+
+## CLI
+
+```bash
+simframe start                 # start the capture loop
+simframe mark                  # hash of the current frame, for --since
+simframe state --since=$H      # what changed, as text
+simframe frame --out=now.png   # newest frame
+simframe wait --since=$H       # change, then settle
+simframe ui                    # labels + tap points (ax and ocr)
+simframe recall                # what happened in the last minute
+simframe recall --ago=15000    # the frame from 15s ago
+simframe strip --count=6       # contact sheet
+simframe status / stop [--force] / devices / doctor
+```
 
 ## Limitations
 
-- Simulators only. `simctl` cannot capture a physical device.
-- simframe **reads** the screen; it does not tap, swipe or type. It is meant to
-  sit alongside whatever already drives input, replacing only the screenshot.
-- Capture tops out near 6 fps, because `simctl io screenshot` costs ~130 ms.
-  Fast animations are sampled, not recorded.
+- Simulators only — `simctl` cannot capture a physical device.
+- Capture tops out near 6 fps, because `simctl io screenshot` costs ~130 ms. Fast
+  animations are sampled, not recorded.
+- Region maps need a baseline inside the ~90 s history window. Older baselines
+  still get a reliable changed / did-not-change, without a map of what moved.
+- Screen memory assumes a screen's layout is stable. A screen that reflows
+  dramatically between visits will simply be rebuilt.
+- It speeds up *confirming* a fix, not *locating* one. A bug living in a memo
+  comparator or a stale closure is not visible in any frame.
 
 ## Roadmap
 
-- A higher-frame-rate backend via `simctl io recordVideo` piped through ffmpeg,
-  used automatically when ffmpeg is present.
-- Optional accessibility-tree text alongside the frame, so an agent can read
-  labels without spending image tokens.
-- Fusing input with settle-and-look, so tap → wait → see is one round trip
-  rather than three.
+- **Verify-after-tap.** A tap can move the screen without doing what you meant —
+  a swipe that animates but does not navigate still reports `changed`. Comparing
+  against the expected destination would catch it.
+- **Reduce the input dependency.** idb is the one heavyweight requirement. Its
+  simulator input is a reimplementation of the Indigo HID transport rather than a
+  public API, so replacing it is real work, not a wrapper — but it is the last
+  thing standing between simframe and a zero-install tool.
+- **Extend the confirm vocabulary beyond English.**
 
 ## Releasing
 
-`npm version <patch|minor|major>` does not update `server.json`, so bump both,
-then push the tag:
+`npm version` does not touch `server.json`, so bump both, then push the tag:
 
 ```bash
-npm version minor --no-git-tag-version      # bumps package.json
-$EDITOR server.json                         # match "version" and packages[0].version
-git commit -am "Release v0.2.0" && git tag v0.2.0
-git push && git push --tags
+npm version minor --no-git-tag-version
+$EDITOR server.json        # match "version" and packages[0].version
+git commit -am "Release vX.Y.Z" && git tag vX.Y.Z && git push && git push --tags
 ```
 
-The `release` workflow then verifies that the tag, `package.json` and
-`server.json` all agree, validates `server.json` against the live registry, and
-publishes to npm and to the MCP Registry. It needs an npm automation token in
-the `NPM_TOKEN` repository secret; the MCP Registry needs no secret, because it
-trusts the workflow's GitHub OIDC identity.
-
-To publish by hand instead:
-
-```bash
-npm publish --access public
-
-curl -fsSL https://github.com/modelcontextprotocol/registry/releases/latest/download/mcp-publisher_darwin_arm64.tar.gz | tar -xz mcp-publisher
-./mcp-publisher validate
-./mcp-publisher login github
-./mcp-publisher publish
-```
+The `release` workflow verifies tag/`package.json`/`server.json` agree, validates
+`server.json` against the live registry, and publishes to npm and the MCP
+Registry. It needs `NPM_TOKEN`; the registry uses GitHub OIDC and needs no secret.
 
 ## License
 

package/native/ocr.swift

@@ -0,0 +1,29 @@
+import Foundation
+import Vision
+import AppKit
+
+let args = CommandLine.arguments
+guard args.count > 1, let img = NSImage(contentsOfFile: args[1]),
+      let cg = img.cgImage(forProposedRect: nil, context: nil, hints: nil) else {
+    FileHandle.standardError.write("cannot read image\n".data(using: .utf8)!); exit(1)
+}
+let req = VNRecognizeTextRequest()
+req.recognitionLevel = .accurate
+req.usesLanguageCorrection = false
+try! VNImageRequestHandler(cgImage: cg, options: [:]).perform([req])
+let w = Double(cg.width), h = Double(cg.height)
+var out: [[String: Any]] = []
+for obs in (req.results ?? []) {
+    guard let top = obs.topCandidates(1).first else { continue }
+    let b = obs.boundingBox   // normalized, origin bottom-left
+    out.append([
+        "text": top.string,
+        "confidence": top.confidence,
+        "x": b.origin.x * w,
+        "y": (1 - b.origin.y - b.size.height) * h,
+        "width": b.size.width * w,
+        "height": b.size.height * h,
+    ])
+}
+let data = try! JSONSerialization.data(withJSONObject: out)
+FileHandle.standardOutput.write(data)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "simframe",
-  "version": "0.1.0",
+  "version": "0.4.1",
   "mcpName": "io.github.lvlrSajjad/simframe",
   "description": "Always-warm iOS Simulator frames: agents read the screen in ~20ms instead of waiting on screenshots. MCP server + CLI.",
   "keywords": [
@@ -40,6 +40,7 @@
   },
   "files": [
     "src",
+    "native",
     "README.md",
     "LICENSE"
   ],