simframe 0.4.2 → 0.5.0

package/README.md

@@ -32,13 +32,25 @@ Same four-tab navigation flow, on a real production app:
 | --- | --- | --- |
 | Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
 | "Did anything change?" | a full image | **~2 ms**, text only |
-| A 5-step flow | 5+ model round trips | **1 call**, ~7 s |
 | Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
-| Same flow, 3rd run | no improvement — every run is the first | **3304 ms, 4/4 from memory** |
+| A 4-step flow, verified | 4+ model round trips | **1 call**, 3.6 s |
+| Same flow, 3rd run | no improvement — every run is the first | **3.7 s, 4/4 from memory, 4/4 verified** |
 
-The same four-tab tour, run three times back to back: **7370 ms → 5160 ms →
-3304 ms**, with 1, then 3, then 4 of the four controls resolved from memory and
-no mis-taps. What is left is mostly the app's own animation and data load.
+The four-tab tour, three times back to back from a cleared memory:
+
+| Pass | Wall clock | Steps verified | Controls from memory |
+| --- | --- | --- | --- |
+| 1 | 10.2 s | 0/4 — nothing is known yet | **4/4** |
+| 2 | **3.6 s** | **4/4** | **4/4** |
+| 3 | **3.7 s** | **4/4** | **4/4** |
+
+Every step is checked against what the same action did last time, and the run
+records its own preconditions — which input path, which daemon, whether the
+daemon was replaced mid-run — so a regression shows up in the measurement rather
+than hiding inside it. Earlier versions of this table quoted 7.4 s → 3.3 s with
+verification switched off; those numbers were measured while input was silently
+falling back to a slower path and the capture daemon was being replaced by every
+command, so they measured two bugs rather than the tool.
 
 ## Install
 
@@ -47,6 +59,11 @@ npm install -g simframe
 simframe doctor
 ```
 
+The first `simframe start` builds a small Swift daemon from source — a few
+seconds, once. It needs the Xcode command line tools, which you already have if
+you have a simulator. Without them simframe falls back to the original
+`simctl` loop and says so.
+
 `doctor` checks each capability separately and tells you what you have:
 
 ```
@@ -86,7 +103,12 @@ have. **Observation needs nothing but Xcode.**
 | --- | --- | --- |
 | Watch the screen, wait, recall | nothing extra | — |
 | Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
-| Tap, type, swipe | [`idb`](https://fbidb.io) | simframe observes but cannot touch |
+| Tap, type, swipe | nothing extra, or [`idb`](https://fbidb.io) as a fallback | simframe observes but cannot touch |
+| Accessibility tree | [`idb`](https://fbidb.io) | OCR alone still yields labels and coordinates |
+
+`simframe doctor` names which engine is carrying each capability, per device.
+**idb is now required only for the accessibility tree** — capture, input and text
+recognition all run in-process.
 
 ```bash
 # input, optional
@@ -144,7 +166,8 @@ So simframe reads the screen two ways and remembers the result:
 - **Accessibility** gives real hit targets, types and enabled state.
 - **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
   label a person can actually see, with coordinates.
-- The merge is keyed by a **layout hash**, so the next visit is a file read.
+- The merge is keyed by a **structural fingerprint**, so the next visit is a
+  file read. What that fingerprint is, and why it is not a pixel hash, is below.
 
 ```
 first visit to a screen   ~1000 ms   read tree + OCR, store the map
@@ -164,34 +187,92 @@ Two details that matter:
   a tab returns an error listing both with coordinates, because silently tapping
   the title looks exactly like nothing happening.
 
-### Why a layout hash, not a frame hash
+### Two hashes, because there are two questions
 
+"Did this move?" and "is this the same screen?" look like one question and are
+not. simframe answers them separately, and getting that wrong was the single
+most expensive mistake in its development.
+
+**Change and settle** are questions about pixels, so a pixel hash answers them.
 The frame hash changes whenever any pixel group changes — a clock digit, one new
-row of data — which makes it useless as a key for "have I seen this screen
-before?". The layout hash crops the status bar and takes a difference hash over a
-12×24 grid.
+row — which is exactly right for "did anything happen?" and useless as a key for
+"have I been here before?". For change detection there is a layout hash: status
+bar cropped, difference hash over a 12×24 grid.
 
 A mean-threshold hash was tried first and was actively dangerous: low-contrast
-app screens collapsed onto identical values, so unrelated screens matched at
-distance 0 and taps landed on the wrong control. Measured on a real app:
-
-| | Hamming distance |
+screens collapsed onto identical values, so unrelated screens matched at distance
+0 and taps landed on the wrong control. The difference hash fixed that.
+
+**Identity is not a question about pixels**, and this is the part that took three
+attempts. Content *is* pixels: a list whose rows changed drifts as far as a
+different screen does. Measured on a real app, same-screen revisits reached 62
+bits against a different-screen floor of 74 — overlapping, with no threshold
+available to choose. An earlier calibration had suggested a comfortable margin
+(0–4 against 77–113), but it was measured on screens whose content happened to be
+stable and did not survive contact with a real list.
+
+So identity is **structural**. The fingerprint is built from element roles,
+frames quantised to a 24 px grid, the region each element sits in, and repeated
+siblings bucketed as "one" or "many" rather than counted. Deliberately included:
+the labels of chrome elements only — nav title, tab labels, toolbar buttons —
+because two list screens with identical structure are told apart by their title
+and nothing else. Deliberately excluded: all content text and values, the status
+bar, and the keyboard region when a keyboard is up.
+
+It does not depend on the accessibility tree. Fingerprinting from OCR boxes
+alone, with the tree discarded entirely, still separates screens — different
+screens ceiling 0.35 against the same threshold.
+
+| | Jaccard similarity |
 | --- | --- |
-| Same screen, revisited while settled | **0–4** |
-| Same screen, but loading vs loaded | 48–98 |
-| **Different screens** | **77–113** |
-
-Only one of those errors is dangerous. Matching the *wrong* screen would tap the
-wrong control, and that needs two different screens to land within 12 bits of
-each other — the closest pair ever measured was 77. Failing to recognise a screen
-you have seen is harmless: it rebuilds the map, costs ~600 ms, and taps correctly.
-So the tolerance is deliberately far below the collision floor rather than tuned
-to maximise hits.
-
-That middle row is worth knowing about: a screen mid-load genuinely does not look
-like the same screen loaded, so the first visit after a cold launch usually
-rebuilds. Hit rates climb as an app warms up, which is exactly what the three-pass
-numbers above show.
+| Same screen, revisited | 0.41–1.00 |
+| **Different screens** | **0.00–0.31** |
+
+The threshold sits in that gap, but the gap is narrower than anyone would want,
+and one screen causes it: a screen whose sections load from different sources has
+more than one genuine settled structure, and two structures of one screen are as
+far apart as two different screens.
+
+No threshold can express that, so a screen may hold **several** accepted
+fingerprints instead. A new one is admitted only when a known edge lands
+somewhere its target does not recognise — the edge is the evidence that it is the
+same place — and only if no other stored screen claims that reading. Identity
+stays exact rather than being loosened, and the count is capped, so a
+non-deterministic action shows up as a node collecting variants rather than as
+screens silently merging.
+
+It earns its place on real apps: an app reconnecting to its bundler put an alert
+over one screen, and that screen gained a variant instead of a duplicate
+appearing.
+
+Failing to recognise a screen you have seen is harmless — it rebuilds the map and
+taps correctly. Matching the *wrong* screen taps the wrong control. The threshold
+is set to err toward the first.
+
+## Navigating by memory
+
+Once simframe knows which screens exist and which action leads from one to the
+next, getting somewhere is a search over known edges rather than a question for a
+model:
+
+```bash
+simframe screens              # what this device has learned
+simframe goto invoices        # walk there, verifying every step
+```
+
+Measured on a four-tab tour, `goto` plans and walks three-step routes with every
+step verified and no model call. It fails rather than guesses: an unknown
+destination, a query matching two screens equally, or no path of known edges all
+report themselves instead of tapping hopefully.
+
+Flows work the same way and refuse to save if any step went unverified —
+replaying a recording of something that may not have worked just reproduces the
+doubt.
+
+```bash
+simframe flow save checkout ./checkout.json
+simframe flow run checkout
+```
 
 ## Does this work on *your* app?
 
@@ -219,37 +300,65 @@ iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
 
 | | |
 | --- | --- |
+| Frame capture, whole pipeline | **6.6 ms** |
+| Frame grab alone | **0.13 ms** |
+| The `simctl` + `sips` path it replaces | ~210 ms |
 | Warm frame read (`sim_look`) | ~20 ms |
 | State check (`sim_state`) | ~2 ms |
-| Contact sheet (`sim_strip`, 5 frames) | ~30 ms |
-| Accessibility tree read | ~570 ms |
-| On-device OCR of a full frame | ~290 ms |
-| Screen map: first visit / remembered | ~600 ms / **~1 ms** |
-| Raw `simctl io screenshot`, for comparison | ~130 ms, every look |
-| Cold start, first frame | ~400 ms, once |
+| Input round trip (`ping`) | **0 ms** |
+| Tap (70 ms hold / 10 ms hold) | 76 ms / 13 ms |
+| Text recognition, in-process | **~174 ms** |
+| Text recognition, via PNG + helper (fallback) | ~555 ms |
+| Accessibility tree read (idb) | ~570 ms |
+| Screen map: first visit / remembered | ~305 ms / **~1 ms** |
 | CPU | 1.1 % idle · 3.1 % active |
 | Frame memory | ~60 s of screen, ~2.7 MB |
 
+Reproduce all of it with `npm run bench`, which prints the same table against
+your machine. Full detail, including the measurement traps, is in
+[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
+
 ## How it works
 
 ```
-┌──────────────────── background, one loop per simulator ─────────────────────┐
-│   simctl screenshot ──► sips ──► decode ──► hash + diff ──► rename into      │
-│      ~130 ms            ~30 ms    ~6 ms                  ~/.simframe/<udid>/ │
+┌─── simframed — one Swift daemon per simulator ──────────────────────────────┐
+│                                                                             │
+│   display damage callback ──► read IOSurface ──► scale ──► hash             │
+│        the screen tells us          0.13 ms      6.6 ms total               │
+│                                          │                                  │
+│   Vision OCR reads the same surface ─────┤   no PNG, no file, no spawn      │
+│                                          │                                  │
+│   Indigo HID ◄── control socket ◄────────┤   0600, one JSON object per line │
+│   taps, swipes, text                     │                                  │
+│                                          ▼                                  │
+│                        ~/.simframe/<udid>/  frames · state.json · meta.json │
 └─────────────────────────────────────────────────────────────────────────────┘
                                      │  a rename is atomic
 ┌────────────────────────────────────▼────────────────────────────────────────┐
-│   MCP server / CLI: stat + read. No simctl anywhere in the request path.     │
-│   Screen memory: layout hash ──► label → point, built once per screen.       │
+│   MCP server / CLI: stat + read, or one socket round trip for input.        │
+│   Screen memory: layout hash ──► label → point, built once per screen.      │
 └─────────────────────────────────────────────────────────────────────────────┘
 ```
 
-- **Files are the IPC.** The loop renames completed frames into place; readers
-  just read them. A rename is atomic, so a reader can never see a half-written
-  frame, and there is no socket or protocol to get wrong.
-- **Almost no dependencies.** Resizing uses `sips`; PNG codec, hashing and frame
-  comparison are plain JavaScript over `node:zlib`. The only runtime dependency
-  is the MCP SDK. OCR is a ~60-line Swift file compiled on first use.
+The daemon links CoreSimulator and SimulatorKit, which are private frameworks
+with no documentation and no stability promise. Everything it calls is recorded
+in [`docs/PRIVATE_API.md`](docs/PRIVATE_API.md) with the evidence behind it, so
+an Xcode upgrade that moves something is a bounded fix rather than an
+archaeology project. If a layer breaks, simframe degrades to the layer below
+and `doctor` says which.
+
+Run `simframe start --engine=simctl` to use the original loop instead.
+
+- **Files are the IPC for reads.** The daemon renames completed frames into
+  place; readers just read them. A rename is atomic, so a reader can never see a
+  half-written frame. Input is the one thing that needs a reply, and it goes
+  over a `0600` Unix socket — the file system is the whole permission model.
+- **Almost no dependencies.** The only runtime npm dependency is the MCP SDK.
+  The daemon is Swift built from source against frameworks already on the
+  machine.
+- **The screen says when it changed.** The capture loop is driven by the
+  display's damage callback rather than a timer, so an idle screen costs
+  nothing and a moving one is picked up at once.
 - **It backs off when nothing happens.** 4 fps while the screen moves, 1.5 fps
   once still, snapping back instantly on change.
 - **One writer per device.** Ownership lives in `meta.json`; `stop` refuses to
@@ -273,14 +382,72 @@ simframe ui                    # labels + tap points (ax and ocr)
 simframe recall                # what happened in the last minute
 simframe recall --ago=15000    # the frame from 15s ago
 simframe strip --count=6       # contact sheet
+simframe screens               # screens this device has learned
+simframe goto invoices         # walk to a known screen over known steps
+simframe flow save|run|list    # record a verified flow, replay it
+simframe doctor --json         machine-readable; --strict fails on any downgrade
 simframe status / stop [--force] / devices / doctor
 ```
 
+## Degrading is allowed. Degrading quietly is not
+
+simframe is built to degrade rather than fail: no Swift toolchain still gives
+you frames through `simctl`, no accessibility tree still gives you OCR. That
+policy is right, and it nearly sank the tool twice — because a downgrade looked
+exactly like everything working.
+
+Once, one file was missing from the published package, so `Package.swift`
+declared a test target with no directory, SwiftPM reported overlapping sources,
+and **every install silently fell back to the slow engine**. Another time OCR
+shipped disabled the same way. Both passed the tests. Both printed nothing. The
+bug was never the missing file; it was the silence.
+
+So every downgrade now announces itself:
+
+- `simframe start` prints the engine it chose, and if it is the slow one, why —
+  build error, missing sources, or "reason unrecorded" if the daemon was started
+  by an earlier process.
+- `simframe doctor` marks a degraded layer `WARN`, not `ok`, and summarises what
+  is degraded and what that costs.
+- A dependency that is simply not installed is `--`, not `WARN`. The distinction
+  is deliberate: `WARN` means this machine could be doing better and silently is
+  not, which is the failure worth shouting about. idb missing on a fresh machine
+  has not degraded from anything, and `--strict` ignores it.
+- `--strict`, or `SIMFRAME_STRICT=1`, turns any downgrade into a non-zero exit.
+  CI runs strict, so a release cannot ship in the state that shipped twice.
+
+```
+$ simframe doctor
+ok   capture engine        simframed
+WARN input driver          idb — the daemon's control socket is not up
+--   accessibility tree    not installed: idb is not installed
+```
+
+Two checks enforce it. A packaging check derives the required file list from the
+build's own inputs — a hand-written list is what rotted last time — and runs in
+seconds without a simulator. An integration job installs the packed tarball on a
+real simulator and asserts `capture.engine`, `input.driver` and `ocr.available`
+are all the good values, under `--strict`.
+
+That last check found a real bug the day it was written: a daemon shutting down
+unlinked the control socket unconditionally, so restarting deleted the *new*
+daemon's socket. Capture kept working, input quietly dropped to idb, and nothing
+said a word — the exact failure shape, found by the thing built to catch it.
+
 ## Limitations
 
-- Simulators only — `simctl` cannot capture a physical device.
-- Capture tops out near 6 fps, because `simctl io screenshot` costs ~130 ms. Fast
-  animations are sampled, not recorded.
+- Simulators only. Neither the framebuffer nor `simctl` can reach a physical
+  device.
+- The daemon depends on private frameworks. They are stable enough to build on —
+  capture and accessibility survived the iOS 26 transition — but an Xcode
+  upgrade can move a symbol. `doctor` reports each layer separately so a break
+  is visible rather than mysterious, and `--engine=simctl` still works.
+- Hardware buttons: only `home` is implemented. The other Indigo codes are
+  unverified, and a wrong one can crash `backboardd` or lock the device, so they
+  return an error rather than a guess.
+- Typing sends key positions, which iOS maps through the device's active
+  keyboard layout. Text that must be exact goes through the pasteboard, which
+  `sim_do` does by default.
 - Region maps need a baseline inside the ~90 s history window. Older baselines
   still get a reliable changed / did-not-change, without a map of what moved.
 - Screen memory assumes a screen's layout is stable. A screen that reflows
@@ -290,9 +457,16 @@ simframe status / stop [--force] / devices / doctor
 
 ## Roadmap
 
-- **Verify-after-tap.** A tap can move the screen without doing what you meant —
-  a swipe that animates but does not navigate still reports `changed`. Comparing
-  against the expected destination would catch it.
+- **The accessibility tree without idb.** `AXPTranslator` would remove the last
+  heavyweight install. Capture, input and geometry already come from the daemon;
+  the tree is all that is left.
+- **A compact state for the agent.** The element list, regions, what changed and
+  the last verdict, shaped so a model spends tokens on deciding rather than on
+  reading. This is where the token savings actually land.
+- **Region bands from where elements cluster**, rather than fractions of screen
+  height. A date banner sitting above the tab bar was classified as a tab label
+  and its text entered a screen's identity, which would have expired at
+  midnight. Fixed for that case by a width rule; the underlying cause remains.
 - **Reduce the input dependency.** idb is the one heavyweight requirement. Its
   simulator input is a reimplementation of the Indigo HID transport rather than a
   public API, so replacing it is real work, not a wrapper — but it is the last

package/native/simframed/Package.swift

@@ -0,0 +1,16 @@
+// swift-tools-version:5.9
+import PackageDescription
+
+let package = Package(
+    name: "simframed",
+    platforms: [.macOS(.v13)],
+    targets: [
+        // Every private-framework call lives here, behind a protocol, so a
+        // signature change is a one-file fix and tests can run without a device.
+        .target(name: "PrivateAPI"),
+        // Platform-independent: hashing, downscaling, the on-disk frame layout.
+        .target(name: "SimframeCore", dependencies: ["PrivateAPI"]),
+        .executableTarget(name: "simframed", dependencies: ["SimframeCore", "PrivateAPI"]),
+        .testTarget(name: "SimframeCoreTests", dependencies: ["SimframeCore"]),
+    ]
+)