simframe 0.4.2 → 0.6.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/README.md +334 -85
  2. package/native/simframed/Package.swift +16 -0
  3. package/native/simframed/Sources/PrivateAPI/AccessibilityBridge.swift +379 -0
  4. package/native/simframed/Sources/PrivateAPI/CoreSimulatorPlatform.swift +523 -0
  5. package/native/simframed/Sources/PrivateAPI/HIDKeyboard.swift +70 -0
  6. package/native/simframed/Sources/PrivateAPI/IndigoHID.swift +121 -0
  7. package/native/simframed/Sources/PrivateAPI/PrivateAPI.swift +149 -0
  8. package/native/simframed/Sources/PrivateAPI/StubPlatform.swift +112 -0
  9. package/native/simframed/Sources/SimframeCore/Bitmap.swift +61 -0
  10. package/native/simframed/Sources/SimframeCore/ControlSocket.swift +122 -0
  11. package/native/simframed/Sources/SimframeCore/CoreGraphicsScaler.swift +70 -0
  12. package/native/simframed/Sources/SimframeCore/Element.swift +148 -0
  13. package/native/simframed/Sources/SimframeCore/FrameStore.swift +303 -0
  14. package/native/simframed/Sources/SimframeCore/Hashing.swift +119 -0
  15. package/native/simframed/Sources/SimframeCore/Motion.swift +431 -0
  16. package/native/simframed/Sources/SimframeCore/PNGWriter.swift +40 -0
  17. package/native/simframed/Sources/SimframeCore/VisionOCR.swift +75 -0
  18. package/native/simframed/Sources/simframed/main.swift +485 -0
  19. package/native/simframed/Tests/SimframeCoreTests/HashingTests.swift +270 -0
  20. package/package.json +12 -4
  21. package/scripts/bench-flow.mjs +54 -0
  22. package/scripts/bench.sh +98 -0
  23. package/scripts/check-package.mjs +99 -0
  24. package/scripts/ci-memory.mjs +416 -0
  25. package/scripts/eval-fingerprint.mjs +192 -0
  26. package/scripts/smoke.mjs +76 -0
  27. package/scripts/sync-server-version.mjs +39 -0
  28. package/scripts/verify-baseline.mjs +65 -0
  29. package/skills/simframe/SKILL.md +173 -0
  30. package/src/actions.js +264 -18
  31. package/src/cli.js +561 -89
  32. package/src/control.js +77 -0
  33. package/src/daemon.js +8 -1
  34. package/src/engine.js +99 -0
  35. package/src/fingerprint.js +183 -0
  36. package/src/graph.js +411 -0
  37. package/src/index.js +351 -24
  38. package/src/input.js +179 -2
  39. package/src/matching.js +265 -0
  40. package/src/mcp.js +425 -112
  41. package/src/navigate.js +120 -0
  42. package/src/refs.js +141 -0
  43. package/src/regions.js +267 -0
  44. package/src/screenmap.js +119 -22
  45. package/src/simctl.js +74 -5
  46. package/src/store.js +8 -0
  47. package/src/view.js +342 -0
package/README.md CHANGED
@@ -20,9 +20,15 @@ one is obvious:
20
20
  3. **Nothing is remembered.** The same screen gets re-read and re-reasoned about
21
21
  every single time it appears.
22
22
 
23
- simframe attacks all three: a background loop keeps the newest frame warm, whole
24
- flows run in one call, and screens the agent has seen before are answered from
25
- memory.
23
+ And there is a fourth that is pure waste: **an image is the most expensive way
24
+ to ask what is on screen.** A screenshot costs ~1,600 tokens when it is handled
25
+ as a native image block and 15,000–25,000 when it is not, and it does not tell
26
+ you what is tappable or where — you have to measure that by eye.
27
+
28
+ simframe attacks all four: a background loop keeps the newest frame warm, whole
29
+ flows run in one call, screens the agent has seen before are answered from
30
+ memory, and every answer is text with tap points in it. Nothing returns an
31
+ image unless you ask for one.
26
32
 
27
33
  ## What changed, measured
28
34
 
@@ -32,13 +38,27 @@ Same four-tab navigation flow, on a real production app:
32
38
  | --- | --- | --- |
33
39
  | Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
34
40
  | "Did anything change?" | a full image | **~2 ms**, text only |
35
- | A 5-step flow | 5+ model round trips | **1 call**, ~7 s |
36
41
  | Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
37
- | Same flow, 3rd run | no improvement — every run is the first | **3304 ms, 4/4 from memory** |
38
-
39
- The same four-tab tour, run three times back to back: **7370 ms → 5160 ms →
40
- 3304 ms**, with 1, then 3, then 4 of the four controls resolved from memory and
41
- no mis-taps. What is left is mostly the app's own animation and data load.
42
+ | A 4-step flow, verified | 4+ model round trips | **1 call**, 3.6 s |
43
+ | Same flow, 3rd run | no improvement — every run is the first | **3.7 s, 4/4 from memory, 4/4 verified** |
44
+ | A 10-step flow | 10 turns, 10 images (~16,000 tokens at best) | **1 turn, 0 images, ~1,650 characters** |
45
+ | Reading a screen | an image: ~1,600 tokens, no tap points | **~330 tokens** of text, with tap points |
46
+
47
+ The four-tab tour, three times back to back from a cleared memory:
48
+
49
+ | Pass | Wall clock | Steps verified | Controls from memory |
50
+ | --- | --- | --- | --- |
51
+ | 1 | 10.2 s | 0/4 — nothing is known yet | **4/4** |
52
+ | 2 | **3.6 s** | **4/4** | **4/4** |
53
+ | 3 | **3.7 s** | **4/4** | **4/4** |
54
+
55
+ Every step is checked against what the same action did last time, and the run
56
+ records its own preconditions — which input path, which daemon, whether the
57
+ daemon was replaced mid-run — so a regression shows up in the measurement rather
58
+ than hiding inside it. Earlier versions of this table quoted 7.4 s → 3.3 s with
59
+ verification switched off; those numbers were measured while input was silently
60
+ falling back to a slower path and the capture daemon was being replaced by every
61
+ command, so they measured two bugs rather than the tool.
42
62
 
43
63
  ## Install
44
64
 
@@ -47,12 +67,18 @@ npm install -g simframe
47
67
  simframe doctor
48
68
  ```
49
69
 
70
+ The first `simframe start` builds a small Swift daemon from source — a few
71
+ seconds, once. It needs the Xcode command line tools, which you already have if
72
+ you have a simulator. Without them simframe falls back to the original
73
+ `simctl` loop and says so.
74
+
50
75
  `doctor` checks each capability separately and tells you what you have:
51
76
 
52
77
  ```
53
78
  ok xcrun xcrun version 72.
54
79
  ok sips available
55
- ok input driver (idb) companion built Sep 1 2026
80
+ ok input driver simframed: Indigo HID
81
+ ok accessibility tree simframed: AXPTranslator, host-side
56
82
  ok on-device OCR available
57
83
  ok booted simulator iPhone 17 Pro (iOS 26.5)
58
84
  ok capture frame #888 322x700 in 2ms (age 538ms)
@@ -86,28 +112,73 @@ have. **Observation needs nothing but Xcode.**
86
112
  | --- | --- | --- |
87
113
  | Watch the screen, wait, recall | nothing extra | — |
88
114
  | Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
89
- | Tap, type, swipe | [`idb`](https://fbidb.io) | simframe observes but cannot touch |
115
+ | Tap, type, swipe | nothing extra | simframe observes but cannot touch |
116
+ | Accessibility tree | nothing extra | OCR alone still yields labels and coordinates |
117
+
118
+ `simframe doctor` names which engine is carrying each capability, per device.
119
+ **Nothing beyond Xcode is required.** Capture, input, text recognition and the
120
+ accessibility tree all run in-process, in one daemon.
121
+
122
+ [`idb`](https://fbidb.io) is still accepted as a fallback for input and for the
123
+ tree, for a machine where the daemon cannot run — and `SIMFRAME_AX_DRIVER=idb`
124
+ forces the tree back onto it, which is the escape hatch if an Xcode upgrade
125
+ breaks the host-side path.
90
126
 
91
127
  ```bash
92
- # input, optional
128
+ # optional fallback, not a requirement
93
129
  brew tap facebook/fb && brew install idb-companion && pipx install fb-idb
94
130
  ```
95
131
 
96
- Homebrew may ask you to trust the tap first; that is a deliberate prompt for a
97
- human, and the narrow form is `brew trust --formula facebook/fb/idb-companion`.
98
-
99
132
  ## The tools
100
133
 
134
+ Read first, act in batches, and look at pixels only when the question is about
135
+ pixels. Every tool description says so, because a tool surface that does not
136
+ steer the model is a tool surface the model uses wrong.
137
+
101
138
  | Tool | What it does |
102
139
  | --- | --- |
103
- | `sim_look` | Newest frame as an image, no capture wait. |
104
- | `sim_state` | Text only: screen hash, what changed **since your last look**, region movement map. |
140
+ | `sim_ui` | **Start here.** The screen as a numbered text map: region, type, label, state, tap point, source. A tenth the cost of a screenshot and strictly more useful. |
141
+ | `sim_do` | **The main tool.** A whole flow in one call — tap, type, scroll, wait, assert — each step settling before the next and verified against what it did last time. |
142
+ | `sim_state` | The cheapest question there is: has anything changed **since your last look**, and which regions moved. |
143
+ | `sim_goto` | Walk to a screen simframe has been to before, planning the route through remembered transitions. |
144
+ | `sim_flow_run` | Replay a flow that verified end to end. |
145
+ | `sim_find` | Resolve an intent to one control, without acting on it. |
146
+ | `sim_tap` · `sim_type_into` · `sim_scroll_to` · `sim_wait_for` · `sim_assert` | Single actions, for when you genuinely only have one step. Each is one `sim_do` step underneath. |
147
+ | `sim_launch` · `sim_open_url` · `sim_permission` | Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert. |
105
148
  | `sim_wait` | Waits for the screen to change *and then* settle. |
106
- | `sim_do` | A whole flow in one call — tap, type, scroll, assert — each step settling before the next. |
107
- | `sim_ui` | The screen as labels + tap coordinates, from accessibility **and** OCR. |
108
- | `sim_recall` | Look backwards: a timeline of what happened, or the frame from N seconds ago. |
109
- | `sim_strip` | Recent frames tiled into one image. |
110
- | `sim_capture` / `sim_devices` | Manage capture loops; list simulators. |
149
+ | `sim_look` | **The only tool that returns an image**, capped at 1024 px. For layout, colour, spacing — questions text cannot answer. |
150
+ | `sim_recall` · `sim_strip` | Look backwards: a text timeline of what happened, or recent frames tiled into one image. |
151
+ | `sim_capture` · `sim_devices` | Manage capture loops; list simulators. |
152
+
153
+ ### What the screen looks like as text
154
+
155
+ ```
156
+ iPhone 17 Pro · 402x874pt · screen a1b2c3d4 "Inbox" (known, 3 known exits)
157
+ last action: [2] tap — ok: matches the outcome seen 5x before
158
+ nav-bar:
159
+ #1 button 24,64 Back
160
+ #2 text 201,64 Inbox
161
+ content:
162
+ #3 cell 201,140 Weekly digest
163
+ #4 cell 201,196 Payment received
164
+ tab-bar:
165
+ #5 text 62,835 Inbox
166
+ #6 text 201,835 Settings
167
+ ```
168
+
169
+ Region first, because "Inbox" the title and "Inbox" the tab differ only by where
170
+ they are. A tap point, because that is what an action needs. And a number, which
171
+ is a selector: whatever this calls `#3`, the next call can tap as `#3` without
172
+ describing it. A ref is valid only while that screen is showing — used on a
173
+ different screen it refuses rather than tapping whatever now sits there.
174
+
175
+ Three ways to name a control, anywhere one is named:
176
+
177
+ | | |
178
+ | --- | --- |
179
+ | `#3` | the number the map gave it. Cheapest, unambiguous. |
180
+ | `"Save"` · `the Assets tab` · `back` | resolved by intent — verbs, typos, synonyms, icon-only controls by their common name |
181
+ | `@120,400` | raw point coordinates. Last resort: it cannot tell you it missed. |
111
182
 
112
183
  ## Baselines: the thing to understand
113
184
 
@@ -144,7 +215,8 @@ So simframe reads the screen two ways and remembers the result:
144
215
  - **Accessibility** gives real hit targets, types and enabled state.
145
216
  - **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
146
217
  label a person can actually see, with coordinates.
147
- - The merge is keyed by a **layout hash**, so the next visit is a file read.
218
+ - The merge is keyed by a **structural fingerprint**, so the next visit is a
219
+ file read. What that fingerprint is, and why it is not a pixel hash, is below.
148
220
 
149
221
  ```
150
222
  first visit to a screen ~1000 ms read tree + OCR, store the map
@@ -164,34 +236,92 @@ Two details that matter:
164
236
  a tab returns an error listing both with coordinates, because silently tapping
165
237
  the title looks exactly like nothing happening.
166
238
 
167
- ### Why a layout hash, not a frame hash
239
+ ### Two hashes, because there are two questions
240
+
241
+ "Did this move?" and "is this the same screen?" look like one question and are
242
+ not. simframe answers them separately, and getting that wrong was the single
243
+ most expensive mistake in its development.
168
244
 
245
+ **Change and settle** are questions about pixels, so a pixel hash answers them.
169
246
  The frame hash changes whenever any pixel group changes — a clock digit, one new
170
- row of data — which makes it useless as a key for "have I seen this screen
171
- before?". The layout hash crops the status bar and takes a difference hash over a
172
- 12×24 grid.
247
+ row — which is exactly right for "did anything happen?" and useless as a key for
248
+ "have I been here before?". For change detection there is a layout hash: status
249
+ bar cropped, difference hash over a 12×24 grid.
173
250
 
174
251
  A mean-threshold hash was tried first and was actively dangerous: low-contrast
175
- app screens collapsed onto identical values, so unrelated screens matched at
176
- distance 0 and taps landed on the wrong control. Measured on a real app:
177
-
178
- | | Hamming distance |
252
+ screens collapsed onto identical values, so unrelated screens matched at distance
253
+ 0 and taps landed on the wrong control. The difference hash fixed that.
254
+
255
+ **Identity is not a question about pixels**, and this is the part that took three
256
+ attempts. Content *is* pixels: a list whose rows changed drifts as far as a
257
+ different screen does. Measured on a real app, same-screen revisits reached 62
258
+ bits against a different-screen floor of 74 — overlapping, with no threshold
259
+ available to choose. An earlier calibration had suggested a comfortable margin
260
+ (0–4 against 77–113), but it was measured on screens whose content happened to be
261
+ stable and did not survive contact with a real list.
262
+
263
+ So identity is **structural**. The fingerprint is built from element roles,
264
+ frames quantised to a 24 px grid, the region each element sits in, and repeated
265
+ siblings bucketed as "one" or "many" rather than counted. Deliberately included:
266
+ the labels of chrome elements only — nav title, tab labels, toolbar buttons —
267
+ because two list screens with identical structure are told apart by their title
268
+ and nothing else. Deliberately excluded: all content text and values, the status
269
+ bar, and the keyboard region when a keyboard is up.
270
+
271
+ It does not depend on the accessibility tree. Fingerprinting from OCR boxes
272
+ alone, with the tree discarded entirely, still separates screens — different
273
+ screens ceiling 0.35 against the same threshold.
274
+
275
+ | | Jaccard similarity |
179
276
  | --- | --- |
180
- | Same screen, revisited while settled | **0–4** |
181
- | Same screen, but loading vs loaded | 48–98 |
182
- | **Different screens** | **77–113** |
183
-
184
- Only one of those errors is dangerous. Matching the *wrong* screen would tap the
185
- wrong control, and that needs two different screens to land within 12 bits of
186
- each other — the closest pair ever measured was 77. Failing to recognise a screen
187
- you have seen is harmless: it rebuilds the map, costs ~600 ms, and taps correctly.
188
- So the tolerance is deliberately far below the collision floor rather than tuned
189
- to maximise hits.
190
-
191
- That middle row is worth knowing about: a screen mid-load genuinely does not look
192
- like the same screen loaded, so the first visit after a cold launch usually
193
- rebuilds. Hit rates climb as an app warms up, which is exactly what the three-pass
194
- numbers above show.
277
+ | Same screen, revisited | 0.41–1.00 |
278
+ | **Different screens** | **0.00–0.31** |
279
+
280
+ The threshold sits in that gap, but the gap is narrower than anyone would want,
281
+ and one screen causes it: a screen whose sections load from different sources has
282
+ more than one genuine settled structure, and two structures of one screen are as
283
+ far apart as two different screens.
284
+
285
+ No threshold can express that, so a screen may hold **several** accepted
286
+ fingerprints instead. A new one is admitted only when a known edge lands
287
+ somewhere its target does not recognise — the edge is the evidence that it is the
288
+ same place — and only if no other stored screen claims that reading. Identity
289
+ stays exact rather than being loosened, and the count is capped, so a
290
+ non-deterministic action shows up as a node collecting variants rather than as
291
+ screens silently merging.
292
+
293
+ It earns its place on real apps: an app reconnecting to its bundler put an alert
294
+ over one screen, and that screen gained a variant instead of a duplicate
295
+ appearing.
296
+
297
+ Failing to recognise a screen you have seen is harmless — it rebuilds the map and
298
+ taps correctly. Matching the *wrong* screen taps the wrong control. The threshold
299
+ is set to err toward the first.
300
+
301
+ ## Navigating by memory
302
+
303
+ Once simframe knows which screens exist and which action leads from one to the
304
+ next, getting somewhere is a search over known edges rather than a question for a
305
+ model:
306
+
307
+ ```bash
308
+ simframe screens # what this device has learned
309
+ simframe goto invoices # walk there, verifying every step
310
+ ```
311
+
312
+ Measured on a four-tab tour, `goto` plans and walks three-step routes with every
313
+ step verified and no model call. It fails rather than guesses: an unknown
314
+ destination, a query matching two screens equally, or no path of known edges all
315
+ report themselves instead of tapping hopefully.
316
+
317
+ Flows work the same way and refuse to save if any step went unverified —
318
+ replaying a recording of something that may not have worked just reproduces the
319
+ doubt.
320
+
321
+ ```bash
322
+ simframe flow save checkout ./checkout.json
323
+ simframe flow run checkout
324
+ ```
195
325
 
196
326
  ## Does this work on *your* app?
197
327
 
@@ -219,37 +349,66 @@ iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
219
349
 
220
350
  | | |
221
351
  | --- | --- |
352
+ | Frame capture, whole pipeline | **6.6 ms** |
353
+ | Frame grab alone | **0.13 ms** |
354
+ | The `simctl` + `sips` path it replaces | ~210 ms |
222
355
  | Warm frame read (`sim_look`) | ~20 ms |
223
356
  | State check (`sim_state`) | ~2 ms |
224
- | Contact sheet (`sim_strip`, 5 frames) | ~30 ms |
225
- | Accessibility tree read | ~570 ms |
226
- | On-device OCR of a full frame | ~290 ms |
227
- | Screen map: first visit / remembered | ~600 ms / **~1 ms** |
228
- | Raw `simctl io screenshot`, for comparison | ~130 ms, every look |
229
- | Cold start, first frame | ~400 ms, once |
357
+ | Input round trip (`ping`) | **0 ms** |
358
+ | Tap (70 ms hold / 10 ms hold) | 76 ms / 13 ms |
359
+ | Text recognition, in-process | **~174 ms** |
360
+ | Text recognition, via PNG + helper (fallback) | ~555 ms |
361
+ | Accessibility tree read, in-process | **~45 ms** |
362
+ | Accessibility tree read, via idb (fallback) | ~203 ms |
363
+ | Screen map: first visit / remembered | ~305 ms / **~1 ms** |
230
364
  | CPU | 1.1 % idle · 3.1 % active |
231
365
  | Frame memory | ~60 s of screen, ~2.7 MB |
232
366
 
367
+ Reproduce all of it with `npm run bench`, which prints the same table against
368
+ your machine. Full detail, including the measurement traps, is in
369
+ [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
370
+
233
371
  ## How it works
234
372
 
235
373
  ```
236
- ┌──────────────────── background, one loop per simulator ─────────────────────┐
237
- │ simctl screenshot ──► sips ──► decode ──► hash + diff ──► rename into │
238
- │ ~130 ms ~30 ms ~6 ms ~/.simframe/<udid>/ │
374
+ ┌─── simframed — one Swift daemon per simulator ──────────────────────────────┐
375
+ │ │
376
+ │ display damage callback ──► read IOSurface ──► scale ──► hash │
377
+ │ the screen tells us 0.13 ms 6.6 ms total │
378
+ │ │ │
379
+ │ Vision OCR reads the same surface ─────┤ no PNG, no file, no spawn │
380
+ │ │ │
381
+ │ Indigo HID ◄── control socket ◄────────┤ 0600, one JSON object per line │
382
+ │ taps, swipes, text │ │
383
+ │ ▼ │
384
+ │ ~/.simframe/<udid>/ frames · state.json · meta.json │
239
385
  └─────────────────────────────────────────────────────────────────────────────┘
240
386
  │ a rename is atomic
241
387
  ┌────────────────────────────────────▼────────────────────────────────────────┐
242
- │ MCP server / CLI: stat + read. No simctl anywhere in the request path. │
243
- │ Screen memory: layout hash ──► label → point, built once per screen. │
388
+ │ MCP server / CLI: stat + read, or one socket round trip for input. │
389
+ │ Screen memory: layout hash ──► label → point, built once per screen. │
244
390
  └─────────────────────────────────────────────────────────────────────────────┘
245
391
  ```
246
392
 
247
- - **Files are the IPC.** The loop renames completed frames into place; readers
248
- just read them. A rename is atomic, so a reader can never see a half-written
249
- frame, and there is no socket or protocol to get wrong.
250
- - **Almost no dependencies.** Resizing uses `sips`; PNG codec, hashing and frame
251
- comparison are plain JavaScript over `node:zlib`. The only runtime dependency
252
- is the MCP SDK. OCR is a ~60-line Swift file compiled on first use.
393
+ The daemon links CoreSimulator and SimulatorKit, which are private frameworks
394
+ with no documentation and no stability promise. Everything it calls is recorded
395
+ in [`docs/PRIVATE_API.md`](docs/PRIVATE_API.md) with the evidence behind it, so
396
+ an Xcode upgrade that moves something is a bounded fix rather than an
397
+ archaeology project. If a layer breaks, simframe degrades to the layer below
398
+ and `doctor` says which.
399
+
400
+ Run `simframe start --engine=simctl` to use the original loop instead.
401
+
402
+ - **Files are the IPC for reads.** The daemon renames completed frames into
403
+ place; readers just read them. A rename is atomic, so a reader can never see a
404
+ half-written frame. Input is the one thing that needs a reply, and it goes
405
+ over a `0600` Unix socket — the file system is the whole permission model.
406
+ - **Almost no dependencies.** The only runtime npm dependency is the MCP SDK.
407
+ The daemon is Swift built from source against frameworks already on the
408
+ machine.
409
+ - **The screen says when it changed.** The capture loop is driven by the
410
+ display's damage callback rather than a timer, so an idle screen costs
411
+ nothing and a moving one is picked up at once.
253
412
  - **It backs off when nothing happens.** 4 fps while the screen moves, 1.5 fps
254
413
  once still, snapping back instantly on change.
255
414
  - **One writer per device.** Ownership lives in `meta.json`; `stop` refuses to
@@ -263,24 +422,107 @@ iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
263
422
 
264
423
  ## CLI
265
424
 
425
+ The CLI is the low-token path, and it is a first-class one: `--json` is on every
426
+ command, so nothing has to be parsed out of prose.
427
+
266
428
  ```bash
267
- simframe start # start the capture loop
268
- simframe mark # hash of the current frame, for --since
429
+ simframe ui # the numbered screen map — start here
430
+ simframe ui --json | jq '.elements[] | select(.type=="button") | .label'
431
+ simframe do flow.json # a whole flow, verified, then the end-state map
432
+ simframe do flow.json --save=checkout # save it if every step verified
433
+ simframe flow run checkout # replay it
434
+ simframe tap "#3" # or "Save", or "@120,400"
435
+ simframe find "the save button" # resolve an intent without acting
436
+ simframe screens # screens this device has learned
437
+ simframe goto invoices # walk to a known screen over known steps
269
438
  simframe state --since=$H # what changed, as text
270
- simframe frame --out=now.png # newest frame
439
+ simframe mark # hash of the current frame, for --since
271
440
  simframe wait --since=$H # change, then settle
272
- simframe ui # labels + tap points (ax and ocr)
273
- simframe recall # what happened in the last minute
441
+ simframe recall # what happened in the last minute, as text
274
442
  simframe recall --ago=15000 # the frame from 15s ago
275
- simframe strip --count=6 # contact sheet
276
- simframe status / stop [--force] / devices / doctor
443
+ simframe frame --out=now.png # newest frame, native resolution, to a file
444
+ simframe strip --count=6 # contact sheet, for an animation
445
+ simframe doctor --strict # any degraded layer is a non-zero exit
446
+ simframe start / status / stop [--force] / devices
277
447
  ```
278
448
 
449
+ ### The Claude Code skill
450
+
451
+ [`skills/simframe/SKILL.md`](skills/simframe/SKILL.md) teaches the CLI path
452
+ directly: the cheap-to-expensive order, the selector grammar, what each verdict
453
+ means and what to do about it. It ships with the package, so an installed copy
454
+ has it.
455
+
456
+ ```bash
457
+ mkdir -p ~/.claude/skills
458
+ ln -s "$(npm root -g)/simframe/skills/simframe" ~/.claude/skills/simframe
459
+ ```
460
+
461
+ A skill and an MCP server are not redundant. The MCP server is discoverable —
462
+ it appears in the tool list without anybody setting it up. The skill is cheaper:
463
+ Claude reads 3–5 lines of CLI output instead of a tool result, and none of the
464
+ MCP schema is in context until a tool is actually used. Ship both, use whichever
465
+ the client makes easy.
466
+
467
+ ## Degrading is allowed. Degrading quietly is not
468
+
469
+ simframe is built to degrade rather than fail: no Swift toolchain still gives
470
+ you frames through `simctl`, no accessibility tree still gives you OCR. That
471
+ policy is right, and it nearly sank the tool twice — because a downgrade looked
472
+ exactly like everything working.
473
+
474
+ Once, one file was missing from the published package, so `Package.swift`
475
+ declared a test target with no directory, SwiftPM reported overlapping sources,
476
+ and **every install silently fell back to the slow engine**. Another time OCR
477
+ shipped disabled the same way. Both passed the tests. Both printed nothing. The
478
+ bug was never the missing file; it was the silence.
479
+
480
+ So every downgrade now announces itself:
481
+
482
+ - `simframe start` prints the engine it chose, and if it is the slow one, why —
483
+ build error, missing sources, or "reason unrecorded" if the daemon was started
484
+ by an earlier process.
485
+ - `simframe doctor` marks a degraded layer `WARN`, not `ok`, and summarises what
486
+ is degraded and what that costs.
487
+ - A dependency that is simply not installed is `--`, not `WARN`. The distinction
488
+ is deliberate: `WARN` means this machine could be doing better and silently is
489
+ not, which is the failure worth shouting about. An optional fallback missing on
490
+ a fresh machine has not degraded from anything, and `--strict` ignores it.
491
+ - `--strict`, or `SIMFRAME_STRICT=1`, turns any downgrade into a non-zero exit.
492
+ CI runs strict, so a release cannot ship in the state that shipped twice.
493
+
494
+ ```
495
+ $ simframe doctor
496
+ ok capture engine simframed
497
+ WARN input driver idb — the daemon's control socket is not up
498
+ WARN accessibility tree idb — the host-side translator did not load
499
+ ```
500
+
501
+ Two checks enforce it. A packaging check derives the required file list from the
502
+ build's own inputs — a hand-written list is what rotted last time — and runs in
503
+ seconds without a simulator. An integration job installs the packed tarball on a
504
+ real simulator and asserts `capture.engine`, `input.driver` and `ocr.available`
505
+ are all the good values, under `--strict`.
506
+
507
+ That last check found a real bug the day it was written: a daemon shutting down
508
+ unlinked the control socket unconditionally, so restarting deleted the *new*
509
+ daemon's socket. Capture kept working, input quietly dropped to idb, and nothing
510
+ said a word — the exact failure shape, found by the thing built to catch it.
511
+
279
512
  ## Limitations
280
513
 
281
- - Simulators only — `simctl` cannot capture a physical device.
282
- - Capture tops out near 6 fps, because `simctl io screenshot` costs ~130 ms. Fast
283
- animations are sampled, not recorded.
514
+ - Simulators only. Neither the framebuffer nor `simctl` can reach a physical
515
+ device.
516
+ - The daemon depends on private frameworks. They are stable enough to build on —
517
+ capture and accessibility survived the iOS 26 transition — but an Xcode
518
+ upgrade can move a symbol. `doctor` reports each layer separately so a break
519
+ is visible rather than mysterious, and `--engine=simctl` still works.
520
+ - Hardware buttons: only `home` is implemented. The other Indigo codes are
521
+ unverified, and a wrong one can crash `backboardd` or lock the device, so they
522
+ return an error rather than a guess.
523
+ - Typing sends key positions, which iOS maps through the device's active
524
+ keyboard layout. Text that must be exact goes through the pasteboard, which
525
+ `sim_do` does by default.
284
526
  - Region maps need a baseline inside the ~90 s history window. Older baselines
285
527
  still get a reliable changed / did-not-change, without a map of what moved.
286
528
  - Screen memory assumes a screen's layout is stable. A screen that reflows
@@ -290,28 +532,35 @@ simframe status / stop [--force] / devices / doctor
290
532
 
291
533
  ## Roadmap
292
534
 
293
- - **Verify-after-tap.** A tap can move the screen without doing what you meant —
294
- a swipe that animates but does not navigate still reports `changed`. Comparing
295
- against the expected destination would catch it.
296
- - **Reduce the input dependency.** idb is the one heavyweight requirement. Its
297
- simulator input is a reimplementation of the Indigo HID transport rather than a
298
- public API, so replacing it is real work, not a wrapper — but it is the last
299
- thing standing between simframe and a zero-install tool.
535
+ - **Android, as a second backend.** Everything above the platform boundary is
536
+ already platform-agnostic; nothing above it imports a simulator framework.
300
537
  - **Extend the confirm vocabulary beyond English.**
301
538
 
302
539
  ## Releasing
303
540
 
304
- `npm version` does not touch `server.json`, so bump both, then push the tag:
541
+ `npm version` runs a `version` hook that rewrites `server.json` to match and
542
+ stages it, so one command covers both files:
305
543
 
306
544
  ```bash
307
- npm version minor --no-git-tag-version
308
- $EDITOR server.json # match "version" and packages[0].version
309
- git commit -am "Release vX.Y.Z" && git tag vX.Y.Z && git push && git push --tags
545
+ npm version minor # bumps package.json + server.json, commits, tags
546
+ git push --follow-tags
310
547
  ```
311
548
 
549
+ Before that hook existed, `server.json` had to be hand-edited between two
550
+ commands, and the release that forgot failed at the workflow's own agreement
551
+ check — which is the one thing that check is for.
552
+
312
553
  The `release` workflow verifies tag/`package.json`/`server.json` agree, validates
313
554
  `server.json` against the live registry, and publishes to npm and the MCP
314
- Registry. It needs `NPM_TOKEN`; the registry uses GitHub OIDC and needs no secret.
555
+ Registry. It holds **no secrets** — both halves authenticate with the workflow's
556
+ GitHub OIDC identity.
557
+
558
+ That needs one setup step on npmjs.com, not in this repo: the package must have a
559
+ Trusted Publisher pointing at this repository and `release.yml` (Package →
560
+ Settings → Trusted Publisher → GitHub Actions). Without it npm has nothing to
561
+ trust and fails with `ENEEDAUTH`. npm is ending token publishing in January 2027,
562
+ and the tokens that work in CI need 2FA bypass, which npm's own UI warns against —
563
+ so OIDC is the durable path, not merely the tidier one.
315
564
 
316
565
  ## License
317
566
 
@@ -0,0 +1,16 @@
1
+ // swift-tools-version:5.9
2
+ import PackageDescription
3
+
4
+ let package = Package(
5
+ name: "simframed",
6
+ platforms: [.macOS(.v13)],
7
+ targets: [
8
+ // Every private-framework call lives here, behind a protocol, so a
9
+ // signature change is a one-file fix and tests can run without a device.
10
+ .target(name: "PrivateAPI"),
11
+ // Platform-independent: hashing, downscaling, the on-disk frame layout.
12
+ .target(name: "SimframeCore", dependencies: ["PrivateAPI"]),
13
+ .executableTarget(name: "simframed", dependencies: ["SimframeCore", "PrivateAPI"]),
14
+ .testTarget(name: "SimframeCoreTests", dependencies: ["SimframeCore"]),
15
+ ]
16
+ )