simframe 0.4.2 → 0.6.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +334 -85
- package/native/simframed/Package.swift +16 -0
- package/native/simframed/Sources/PrivateAPI/AccessibilityBridge.swift +379 -0
- package/native/simframed/Sources/PrivateAPI/CoreSimulatorPlatform.swift +523 -0
- package/native/simframed/Sources/PrivateAPI/HIDKeyboard.swift +70 -0
- package/native/simframed/Sources/PrivateAPI/IndigoHID.swift +121 -0
- package/native/simframed/Sources/PrivateAPI/PrivateAPI.swift +149 -0
- package/native/simframed/Sources/PrivateAPI/StubPlatform.swift +112 -0
- package/native/simframed/Sources/SimframeCore/Bitmap.swift +61 -0
- package/native/simframed/Sources/SimframeCore/ControlSocket.swift +122 -0
- package/native/simframed/Sources/SimframeCore/CoreGraphicsScaler.swift +70 -0
- package/native/simframed/Sources/SimframeCore/Element.swift +148 -0
- package/native/simframed/Sources/SimframeCore/FrameStore.swift +303 -0
- package/native/simframed/Sources/SimframeCore/Hashing.swift +119 -0
- package/native/simframed/Sources/SimframeCore/Motion.swift +431 -0
- package/native/simframed/Sources/SimframeCore/PNGWriter.swift +40 -0
- package/native/simframed/Sources/SimframeCore/VisionOCR.swift +75 -0
- package/native/simframed/Sources/simframed/main.swift +485 -0
- package/native/simframed/Tests/SimframeCoreTests/HashingTests.swift +270 -0
- package/package.json +12 -4
- package/scripts/bench-flow.mjs +54 -0
- package/scripts/bench.sh +98 -0
- package/scripts/check-package.mjs +99 -0
- package/scripts/ci-memory.mjs +416 -0
- package/scripts/eval-fingerprint.mjs +192 -0
- package/scripts/smoke.mjs +76 -0
- package/scripts/sync-server-version.mjs +39 -0
- package/scripts/verify-baseline.mjs +65 -0
- package/skills/simframe/SKILL.md +173 -0
- package/src/actions.js +264 -18
- package/src/cli.js +561 -89
- package/src/control.js +77 -0
- package/src/daemon.js +8 -1
- package/src/engine.js +99 -0
- package/src/fingerprint.js +183 -0
- package/src/graph.js +411 -0
- package/src/index.js +351 -24
- package/src/input.js +179 -2
- package/src/matching.js +265 -0
- package/src/mcp.js +425 -112
- package/src/navigate.js +120 -0
- package/src/refs.js +141 -0
- package/src/regions.js +267 -0
- package/src/screenmap.js +119 -22
- package/src/simctl.js +74 -5
- package/src/store.js +8 -0
- package/src/view.js +342 -0
package/README.md
CHANGED
|
@@ -20,9 +20,15 @@ one is obvious:
|
|
|
20
20
|
3. **Nothing is remembered.** The same screen gets re-read and re-reasoned about
|
|
21
21
|
every single time it appears.
|
|
22
22
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
23
|
+
And there is a fourth that is pure waste: **an image is the most expensive way
|
|
24
|
+
to ask what is on screen.** A screenshot costs ~1,600 tokens when it is handled
|
|
25
|
+
as a native image block and 15,000–25,000 when it is not, and it does not tell
|
|
26
|
+
you what is tappable or where — you have to measure that by eye.
|
|
27
|
+
|
|
28
|
+
simframe attacks all four: a background loop keeps the newest frame warm, whole
|
|
29
|
+
flows run in one call, screens the agent has seen before are answered from
|
|
30
|
+
memory, and every answer is text with tap points in it. Nothing returns an
|
|
31
|
+
image unless you ask for one.
|
|
26
32
|
|
|
27
33
|
## What changed, measured
|
|
28
34
|
|
|
@@ -32,13 +38,27 @@ Same four-tab navigation flow, on a real production app:
|
|
|
32
38
|
| --- | --- | --- |
|
|
33
39
|
| Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
|
|
34
40
|
| "Did anything change?" | a full image | **~2 ms**, text only |
|
|
35
|
-
| A 5-step flow | 5+ model round trips | **1 call**, ~7 s |
|
|
36
41
|
| Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
|
|
37
|
-
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
+
| A 4-step flow, verified | 4+ model round trips | **1 call**, 3.6 s |
|
|
43
|
+
| Same flow, 3rd run | no improvement — every run is the first | **3.7 s, 4/4 from memory, 4/4 verified** |
|
|
44
|
+
| A 10-step flow | 10 turns, 10 images (~16,000 tokens at best) | **1 turn, 0 images, ~1,650 characters** |
|
|
45
|
+
| Reading a screen | an image: ~1,600 tokens, no tap points | **~330 tokens** of text, with tap points |
|
|
46
|
+
|
|
47
|
+
The four-tab tour, three times back to back from a cleared memory:
|
|
48
|
+
|
|
49
|
+
| Pass | Wall clock | Steps verified | Controls from memory |
|
|
50
|
+
| --- | --- | --- | --- |
|
|
51
|
+
| 1 | 10.2 s | 0/4 — nothing is known yet | **4/4** |
|
|
52
|
+
| 2 | **3.6 s** | **4/4** | **4/4** |
|
|
53
|
+
| 3 | **3.7 s** | **4/4** | **4/4** |
|
|
54
|
+
|
|
55
|
+
Every step is checked against what the same action did last time, and the run
|
|
56
|
+
records its own preconditions — which input path, which daemon, whether the
|
|
57
|
+
daemon was replaced mid-run — so a regression shows up in the measurement rather
|
|
58
|
+
than hiding inside it. Earlier versions of this table quoted 7.4 s → 3.3 s with
|
|
59
|
+
verification switched off; those numbers were measured while input was silently
|
|
60
|
+
falling back to a slower path and the capture daemon was being replaced by every
|
|
61
|
+
command, so they measured two bugs rather than the tool.
|
|
42
62
|
|
|
43
63
|
## Install
|
|
44
64
|
|
|
@@ -47,12 +67,18 @@ npm install -g simframe
|
|
|
47
67
|
simframe doctor
|
|
48
68
|
```
|
|
49
69
|
|
|
70
|
+
The first `simframe start` builds a small Swift daemon from source — a few
|
|
71
|
+
seconds, once. It needs the Xcode command line tools, which you already have if
|
|
72
|
+
you have a simulator. Without them simframe falls back to the original
|
|
73
|
+
`simctl` loop and says so.
|
|
74
|
+
|
|
50
75
|
`doctor` checks each capability separately and tells you what you have:
|
|
51
76
|
|
|
52
77
|
```
|
|
53
78
|
ok xcrun xcrun version 72.
|
|
54
79
|
ok sips available
|
|
55
|
-
ok input driver
|
|
80
|
+
ok input driver simframed: Indigo HID
|
|
81
|
+
ok accessibility tree simframed: AXPTranslator, host-side
|
|
56
82
|
ok on-device OCR available
|
|
57
83
|
ok booted simulator iPhone 17 Pro (iOS 26.5)
|
|
58
84
|
ok capture frame #888 322x700 in 2ms (age 538ms)
|
|
@@ -86,28 +112,73 @@ have. **Observation needs nothing but Xcode.**
|
|
|
86
112
|
| --- | --- | --- |
|
|
87
113
|
| Watch the screen, wait, recall | nothing extra | — |
|
|
88
114
|
| Read labels + coordinates from pixels | `swiftc` (Xcode CLT) | falls back to the accessibility tree alone |
|
|
89
|
-
| Tap, type, swipe |
|
|
115
|
+
| Tap, type, swipe | nothing extra | simframe observes but cannot touch |
|
|
116
|
+
| Accessibility tree | nothing extra | OCR alone still yields labels and coordinates |
|
|
117
|
+
|
|
118
|
+
`simframe doctor` names which engine is carrying each capability, per device.
|
|
119
|
+
**Nothing beyond Xcode is required.** Capture, input, text recognition and the
|
|
120
|
+
accessibility tree all run in-process, in one daemon.
|
|
121
|
+
|
|
122
|
+
[`idb`](https://fbidb.io) is still accepted as a fallback for input and for the
|
|
123
|
+
tree, for a machine where the daemon cannot run — and `SIMFRAME_AX_DRIVER=idb`
|
|
124
|
+
forces the tree back onto it, which is the escape hatch if an Xcode upgrade
|
|
125
|
+
breaks the host-side path.
|
|
90
126
|
|
|
91
127
|
```bash
|
|
92
|
-
#
|
|
128
|
+
# optional fallback, not a requirement
|
|
93
129
|
brew tap facebook/fb && brew install idb-companion && pipx install fb-idb
|
|
94
130
|
```
|
|
95
131
|
|
|
96
|
-
Homebrew may ask you to trust the tap first; that is a deliberate prompt for a
|
|
97
|
-
human, and the narrow form is `brew trust --formula facebook/fb/idb-companion`.
|
|
98
|
-
|
|
99
132
|
## The tools
|
|
100
133
|
|
|
134
|
+
Read first, act in batches, and look at pixels only when the question is about
|
|
135
|
+
pixels. Every tool description says so, because a tool surface that does not
|
|
136
|
+
steer the model is a tool surface the model uses wrong.
|
|
137
|
+
|
|
101
138
|
| Tool | What it does |
|
|
102
139
|
| --- | --- |
|
|
103
|
-
| `
|
|
104
|
-
| `
|
|
140
|
+
| `sim_ui` | **Start here.** The screen as a numbered text map: region, type, label, state, tap point, source. A tenth the cost of a screenshot and strictly more useful. |
|
|
141
|
+
| `sim_do` | **The main tool.** A whole flow in one call — tap, type, scroll, wait, assert — each step settling before the next and verified against what it did last time. |
|
|
142
|
+
| `sim_state` | The cheapest question there is: has anything changed **since your last look**, and which regions moved. |
|
|
143
|
+
| `sim_goto` | Walk to a screen simframe has been to before, planning the route through remembered transitions. |
|
|
144
|
+
| `sim_flow_run` | Replay a flow that verified end to end. |
|
|
145
|
+
| `sim_find` | Resolve an intent to one control, without acting on it. |
|
|
146
|
+
| `sim_tap` · `sim_type_into` · `sim_scroll_to` · `sim_wait_for` · `sim_assert` | Single actions, for when you genuinely only have one step. Each is one `sim_do` step underneath. |
|
|
147
|
+
| `sim_launch` · `sim_open_url` · `sim_permission` | Launch with arguments and environment; open a deep link; grant a privacy permission instead of tapping a system alert. |
|
|
105
148
|
| `sim_wait` | Waits for the screen to change *and then* settle. |
|
|
106
|
-
| `
|
|
107
|
-
| `
|
|
108
|
-
| `
|
|
109
|
-
|
|
110
|
-
|
|
149
|
+
| `sim_look` | **The only tool that returns an image**, capped at 1024 px. For layout, colour, spacing — questions text cannot answer. |
|
|
150
|
+
| `sim_recall` · `sim_strip` | Look backwards: a text timeline of what happened, or recent frames tiled into one image. |
|
|
151
|
+
| `sim_capture` · `sim_devices` | Manage capture loops; list simulators. |
|
|
152
|
+
|
|
153
|
+
### What the screen looks like as text
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
iPhone 17 Pro · 402x874pt · screen a1b2c3d4 "Inbox" (known, 3 known exits)
|
|
157
|
+
last action: [2] tap — ok: matches the outcome seen 5x before
|
|
158
|
+
nav-bar:
|
|
159
|
+
#1 button 24,64 Back
|
|
160
|
+
#2 text 201,64 Inbox
|
|
161
|
+
content:
|
|
162
|
+
#3 cell 201,140 Weekly digest
|
|
163
|
+
#4 cell 201,196 Payment received
|
|
164
|
+
tab-bar:
|
|
165
|
+
#5 text 62,835 Inbox
|
|
166
|
+
#6 text 201,835 Settings
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Region first, because "Inbox" the title and "Inbox" the tab differ only by where
|
|
170
|
+
they are. A tap point, because that is what an action needs. And a number, which
|
|
171
|
+
is a selector: whatever this calls `#3`, the next call can tap as `#3` without
|
|
172
|
+
describing it. A ref is valid only while that screen is showing — used on a
|
|
173
|
+
different screen it refuses rather than tapping whatever now sits there.
|
|
174
|
+
|
|
175
|
+
Three ways to name a control, anywhere one is named:
|
|
176
|
+
|
|
177
|
+
| | |
|
|
178
|
+
| --- | --- |
|
|
179
|
+
| `#3` | the number the map gave it. Cheapest, unambiguous. |
|
|
180
|
+
| `"Save"` · `the Assets tab` · `back` | resolved by intent — verbs, typos, synonyms, icon-only controls by their common name |
|
|
181
|
+
| `@120,400` | raw point coordinates. Last resort: it cannot tell you it missed. |
|
|
111
182
|
|
|
112
183
|
## Baselines: the thing to understand
|
|
113
184
|
|
|
@@ -144,7 +215,8 @@ So simframe reads the screen two ways and remembers the result:
|
|
|
144
215
|
- **Accessibility** gives real hit targets, types and enabled state.
|
|
145
216
|
- **On-device OCR** (Apple's Vision, ~290 ms, no model round trip) gives every
|
|
146
217
|
label a person can actually see, with coordinates.
|
|
147
|
-
- The merge is keyed by a **
|
|
218
|
+
- The merge is keyed by a **structural fingerprint**, so the next visit is a
|
|
219
|
+
file read. What that fingerprint is, and why it is not a pixel hash, is below.
|
|
148
220
|
|
|
149
221
|
```
|
|
150
222
|
first visit to a screen ~1000 ms read tree + OCR, store the map
|
|
@@ -164,34 +236,92 @@ Two details that matter:
|
|
|
164
236
|
a tab returns an error listing both with coordinates, because silently tapping
|
|
165
237
|
the title looks exactly like nothing happening.
|
|
166
238
|
|
|
167
|
-
###
|
|
239
|
+
### Two hashes, because there are two questions
|
|
240
|
+
|
|
241
|
+
"Did this move?" and "is this the same screen?" look like one question and are
|
|
242
|
+
not. simframe answers them separately, and getting that wrong was the single
|
|
243
|
+
most expensive mistake in its development.
|
|
168
244
|
|
|
245
|
+
**Change and settle** are questions about pixels, so a pixel hash answers them.
|
|
169
246
|
The frame hash changes whenever any pixel group changes — a clock digit, one new
|
|
170
|
-
row
|
|
171
|
-
|
|
172
|
-
12×24 grid.
|
|
247
|
+
row — which is exactly right for "did anything happen?" and useless as a key for
|
|
248
|
+
"have I been here before?". For change detection there is a layout hash: status
|
|
249
|
+
bar cropped, difference hash over a 12×24 grid.
|
|
173
250
|
|
|
174
251
|
A mean-threshold hash was tried first and was actively dangerous: low-contrast
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
252
|
+
screens collapsed onto identical values, so unrelated screens matched at distance
|
|
253
|
+
0 and taps landed on the wrong control. The difference hash fixed that.
|
|
254
|
+
|
|
255
|
+
**Identity is not a question about pixels**, and this is the part that took three
|
|
256
|
+
attempts. Content *is* pixels: a list whose rows changed drifts as far as a
|
|
257
|
+
different screen does. Measured on a real app, same-screen revisits reached 62
|
|
258
|
+
bits against a different-screen floor of 74 — overlapping, with no threshold
|
|
259
|
+
available to choose. An earlier calibration had suggested a comfortable margin
|
|
260
|
+
(0–4 against 77–113), but it was measured on screens whose content happened to be
|
|
261
|
+
stable and did not survive contact with a real list.
|
|
262
|
+
|
|
263
|
+
So identity is **structural**. The fingerprint is built from element roles,
|
|
264
|
+
frames quantised to a 24 px grid, the region each element sits in, and repeated
|
|
265
|
+
siblings bucketed as "one" or "many" rather than counted. Deliberately included:
|
|
266
|
+
the labels of chrome elements only — nav title, tab labels, toolbar buttons —
|
|
267
|
+
because two list screens with identical structure are told apart by their title
|
|
268
|
+
and nothing else. Deliberately excluded: all content text and values, the status
|
|
269
|
+
bar, and the keyboard region when a keyboard is up.
|
|
270
|
+
|
|
271
|
+
It does not depend on the accessibility tree. Fingerprinting from OCR boxes
|
|
272
|
+
alone, with the tree discarded entirely, still separates screens — different
|
|
273
|
+
screens ceiling 0.35 against the same threshold.
|
|
274
|
+
|
|
275
|
+
| | Jaccard similarity |
|
|
179
276
|
| --- | --- |
|
|
180
|
-
| Same screen, revisited
|
|
181
|
-
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
277
|
+
| Same screen, revisited | 0.41–1.00 |
|
|
278
|
+
| **Different screens** | **0.00–0.31** |
|
|
279
|
+
|
|
280
|
+
The threshold sits in that gap, but the gap is narrower than anyone would want,
|
|
281
|
+
and one screen causes it: a screen whose sections load from different sources has
|
|
282
|
+
more than one genuine settled structure, and two structures of one screen are as
|
|
283
|
+
far apart as two different screens.
|
|
284
|
+
|
|
285
|
+
No threshold can express that, so a screen may hold **several** accepted
|
|
286
|
+
fingerprints instead. A new one is admitted only when a known edge lands
|
|
287
|
+
somewhere its target does not recognise — the edge is the evidence that it is the
|
|
288
|
+
same place — and only if no other stored screen claims that reading. Identity
|
|
289
|
+
stays exact rather than being loosened, and the count is capped, so a
|
|
290
|
+
non-deterministic action shows up as a node collecting variants rather than as
|
|
291
|
+
screens silently merging.
|
|
292
|
+
|
|
293
|
+
It earns its place on real apps: an app reconnecting to its bundler put an alert
|
|
294
|
+
over one screen, and that screen gained a variant instead of a duplicate
|
|
295
|
+
appearing.
|
|
296
|
+
|
|
297
|
+
Failing to recognise a screen you have seen is harmless — it rebuilds the map and
|
|
298
|
+
taps correctly. Matching the *wrong* screen taps the wrong control. The threshold
|
|
299
|
+
is set to err toward the first.
|
|
300
|
+
|
|
301
|
+
## Navigating by memory
|
|
302
|
+
|
|
303
|
+
Once simframe knows which screens exist and which action leads from one to the
|
|
304
|
+
next, getting somewhere is a search over known edges rather than a question for a
|
|
305
|
+
model:
|
|
306
|
+
|
|
307
|
+
```bash
|
|
308
|
+
simframe screens # what this device has learned
|
|
309
|
+
simframe goto invoices # walk there, verifying every step
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Measured on a four-tab tour, `goto` plans and walks three-step routes with every
|
|
313
|
+
step verified and no model call. It fails rather than guesses: an unknown
|
|
314
|
+
destination, a query matching two screens equally, or no path of known edges all
|
|
315
|
+
report themselves instead of tapping hopefully.
|
|
316
|
+
|
|
317
|
+
Flows work the same way and refuse to save if any step went unverified —
|
|
318
|
+
replaying a recording of something that may not have worked just reproduces the
|
|
319
|
+
doubt.
|
|
320
|
+
|
|
321
|
+
```bash
|
|
322
|
+
simframe flow save checkout ./checkout.json
|
|
323
|
+
simframe flow run checkout
|
|
324
|
+
```
|
|
195
325
|
|
|
196
326
|
## Does this work on *your* app?
|
|
197
327
|
|
|
@@ -219,37 +349,66 @@ iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
|
|
|
219
349
|
|
|
220
350
|
| | |
|
|
221
351
|
| --- | --- |
|
|
352
|
+
| Frame capture, whole pipeline | **6.6 ms** |
|
|
353
|
+
| Frame grab alone | **0.13 ms** |
|
|
354
|
+
| The `simctl` + `sips` path it replaces | ~210 ms |
|
|
222
355
|
| Warm frame read (`sim_look`) | ~20 ms |
|
|
223
356
|
| State check (`sim_state`) | ~2 ms |
|
|
224
|
-
|
|
|
225
|
-
|
|
|
226
|
-
|
|
|
227
|
-
|
|
|
228
|
-
|
|
|
229
|
-
|
|
|
357
|
+
| Input round trip (`ping`) | **0 ms** |
|
|
358
|
+
| Tap (70 ms hold / 10 ms hold) | 76 ms / 13 ms |
|
|
359
|
+
| Text recognition, in-process | **~174 ms** |
|
|
360
|
+
| Text recognition, via PNG + helper (fallback) | ~555 ms |
|
|
361
|
+
| Accessibility tree read, in-process | **~45 ms** |
|
|
362
|
+
| Accessibility tree read, via idb (fallback) | ~203 ms |
|
|
363
|
+
| Screen map: first visit / remembered | ~305 ms / **~1 ms** |
|
|
230
364
|
| CPU | 1.1 % idle · 3.1 % active |
|
|
231
365
|
| Frame memory | ~60 s of screen, ~2.7 MB |
|
|
232
366
|
|
|
367
|
+
Reproduce all of it with `npm run bench`, which prints the same table against
|
|
368
|
+
your machine. Full detail, including the measurement traps, is in
|
|
369
|
+
[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
|
|
370
|
+
|
|
233
371
|
## How it works
|
|
234
372
|
|
|
235
373
|
```
|
|
236
|
-
|
|
237
|
-
│
|
|
238
|
-
│
|
|
374
|
+
┌─── simframed — one Swift daemon per simulator ──────────────────────────────┐
|
|
375
|
+
│ │
|
|
376
|
+
│ display damage callback ──► read IOSurface ──► scale ──► hash │
|
|
377
|
+
│ the screen tells us 0.13 ms 6.6 ms total │
|
|
378
|
+
│ │ │
|
|
379
|
+
│ Vision OCR reads the same surface ─────┤ no PNG, no file, no spawn │
|
|
380
|
+
│ │ │
|
|
381
|
+
│ Indigo HID ◄── control socket ◄────────┤ 0600, one JSON object per line │
|
|
382
|
+
│ taps, swipes, text │ │
|
|
383
|
+
│ ▼ │
|
|
384
|
+
│ ~/.simframe/<udid>/ frames · state.json · meta.json │
|
|
239
385
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
240
386
|
│ a rename is atomic
|
|
241
387
|
┌────────────────────────────────────▼────────────────────────────────────────┐
|
|
242
|
-
│ MCP server / CLI: stat + read
|
|
243
|
-
│ Screen memory: layout hash ──► label → point, built once per screen.
|
|
388
|
+
│ MCP server / CLI: stat + read, or one socket round trip for input. │
|
|
389
|
+
│ Screen memory: layout hash ──► label → point, built once per screen. │
|
|
244
390
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
245
391
|
```
|
|
246
392
|
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
393
|
+
The daemon links CoreSimulator and SimulatorKit, which are private frameworks
|
|
394
|
+
with no documentation and no stability promise. Everything it calls is recorded
|
|
395
|
+
in [`docs/PRIVATE_API.md`](docs/PRIVATE_API.md) with the evidence behind it, so
|
|
396
|
+
an Xcode upgrade that moves something is a bounded fix rather than an
|
|
397
|
+
archaeology project. If a layer breaks, simframe degrades to the layer below
|
|
398
|
+
and `doctor` says which.
|
|
399
|
+
|
|
400
|
+
Run `simframe start --engine=simctl` to use the original loop instead.
|
|
401
|
+
|
|
402
|
+
- **Files are the IPC for reads.** The daemon renames completed frames into
|
|
403
|
+
place; readers just read them. A rename is atomic, so a reader can never see a
|
|
404
|
+
half-written frame. Input is the one thing that needs a reply, and it goes
|
|
405
|
+
over a `0600` Unix socket — the file system is the whole permission model.
|
|
406
|
+
- **Almost no dependencies.** The only runtime npm dependency is the MCP SDK.
|
|
407
|
+
The daemon is Swift built from source against frameworks already on the
|
|
408
|
+
machine.
|
|
409
|
+
- **The screen says when it changed.** The capture loop is driven by the
|
|
410
|
+
display's damage callback rather than a timer, so an idle screen costs
|
|
411
|
+
nothing and a moving one is picked up at once.
|
|
253
412
|
- **It backs off when nothing happens.** 4 fps while the screen moves, 1.5 fps
|
|
254
413
|
once still, snapping back instantly on change.
|
|
255
414
|
- **One writer per device.** Ownership lives in `meta.json`; `stop` refuses to
|
|
@@ -263,24 +422,107 @@ iPhone 17 Pro, iOS 26.5, Apple Silicon, default settings.
|
|
|
263
422
|
|
|
264
423
|
## CLI
|
|
265
424
|
|
|
425
|
+
The CLI is the low-token path, and it is a first-class one: `--json` is on every
|
|
426
|
+
command, so nothing has to be parsed out of prose.
|
|
427
|
+
|
|
266
428
|
```bash
|
|
267
|
-
simframe
|
|
268
|
-
simframe
|
|
429
|
+
simframe ui # the numbered screen map — start here
|
|
430
|
+
simframe ui --json | jq '.elements[] | select(.type=="button") | .label'
|
|
431
|
+
simframe do flow.json # a whole flow, verified, then the end-state map
|
|
432
|
+
simframe do flow.json --save=checkout # save it if every step verified
|
|
433
|
+
simframe flow run checkout # replay it
|
|
434
|
+
simframe tap "#3" # or "Save", or "@120,400"
|
|
435
|
+
simframe find "the save button" # resolve an intent without acting
|
|
436
|
+
simframe screens # screens this device has learned
|
|
437
|
+
simframe goto invoices # walk to a known screen over known steps
|
|
269
438
|
simframe state --since=$H # what changed, as text
|
|
270
|
-
simframe
|
|
439
|
+
simframe mark # hash of the current frame, for --since
|
|
271
440
|
simframe wait --since=$H # change, then settle
|
|
272
|
-
simframe
|
|
273
|
-
simframe recall # what happened in the last minute
|
|
441
|
+
simframe recall # what happened in the last minute, as text
|
|
274
442
|
simframe recall --ago=15000 # the frame from 15s ago
|
|
275
|
-
simframe
|
|
276
|
-
simframe
|
|
443
|
+
simframe frame --out=now.png # newest frame, native resolution, to a file
|
|
444
|
+
simframe strip --count=6 # contact sheet, for an animation
|
|
445
|
+
simframe doctor --strict # any degraded layer is a non-zero exit
|
|
446
|
+
simframe start / status / stop [--force] / devices
|
|
277
447
|
```
|
|
278
448
|
|
|
449
|
+
### The Claude Code skill
|
|
450
|
+
|
|
451
|
+
[`skills/simframe/SKILL.md`](skills/simframe/SKILL.md) teaches the CLI path
|
|
452
|
+
directly: the cheap-to-expensive order, the selector grammar, what each verdict
|
|
453
|
+
means and what to do about it. It ships with the package, so an installed copy
|
|
454
|
+
has it.
|
|
455
|
+
|
|
456
|
+
```bash
|
|
457
|
+
mkdir -p ~/.claude/skills
|
|
458
|
+
ln -s "$(npm root -g)/simframe/skills/simframe" ~/.claude/skills/simframe
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
A skill and an MCP server are not redundant. The MCP server is discoverable —
|
|
462
|
+
it appears in the tool list without anybody setting it up. The skill is cheaper:
|
|
463
|
+
Claude reads 3–5 lines of CLI output instead of a tool result, and none of the
|
|
464
|
+
MCP schema is in context until a tool is actually used. Ship both, use whichever
|
|
465
|
+
the client makes easy.
|
|
466
|
+
|
|
467
|
+
## Degrading is allowed. Degrading quietly is not
|
|
468
|
+
|
|
469
|
+
simframe is built to degrade rather than fail: no Swift toolchain still gives
|
|
470
|
+
you frames through `simctl`, no accessibility tree still gives you OCR. That
|
|
471
|
+
policy is right, and it nearly sank the tool twice — because a downgrade looked
|
|
472
|
+
exactly like everything working.
|
|
473
|
+
|
|
474
|
+
Once, one file was missing from the published package, so `Package.swift`
|
|
475
|
+
declared a test target with no directory, SwiftPM reported overlapping sources,
|
|
476
|
+
and **every install silently fell back to the slow engine**. Another time OCR
|
|
477
|
+
shipped disabled the same way. Both passed the tests. Both printed nothing. The
|
|
478
|
+
bug was never the missing file; it was the silence.
|
|
479
|
+
|
|
480
|
+
So every downgrade now announces itself:
|
|
481
|
+
|
|
482
|
+
- `simframe start` prints the engine it chose, and if it is the slow one, why —
|
|
483
|
+
build error, missing sources, or "reason unrecorded" if the daemon was started
|
|
484
|
+
by an earlier process.
|
|
485
|
+
- `simframe doctor` marks a degraded layer `WARN`, not `ok`, and summarises what
|
|
486
|
+
is degraded and what that costs.
|
|
487
|
+
- A dependency that is simply not installed is `--`, not `WARN`. The distinction
|
|
488
|
+
is deliberate: `WARN` means this machine could be doing better and silently is
|
|
489
|
+
not, which is the failure worth shouting about. An optional fallback missing on
|
|
490
|
+
a fresh machine has not degraded from anything, and `--strict` ignores it.
|
|
491
|
+
- `--strict`, or `SIMFRAME_STRICT=1`, turns any downgrade into a non-zero exit.
|
|
492
|
+
CI runs strict, so a release cannot ship in the state that shipped twice.
|
|
493
|
+
|
|
494
|
+
```
|
|
495
|
+
$ simframe doctor
|
|
496
|
+
ok capture engine simframed
|
|
497
|
+
WARN input driver idb — the daemon's control socket is not up
|
|
498
|
+
WARN accessibility tree idb — the host-side translator did not load
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
Two checks enforce it. A packaging check derives the required file list from the
|
|
502
|
+
build's own inputs — a hand-written list is what rotted last time — and runs in
|
|
503
|
+
seconds without a simulator. An integration job installs the packed tarball on a
|
|
504
|
+
real simulator and asserts `capture.engine`, `input.driver` and `ocr.available`
|
|
505
|
+
are all the good values, under `--strict`.
|
|
506
|
+
|
|
507
|
+
That last check found a real bug the day it was written: a daemon shutting down
|
|
508
|
+
unlinked the control socket unconditionally, so restarting deleted the *new*
|
|
509
|
+
daemon's socket. Capture kept working, input quietly dropped to idb, and nothing
|
|
510
|
+
said a word — the exact failure shape, found by the thing built to catch it.
|
|
511
|
+
|
|
279
512
|
## Limitations
|
|
280
513
|
|
|
281
|
-
- Simulators only
|
|
282
|
-
|
|
283
|
-
|
|
514
|
+
- Simulators only. Neither the framebuffer nor `simctl` can reach a physical
|
|
515
|
+
device.
|
|
516
|
+
- The daemon depends on private frameworks. They are stable enough to build on —
|
|
517
|
+
capture and accessibility survived the iOS 26 transition — but an Xcode
|
|
518
|
+
upgrade can move a symbol. `doctor` reports each layer separately so a break
|
|
519
|
+
is visible rather than mysterious, and `--engine=simctl` still works.
|
|
520
|
+
- Hardware buttons: only `home` is implemented. The other Indigo codes are
|
|
521
|
+
unverified, and a wrong one can crash `backboardd` or lock the device, so they
|
|
522
|
+
return an error rather than a guess.
|
|
523
|
+
- Typing sends key positions, which iOS maps through the device's active
|
|
524
|
+
keyboard layout. Text that must be exact goes through the pasteboard, which
|
|
525
|
+
`sim_do` does by default.
|
|
284
526
|
- Region maps need a baseline inside the ~90 s history window. Older baselines
|
|
285
527
|
still get a reliable changed / did-not-change, without a map of what moved.
|
|
286
528
|
- Screen memory assumes a screen's layout is stable. A screen that reflows
|
|
@@ -290,28 +532,35 @@ simframe status / stop [--force] / devices / doctor
|
|
|
290
532
|
|
|
291
533
|
## Roadmap
|
|
292
534
|
|
|
293
|
-
- **
|
|
294
|
-
|
|
295
|
-
against the expected destination would catch it.
|
|
296
|
-
- **Reduce the input dependency.** idb is the one heavyweight requirement. Its
|
|
297
|
-
simulator input is a reimplementation of the Indigo HID transport rather than a
|
|
298
|
-
public API, so replacing it is real work, not a wrapper — but it is the last
|
|
299
|
-
thing standing between simframe and a zero-install tool.
|
|
535
|
+
- **Android, as a second backend.** Everything above the platform boundary is
|
|
536
|
+
already platform-agnostic; nothing above it imports a simulator framework.
|
|
300
537
|
- **Extend the confirm vocabulary beyond English.**
|
|
301
538
|
|
|
302
539
|
## Releasing
|
|
303
540
|
|
|
304
|
-
`npm version`
|
|
541
|
+
`npm version` runs a `version` hook that rewrites `server.json` to match and
|
|
542
|
+
stages it, so one command covers both files:
|
|
305
543
|
|
|
306
544
|
```bash
|
|
307
|
-
npm version minor
|
|
308
|
-
|
|
309
|
-
git commit -am "Release vX.Y.Z" && git tag vX.Y.Z && git push && git push --tags
|
|
545
|
+
npm version minor # bumps package.json + server.json, commits, tags
|
|
546
|
+
git push --follow-tags
|
|
310
547
|
```
|
|
311
548
|
|
|
549
|
+
Before that hook existed, `server.json` had to be hand-edited between two
|
|
550
|
+
commands, and the release that forgot failed at the workflow's own agreement
|
|
551
|
+
check — which is the one thing that check is for.
|
|
552
|
+
|
|
312
553
|
The `release` workflow verifies tag/`package.json`/`server.json` agree, validates
|
|
313
554
|
`server.json` against the live registry, and publishes to npm and the MCP
|
|
314
|
-
Registry. It
|
|
555
|
+
Registry. It holds **no secrets** — both halves authenticate with the workflow's
|
|
556
|
+
GitHub OIDC identity.
|
|
557
|
+
|
|
558
|
+
That needs one setup step on npmjs.com, not in this repo: the package must have a
|
|
559
|
+
Trusted Publisher pointing at this repository and `release.yml` (Package →
|
|
560
|
+
Settings → Trusted Publisher → GitHub Actions). Without it npm has nothing to
|
|
561
|
+
trust and fails with `ENEEDAUTH`. npm is ending token publishing in January 2027,
|
|
562
|
+
and the tokens that work in CI need 2FA bypass, which npm's own UI warns against —
|
|
563
|
+
so OIDC is the durable path, not merely the tidier one.
|
|
315
564
|
|
|
316
565
|
## License
|
|
317
566
|
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
// swift-tools-version:5.9
|
|
2
|
+
import PackageDescription
|
|
3
|
+
|
|
4
|
+
let package = Package(
|
|
5
|
+
name: "simframed",
|
|
6
|
+
platforms: [.macOS(.v13)],
|
|
7
|
+
targets: [
|
|
8
|
+
// Every private-framework call lives here, behind a protocol, so a
|
|
9
|
+
// signature change is a one-file fix and tests can run without a device.
|
|
10
|
+
.target(name: "PrivateAPI"),
|
|
11
|
+
// Platform-independent: hashing, downscaling, the on-disk frame layout.
|
|
12
|
+
.target(name: "SimframeCore", dependencies: ["PrivateAPI"]),
|
|
13
|
+
.executableTarget(name: "simframed", dependencies: ["SimframeCore", "PrivateAPI"]),
|
|
14
|
+
.testTarget(name: "SimframeCoreTests", dependencies: ["SimframeCore"]),
|
|
15
|
+
]
|
|
16
|
+
)
|