simframe 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -7
- package/package.json +4 -2
- package/scripts/check-package.mjs +15 -0
- package/scripts/ci-memory.mjs +15 -2
- package/src/actions.js +41 -22
- package/src/cli.js +69 -7
- package/src/daemon.js +6 -0
- package/src/index.js +11 -1
- package/src/input.js +42 -1
- package/src/navigate.js +8 -1
- package/src/platform/android.js +47 -6
- package/src/platform/index.js +21 -6
- package/src/platform/ios.js +23 -0
package/README.md
CHANGED
|
@@ -4,11 +4,11 @@
|
|
|
4
4
|
[](https://www.npmjs.com/package/simframe)
|
|
5
5
|
[](./LICENSE)
|
|
6
6
|
|
|
7
|
-
**Eyes, hands and memory for an agent driving the iOS Simulator.**
|
|
7
|
+
**Eyes, hands and memory for an agent driving the iOS Simulator or an Android emulator.**
|
|
8
8
|
|
|
9
9
|
[Website](https://lvlrsajjad.github.io/simframe/) · [npm](https://www.npmjs.com/package/simframe)
|
|
10
10
|
|
|
11
|
-
An agent driving
|
|
11
|
+
An agent driving a simulator is slow for three reasons, and only the first
|
|
12
12
|
one is obvious:
|
|
13
13
|
|
|
14
14
|
1. **Every look is a wait.** `simctl io screenshot` costs ~130 ms of blocking
|
|
@@ -37,6 +37,7 @@ Same four-tab navigation flow, on a real production app:
|
|
|
37
37
|
| | Before | With simframe |
|
|
38
38
|
| --- | --- | --- |
|
|
39
39
|
| Look at the screen | ~130–400 ms, blocking | **~20 ms**, already captured |
|
|
40
|
+
|
|
40
41
|
| "Did anything change?" | a full image | **~2 ms**, text only |
|
|
41
42
|
| Finding a control | read tree (~570 ms) + reason | **~1 ms** from memory |
|
|
42
43
|
| A 4-step flow, verified | 4+ model round trips | **1 call**, 3.6 s |
|
|
@@ -44,6 +45,13 @@ Same four-tab navigation flow, on a real production app:
|
|
|
44
45
|
| A 10-step flow | 10 turns, 10 images (~16,000 tokens at best) | **1 turn, 0 images, ~1,650 characters** |
|
|
45
46
|
| Reading a screen | an image: ~1,600 tokens, no tap points | **~330 tokens** of text, with tap points |
|
|
46
47
|
|
|
48
|
+
Every figure above is the cost inside a live process — the MCP server, or the
|
|
49
|
+
daemon answering a socket — which is how an agent actually uses simframe. A
|
|
50
|
+
one-shot `simframe` command from a shell pays about 200 ms of Node startup on
|
|
51
|
+
top, and a frame sitting on an idle screen can be older than 20 ms because the
|
|
52
|
+
capture loop throttles when nothing moves. `~20 ms` is the read, not the
|
|
53
|
+
process.
|
|
54
|
+
|
|
47
55
|
The four-tab tour, three times back to back from a cleared memory:
|
|
48
56
|
|
|
49
57
|
| Pass | Wall clock | Steps verified | Controls from memory |
|
|
@@ -67,10 +75,16 @@ npm install -g simframe
|
|
|
67
75
|
simframe doctor
|
|
68
76
|
```
|
|
69
77
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
78
|
+
**Whichever of those two commands you run first** builds a small Swift daemon
|
|
79
|
+
from source — including `doctor`, which is why a cold `doctor` takes around 15
|
|
80
|
+
seconds and every later one takes two. It needs the Xcode command line tools,
|
|
81
|
+
which you already have if you have a simulator. Without them simframe falls
|
|
82
|
+
back to the original `simctl` loop and says so.
|
|
83
|
+
|
|
84
|
+
If more than one simulator is booted, name the one you mean — `--device=<udid>`,
|
|
85
|
+
or `export SIMFRAME_DEVICE=<udid>` once per shell. simframe refuses to choose
|
|
86
|
+
for you, because the first booted device is nobody's idea of "yours" and the
|
|
87
|
+
command that would act on it is a tap.
|
|
74
88
|
|
|
75
89
|
`doctor` checks each capability separately and tells you what you have:
|
|
76
90
|
|
|
@@ -143,7 +157,7 @@ boundary hands it frames and nothing above it knows what a simulator is.
|
|
|
143
157
|
| Read labels + coordinates from pixels | yes | the same Vision OCR + CV, off the same PNG |
|
|
144
158
|
| Screen map, refs, screen memory, the graph | yes | unchanged above the boundary |
|
|
145
159
|
| Tap, type, swipe, keys | yes | the console's `event mouse` as a real down/move/up, `event text` for characters, `input keyevent` for keys |
|
|
146
|
-
| Clipboard | yes | the emulator's gRPC `setClipboard`, over `node:http2`, no dependency |
|
|
160
|
+
| Clipboard, and `paste` into a field | yes | the emulator's gRPC `setClipboard`, over `node:http2`, no dependency, then `KEYCODE_PASTE` to deliver it |
|
|
147
161
|
| List/resolve devices, launch, terminate, open a URL, permissions | yes | `adb`, with the permission state read back off the device |
|
|
148
162
|
| Accessibility tree | **not available (OCR + CV only)** | `uiautomator dump` costs **2,012 ms** a read, against 45 ms for the iOS tree. See [`docs/DEFERRED.md`](docs/DEFERRED.md) |
|
|
149
163
|
|
|
@@ -154,6 +168,17 @@ simframe ui --device=emulator-5554
|
|
|
154
168
|
simframe do --device=emulator-5554 flow.json
|
|
155
169
|
```
|
|
156
170
|
|
|
171
|
+
A host with a booted simulator **and** a booted emulator has no default, and
|
|
172
|
+
simframe will not pick one for you: preferring iOS because it came first would
|
|
173
|
+
tap a simulator while you were driving an emulator, and acting on the wrong
|
|
174
|
+
device is worse than refusing. So a command with no device names both and stops.
|
|
175
|
+
`--device` answers it per command; `SIMFRAME_DEVICE` answers it per shell:
|
|
176
|
+
|
|
177
|
+
```bash
|
|
178
|
+
export SIMFRAME_DEVICE=emulator-5554
|
|
179
|
+
simframe ui # the emulator, without saying so every time
|
|
180
|
+
```
|
|
181
|
+
|
|
157
182
|
The tree is a deliberate omission, not an oversight. Making it fast needs a
|
|
158
183
|
resident instrumentation APK on the device — the shape uiautomator2, Maestro and
|
|
159
184
|
Appium all converged on — and that would be simframe's first runtime artifact
|
|
@@ -517,8 +542,14 @@ simframe frame --out=now.png # newest frame, native resolution, to a file
|
|
|
517
542
|
simframe strip --count=6 # contact sheet, for an animation
|
|
518
543
|
simframe doctor --strict # any degraded layer is a non-zero exit
|
|
519
544
|
simframe start / status / stop [--force] / devices
|
|
545
|
+
simframe ui --device=emulator-5554 # or export SIMFRAME_DEVICE once
|
|
520
546
|
```
|
|
521
547
|
|
|
548
|
+
`--device` takes `--device=X` and `--device X` alike. It used to take only the
|
|
549
|
+
first: the space form set the flag to `true` and then resolved a device named
|
|
550
|
+
"true", which is a poor answer to a flag `doctor`'s own advice tells you to
|
|
551
|
+
type.
|
|
552
|
+
|
|
522
553
|
### The Claude Code skill
|
|
523
554
|
|
|
524
555
|
[`skills/simframe/SKILL.md`](skills/simframe/SKILL.md) teaches the CLI path
|
package/package.json
CHANGED
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "simframe",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.0",
|
|
4
4
|
"mcpName": "io.github.lvlrSajjad/simframe",
|
|
5
|
-
"description": "Always-warm iOS Simulator frames: agents read the screen in ~20ms instead of waiting on screenshots. MCP server + CLI.",
|
|
5
|
+
"description": "Always-warm iOS Simulator and Android emulator frames: agents read the screen in ~20ms instead of waiting on screenshots. MCP server + CLI.",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"ios",
|
|
8
8
|
"simulator",
|
|
9
|
+
"android",
|
|
10
|
+
"emulator",
|
|
9
11
|
"mcp",
|
|
10
12
|
"model-context-protocol",
|
|
11
13
|
"claude",
|
|
@@ -123,4 +123,19 @@ if (missing.length) {
|
|
|
123
123
|
);
|
|
124
124
|
process.exit(1);
|
|
125
125
|
}
|
|
126
|
+
// The MCP Registry rejects a description over 100 characters, and it does so at
|
|
127
|
+
// `mcp-publisher validate` — after the tag is pushed and the release has begun.
|
|
128
|
+
// A 0.7.1 release found that out the hard way with a 113-character one. The
|
|
129
|
+
// constraint is the registry's; discovering it locally is this script's job.
|
|
130
|
+
const DESCRIPTION_MAX = 100;
|
|
131
|
+
const server = JSON.parse(fs.readFileSync(path.join(ROOT, 'server.json'), 'utf8'));
|
|
132
|
+
if (server.description.length > DESCRIPTION_MAX) {
|
|
133
|
+
console.error(
|
|
134
|
+
`server.json description is ${server.description.length} characters; the MCP Registry accepts ` +
|
|
135
|
+
`${DESCRIPTION_MAX}. It would fail at validate, with the tag already pushed.`,
|
|
136
|
+
);
|
|
137
|
+
process.exit(1);
|
|
138
|
+
}
|
|
139
|
+
console.log(`server.json description ${server.description.length}/${DESCRIPTION_MAX} chars`);
|
|
140
|
+
|
|
126
141
|
console.log('ok — every build input ships');
|
package/scripts/ci-memory.mjs
CHANGED
|
@@ -257,7 +257,14 @@ if (first) {
|
|
|
257
257
|
// to defend — and it is how this check has failed twice.
|
|
258
258
|
if (moved) {
|
|
259
259
|
const stale = await cli(['find', `#${first.ref}`], { expectFail: true });
|
|
260
|
-
check
|
|
260
|
+
// Matched on prose, which is this check's weakness: simframe refused
|
|
261
|
+
// correctly with "#1 cannot be trusted here — simframe does not recognise
|
|
262
|
+
// this screen. Read it again", and the check failed because that wording was
|
|
263
|
+
// not one of the two it knew. The refusal is what matters, so the
|
|
264
|
+
// alternation covers how a refusal is actually phrased; the durable fix is a
|
|
265
|
+
// machine-readable reason on the failure, which `find --json` does not yet
|
|
266
|
+
// carry.
|
|
267
|
+
check(/different screen|read the screen|read it again|does not recognise this screen|cannot be trusted/i.test(stale),
|
|
261
268
|
'a ref numbered on another screen refuses instead of tapping those coordinates',
|
|
262
269
|
stale.trim().split('\n')[0]?.slice(0, 90));
|
|
263
270
|
}
|
|
@@ -398,7 +405,13 @@ check(Array.isArray(nonsense.known) && nonsense.known.length > 0,
|
|
|
398
405
|
|
|
399
406
|
const target = screens[0];
|
|
400
407
|
const walked = await jsonRetry(['goto', target.hash], { allowFail: true });
|
|
401
|
-
|
|
408
|
+
// `no-identity` is new in 0.7.2 and belongs here: `goto` used to *throw*
|
|
409
|
+
// `Cannot read properties of null (reading 'slice')` on a screen whose token set
|
|
410
|
+
// was empty, because `hashTokens` returns null for one on purpose. A crash is
|
|
411
|
+
// neither walking there nor naming why it cannot, so this check failed with an
|
|
412
|
+
// empty detail — the reason was `undefined` — and the check was right to fail.
|
|
413
|
+
// Now there is a name for it, and the list has to know the name.
|
|
414
|
+
const outcomes = ['no-route', 'unreplayable-edge', 'ambiguous', 'unknown-screen', 'no-identity'];
|
|
402
415
|
check(walked.ok === true || outcomes.includes(walked.reason),
|
|
403
416
|
'and asked for a screen it knows, it either walks there or names why it cannot',
|
|
404
417
|
walked.ok ? (walked.already ? 'already there' : `walked ${walked.ranSteps} step(s)`) : walked.reason);
|
package/src/actions.js
CHANGED
|
@@ -6,7 +6,7 @@ import * as api from './index.js';
|
|
|
6
6
|
import * as graph from './graph.js';
|
|
7
7
|
import * as input from './input.js';
|
|
8
8
|
import * as intent from './intent.js';
|
|
9
|
-
import { launchApp, openUrl,
|
|
9
|
+
import { launchApp, openUrl, setPermission, terminateApp } from './platform/index.js';
|
|
10
10
|
|
|
11
11
|
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
|
12
12
|
const MAX_PAUSE_MS = 5000;
|
|
@@ -272,6 +272,33 @@ export async function runScript(
|
|
|
272
272
|
};
|
|
273
273
|
}
|
|
274
274
|
|
|
275
|
+
/**
|
|
276
|
+
* Tap a field and wait for it to take focus, for the steps that then put text
|
|
277
|
+
* in it.
|
|
278
|
+
*
|
|
279
|
+
* `locate`, not `tapLabel`: tapLabel asks the accessibility tree directly, so a
|
|
280
|
+
* field only OCR can see was untypeable and a selector (`#4`, `@x,y`) meant
|
|
281
|
+
* nothing here. The returned `quiet` says when the field never visibly took
|
|
282
|
+
* focus — usually fine, since a field that already had focus does not move, but
|
|
283
|
+
* also exactly what a tap that missed looks like, so the caller should be told.
|
|
284
|
+
*/
|
|
285
|
+
async function focusField(deviceQuery, udid, step, ctx) {
|
|
286
|
+
const found = await api.locate(deviceQuery, step.into, { index: step.index, refresh: step.refresh });
|
|
287
|
+
await input.tapPoint(udid, found.target.x, found.target.y);
|
|
288
|
+
const focused = await api.waitFor(deviceQuery, {
|
|
289
|
+
mode: 'settle',
|
|
290
|
+
stableMs: FOCUS_STABLE_MS,
|
|
291
|
+
reactionMs: FOCUS_REACTION_MS,
|
|
292
|
+
timeoutMs: FOCUS_TIMEOUT_MS,
|
|
293
|
+
options: ctx.options,
|
|
294
|
+
});
|
|
295
|
+
return {
|
|
296
|
+
found,
|
|
297
|
+
where: `"${found.target.label}" at ${found.target.x},${found.target.y}`,
|
|
298
|
+
quiet: focused.satisfied ? '' : ' [the field did not visibly take focus]',
|
|
299
|
+
};
|
|
300
|
+
}
|
|
301
|
+
|
|
275
302
|
async function runStep(deviceQuery, udid, step, ctx) {
|
|
276
303
|
switch (step.action) {
|
|
277
304
|
case 'tap': {
|
|
@@ -298,33 +325,25 @@ async function runStep(deviceQuery, udid, step, ctx) {
|
|
|
298
325
|
}
|
|
299
326
|
case 'type': {
|
|
300
327
|
if (step.into) {
|
|
301
|
-
|
|
302
|
-
// so a field that only OCR can see was untypeable, and a selector
|
|
303
|
-
// (`#4`, `@x,y`) meant nothing here.
|
|
304
|
-
const found = await api.locate(deviceQuery, step.into, { index: step.index, refresh: step.refresh });
|
|
305
|
-
await input.tapPoint(udid, found.target.x, found.target.y);
|
|
306
|
-
const focused = await api.waitFor(deviceQuery, {
|
|
307
|
-
mode: 'settle',
|
|
308
|
-
stableMs: FOCUS_STABLE_MS,
|
|
309
|
-
reactionMs: FOCUS_REACTION_MS,
|
|
310
|
-
timeoutMs: FOCUS_TIMEOUT_MS,
|
|
311
|
-
options: ctx.options,
|
|
312
|
-
});
|
|
328
|
+
const field = await focusField(deviceQuery, udid, step, ctx);
|
|
313
329
|
await input.typeText(udid, step.text ?? step.value);
|
|
314
|
-
|
|
315
|
-
// field that was already focused does not move — but it is also what a
|
|
316
|
-
// tap that missed looks like, and the caller should be able to tell.
|
|
317
|
-
const quiet = focused.satisfied ? '' : ' [the field did not visibly take focus]';
|
|
318
|
-
return `typed into "${found.target.label}" at ${found.target.x},${found.target.y}${quiet}`;
|
|
330
|
+
return `typed into ${field.where}${field.quiet}`;
|
|
319
331
|
}
|
|
320
332
|
await input.typeText(udid, step.text ?? step.value);
|
|
321
333
|
return 'typed text';
|
|
322
334
|
}
|
|
323
335
|
case 'paste': {
|
|
324
|
-
// Long strings are much faster on the pasteboard than through the
|
|
325
|
-
|
|
326
|
-
if
|
|
327
|
-
|
|
336
|
+
// Long strings are much faster on the pasteboard than through the
|
|
337
|
+
// keyboard. `pasteText` delivers the keystroke as well as setting the
|
|
338
|
+
// pasteboard, and throws if it cannot — this step used to do neither and
|
|
339
|
+
// report success anyway.
|
|
340
|
+
if (step.into) {
|
|
341
|
+
const field = await focusField(deviceQuery, udid, step, ctx);
|
|
342
|
+
await input.pasteText(udid, step.text ?? step.value);
|
|
343
|
+
return `pasted into ${field.where}${field.quiet}`;
|
|
344
|
+
}
|
|
345
|
+
await input.pasteText(udid, step.text ?? step.value);
|
|
346
|
+
return 'pasted into the focused field';
|
|
328
347
|
}
|
|
329
348
|
case 'swipe': {
|
|
330
349
|
const from = { x: step.from?.[0] ?? step.from?.x, y: step.from?.[1] ?? step.from?.y };
|
package/src/cli.js
CHANGED
|
@@ -86,14 +86,39 @@ The reliable pattern around an action is:
|
|
|
86
86
|
simframe state --since=$H
|
|
87
87
|
`;
|
|
88
88
|
|
|
89
|
+
/**
|
|
90
|
+
* Flags that take a value, so `--flag value` can mean what it looks like.
|
|
91
|
+
*
|
|
92
|
+
* Deliberately not every flag: `--json`, `--refresh` and friends have a bare
|
|
93
|
+
* form, and letting those swallow the next argument would turn
|
|
94
|
+
* `simframe tap --refresh Save` into a tap on nothing.
|
|
95
|
+
*/
|
|
96
|
+
const VALUE_FLAGS = new Set([
|
|
97
|
+
'ago', 'count', 'detail', 'device', 'durationMs', 'engine', 'filter', 'fps', 'index', 'maxDim',
|
|
98
|
+
'mode', 'out', 'ringSize', 'since', 'spanMs', 'stableMs', 'timeoutMs',
|
|
99
|
+
]);
|
|
100
|
+
|
|
89
101
|
function parseArgs(argv) {
|
|
90
102
|
const flags = {};
|
|
91
103
|
const positional = [];
|
|
92
|
-
for (
|
|
104
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
105
|
+
const arg = argv[i];
|
|
93
106
|
if (arg.startsWith('--')) {
|
|
94
107
|
const [key, value] = arg.slice(2).split('=');
|
|
95
108
|
const camel = key.replace(/-([a-z])/g, (_, c) => c.toUpperCase());
|
|
96
|
-
|
|
109
|
+
if (value !== undefined) {
|
|
110
|
+
flags[camel] = value;
|
|
111
|
+
} else if (VALUE_FLAGS.has(camel) && argv[i + 1] != null && !argv[i + 1].startsWith('--')) {
|
|
112
|
+
// `--device X` as well as `--device=X`. Only for flags whose bare form
|
|
113
|
+
// means nothing: `simframe doctor --device B55AB0AE` used to set
|
|
114
|
+
// `device` to `true`, push the udid to positional, and resolve the
|
|
115
|
+
// literal string "true" — and doctor's own advice is to name a device
|
|
116
|
+
// with --device, which is exactly how someone would write it.
|
|
117
|
+
flags[camel] = argv[i + 1];
|
|
118
|
+
i += 1;
|
|
119
|
+
} else {
|
|
120
|
+
flags[camel] = true;
|
|
121
|
+
}
|
|
97
122
|
} else {
|
|
98
123
|
positional.push(arg);
|
|
99
124
|
}
|
|
@@ -218,6 +243,11 @@ async function main() {
|
|
|
218
243
|
`stopped ${stopped} daemon${stopped === 1 ? '' : 's'}` +
|
|
219
244
|
(inUse ? `; left ${inUse} in use by another client (pass --force to stop anyway)` : ''),
|
|
220
245
|
);
|
|
246
|
+
// Asked to stop one device, refused, and exited 0 — which is a success
|
|
247
|
+
// code for work not done, and a script checking `$?` could not tell the
|
|
248
|
+
// difference. `--all` is informational by nature, so it keeps exiting 0
|
|
249
|
+
// when it skips a device somebody else holds.
|
|
250
|
+
if (!flags.all && inUse && !stopped) process.exitCode = 1;
|
|
221
251
|
return;
|
|
222
252
|
}
|
|
223
253
|
|
|
@@ -800,6 +830,25 @@ async function doctor({ json = false, strict = false, device } = {}) {
|
|
|
800
830
|
const wanted = await resolveDevice(device);
|
|
801
831
|
booted = booted.filter((d) => d.udid === wanted.udid);
|
|
802
832
|
}
|
|
833
|
+
// Which devices get *probed*, as opposed to listed. The probes below start
|
|
834
|
+
// a capture loop and read frames, and doctor used to do that to every
|
|
835
|
+
// booted device on the host. On a shared machine that means starting a
|
|
836
|
+
// daemon on a colleague's simulator and capturing their screen to answer a
|
|
837
|
+
// question about this one. Listing is free and stays; probing is not, so
|
|
838
|
+
// without --device it goes to a device already running its own capture loop
|
|
839
|
+
// (nothing new is started), or to the only booted device, and otherwise to
|
|
840
|
+
// none, with a line saying which flag would pick one.
|
|
841
|
+
let probed = booted;
|
|
842
|
+
if (!device && booted.length > 1) {
|
|
843
|
+
probed = booted.filter((d) => engineModule.runningEngine(d.udid));
|
|
844
|
+
if (probed.length !== 1) {
|
|
845
|
+
probed = [];
|
|
846
|
+
add('device probes', 'warn',
|
|
847
|
+
`${booted.length} devices are booted and none is clearly yours — name one with --device ` +
|
|
848
|
+
'to check its capture, input and accessibility layers',
|
|
849
|
+
{ key: 'probes.skipped', value: booted.length });
|
|
850
|
+
}
|
|
851
|
+
}
|
|
803
852
|
// `deviceNoun` earns its place here: one platform's devices are called by
|
|
804
853
|
// its own word, and a mixed set by the neutral one. An emulator reported as
|
|
805
854
|
// a "booted simulator" is the same small lie as an emulator reported as
|
|
@@ -807,7 +856,7 @@ async function doctor({ json = false, strict = false, device } = {}) {
|
|
|
807
856
|
const nouns = [...new Set(booted.map((d) => capabilitiesFor(d.udid) && PLATFORMS[d.platform].deviceNoun))];
|
|
808
857
|
add(`booted ${nouns.length === 1 ? nouns[0] : 'device'}`, booted.length ? 'ok' : 'warn',
|
|
809
858
|
booted.map((d) => `${d.name} (${d.runtime})`).join(', ') || 'none');
|
|
810
|
-
for (const d of
|
|
859
|
+
for (const d of probed) {
|
|
811
860
|
const input = await import('./input.js');
|
|
812
861
|
const control = await import('./control.js');
|
|
813
862
|
// What this device's platform can do at all. Without asking, doctor
|
|
@@ -871,9 +920,9 @@ async function doctor({ json = false, strict = false, device } = {}) {
|
|
|
871
920
|
ax.available ? `${ax.name}: ${ax.version}` : `unavailable: ${ax.reason}`,
|
|
872
921
|
{ key: 'ax.driver', value: ax.name });
|
|
873
922
|
}
|
|
874
|
-
if (
|
|
923
|
+
if (probed.length) {
|
|
875
924
|
const t0 = Date.now();
|
|
876
|
-
const res = await api.getFrame(
|
|
925
|
+
const res = await api.getFrame(probed[0].udid);
|
|
877
926
|
add('capture', 'ok',
|
|
878
927
|
`frame #${res.state.seq} ${res.width}x${res.height} in ${Date.now() - t0}ms (age ${res.ageMs}ms)`,
|
|
879
928
|
{ key: 'capture.frames', value: res.state.seq });
|
|
@@ -881,7 +930,7 @@ async function doctor({ json = false, strict = false, device } = {}) {
|
|
|
881
930
|
// to ask the capture loop rather than look at the frames. `fail`, not
|
|
882
931
|
// `warn`: nothing here is degraded-but-working, and the cure is a device
|
|
883
932
|
// restart that simframe deliberately does not perform.
|
|
884
|
-
for (const d of
|
|
933
|
+
for (const d of probed) {
|
|
885
934
|
const live = api.liveness(d.udid, (await api.getState(d.udid)).state);
|
|
886
935
|
if (live.stalled) add(`capture health (${d.name})`, 'fail', live.note, { key: 'capture.stalled', value: true });
|
|
887
936
|
}
|
|
@@ -893,8 +942,21 @@ async function doctor({ json = false, strict = false, device } = {}) {
|
|
|
893
942
|
// doctor is a diagnostic, not a way to start things. If it had to start a
|
|
894
943
|
// daemon to answer "which engine is in use", it stops it again rather than
|
|
895
944
|
// leaving a detached process behind.
|
|
945
|
+
//
|
|
946
|
+
// And it says when it could not. This was `catch { /* best effort */ }`, and
|
|
947
|
+
// best effort silently failed: a stop refused because another client holds
|
|
948
|
+
// the device left a capture loop running on a machine somebody else was
|
|
949
|
+
// using, with doctor reporting a clean bill of health. A diagnostic that
|
|
950
|
+
// leaves something behind has to name it.
|
|
896
951
|
for (const udid of startedHere) {
|
|
897
|
-
try {
|
|
952
|
+
try {
|
|
953
|
+
await api.stopDaemon(udid);
|
|
954
|
+
} catch (err) {
|
|
955
|
+
add('cleanup', 'warn',
|
|
956
|
+
`started a capture loop on ${udid} to answer a question and could not stop it again ` +
|
|
957
|
+
`(${err.message}) — stop it with: simframe stop --device=${udid}`,
|
|
958
|
+
{ key: 'cleanup.left', value: udid });
|
|
959
|
+
}
|
|
898
960
|
}
|
|
899
961
|
|
|
900
962
|
const failed = checks.filter((c) => c.level === 'fail');
|
package/src/daemon.js
CHANGED
|
@@ -159,6 +159,12 @@ export async function runDaemon(device, options = {}) {
|
|
|
159
159
|
store.writeCaptureHealth(udid, null);
|
|
160
160
|
}
|
|
161
161
|
consecutiveErrors = 0;
|
|
162
|
+
// And the clock, which it did not. A second stall then kept the first
|
|
163
|
+
// one's timestamp — `stalledSince ??=` only fills a null — and reported
|
|
164
|
+
// "unreadable for 4 hours" about a wedge four seconds old. The Swift loop
|
|
165
|
+
// resets it (main.swift), which is what made this a missed line rather
|
|
166
|
+
// than a difference of opinion between the two engines.
|
|
167
|
+
stalledSince = null;
|
|
162
168
|
|
|
163
169
|
const hash = frameHash(bmp);
|
|
164
170
|
const layout = layoutHash(bmp);
|
package/src/index.js
CHANGED
|
@@ -223,9 +223,19 @@ function spawnNodeDaemon(udid, options) {
|
|
|
223
223
|
for (const key of ['fps', 'maxDim', 'ringSize', 'idleExitMs']) {
|
|
224
224
|
if (options[key] != null) args.push(`--${key}=${options[key]}`);
|
|
225
225
|
}
|
|
226
|
+
// stderr goes to the device's own log, the way the Swift daemon's does
|
|
227
|
+
// (engine.js). It was `stdio: 'ignore'`, so anything this loop said about
|
|
228
|
+
// itself went to /dev/null — including the Android backend's notice that the
|
|
229
|
+
// console capture path had failed and it had fallen back to adb at five times
|
|
230
|
+
// the latency. A silent fallback is the failure mode this project has been
|
|
231
|
+
// bitten by twice; it does not get a third time for want of a file handle.
|
|
232
|
+
// The directory may not exist yet on a first start, and `writeCaptureHealth`
|
|
233
|
+
// already taught this project that an unwritable path here fails silently.
|
|
234
|
+
fs.mkdirSync(store.paths(udid).dir, { recursive: true });
|
|
235
|
+
const log = fs.openSync(store.paths(udid).log, 'a');
|
|
226
236
|
const child = spawn(process.execPath, args, {
|
|
227
237
|
detached: true,
|
|
228
|
-
stdio: 'ignore',
|
|
238
|
+
stdio: ['ignore', log, log],
|
|
229
239
|
env: process.env,
|
|
230
240
|
});
|
|
231
241
|
child.unref();
|
package/src/input.js
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
// even available?" before it answers anything else.
|
|
4
4
|
import { execFile } from 'node:child_process';
|
|
5
5
|
import * as control from './control.js';
|
|
6
|
-
import { geometryFor, inputDriverFor } from './platform/index.js';
|
|
6
|
+
import { capabilitiesFor, geometryFor, inputDriverFor, setPasteboard } from './platform/index.js';
|
|
7
7
|
import { promisify } from 'node:util';
|
|
8
8
|
|
|
9
9
|
const run = promisify(execFile);
|
|
@@ -207,6 +207,14 @@ export async function describeAll(udid) {
|
|
|
207
207
|
/* daemon went away mid-call; fall through to idb */
|
|
208
208
|
}
|
|
209
209
|
}
|
|
210
|
+
// A platform with no accessibility tree is not a machine missing idb.
|
|
211
|
+
// `doctor` learned that when it reported "input driver: idb" for an emulator;
|
|
212
|
+
// this path had not, so every Android screen map carried a note telling the
|
|
213
|
+
// reader to brew-install a tool that has never spoken to an Android device —
|
|
214
|
+
// and on a machine where idb *is* installed, spawned it against an emulator
|
|
215
|
+
// serial on every map build.
|
|
216
|
+
const ax = capabilitiesFor(udid).ax;
|
|
217
|
+
if (!ax.supported) throw new Error(`no accessibility tree on this device — ${ax.note}`);
|
|
210
218
|
// Passing --json here yields empty output; the default already emits JSON.
|
|
211
219
|
const out = await idb(['ui', 'describe-all', '--udid', udid]);
|
|
212
220
|
const nodes = [];
|
|
@@ -357,6 +365,39 @@ export async function typeText(udid, value) {
|
|
|
357
365
|
await idb(['ui', 'text', '--udid', udid, String(value)]);
|
|
358
366
|
}
|
|
359
367
|
|
|
368
|
+
/**
|
|
369
|
+
* Put text on the pasteboard **and deliver it** into the focused field.
|
|
370
|
+
*
|
|
371
|
+
* The delivery is the whole point, and it is what was missing: the `paste` step
|
|
372
|
+
* used to set the pasteboard, long-press the field, and report success while
|
|
373
|
+
* the field stayed empty — on both platforms, not just the one it was filed
|
|
374
|
+
* against. iOS had the mechanism and did not use it (the daemon's `paste` is
|
|
375
|
+
* pbcopy *plus* Cmd-V) and Android had the keycode sitting unused in `KEYS`.
|
|
376
|
+
*
|
|
377
|
+
* When nothing can deliver the keystroke this throws rather than returning a
|
|
378
|
+
* cheerful description of half the job. A step that cannot do what it says has
|
|
379
|
+
* to say so; `type` still works on every device.
|
|
380
|
+
*/
|
|
381
|
+
export async function pasteText(udid, value) {
|
|
382
|
+
const own = inputDriverFor(udid);
|
|
383
|
+
if (own?.key) {
|
|
384
|
+
// The clipboard goes over gRPC; KEYCODE_PASTE is what puts it in the field.
|
|
385
|
+
await setPasteboard(udid, String(value));
|
|
386
|
+
await own.key(udid, 'paste');
|
|
387
|
+
return;
|
|
388
|
+
}
|
|
389
|
+
if (control.available(udid)) {
|
|
390
|
+
// One round trip: the daemon copies and presses Cmd-V.
|
|
391
|
+
await control.paste(udid, String(value));
|
|
392
|
+
return;
|
|
393
|
+
}
|
|
394
|
+
await setPasteboard(udid, String(value));
|
|
395
|
+
throw new Error(
|
|
396
|
+
'the text is on the pasteboard but nothing here can paste it: the keystroke needs the simframe ' +
|
|
397
|
+
'daemon (start it with `simframe start`), or use `type`, which carries the characters itself',
|
|
398
|
+
);
|
|
399
|
+
}
|
|
400
|
+
|
|
360
401
|
/** Key events rather than text: for shortcuts and search-as-you-type. */
|
|
361
402
|
export async function typeKeys(udid, value) {
|
|
362
403
|
const own = inputDriverFor(udid);
|
package/src/navigate.js
CHANGED
|
@@ -51,6 +51,13 @@ export async function goto(deviceQuery, target, { options, ...runOptions } = {})
|
|
|
51
51
|
return { ok: true, already: true, screen: found.name, steps: [] };
|
|
52
52
|
}
|
|
53
53
|
|
|
54
|
+
// `hashTokens` returns null for an empty token set on purpose — a constant
|
|
55
|
+
// hash for "I could read nothing" is the self-confirming-emptiness bug. So a
|
|
56
|
+
// screen with no identity has to be reported, not sliced: this threw
|
|
57
|
+
// `Cannot read properties of null (reading 'slice')` instead of answering.
|
|
58
|
+
// Not hypothetical on Android, where README's own table puts the launcher at
|
|
59
|
+
// one token.
|
|
60
|
+
if (!here.hash) return { ok: false, reason: 'no-identity', to: found.name };
|
|
54
61
|
const path_ = graph.route(udid, { hash: here.hash, tokens: here.tokens }, found.node.hash);
|
|
55
62
|
if (!path_) return { ok: false, reason: 'no-route', from: here.hash.slice(0, 8), to: found.name };
|
|
56
63
|
|
|
@@ -65,7 +72,7 @@ export async function goto(deviceQuery, target, { options, ...runOptions } = {})
|
|
|
65
72
|
steps,
|
|
66
73
|
ranSteps: result.ranSteps,
|
|
67
74
|
results: result.results,
|
|
68
|
-
arrived: arrived.hash.slice(0, 8),
|
|
75
|
+
arrived: arrived.hash ? arrived.hash.slice(0, 8) : null,
|
|
69
76
|
};
|
|
70
77
|
}
|
|
71
78
|
|
package/src/platform/android.js
CHANGED
|
@@ -10,6 +10,10 @@
|
|
|
10
10
|
// below, and the reason two of them are not the obvious command:
|
|
11
11
|
//
|
|
12
12
|
// emulator console `screenrecord screenshot <dir>` 20 ms, written host-side
|
|
13
|
+
//
|
|
14
|
+
// Those are screenshot costs. A *frame* — screenshot plus the resize that makes
|
|
15
|
+
// it readable — measures 41 ms, which is the number README quotes; the two were
|
|
16
|
+
// quoted interchangeably in three files and agreed with each other in none.
|
|
13
17
|
// adb exec-out screencap -p 113 ms, 9 KB over adb
|
|
14
18
|
// adb exec-out screencap (raw RGBA, 3.7 MB) 218 ms — the transfer, not the encode
|
|
15
19
|
// adb shell getprop x4 in one hop 28 ms
|
|
@@ -107,7 +111,14 @@ async function fetchDevices() {
|
|
|
107
111
|
const serials = [];
|
|
108
112
|
for (const line of stdout.split('\n').slice(1)) {
|
|
109
113
|
const [serial, state] = line.trim().split(/\s+/);
|
|
110
|
-
|
|
114
|
+
// Only emulators. Physical devices are a stated non-goal, and this list is
|
|
115
|
+
// "devices simframe can drive" — `ownsUdid` claims emulator serials and
|
|
116
|
+
// nothing else, so listing a plugged-in phone here made listing and routing
|
|
117
|
+
// disagree: `devices` offered it, a bare `resolveDevice` could return it,
|
|
118
|
+
// and the tap that followed failed with a complaint about the serial rather
|
|
119
|
+
// than an honest "not supported". A unit test asserts the two agree; it
|
|
120
|
+
// passed only because nobody had a phone attached.
|
|
121
|
+
if (serial && state && ownsUdid(serial)) serials.push({ serial, adbState: state });
|
|
111
122
|
}
|
|
112
123
|
const out = [];
|
|
113
124
|
for (const { serial, adbState } of serials) {
|
|
@@ -165,6 +176,18 @@ async function resolveDevice(query, opts) {
|
|
|
165
176
|
: 'no booted emulator (start one with `emulator -avd <name>`)',
|
|
166
177
|
);
|
|
167
178
|
}
|
|
179
|
+
// See the same guard in ios.js: an arbitrary pick is a tap on the wrong
|
|
180
|
+
// device, and this backend can act too.
|
|
181
|
+
if (booted.length > 1) {
|
|
182
|
+
throw Object.assign(
|
|
183
|
+
new Error(
|
|
184
|
+
`${booted.length} emulators are running and none was named: ` +
|
|
185
|
+
`${booted.map((d) => `${d.name} (${d.udid})`).join(', ')} — name one with --device, ` +
|
|
186
|
+
'or set SIMFRAME_DEVICE to pick a default for this shell',
|
|
187
|
+
),
|
|
188
|
+
{ ambiguous: true },
|
|
189
|
+
);
|
|
190
|
+
}
|
|
168
191
|
return booted[0];
|
|
169
192
|
}
|
|
170
193
|
const q = loose(query);
|
|
@@ -556,6 +579,12 @@ async function screenshot(udid, outFile, { mask: _mask = 'ignored' } = {}) {
|
|
|
556
579
|
try {
|
|
557
580
|
const { stdout } = await adb(udid, ['exec-out', 'screencap', '-p'], { encoding: 'buffer', timeout: 20_000 });
|
|
558
581
|
fs.writeFileSync(outFile, stdout);
|
|
582
|
+
// stderr, not a store write: a backend below the boundary has no business
|
|
583
|
+
// knowing where simframe keeps its files. The capture loop is now spawned
|
|
584
|
+
// with its stderr pointed at `daemon.log` (src/index.js), which is what
|
|
585
|
+
// makes this line readable at all — it used to go to a daemon started
|
|
586
|
+
// `stdio: 'ignore'`, so a five-times-slower capture path could be in
|
|
587
|
+
// effect for a whole session with no trace of it anywhere.
|
|
559
588
|
process.env.SIMFRAME_QUIET === '1' ||
|
|
560
589
|
process.stderr.write(`simframe: emulator console unavailable (${consoleErr.message}); used adb screencap\n`);
|
|
561
590
|
} catch (adbErr) {
|
|
@@ -796,7 +825,14 @@ async function launchApp(udid, bundleId, { args = [], env = {}, terminateFirst =
|
|
|
796
825
|
}
|
|
797
826
|
|
|
798
827
|
async function terminateApp(udid, bundleId) {
|
|
799
|
-
|
|
828
|
+
// `am` reports failure on stdout and still exits 0 — the same trap launchApp
|
|
829
|
+
// and openUrl already check for. Without this, terminating a package that is
|
|
830
|
+
// not installed answered "terminated com.typo.app".
|
|
831
|
+
const { stdout, stderr } = await adb(udid, ['shell', 'am', 'force-stop', bundleId]);
|
|
832
|
+
const error = /^Error:.*$/m.exec(`${stdout}${stderr}`);
|
|
833
|
+
if (error) {
|
|
834
|
+
throw new Error(`could not terminate ${bundleId}: ${error[0].replace(/^Error:\s*/, '')}`);
|
|
835
|
+
}
|
|
800
836
|
}
|
|
801
837
|
|
|
802
838
|
async function openUrl(udid, url) {
|
|
@@ -928,10 +964,15 @@ function toolchain() {
|
|
|
928
964
|
* There is no framebuffer engine, so `screenshot` is not a downgrade on this
|
|
929
965
|
* platform and doctor must not report it as one.
|
|
930
966
|
*
|
|
931
|
-
* Input
|
|
932
|
-
*
|
|
933
|
-
*
|
|
934
|
-
*
|
|
967
|
+
* Input: yes, through the emulator console — `event mouse` puts a real
|
|
968
|
+
* down/move/up on the touch screen, and `event text` carries characters rather
|
|
969
|
+
* than key positions, so a non-Latin host layout cannot reinterpret them. This
|
|
970
|
+
* comment said "not yet" for a release after it shipped; a stale comment above
|
|
971
|
+
* a live value is worse than no comment, because the next reader believes it.
|
|
972
|
+
*
|
|
973
|
+
* The accessibility tree: no, and said so rather than answered with the iOS
|
|
974
|
+
* driver's name. `uiautomator dump` costs 2,012 ms a read, which is why it is
|
|
975
|
+
* not the answer; see docs/DEFERRED.md for the shape of the one that would be.
|
|
935
976
|
*/
|
|
936
977
|
function capabilities() {
|
|
937
978
|
return {
|
package/src/platform/index.js
CHANGED
|
@@ -147,18 +147,30 @@ export async function bootedDevices(opts) {
|
|
|
147
147
|
* device is the wrong-device bug wearing a different hat.
|
|
148
148
|
*
|
|
149
149
|
* Cross-platform ambiguity — one name matching a simulator and an emulator — is
|
|
150
|
-
* reported rather than guessed at
|
|
151
|
-
*
|
|
152
|
-
*
|
|
150
|
+
* reported rather than guessed at, and so is a *bare* query on a host with a
|
|
151
|
+
* booted device on both platforms. That case is not exotic: it is the mixed
|
|
152
|
+
* setup the second backend exists for, and it used to break every command that
|
|
153
|
+
* did not name a device, with the word "undefined" standing in for the query.
|
|
154
|
+
*
|
|
155
|
+
* There is no defensible way to pick for you. Preferring iOS because it came
|
|
156
|
+
* first would silently tap a simulator while you were driving an emulator,
|
|
157
|
+
* which is the one failure this project will not ship. So a bare query on a
|
|
158
|
+
* mixed host says so and names both devices — and `SIMFRAME_DEVICE` exists so
|
|
159
|
+
* the answer can be given once per shell instead of on every command.
|
|
153
160
|
*
|
|
154
161
|
* @returns {Promise<Device>}
|
|
155
162
|
*/
|
|
156
163
|
export async function resolveDevice(query, opts) {
|
|
157
|
-
return resolveAcross(query, opts, backends());
|
|
164
|
+
return resolveAcross(query ?? process.env.SIMFRAME_DEVICE ?? undefined, opts, backends());
|
|
158
165
|
}
|
|
159
166
|
|
|
160
167
|
/** As above, over a given set of backends — the testable half. @returns {Promise<Device>} */
|
|
161
168
|
export async function resolveAcross(query, opts, all) {
|
|
169
|
+
// A backend is entitled to assume a query is a string. `--device X` used to
|
|
170
|
+
// parse as `device: true`, which reached ios.js as `query.toLowerCase` and
|
|
171
|
+
// crashed with a TypeError where an unmatched device should have been a
|
|
172
|
+
// sentence. The parser no longer produces that, and this makes it unable to.
|
|
173
|
+
if (query != null && typeof query !== 'string') query = String(query);
|
|
162
174
|
const hits = [];
|
|
163
175
|
const misses = [];
|
|
164
176
|
for (const backend of all) {
|
|
@@ -171,9 +183,12 @@ export async function resolveAcross(query, opts, all) {
|
|
|
171
183
|
}
|
|
172
184
|
if (hits.length === 1) return hits[0];
|
|
173
185
|
if (hits.length > 1) {
|
|
186
|
+
const named = hits.map((d) => `${d.name} (${d.platform}, ${d.udid})`).join(', ');
|
|
174
187
|
throw new Error(
|
|
175
|
-
|
|
176
|
-
|
|
188
|
+
(query == null || query === ''
|
|
189
|
+
? 'no device named, and there is a booted device on more than one platform'
|
|
190
|
+
: `"${query}" matches a device on more than one platform`) +
|
|
191
|
+
`: ${named} — name one by its id, or set SIMFRAME_DEVICE to pick a default`,
|
|
177
192
|
);
|
|
178
193
|
}
|
|
179
194
|
// One backend: its own message, unchanged. It is the better message, because
|
package/src/platform/ios.js
CHANGED
|
@@ -66,6 +66,29 @@ async function resolveDevice(query, opts) {
|
|
|
66
66
|
const booted = all.filter((d) => d.state === 'Booted');
|
|
67
67
|
if (!query) {
|
|
68
68
|
if (booted.length === 0) throw new Error('no booted simulator (open Simulator.app or run `xcrun simctl boot <udid>`)');
|
|
69
|
+
// `booted[0]` was the wrong-device bug, and it was worse than it looked.
|
|
70
|
+
// simctl's order is not "yours" by any definition, so on a machine with
|
|
71
|
+
// more than one booted simulator a bare `simframe ui` read whichever came
|
|
72
|
+
// first — and a bare `simframe tap` would have *injected input* into it. A
|
|
73
|
+
// reviewer reproduced it deterministically against a colleague's simulator.
|
|
74
|
+
//
|
|
75
|
+
// `doctor` got a guard for its own fan-out and this default did not, which
|
|
76
|
+
// is how the same command set could pick two different devices in one
|
|
77
|
+
// moment. Refusing is the only safe answer here: the seam cannot see which
|
|
78
|
+
// device simframe is already driving (that is store state, above the
|
|
79
|
+
// boundary), and a backend must never guess when the cost of guessing wrong
|
|
80
|
+
// is a tap on somebody else's screen. `ambiguous` so that a second platform
|
|
81
|
+
// matching cleanly cannot override this — see resolveAcross.
|
|
82
|
+
if (booted.length > 1) {
|
|
83
|
+
throw Object.assign(
|
|
84
|
+
new Error(
|
|
85
|
+
`${booted.length} simulators are booted and none was named: ` +
|
|
86
|
+
`${booted.map((d) => `${d.name} (${d.udid})`).join(', ')} — name one with --device, ` +
|
|
87
|
+
'or set SIMFRAME_DEVICE to pick a default for this shell',
|
|
88
|
+
),
|
|
89
|
+
{ ambiguous: true },
|
|
90
|
+
);
|
|
91
|
+
}
|
|
69
92
|
return booted[0];
|
|
70
93
|
}
|
|
71
94
|
const q = query.toLowerCase();
|