simframe 0.10.0 → 0.12.0

package/README.md

@@ -96,8 +96,14 @@ ok   accessibility tree simframed: AXPTranslator, host-side
 ok   on-device OCR      available
 ok   booted simulator   iPhone 17 Pro (iOS 26.5)
 ok   capture            frame #888 322x700 in 2ms (age 538ms)
+ok   sensor mode        full — accessibility and OCR fused on every read (~164ms)
+ok   local supervisor   none — not requested (SIMFRAME_SUPERVISOR is unset)
+ok   local planner      none — not requested (SIMFRAME_PLANNER is unset)
 ```
 
+The last three are experiments and `none` is their normal answer. See
+[Local tiers, off by default](#local-tiers-off-by-default).
+
 ### Claude Code
 
 ```bash
@@ -117,6 +123,75 @@ registered only for the directory you ran the command in.
 }
 ```
 
+## Recovering without a round trip
+
+The measured cost of driving an app is not perception — warm, an
+accessibility-only read is 85 ms and a fused read 142 ms. It is **round trips**:
+in one instrumented run, 75% of the wall time was the agent thinking and the
+call boundary, not simframe working. So the tools that matter most are the ones
+that let a batch survive a problem instead of handing it back.
+
+**Fallback selectors.** `{"tap": "Save", "or": ["Done", "Confirm"]}` — tried
+locally in order, only an exhausted list reaching the model. Eligible after a
+selector that did not *resolve* and nothing else, because retrying from a screen
+you did not expect to be on is a second guess. A destructive-looking label is
+refused as a substitute even if you list it.
+
+**`{"seek": "change username", "budget": 6}`** opens containers, checks, and
+comes back, depth first, inside a hard budget. It **acts** — opening a door
+changes state — and it refuses to open anything that commits, abandons or
+answers. It does not tap the target; it leaves you on the screen where the target
+resolves.
+
+**`{"sweep": "all", "fill": {…}}`** reads a long screen a viewport at a time and
+fills each field while it is on screen. A form taller than the screen is only
+knowable in pieces — the tree publishes what is rendered — and one scroll gesture
+travels a non-deterministic distance, so finding a field and scrolling back to it
+does not work. Sweeping does: on a real web form it filled every field in one
+call. It detects both ends by measuring how far the *content* moved, ignoring
+fixed chrome, which is the only reliable signal available since nothing reports a
+scroll offset.
+
+**`worked here before:`** puts the graph's own vocabulary in the map, most-used
+first, rather than reporting a count. When the remembered controls are *not* on
+the screen it says so instead, because that means two screens share one
+fingerprint — and confident advice on a misidentified screen is how a remembered
+label ends up pointing at a submit button.
+
+## Local tiers, off by default
+
+Two on-device model experiments, both `none` unless asked for, both degrading to
+the existing matcher-then-model ladder, and CI runs with both off. They ship no
+weights: Apple's Foundation Models framework has nothing to download, which is
+the whole reason it clears this project's non-goal on shipping model weights.
+
+| flag | what it does |
+|---|---|
+| `--sensor=ax-first` | read the accessibility tree alone (~50 ms) and pay for OCR only when a resolve fails |
+| `--planner=apple` | order the containers `seek` opens; it cannot choose an action |
+| `--supervisor=apple` | when a step fails, answer `wait`, `retry` or `stop` — nothing else — before the failure reaches the model |
+
+All three are also per-call arguments on every MCP tool, because an MCP server's
+environment is fixed when it spawns and comparing two modes inside one session
+was otherwise impossible.
+
+**What is measured and what is not.** The ranker: 5 of 6 top-1 on hand-written
+cases, median 564 ms warm, and on a real exploration it went to the right region
+in two steps where reading order wandered into version strings. The supervisor:
+correct on four real batch-killers once the plan briefed it, 689–751 ms warm —
+**on a bench, not in the field.** `ax-first` made no measurable difference to how
+an agent drove a real app, with one small regression and one small win. Numbers
+and conditions are in [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md); the judgements,
+including a phase cancelled by its own measurement, are in
+[`docs/DECISIONS.md`](docs/DECISIONS.md).
+
+The supervisor's whole vocabulary is three words on purpose. It cannot invent a
+step, skip one, substitute a target or continue past an unexpected screen — not
+because a threshold forbids it but because those are not answers it can give.
+That constraint replaced an earlier version of the same idea that was given
+latitude over *what* to open and pressed a button labelled "YES, THIS FIXED MY
+PROBLEM" in a live app.
+
 ## Capabilities are independent
 
 Each layer works without the ones above it, and `doctor` tells you which you
@@ -284,10 +359,15 @@ Three ways to name a control, anywhere one is named:
 
 | | |
 | --- | --- |
-| `#3` | the number the map gave it. Cheapest, unambiguous. |
-| `"Save"` · `the Assets tab` · `back` | resolved by intent — verbs, typos, synonyms, icon-only controls by their common name |
+| `"Save"` · `the Assets tab` · `back` | **start here** — resolved by intent: verbs, typos, synonyms, icon-only controls by their common name |
+| `#3` | the number the map gave it. Cheap and exact, but only within the round trip that numbered it |
 | `@120,400` | raw point coordinates. Last resort: it cannot tell you it missed. |
 
+The order is deliberate and it used to be the other way round. Four peer rounds
+reported that intent resolution worked every time while refs renumbered
+underneath them, so a table that led with `#3` and called it "unambiguous" was
+recommending the more brittle of the two.
+
 ## Baselines: the thing to understand
 
 Every change question is really "changed **since when**?" — and the answer is
@@ -556,9 +636,11 @@ simframe strip --count=6       # contact sheet, for an animation
 simframe doctor --strict       # any degraded layer is a non-zero exit
 simframe escalations           # why simframe still needs a model, by reason
 simframe escalations --session # ...this agent only, not every agent on the device
+simframe supervisions          # local supervisor rulings, and what came of each
 simframe hpi                   # speed and accuracy against a human baseline
 simframe baseline record settings-larger-text --runs=5   # record the human
 simframe input reset           # rebuild the HID session, without restarting anything
+simframe revive                # power-cycle a wedged device: stop, shutdown, boot, start, reset input
 simframe start / status / stop [--force] / devices
 simframe ui --device=emulator-5554      # or export SIMFRAME_DEVICE once
 ```
@@ -568,6 +650,88 @@ first: the space form set the flag to `true` and then resolved a device named
 "true", which is a poor answer to a flag `doctor`'s own advice tells you to
 type.
 
+### When the simulator stops rendering
+
+A simulator driven hard for several minutes can stop rendering: capture fails,
+`simctl screenshot` fails too, and the daemon's own recoveries — re-resolving the
+display port, then rebinding the device — do not help. It reports the state
+rather than acting on it, because a capture loop that rebooted the device it was
+watching would be a tool reaching for the mains:
+
+```
+capture is wedged and both recoveries are spent (2 port re-resolves, 2 device
+rebinds, no frame since). This needs the device restarted —
+`simframe revive --device=<udid>`. Backing off until a frame arrives.
+```
+
+`simframe revive` is that restart, in the order that matters — stop the daemon,
+shut the device down, boot it and *wait for the boot to finish*, start capture,
+rebuild the HID session — and it ends by checking frames are flowing again
+rather than by reporting that the steps ran. It is a command and not a
+behaviour: the decision stays yours.
+
+### Reading what the local supervisor decided
+
+With `SIMFRAME_SUPERVISOR=apple`, every consultation is written to
+`~/.simframe/<udid>/supervisions.jsonl` with what the executor observed
+afterwards, which is what makes a ruling scoreable rather than merely recorded.
+`simframe supervisions` reads it:
+
+```
+14 supervisor rulings on iPhone 17 Pro
+
+  stop -> stopped                    7
+  wait -> recovered                  5
+  wait -> still_failed               2
+
+sourced: model 14
+median latency: 1440ms
+edges the graph had timed: 0/14 — 14 ruling(s) are on edges with no p95
+```
+
+That last line is the honest one: a step that failed is usually a step that has
+never succeeded on that edge, so the graph has no timing to compare against.
+
+### Keeping a session cheap
+
+The expensive part of driving a simulator with an agent is not the tapping, it
+is the thinking between taps — observe, think, tap, observe, think. Measured
+over one real session against a third-party app: **62 tool calls for 179 steps**,
+and 48 of those calls were three steps or fewer. A twelve-step flow arrived as
+five calls, and every boundary between them was a think.
+
+Three things move that number, and simframe does the first two for you:
+
+- **Batch.** `sim_do` runs a whole flow in one call, with an assert after each
+  step that matters. The asserts are what make it safe not to look in between:
+  a step that lands somewhere unplanned halts the flow instead of letting the
+  next four run against the wrong screen.
+- **A `next:` line on every action result** — from the CLI and the MCP server
+  alike — computed locally from what the daemon already knows — whether the screen settled, whether the graph
+  recognises it, how many elements it has, whether any labels repeat. When it
+  says *nothing ambiguous — chain the next steps in one sim_do without looking
+  again*, that is the tool telling the agent it does not need to think.
+- **A trailing map that was re-read, not recalled.** An action pays one
+  perception pass — a few hundred milliseconds, locally — so the map it returns
+  is the screen as it is now. The alternative was an agent spending a whole turn
+  on `ui --refresh` because it could not trust the one it was given.
+- **One goal per session.** Sessions get slower with every turn. A flow that
+  runs as one call adds one exchange to the context instead of twelve.
+
+And one thing to know about waiting: `settle` asks whether the screen stopped
+moving, and a screen waiting on a network call has stopped moving. For anything
+that arrives over the network, assert on the content you expect —
+`{"waitFor": {"value": "Kate Bell"}}` — rather than on stillness. The map says
+`STILL LOADING` when the classifier can see a load in flight, but only you know
+what "arrived" means.
+
+And the expensive habit worth naming: in that session, **28 of 62 calls returned
+a screenshot** — about a third of its entire token cost — because the text map
+could not report what a text field contained. It can now, so check the map
+before reaching for pixels: a row carries the element's contents (`= Fryer 3`)
+and its state (`disabled`), and the flow's own verdict already said whether the
+action worked.
+
 ### The Claude Code skill
 
 [`skills/simframe/SKILL.md`](skills/simframe/SKILL.md) teaches the CLI path
@@ -672,6 +836,21 @@ said a word — the exact failure shape, found by the thing built to catch it.
   black screen as a calm one, but it cannot fix it: restarting the device is
   the cure that always works, and it usually recovers on its own.
 
+## What was decided, and what was not built
+
+[`docs/DECISIONS.md`](docs/DECISIONS.md) is the register of judgements that
+changed the plan: a phase cancelled by its own measurement, two features built
+and reverted for cause, and the premises that turned out to be false. It is
+short on purpose — the numbers live in
+[`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) and the open work in
+[`docs/DEFERRED.md`](docs/DEFERRED.md).
+
+The most useful entry is a **no-go**: Phase 17 proposed a small on-device model
+to choose the next element, and measuring the prize before the model showed the
+matcher already resolves 37 of 40 real decisions. By the time a step reaches
+simframe the decision has already been made — the goal names the option,
+because the agent chose it and then asked for it by name.
+
 ## Roadmap
 
 - **Extend the confirm vocabulary beyond English.**

package/data/vocabulary/en.json

@@ -0,0 +1,148 @@
+{
+  "locale": "en",
+  "note": "Data, not code. CLAUDE.md: reflexes and the destructive vocabulary live in a locale-keyed file so adding a language is not a code change. Referenced by the verify barrier, per-step alternatives, exploration and (when built) reflexes.",
+  "destructive": {
+    "note": "Matched against a control's label, case-insensitively, on word boundaries. Nothing local may act on one of these: no retry, no alternative, no exploration, no reflex, no speculation. A model call is cheaper than a wrong tap.",
+    "words": [
+      "delete",
+      "remove",
+      "erase",
+      "clear",
+      "wipe",
+      "destroy",
+      "discard",
+      "pay",
+      "purchase",
+      "buy",
+      "subscribe",
+      "checkout",
+      "confirm payment",
+      "send",
+      "submit",
+      "post",
+      "publish",
+      "share",
+      "invite",
+      "sign out",
+      "log out",
+      "logout",
+      "deactivate",
+      "deregister",
+      "reset",
+      "restore",
+      "factory",
+      "unpair",
+      "forget",
+      "block",
+      "report",
+      "ban",
+      "unfriend",
+      "unfollow",
+      "cancel subscription",
+      "close account",
+      "delete account",
+      "accept",
+      "agree",
+      "allow",
+      "grant",
+      "trust",
+      "call",
+      "dial",
+      "transfer",
+      "withdraw",
+      "deposit",
+      "place order",
+      "confirm order",
+      "submit order",
+      "cancel order"
+    ],
+    "notWords": {
+      "note": "Labels that are NOT destructive despite containing a destructive word. Matched against the WHOLE label, not as a phrase inside it: a bare 'Cancel' declines a dialog and must stay tappable, while 'Cancel order' is a different act and falls through to the list above. The bare noun 'order' is deliberately absent from that list for the same reason - 'Work Orders' and 'Order Details' are navigation, and only the verb phrase places one.",
+      "words": [
+        "cancel",
+        "clear all filters",
+        "clear search",
+        "clear filter"
+      ]
+    }
+  },
+  "leavesTheApp": {
+    "note": "Nothing local may follow one of these either: it takes the run somewhere simframe was not asked to drive.",
+    "words": [
+      "open in safari",
+      "view in browser",
+      "app store",
+      "settings app",
+      "privacy policy",
+      "terms of service",
+      "contact support"
+    ]
+  },
+  "exploration": {
+    "note": "Exploration is a different question from substitution, and conflating them cost a real run. `seek` opened CANCEL first — because 'cancel' is listed as safe above, so that a local tier may DECLINE a dialog — then AI TROUBLESHOOTING, then 'YES, THIS FIXED MY PROBLEM', ending five screens deep in a support chat with a half-built service request destroyed. May-I-tap-this-to-decline and may-I-open-this-as-a-door are not the same permission. This list answers the second, and it errs toward refusing: a door missed costs one step, a door taken wrongly costs the run.",
+    "neverOpen": [
+      "cancel",
+      "discard",
+      "abandon",
+      "close",
+      "dismiss",
+      "back",
+      "done",
+      "finish",
+      "yes",
+      "no",
+      "ok",
+      "okay",
+      "confirm",
+      "continue",
+      "proceed",
+      "next",
+      "skip",
+      "save",
+      "apply",
+      "submit",
+      "send",
+      "post",
+      "publish",
+      "accept",
+      "agree",
+      "allow",
+      "deny",
+      "block",
+      "grant",
+      "retry",
+      "try again",
+      "start over",
+      "help",
+      "support",
+      "contact",
+      "chat",
+      "feedback",
+      "rate",
+      "review us",
+      "sign in",
+      "sign up",
+      "log in",
+      "register",
+      "upgrade",
+      "buy",
+      "pay",
+      "activate",
+      "enable",
+      "disable",
+      "turn on",
+      "turn off",
+      "reset"
+    ],
+    "neverOpenPatterns": [
+      "^yes[,. ]",
+      "^no[,. ]",
+      "^ok[,. ]",
+      "^don't ",
+      "^do not ",
+      "activate to ",
+      "^tap to ",
+      "^click to "
+    ]
+  }
+}

package/native/ocr.swift

@@ -7,8 +7,20 @@ guard args.count > 1, let img = NSImage(contentsOfFile: args[1]),
       let cg = img.cgImage(forProposedRect: nil, context: nil, hints: nil) else {
     FileHandle.standardError.write("cannot read image\n".data(using: .utf8)!); exit(1)
 }
+// Recognition level is a switch now, not a constant.
+//
+// CLAUDE.md fixes `.accurate` with language correction off, and that was chosen
+// without a comparison — which is a threshold nobody had scored. `SIMFRAME_OCR`
+// selects it so the two can be measured against each other on the perception
+// harness: accuracy lost against milliseconds gained.
+//
+// The theory being tested is a human one: we read imprecisely and gain speed by
+// it, tolerated by a forgiving match. simframe's matcher already forgives a
+// great deal — prefixes, synonyms, typo distance, a Cyrillic-for-Latin fold —
+// so a worse reading may cost nothing that matters.
 let req = VNRecognizeTextRequest()
-req.recognitionLevel = .accurate
+let level = ProcessInfo.processInfo.environment["SIMFRAME_OCR"]?.lowercased() ?? "accurate"
+req.recognitionLevel = (level == "fast") ? .fast : .accurate
 req.usesLanguageCorrection = false
 try! VNImageRequestHandler(cgImage: cg, options: [:]).perform([req])
 let w = Double(cg.width), h = Double(cg.height)

package/native/rank.swift

@@ -0,0 +1,87 @@
+// A local ranker, and nothing more than a ranker.
+//
+// Reads one JSON object per line on stdin and writes one per line on stdout, so
+// the caller pays model load once instead of once per question: measured, the
+// first answer in a process costs ~880ms and every later one ~564ms.
+//
+//   in   {"goal":"change my username","options":["General","Account"]}
+//   out  {"order":["Account","General"],"ms":564}
+//
+// It may only reorder the labels it was given. It never invents one, never says
+// what to do, and never sees pixels — the caller has already decided that every
+// option is permitted (see src/vocabulary.js) and will try them in some order
+// regardless. This only decides which order.
+//
+// Unavailable is a normal answer, not an error: no Apple Intelligence, no Apple
+// Silicon, an older macOS, or a user who has turned it off. It says so on the
+// first line and exits, and `doctor` reports `planner: none` while the existing
+// matcher-then-Claude ladder carries on unchanged.
+import Foundation
+#if canImport(FoundationModels)
+import FoundationModels
+
+@available(macOS 26.0, *)
+@Generable
+struct Ranking {
+    @Guide(description: "The given labels, most likely to lead to the goal first")
+    var order: [String]
+}
+
+struct Question: Decodable { let goal: String; let options: [String] }
+
+func emit(_ object: [String: Any]) {
+    guard let data = try? JSONSerialization.data(withJSONObject: object),
+          let line = String(data: data, encoding: .utf8) else { return }
+    print(line)
+    fflush(stdout)
+}
+
+@available(macOS 26.0, *)
+func serve() async {
+    switch SystemLanguageModel.default.availability {
+    case .available: break
+    case .unavailable(let reason):
+        emit(["unavailable": "\(reason)"])
+        return
+    @unknown default:
+        emit(["unavailable": "unknown availability"])
+        return
+    }
+    let session = LanguageModelSession(instructions: """
+    You rank user-interface controls. You are given something the user is looking \
+    for and a list of labels visible on one screen. Order the labels from most to \
+    least likely to lead to what they want. Use only the labels you were given, \
+    copied verbatim. Do not invent labels and do not explain.
+    """)
+    emit(["ready": true])
+    while let line = readLine(strippingNewline: true) {
+        if line.isEmpty { continue }
+        guard let data = line.data(using: .utf8),
+              let q = try? JSONDecoder().decode(Question.self, from: data) else {
+            emit(["error": "could not parse that line"])
+            continue
+        }
+        let started = Date()
+        do {
+            let answer = try await session.respond(
+                to: "Looking for: \(q.goal)\nLabels: \(q.options.joined(separator: ", "))",
+                generating: Ranking.self)
+            // Only labels we handed it, and never the same one twice: a model
+            // that paraphrases must not be able to smuggle in a new target.
+            var seen = Set<String>()
+            let kept = answer.content.order.filter { q.options.contains($0) && seen.insert($0).inserted }
+            emit(["order": kept, "ms": Int(Date().timeIntervalSince(started) * 1000)])
+        } catch {
+            emit(["error": "\(error)"])
+        }
+    }
+}
+
+if #available(macOS 26.0, *) {
+    await serve()
+} else {
+    emit(["unavailable": "the on-device model needs macOS 26 or newer"])
+}
+#else
+print("{\"unavailable\":\"this toolchain cannot import FoundationModels\"}")
+#endif

package/native/simframed/Sources/PrivateAPI/CoreSimulatorPlatform.swift

@@ -167,6 +167,9 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
     /// session survives. Input is warmed too, because a session that outlived
     /// its device is dead anyway.
     public func reattachDevice(udid: String?) throws -> DeviceInfo {
+        // Before the port reference goes, not after: unregistering needs the
+        // port it was registered on.
+        unregisterChangeCallback()
         device = nil
         display = nil
         return try attach(udid: udid)
@@ -183,6 +186,10 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
     }
 
     private func resolveDisplay(on device: NSObject, warmInput: Bool) throws -> DeviceInfo {
+        // This re-walks the same `ioPorts` array and frequently finds the *same*
+        // port object, so replacing `display` without releasing the callback on
+        // the outgoing one is how registrations accumulated on a single port.
+        unregisterChangeCallback()
         guard let io = device.value(forKey: "io") as? NSObject,
               let ports = io.value(forKey: "ioPorts") as? [NSObject] else {
             throw PrivateAPIError.noDisplayPort
@@ -307,11 +314,37 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
         return raw is IOSurface
     }
 
+    /// How long an on-demand surface read waits for a surface to exist.
+    ///
+    /// The capture *loop* has re-resolve and rebind around this same call. This
+    /// path had nothing: it asked once and threw. On a loaded hosted runner
+    /// that cost a CI job — the display renders intermittently, one read landed
+    /// in a gap, text recognition reported "the display surface could not be
+    /// read", and the screen map came back empty. Four checks failed on a
+    /// device that was healthy before and after, and whose capture loop never
+    /// logged a single failure in the whole run.
+    static let surfaceRetryBudgetMs = 600
+    static let surfaceRetryStepMs = 50
+
     public func withFrame<T>(_ body: (RawFrame) throws -> T) throws -> T {
         guard let display else { throw PrivateAPIError.noDisplayPort }
-        guard let raw = display.perform(NSSelectorFromString("framebufferSurface"))?.takeUnretainedValue(),
-              let surface = raw as? IOSurface else {
-            throw PrivateAPIError.surfaceUnavailable
+        var raw = display.perform(NSSelectorFromString("framebufferSurface"))?.takeUnretainedValue()
+        // Wait briefly rather than failing on the first miss. A display with
+        // nothing to draw can be between surfaces for a few tens of
+        // milliseconds, which is not the same condition as a display that has
+        // stopped rendering — and until now both said the same sentence.
+        var waitedMs = 0
+        while !(raw is IOSurface) && waitedMs < Self.surfaceRetryBudgetMs {
+            usleep(UInt32(Self.surfaceRetryStepMs) * 1000)
+            waitedMs += Self.surfaceRetryStepMs
+            raw = display.perform(NSSelectorFromString("framebufferSurface"))?.takeUnretainedValue()
+        }
+        guard let surface = raw as? IOSurface else {
+            // Two different conditions, and giving them one sentence bought two
+            // wrong diagnoses in a row: a transient miss reads exactly like the
+            // permanent wedge, so "it is the documented wedge, re-run it" was
+            // the advice both times. It said so here for the transient case.
+            throw PrivateAPIError.surfaceMissing(afterMs: waitedMs)
         }
         surface.lock(options: .readOnly, seed: nil)
         defer { surface.unlock(options: .readOnly, seed: nil) }
@@ -333,6 +366,17 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
     /// to dismiss as broken.
     public func observeChanges(_ handler: @escaping () -> Void) throws {
         guard let display else { throw PrivateAPIError.noDisplayPort }
+        // Registration is not idempotent and this used to be called as though it
+        // were. Every call minted a fresh UUID and left the previous
+        // registration live, and the recovery loop calls it on every attempt:
+        // one log from 2026-09-12 shows **670 port re-resolves**, which is up to
+        // 670 damage callbacks registered on one port, each invoked per redraw
+        // (~52/s while an app switches).
+        //
+        // That is the best available explanation for a wedge that gets worse the
+        // longer it runs and that only a device restart cures — and it means the
+        // recovery was a cause as well as a response. Unregister first, always.
+        unregisterChangeCallback()
         let sel = NSSelectorFromString("registerCallbackWithUUID:damageRectanglesCallback:")
         guard display.responds(to: sel), let imp = display.method(for: sel) else {
             throw PrivateAPIError.frameworksUnavailable("damageRectanglesCallback missing")
@@ -345,7 +389,13 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
         unsafeBitCast(imp, to: RegFn.self)(display, sel, uuid, block as AnyObject)
     }
 
-    public func detach() {
+    /// Drop the damage callback we registered, if we registered one.
+    ///
+    /// Safe to call when there is nothing to drop, which is what lets both
+    /// `observeChanges` and the port-replacement paths call it unconditionally.
+    /// Unregisters against the port it was registered on — the one currently in
+    /// `display` — so it must run *before* that reference is replaced.
+    func unregisterChangeCallback() {
         if let display, let uuid = changeUUID {
             let sel = NSSelectorFromString("unregisterDamageRectanglesCallbackWithUUID:")
             if display.responds(to: sel), let imp = display.method(for: sel) {
@@ -355,6 +405,10 @@ public final class CoreSimulatorPlatform: SimulatorPlatform {
         }
         changeCallback = nil
         changeUUID = nil
+    }
+
+    public func detach() {
+        unregisterChangeCallback()
         display = nil
         attached = nil
     }
@@ -573,6 +627,20 @@ extension CoreSimulatorPlatform {
         guard hid != nil else { throw PrivateAPIError.hidUnavailable("could not rebuild the HID client") }
     }
 
+    public func pressKey(usage: UInt32, modifiers: [UInt32]) throws {
+        let (hid, _) = try requireHID()
+        for m in modifiers { hid.key(usage: m, op: .down) }
+        defer { for m in modifiers.reversed() { hid.key(usage: m, op: .up) } }
+        // The usage-code path, not the character path. `type` sends characters
+        // and is therefore at the mercy of whichever keyboard layout iOS has
+        // active — which is why this device's own doctor warns about the fa and
+        // hy layouts. A usage code names a key *position* and is not translated,
+        // so Return is Return whatever is installed.
+        hid.key(usage: usage, op: .down)
+        Thread.sleep(forTimeInterval: 0.06)
+        hid.key(usage: usage, op: .up)
+    }
+
     public func press(_ button: HardwareButton) throws {
         let (hid, _) = try requireHID()
         guard let code = HIDKeyboard.buttonCode(button) else {

package/native/simframed/Sources/PrivateAPI/PrivateAPI.swift

@@ -52,6 +52,12 @@ public enum PrivateAPIError: Error, CustomStringConvertible {
     case deviceNotFound(String)
     case noDisplayPort
     case surfaceUnavailable
+    /// The display had no surface *right now*, after waiting. Distinct from
+    /// `surfaceUnavailable`, which is the wedge: that one never heals without a
+    /// device restart, and this one is usually gone by the next frame. One
+    /// sentence for both is what made two sessions in a row diagnose a
+    /// transient miss as the wedge and advise a re-run.
+    case surfaceMissing(afterMs: Int)
     case hidUnavailable(String)
     case simctlFailed(String)
 
@@ -62,6 +68,10 @@ public enum PrivateAPIError: Error, CustomStringConvertible {
         case .deviceNotFound(let u): return "no simulator matching \(u)"
         case .noDisplayPort: return "the device exposes no active display port"
         case .surfaceUnavailable: return "the display surface could not be read"
+        case .surfaceMissing(let ms):
+            return "no frame was available from the display for \(ms)ms"
+                + " — this is usually transient; a display that has stopped rendering says"
+                + " \"the display surface could not be read\" instead"
         case .hidUnavailable(let d): return "input is unavailable: \(d)"
         case .simctlFailed(let d): return "simctl \(d)"
         }
@@ -122,6 +132,23 @@ public protocol SimulatorPlatform: AnyObject {
     /// it is the only reliable route for content that must be exact.
     func paste(_ text: String) throws
     func press(_ button: HardwareButton) throws
+    /// Press one keyboard key by HID usage code.
+    ///
+    /// Separate from `press`, which is the hardware buttons (HOME, LOCK, SIRI).
+    /// A peer was blocked outright for want of this: half of mobile search
+    /// fields submit on the keyboard return key, and there was no way to send
+    /// one. Typing "\n" as text goes through the active keyboard layout and
+    /// mangles the field instead — measured, it turned "Coke Display" into
+    /// "Coke In Display".
+    /// Press one keyboard key, optionally while holding modifiers.
+    ///
+    /// Modifiers are usage codes too (Left GUI `0xE3`, Shift `0xE1`, Control
+    /// `0xE0`, Alt `0xE2`), held in order and released in reverse. This is what
+    /// makes clearing a field possible at all: nothing in XCUITest, Appium or
+    /// idb has a clear primitive, and the standard answer is Command-A followed
+    /// by Delete — which is layout-independent, because a modifier and Delete
+    /// are key *positions* and so is the `a` in Command-A.
+    func pressKey(usage: UInt32, modifiers: [UInt32]) throws
     /// Rebuild the HID session.
     ///
     /// Input has no feedback channel: a dispatched Indigo message reports