simframe 0.11.0 → 0.12.0

package/skills/simframe/SKILL.md

@@ -230,7 +230,10 @@ Measured against the same screen as an image: **~460 tokens of text versus
 image path degrades to base64-as-text. The text also says what is *tappable*
 and where, which an image does not.
 
-The numbers are selectors. Whatever `ui` calls `#3`, you can tap as `#3`.
+The numbers are selectors, and so are the labels beside them. Act by **name** —
+whatever `ui` calls `Weekly digest`, you can tap as `"Weekly digest"`. A `#3`
+is exact but only until the screen moves; see Selectors below for why that
+order is a correction rather than a preference.
 
 **Reach for an image only when the text genuinely cannot answer the question:**
 visual layout, colour, spacing, an animation, or something neither the
@@ -246,7 +249,7 @@ a baseline captured *before* it, so steps cannot race the UI.
 cat > /tmp/flow.json <<'JSON'
 [{"tap": "Inbox tab"},
  {"assert": {"value": "Weekly digest", "is": "visible"}},
- {"tap": "#3"},
+ {"tap": "Weekly digest"},
  {"type": {"into": "Reply", "text": "on it"}},
  {"scrollTo": "Send"},
  {"tap": "Send"},
@@ -312,6 +315,9 @@ the next call should change.
 | `#N cannot be trusted here — … this is a different screen` | the screen really did change. Read it again |
 | `autoSettle was off, so the map below was read without waiting` | the map may describe the screen *before* the last action landed |
 | `"iPhone 17 Pro" names more than one booted device` | a name cannot identify which device answered. Pass `device` with a UDID |
+| `frame 2.5s old` | the map was read from a frame that old. Usually fine after a deliberate pause; worth noticing if you expected to have just acted |
+| `WARNING this frame is Ns old — … a description of the past` | the screen may have moved on entirely. **Pass `refresh`** before believing any element below it. Reported from the field against 0.11.0: a complete 20-element map of a screen the app was not on, with nothing saying so |
+| `capture is wedged and both recoveries are spent` | the simulator has stopped rendering and neither of simframe's recoveries helped. Run `simframe revive --device=<udid>`; nothing else will work until you do |
 
 ## Every step is verified, and the verdict means something
 
@@ -388,8 +394,19 @@ as one call adds one exchange to the context instead of twelve.
 ```bash
 simframe doctor           # capture engine, input driver, a11y, OCR — each honestly
 simframe doctor --strict  # any degraded layer is a non-zero exit
+simframe input reset      # rebuild the HID session, without restarting anything
+simframe revive           # power-cycle a device that has stopped rendering
 ```
 
+**If capture has wedged**, `doctor` says so and nothing below it will work. A
+simulator driven hard for several minutes can stop rendering — `simctl
+screenshot` fails too, so it is the device and not simframe's view of it — and
+the daemon tries two recoveries, reports the state, and stops there. `simframe
+revive` does the restart in the order that matters and confirms frames are
+flowing again afterwards. It is a command rather than a behaviour on purpose:
+a capture loop that rebooted the device it was watching would be a tool
+reaching for the mains.
+
 simframe falls back when it must — the simctl capture loop instead of the
 daemon, idb instead of the in-process input and accessibility paths — but it
 never falls back quietly. If
@@ -405,6 +422,7 @@ simframe recall             # what happened in the last ~60s, as text
 simframe strip              # recent frames tiled into one image, for an animation
 simframe find "the save button"   # resolve an intent without acting on it
 simframe wait --mode=settle       # block until the screen stops reacting
+simframe supervisions             # what the local supervisor decided, and what came of it
 ```
 
 `recall` matters more than it looks: if you look up and the screen is already

package/src/actions.js

@@ -209,6 +209,62 @@ export async function runScript(
       /* instrumentation must not be able to fail a flow it is only watching */
     }
   };
+  /**
+   * What the caller is shown for each outcome, keyed by what the log stores.
+   *
+   * Two vocabularies on purpose. The log wants tokens that will still parse in
+   * six months; the result and the CLI line want English, and a test already
+   * pins those words. Mapping them here is what stops the two drifting.
+   */
+  const SHOWN = {
+    recovered: 'recovered',
+    still_failed: 'still failed',
+    stopped: 'stopped the run',
+    no_ruling: 'no ruling',
+  };
+  /**
+   * Record one ruling, in memory for the caller and on disk for the analysis.
+   *
+   * One helper rather than four call sites because the two had already drifted
+   * apart once — `from` was on the `stop` push and on none of the others, so a
+   * rule-sourced wait was indistinguishable from a model-sourced one in the
+   * only place that reported them. A ruling that reaches the caller and not the
+   * log is the state item 101a exists to end: three rulings had ever existed
+   * anywhere, and 101, 106 and 96 are all waiting on a population.
+   */
+  const noteRuling = (index, ruling, outcome, { step, expect, failure, why } = {}) => {
+    const shown = SHOWN[outcome] ?? outcome;
+    const unanswered = why?.kind
+      ? `the supervisor did not answer: ${why.kind}`
+      : 'the supervisor did not answer';
+    supervisions.push({
+      index,
+      decision: ruling?.decision ?? 'unavailable',
+      reason: ruling?.reason ?? unanswered,
+      from: ruling?.from,
+      outcome: shown,
+    });
+    try {
+      metrics.recordSupervision(udid, {
+        index,
+        step: step ? graph.actionSignature(step) : ruling?.context?.action ?? null,
+        edge: ruling?.context?.edge ?? null,
+        screen: ruling?.context?.screen ?? null,
+        decision: ruling?.decision ?? 'unavailable',
+        from: ruling?.from ?? (ruling ? 'model' : 'none'),
+        reason: ruling?.reason ?? unanswered,
+        ms: ruling?.ms,
+        stillMs: ruling?.context?.stillMs,
+        p95: ruling?.context?.p95,
+        samples: ruling?.context?.samples,
+        expect,
+        failure,
+        outcome,
+      });
+    } catch {
+      /* instrumentation must not be able to fail a flow it is only watching */
+    }
+  };
 
   const needsInput = steps.some((s) => ACTION_STEPS.has(normalizeStep(s).action));
   if (needsInput) {
@@ -309,8 +365,15 @@ export async function runScript(
         // Ask the supervisor before anything is abandoned. It sits behind the
         // hands and in front of the reasoner: first responder, not
         // decision-maker, and its whole vocabulary is wait/retry/stop.
+        // Why a consultation produced nothing, for the log. Every failure used
+        // to arrive as "the supervisor did not answer" — a timeout, a guardrail
+        // refusal and a model that was never installed reading identically.
+        const why = {};
         const ruling = await superviseFailure(deviceQuery, {
-          goal: supervise ?? flowName, step, expected: step.expect, err, options,
+          goal: supervise ?? flowName, step, expected: step.expect, err, options, udid, detail: why,
+        });
+        const ruled = (outcome) => noteRuling(i, ruling, outcome, {
+          step, expect: step.expect, failure: err.message, why,
         });
         if (ruling?.decision === 'wait' || ruling?.decision === 'retry') {
           // Both wait and retry settle first, differing only in how long.
@@ -332,10 +395,10 @@ export async function runScript(
           try {
             detail = await runStep(deviceQuery, udid, step, { screen, options, frames, focus });
             detail += ` [the local supervisor said ${ruling.decision}; it worked on the second attempt]`;
-            supervisions.push({ index: i, decision: ruling.decision, reason: ruling.reason, outcome: 'recovered' });
+            ruled('recovered');
             continue;
           } catch (again) {
-            supervisions.push({ index: i, decision: ruling.decision, reason: ruling.reason, outcome: 'still failed' });
+            ruled('still_failed');
             err = again;
           }
         } else if (!ruling && supervisor.requested(options)) {
@@ -345,10 +408,12 @@ export async function runScript(
           // model healthy. The reporter's own words — *"had I run flow 2 alone
           // I would have reported the supervisor makes no difference without
           // realising it had never run"*.
-          supervisions.push({ index: i, decision: 'unavailable', reason: 'the supervisor did not answer', outcome: 'no ruling' });
-          err.message += ' — the local supervisor was consulted and did not answer, so this failure was not judged.';
+          ruled('no_ruling');
+          err.message += ' — the local supervisor was consulted and did not answer'
+            + (why.kind ? ` (${why.kind})` : '')
+            + ', so this failure was not judged.';
         } else if (ruling?.decision === 'stop') {
-          supervisions.push({ index: i, decision: 'stop', reason: ruling.reason, from: ruling.from, outcome: 'stopped the run' });
+          ruled('stopped');
           const remaining = steps.slice(i);
           err.message += ` — the local supervisor stopped the run here.`
             + ` ${remaining.length} step(s) were not attempted.`
@@ -1123,14 +1188,34 @@ const SUPERVISOR_RETRY_MS = 900;
 /** A scroll moves at once or not at all; it does not need a transition's budget. */
 const SCROLL_SETTLE_MS = 800;
 
-async function superviseFailure(deviceQuery, { goal, step, expected, err, options }) {
+async function superviseFailure(deviceQuery, { goal, step, expected, err, options, udid, detail }) {
   if (!supervisor.requested(options)) return null;
+  // The action half of the edge is free — the step is in hand — so a rule-sourced
+  // ruling still records which action it was about even though it never reads
+  // the screen. Deliberately no `screenMap` call on this path: the rule exists
+  // to answer without one, and paying for a read to make the log prettier would
+  // slow the fast path to improve the bookkeeping.
+  const action = graph.actionSignature(step);
   const settled = deterministicRuling(err);
-  if (settled) return { decision: settled.decision, reason: settled.why, from: 'rule' };
+  if (settled) {
+    return {
+      decision: settled.decision, reason: settled.why, from: 'rule',
+      context: { action, edge: action, screen: null, stillMs: null, p95: null, samples: null },
+    };
+  }
   try {
     const map = await view.screenMap(deviceQuery, { options, refresh: false });
     const stillMs = map.identity?.state?.motion?.stillForMs;
-    return await supervisor.judge({
+    const screen = map.identity?.hash ?? null;
+    // What the graph knew about this edge at the moment of the ruling. Read
+    // here and not at analysis time on purpose: the graph keeps learning, so a
+    // p95 looked up next week is not the number this ruling was competing
+    // with, and item 101 asks exactly "would a p95 lookup have got this right".
+    let timing = null;
+    try {
+      timing = udid && map.identity ? graph.timingFor(udid, map.identity, step) : null;
+    } catch { /* an unknown edge is a fact about the graph, not a failure here */ }
+    const ruling = await supervisor.judge({
       goal,
       step: `${step.action} ${JSON.stringify(String(step.value ?? step.target ?? step.into ?? step.seek ?? '').slice(0, 60))}`,
       expected,
@@ -1139,7 +1224,21 @@ async function superviseFailure(deviceQuery, { goal, step, expected, err, option
       stillMs,
       note: stillFillingIn(map.identity?.entry),
       options,
+      detail,
     });
+    if (!ruling) return null;
+    return {
+      ...ruling,
+      from: ruling.from ?? 'model',
+      context: {
+        action,
+        edge: screen ? `${String(screen).slice(0, 12)}:${action}` : action,
+        screen,
+        stillMs,
+        p95: timing?.p95 ?? null,
+        samples: timing?.samples ?? null,
+      },
+    };
   } catch {
     return null;
   }

package/src/cli.js

@@ -3,7 +3,7 @@ import fs from 'node:fs';
 import os from 'node:os';
 import path from 'node:path';
 import { runDaemon, DEFAULTS } from './daemon.js';
-import { bootedDevices, capabilitiesFor, listDevices, PLATFORMS, resolveDevice, screenshot, toolchainChecks } from './platform/index.js';
+import { bootedDevices, capabilitiesFor, listDevices, PLATFORMS, resolveDevice, restartDevice, screenshot, toolchainChecks } from './platform/index.js';
 import * as actions from './actions.js';
 import * as analyze from './analyze.js';
 import * as api from './index.js';
@@ -47,6 +47,8 @@ const USAGE = `simframe — always-warm iOS Simulator frames
   simframe baseline list             recorded runs per flow, and what is committed
   simframe hpi     [device]          Human Parity Index, per flow and overall
   simframe escalations [device]      why simframe handed decisions back, by reason
+  simframe supervisions [device]     local supervisor rulings, and what came of each
+  simframe revive [device]           power-cycle a wedged device: stop, shutdown, boot, start, reset input
                                      (--session=<id> narrows to one agent; the
                                      ids are listed in the output. SIMFRAME_SESSION
                                      names one, but only at process start — an
@@ -56,10 +58,12 @@ const USAGE = `simframe — always-warm iOS Simulator frames
                                      (--strict, or SIMFRAME_STRICT=1, makes any
                                       degraded layer a non-zero exit)
 
-Selectors — anywhere a control is named
-  #3            the number \`simframe ui\` gave it. Cheapest, unambiguous.
-  "Save"        a label or a phrase, resolved by intent (verbs, typos, synonyms)
-  @120,400      raw point coordinates
+Selectors — anywhere a control is named, best first
+  "Save"        a label or a phrase, resolved by intent (verbs, typos, synonyms,
+                icon-only controls by their common name). Start here.
+  #3            the number \`simframe ui\` gave it. Exact, but only inside the
+                round trip that numbered it — the screen moves and it does not.
+  @120,400      raw point coordinates. Last resort: it cannot tell you it missed.
 
 Measuring against a human — the Human Parity Index
 
@@ -378,6 +382,51 @@ async function main() {
       return;
     }
 
+    // The power cycle, when the narrow remedies are spent.
+    //
+    // Deliberately a command and not a behaviour. The capture loop tries two
+    // things — re-resolve the display port, then rebind the device — and then
+    // reports `stalled` and stops, because a capture loop that rebooted the
+    // device it was watching would be a tool reaching for the mains when a
+    // reading looks wrong. Restarting is the operator's call.
+    //
+    // But it was the operator's call *and* their four commands, remembered from
+    // a handoff note: stop the daemon, shut the device down, boot it and wait,
+    // start capture, rebuild the HID session. Done by hand three times in one
+    // afternoon, in that order, because any other order leaves a daemon holding
+    // a dead device. So the tool knows the order now; the decision is still
+    // yours.
+    case 'revive': {
+      const dev = await resolveDevice(device);
+      const say = (line) => { if (!flags.json) console.log(line); };
+      const steps = [];
+      const did = async (what, fn) => {
+        try { await fn(); steps.push({ step: what, ok: true }); say(`  ok   ${what}`); } catch (err) {
+          steps.push({ step: what, ok: false, error: err.message });
+          say(`  ..   ${what} — ${err.message.split('\n')[0]}`);
+        }
+      };
+      say(`reviving ${dev.name}`);
+      // Forced: the point of this command is that the device is wedged, so
+      // something is certainly still holding it.
+      await did('stopped the daemon', async () => { api.stopDaemon(dev.udid, { force: true }); });
+      // Through the boundary, which is the whole point of the boundary: the
+      // first version of this shelled out to `xcrun` from here and the test
+      // that forbids it failed immediately, correctly.
+      await did('restarted the device, and waited for the boot to finish',
+        () => restartDevice(dev.udid));
+      await did('started capture', () => api.ensureDaemon(dev.udid));
+      await did('rebuilt the HID session', () => input.resetSession(dev.udid));
+      const health = await api.getState(dev.udid).then((s) => s?.state ?? null).catch(() => null);
+      const alive = Boolean(health?.hash);
+      emit(flags, { ok: alive, device: dev.udid, steps }, alive
+        ? `\n${dev.name} is producing frames again`
+        : `\n${dev.name} is still not producing frames. This is past what simframe can do —`
+          + ' check Simulator.app is not showing an error, and see docs/DEFERRED.md item 95.');
+      if (!alive) process.exitCode = 1;
+      return;
+    }
+
     case 'status': {
       const udids = device
         ? [(await resolveDevice(device)).udid]
@@ -1027,6 +1076,37 @@ async function main() {
       return;
     }
 
+    case 'supervisions': {
+      const dev = await resolveDevice(flags.device);
+      const records = metrics.readSupervisions(dev.udid, { limit: flags.last ? num(flags.last) : undefined });
+      const b = metrics.supervisionBreakdown(records);
+      if (flags.out) store.writeAtomic(String(flags.out), `${JSON.stringify({ ...b, records }, null, 2)}\n`);
+      emit(flags, { ...b, records: flags.verbose ? records : undefined }, [
+        `${b.total} supervisor ruling${b.total === 1 ? '' : 's'} on ${dev.name}`,
+        b.total ? '' : 'Nothing has been judged on this device yet. The supervisor is off unless'
+          + ' SIMFRAME_SUPERVISOR=apple, and a ruling is only recorded when a step actually fails.',
+        ...Object.entries(b.decision_to_outcome)
+          .sort((a, c) => c[1] - a[1])
+          .map(([k, n]) => `  ${k.padEnd(28)} ${String(n).padStart(4)}`),
+        b.total ? '' : null,
+        b.total ? `sourced: ${Object.entries(b.by_from).map(([k, n]) => `${k} ${n}`).join(', ')}` : null,
+        b.median_latency_ms != null ? `median latency: ${b.median_latency_ms}ms` : null,
+        // Said out loud, because the first version of item 101 claimed its
+        // measurement ran "on logs we already have" when nothing persisted a
+        // ruling at all. This line is what stops that claim being made twice.
+        b.total
+          ? `edges the graph had timed: ${b.p95_known}/${b.total}`
+            + (b.p95_unknown
+              ? ` — ${b.p95_unknown} ruling(s) are on edges with no p95, so they cannot take part in 101's comparison`
+              : '')
+          : null,
+        b.sessions.length > 1
+          ? `WARNING ${b.sessions.length} sessions are pooled here; two agents on one device write one file`
+          : null,
+      ].filter((l) => l !== null).join('\n'));
+      break;
+    }
+
     case 'escalations': {
       const dev = await resolveDevice(flags.device);
       const records = metrics.readEscalations(dev.udid, { limit: flags.last ? num(flags.last) : undefined });

package/src/fingerprint.js

@@ -48,8 +48,32 @@ import * as regions from './regions.js';
  *     extends upward while the rows above keep being key-shaped, which page
  *     content is not. Any screen fingerprinted with a keyboard up hashes
  *     differently, so the stored graph and maps must go again.
+ *
+ * 7 — a screen whose own name is an iOS **large title** had no name at all in
+ *     its identity. Chrome labels are the only text these tokens keep, and a
+ *     large title is drawn tight against the content it heads — 79 pt of inset
+ *     above it and 5.3 pt below, against a boundary bar of 66.5 — so the top
+ *     chrome detector, which looks for the gap *beneath* a bar, never found it
+ *     and the title was discarded as content. Measured on the Settings root in
+ *     both sensor modes: **0 named tokens**, and the same for Contacts and
+ *     Reminders. On a hosted runner two sparse nameless readings then matched
+ *     exactly and one hash stood for two different screens. `regions.bands`
+ *     now finds a large title by the inset above it; every affected screen
+ *     hashes differently, so stored graphs and maps go again.
+ *
+ * 8 — the same rule, keyed on the wrong thing. Its "is this a real inset"
+ *     test compared the gap above the title against the screen's **median row
+ *     gap**, so whether a system-drawn title counted as chrome depended on how
+ *     many rows happened to sit below it. Caught on the React Native testbed's
+ *     first day, with two screens of one app: a list of 24 rows has a median
+ *     gap of 0 and the rule fired, a list of 4 above a tab bar has a median gap
+ *     of **414** and it did not. Same title, same 62.9pt inset, opposite
+ *     answers — so one screen carried a name and the other did not, and the
+ *     graph merged them. Both bounds are absolute now. Screens that were
+ *     missed at 7 hash differently at 8, and unlike a stale hash that matches
+ *     nothing, these matched the *wrong* thing.
  */
-export const TOKEN_RULES_VERSION = 6;
+export const TOKEN_RULES_VERSION = 8;
 
 /** Frames are quantised to this, so sub-pixel drift and a nudged row do not matter. */
 export const GRID = 24;

package/src/graph.js

@@ -258,6 +258,28 @@ export function nearestScreen(udid, screen, { threshold = SIMILARITY_THRESHOLD }
   for (const node of nodes) {
     for (const f of fingerprintsOf(node)) {
       if (!f.tokens?.length) continue;
+      // A screen that says it is called something else is not this screen.
+      //
+      // Similarity weighs every token equally, and a chrome label is not an
+      // equal token — it is the only text in a fingerprint and the only thing
+      // that distinguishes two screens of the same shape. `fingerprint.js` has
+      // said so in a comment since it was written: "two list screens with
+      // identical structure differ by their title, and nothing else says so".
+      // Nothing enforced it.
+      //
+      // Measured on the RN testbed, which is what found this: a list of plants
+      // and a list of forms, each a large title over full-width rows above a
+      // tab bar, scored **0.50** against a 0.36 threshold and became one node.
+      // Both were correctly named by then; the name was simply outvoted, being
+      // one token of a union of six. The graph then offered one screen's
+      // controls on the other, which is the failure `verify` exists to stop.
+      //
+      // Both sides must actually carry names for this to apply. A reading whose
+      // tree did not answer has no names through no fault of the screen's, and
+      // the two sensors are already known to disagree about 0.33-0.47 of a
+      // token set — so "one has names, the other does not" is a fact about the
+      // sensors and must not be read as a fact about identity.
+      if (disagreeOnName(f.tokens, key.tokens)) continue;
       const s = fingerprint.similarity(f.tokens, key.tokens);
       if (s > bestSimilarity) {
         bestSimilarity = s;
@@ -268,6 +290,31 @@ export function nearestScreen(udid, screen, { threshold = SIMILARITY_THRESHOLD }
   return best && bestSimilarity >= threshold ? { node: best, similarity: bestSimilarity } : null;
 }
 
+/** The chrome labels in a token set — the only text a fingerprint keeps. */
+function namesIn(tokens) {
+  const out = new Set();
+  for (const t of tokens ?? []) {
+    const open = t.indexOf('"');
+    if (open < 0) continue;
+    const close = t.lastIndexOf('"');
+    if (close > open) out.add(t.slice(open + 1, close));
+  }
+  return out;
+}
+
+/**
+ * Do these two readings name themselves, and name themselves differently?
+ *
+ * Only a positive disagreement counts. Silence on either side is not evidence.
+ */
+function disagreeOnName(a, b) {
+  const left = namesIn(a);
+  const right = namesIn(b);
+  if (!left.size || !right.size) return false;
+  for (const name of left) if (right.has(name)) return false;
+  return true;
+}
+
 /**
  * Teach a node that it also looks like this.
  *

package/src/index.js

@@ -303,6 +303,18 @@ export const STALE_FRAME_MS = 2500;
  */
 export const MEMORY_SETTLE_MS = 250;
 
+/**
+ * How well a relabelled ref must match before it is acted on.
+ *
+ * Above `matching.MINIMUM_SCORE` (0.45) on purpose: that floor is for a label
+ * the caller wrote, and this is a label simframe substituted after refusing
+ * their `#n`. A fuzzy name match returns `similarity * 0.72` and a prefix match
+ * is scaled by its coverage, so neither can reach 0.8 on the name alone — which
+ * makes this "the label matched nearly exactly", not a tuned constant. The
+ * recovery that made this necessary scored 0.64.
+ */
+export const RELABEL_MIN_SCORE = 0.8;
+
 /** Below this a "change" is a clock digit or a caret, not a new screen. */
 export const MINOR_CHANGE = 0.004;
 export const MAJOR_CHANGE = 0.03;
@@ -1251,6 +1263,35 @@ async function locateWith(
         second.staleLabel = err.staleLabel;
         throw second;
       }
+      // A recovery is held to a higher bar than the lookup it stands in for,
+      // and to one the caller never has to think about.
+      //
+      // `MINIMUM_SCORE` (0.45) is the bar for a label the caller actually
+      // wrote. This label is one *we substituted on their behalf* after
+      // refusing their `#n`, so a weak match here is not "close enough" — it is
+      // us choosing a target nobody named. The reported wrong answer scored
+      // **0.64** and cleared the ordinary floor comfortably.
+      //
+      // 0.8 is structural rather than fitted to that incident: a fuzzy name
+      // match returns `similarity * 0.72` and a prefix match is scaled by its
+      // coverage, so neither reaches 0.8 on the name alone. Only a near-exact
+      // name does. And the region check needs no tuned number at all — if the
+      // map would not offer this target, a recovery may not silently pick it.
+      const region = again.target?.region ?? 'content';
+      const weak = Number.isFinite(again.score) && again.score < RELABEL_MIN_SCORE;
+      if (weak || !regions.offerable(region)) {
+        err.message = `#${selector.ref} cannot be trusted here, and "${err.staleLabel}" was not`
+          + ' safely re-findable either:'
+          + (weak ? ` the best match scored ${again.score.toFixed(2)}, below the ${RELABEL_MIN_SCORE} a`
+            + ' relabel needs (a number you did not ask for may not become a tap on a guess).' : '')
+          + (!regions.offerable(region) ? ` the best match sits in the ${region}, which sim_ui does not`
+            + ' offer as something to act on.' : '')
+          + ' Read the screen again (sim_ui) and name the target.';
+        err.staleRef = true;
+        err.staleLabel = err.staleLabel;
+        err.relabelRefused = { score: again.score ?? null, region };
+        throw err;
+      }
       return {
         ...again,
         from: 'ref-relabelled',

package/src/localhelper.js

@@ -94,7 +94,11 @@ export function lineServer({ ensureBinary, what }) {
           try { msg = JSON.parse(line); } catch { continue; }
           if (!settled) {
             settled = true;
-            if (msg.ready) resolve({ ok: true, child, waiters });
+            // The ready line may carry facts about the helper worth keeping —
+            // the model's context window, for one, which used to be a constant
+            // we repeated in comments. Passed through rather than parsed here,
+            // because this file knows about lines and not about models.
+            if (msg.ready) resolve({ ok: true, child, waiters, hello: msg });
             else resolve({ ok: false, reason: msg.unavailable ?? `the ${what} did not become ready` });
             continue;
           }
@@ -145,7 +149,9 @@ export function lineServer({ ensureBinary, what }) {
     },
     async status() {
       const live = await open();
-      return live?.ok ? { ok: true } : { ok: false, reason: live?.reason ?? 'unavailable' };
+      return live?.ok
+        ? { ok: true, hello: live.hello ?? {} }
+        : { ok: false, reason: live?.reason ?? 'unavailable' };
     },
     close() {
       try { session?.child?.kill(); } catch { /* already gone */ }

package/src/mcp.js

@@ -78,13 +78,22 @@ const deviceProp = {
 };
 
 /**
- * One selector grammar everywhere.
+ * One selector grammar everywhere, and the order is the recommendation.
  *
- * `#3` is the cheapest thing a caller can say and the least ambiguous, because
- * simframe numbered it; a bare phrase is resolved by intent, which is more
- * forgiving and occasionally has to ask which one was meant.
+ * It used to lead with `#3` and call it "cheapest and unambiguous". Four peer
+ * rounds running reported the opposite: intent resolution worked every time,
+ * while refs renumbered underneath them and were only safe inside the round
+ * trip that issued them. The README was corrected and these descriptions were
+ * not, which is the half a caller actually reads.
+ *
+ * "Unambiguous" was also the wrong word for it. A ref is exact about which
+ * element simframe meant and says nothing about whether that element is still
+ * there — the failure mode that needed `staleKind` to tell a moved layout from
+ * a different screen, and then a score floor and a region check on top of that
+ * before a relabelled ref could be trusted. A label carries its own evidence;
+ * a number carries none.
  */
-const SELECTOR = 'Selector: "#3" (a number from the last screen map — cheapest and unambiguous), a label or phrase like "Save" or "the Assets tab" (resolved by intent), or "@120,400" for raw point coordinates.';
+const SELECTOR = 'Selector: a label or phrase like "Save" or "the Assets tab" or "back" (resolved by intent — verbs, typos, synonyms, icon-only controls: START HERE), "#3" (a number from the last screen map — exact, but only inside the round trip that numbered it), or "@120,400" for raw point coordinates (last resort: it cannot tell you it missed).';
 
 const selectorProp = (what = 'What to act on') => ({
   sel: { type: 'string', description: `${what}. ${SELECTOR}` },
@@ -94,7 +103,7 @@ const TOOLS = [
   {
     name: 'sim_ui',
     description:
-      'READ THE SCREEN as text: every element numbered, with region, type, label, state, contents and tap point, plus which screen this is and what simframe knows about it. A tenth the cost of a screenshot and more useful, because it says what is tappable and where. Whatever it calls #3, you can tap as "#3". Start here, never with sim_look.',
+      'READ THE SCREEN as text: every element numbered, with region, type, label, state, contents and tap point, plus which screen this is and what simframe knows about it. A tenth the cost of a screenshot and more useful, because it says what is tappable and where. Act on what it shows by NAME — whatever it calls "General", you can tap as "General"; the #3 numbers are exact but only until the screen moves. Start here, never with sim_look.',
     inputSchema: {
       type: 'object',
       properties: {
@@ -123,7 +132,7 @@ const TOOLS = [
         steps: {
           type: 'array',
           description:
-            'Ordered steps. Every selector below accepts "#3" | "Save" | "@120,400". Act: {"tap":"Save"} (add "index" if a label is ambiguous), {"type":{"into":"Name","text":"Fryer 3"}}, {"paste":{"into":"Notes","text":"long text"}}, {"clear":"Notes"} to empty a field and "clear":true on a type/paste to replace rather than append (drop "into" to type into whatever already has focus, which is how you follow a browser next-field chevron — nothing can be read back then, and the step says so), {"scroll":"down"}, {"scrollTo":"Delete account"}, {"swipe":{"from":[x,y],"to":[x,y]}}, {"button":"HOME"}, {"key":"return"} (the keyboard return/enter key, which is how a mobile search field submits — also escape, tab, space, backspace, and the arrows), {"launch":{"value":"com.example.app","relaunch":true,"args":["-uiTest","1"]}}, {"openUrl":"myapp://x"}, {"permission":{"value":"photos","grant":"grant","bundleId":"com.example.app"}}. Check: {"assert":{"value":"Saved","is":"visible"}} (also gone | enabled | disabled | value with "equals"), {"waitFor":{"value":"Saved","timeoutMs":5000}}, {"settle":{"stableMs":600}}, {"pause":300}. Recover without a round trip: add "or" to any step for fallback selectors tried locally — {"tap":"Save","or":["Done","Confirm"]} — and {"seek":"change username","budget":6} explores for something not on this screen: it OPENS containers (a real action — state changes), checks, and returns to where it started, refusing to open anything that commits, abandons or answers. It does not tap the target; it leaves you on the screen where the target resolves so you tap it next. Do not point it into a flow whose progress you cannot afford to lose. A long screen is only knowable a viewport at a time, so {"sweep":"all","fill":{"Last Name":"Asadi","Email":"a@b.c"}} goes to the top, then reads and fills section by section to the bottom — filling each field while it is on screen, which beats finding one and scrolling back. Add "from":"here" to sweep down from where you are. It reports which section each element was in, what it filled, and what it never found at any scroll position. Prefer it to scrollTo on forms and long lists. Brief the supervisor from the plan: top-level "supervise" is standing guidance for the whole batch ("lists here render a count header before rows; REVIEW stays disabled until a provider is chosen") and per-step "expect" adds to it. When it stops a run the result names the steps it did not attempt — re-issue them with a corrected "supervise" note if the judgement was wrong.',
+            'Ordered steps. Every selector below accepts "Save" | "#3" | "@120,400", in that order of preference. Act: {"tap":"Save"} (add "index" if a label is ambiguous), {"type":{"into":"Name","text":"Fryer 3"}}, {"paste":{"into":"Notes","text":"long text"}}, {"clear":"Notes"} to empty a field and "clear":true on a type/paste to replace rather than append (drop "into" to type into whatever already has focus, which is how you follow a browser next-field chevron — nothing can be read back then, and the step says so), {"scroll":"down"}, {"scrollTo":"Delete account"}, {"swipe":{"from":[x,y],"to":[x,y]}}, {"button":"HOME"}, {"key":"return"} (the keyboard return/enter key, which is how a mobile search field submits — also escape, tab, space, backspace, and the arrows), {"launch":{"value":"com.example.app","relaunch":true,"args":["-uiTest","1"]}}, {"openUrl":"myapp://x"}, {"permission":{"value":"photos","grant":"grant","bundleId":"com.example.app"}}. Check: {"assert":{"value":"Saved","is":"visible"}} (also gone | enabled | disabled | value with "equals"), {"waitFor":{"value":"Saved","timeoutMs":5000}}, {"settle":{"stableMs":600}}, {"pause":300}. Recover without a round trip: add "or" to any step for fallback selectors tried locally — {"tap":"Save","or":["Done","Confirm"]} — and {"seek":"change username","budget":6} explores for something not on this screen: it OPENS containers (a real action — state changes), checks, and returns to where it started, refusing to open anything that commits, abandons or answers. It does not tap the target; it leaves you on the screen where the target resolves so you tap it next. Do not point it into a flow whose progress you cannot afford to lose. A long screen is only knowable a viewport at a time, so {"sweep":"all","fill":{"Last Name":"Asadi","Email":"a@b.c"}} goes to the top, then reads and fills section by section to the bottom — filling each field while it is on screen, which beats finding one and scrolling back. Add "from":"here" to sweep down from where you are. It reports which section each element was in, what it filled, and what it never found at any scroll position. Prefer it to scrollTo on forms and long lists. Brief the supervisor from the plan: top-level "supervise" is standing guidance for the whole batch ("lists here render a count header before rows; REVIEW stays disabled until a provider is chosen") and per-step "expect" adds to it. When it stops a run the result names the steps it did not attempt — re-issue them with a corrected "supervise" note if the judgement was wrong.',
           items: { type: 'object' },
         },
         autoSettle: {