verikun 0.27.0 → 0.28.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -530,10 +530,12 @@ device that failed and is now on another one. What that means depends on the lin
530
530
  the failure was real on A, and B has none of the state your flow built up. Start the
531
531
  flow again from the top if you want it on B.
532
532
 
533
- The server rules the bad device out until it is power-cycled;
534
- `vk devices --server <url>` shows why in its `NOTE` column. On a pool the replacement
535
- joins the pool, so capacity holds and the last device is never shed, so its own error
536
- keeps reaching you rather than a bare "no device attached".
533
+ The server rules the bad device out; `vk devices --server <url>` shows why in its `NOTE`
534
+ column, and a pooled server re-adopts a device that comes back within a minute. A device
535
+ that is still attached keeps its place and is simply dealt last, so its own error keeps
536
+ reaching you rather than a bare "no device attached". A device that is **gone** leaves the
537
+ pool — so `capacity` can drop mid-job, and an install can come back `exit 0` having skipped
538
+ it. That is a success: nothing can be dealt a device running the previous build.
537
539
 
538
540
  ## The device is missing or wedged
539
541
 
@@ -613,7 +615,7 @@ owns the redaction and the review-first flow.
613
615
  ## Gotchas
614
616
 
615
617
  - **Prepare the device once** for reliable dumps: `vk device prep` (a physical device
616
- needs `--device <serial>`). Live animations can make `vk ui` flaky (it already retries 3×).
618
+ needs `--device <serial>`). Live animations can make `vk ui` flaky.
617
619
  - **A slept device returns the LOCK SCREEN, not an error.** The dump succeeds and hands
618
620
  back `com.android.systemui` — so selectors miss for a reason unrelated to the app.
619
621
  verikun detects this, wakes the device and clears a *swipe* lock automatically; on a
package/CHANGELOG.md CHANGED
@@ -6,6 +6,43 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.28.0] - 2026-09-14
10
+
11
+ A device that has gone now leaves a pooled `vk server` instead of being dealt out until
12
+ somebody notices.
13
+
14
+ ### Fixed
15
+ - **`vk server --devices`** drops a device that is gone from the pool instead of dealing it
16
+ forever; the sweep readmits it when it returns. ([#139])
17
+ - **`vk install --server`** exits `0` when some pooled devices take the build, naming the rest
18
+ as `skipped`; only a build that fails everywhere is an error. ([#139])
19
+ - **A `--server` call that outlives Node's 300s fetch ceiling** now says so, and says the clock
20
+ was the client's, instead of a bare `fetch failed` the suite read as a dead device. ([#139])
21
+
22
+ ### Changed
23
+ - **`/v1/health` `capacity` can now drop mid-job** on a pooled server, to `0`. A plain
24
+ `vk server` still keeps its only device and answers with that device's own error. ([#139])
25
+ - **`POST /v1/install`** answers `200 {devices, skipped}` where it used to answer `500` on a
26
+ partial failure. Older clients see a success with an unknown field. ([#139])
27
+
28
+ [#139]: https://github.com/ddikman/verikun/issues/139
29
+
30
+ ## [0.27.1] - 2026-09-14
31
+
32
+ A hierarchy read the device killed is now waited out instead of ending the test.
33
+
34
+ ### Fixed
35
+ - **Hierarchy reads** ride out a dump the device killed, for the caller's full wait budget,
36
+ instead of aborting after three fast retries. ([#137])
37
+ - **`assert --gone` / `wait --gone`** no longer count a killed read as an absence; a window of
38
+ only killed reads exits `3` instead of reporting a miss. ([#137])
39
+
40
+ ### Changed
41
+ - **A killed dump** names memory pressure and is never failed over or retired — the device is
42
+ busy, not broken. ([#137])
43
+
44
+ [#137]: https://github.com/ddikman/verikun/issues/137
45
+
9
46
  ## [0.27.0] - 2026-09-13
10
47
 
11
48
  ### Added
@@ -89,8 +89,10 @@ exports.DEFAULT_GUARD_SETTLE_MS = 1500;
89
89
  /** Re-dump cadence inside a guard's settle window. */
90
90
  const GUARD_POLL_MS = 150;
91
91
  /**
92
- * How long a guard keeps looking at a screen it cannot read at all, when the reason is
93
- * NoWindowError the app was force-stopped or is mid-launch and has genuinely not drawn.
92
+ * How long a guard keeps looking at a screen it cannot read at all, when the reason is one
93
+ * that CLEARS ON ITS OWN (`TransientReadError`): the app force-stopped or mid-launch and not
94
+ * yet drawn (`NoWindowError`), or the dumper SIGKILLed under memory pressure
95
+ * (`DumpKilledError`, issue #137).
94
96
  *
95
97
  * Separate from `settleMs`, which answers "how long before I believe this selector is
96
98
  * absent?". This answers "how long before I believe there is no screen to ask?" — a
@@ -113,7 +115,7 @@ const GUARD_POLL_MS = 150;
113
115
  * Not configurable on purpose: a dial here is one more thing to explain, and every value a
114
116
  * user might pick is worse than the measurement.
115
117
  */
116
- const NO_WINDOW_GRACE_MS = 10_000;
118
+ const TRANSIENT_READ_GRACE_MS = 10_000;
117
119
  /** Consecutive identical screen snapshots before a loop is believed to be stuck.
118
120
  *
119
121
  * This check is a TIME SAVER and nothing more. A loop already fails when its exit
@@ -205,8 +207,9 @@ async function runPlan(plan, deps) {
205
207
  * probe a loop-exit check needs.
206
208
  * So one dump attempt always happens regardless of the window.
207
209
  *
208
- * A THIRD clock sits beside both: a screen that cannot be read because the app has not
209
- * drawn (NoWindowError) is retried against NO_WINDOW_GRACE_MS, not against `settleMs`.
210
+ * A THIRD clock sits beside both: a screen that is not READABLE AT ALL for a reason that
211
+ * clears on its own (a TransientReadError) is retried against TRANSIENT_READ_GRACE_MS,
212
+ * not against `settleMs`.
210
213
  * That is deliberately independent — "is this selector absent?" and "is there a screen to
211
214
  * ask at all?" are different questions, and only the second one aborts the run.
212
215
  *
@@ -229,10 +232,10 @@ async function runPlan(plan, deps) {
229
232
  if (deps.platform)
230
233
  (0, state_support_1.assertStateSupported)(sel, deps.platform);
231
234
  const deadline = Date.now() + Math.max(0, settleMs);
232
- // A screen that cannot be read AT ALL gets its own, longer clock — see NO_WINDOW_GRACE_MS.
235
+ // A screen that cannot be read AT ALL gets its own, longer clock — see TRANSIENT_READ_GRACE_MS.
233
236
  // Clamped by the run deadline so a guard can never push a run past --timeout: the grace
234
237
  // exists to spend budget the caller already has, never to invent more.
235
- const noWindowDeadline = Math.min(Date.now() + NO_WINDOW_GRACE_MS, deps.deadline ?? Infinity);
238
+ const transientDeadline = Math.min(Date.now() + TRANSIENT_READ_GRACE_MS, deps.deadline ?? Infinity);
236
239
  // A non-zero window must buy at least one SECOND look, independent of the clock.
237
240
  // Measured on emulator-5554: one uiautomator dump costs ~2.4s, which already exceeds
238
241
  // a 1.5s window — so a purely time-boxed loop returns after a single dump and the
@@ -280,7 +283,7 @@ async function runPlan(plan, deps) {
280
283
  // One successful read — even an empty tree — and the ordinary semantics resume exactly:
281
284
  // settleMs=0 is still a single-shot probe. It must never make a merely ABSENT selector
282
285
  // more patient, or every guard silently costs 10s.
283
- if (!everRead && lastErr instanceof errors_1.NoWindowError && Date.now() < noWindowDeadline) {
286
+ if (!everRead && lastErr instanceof errors_1.TransientReadError && Date.now() < transientDeadline) {
284
287
  await (0, wait_1.sleep)(GUARD_POLL_MS);
285
288
  continue;
286
289
  }
@@ -10,6 +10,7 @@
10
10
  // local one. Recording stays a caller concern: this module never touches ./.verikun.
11
11
  Object.defineProperty(exports, "__esModule", { value: true });
12
12
  exports.describeStatus = describeStatus;
13
+ exports.transportReason = transportReason;
13
14
  exports.pingServer = pingServer;
14
15
  exports.remoteDeviceOp = remoteDeviceOp;
15
16
  exports.remoteDeviceList = remoteDeviceList;
@@ -18,16 +19,36 @@ const node_fs_1 = require("node:fs");
18
19
  const node_crypto_1 = require("node:crypto");
19
20
  const node_path_1 = require("node:path");
20
21
  const errors_1 = require("../errors");
22
+ const output_1 = require("../output");
21
23
  const rpc_1 = require("../rpc");
24
+ /**
25
+ * The ceiling NONE of the per-call timeouts below can exceed, whatever they say.
26
+ *
27
+ * Node's global `fetch` is undici, whose `headersTimeout` and `bodyTimeout` both default to
28
+ * 300s, and there is no dependency-free way to raise them: a `dispatcher` needs `undici`
29
+ * itself, which is bundled but not importable. The `AbortController` below is therefore a
30
+ * FLOOR on how long a call may take, never a ceiling — `EXEC_TIMEOUT_MS` says 600s and gets
31
+ * 300s.
32
+ *
33
+ * MEASURED on Node v20.20.2 against a server that held its headers for 310s: the fetch
34
+ * rejected at 301s with `TypeError: fetch failed`, cause `HeadersTimeoutError`, code
35
+ * `UND_ERR_HEADERS_TIMEOUT`. The bare `fetch failed` is the whole problem — `describeStatus`
36
+ * never sees it, the suite reads the resulting exit 3 as the DEVICE being unreachable, and a
37
+ * healthy phone gets retired for a client-side clock. Named in `request` below so it says so.
38
+ */
39
+ const FETCH_HEADERS_CEILING_MS = 300_000;
22
40
  // Per-call ceilings. exec is generous: a single leaf may legitimately block for its
23
- // whole auto-wait window or an explicit `wait --timeout`, plus device time.
41
+ // whole auto-wait window or an explicit `wait --timeout`, plus device time. Anything here
42
+ // above FETCH_HEADERS_CEILING_MS is aspirational — see that constant.
24
43
  const HEALTH_TIMEOUT_MS = 10_000;
25
44
  const ELEMENTS_TIMEOUT_MS = 60_000;
26
45
  const EXEC_TIMEOUT_MS = 10 * 60_000;
27
46
  const INSTALL_TIMEOUT_MS = 15 * 60_000;
28
47
  const DEVICE_LIST_TIMEOUT_MS = 30_000;
29
- // Above the server's own 4-minute boot ceiling, so the SERVER reports why a boot
30
- // timed out rather than the client aborting into a generic "timed out after 300s".
48
+ // Meant to sit above the server's own 4-minute boot ceiling, so the SERVER reports why a
49
+ // boot timed out rather than the client aborting first. It is exactly AT
50
+ // FETCH_HEADERS_CEILING_MS, so a boot that runs the full four minutes and then some is a
51
+ // photo finish — which is survivable only because `transportReason` now names the loser.
31
52
  const DEVICE_START_TIMEOUT_MS = 5 * 60_000;
32
53
  const DEVICE_STOP_TIMEOUT_MS = 60_000;
33
54
  const trimUrl = (url) => url.replace(/\/+$/, '');
@@ -66,6 +87,28 @@ function describeStatus(status, body, url) {
66
87
  }
67
88
  return new errors_1.CliError(`verikun server error ${status} at ${url}${detail}`, exitCode);
68
89
  }
90
+ /**
91
+ * Why the transport failed, in words an operator can act on. PURE — exported for the tests.
92
+ *
93
+ * The undici arm is the one that earns its keep. `fetch` reports its own header/body
94
+ * timeouts as a bare `TypeError: fetch failed` with the real cause one level down, and that
95
+ * string is indistinguishable from a server that is genuinely unreachable — which is how a
96
+ * five-minute install came to look like a dead phone, and how a lane came to be retired for
97
+ * it. Say which clock ran out, and say whose it was.
98
+ */
99
+ function transportReason(e, timeoutMs) {
100
+ const ex = e;
101
+ if (ex?.name === 'AbortError')
102
+ return `timed out after ${Math.round(timeoutMs / 1000)}s`;
103
+ const code = ex?.cause?.code;
104
+ if (code === 'UND_ERR_HEADERS_TIMEOUT' || code === 'UND_ERR_BODY_TIMEOUT') {
105
+ const what = code === 'UND_ERR_HEADERS_TIMEOUT' ? 'send a response' : 'finish its response';
106
+ return (`the server did not ${what} within ${Math.round(FETCH_HEADERS_CEILING_MS / 1000)}s — ` +
107
+ "this is Node's own fetch ceiling on the CLIENT, not the device. " +
108
+ 'The server may still be working; check its log before blaming the device');
109
+ }
110
+ return ex?.message ?? String(e);
111
+ }
69
112
  async function readBody(res) {
70
113
  try {
71
114
  return (await res.json());
@@ -103,8 +146,7 @@ class RemoteTransport {
103
146
  });
104
147
  }
105
148
  catch (e) {
106
- const reason = e.name === 'AbortError' ? `timed out after ${Math.round(timeoutMs / 1000)}s` : e.message;
107
- throw new errors_1.CliError(`cannot reach verikun server at ${url} (${reason})`, 3);
149
+ throw new errors_1.CliError(`cannot reach verikun server at ${url} (${transportReason(e, timeoutMs)})`, 3);
108
150
  }
109
151
  finally {
110
152
  clearTimeout(timer);
@@ -229,6 +271,14 @@ function createRemoteBackend(opts, health) {
229
271
  // the build DID land — on a different device than the one we started with.
230
272
  if (res.deviceChanged)
231
273
  opts.onDeviceChange?.(res.deviceChanged);
274
+ // A PARTIAL install is a success, and it must not be a silent one: capacity just
275
+ // dropped, and the operator's next question is which phone to go and look at.
276
+ if (res.skipped?.length) {
277
+ (0, output_1.err)(`[verikun] server installed on ${(res.devices ?? []).join(', ') || '(none)'}; ` +
278
+ `${res.skipped.length} device(s) could not take this build and left the pool — ` +
279
+ res.skipped.map((s) => `${s.serial} (${s.reason})`).join('; '));
280
+ opts.onInstallSkipped?.(res.skipped);
281
+ }
232
282
  },
233
283
  async reset(appId) {
234
284
  // Between-test housekeeping (vk suite): the step is deliberately NOT spliced
package/dist/cli.js CHANGED
@@ -917,10 +917,13 @@ async function cmdWait(ctx) {
917
917
  const timeout = (0, args_1.flagNum)(ctx.flags, 'timeout') ?? 10000;
918
918
  const interval = (0, args_1.flagNum)(ctx.flags, 'interval') ?? 400;
919
919
  const deadline = Date.now() + timeout;
920
- const barrier = new auto_wait_1.BarrierTally(ctx);
920
+ const barrier = new auto_wait_1.ReadTally(ctx);
921
921
  while (Date.now() < deadline) {
922
- const { matches, tier } = (0, selector_1.matchElements)(barrier.note((0, auto_wait_1.readForPoll)(ctx)), sel);
923
- if (gone ? matches.length === 0 : matches.length > 0) {
922
+ const { matches, tier } = (0, selector_1.matchElements)((0, auto_wait_1.readForPoll)(ctx, barrier), sel);
923
+ // A read that did not happen proves nothing, and `--gone` is satisfied by an EMPTY one —
924
+ // so without this a kill storm answers "gone" on its first poll, exit 0. Absence has to be
925
+ // observed. (Measured on a Pixel 3a while fixing #137.)
926
+ if (!barrier.lastWasBlind() && (gone ? matches.length === 0 : matches.length > 0)) {
924
927
  ctx.record?.note({ selector: sel, tier, element: matches[0], message: gone ? 'gone' : `${matches.length} match(es)` });
925
928
  if (gone)
926
929
  (0, output_1.out)(`gone: '${sel.raw}'`);
@@ -930,6 +933,11 @@ async function cmdWait(ctx) {
930
933
  }
931
934
  await (0, wait_1.sleep)(interval);
932
935
  }
936
+ // A window that never once READ the screen has no timeout to report — it has an environment
937
+ // failure, and it throws (issue #137) rather than reaching the `return 1` below. That is also
938
+ // what a killed dump did before this fix, so the recorded step keeps its `error` shape
939
+ // instead of turning into a `failure`.
940
+ barrier.rethrowIfBlind();
933
941
  // A barrier can only explain a miss: with --gone the element is absent and the wait passed above.
934
942
  const why = withStop(gone ? '' : barrier.clause());
935
943
  ctx.record?.note({ selector: sel, message: `timeout after ${timeout}ms${gone ? ' (still present)' : ' (never appeared)'}${why}` });
@@ -979,12 +987,23 @@ async function cmdAssert(ctx) {
979
987
  // Auto-wait subsumes the common "wait then assert": poll until the assertion
980
988
  // passes or the window elapses. `--gone` therefore waits for disappearance.
981
989
  const deadline = Date.now() + (0, auto_wait_1.waitWindowMs)(ctx.flags);
982
- const barrier = new auto_wait_1.BarrierTally(ctx);
983
- let result = evalAssert(barrier.note((0, auto_wait_1.readForPoll)(ctx)), sel, ctx.flags);
990
+ const barrier = new auto_wait_1.ReadTally(ctx);
991
+ // A verdict is only worth banking if the read behind it happened. `--gone` (and `--count 0`)
992
+ // pass on an EMPTY tree, which is exactly what an absorbed transient hands back — so a single
993
+ // killed dump mid-window could otherwise bank a green the screen never showed. Poll again
994
+ // instead; `rethrowIfBlind()` below handles a window that stayed blind to the end.
995
+ const look = () => {
996
+ const result = evalAssert((0, auto_wait_1.readForPoll)(ctx, barrier), sel, ctx.flags);
997
+ return barrier.lastWasBlind() ? { ...result, pass: false } : result;
998
+ };
999
+ let result = look();
984
1000
  while (!result.pass && Date.now() < deadline) {
985
1001
  await (0, wait_1.sleep)((0, auto_wait_1.pollStep)(ctx.flags, deadline));
986
- result = evalAssert(barrier.note((0, auto_wait_1.readForPoll)(ctx)), sel, ctx.flags);
1002
+ result = look();
987
1003
  }
1004
+ // `--gone` PASSES on an empty read, so a window of nothing but killed dumps would report a
1005
+ // green earned from a screen nobody could read. Checked before `pass` is consumed (issue #137).
1006
+ barrier.rethrowIfBlind();
988
1007
  const { pass, matches } = result;
989
1008
  // Only a "not found" can be explained by a barrier: `--gone` passed if the tree was
990
1009
  // barrier-only, and a text mismatch found the element.
@@ -2250,10 +2269,12 @@ async function resolveBackend(platform, device, flags) {
2250
2269
  // every recorded command, which beats a timer: it fires when work happens.
2251
2270
  grant: (0, grant_1.processClaimGrant)(device, claims_1.releaseOwnClaims),
2252
2271
  moves: [],
2272
+ skipped: [],
2253
2273
  };
2254
2274
  }
2255
2275
  let runCtx = { platform, device };
2256
2276
  const moves = [];
2277
+ const skipped = [];
2257
2278
  /** Set by the last move; the preflight below reads it to decide whether re-asking is
2258
2279
  * warranted, then clears it. */
2259
2280
  let movedDuringCall;
@@ -2264,6 +2285,7 @@ async function resolveBackend(platform, device, flags) {
2264
2285
  // is identical to a local run's. logStart travels from the server's device clock
2265
2286
  // so archive-time / vk log scoping works without a local driver.
2266
2287
  onStep: (step, artifacts, logStart) => run_1.Recorder.appendForeignStep(step, artifacts, { ...runCtx, logStart }),
2288
+ onInstallSkipped: (s) => skipped.push(...s),
2267
2289
  onDeviceChange: (c) => {
2268
2290
  moves.push(c);
2269
2291
  movedDuringCall = c;
@@ -2342,6 +2364,7 @@ async function resolveBackend(platform, device, flags) {
2342
2364
  grant: (0, grant_1.leaseGrant)(remote, serial),
2343
2365
  remote: { url: server, version: health.version, reads },
2344
2366
  moves,
2367
+ skipped,
2345
2368
  };
2346
2369
  }
2347
2370
  /**
@@ -2640,7 +2663,7 @@ async function cmdInstall(positionals, flags) {
2640
2663
  if (!(0, node_fs_1.existsSync)(path))
2641
2664
  throw new errors_1.CliError(`install: '${appPath}' does not exist`, 2);
2642
2665
  const platform = platformFromFlags(flags);
2643
- const { backend, remote, moves } = await resolveBackend(platform, deviceFromFlags(flags, platform), flags);
2666
+ const { backend, remote, moves, skipped } = await resolveBackend(platform, deviceFromFlags(flags, platform), flags);
2644
2667
  (0, output_1.err)(`[verikun] installing ${appPath}${remote ? ` via ${remote.url}` : ''}…`);
2645
2668
  try {
2646
2669
  await backend.install(path);
@@ -2657,11 +2680,22 @@ async function cmdInstall(positionals, flags) {
2657
2680
  // device than the one the run started against, and a caller acting on the old serial
2658
2681
  // (`adb -s … shell am start`) would be driving a phone without the build.
2659
2682
  const moved = moves.length ? moves[moves.length - 1] : undefined;
2683
+ // A pooled server may have installed on some devices and dropped the rest. That is a
2684
+ // SUCCESS — the ones that missed the build are no longer leasable, so no later lane can
2685
+ // run the previous build and report green — but it is not a silent one: capacity changed.
2660
2686
  if ((0, args_1.flagBool)(flags, 'json')) {
2661
- (0, output_1.json)({ installed: appPath, ...(remote ? { server: remote.url } : {}), ...(moved ? { deviceChanged: moved } : {}) });
2687
+ (0, output_1.json)({
2688
+ installed: appPath,
2689
+ ...(remote ? { server: remote.url } : {}),
2690
+ ...(moved ? { deviceChanged: moved } : {}),
2691
+ ...(skipped.length ? { skipped } : {}),
2692
+ });
2662
2693
  }
2663
2694
  else {
2664
2695
  (0, output_1.out)(`installed ${appPath}${moved ? ` on ${moved.to}` : ''}`);
2696
+ if (skipped.length) {
2697
+ (0, output_1.out)(`skipped ${skipped.length} device(s), now out of the pool: ${skipped.map((s) => s.serial).join(', ')}`);
2698
+ }
2665
2699
  }
2666
2700
  return 0;
2667
2701
  }
@@ -10,7 +10,7 @@
10
10
  // waiting lives here. The rules a new selector-resolving command must follow are in
11
11
  // CLAUDE.md, "Selector auto-wait".
12
12
  Object.defineProperty(exports, "__esModule", { value: true });
13
- exports.BarrierTally = void 0;
13
+ exports.ReadTally = void 0;
14
14
  exports.parseDuration = parseDuration;
15
15
  exports.waitWindowMs = waitWindowMs;
16
16
  exports.waitNote = waitNote;
@@ -53,48 +53,63 @@ function pollStep(flags, deadline) {
53
53
  return Math.min(interval, Math.max(0, deadline - Date.now()));
54
54
  }
55
55
  /**
56
- * Read the hierarchy for a caller that is polling, treating "no window yet" as "nothing on
57
- * screen yet" rather than a fatal environment error.
56
+ * Read the hierarchy for a caller that is polling, treating a read that will clear on its own
57
+ * as "nothing on screen yet" rather than as a fatal environment error.
58
58
  *
59
- * A `NoWindowError` means the device genuinely had nothing to show — `launch --clear` and
60
- * `launch` both leave a gap where the app has been stopped and has not drawn. That clears in
61
- * a second or two, so a caller that has a wait budget should keep polling; escalating to
59
+ * A `TransientReadError` means the device could not answer *right now* — `launch --clear` and
60
+ * `launch` both leave a gap where the app has been stopped and has not drawn (`NoWindowError`),
61
+ * and a memory-pressured phone SIGKILLs the dumper outright (`DumpKilledError`, issue #137).
62
+ * Both clear in seconds, so a caller that has a wait budget should keep polling; escalating to
62
63
  * exit 3 throws away the budget it was explicitly given. MEASURED: a `wait --timeout 120000`
63
- * used to abort at ~20s with 100 seconds unspent.
64
+ * used to abort at ~20s with 100 seconds unspent, and a `wait --timeout 30000` at 2.5s.
64
65
  *
65
66
  * Every OTHER capture failure still propagates untouched — a missing adb, an unauthorised
66
67
  * device or a wedged dumper is a machine to fix, and polling it for two minutes helps nobody.
68
+ *
69
+ * Pass the tally so the window can tell "the screen said nothing was there" from "nobody ever
70
+ * read the screen"; see `ReadTally.rethrowIfBlind`.
67
71
  */
68
- function readForPoll(ctx, opts = {}) {
72
+ function readForPoll(ctx, tally, opts = {}) {
69
73
  try {
70
- return ctx.driver.getElements(opts);
74
+ const els = ctx.driver.getElements(opts);
75
+ return tally ? tally.note(els) : els;
71
76
  }
72
77
  catch (e) {
73
- if (e instanceof errors_1.NoWindowError)
78
+ if (e instanceof errors_1.TransientReadError) {
79
+ tally?.noteBlind(e);
74
80
  return [];
81
+ }
75
82
  throw e;
76
83
  }
77
84
  }
78
85
  /**
79
- * Keeps track of whether the snapshots a poll loop read were barrier-only trees (see
80
- * ui/barrier.ts), so the failure at the end can say so instead of "never appeared".
86
+ * What every read in ONE poll window saw. This is the only layer that sees all of them, so
87
+ * both of the things a miss message needs to be honest about live here.
88
+ *
89
+ * **Was the tree barrier-only?** (issue #131) A sheet's barrier that outlives the whole wait
90
+ * made the step report "never appeared", sending the reader looking for a missing identifier
91
+ * in app code. Naming the barrier is the cheap half of that fix.
81
92
  *
82
- * A sheet's barrier that outlives the whole wait is the report behind issue #131: the
83
- * step waited 30s on a painted sheet and the message sent the reader looking for a missing
84
- * identifier in app code. Naming the barrier is the cheap half of that fix, and it belongs
85
- * to whoever owns the wait — this is the only layer that saw every read.
93
+ * **Did anyone ever read the screen at all?** (issue #137) An absorbed `DumpKilledError` costs
94
+ * a read and yields no elements, and a window made entirely of those has no grounds to call
95
+ * anything absent see `rethrowIfBlind`.
86
96
  */
87
- class BarrierTally {
97
+ class ReadTally {
88
98
  ctx;
89
99
  reads = 0;
100
+ okReads = 0;
90
101
  barrierReads = 0;
91
102
  last = null;
103
+ blind;
104
+ lastBlind = false;
92
105
  constructor(ctx) {
93
106
  this.ctx = ctx;
94
107
  }
95
108
  /** Record one snapshot. Returns it, so it can wrap a read in place. */
96
109
  note(els) {
97
110
  this.reads++;
111
+ this.okReads++;
112
+ this.lastBlind = false;
98
113
  this.last = (0, barrier_1.modalBarrierOnly)(els, this.ctx.driver.viewport());
99
114
  if (this.last)
100
115
  this.barrierReads++;
@@ -106,8 +121,47 @@ class BarrierTally {
106
121
  return '';
107
122
  return (0, barrier_1.barrierClause)(this.last, this.barrierReads === this.reads);
108
123
  }
124
+ /** Record a read that never happened — a transient failure `readForPoll` absorbed as `[]`. */
125
+ noteBlind(e) {
126
+ this.reads++;
127
+ this.blind = e;
128
+ this.lastBlind = true;
129
+ this.last = null; // a read that did not happen is not a barrier, and must not read as one
130
+ }
131
+ /**
132
+ * Did the most recent read fail to happen? Then it proves NOTHING, and least of all an
133
+ * absence — which `--gone` counts as a pass.
134
+ *
135
+ * Separate from `rethrowIfBlind`, and it has to be: that one asks about the whole window and
136
+ * fires at the deadline, but a `--gone` predicate is satisfied by the FIRST empty read and
137
+ * returns from inside the poll loop, so a window-level check never runs. MEASURED on a
138
+ * Pixel 3a while fixing #137: `wait --gone` under a kill storm exited 0 reporting "gone".
139
+ *
140
+ * Both `NoWindowError` and `DumpKilledError` count here. Unlike the deadline rule, the two
141
+ * need no asymmetry: an absorbed read yielded no elements to judge either way, so polling
142
+ * once more is right for both and costs a merely-absent selector nothing.
143
+ */
144
+ lastWasBlind() {
145
+ return this.lastBlind;
146
+ }
147
+ /**
148
+ * Refuse to report an absence this window never actually observed (issue #137).
149
+ *
150
+ * ONLY for a killed dump. The asymmetry is the point: a null root is the device ANSWERING
151
+ * "nothing is drawn", so "the selector is absent" is a true reading of it and
152
+ * `NoWindowError` keeps its existing behaviour exactly. A kill is no answer at all — and
153
+ * `assert --gone` turns "absent" into a PASS, so absorbing it silently would manufacture a
154
+ * green from a screen nobody could read.
155
+ *
156
+ * Gated on `okReads === 0`, the same `everRead` rule the engine's guard grace uses: one good
157
+ * read anywhere in the window means the screen was legible and an ordinary miss is honest.
158
+ */
159
+ rethrowIfBlind() {
160
+ if (this.okReads === 0 && this.blind instanceof errors_1.DumpKilledError)
161
+ throw this.blind;
162
+ }
109
163
  }
110
- exports.BarrierTally = BarrierTally;
164
+ exports.ReadTally = ReadTally;
111
165
  /**
112
166
  * matchElements with auto-wait: re-capture + re-match until at least one element
113
167
  * matches or the window elapses. Returns the final result either way (empty on miss),
@@ -115,11 +169,16 @@ exports.BarrierTally = BarrierTally;
115
169
  */
116
170
  async function matchWaiting(ctx, sel, opts = {}) {
117
171
  const deadline = Date.now() + waitWindowMs(ctx.flags);
118
- const barrier = new BarrierTally(ctx);
172
+ const barrier = new ReadTally(ctx);
119
173
  for (;;) {
120
- const res = (0, selector_1.matchElements)(barrier.note(readForPoll(ctx, opts)), sel);
121
- if (res.matches.length > 0 || Date.now() >= deadline)
174
+ const res = (0, selector_1.matchElements)(readForPoll(ctx, barrier, opts), sel);
175
+ if (res.matches.length > 0)
122
176
  return { ...res, barrier };
177
+ if (Date.now() >= deadline) {
178
+ // Before ANY caller can read this as an absence — `assert --gone` calls it a pass.
179
+ barrier.rethrowIfBlind();
180
+ return { ...res, barrier };
181
+ }
123
182
  await (0, wait_1.sleep)(pollStep(ctx.flags, deadline));
124
183
  }
125
184
  }
@@ -132,9 +191,9 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
132
191
  const windowMs = waitWindowMs(ctx.flags);
133
192
  const start = Date.now();
134
193
  const deadline = start + windowMs;
135
- const barrier = new BarrierTally(ctx);
194
+ const barrier = new ReadTally(ctx);
136
195
  for (;;) {
137
- const els = barrier.note(readForPoll(ctx, opts));
196
+ const els = readForPoll(ctx, barrier, opts);
138
197
  if ((0, selector_1.matchElements)(els, sel).matches.length >= 1) {
139
198
  const { element, tier } = (0, selector_1.resolveOne)(els, sel); // 1 → resolved; >1 → throws ambiguity
140
199
  // The snapshot rides along: scroll-into-view needs the scrollable containers
@@ -143,6 +202,7 @@ async function resolveOneWaiting(ctx, sel, opts = {}) {
143
202
  return { element, tier, waitedMs: Date.now() - start, elements: els };
144
203
  }
145
204
  if (Date.now() >= deadline) {
205
+ barrier.rethrowIfBlind();
146
206
  const waited = windowMs > 0 ? ` after ${(windowMs / 1000).toFixed(1)}s` : '';
147
207
  throw new errors_1.SelectorNotFoundError(`No element matched selector '${sel.raw}'${waited}.${barrier.clause()} Run \`verikun ui\` to inspect the current screen.`);
148
208
  }
@@ -84,16 +84,22 @@ function exitCodeOf(e) {
84
84
  return e instanceof errors_1.CliError ? e.exitCode : 3;
85
85
  }
86
86
  const messageOf = (e) => (e instanceof Error ? e.message : String(e ?? ''));
87
+ /** Why this device stays. Named per class so the operator reads the actual cause, not
88
+ * "transient" — the two want different responses (wait vs free some memory). */
89
+ const transientReason = (e) => e instanceof errors_1.DumpKilledError
90
+ ? 'the hierarchy dump was killed — the device is under memory pressure, not broken'
91
+ : 'the app has not drawn yet — this clears on its own';
87
92
  /**
88
93
  * The arms share everything except what an unrecognised exit-3 means, so they share
89
94
  * this and differ only in `fallback`.
90
95
  */
91
96
  function classify(e, fallback) {
92
- // Identity first, never message text: NoWindowError is exit 3 and its wording could
93
- // plausibly be matched by another rule, and getting this one wrong means rotating the
94
- // pool every time an app is mid-launch.
95
- if (e instanceof errors_1.NoWindowError) {
96
- return { move: false, kind: 'transient', reason: 'the app has not drawn yet — this clears on its own' };
97
+ // Identity first, never message text: both transient reads are exit 3 and their wording could
98
+ // plausibly be matched by another rule, and getting this one wrong means rotating the pool
99
+ // every time an app is mid-launch — or, for a killed dump, retiring a phone for being busy
100
+ // (issue #137).
101
+ if (e instanceof errors_1.TransientReadError) {
102
+ return { move: false, kind: 'transient', reason: transientReason(e) };
97
103
  }
98
104
  const code = exitCodeOf(e);
99
105
  if (code === 0 || code === 1)
@@ -140,7 +146,7 @@ function classifyInstallFailure(e) {
140
146
  const code = exitCodeOf(e);
141
147
  // Only an environment failure is ever the device's fault; a usage error (a rejected
142
148
  // extension, an unreadable path) is the caller's and no device fixes it.
143
- if (code === 3 && !(e instanceof errors_1.NoWindowError)) {
149
+ if (code === 3 && !(e instanceof errors_1.TransientReadError)) {
144
150
  const message = messageOf(e);
145
151
  // Same order as `classify` below, so the two arms can only ever differ in their
146
152
  // DEFAULT — which is the one difference between them that is meant to exist.
@@ -5,6 +5,7 @@ exports.probeAdb = probeAdb;
5
5
  exports.parseLockKind = parseLockKind;
6
6
  exports.looksLikeSystemUi = looksLikeSystemUi;
7
7
  exports.lockKindOf = lockKindOf;
8
+ exports.dumpWasKilled = dumpWasKilled;
8
9
  exports.escapeText = escapeText;
9
10
  exports.adbTransport = adbTransport;
10
11
  exports.severanceRisk = severanceRisk;
@@ -204,6 +205,26 @@ const BARRIER_SETTLE_MS = 300;
204
205
  * companion reports the same condition from getRootInActiveWindow(). Transient — see
205
206
  * NoWindowError. */
206
207
  const NULL_ROOT = /null root node/i;
208
+ /** 128 + SIGKILL — what a shell reports for a command the kernel killed outright. */
209
+ const EXIT_SIGKILL = 137;
210
+ /** The device shell's word for it, when the shell itself survived to say so. */
211
+ const KILLED_TEXT = /\bKilled\b/;
212
+ /**
213
+ * Was this read SIGKILLed rather than merely unsuccessful? (issue #137)
214
+ *
215
+ * THE EXIT CODE IS THE PRIMARY SIGNAL, not the text. MEASURED on a Pixel 3a: in the
216
+ * `adb shell '<cmd>'` form the device shell prints NOTHING when it is killed — both streams
217
+ * come back empty and only the status says 137. That is why the old message read
218
+ * "Failed to capture UI hierarchy after 3 attempts." with nothing after it. The word "Killed"
219
+ * does appear on some shells (issue #137 was reported with it), so it is kept as a second
220
+ * signal for an adb too old to propagate the remote status — but a matcher built on the text
221
+ * alone would have missed the very device this was reported from.
222
+ *
223
+ * Pure and exported for the unit suite only; nothing else imports it.
224
+ */
225
+ function dumpWasKilled(code, text) {
226
+ return code === EXIT_SIGKILL || KILLED_TEXT.test(text);
227
+ }
207
228
  /** Header sizes `screencap` writes before the pixels: width/height/format, plus a
208
229
  * colorspace word since Android 9. Newest first — see `screenshotRaw`. */
209
230
  const RAW_HEADER_SIZES = [16, 12];
@@ -898,6 +919,16 @@ class AdbDriver {
898
919
  if (NULL_ROOT.test(lastErr)) {
899
920
  throw new errors_1.NoWindowError();
900
921
  }
922
+ // A KILLED dump is the same deal, with one difference: attempt 0 has a real cure for
923
+ // one of its two causes (a companion holding the UiAutomation connection SIGKILLs a
924
+ // competing dump), so it still gets the remedy below and one more try. Once THAT is
925
+ // killed too, stop — a third back-to-back attempt is a third sample of the same instant
926
+ // (#137 measured all three losing), and the caller's poll interval is the spacing that
927
+ // actually helps. Judged on the DUMP's own result, never the combined text: a `cat` that
928
+ // reports a missing file is the kill's consequence, not evidence of one.
929
+ if (attempt > 0 && dumpWasKilled(dump.code, `${dump.stdout} ${dump.stderr}`)) {
930
+ throw new errors_1.DumpKilledError((0, errors_1.dumpKilledMessage)(lastErr));
931
+ }
901
932
  if (attempt === 0) {
902
933
  // A sleeping display is the other documented cause of a failed read. `ensureAwake` ran
903
934
  // before the dump, so reaching here means its answer went stale during a slow read (or
package/dist/errors.js CHANGED
@@ -6,7 +6,7 @@
6
6
  // 2 usage error, ambiguous selector, or a device another job is driving (caller must refine)
7
7
  // 3 environment error (adb/simctl missing, no usable device, dump failed)
8
8
  Object.defineProperty(exports, "__esModule", { value: true });
9
- exports.AmbiguousSelectorError = exports.NO_WINDOW_MESSAGE = exports.NoWindowError = exports.SelectorNotFoundError = exports.probeFailure = exports.envError = exports.usageError = exports.CliError = void 0;
9
+ exports.AmbiguousSelectorError = exports.DUMP_KILLED_MESSAGE = exports.dumpKilledMessage = exports.DumpKilledError = exports.NO_WINDOW_MESSAGE = exports.NoWindowError = exports.TransientReadError = exports.SelectorNotFoundError = exports.probeFailure = exports.envError = exports.usageError = exports.CliError = void 0;
10
10
  exports.isEnvError = isEnvError;
11
11
  class CliError extends Error {
12
12
  exitCode;
@@ -51,6 +51,21 @@ class SelectorNotFoundError extends CliError {
51
51
  }
52
52
  }
53
53
  exports.SelectorNotFoundError = SelectorNotFoundError;
54
+ /**
55
+ * A hierarchy read that failed for a reason which CLEARS ON ITS OWN within seconds.
56
+ *
57
+ * The base exists so the three layers that mean "ride this out" — `readForPoll`, the engine's
58
+ * guard grace, and the failover classifier — say so once, by class, instead of listing
59
+ * subclasses they would each have to be remembered to update. Everything narrower than "this
60
+ * is transient" keeps checking the concrete class (the companion only stands down for a
61
+ * NoWindowError, not for a kill it had nothing to do with).
62
+ *
63
+ * Still exit 3, so a caller with NO budget is unaffected: an unabsorbed one exits exactly as
64
+ * it did before. What the class buys is the right to be polled through, not a softer exit.
65
+ */
66
+ class TransientReadError extends CliError {
67
+ }
68
+ exports.TransientReadError = TransientReadError;
54
69
  /**
55
70
  * There is no window to read right now — the app was just force-stopped, or is mid-launch
56
71
  * and has not drawn yet. `getRootInActiveWindow()` returns null and the platform says so.
@@ -66,7 +81,7 @@ exports.SelectorNotFoundError = SelectorNotFoundError;
66
81
  * and a `wait --timeout 120000` would abort at ~20s with 100 seconds of its budget unspent.
67
82
  * The retry belongs to the caller that knows how long it is willing to wait.
68
83
  */
69
- class NoWindowError extends CliError {
84
+ class NoWindowError extends TransientReadError {
70
85
  constructor(message = exports.NO_WINDOW_MESSAGE) {
71
86
  super(message, 3);
72
87
  this.name = 'NoWindowError';
@@ -86,6 +101,45 @@ exports.NoWindowError = NoWindowError;
86
101
  */
87
102
  exports.NO_WINDOW_MESSAGE = 'No window to read: the app has no drawn window right now — force-stopped, mid-launch, or ' +
88
103
  'its main thread is busy mid-transition. This normally clears within a few seconds.';
104
+ /**
105
+ * The dump process was SIGKILLed before it could answer (issue #137).
106
+ *
107
+ * NOT the same signal as NoWindowError, and the difference is the whole reason this is its own
108
+ * class. A null root is the device ANSWERING "nothing is drawn", so "the selector is absent" is
109
+ * a true reading of it. A kill is no answer at all, so calling the selector absent would be a
110
+ * fabrication — which is why a poll window that never once read the screen re-throws this
111
+ * instead of reporting a miss (`ReadTally.rethrowIfBlind`).
112
+ *
113
+ * MEASURED on a 4 GB-class phone (#137): the OS reaps the dumper while an app cold-starts, and
114
+ * the driver's three attempts fired back-to-back all landed inside the same second — so a
115
+ * `wait` holding a two-minute budget aborted at ~2.5s with 117 seconds unspent. Same rule as
116
+ * NoWindowError: the driver hands it up, the caller spends its own clock on it.
117
+ */
118
+ class DumpKilledError extends TransientReadError {
119
+ constructor(message = exports.DUMP_KILLED_MESSAGE) {
120
+ super(message, 3);
121
+ this.name = 'DumpKilledError';
122
+ }
123
+ }
124
+ exports.DumpKilledError = DumpKilledError;
125
+ /** The wording plus whatever the device actually said, for the one thrower that has evidence.
126
+ * Separate from the constructor so the wire codec can rebuild a message losslessly rather
127
+ * than re-prefixing one that already carries its detail. */
128
+ const dumpKilledMessage = (detail) => detail ? `${exports.DUMP_KILLED_MESSAGE} (${detail})` : exports.DUMP_KILLED_MESSAGE;
129
+ exports.dumpKilledMessage = dumpKilledMessage;
130
+ /**
131
+ * Names the CAUSE, because "the dump was killed" and "the device left adb" both used to arrive
132
+ * as `Failed to capture UI hierarchy after 3 attempts` and want opposite responses from whoever
133
+ * reads the report — wait vs go and find the phone.
134
+ *
135
+ * Both causes are named because both are real and the fix for each is different: memory pressure
136
+ * (wait, or test on a device with more headroom) and a competing UiAutomation client (stop it).
137
+ * verikun's own companion is the second one, and the driver already tries to clear that itself
138
+ * before this is ever thrown.
139
+ */
140
+ exports.DUMP_KILLED_MESSAGE = 'The UI hierarchy dump was killed before it could answer — the device reclaiming memory ' +
141
+ 'while an app starts, or another tool holding the one UiAutomation connection. This ' +
142
+ 'normally clears within seconds.';
89
143
  /** Selector matched >1 element. Exit 2. Carries the candidates so the agent runner
90
144
  * can ask the model to disambiguate (a heal trigger) instead of aborting. */
91
145
  class AmbiguousSelectorError extends CliError {
package/dist/rpc.js CHANGED
@@ -22,11 +22,15 @@ function describeError(e) {
22
22
  if (e instanceof errors_1.SelectorNotFoundError) {
23
23
  return { kind: 'SelectorNotFoundError', name: e.name, message: e.message, exitCode: e.exitCode };
24
24
  }
25
- // BEFORE the CliError arm — NoWindowError extends it, so a subclass check must come
26
- // first or the identity is flattened away. device/failover.ts classifies on
27
- // `instanceof NoWindowError` deliberately ("identity first, never message text"), and
28
- // losing it turns every mid-launch gap into an unknown that costs two device probes and
29
- // can quarantine a perfectly healthy phone.
25
+ // BEFORE the CliError arm — both transient reads extend it, so a subclass check must come
26
+ // first or the identity is flattened away. device/failover.ts classifies on the CLASS
27
+ // deliberately ("identity first, never message text"), and losing it turns every mid-launch
28
+ // gap into an unknown that costs two device probes and can quarantine a perfectly healthy
29
+ // phone. Flattening DumpKilledError also costs a poller its ride-out, which is the whole of
30
+ // issue #137 — and #137 was reported through a pooled `vk server`, i.e. across this wire.
31
+ if (e instanceof errors_1.DumpKilledError) {
32
+ return { kind: 'DumpKilledError', name: e.name, message: e.message, exitCode: e.exitCode };
33
+ }
30
34
  if (e instanceof errors_1.NoWindowError) {
31
35
  return { kind: 'NoWindowError', name: e.name, message: e.message, exitCode: e.exitCode };
32
36
  }
@@ -45,6 +49,8 @@ function rebuildError(d) {
45
49
  return new errors_1.SelectorNotFoundError(d.message);
46
50
  case 'NoWindowError':
47
51
  return new errors_1.NoWindowError(d.message);
52
+ case 'DumpKilledError':
53
+ return new errors_1.DumpKilledError(d.message);
48
54
  case 'CliError':
49
55
  return new errors_1.CliError(d.message, d.exitCode);
50
56
  default: {
package/dist/server.js CHANGED
@@ -519,7 +519,26 @@ function buildServer(config) {
519
519
  : null;
520
520
  // Like the reconcile timer: must never hold the process open at Ctrl-C.
521
521
  recycleTimer?.unref?.();
522
- const pickFailoverDevice = (failed, reason) => serializeFailover(() => pickFailoverDeviceLocked(failed, reason));
522
+ /**
523
+ * Is this failure grounds to REMOVE the device from the pool, rather than merely deal it
524
+ * last? Two conditions, and both are necessary.
525
+ *
526
+ * `unreachable` is the only kind that qualifies, because it is the only one that says the
527
+ * device is not there. Every other kind describes a device that is present and unhappy —
528
+ * a full disk, a wedged app, an exit 3 nobody has classified — and for those, demotion
529
+ * plus recovery-by-traffic is right and this must not change: they can still produce the
530
+ * traffic that clears them. An absent device cannot, which is the whole defect (#139): the
531
+ * demotion is a sort key (`leaseFor`), so "dealt last" is still dealt, every round, and
532
+ * `restoreDevice` can never fire for a device that will never answer again.
533
+ *
534
+ * And only on a POOLED server, because only a pooled server sweeps. `reconcileOnce`
535
+ * returns immediately without `poolSpec` (`wantedSerials`) and its timer is never even
536
+ * created — see `ServerConfig.poolSpec`, "deliberately does not reconcile". Shedding
537
+ * where nothing readmits would trade a device that fails loudly for a server that is
538
+ * empty until someone restarts it: a worse failure, and a new one.
539
+ */
540
+ const shedOnFailure = (kind) => kind === 'unreachable' && config.poolSpec !== undefined;
541
+ const pickFailoverDevice = (failed, reason, kind) => serializeFailover(() => pickFailoverDeviceLocked(failed, reason, kind));
523
542
  /**
524
543
  * Bring in a healthy replacement for `failed`. Returns the serial moved to, or null
525
544
  * when none remains (which is not an error here — the caller reports the ORIGINAL
@@ -533,7 +552,7 @@ function buildServer(config) {
533
552
  * only reports ready once its OWN `preflight()` has passed, so starting the worker IS
534
553
  * the probe, run on the thread that will go on to use it.
535
554
  */
536
- const pickFailoverDeviceLocked = async (failed, reason) => {
555
+ const pickFailoverDeviceLocked = async (failed, reason, kind) => {
537
556
  const policy = config.failover;
538
557
  if (!policy)
539
558
  return null;
@@ -549,23 +568,29 @@ function buildServer(config) {
549
568
  // see and where the server will actually go cannot drift. A pool member's own driver
550
569
  // is not: it may be pointed at a corpse.
551
570
  /**
552
- * Nothing healthier exists. Shed the failed device — continuing to hand it out is
553
- * what makes a pool a coin flip per lease — but ONLY while another remains.
571
+ * Nothing healthier exists. Decide what becomes of the failed device itself THREE
572
+ * outcomes, not two, and which one applies is `shedOnFailure`'s question:
554
573
  *
555
- * The LAST device stays, deliberately. A server that shed it would answer every
556
- * later request `503 no device attached`, replacing the device's own error (full
557
- * disk, no space, whatever it actually was) with a message that names nothing. A
558
- * caller stuck on one broken device is better served by the truth about it.
574
+ * - GONE, on a pooled server shed it. It cannot serve and cannot recover by
575
+ * traffic, so leaving it in the pool means dealing it forever (#139). The sweep
576
+ * owns readmission, so capacity comes back on its own.
577
+ * - present but unhappy demote it: worker, claim and slot kept, dealt last,
578
+ * restored by the first command that works.
579
+ * - already left on its own (its worker died) — nothing to remove, just clean up.
580
+ *
581
+ * The last two share a tail with the first, because "stop serving this device" has the
582
+ * same consequences however it came about.
559
583
  */
560
584
  const shrink = async () => {
561
585
  // A device whose worker DIED is already out of the pool, so there is nothing left to
562
586
  // shed — but its holder still has to be evicted and its claim and companion handed
563
- // back, and the "last device stays" guard below must not skip that. Asking whether
564
- // it is still a member is what separates the two cases.
587
+ // back. Asking whether it is still a member is what separates that case from a
588
+ // device we are removing ourselves.
565
589
  const serving = pool.serials().includes(failed);
566
- if (serving) {
567
- // DEMOTE, never shed. The device keeps its worker, its claim and its place in the
568
- // pool; it is simply dealt last until it does some work (see `degradeDevice`).
590
+ if (serving && !shedOnFailure(kind)) {
591
+ // DEMOTE the device is still THERE. It keeps its worker, its claim and its place
592
+ // in the pool; it is simply dealt last until it does some work (see
593
+ // `degradeDevice`). Contrast the shed below, which is only for a device that is not.
569
594
  //
570
595
  // This replaces "nothing healthier to move to — X left the pool". That rule read
571
596
  // correctly on a SINGLE-device server, where it never actually fired (the last
@@ -573,10 +598,16 @@ function buildServer(config) {
573
598
  // verdict — because a pool's own members are excluded from its candidate list, so
574
599
  // "no candidate" is the normal case rather than the exceptional one. The argument
575
600
  // for shedding was that continuing to hand out a broken device makes a pool a coin
576
- // flip per lease; that is answered by ORDERING (a degraded device is chosen only
577
- // when nothing else is free), which costs no capacity. And it is the same judgement
578
- // the last-device branch already made out loud: a caller stuck on one broken device
579
- // is better served by the truth about it than by a server that quietly halved.
601
+ // flip per lease; for a device that is PRESENT that is answered by ORDERING (a
602
+ // degraded device is chosen only when nothing else is free), which costs no
603
+ // capacity, and a caller that does reach it is better served by the truth about it
604
+ // than by a server that quietly halved.
605
+ //
606
+ // Ordering answers it only while the device can still come back, though. It cannot
607
+ // answer for a device that is GONE — "dealt last" is still dealt once the healthy
608
+ // devices are busy, which on a suite sized to the pool is every round, and no
609
+ // amount of ordering produces the traffic `restoreDevice` needs. That case is
610
+ // shed above, by `shedOnFailure`.
580
611
  //
581
612
  // The holder keeps its lease too: its device did not go anywhere, so there is no
582
613
  // `deviceChanged` to send and nothing for the run to seal. The step that failed
@@ -585,7 +616,28 @@ function buildServer(config) {
585
616
  degradeDevice(failed, reason);
586
617
  return null;
587
618
  }
588
- // NOT serving — its worker already died, so the device left on its own and there is
619
+ if (serving) {
620
+ // SHED. The device is gone and this server sweeps, so removing it is not the
621
+ // one-way ratchet it was before the sweep existed (#114): `reconcileOnce` lists it
622
+ // as missing from what `--devices` asked for, retries with backoff, and
623
+ // `rejoinDevice` readmits it — bringing it up to `lastInstall` first — the moment
624
+ // it answers again. Capacity returns without anyone restarting anything.
625
+ //
626
+ // It keeps its QUARANTINE, unlike the demote branch above, and that asymmetry is
627
+ // the point: quarantine means "not serving, and ruled out", degradation means
628
+ // "serving but suspect", and the two are disjoint precisely so `/v1/health` and
629
+ // `exhaustedNote` can be read. A shed device genuinely is not serving, so it
630
+ // belongs in the same list as one whose worker died — which is the tail below,
631
+ // reached from here. `rejoinDevice` clears it on evidence, never on a clock.
632
+ //
633
+ // `degraded` must be given up though: it is defined as pool MEMBERS that recently
634
+ // failed, and a non-member left in it would have `/v1/health` reporting a device it
635
+ // no longer serves, in a list whose whole meaning is that it still does.
636
+ pool.retire(failed);
637
+ degraded.delete(failed);
638
+ (0, output_1.err)(`[server] pool: ${failed} left the pool — ${reason} (the sweep readmits it when it answers again)`);
639
+ }
640
+ // NOT serving — its worker died, or the shed above just removed it, so there is
589
641
  // nothing to demote. The holder is EVICTED, not migrated: without a replacement there
590
642
  // is no `deviceChanged` to send, so the client never learns to seal its run — and
591
643
  // merely dropping the lease would let its next request silently draw some other device
@@ -690,7 +742,14 @@ function buildServer(config) {
690
742
  * Is this device actually gone? Two probes a second apart, because that gap is the only
691
743
  * thing separating a USB re-enumeration or a mid-`launch --clear` gap from a dead box —
692
744
  * and quarantining a healthy device is the expensive mistake here. Returns the reason
693
- * when dead, undefined when it was a blip.
745
+ * AND the probe's own verdict kind when dead, undefined when it was a blip.
746
+ *
747
+ * The kind is carried out because the probe is often the better-classified of the two
748
+ * failures. The operation that brought us here may have failed with a string nothing
749
+ * recognises (an unclassified exit 3, which is what earns a probe in the first place),
750
+ * while `preflight` on a detached phone says `device '<serial>' not found` — the exact
751
+ * `UNREACHABLE_RULES` wording. Reporting the ORIGINAL verdict's kind there would decide
752
+ * "shed or demote" from the vaguer of two answers about the same device.
694
753
  */
695
754
  const deviceIsDead = async (handle) => {
696
755
  let last = '';
@@ -722,7 +781,7 @@ function buildServer(config) {
722
781
  (0, output_1.err)(`[server] probe on ${handle.serial}: ${verdict.reason} (${verdict.kind}) — a host problem, not this device`);
723
782
  return undefined;
724
783
  }
725
- return last || 'the device stopped answering';
784
+ return { reason: last || 'the device stopped answering', kind: verdict.kind };
726
785
  };
727
786
  /**
728
787
  * A non-install operation failed. Move off the device if it is genuinely at fault —
@@ -761,6 +820,7 @@ function buildServer(config) {
761
820
  try {
762
821
  const verdict = (0, failover_1.classifyFailure)(e);
763
822
  let reason = verdict.reason;
823
+ let kind = verdict.kind;
764
824
  if (!verdict.move) {
765
825
  // Only an unrecognised exit 3 earns a probe; `transient` and `toolchain` set
766
826
  // probe:false precisely so a mid-launch gap or a missing adb cannot become a move.
@@ -777,15 +837,21 @@ function buildServer(config) {
777
837
  (0, output_1.err)(`[server] ${what}: ${from} failed but probes healthy — staying (${verdict.reason})`);
778
838
  return undefined; // a blip — the test rerun is the right answer, not a new device
779
839
  }
780
- reason = dead;
840
+ reason = dead.reason;
841
+ // The PROBE's verdict, not the original failure's. We are here because the
842
+ // operation failed with something nothing recognised; `preflight` on a detached
843
+ // phone says `device '<serial>' not found`, which is classified. Taking the vaguer
844
+ // of two answers about the same device is how a detachment that first showed up as
845
+ // an odd exit 3 would be demoted forever instead of shed.
846
+ kind = dead.kind;
781
847
  noteVerdict({ ...verdict, move: true }, e, what);
782
848
  }
783
849
  (0, output_1.err)(`[server] ${what}: FAILED on ${from} — ${reason}`);
784
850
  quarantineDevice(from, reason);
785
- // pickFailoverDevice has already said which of the two no-move outcomes happened —
786
- // the device was shed, or it was the last one and stayed. A second line here would
787
- // contradict one of them.
788
- const to = await pickFailoverDevice(from, reason);
851
+ // pickFailoverDevice has already said which no-move outcome happened — the device
852
+ // was shed, or demoted, or had already left. A second line here would contradict
853
+ // one of them.
854
+ const to = await pickFailoverDevice(from, reason, kind);
789
855
  if (!to)
790
856
  return undefined;
791
857
  return { from, to, reason, retried: false };
@@ -958,6 +1024,9 @@ function buildServer(config) {
958
1024
  return new server_http_1.HttpError(409, 'the device this run was using left the pool and nothing healthy replaced it — ' +
959
1025
  'start a fresh run; this one cannot continue on another device', 3);
960
1026
  }
1027
+ // An empty pool never reaches here: the deviceless guard in the router answers 503 for
1028
+ // every route that takes a lease, and it names `lostDevice` while doing it. So `n` is
1029
+ // always >= 1 and this only ever describes CONTENTION, which is what 409 means.
961
1030
  const n = pool.serials().length;
962
1031
  return new server_http_1.HttpError(409, n > 1
963
1032
  ? `all ${n} devices are leased by other active runs — retry when one finishes`
@@ -1197,9 +1266,9 @@ function buildServer(config) {
1197
1266
  * wrapper keeps no result on a throw), so dropping it leaves the operator holding a
1198
1267
  * serial the server has already left.
1199
1268
  */
1200
- const hopOrThrow = async (why, giveUp) => {
1269
+ const hopOrThrow = async (why, giveUp, kind) => {
1201
1270
  quarantineDevice(from, why);
1202
- const to = await pickFailoverDevice(from, why);
1271
+ const to = await pickFailoverDevice(from, why, kind);
1203
1272
  if (to === null) {
1204
1273
  // A pool that emptied with nothing having moved keeps its own 503 — a more
1205
1274
  // accurate status than a wrapped 500.
@@ -1224,7 +1293,9 @@ function buildServer(config) {
1224
1293
  const gone = firstError ?? new server_http_1.HttpError(503, `device ${from} is no longer attached`, 3);
1225
1294
  if (!config.failover || hop >= MAX_FAILOVER_HOPS)
1226
1295
  throw gone;
1227
- from = await hopOrThrow('the device left the pool mid-install', gone);
1296
+ // `unreachable` is the literal truth — it is not in the pool and it is also
1297
+ // inert here: `shrink` sees a non-member and takes its cleanup tail either way.
1298
+ from = await hopOrThrow('the device left the pool mid-install', gone, 'unreachable');
1228
1299
  continue;
1229
1300
  }
1230
1301
  try {
@@ -1241,7 +1312,7 @@ function buildServer(config) {
1241
1312
  // hops: report the first failure unchanged, exactly as before this feature.
1242
1313
  if (!verdict.move || !config.failover || hop >= MAX_FAILOVER_HOPS)
1243
1314
  throw firstError;
1244
- from = await hopOrThrow(verdict.reason, firstError);
1315
+ from = await hopOrThrow(verdict.reason, firstError, verdict.kind);
1245
1316
  }
1246
1317
  }
1247
1318
  }
@@ -1300,6 +1371,12 @@ function buildServer(config) {
1300
1371
  }));
1301
1372
  const failed = outcomes.filter((o) => o.error);
1302
1373
  const moved = outcomes.filter((o) => o.change);
1374
+ // Where each outcome ENDED UP. An install that moved ran on its replacement, not on
1375
+ // the serial it started from, so the device that holds this build — or conspicuously
1376
+ // does not — is the last one it was on, never `o.serial`.
1377
+ const landedOn = (o) => o.change?.to ?? o.serial;
1378
+ const installed = outcomes.filter((o) => !o.error).map(landedOn);
1379
+ let skipped = [];
1303
1380
  if (failed.length) {
1304
1381
  // One artifact, many devices: if it failed everywhere the file is the suspect, so
1305
1382
  // surface the FIRST device's error unchanged rather than a summary that buries it.
@@ -1316,26 +1393,69 @@ function buildServer(config) {
1316
1393
  }
1317
1394
  throw failed[0].error;
1318
1395
  }
1319
- // Carry a move that DID happen even though the overall install failed: the
1320
- // client re-points its run context on `deviceChanged`, and dropping it here
1321
- // would leave the operator holding a serial the server has already left.
1322
- throw new server_http_1.HttpError(500, `install failed on ${failed.map((f) => `${f.serial} (${(0, server_http_1.firstLine)(f.error.message)})`).join('; ')}`, 3, moved[0]?.change);
1396
+ // PARTIAL. Two healthy phones took the build and one did not. Answering 500 for the
1397
+ // whole pool is what turned one detached device into a dead CI job (#139) — and it
1398
+ // dies at the install step, so the run has already paid for an app build and tested
1399
+ // nothing.
1400
+ //
1401
+ // What made the 500 defensible is the fan-out's own rule, one line up: a lane dealt
1402
+ // a device that missed this build runs the PREVIOUS one and reports green, which is
1403
+ // the worst result this server can produce. The answer is not to soften that rule
1404
+ // but to SATISFY it — a device that did not take the build leaves the pool, so no
1405
+ // lease can reach it. `rejoinDevice` already makes exactly this call out loud
1406
+ // ("serving the wrong build is worse than not serving") and offers the same remedy:
1407
+ // the sweep readmits it and installs `lastInstall` before it is dealt any work.
1408
+ //
1409
+ // Done regardless of `config.failover`. The kill switch governs MOVING BETWEEN
1410
+ // devices; it was never a licence to serve a stale build, and the sweep that brings
1411
+ // the device back is gated on `poolSpec`, not on failover.
1412
+ skipped = failed.map((f) => {
1413
+ const serial = landedOn(f);
1414
+ const reason = (0, server_http_1.firstLine)(f.error.message);
1415
+ if (pool.serials().includes(serial)) {
1416
+ pool.retire(serial);
1417
+ // Same two rules as the shed in `shrink`: `degraded` is for MEMBERS, and a
1418
+ // device that is not serving belongs in `quarantine` — which `rejoinDevice`
1419
+ // clears on the evidence of a worker that started and a build that installed.
1420
+ degraded.delete(serial);
1421
+ quarantineDevice(serial, `did not take the current build — ${reason}`);
1422
+ evictHoldersOf(serial, `${serial} left the pool without the current build`);
1423
+ (0, manager_1.releaseCompanionOn)(serial);
1424
+ if ((0, claims_1.claimsEnabled)(claimEnv))
1425
+ (0, claims_1.releaseClaim)(serial, { ...claimOpts, mineOnly: true });
1426
+ }
1427
+ return { serial, reason };
1428
+ });
1429
+ (0, output_1.err)(`[server] install: partial — ${installed.join(', ')} took the build; ` +
1430
+ `removed from the pool: ${skipped.map((s) => `${s.serial} (${s.reason})`).join('; ')}`);
1323
1431
  }
1324
1432
  for (const m of moved)
1325
1433
  (0, output_1.err)(`[server] install: ${m.serial} → ${m.change.to} (${m.moves} move(s))`);
1326
- (0, output_1.err)(`[server] install: done on ${pool.serials().join(', ')}`);
1434
+ (0, output_1.err)(`[server] install: done on ${installed.join(', ')}`);
1327
1435
  // Retain the artifact so a device that rejoins later can be brought up to this build
1328
1436
  // (see `rejoinDevice`). Renamed out of the per-request temp name into one stable slot,
1329
1437
  // so at most one build is ever held and each install replaces the last.
1438
+ //
1439
+ // Reached on a PARTIAL install too, and load-bearing there: the devices just removed
1440
+ // are precisely the ones the sweep will readmit, and `rejoinDevice` brings a returning
1441
+ // device up to `lastInstall`. Retaining only on a clean sweep would hand each of them
1442
+ // the PREVIOUS build on the way back in — and `rejoinDevice`'s own check would pass,
1443
+ // because an install that succeeds is all it can see.
1330
1444
  retainInstall(tmpPath, ext);
1331
1445
  retained = true;
1446
+ // Only a move whose destination SURVIVED is worth reporting. The client re-points its
1447
+ // run context on `deviceChanged`, so naming a device the partial branch retired three
1448
+ // lines ago would send its next step to a serial this server no longer serves.
1449
+ const survivors = new Set(pool.serials());
1450
+ const reportableMove = moved.map((m) => m.change).find((c) => survivors.has(c.to));
1332
1451
  const body = {
1333
1452
  ok: true,
1334
1453
  bytes: size,
1335
1454
  sha256: digest,
1336
- devices: pool.serials(),
1455
+ devices: installed,
1456
+ ...(skipped.length ? { skipped } : {}),
1337
1457
  // The wire field is singular; a pool that moved more than one device logs the rest.
1338
- ...(moved.length ? { deviceChanged: moved[0].change } : {}),
1458
+ ...(reportableMove ? { deviceChanged: reportableMove } : {}),
1339
1459
  };
1340
1460
  (0, server_http_1.sendJson)(res, 200, body);
1341
1461
  }
@@ -1360,10 +1480,12 @@ function buildServer(config) {
1360
1480
  * phone another job is mid-test on. Refusing plainly beats a rule nobody can predict.
1361
1481
  * The GET listing stays available, because reading what is attached is safe.
1362
1482
  *
1363
- * KNOWN COST, stated in the refusal so nobody has to discover it: this is also the only
1364
- * thing that clears a quarantine, so on a pool a device ruled out by failover stays out
1365
- * until the server is restarted. That is the price of refusing rather than guessing;
1366
- * lifting it would mean letting a NAMED, allowlisted target act on one pool member.
1483
+ * This used to carry a known cost that it was the ONLY thing clearing a quarantine, so
1484
+ * on a pool a device ruled out by failover stayed out until the server was restarted.
1485
+ * That is no longer true and the refusal no longer says it: `rejoinDevice` clears the
1486
+ * quarantine (and `degraded`, and `failedOver`) when the sweep readmits a device, on the
1487
+ * evidence of a worker that started and a build that installed. The refusal itself stands
1488
+ * — power-cycling one member of a pool another job is mid-test on is what it prevents.
1367
1489
  */
1368
1490
  function requireSingleDevice(op) {
1369
1491
  const n = pool.serials().length;
package/dist/version.js CHANGED
@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.VERSION = void 0;
4
4
  // GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
5
5
  // (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
6
- exports.VERSION = '0.27.0';
6
+ exports.VERSION = '0.28.0';
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "verikun",
3
- "version": "0.27.0",
3
+ "version": "0.28.0",
4
4
  "description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
5
5
  "keywords": [
6
6
  "android",