verikun 0.27.1 → 0.29.0

package/.claude/skills/verikun/SKILL.md

@@ -514,6 +514,11 @@ installs again, warning on stderr that **that build's app data is gone**. A same
 install still keeps its data. Do not treat the warning as a failure — it is how the
 install succeeded. iOS has no equivalent recovery; the install just fails there.
 
+**`vk install` also allows a version downgrade by default, when the build is debuggable.**
+A build with a lower version code than what's already on the device installs normally
+instead of failing with `INSTALL_FAILED_VERSION_DOWNGRADE`. A release-signed build is a
+genuine exception — Android still refuses it, and that is not something to retry.
+
 `vk server --devices all` (or `all-android` / `all-ios` / a serial list) serves a **pool**
 from one address. Each run leases one device for its whole life, so a run's steps and
 repairs always land on the same phone. `vk install --server` then installs on every device;
@@ -530,10 +535,12 @@ device that failed and is now on another one. What that means depends on the lin
   the failure was real on A, and B has none of the state your flow built up. Start the
   flow again from the top if you want it on B.
 
-The server rules the bad device out until it is power-cycled;
-`vk devices --server <url>` shows why in its `NOTE` column. On a pool the replacement
-joins the pool, so capacity holds — and the last device is never shed, so its own error
-keeps reaching you rather than a bare "no device attached".
+The server rules the bad device out; `vk devices --server <url>` shows why in its `NOTE`
+column, and a pooled server re-adopts a device that comes back within a minute. A device
+that is still attached keeps its place and is simply dealt last, so its own error keeps
+reaching you rather than a bare "no device attached". A device that is **gone** leaves the
+pool — so `capacity` can drop mid-job, and an install can come back `exit 0` having skipped
+it. That is a success: nothing can be dealt a device running the previous build.
 
 ## The device is missing or wedged
 

package/CHANGELOG.md

@@ -6,6 +6,45 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [0.29.0] - 2026-09-16
+
+### Changed
+- **`vk install`** (Android) allows a version downgrade by default (`adb install -d`),
+  instead of failing with `INSTALL_FAILED_VERSION_DOWNGRADE`, when the build is debuggable.
+
+## [0.28.1] - 2026-09-15
+
+### Fixed
+- **`vk install --server`** sheds a pooled device whose install stalls, while the client now
+  honors its 15-minute request budget. ([#143])
+
+### Changed
+- **Docs rule**: a bug fix is changelog-only; `docs/` changes only when a contract moves or a page
+  went false. ([#140])
+
+[#140]: https://github.com/ddikman/verikun/issues/140
+[#143]: https://github.com/ddikman/verikun/issues/143
+
+## [0.28.0] - 2026-09-14
+
+A device that has gone now leaves a pooled `vk server` instead of being dealt out until
+somebody notices.
+
+### Fixed
+- **`vk server --devices`** drops a device that is gone from the pool instead of dealing it
+  forever; the sweep readmits it when it returns. ([#139])
+- **`vk install --server`** exits `0` when some pooled devices take the build, naming the rest
+  as `skipped`; only a build that fails everywhere is an error. ([#139])
+- **A `--server` call that outlives Node's 300s fetch ceiling** now says so, and says the clock
+  was the client's, instead of a bare `fetch failed` the suite read as a dead device. ([#139])
+
+### Changed
+- **`/v1/health` `capacity` can now drop mid-job** on a pooled server, to `0`. A plain
+  `vk server` still keeps its only device and answers with that device's own error. ([#139])
+- **`POST /v1/install`** answers `200 {devices, skipped}` where it used to answer `500` on a
+  partial failure. Older clients see a success with an unknown field. ([#139])
+
+[#139]: https://github.com/ddikman/verikun/issues/139
 ## [0.27.1] - 2026-09-14
 
 A hierarchy read the device killed is now waited out instead of ending the test.

package/dist/agent/remote.js

@@ -10,27 +10,98 @@
 // local one. Recording stays a caller concern: this module never touches ./.verikun.
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.describeStatus = describeStatus;
+exports.transportReason = transportReason;
 exports.pingServer = pingServer;
 exports.remoteDeviceOp = remoteDeviceOp;
 exports.remoteDeviceList = remoteDeviceList;
 exports.createRemoteBackend = createRemoteBackend;
 const node_fs_1 = require("node:fs");
 const node_crypto_1 = require("node:crypto");
+const node_http_1 = require("node:http");
+const node_https_1 = require("node:https");
 const node_path_1 = require("node:path");
 const errors_1 = require("../errors");
+const install_timeouts_1 = require("../install-timeouts");
+const output_1 = require("../output");
 const rpc_1 = require("../rpc");
+/**
+ * The ceiling fetch-based calls below cannot exceed, whatever their own budgets say.
+ *
+ * Node's global `fetch` is undici, whose `headersTimeout` and `bodyTimeout` both default to
+ * 300s, and there is no dependency-free way to raise them: a `dispatcher` needs `undici`
+ * itself, which is bundled but not importable. Long-running install uploads therefore use
+ * Node's built-in http/https client instead; its explicit 15-minute timer is the real
+ * upload-and-response ceiling. The other AbortControllers remain a floor when set above 300s.
+ *
+ * MEASURED on Node v20.20.2 against a server that held its headers for 310s: the fetch
+ * rejected at 301s with `TypeError: fetch failed`, cause `HeadersTimeoutError`, code
+ * `UND_ERR_HEADERS_TIMEOUT`. The bare `fetch failed` is the whole problem — `describeStatus`
+ * never sees it, the suite reads the resulting exit 3 as the DEVICE being unreachable, and a
+ * healthy phone gets retired for a client-side clock. Named in `request` below so it says so.
+ */
+const FETCH_HEADERS_CEILING_MS = 300_000;
 // Per-call ceilings. exec is generous: a single leaf may legitimately block for its
-// whole auto-wait window or an explicit `wait --timeout`, plus device time.
+// whole auto-wait window or an explicit `wait --timeout`, plus device time. Anything here
+// above FETCH_HEADERS_CEILING_MS is aspirational unless it uses requestWithNodeHttp.
 const HEALTH_TIMEOUT_MS = 10_000;
 const ELEMENTS_TIMEOUT_MS = 60_000;
 const EXEC_TIMEOUT_MS = 10 * 60_000;
-const INSTALL_TIMEOUT_MS = 15 * 60_000;
 const DEVICE_LIST_TIMEOUT_MS = 30_000;
-// Above the server's own 4-minute boot ceiling, so the SERVER reports why a boot
-// timed out rather than the client aborting into a generic "timed out after 300s".
+// Meant to sit above the server's own 4-minute boot ceiling, so the SERVER reports why a
+// boot timed out rather than the client aborting first. It is exactly AT
+// FETCH_HEADERS_CEILING_MS, so a boot that runs the full four minutes and then some is a
+// photo finish — which is survivable only because `transportReason` now names the loser.
 const DEVICE_START_TIMEOUT_MS = 5 * 60_000;
 const DEVICE_STOP_TIMEOUT_MS = 60_000;
 const trimUrl = (url) => url.replace(/\/+$/, '');
+/**
+ * A dependency-free request path whose caller-owned timeout covers BOTH upload and response.
+ *
+ * Global fetch cannot wait past undici's fixed 300s header/body ceilings. An install may
+ * legitimately need 15 minutes, so it uses Node's native protocol clients instead of claiming
+ * a budget the transport cannot honor. Buffering the response preserves the Response boundary
+ * used below; install responses are small JSON descriptors, never artifacts.
+ */
+function requestWithNodeHttp(url, method, headers, body, timeoutMs) {
+    const endpoint = new URL(url);
+    const send = endpoint.protocol === 'http:' ? node_http_1.request : endpoint.protocol === 'https:' ? node_https_1.request : null;
+    if (!send)
+        return Promise.reject(new Error(`unsupported server protocol '${endpoint.protocol}'`));
+    const payload = body === undefined ? undefined : Buffer.isBuffer(body) ? body : Buffer.from(body);
+    const requestHeaders = {
+        ...headers,
+        ...(payload ? { 'content-length': String(payload.byteLength) } : {}),
+    };
+    return new Promise((resolve, reject) => {
+        let settled = false;
+        let timer;
+        const finish = (fn) => {
+            if (settled)
+                return;
+            settled = true;
+            if (timer)
+                clearTimeout(timer);
+            fn();
+        };
+        const req = send(endpoint, { method, headers: requestHeaders }, (res) => {
+            const chunks = [];
+            res.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
+            res.on('error', (e) => finish(() => reject(e)));
+            res.on('aborted', () => finish(() => reject(new Error('the server aborted its response'))));
+            res.on('end', () => {
+                const status = res.statusCode ?? 500;
+                finish(() => resolve(new Response(Buffer.concat(chunks), { status })));
+            });
+        });
+        req.on('error', (e) => finish(() => reject(e)));
+        timer = setTimeout(() => {
+            const timeout = Object.assign(new Error('This operation was aborted'), { name: 'AbortError' });
+            req.destroy(timeout);
+        }, timeoutMs);
+        timer.unref?.();
+        req.end(payload);
+    });
+}
 /**
  * Turn a non-2xx into the error the caller sees.
  *
@@ -66,6 +137,28 @@ function describeStatus(status, body, url) {
     }
     return new errors_1.CliError(`verikun server error ${status} at ${url}${detail}`, exitCode);
 }
+/**
+ * Why the transport failed, in words an operator can act on. PURE — exported for the tests.
+ *
+ * The undici arm is the one that earns its keep. `fetch` reports its own header/body
+ * timeouts as a bare `TypeError: fetch failed` with the real cause one level down, and that
+ * string is indistinguishable from a server that is genuinely unreachable — which is how a
+ * five-minute install came to look like a dead phone, and how a lane came to be retired for
+ * it. Say which clock ran out, and say whose it was.
+ */
+function transportReason(e, timeoutMs) {
+    const ex = e;
+    if (ex?.name === 'AbortError')
+        return `timed out after ${Math.round(timeoutMs / 1000)}s`;
+    const code = ex?.cause?.code;
+    if (code === 'UND_ERR_HEADERS_TIMEOUT' || code === 'UND_ERR_BODY_TIMEOUT') {
+        const what = code === 'UND_ERR_HEADERS_TIMEOUT' ? 'send a response' : 'finish its response';
+        return (`the server did not ${what} within ${Math.round(FETCH_HEADERS_CEILING_MS / 1000)}s — ` +
+            "this is Node's own fetch ceiling on the CLIENT, not the device. " +
+            'The server may still be working; check its log before blaming the device');
+    }
+    return ex?.message ?? String(e);
+}
 async function readBody(res) {
     try {
         return (await res.json());
@@ -89,25 +182,31 @@ class RemoteTransport {
             h.authorization = `Bearer ${this.opts.authKey}`;
         return h;
     }
-    async request(method, path, body, timeoutMs, extraHeaders = {}) {
+    async request(method, path, body, timeoutMs, extraHeaders = {}, transport = 'fetch') {
         const url = `${this.base}${path}`;
-        const controller = new AbortController();
-        const timer = setTimeout(() => controller.abort(), timeoutMs);
+        let timer;
         let res;
         try {
-            res = await fetch(url, {
-                method,
-                headers: this.headers(extraHeaders),
-                body,
-                signal: controller.signal,
-            });
+            if (transport === 'node-http') {
+                res = await requestWithNodeHttp(url, method, this.headers(extraHeaders), body, timeoutMs);
+            }
+            else {
+                const controller = new AbortController();
+                timer = setTimeout(() => controller.abort(), timeoutMs);
+                res = await fetch(url, {
+                    method,
+                    headers: this.headers(extraHeaders),
+                    body,
+                    signal: controller.signal,
+                });
+            }
         }
         catch (e) {
-            const reason = e.name === 'AbortError' ? `timed out after ${Math.round(timeoutMs / 1000)}s` : e.message;
-            throw new errors_1.CliError(`cannot reach verikun server at ${url} (${reason})`, 3);
+            throw new errors_1.CliError(`cannot reach verikun server at ${url} (${transportReason(e, timeoutMs)})`, 3);
         }
         finally {
-            clearTimeout(timer);
+            if (timer)
+                clearTimeout(timer);
         }
         if (!res.ok) {
             const body = await readBody(res);
@@ -220,15 +319,23 @@ function createRemoteBackend(opts, health) {
                 throw new errors_1.CliError(`install: cannot read '${appPath}' (${e.message})`, 2);
             }
             const sha256 = (0, node_crypto_1.createHash)('sha256').update(buf).digest('hex');
-            const res = await t.request('POST', '/v1/install', buf, INSTALL_TIMEOUT_MS, {
+            const res = await t.request('POST', '/v1/install', buf, install_timeouts_1.REMOTE_INSTALL_TIMEOUT_MS, {
                 'content-type': 'application/octet-stream',
                 'x-verikun-ext': ext,
                 'x-verikun-sha256': sha256,
-            });
+            }, 'node-http');
             // Install is the one operation the server replays elsewhere, so a move here means
             // the build DID land — on a different device than the one we started with.
             if (res.deviceChanged)
                 opts.onDeviceChange?.(res.deviceChanged);
+            // A PARTIAL install is a success, and it must not be a silent one: capacity just
+            // dropped, and the operator's next question is which phone to go and look at.
+            if (res.skipped?.length) {
+                (0, output_1.err)(`[verikun] server installed on ${(res.devices ?? []).join(', ') || '(none)'}; ` +
+                    `${res.skipped.length} device(s) could not take this build and left the pool — ` +
+                    res.skipped.map((s) => `${s.serial} (${s.reason})`).join('; '));
+                opts.onInstallSkipped?.(res.skipped);
+            }
         },
         async reset(appId) {
             // Between-test housekeeping (vk suite): the step is deliberately NOT spliced

package/dist/cli.js

@@ -2269,10 +2269,12 @@ async function resolveBackend(platform, device, flags) {
             // every recorded command, which beats a timer: it fires when work happens.
             grant: (0, grant_1.processClaimGrant)(device, claims_1.releaseOwnClaims),
             moves: [],
+            skipped: [],
         };
     }
     let runCtx = { platform, device };
     const moves = [];
+    const skipped = [];
     /** Set by the last move; the preflight below reads it to decide whether re-asking is
      *  warranted, then clears it. */
     let movedDuringCall;
@@ -2283,6 +2285,7 @@ async function resolveBackend(platform, device, flags) {
         // is identical to a local run's. logStart travels from the server's device clock
         // so archive-time / vk log scoping works without a local driver.
         onStep: (step, artifacts, logStart) => run_1.Recorder.appendForeignStep(step, artifacts, { ...runCtx, logStart }),
+        onInstallSkipped: (s) => skipped.push(...s),
         onDeviceChange: (c) => {
             moves.push(c);
             movedDuringCall = c;
@@ -2361,6 +2364,7 @@ async function resolveBackend(platform, device, flags) {
         grant: (0, grant_1.leaseGrant)(remote, serial),
         remote: { url: server, version: health.version, reads },
         moves,
+        skipped,
     };
 }
 /**
@@ -2659,7 +2663,7 @@ async function cmdInstall(positionals, flags) {
     if (!(0, node_fs_1.existsSync)(path))
         throw new errors_1.CliError(`install: '${appPath}' does not exist`, 2);
     const platform = platformFromFlags(flags);
-    const { backend, remote, moves } = await resolveBackend(platform, deviceFromFlags(flags, platform), flags);
+    const { backend, remote, moves, skipped } = await resolveBackend(platform, deviceFromFlags(flags, platform), flags);
     (0, output_1.err)(`[verikun] installing ${appPath}${remote ? ` via ${remote.url}` : ''}…`);
     try {
         await backend.install(path);
@@ -2676,11 +2680,22 @@ async function cmdInstall(positionals, flags) {
     // device than the one the run started against, and a caller acting on the old serial
     // (`adb -s … shell am start`) would be driving a phone without the build.
     const moved = moves.length ? moves[moves.length - 1] : undefined;
+    // A pooled server may have installed on some devices and dropped the rest. That is a
+    // SUCCESS — the ones that missed the build are no longer leasable, so no later lane can
+    // run the previous build and report green — but it is not a silent one: capacity changed.
     if ((0, args_1.flagBool)(flags, 'json')) {
-        (0, output_1.json)({ installed: appPath, ...(remote ? { server: remote.url } : {}), ...(moved ? { deviceChanged: moved } : {}) });
+        (0, output_1.json)({
+            installed: appPath,
+            ...(remote ? { server: remote.url } : {}),
+            ...(moved ? { deviceChanged: moved } : {}),
+            ...(skipped.length ? { skipped } : {}),
+        });
     }
     else {
         (0, output_1.out)(`installed ${appPath}${moved ? ` on ${moved.to}` : ''}`);
+        if (skipped.length) {
+            (0, output_1.out)(`skipped ${skipped.length} device(s), now out of the pool: ${skipped.map((s) => s.serial).join(', ')}`);
+        }
     }
     return 0;
 }

package/dist/device/failover.js

@@ -33,6 +33,7 @@ const ARTIFACT_RULES = [
 const TRANSIENT_RULES = [[/INSTALL_FAILED_ABORTED/, 'the install session was aborted']];
 // --- fast paths: name the reason and skip the probe. NEVER the gate. ----------
 const UNREACHABLE_RULES = [
+    [/stopped responding.*install/i, 'the device stopped responding during install'],
     [/device (?:'[^']*' )?not found/i, 'the device is not attached'],
     [/no devices\/emulators found/i, 'no device is attached'],
     [/device offline/i, 'the device is offline'],
@@ -49,6 +50,9 @@ const DEVICE_STATE_RULES = [
     // `install` already tried removing it and reinstalling (drivers/adb.ts); reaching here
     // means that did not work, so the wording must not imply nothing was attempted.
     [/INSTALL_FAILED_UPDATE_INCOMPATIBLE/, 'a differently-signed build could not be replaced on the device'],
+    // `install` already retries with `-d` (drivers/adb.ts); reaching here means the build
+    // or device isn't debuggable, so Android is still refusing — a genuine per-device
+    // conflict, not a sign the retry never ran.
     [/INSTALL_FAILED_VERSION_DOWNGRADE/, 'the device holds a newer build of this package'],
     [/INSTALL_FAILED_ALREADY_EXISTS/, 'the package is already installed on the device'],
     [/INSTALL_FAILED_DUPLICATE_PERMISSION/, 'another app on the device declares one of these permissions'],
@@ -166,8 +170,9 @@ function classifyInstallFailure(e) {
         if (named)
             return { move: true, kind: 'device-state', reason: named };
         // The inversion. Not in the denylist ⇒ it is about the device, even though we have
-        // never seen this wording. Bounded by MAX_FAILOVER_HOPS, and the caller reports the
-        // FIRST device's error on exhaustion, so a wrong guess costs time, not diagnosis.
+        // never seen this wording. Bounded by the request-derived install hop limit, and the
+        // caller reports the FIRST device's error on exhaustion, so a wrong guess costs time,
+        // not diagnosis.
         return { move: true, kind: 'device-state', reason: 'the device could not install this build', unclassified: true };
     }
     return classify(e, UNKNOWN_STAY);

package/dist/drivers/adb.js

@@ -1104,21 +1104,27 @@ class AdbDriver {
             throw new errors_1.CliError(signatureConflictHelp(appPath, pkg, 'retry-failed', second), 3);
     }
     /**
-     * One `adb install -r` attempt: null on success, else adb's collapsed output.
+     * One `adb install -r -d` attempt: null on success, else adb's collapsed output.
      *
      * Returns rather than throws because the caller has to READ the failure to decide
      * whether it is recoverable — and a thrown-and-caught CliError would put the final
      * message's own prefix inside the message it is composing.
      *
      * `-r` reinstalls over an existing package keeping its data (the common
-     * update-the-build-under-test case). A large APK can legitimately take minutes to
-     * stream + install, so the timeout is far above the 30s default. adb reports failures
-     * both as a non-zero exit AND as a `Failure [REASON]` line on stdout with exit 0
-     * (varies by adb version) — check both, and require `Success` positively rather than
-     * merely inferring it from the absence of `Failure`.
+     * update-the-build-under-test case). `-d` allows a lower versionCode to replace a
+     * higher one already on the device — unconditional here, not a retry like the
+     * signature-conflict replace below, because it carries no data-loss trade-off (inert
+     * when there is no downgrade, and keeps data the same way `-r` does when there is).
+     * Android only honors it for a debuggable build, so a release-signed downgrade still
+     * surfaces `INSTALL_FAILED_VERSION_DOWNGRADE` unresolved — see the comment on that
+     * code in `device/failover.ts`. A large APK can legitimately take minutes to stream +
+     * install, so the timeout is far above the 30s default. adb reports failures both as a
+     * non-zero exit AND as a `Failure [REASON]` line on stdout with exit 0 (varies by adb
+     * version) — check both, and require `Success` positively rather than merely inferring
+     * it from the absence of `Failure`.
      */
     tryInstall(appPath) {
-        const r = (0, exec_1.runText)(ADB, this.withSerial(['install', '-r', appPath]), { timeout: 10 * 60 * 1000 });
+        const r = (0, exec_1.runText)(ADB, this.withSerial(['install', '-r', '-d', appPath]), { timeout: 10 * 60 * 1000 });
         const combined = `${r.stdout}\n${r.stderr}`;
         if (r.code !== 0 || /^Failure\b/im.test(combined) || !/^Success\b/im.test(combined)) {
             return combined.replace(/\s+/g, ' ').trim() || `exit code ${r.code}`;

package/dist/install-timeouts.js

@@ -0,0 +1,12 @@
+"use strict";
+// The remote-install time budget is shared by the client that owns the request and the
+// server that may retry it. Keeping the numbers together prevents a server-side retry plan
+// that can never finish before the client gives up.
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MAX_INSTALL_FAILOVER_HOPS = exports.INSTALL_DEVICE_TIMEOUT_MS = exports.REMOTE_INSTALL_TIMEOUT_MS = void 0;
+/** Whole upload + install request, enforced by the remote client. */
+exports.REMOTE_INSTALL_TIMEOUT_MS = 15 * 60_000;
+/** One device gets four minutes; a measured ~170s large install still has useful margin. */
+exports.INSTALL_DEVICE_TIMEOUT_MS = 4 * 60_000;
+/** Three attempts consume at most 12m, leaving 3m of the request budget for the upload. */
+exports.MAX_INSTALL_FAILOVER_HOPS = Math.max(0, Math.floor(exports.REMOTE_INSTALL_TIMEOUT_MS / exports.INSTALL_DEVICE_TIMEOUT_MS) - 1);

package/dist/server.js

@@ -45,6 +45,7 @@ const node_stream_1 = require("node:stream");
 const promises_1 = require("node:stream/promises");
 const args_1 = require("./args");
 const errors_1 = require("./errors");
+const install_timeouts_1 = require("./install-timeouts");
 const drivers_1 = require("./drivers");
 const manager_1 = require("./companion/manager");
 const claims_1 = require("./device/claims");
@@ -86,10 +87,6 @@ const INSTALL_BODY_CAP = 512 * 1024 * 1024; // 512 MB app build
 // to survive a client-side compile/repair pause, short enough that a crashed
 // caller doesn't wedge the device.
 const LOCK_IDLE_MS = 5 * 60 * 1000;
-// How many times ONE request may move device. 2 moves = 3 devices tried, which sits
-// comfortably inside the client's 15-minute install ceiling at ~1 minute an install,
-// while a farm of ten wedged emulators cannot burn ten installs inside one request.
-const MAX_FAILOVER_HOPS = 2;
 // How often to ask whether the host's adb server has rotted. Generous on purpose: the
 // check shells out to `log show` (~1s) and the condition it looks for accumulates over
 // DAYS, so a tight interval would buy nothing and spend host time on every idle server.
@@ -108,6 +105,16 @@ const ADB_RECYCLE_CHECK_MS = 10 * 60 * 1000;
  */
 const RECONCILE_INTERVAL_MS = 60_000;
 const PROBE_RETRY_MS = 1000;
+/** Kept distinct so an all-device timeout is never mistaken for evidence of a bad build. */
+class InstallAttemptTimeoutError extends errors_1.CliError {
+    constructor(serial, timeoutMs) {
+        const duration = timeoutMs >= 1000 ? `${Math.round(timeoutMs / 1000)}s` : `${timeoutMs}ms`;
+        super(`device ${serial} stopped responding: install exceeded its ${duration} per-device deadline`, 3);
+        this.name = 'InstallAttemptTimeoutError';
+    }
+}
+/** Failover wraps an exhausted attempt in HttpError, so retain a wire-safe marker too. */
+const isInstallAttemptTimeout = (e) => e instanceof InstallAttemptTimeoutError || (e instanceof Error && /per-device deadline/.test(e.message));
 // Deliberately below the client's 5-minute ceiling, so a slow boot is reported by the
 // side that knows WHY ("did not finish booting within 240s") rather than as a generic
 // client-side abort.
@@ -519,7 +526,26 @@ function buildServer(config) {
         : null;
     // Like the reconcile timer: must never hold the process open at Ctrl-C.
     recycleTimer?.unref?.();
-    const pickFailoverDevice = (failed, reason) => serializeFailover(() => pickFailoverDeviceLocked(failed, reason));
+    /**
+     * Is this failure grounds to REMOVE the device from the pool, rather than merely deal it
+     * last? Two conditions, and both are necessary.
+     *
+     * `unreachable` is the only kind that qualifies, because it is the only one that says the
+     * device is not there. Every other kind describes a device that is present and unhappy —
+     * a full disk, a wedged app, an exit 3 nobody has classified — and for those, demotion
+     * plus recovery-by-traffic is right and this must not change: they can still produce the
+     * traffic that clears them. An absent device cannot, which is the whole defect (#139): the
+     * demotion is a sort key (`leaseFor`), so "dealt last" is still dealt, every round, and
+     * `restoreDevice` can never fire for a device that will never answer again.
+     *
+     * And only on a POOLED server, because only a pooled server sweeps. `reconcileOnce`
+     * returns immediately without `poolSpec` (`wantedSerials`) and its timer is never even
+     * created — see `ServerConfig.poolSpec`, "deliberately does not reconcile". Shedding
+     * where nothing readmits would trade a device that fails loudly for a server that is
+     * empty until someone restarts it: a worse failure, and a new one.
+     */
+    const shedOnFailure = (kind) => kind === 'unreachable' && config.poolSpec !== undefined;
+    const pickFailoverDevice = (failed, reason, kind) => serializeFailover(() => pickFailoverDeviceLocked(failed, reason, kind));
     /**
      * Bring in a healthy replacement for `failed`. Returns the serial moved to, or null
      * when none remains (which is not an error here — the caller reports the ORIGINAL
@@ -533,7 +559,7 @@ function buildServer(config) {
      * only reports ready once its OWN `preflight()` has passed, so starting the worker IS
      * the probe, run on the thread that will go on to use it.
      */
-    const pickFailoverDeviceLocked = async (failed, reason) => {
+    const pickFailoverDeviceLocked = async (failed, reason, kind) => {
         const policy = config.failover;
         if (!policy)
             return null;
@@ -549,23 +575,29 @@ function buildServer(config) {
         // see and where the server will actually go cannot drift. A pool member's own driver
         // is not: it may be pointed at a corpse.
         /**
-         * Nothing healthier exists. Shed the failed device — continuing to hand it out is
-         * what makes a pool a coin flip per lease — but ONLY while another remains.
+         * Nothing healthier exists. Decide what becomes of the failed device itself — THREE
+         * outcomes, not two, and which one applies is `shedOnFailure`'s question:
          *
-         * The LAST device stays, deliberately. A server that shed it would answer every
-         * later request `503 no device attached`, replacing the device's own error (full
-         * disk, no space, whatever it actually was) with a message that names nothing. A
-         * caller stuck on one broken device is better served by the truth about it.
+         *  - GONE, on a pooled server — shed it. It cannot serve and cannot recover by
+         *    traffic, so leaving it in the pool means dealing it forever (#139). The sweep
+         *    owns readmission, so capacity comes back on its own.
+         *  - present but unhappy — demote it: worker, claim and slot kept, dealt last,
+         *    restored by the first command that works.
+         *  - already left on its own (its worker died) — nothing to remove, just clean up.
+         *
+         * The last two share a tail with the first, because "stop serving this device" has the
+         * same consequences however it came about.
          */
         const shrink = async () => {
             // A device whose worker DIED is already out of the pool, so there is nothing left to
             // shed — but its holder still has to be evicted and its claim and companion handed
-            // back, and the "last device stays" guard below must not skip that. Asking whether
-            // it is still a member is what separates the two cases.
+            // back. Asking whether it is still a member is what separates that case from a
+            // device we are removing ourselves.
             const serving = pool.serials().includes(failed);
-            if (serving) {
-                // DEMOTE, never shed. The device keeps its worker, its claim and its place in the
-                // pool; it is simply dealt last until it does some work (see `degradeDevice`).
+            if (serving && !shedOnFailure(kind)) {
+                // DEMOTE — the device is still THERE. It keeps its worker, its claim and its place
+                // in the pool; it is simply dealt last until it does some work (see
+                // `degradeDevice`). Contrast the shed below, which is only for a device that is not.
                 //
                 // This replaces "nothing healthier to move to — X left the pool". That rule read
                 // correctly on a SINGLE-device server, where it never actually fired (the last
@@ -573,10 +605,16 @@ function buildServer(config) {
                 // verdict — because a pool's own members are excluded from its candidate list, so
                 // "no candidate" is the normal case rather than the exceptional one. The argument
                 // for shedding was that continuing to hand out a broken device makes a pool a coin
-                // flip per lease; that is answered by ORDERING (a degraded device is chosen only
-                // when nothing else is free), which costs no capacity. And it is the same judgement
-                // the last-device branch already made out loud: a caller stuck on one broken device
-                // is better served by the truth about it than by a server that quietly halved.
+                // flip per lease; for a device that is PRESENT that is answered by ORDERING (a
+                // degraded device is chosen only when nothing else is free), which costs no
+                // capacity, and a caller that does reach it is better served by the truth about it
+                // than by a server that quietly halved.
+                //
+                // Ordering answers it only while the device can still come back, though. It cannot
+                // answer for a device that is GONE — "dealt last" is still dealt once the healthy
+                // devices are busy, which on a suite sized to the pool is every round, and no
+                // amount of ordering produces the traffic `restoreDevice` needs. That case is
+                // shed above, by `shedOnFailure`.
                 //
                 // The holder keeps its lease too: its device did not go anywhere, so there is no
                 // `deviceChanged` to send and nothing for the run to seal. The step that failed
@@ -585,7 +623,28 @@ function buildServer(config) {
                 degradeDevice(failed, reason);
                 return null;
             }
-            // NOT serving — its worker already died, so the device left on its own and there is
+            if (serving) {
+                // SHED. The device is gone and this server sweeps, so removing it is not the
+                // one-way ratchet it was before the sweep existed (#114): `reconcileOnce` lists it
+                // as missing from what `--devices` asked for, retries with backoff, and
+                // `rejoinDevice` readmits it — bringing it up to `lastInstall` first — the moment
+                // it answers again. Capacity returns without anyone restarting anything.
+                //
+                // It keeps its QUARANTINE, unlike the demote branch above, and that asymmetry is
+                // the point: quarantine means "not serving, and ruled out", degradation means
+                // "serving but suspect", and the two are disjoint precisely so `/v1/health` and
+                // `exhaustedNote` can be read. A shed device genuinely is not serving, so it
+                // belongs in the same list as one whose worker died — which is the tail below,
+                // reached from here. `rejoinDevice` clears it on evidence, never on a clock.
+                //
+                // `degraded` must be given up though: it is defined as pool MEMBERS that recently
+                // failed, and a non-member left in it would have `/v1/health` reporting a device it
+                // no longer serves, in a list whose whole meaning is that it still does.
+                pool.retire(failed);
+                degraded.delete(failed);
+                (0, output_1.err)(`[server] pool: ${failed} left the pool — ${reason} (the sweep readmits it when it answers again)`);
+            }
+            // NOT serving — its worker died, or the shed above just removed it, so there is
             // nothing to demote. The holder is EVICTED, not migrated: without a replacement there
             // is no `deviceChanged` to send, so the client never learns to seal its run — and
             // merely dropping the lease would let its next request silently draw some other device
@@ -690,7 +749,14 @@ function buildServer(config) {
      * Is this device actually gone? Two probes a second apart, because that gap is the only
      * thing separating a USB re-enumeration or a mid-`launch --clear` gap from a dead box —
      * and quarantining a healthy device is the expensive mistake here. Returns the reason
-     * when dead, undefined when it was a blip.
+     * AND the probe's own verdict kind when dead, undefined when it was a blip.
+     *
+     * The kind is carried out because the probe is often the better-classified of the two
+     * failures. The operation that brought us here may have failed with a string nothing
+     * recognises (an unclassified exit 3, which is what earns a probe in the first place),
+     * while `preflight` on a detached phone says `device '<serial>' not found` — the exact
+     * `UNREACHABLE_RULES` wording. Reporting the ORIGINAL verdict's kind there would decide
+     * "shed or demote" from the vaguer of two answers about the same device.
      */
     const deviceIsDead = async (handle) => {
         let last = '';
@@ -722,7 +788,7 @@ function buildServer(config) {
             (0, output_1.err)(`[server] probe on ${handle.serial}: ${verdict.reason} (${verdict.kind}) — a host problem, not this device`);
             return undefined;
         }
-        return last || 'the device stopped answering';
+        return { reason: last || 'the device stopped answering', kind: verdict.kind };
     };
     /**
      * A non-install operation failed. Move off the device if it is genuinely at fault —
@@ -761,6 +827,7 @@ function buildServer(config) {
         try {
             const verdict = (0, failover_1.classifyFailure)(e);
             let reason = verdict.reason;
+            let kind = verdict.kind;
             if (!verdict.move) {
                 // Only an unrecognised exit 3 earns a probe; `transient` and `toolchain` set
                 // probe:false precisely so a mid-launch gap or a missing adb cannot become a move.
@@ -777,15 +844,21 @@ function buildServer(config) {
                     (0, output_1.err)(`[server] ${what}: ${from} failed but probes healthy — staying (${verdict.reason})`);
                     return undefined; // a blip — the test rerun is the right answer, not a new device
                 }
-                reason = dead;
+                reason = dead.reason;
+                // The PROBE's verdict, not the original failure's. We are here because the
+                // operation failed with something nothing recognised; `preflight` on a detached
+                // phone says `device '<serial>' not found`, which is classified. Taking the vaguer
+                // of two answers about the same device is how a detachment that first showed up as
+                // an odd exit 3 would be demoted forever instead of shed.
+                kind = dead.kind;
                 noteVerdict({ ...verdict, move: true }, e, what);
             }
             (0, output_1.err)(`[server] ${what}: FAILED on ${from} — ${reason}`);
             quarantineDevice(from, reason);
-            // pickFailoverDevice has already said which of the two no-move outcomes happened —
-            // the device was shed, or it was the last one and stayed. A second line here would
-            // contradict one of them.
-            const to = await pickFailoverDevice(from, reason);
+            // pickFailoverDevice has already said which no-move outcome happened — the device
+            // was shed, or demoted, or had already left. A second line here would contradict
+            // one of them.
+            const to = await pickFailoverDevice(from, reason, kind);
             if (!to)
                 return undefined;
             return { from, to, reason, retried: false };
@@ -958,6 +1031,9 @@ function buildServer(config) {
             return new server_http_1.HttpError(409, 'the device this run was using left the pool and nothing healthy replaced it — ' +
                 'start a fresh run; this one cannot continue on another device', 3);
         }
+        // An empty pool never reaches here: the deviceless guard in the router answers 503 for
+        // every route that takes a lease, and it names `lostDevice` while doing it. So `n` is
+        // always >= 1 and this only ever describes CONTENTION, which is what 409 means.
         const n = pool.serials().length;
         return new server_http_1.HttpError(409, n > 1
             ? `all ${n} devices are leased by other active runs — retry when one finishes`
@@ -1182,7 +1258,7 @@ function buildServer(config) {
      * On exhaustion it throws the FIRST device's error, never the last. That inversion is
      * what makes move-by-default safe: a wrong move costs time, not the diagnosis.
      */
-    async function installWithFailover(serial, tmpPath) {
+    async function installWithFailover(serial, tmpPath, timedOut) {
         let change;
         let moves = 0;
         let firstError;
@@ -1197,9 +1273,9 @@ function buildServer(config) {
          * wrapper keeps no result on a throw), so dropping it leaves the operator holding a
          * serial the server has already left.
          */
-        const hopOrThrow = async (why, giveUp) => {
+        const hopOrThrow = async (why, giveUp, kind) => {
             quarantineDevice(from, why);
-            const to = await pickFailoverDevice(from, why);
+            const to = await pickFailoverDevice(from, why, kind);
             if (to === null) {
                 // A pool that emptied with nothing having moved keeps its own 503 — a more
                 // accurate status than a wrapped 500.
@@ -1222,13 +1298,36 @@ function buildServer(config) {
                 // are still on our disk), so it hops rather than throwing a bare 503 that never
                 // reaches the failover machinery at all.
                 const gone = firstError ?? new server_http_1.HttpError(503, `device ${from} is no longer attached`, 3);
-                if (!config.failover || hop >= MAX_FAILOVER_HOPS)
+                if (!config.failover || hop >= install_timeouts_1.MAX_INSTALL_FAILOVER_HOPS)
                     throw gone;
-                from = await hopOrThrow('the device left the pool mid-install', gone);
+                // `unreachable` is the literal truth — it is not in the pool — and it is also
+                // inert here: `shrink` sees a non-member and takes its cleanup tail either way.
+                from = await hopOrThrow('the device left the pool mid-install', gone, 'unreachable');
                 continue;
             }
             try {
-                await handle.install(tmpPath);
+                const timeoutMs = config.installAttemptMs ?? install_timeouts_1.INSTALL_DEVICE_TIMEOUT_MS;
+                await new Promise((resolve, reject) => {
+                    let settled = false;
+                    const finish = (fn) => {
+                        if (settled)
+                            return;
+                        settled = true;
+                        clearTimeout(timer);
+                        fn();
+                    };
+                    const timer = setTimeout(() => {
+                        const timeout = new InstallAttemptTimeoutError(from, timeoutMs);
+                        // A race alone would leave the worker blocked in adb forever. Retiring it is
+                        // what releases the transport and makes the deadline a recovery mechanism.
+                        timedOut.add(from);
+                        quarantineDevice(from, timeout.message);
+                        pool.retire(from);
+                        finish(() => reject(timeout));
+                    }, timeoutMs);
+                    timer.unref?.();
+                    void handle.install(tmpPath).then(() => finish(resolve), (e) => finish(() => reject(e)));
+                });
                 return { change, moves };
             }
             catch (e) {
@@ -1239,9 +1338,19 @@ function buildServer(config) {
                 noteVerdict(verdict, e, 'install');
                 // The artifact is broken / the caller is wrong / failover is off / we are out of
                 // hops: report the first failure unchanged, exactly as before this feature.
-                if (!verdict.move || !config.failover || hop >= MAX_FAILOVER_HOPS)
+                if (!verdict.move || !config.failover || hop >= install_timeouts_1.MAX_INSTALL_FAILOVER_HOPS) {
+                    // A timeout retired its worker before this catch. If it cannot move, finish the
+                    // cleanup that pickFailoverDevice would otherwise own; especially, never leave
+                    // a host-global claim pinned to a worker that no longer exists.
+                    if (isInstallAttemptTimeout(e)) {
+                        evictHoldersOf(from, `${from} left the pool after its install timed out`);
+                        (0, manager_1.releaseCompanionOn)(from);
+                        if ((0, claims_1.claimsEnabled)(claimEnv))
+                            (0, claims_1.releaseClaim)(from, { ...claimOpts, mineOnly: true });
+                    }
                     throw firstError;
-                from = await hopOrThrow(verdict.reason, firstError);
+                }
+                from = await hopOrThrow(verdict.reason, firstError, verdict.kind);
             }
         }
     }
@@ -1290,9 +1399,10 @@ function buildServer(config) {
                 throw new server_http_1.HttpError(503, `no device is left to install onto${lostDevice ? ` — last loss: ${lostDevice}` : ''}`, 3);
             }
             (0, output_1.err)(`[server] install: received ${size} bytes (.${ext}), installing on ${targets.join(', ')}…`);
+            const timedOut = new Set();
             const outcomes = await Promise.all(targets.map(async (serial) => {
                 try {
-                    return { serial, ...(await installWithFailover(serial, tmpPath)), error: null };
+                    return { serial, ...(await installWithFailover(serial, tmpPath, timedOut)), error: null };
                 }
                 catch (e) {
                     return { serial, change: undefined, moves: 0, error: e };
@@ -1300,6 +1410,12 @@ function buildServer(config) {
             }));
             const failed = outcomes.filter((o) => o.error);
             const moved = outcomes.filter((o) => o.change);
+            // Where each outcome ENDED UP. An install that moved ran on its replacement, not on
+            // the serial it started from, so the device that holds this build — or conspicuously
+            // does not — is the last one it was on, never `o.serial`.
+            const landedOn = (o) => o.change?.to ?? o.serial;
+            const installed = outcomes.filter((o) => !o.error).map(landedOn);
+            let skipped = [];
             if (failed.length) {
                 // One artifact, many devices: if it failed everywhere the file is the suspect, so
                 // surface the FIRST device's error unchanged rather than a summary that buries it.
@@ -1310,32 +1426,91 @@ function buildServer(config) {
                     // polarity, since the device side is open-ended and OEM-specific. That is right
                     // for ONE device failing; applied to every device at once it condemns the whole
                     // pool for what this very branch has just concluded is a bad build. Undo them.
-                    const condemned = targets.filter((t) => quarantine.delete(t));
-                    if (condemned.length) {
-                        (0, output_1.err)(`[server] install: failed on every device, so the build is the suspect — un-quarantining ${condemned.join(', ')}`);
+                    const hasTimeout = timedOut.size > 0;
+                    if (!hasTimeout) {
+                        // No lane succeeded and none hung: the common input is now the stronger
+                        // suspect, so undo the per-device quarantines made by the move-by-default
+                        // classifier.
+                        const condemned = targets.filter((t) => quarantine.delete(t));
+                        if (condemned.length) {
+                            (0, output_1.err)(`[server] install: failed on every device, so the build is the suspect — un-quarantining ${condemned.join(', ')}`);
+                        }
+                    }
+                    else {
+                        // A deadline is direct evidence about a device, not the artifact. Preserve
+                        // every quarantine when any lane timed out; otherwise the next request could
+                        // immediately be dealt the same wedged worker again.
+                        (0, output_1.err)('[server] install: every device failed and at least one timed out — keeping the device quarantines');
                     }
                     throw failed[0].error;
                 }
-                // Carry a move that DID happen even though the overall install failed: the
-                // client re-points its run context on `deviceChanged`, and dropping it here
-                // would leave the operator holding a serial the server has already left.
-                throw new server_http_1.HttpError(500, `install failed on ${failed.map((f) => `${f.serial} (${(0, server_http_1.firstLine)(f.error.message)})`).join('; ')}`, 3, moved[0]?.change);
+                // PARTIAL. Two healthy phones took the build and one did not. Answering 500 for the
+                // whole pool is what turned one detached device into a dead CI job (#139) — and it
+                // dies at the install step, so the run has already paid for an app build and tested
+                // nothing.
+                //
+                // What made the 500 defensible is the fan-out's own rule, one line up: a lane dealt
+                // a device that missed this build runs the PREVIOUS one and reports green, which is
+                // the worst result this server can produce. The answer is not to soften that rule
+                // but to SATISFY it — a device that did not take the build leaves the pool, so no
+                // lease can reach it. `rejoinDevice` already makes exactly this call out loud
+                // ("serving the wrong build is worse than not serving") and offers the same remedy:
+                // the sweep readmits it and installs `lastInstall` before it is dealt any work.
+                //
+                // Done regardless of `config.failover`. The kill switch governs MOVING BETWEEN
+                // devices; it was never a licence to serve a stale build, and the sweep that brings
+                // the device back is gated on `poolSpec`, not on failover.
+                skipped = failed.map((f) => {
+                    const serial = landedOn(f);
+                    const reason = (0, server_http_1.firstLine)(f.error.message);
+                    const serving = pool.serials().includes(serial);
+                    if (serving)
+                        pool.retire(serial);
+                    // Same two rules as the shed in `shrink`: `degraded` is for MEMBERS, and a
+                    // device that is not serving belongs in `quarantine` — which `rejoinDevice`
+                    // clears on the evidence of a worker that started and a build that installed.
+                    // The timeout path retired its worker already, but still needs this common
+                    // bookkeeping tail. Other already-absent failures keep their existing handling.
+                    if (serving || isInstallAttemptTimeout(f.error)) {
+                        degraded.delete(serial);
+                        quarantineDevice(serial, `did not take the current build — ${reason}`);
+                        evictHoldersOf(serial, `${serial} left the pool without the current build`);
+                        (0, manager_1.releaseCompanionOn)(serial);
+                        if ((0, claims_1.claimsEnabled)(claimEnv))
+                            (0, claims_1.releaseClaim)(serial, { ...claimOpts, mineOnly: true });
+                    }
+                    return { serial, reason };
+                });
+                (0, output_1.err)(`[server] install: partial — ${installed.join(', ')} took the build; ` +
+                    `removed from the pool: ${skipped.map((s) => `${s.serial} (${s.reason})`).join('; ')}`);
             }
             for (const m of moved)
                 (0, output_1.err)(`[server] install: ${m.serial} → ${m.change.to} (${m.moves} move(s))`);
-            (0, output_1.err)(`[server] install: done on ${pool.serials().join(', ')}`);
+            (0, output_1.err)(`[server] install: done on ${installed.join(', ')}`);
             // Retain the artifact so a device that rejoins later can be brought up to this build
             // (see `rejoinDevice`). Renamed out of the per-request temp name into one stable slot,
             // so at most one build is ever held and each install replaces the last.
+            //
+            // Reached on a PARTIAL install too, and load-bearing there: the devices just removed
+            // are precisely the ones the sweep will readmit, and `rejoinDevice` brings a returning
+            // device up to `lastInstall`. Retaining only on a clean sweep would hand each of them
+            // the PREVIOUS build on the way back in — and `rejoinDevice`'s own check would pass,
+            // because an install that succeeds is all it can see.
             retainInstall(tmpPath, ext);
             retained = true;
+            // Only a move whose destination SURVIVED is worth reporting. The client re-points its
+            // run context on `deviceChanged`, so naming a device the partial branch retired three
+            // lines ago would send its next step to a serial this server no longer serves.
+            const survivors = new Set(pool.serials());
+            const reportableMove = moved.map((m) => m.change).find((c) => survivors.has(c.to));
             const body = {
                 ok: true,
                 bytes: size,
                 sha256: digest,
-                devices: pool.serials(),
+                devices: installed,
+                ...(skipped.length ? { skipped } : {}),
                 // The wire field is singular; a pool that moved more than one device logs the rest.
-                ...(moved.length ? { deviceChanged: moved[0].change } : {}),
+                ...(reportableMove ? { deviceChanged: reportableMove } : {}),
             };
             (0, server_http_1.sendJson)(res, 200, body);
         }
@@ -1360,10 +1535,12 @@ function buildServer(config) {
      * phone another job is mid-test on. Refusing plainly beats a rule nobody can predict.
      * The GET listing stays available, because reading what is attached is safe.
      *
-     * KNOWN COST, stated in the refusal so nobody has to discover it: this is also the only
-     * thing that clears a quarantine, so on a pool a device ruled out by failover stays out
-     * until the server is restarted. That is the price of refusing rather than guessing;
-     * lifting it would mean letting a NAMED, allowlisted target act on one pool member.
+     * This used to carry a known cost — that it was the ONLY thing clearing a quarantine, so
+     * on a pool a device ruled out by failover stayed out until the server was restarted.
+     * That is no longer true and the refusal no longer says it: `rejoinDevice` clears the
+     * quarantine (and `degraded`, and `failedOver`) when the sweep readmits a device, on the
+     * evidence of a worker that started and a build that installed. The refusal itself stands
+     * — power-cycling one member of a pool another job is mid-test on is what it prevents.
      */
     function requireSingleDevice(op) {
         const n = pool.serials().length;

package/dist/version.js

@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.VERSION = void 0;
 // GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
 // (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
-exports.VERSION = '0.27.1';
+exports.VERSION = '0.29.0';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "verikun",
-  "version": "0.27.1",
+  "version": "0.29.0",
   "description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
   "keywords": [
     "android",