verikun 0.16.0 → 0.17.0

package/README.md

@@ -104,7 +104,7 @@ vk screenshot                   # -> ./.verikun/screen.png
 | Command | Description |
 |---|---|
 | `ai <file> [--model m] [--max-cost-usd n] [--timeout dur] [--cost-override in/out] [--effort e] [--package pkg] [--app-build id] [--server url] [--show-plan] [--recompile] [--json]` | Run a plain-English test: compile it to a deterministic plan once, replay it model-free, and self-heal failures via the model. Needs `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` (per model), or no key with `--model codex-cli` / `cursor-cli` (a logged-in `codex` / `cursor-agent` CLI). See [AI](#ai--natural-language-tests). |
-| `suite <dir> [--app <id>] [--name n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
+| `suite <dir> [--app <id>] [--name n] [--retries n] [--server url] [--json]` (+ all `ai` flags) | Run every `*.md` in `<dir>` as one sequential suite with an overview report and a non-zero exit on failure — the CI gate. See [Suites](#suites--run-a-directory-of-tests). |
 
 ### Remote
 | Command | Description |
@@ -287,12 +287,29 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER"   # remote devi
 - **Each test is a full `vk ai` run** — plan cache, self-healing, cost budget, and
   its own archived JUnit + HTML report under `./.verikun/runs/<id>/`. A test that
   fails (or errors) doesn't stop the suite; the rest still run.
-- **But a broken *environment* does stop it.** If a test dies from an environment
-  error (exit 3 — tool gone, device unplugged, server unreachable), the toolchain is
-  re-probed; only if it is *still* broken does the suite abort. That re-probe matters:
-  a transient `uiautomator` dump failure also exits 3, and shouldn't vaporize a
-  20-test run. Continuing on a genuinely dead box just produces one identical red row
-  per remaining test — noise that reads exactly like a mass regression.
+- **`--retries N` recovers from flakes.** A failed test is re-run up to N times
+  (default `0` — opt-in, so CI cost/time stay predictable). If a later attempt
+  passes, the suite exits `0` and the flake is a **warning**, not a hard failure.
+  Failed attempt archives stay linked from the suite overview (`attempts` on the
+  test row + a `warnings` list on the manifest), so flakiness remains visible.
+  Cost and duration sum across attempts.
+- **What earns a retry:** anything that might come out differently — a flaky
+  selector, a wedged app, and **a broken environment**, including a `vk server`
+  connection dropping mid-suite. The bias is intentional: an attempt costs one test,
+  giving up costs the whole suite plus a human rerunning it. Environment retries wait
+  a little longer each time (an outage that survives the health probe usually needs
+  seconds, not milliseconds) and each one lands in `warnings`, so riding out a wobble
+  is never silent. Exactly two failures are never retried, because a rerun cannot
+  change them: a **budget abort** (each attempt gets its own ceiling, so it would just
+  re-abort having spent twice) and a **usage error** (exit `2` — an unreadable test
+  file, a payload the server refuses).
+- **But a broken *environment* does stop it, once the attempts are gone.** If a test
+  dies from an environment error (exit 3 — tool gone, device unplugged, server
+  unreachable), the toolchain is re-probed; only if it is *still* broken **and** no
+  retries remain does the suite abort. That re-probe matters: a transient
+  `uiautomator` dump failure also exits 3, and shouldn't vaporize a 20-test run.
+  Continuing on a genuinely dead box just produces one identical red row per remaining
+  test — noise that reads exactly like a mass regression.
 - **The suite writes an overview** to `./.verikun/suites/<id>/`:
   - **`index.json`** — a stable, `schemaVersion`ed manifest: per-test pass/fail,
     steps, model repairs, cost, duration, and the run id, plus suite totals. This
@@ -300,17 +317,21 @@ vk suite tests/ --app com.example.app --server "$VERIKUN_SERVER"   # remote devi
     it (see the [CI recipe](#ci-recipe)) instead of verikun growing upload plugins.
     On an abort it also carries `aborted: {reason, notRun}`; the not-run tests get
     **no rows and no place in `totals`**, so `passed + failed === tests` still holds
-    and nothing downstream mistakes a skipped test for a regression.
+    and nothing downstream mistakes a skipped test for a regression. Retried flakes
+    add `flaky` / `attempts` on the test row and suite-level `warnings` (additive;
+    `schemaVersion` stays `1`).
   - **`index.html`** — a summary page linking every test's `report.html`, with a
-    banner naming the not-run tests when the suite aborted.
-- **Exit code is the CI gate:** `0` all green · `1` a test failed · `2` bad/empty
-  directory · `3` environment (the provider or the device toolchain is unavailable,
-  or the box broke mid-run). The `1`-vs-`3` split is the point: `1` is a regression
-  to investigate, `3` is a machine to fix. All `ai` flags (`--model`,
-  `--max-cost-usd`, `--timeout`, …) apply to every test; both the provider
-  (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`, or the `codex` / `cursor-agent` CLI for
-  `--model codex-cli` / `cursor-cli`) **and** the device toolchain (`adb` / `idb` +
-  a resolvable device) are checked up front, before anything is compiled.
+    banner naming the not-run tests when the suite aborted, and a warnings banner
+    when a flake recovered on retry (prior failed attempts stay linked).
+- **Exit code is the CI gate:** `0` all green (including flakes that recovered with
+  `--retries`) · `1` a test failed · `2` bad/empty directory · `3` environment (the
+  provider or the device toolchain is unavailable, or the box broke mid-run). The
+  `1`-vs-`3` split is the point: `1` is a regression to investigate, `3` is a
+  machine to fix. All `ai` flags (`--model`, `--max-cost-usd`, `--timeout`, …)
+  apply to every test; both the provider (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY`,
+  or the `codex` / `cursor-agent` CLI for `--model codex-cli` / `cursor-cli`)
+  **and** the device toolchain (`adb` / `idb` + a resolvable device) are checked up
+  front, before anything is compiled.
 
 ## Remote devices — `vk server`
 

package/dist/cli.js

@@ -1537,7 +1537,7 @@ async function cmdInstall(positionals, flags) {
 async function cmdSuiteEntry(positionals, flags) {
     const dirArg = positionals[0];
     if (!dirArg)
-        throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--json]', 2);
+        throw new errors_1.CliError('Usage: verikun suite <dir> [--app <id>] [--server url] [--name n] [--retries n] [--json]', 2);
     const opts = parseAiOptions(flags);
     // Pre-flight the provider BEFORE touching any device/server: every test needs it
     // to compile (on a cache miss) or to repair at runtime.
@@ -1849,14 +1849,19 @@ AI (run a natural-language test — compile once, replay model-free, self-heal)
                                       codex-cli | cursor-cli.
 
 SUITE (run a directory of natural-language tests as one gated suite)
-  suite <dir> [--app <id>] [--name n] [--json]  (+ all \`ai\` flags, incl. --server)
+  suite <dir> [--app <id>] [--name n] [--retries n] [--json]
+                                      (+ all \`ai\` flags, incl. --server)
                                       Run every *.md in <dir> (lexicographic order —
                                       prefix 01-, 02- to sequence; README.md skipped)
                                       through \`vk ai\`. With --app, app data is reset
-                                      between tests (iOS: force-stop). Writes a suite
-                                      overview to ./.verikun/suites/<id>/{index.json,
-                                      index.html} linking each test's report. Exits 1
-                                      if any test failed — the CI gate.
+                                      between tests (iOS: force-stop). --retries N
+                                      re-runs a failed test up to N times; a later
+                                      pass recovers the suite (exit 0) and surfaces a
+                                      warning, keeping failed-attempt evidence in the
+                                      report. Writes a suite overview to
+                                      ./.verikun/suites/<id>/{index.json, index.html}
+                                      linking each test's report. Exits 1 if any test
+                                      failed — the CI gate.
 
 SERVER (expose a locally-connected device to remote verikun clients)
   server [--bind addr] [--port n] [--auth-key k] [--allow-install]

package/dist/report.js

@@ -118,6 +118,7 @@ const STYLE = `
   .summary { display:flex; gap:8px; flex-wrap:wrap; align-items:center; margin-bottom: 20px; }
   .chip { font-weight:600; font-size:13px; padding:4px 10px; border-radius:999px; color:#fff; }
   .chip.pass{background:var(--pass)} .chip.fail{background:var(--fail)} .chip.err{background:var(--err)}
+  .chip.warn{background:var(--err)}
   .chip.muted{ background:#eaeef2; color:var(--muted); }
   ol.steps { list-style:none; margin:0; padding:0; }
   li.step { background:#fff; border:1px solid var(--line); border-left-width:4px; border-radius:8px; margin-bottom:10px; padding:12px 14px; }
@@ -206,19 +207,40 @@ const SUITE_STYLE = `
   table.tests td.num { text-align:right; font-variant-numeric:tabular-nums; white-space:nowrap; }
   table.tests a { color:inherit; }
   .fail-reason { color:var(--fail); font-size:12px; margin-top:2px; }
+  .flake-note { color:var(--err); font-size:12px; margin-top:2px; }
+  .attempts { margin-top:4px; font-size:12px; color:var(--muted); }
+  .attempts a { color:var(--fail); }
   .aborted { background:#fff4e5; border:1px solid #f0b429; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
   .aborted strong { color:#8a5300; }
   .aborted ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
+  .warnings { background:#fff8c5; border:1px solid #d4a72c; border-radius:8px; padding:12px 14px; margin:0 0 14px; font-size:13px; }
+  .warnings strong { color:#7d4e00; }
+  .warnings ul { margin:6px 0 0; padding-left:20px; color:var(--muted); }
 `;
+function suiteAttemptLinks(attempts, linkBase) {
+    const links = attempts
+        .map((a, i) => {
+        const label = `attempt ${i + 1}`;
+        if (!a.id)
+            return htmlEsc(label);
+        return `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(a.id)}/report.html`)}">${htmlEsc(label)}</a>`;
+    })
+        .join(', ');
+    return `<div class="attempts">prior failed: ${links}</div>`;
+}
 function suiteTestRow(t, linkBase) {
     // A test that errored before its run started (id '') has no report to link.
     const label = t.id
         ? `<a href="${htmlEsc(`${linkBase}runs/${encodeURIComponent(t.id)}/report.html`)}">${htmlEsc(t.name)}</a>`
         : htmlEsc(t.name);
     const failure = t.failure ? `<div class="fail-reason">${htmlEsc(t.failure)}</div>` : '';
+    const flake = t.flaky ? `<div class="flake-note">passed on retry (flake)</div>` : '';
+    const prior = t.attempts?.length ? suiteAttemptLinks(t.attempts, linkBase) : '';
+    const status = t.flaky ? 'FLAKY' : t.ok ? 'PASS' : 'FAIL';
+    const statusClass = t.ok ? 'passed' : 'failed';
     return `  <tr>
-    <td><span class="st ${t.ok ? 'passed' : 'failed'}">${t.ok ? 'PASS' : 'FAIL'}</span></td>
-    <td>${label}${failure}</td>
+    <td><span class="st ${statusClass}">${status}</span></td>
+    <td>${label}${flake}${failure}${prior}</td>
     <td class="num">${t.passedSteps}/${t.steps}${t.failedSteps ? ` (${t.failedSteps} failed)` : ''}</td>
     <td class="num">${t.modelRepairs || ''}</td>
     <td class="num">$${t.costUsd.toFixed(4)}</td>
@@ -233,9 +255,11 @@ function suiteTestRow(t, linkBase) {
 function toSuiteHtml(suite, opts = {}) {
     const linkBase = opts.linkBase ?? '../../';
     const t = suite.totals;
+    const flaky = suite.tests.filter((x) => x.flaky).length;
     const chips = [
         `<span class="chip pass">${t.passed} passed</span>`,
         t.failed ? `<span class="chip fail">${t.failed} failed</span>` : '',
+        flaky ? `<span class="chip warn">${flaky} flaky</span>` : '',
         suite.aborted ? `<span class="chip fail">ABORTED</span>` : '',
         `<span class="chip muted">${t.tests} tests &middot; ${t.steps} steps &middot; ${fmtDuration(t.durationMs)} &middot; $${t.costUsd.toFixed(4)}</span>`,
     ]
@@ -250,6 +274,13 @@ function toSuiteHtml(suite, opts = {}) {
 ${suite.aborted.notRun.length
             ? `    <ul>${suite.aborted.notRun.map((f) => `<li>${htmlEsc(f)} — not run</li>`).join('')}</ul>\n`
             : ''}  </div>
+`
+        : '';
+    const warningsBanner = suite.warnings?.length
+        ? `  <div class="warnings">
+    <strong>Warnings</strong>
+    <ul>${suite.warnings.map((w) => `<li>${htmlEsc(w)}</li>`).join('')}</ul>
+  </div>
 `
         : '';
     const metaBits = [
@@ -274,7 +305,7 @@ ${suite.aborted.notRun.length
   <div class="summary">
       ${chips}
   </div>
-${abortedBanner}  <table class="tests">
+${abortedBanner}${warningsBanner}  <table class="tests">
     <thead><tr><th></th><th>Test</th><th>Steps</th><th>Repairs</th><th>Cost</th><th>Duration</th></tr></thead>
     <tbody>
 ${suite.tests.map((x) => suiteTestRow(x, linkBase)).join('\n')}

package/dist/suite.js

@@ -13,6 +13,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.sortTestFiles = sortTestFiles;
 exports.listTestFiles = listTestFiles;
 exports.toSuiteResult = toSuiteResult;
+exports.toSuiteAttempt = toSuiteAttempt;
+exports.mergeSuiteAttempts = mergeSuiteAttempts;
 exports.cmdSuite = cmdSuite;
 const node_fs_1 = require("node:fs");
 const node_path_1 = require("node:path");
@@ -100,6 +102,64 @@ function toSuiteResult(file, r, durationMs) {
         ...(r.ok ? {} : { failure: failure ?? 'failed' }),
     };
 }
+/** Compact one attempt for the `attempts` evidence array (pure). */
+function toSuiteAttempt(r) {
+    return {
+        id: r.id,
+        ok: r.ok,
+        durationMs: r.durationMs,
+        costUsd: r.costUsd,
+        ...(r.failure ? { failure: r.failure } : {}),
+    };
+}
+/**
+ * Merge a sequence of attempt rows into the final suite row: primary `id` is the last
+ * attempt (winning green, or last red), cost/duration/repairs sum across attempts, and
+ * prior attempts are retained as flake evidence.
+ */
+function mergeSuiteAttempts(attempts) {
+    if (attempts.length === 0)
+        throw new Error('mergeSuiteAttempts: empty');
+    const last = attempts[attempts.length - 1];
+    if (attempts.length === 1)
+        return last;
+    const round = (n) => Number(n.toFixed(4));
+    const prior = attempts.slice(0, -1).map(toSuiteAttempt);
+    const flaky = last.ok && prior.some((a) => !a.ok);
+    return {
+        ...last,
+        durationMs: attempts.reduce((a, t) => a + t.durationMs, 0),
+        costUsd: round(attempts.reduce((a, t) => a + t.costUsd, 0)),
+        modelRepairs: attempts.reduce((a, t) => a + t.modelRepairs, 0),
+        attempts: prior,
+        ...(flaky ? { flaky: true } : {}),
+    };
+}
+// What --retries will and won't spend an attempt on. The bias is deliberate and
+// asymmetric: a retry costs one test, while giving up costs the whole suite plus a
+// human rerunning it. So the rule is *retry unless a rerun provably cannot change the
+// outcome* — the two predicates below are the only "provably" cases, everything else
+// (flaky selector, wedged app, a wobbling network to `vk server`) earns another go.
+/** Budget aborts won't heal on retry: each attempt gets its own cost ceiling, so a
+ *  rerun just re-aborts at the same place having spent the money twice. */
+function isRetryable(r) {
+    return !r.ok && !r.abortedForBudget;
+}
+/** A thrown USAGE error (exit 2) is the one throw a rerun cannot change — an unreadable
+ *  test file, a payload the server refuses, a flag it doesn't understand. Everything
+ *  else, including every environment error, is retried while attempts remain. */
+function isRetryableThrow(e) {
+    return !(e instanceof errors_1.CliError && e.exitCode === 2);
+}
+function parseRetries(flags) {
+    const n = (0, args_1.flagNum)(flags, 'retries');
+    if (n === undefined)
+        return 0;
+    if (!Number.isInteger(n) || n < 0) {
+        throw new errors_1.CliError(`--retries must be a non-negative integer, got '${n}'`, 2);
+    }
+    return n;
+}
 async function cmdSuite(dirArg, flags, deps) {
     const dir = (0, node_path_1.resolve)(process.cwd(), dirArg);
     if (!(0, node_fs_1.existsSync)(dir) || !(0, node_fs_1.statSync)(dir).isDirectory()) {
@@ -109,69 +169,133 @@ async function cmdSuite(dirArg, flags, deps) {
     if (files.length === 0) {
         throw new errors_1.CliError(`suite: no test files (*.md) in '${dirArg}'`, 2);
     }
+    const retries = parseRetries(flags);
     const suiteId = (0, run_1.runId)();
     const name = (0, args_1.flagStr)(flags, 'name') || (0, node_path_1.basename)(dir);
     const startedAt = new Date().toISOString();
-    (0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})`);
+    (0, output_1.err)(`[suite] '${name}': ${files.length} test(s) from ${dirArg} (${deps.platform}${deps.device ? ` · ${deps.device}` : ''})${retries > 0 ? ` · up to ${retries} retry(ies) on failure` : ''}`);
     const results = [];
+    const warnings = [];
     let aborted;
+    async function resetApp(label) {
+        // Returns the abort reason when the suite should stop (confirmed env break during reset).
+        if (!deps.reset)
+            return undefined;
+        try {
+            await deps.reset();
+            (0, output_1.err)(`[suite] app state reset${label}`);
+            return undefined;
+        }
+        catch (e) {
+            // A reset that failed because the BOX is broken means nothing after it is
+            // trustworthy — but only if a re-probe agrees. Otherwise surface and continue:
+            // a flaky reset should not zero out the whole suite, and the test itself will
+            // fail loudly if the stale state actually matters.
+            const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
+            if (broken)
+                return broken;
+            (0, output_1.err)(`[suite] reset failed (${e.message}) — continuing`);
+            return undefined;
+        }
+    }
+    /** A confirmed env break with attempts left: say so, pause, and let the loop retry.
+     *  The pause matters — the failures this rides out (a server restart, a wifi drop, a
+     *  USB re-enumeration) clear in seconds, and retrying into the same dead socket
+     *  immediately would burn every attempt inside the outage. */
+    async function noteEnvRetry(file, attempt, reason) {
+        const warn = `${file}: environment error on attempt ${attempt + 1} (${reason}) — retried`;
+        warnings.push(warn);
+        (0, output_1.err)(`[suite] WARN ${warn}`);
+        await sleep((deps.probeRetryMs ?? PROBE_RETRY_MS) * (attempt + 1));
+    }
     for (let i = 0; i < files.length && !aborted; i++) {
         const file = files[i];
         (0, output_1.err)(`[suite] ── (${i + 1}/${files.length}) ${file} ──`);
-        if (deps.reset) {
+        const attemptRows = [];
+        for (let attempt = 0; attempt <= retries; attempt++) {
+            // The last attempt is where a retryable failure becomes the verdict: a confirmed
+            // env break aborts the suite, anything else stands as this test's failed row.
+            const lastAttempt = attempt === retries;
+            if (attempt > 0)
+                (0, output_1.err)(`[suite] retry ${attempt}/${retries} for ${file}`);
+            // Re-isolate before EVERY attempt — between tests and between retries alike.
+            const resetBreak = await resetApp(attempt > 0 ? ' (retry)' : '');
+            if (resetBreak) {
+                if (!lastAttempt) {
+                    await noteEnvRetry(file, attempt, `reset failed: ${resetBreak}`);
+                    continue;
+                }
+                // With no attempt row this test never ran, so notRun starts at the CURRENT file.
+                aborted = {
+                    reason: `reset failed: ${resetBreak}`,
+                    notRun: files.slice(attemptRows.length ? i + 1 : i),
+                };
+                break;
+            }
+            const t0 = Date.now();
             try {
-                await deps.reset();
-                (0, output_1.err)('[suite] app state reset');
+                const r = await deps.runTest((0, node_path_1.join)(dir, file));
+                attemptRows.push(toSuiteResult(file, r, Date.now() - t0));
+                if (r.abortedForEnv) {
+                    const broken = await stillBroken(deps);
+                    if (broken) {
+                        if (!lastAttempt) {
+                            // Even a CONFIRMED break is worth an attempt: the probe window is a couple
+                            // of seconds, which a server restart outlives — and aborting costs the run.
+                            await noteEnvRetry(file, attempt, broken);
+                            continue;
+                        }
+                        aborted = { reason: broken, notRun: files.slice(i + 1) };
+                        break;
+                    }
+                    // Transient env blip: retryable like any other failure.
+                }
+                if (r.ok || !isRetryable(r) || lastAttempt)
+                    break;
             }
             catch (e) {
-                // A reset that failed because the BOX is broken means nothing after it is
-                // trustworthy — but only if a re-probe agrees. Otherwise surface and continue:
-                // a flaky reset should not zero out the whole suite, and the test itself will
-                // fail loudly if the stale state actually matters.
+                // A test that THREW (device gone, server unreachable, bad file) still becomes a
+                // failed row — one broken test must not vaporize the suite report for the tests
+                // that already ran. Out of attempts, a confirmed env break stops the suite.
+                const msg = e instanceof Error ? e.message : String(e);
+                (0, output_1.err)(`[suite] ${file} errored: ${msg}`);
+                attemptRows.push({
+                    id: '',
+                    file,
+                    name: (0, node_path_1.basename)(file, (0, node_path_1.extname)(file)),
+                    ok: false,
+                    durationMs: Date.now() - t0,
+                    costUsd: 0,
+                    steps: 0,
+                    passedSteps: 0,
+                    failedSteps: 0,
+                    modelRepairs: 0,
+                    failure: msg.split('\n')[0],
+                });
                 const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
-                if (broken) {
-                    // This test never ran, so it gets no row — notRun starts at the CURRENT file.
-                    aborted = { reason: `reset failed: ${broken}`, notRun: files.slice(i) };
+                if (lastAttempt) {
+                    if (broken)
+                        aborted = { reason: broken, notRun: files.slice(i + 1) };
                     break;
                 }
-                (0, output_1.err)(`[suite] reset failed (${e.message}) — continuing`);
-            }
-        }
-        const t0 = Date.now();
-        try {
-            const r = await deps.runTest((0, node_path_1.join)(dir, file));
-            results.push(toSuiteResult(file, r, Date.now() - t0));
-            // The test itself reported an environment abort (exit 3 mid-plan). Same rule:
-            // fatal only if the box is still broken. This test HAS a row and a real report,
-            // so notRun starts after it.
-            if (r.abortedForEnv) {
-                const broken = await stillBroken(deps);
+                if (!isRetryableThrow(e))
+                    break;
                 if (broken)
-                    aborted = { reason: broken, notRun: files.slice(i + 1) };
+                    await noteEnvRetry(file, attempt, broken);
             }
         }
-        catch (e) {
-            // A test that THREW (device gone, server unreachable, bad file) still becomes a
-            // failed row — one broken test must not vaporize the suite report for the tests
-            // that already ran. But if it threw because the environment is gone, stop.
-            const msg = e instanceof Error ? e.message : String(e);
-            (0, output_1.err)(`[suite] ${file} errored: ${msg}`);
-            results.push({
-                id: '',
-                file,
-                name: (0, node_path_1.basename)(file, (0, node_path_1.extname)(file)),
-                ok: false,
-                durationMs: Date.now() - t0,
-                costUsd: 0,
-                steps: 0,
-                passedSteps: 0,
-                failedSteps: 0,
-                modelRepairs: 0,
-                failure: msg.split('\n')[0],
-            });
-            const broken = (0, errors_1.isEnvError)(e) ? await stillBroken(deps) : undefined;
-            if (broken)
-                aborted = { reason: broken, notRun: files.slice(i + 1) };
+        if (attemptRows.length === 0) {
+            // Every attempt was blocked by a failing reset, so the test never ran and gets no
+            // row — `aborted.notRun` (set above) already names it. Nothing to merge.
+            break;
+        }
+        const merged = mergeSuiteAttempts(attemptRows);
+        results.push(merged);
+        if (merged.flaky) {
+            const n = merged.attempts?.length ?? 0;
+            const warn = `${file} passed on retry after ${n} failed attempt${n === 1 ? '' : 's'}`;
+            warnings.push(warn);
+            (0, output_1.err)(`[suite] WARN ${warn}`);
         }
     }
     if (aborted) {
@@ -189,6 +313,7 @@ async function cmdSuite(dirArg, flags, deps) {
         totals: (0, report_1.suiteTotals)(results),
         tests: results,
         ...(aborted ? { aborted } : {}),
+        ...(warnings.length ? { warnings } : {}),
     };
     // .verikun/suites/<id>/ sits beside .verikun/runs/<id>/, so index.html reaches a
     // test report at ../../runs/<id>/report.html — the linkBase below.
@@ -198,8 +323,12 @@ async function cmdSuite(dirArg, flags, deps) {
     (0, node_fs_1.writeFileSync)((0, node_path_1.join)(outDir, 'index.html'), (0, report_1.toSuiteHtml)(suite, { linkBase: '../../' }));
     const t = suite.totals;
     (0, output_1.err)(`[suite] ${t.passed}/${t.tests} passed · ${t.steps} steps · $${t.costUsd.toFixed(4)} · ${(t.durationMs / 1000).toFixed(1)}s`);
-    for (const r of results)
-        (0, output_1.err)(`  ${r.ok ? 'PASS' : 'FAIL'} ${r.file}${r.failure ? ` — ${r.failure}` : ''}`);
+    for (const r of results) {
+        const tag = r.flaky ? 'FLAKY' : r.ok ? 'PASS' : 'FAIL';
+        (0, output_1.err)(`  ${tag} ${r.file}${r.failure ? ` — ${r.failure}` : r.flaky ? ' — passed on retry' : ''}`);
+    }
+    if (warnings.length)
+        (0, output_1.err)(`[suite] ${warnings.length} warning(s)`);
     (0, output_1.err)(`[suite] overview: ${(0, node_path_1.join)(outDir, 'index.html')}`);
     if ((0, args_1.flagBool)(flags, 'json'))
         (0, output_1.json)(suite);
@@ -207,6 +336,7 @@ async function cmdSuite(dirArg, flags, deps) {
         (0, output_1.out)(outDir); // primary machine result: the suite directory
     // The CI gate: any failed test fails the invocation (mirrors `vk run archive`). An
     // environment abort exits 3 instead, so CI can tell "the runner is broken" from "the
-    // app regressed" — the whole point of stopping early.
+    // app regressed" — the whole point of stopping early. A flake that recovered is ok
+    // (exit 0) with a warning — that is the whole point of --retries.
     return aborted ? 3 : t.failed > 0 ? 1 : 0;
 }

package/dist/version.js

@@ -3,4 +3,4 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.VERSION = void 0;
 // GENERATED by scripts/gen-version.mjs from package.json's "version" at build time
 // (the `prebuild` script). Do NOT edit by hand; bump package.json instead.
-exports.VERSION = '0.16.0';
+exports.VERSION = '0.17.0';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "verikun",
-  "version": "0.16.0",
+  "version": "0.17.0",
   "description": "Drive Android emulators/devices and iOS simulators for AI agents: tap, type, swipe, screenshot, and inspect the UI hierarchy by semantic identifiers — like Puppeteer for native apps.",
   "keywords": [
     "android",