@aarwitz/tapp 0.16.4 → 0.16.5

package/AGENTS.md

@@ -82,10 +82,13 @@ Rules that prevent 90% of failures:
 
 ## Autonomous QA (`tapp_run_qa`)
 
-Returns `{verdict, confidence, headline, screensExplored, actionsPerformed, findings[]}`.
+Returns `{verdict, confidence, releaseScore, headline, screensExplored, actionsPerformed, findings[]}`.
+Exploratory web runs set `confidence` and `releaseScore` to `null`; report their deterministic
+finding counts, advisory sampled-probe counts, and coverage instead of inventing a scalar.
 
-- `verdict`: `ready` | `caution` | `blocked`. **Trust it — it's deterministic.** Report it to the
-  user as-is; never soften a `blocked` or inflate a `caution`.
+- `verdict`: `ready` | `caution` | `blocked`. Report it as-is; never soften a `blocked` or inflate a
+  `caution`. Judgment is deterministic for a given evidence trace, while adaptive exploration and
+  live target state can still change which evidence a run observes.
 - `inconclusive: true` means the run couldn't see enough (crash on launch, login wall). That is
   **not a pass** — tell the user what blocked exploration and what would unblock it.
 - Login walls: pass `testEmail`/`testPassword` (auto-typed into login forms), `appLaunchArgs`
@@ -95,9 +98,10 @@ Returns `{verdict, confidence, headline, screensExplored, actionsPerformed, find
 - Diff two runs: pass the previous run's `findings` as `baselineFindings` → you get a
   `regression` block (`new` / `persisting` / `resolved`, plus a CI `gate` signal).
 - On web, report the exact verdict but preserve its scope: Tapp deterministically checks technical
-  behavior such as failed requests, missing assets, placeholder links, and inert controls. It does
+  behavior such as failed requests, missing assets, and placeholder links. Dead-control probes are
+  budget-capped advisory findings and do not drive the verdict. Tapp does
   not validate marketing claims against APIs, API field privacy, brand consistency, or subjective
-  marketplace credibility unless an explicit reviewed test/contract covers them.
+  marketplace credibility unless an explicit reviewed test/contract or verifier covers them.
 
 ## Flows (deterministic E2E tests)
 

package/README.md

@@ -168,7 +168,7 @@ Then ask your agent:
 | 📸 | `tapp_screenshot` | Whatever's on the sim right now, as an inline image. |
 | 🌳 | `tapp_ui_tree` | The accessibility tree of the current screen (ids, labels, hittability). |
 | 🕹 | `tapp_session_start/act/end` | **Interactive driving** — the Playwright loop. App launches once; each act (tap/type/swipe/back/wait) returns the fresh tree. |
-| 🧪 | `tapp_run_qa` | **Autonomous QA** — explores with no authored test, returns `{verdict, releaseScore, findings[]}`. Takes `appBundleId` (iOS), `androidAppId` (Android), or `url` (web). |
+| 🧪 | `tapp_run_qa` | **Autonomous QA** — explores with no authored test, returns `{verdict, releaseScore, findings[]}` (`releaseScore` is `null` for exploratory web). Takes `appBundleId` (iOS), `androidAppId` (Android), or `url` (web). |
 | 🧭 | `tapp_init` | **Repository import** — detect targets; optionally explore a real surface; persist the shared UI Map; construct the evidence-classified model and grounded release plan. |
 | 👤 | `tapp_actor_config` | **Actor/session setup** — store roles, isolation/provisioning, and environment-variable names without accepting or persisting credential values. |
 | ✅ | `tapp_release_plan` | **Release-plan lifecycle** — inspect, approve/reject/defer, generate, real-target validate, and explicitly promote proposed guarantees without silent test edits. |
@@ -193,7 +193,7 @@ deterministic per-platform navigation root used for bounded changed-surface repl
 
 **Adaptive exploration, deterministic judgment.** Exploration is adaptive — two runs may
 traverse different paths through your app. Judgment is deterministic: the same evidence
-trace always produces the same findings, the same score, and the same verdict — no LLM variability
+trace always produces the same findings and verdict — no LLM variability
 in the decision loop. PR gating keys on the **regression diff**
 (stable finding signatures vs. a baseline), so it reacts to what *changed*, not to
 run-to-run path variance. For critical user journeys, committed **Tasks and Flows** provide the stable CI
@@ -201,10 +201,11 @@ suite: reusable semantic actions, exact assertions, condition-based waits, fresh
 and evidence on failure. We call this *flake-resistant*, not magically flake-free—backend outages,
 unstable test data, and poorly identified controls can still make any E2E test fail.
 
-**A release score, not "confidence."** The 0–100 number is a heuristic quality score from
-fixed, documented deductions — we don't call it confidence because it isn't calibrated
-probability. Calibrating it against seeded-fault benchmarks is ongoing work; until then it
-ranks runs, it doesn't promise odds.
+**Native has a heuristic release score; exploratory web does not.** The native 0–100 number comes
+from fixed deductions and is not calibrated probability. Web reports deterministic findings,
+advisory budget-capped control probes, and concrete coverage instead of compressing those unlike
+signals into a scalar. Committed Flows, Tasks, contracts, and baseline regressions provide the web
+merge decision.
 
 `tapp_run_qa` explores like a user — accessibility surfaces on iOS/Android and a real browser on web —
 and detects crashes, failed sign-ins, dead buttons, stuck loading screens, error surfaces,
@@ -219,10 +220,18 @@ broken links and assets, and visible placeholder links with no destination). The
   app crashed on launch or a login wall blocked exploration, you get `inconclusive: true`,
   not a false pass. Absence of findings is not a pass.
 
-Web beta presents a `ready` result as **AUTOMATED CHECKS PASSED**, not “ship-ready.” Its report
+Web beta presents a `ready` result as **AUTOMATED CHECKS COMPLETE**, not “ship-ready,” and displays
+no scalar score. Exhaustive checks on each exercised page drive the verdict; sampled control probes
+remain visible findings but are advisory. The report
 explicitly excludes content/claim accuracy, privacy and API data minimization, brand/SEO
 consistency, and subjective visual credibility. Those require reviewed contracts, privacy review,
-or human/vision judgment; a green technical crawl must not imply they were validated.
+or human/vision judgment; an exploratory crawl must not imply they were validated.
+
+For a business guarantee such as “every coach is insured,” use a deterministic app-owned verifier
+endpoint that returns success only when the invariant holds, then require that status and the
+customer-visible claim in a release contract. The current DSL does not yet read arbitrary JSON
+response bodies or compare a cross-origin API payload directly with page copy; use a verifier or an
+explicit CI preflight rather than assuming autonomous QA inferred the guarantee.
 
 Apps behind a login? Pass `testEmail`/`testPassword` (typed into the login form automatically),
 `appLaunchArgs` (e.g. `["--uitesting"]` if your app supports a bypass), or explicit `loginSteps`

package/mcp-server/src/ci-report.js

@@ -26,7 +26,7 @@
 //   any      fail on any finding at all, or any flow failure. Strictest.
 import fs from "fs";
 import path from "node:path";
-import { buildQaReport, computeRegression, computeContentCollapse, computeReachabilityLoss, verdictBadge } from "./report.js";
+import { buildQaReport, computeRegression, computeContentCollapse, computeReachabilityLoss, qaScoreLabel, verdictBadge } from "./report.js";
 import { writeHtmlReport } from "./html-report.js";
 import { buildUiMapFromMarkers, writeUiMap } from "./ui-map.js";
 import { proposeSelectorMaintenance, validateWebMaintenanceProposal } from "./maintenance-proposal.js";
@@ -345,7 +345,10 @@ function renderMarkdown(report, regression, flows, scenarios, contracts, prPlan,
   lines.push("");
   lines.push(report.headline);
   lines.push("");
-  lines.push(`**release score ${report.confidence}/100** · ${report.screensExplored} screens · ${report.actionsPerformed} actions · ${report.findingCounts.total} finding(s)`);
+  lines.push(`**${qaScoreLabel(report)}** · ${report.screensExplored} screens · ${report.actionsPerformed} actions · ${report.findingCounts.total} finding(s)`);
+  if (report.platform === "web") {
+    lines.push(`**Verdict basis:** ${report.verdictFindingCounts?.total || 0} deterministic finding(s); ${report.sampledFindingCounts?.total || 0} sampled probe finding(s) are advisory.`);
+  }
   if (report.uiMap) lines.push(`**UI Map:** ${report.uiMap.nodeCount} states · ${report.uiMap.edgeCount} transitions · ${report.uiMap.controlCount} semantic controls`);
   if (report.findings.length) {
     lines.push("");
@@ -488,11 +491,17 @@ if (collapsed.length) {
   report.findings.push(...collapsed);
   report.findingCounts.high += collapsed.length;
   report.findingCounts.total += collapsed.length;
-  // Keep the displayed verdict consistent with the merged findings (same scoring as report.js:
-  // high costs 10 confidence; any high caps the verdict at caution).
-  report.confidence = Math.max(0, report.confidence - collapsed.length * 10);
-  report.releaseScore = report.confidence;
-  if (report.verdict === "ready") report.verdict = report.confidence < 50 ? "blocked" : "caution";
+  report.verdictFindingCounts.high += collapsed.length;
+  report.verdictFindingCounts.total += collapsed.length;
+  // Keep native scoring compatible. Exploratory web deliberately has no scalar; deterministic
+  // baseline regressions still raise its verdict directly.
+  if (Number.isFinite(report.confidence)) {
+    report.confidence = Math.max(0, report.confidence - collapsed.length * 10);
+    report.releaseScore = report.confidence;
+  }
+  if (report.verdict === "ready") {
+    report.verdict = Number.isFinite(report.confidence) && report.confidence < 50 ? "blocked" : "caution";
+  }
   report.headline = `Proceed with caution — ${collapsed.length} screen(s) regressed vs. baseline (content collapsed or became unreachable).`;
 }
 const regression = computeRegression(report.findings, baseline?.findings ?? null);

package/mcp-server/src/html-report.js

@@ -8,7 +8,7 @@
 
 import fs from "fs";
 import path from "path";
-import { buildQaReport, verdictBadge } from "./report.js";
+import { buildQaReport, qaScoreLabel, verdictBadge } from "./report.js";
 
 const SEV_COLOR = { critical: "#cf222e", high: "#bc4c00", medium: "#9a6700", low: "#57606a" };
 
@@ -101,8 +101,9 @@ export function writeHtmlReport(captureDir, { report, label = "" } = {}) {
 </style>
 </head>
 <body>
-<h1>${esc(verdictBadge(r))} <span class="dim">· release score ${r.releaseScore ?? r.confidence}/100</span></h1>
+<h1>${esc(verdictBadge(r))} <span class="dim">· ${esc(qaScoreLabel(r))}</span></h1>
 <div class="meta">${esc(label)} · ${r.screensExplored} screens · ${r.actionsPerformed} actions · ${r.findingCounts.total} finding(s)</div>
+${r.platform === "web" ? `<div class="meta">Verdict basis: ${r.verdictFindingCounts?.total || 0} deterministic finding(s); ${r.sampledFindingCounts?.total || 0} sampled probe finding(s) are advisory.</div>` : ""}
 <div class="headline">${esc(r.headline)}</div>
 <h2>Findings</h2>
 <ul class="findings">

package/mcp-server/src/index.js

@@ -12,7 +12,7 @@ import {
   ListToolsRequestSchema,
 } from "@modelcontextprotocol/sdk/types.js";
 
-import { parseOcqaMarkers, buildQaReport, computeRegression, verdictBadge } from "./report.js";
+import { parseOcqaMarkers, buildQaReport, computeRegression, qaScoreLabel, verdictBadge } from "./report.js";
 import { existingProjectArtifactPath, projectArtifactDirectory } from "./project-paths.js";
 
 const __filename = fileURLToPath(import.meta.url);
@@ -1326,11 +1326,14 @@ function formatQaReport(report, { regression, inputHint, timedOut, bundleId, aiC
     .filter(Boolean)
     .join(", ");
   const L = [];
-  L.push(`### 🧪 QA complete — ${badge} · release score ${report.confidence}/100${bundleId ? `\n\`${bundleId}\`` : ""}`);
+  L.push(`### 🧪 QA complete — ${badge} · ${qaScoreLabel(report)}${bundleId ? `\n\`${bundleId}\`` : ""}`);
   L.push("");
   L.push(report.headline);
   L.push("");
   L.push(`**Coverage** — ${report.screensExplored} screens · ${report.actionsPerformed} actions${timedOut ? " · ⏱️ hit time limit" : ""}`);
+  if (report.platform === "web") {
+    L.push(`**Verdict basis** — ${report.verdictFindingCounts?.total || 0} deterministic finding(s); ${report.sampledFindingCounts?.total || 0} sampled probe finding(s) are advisory`);
+  }
   if (uiMap) L.push(`**UI Map** — ${uiMap.nodeCount} states · ${uiMap.edgeCount} transitions · ${uiMap.controlCount} semantic controls · ${uiMap.path}`);
   if (reportHtml) L.push(`**Evidence** — 📄 ${reportHtml} (screenshots of every screen + findings, shareable)`);
   if (recording) L.push(`**Recording** — 🎬 ${recording} (full exploration, embedded in the evidence page)`);
@@ -2070,13 +2073,15 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
       description:
         "Run autonomous QA against iOS (appBundleId), Android (androidAppId), OR a web app " +
         "(url — beta, requires Playwright installed) and return a structured " +
-        "ship/no-ship verdict. Use ONLY when the user wants a QA assessment / to find bugs / a verdict — this " +
+        "QA verdict. Use ONLY when the user wants a QA assessment / to find bugs / a verdict — this " +
         "runs for MINUTES exploring the whole app. Do NOT use it just to view, screenshot, or reach a specific " +
         "screen — use tapp_open_app (launch + screenshot) or a session for that. Tapp explores the app " +
         "like a tester (taps, types, navigates, scrolls) and detects real issues — crashes, dead buttons, failed sign-ins, error screens, " +
         "stuck/hung screens; on web also uncaught JS exceptions, failed/5xx requests, broken links and assets. " +
-        "Returns {verdict: ready|caution|blocked, confidence, headline, screensExplored, " +
-        "actionsPerformed, findings:[{type,severity,category,title,screen}]}. The verdict has a coverage floor: " +
+        "Returns {verdict: ready|caution|blocked, confidence, releaseScore, headline, screensExplored, " +
+        "actionsPerformed, findings:[{type,severity,category,title,screen,evaluationTier}]}. Exploratory web " +
+        "sets confidence/releaseScore to null and separates deterministic verdict findings from advisory " +
+        "sampled control probes. The verdict has a coverage floor: " +
         "if the app barely explored (crash on launch / sign-in wall) it returns 'caution' + inconclusive, never a " +
         "false pass. For iOS the app must already be installed on a booted simulator (use tapp_list_simulators / " +
         "tapp_boot_simulator first). For web, only point it at an app/environment you own — it CLICKS things. " +

package/mcp-server/src/report.js

@@ -96,16 +96,28 @@ export const ISSUE_CATEGORY = {
   explore_timeout: "performance_timeout",
 };
 export const CRITICAL_ISSUE_TYPES = new Set(["crash"]);
+export const WEB_SAMPLED_ISSUE_TYPES = new Set(["unresponsive_element"]);
 
 export function severityRank(s) {
   return { critical: 0, high: 1, medium: 2, low: 3 }[s] ?? 4;
 }
 
+export function findingEvaluationTier(finding, platform = "ios") {
+  return platform === "web" && WEB_SAMPLED_ISSUE_TYPES.has(finding?.type) ? "sampled" : "deterministic";
+}
+
 export function verdictBadge(report) {
-  if (report?.platform === "web" && report?.verdict === "ready") return "🟢 AUTOMATED CHECKS PASSED";
+  if (report?.platform === "web" && report?.verdict === "ready") return "🔵 AUTOMATED CHECKS COMPLETE";
   return { ready: "🟢 SHIP-READY", caution: "🟡 CAUTION", blocked: "🔴 BLOCKED" }[report?.verdict] || report?.verdict;
 }
 
+export function qaScoreLabel(report) {
+  const score = report?.releaseScore ?? report?.confidence;
+  if (Number.isFinite(score)) return `release score ${score}/100`;
+  if (report?.platform === "web") return "exploratory web · no scalar score";
+  return "score unavailable";
+}
+
 // Turn a capture's OCQA markers into the same ship/no-ship report Tapp produces:
 // deduped findings + a trustworthy verdict with a coverage floor (mirrors OrchestratorService).
 export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
@@ -186,7 +198,11 @@ export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
     const key = `${i.type}|${i.screen}|${i.target ?? ""}`;
     if (seen.has(key)) continue;
     seen.add(key);
-    findings.push({ ...i, category: ISSUE_CATEGORY[i.type] || i.type });
+    findings.push({
+      ...i,
+      category: ISSUE_CATEGORY[i.type] || i.type,
+      ...(platform === "web" ? { evaluationTier: findingEvaluationTier(i, platform) } : {}),
+    });
   }
   findings.sort((a, b) => severityRank(a.severity) - severityRank(b.severity));
 
@@ -197,24 +213,32 @@ export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
   const high = findings.filter((f) => f.severity === "high").length;
   const med = findings.filter((f) => f.severity === "medium").length;
   const low = findings.filter((f) => f.severity === "low").length;
+  const verdictFindings = platform === "web"
+    ? findings.filter((finding) => finding.evaluationTier !== "sampled")
+    : findings;
+  const verdictCrit = verdictFindings.filter((f) => f.severity === "critical").length;
+  const verdictHigh = verdictFindings.filter((f) => f.severity === "high").length;
+  const verdictMed = verdictFindings.filter((f) => f.severity === "medium").length;
+  const verdictLow = verdictFindings.filter((f) => f.severity === "low").length;
+  const sampledFindings = platform === "web" ? findings.filter((finding) => finding.evaluationTier === "sampled") : [];
 
   // Coverage floor: a verdict is only trustworthy if the app was actually exercised.
   const inconclusive = screensExplored < 2 || actionsPerformed < 3;
-  let confidence = Math.max(0, Math.min(100, 100 - crit * 25 - high * 10 - med * 3));
-  if (inconclusive) confidence = Math.min(confidence, 40);
+  let riskScore = Math.max(0, Math.min(100, 100 - verdictCrit * 25 - verdictHigh * 10 - verdictMed * 3));
+  if (inconclusive) riskScore = Math.min(riskScore, 40);
 
   let verdict;
-  if (crit > 0) verdict = "blocked";
+  if (verdictCrit > 0) verdict = "blocked";
   else if (inconclusive) verdict = "caution";
-  else if (confidence < 50) verdict = "blocked";
-  else if (high > 0 || confidence < 80) verdict = "caution";
+  else if (riskScore < 50) verdict = "blocked";
+  else if (verdictHigh > 0 || riskScore < 80) verdict = "caution";
   else verdict = "ready";
 
   const headline = inconclusive
     ? `Inconclusive — only ${screensExplored} screen(s) / ${actionsPerformed} action(s) explored. The app may have crashed on launch, be stuck behind a sign-in wall, or otherwise prevent exploration. Absence of issues is NOT a pass.`
     : verdict === "ready"
     ? platform === "web"
-      ? "Automated web checks passed — no release-blocking technical issues found in the exercised surfaces. This is not a content, privacy, brand, or business-claim review."
+      ? "Automated web checks completed — no release-blocking deterministic findings in the exercised surfaces. Sampled control probes are advisory. This is not a content, privacy, brand, or business-claim review."
       : "Ship-ready — no release-blocking issues found."
     : verdict === "caution"
     ? `Proceed with caution — ${findings.length} issue(s) to review.`
@@ -230,7 +254,7 @@ export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
   if (platform === "web") {
     checkedFor = [
       "page errors (uncaught exceptions)", "failed/5xx requests", "broken links (404)",
-      "placeholder links with no destination", "dead buttons", "error text on pages", "load timeouts",
+      "placeholder links with no destination", "sampled dead-button probes (advisory)", "error text on pages", "load timeouts",
     ];
     notChecked = [
       "app-specific business logic (cover with Flows: record or generate, then assert)",
@@ -272,11 +296,14 @@ export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
 
   return {
     verdict,
-    // `releaseScore` is the honest name: a heuristic quality score from fixed deductions,
-    // NOT calibrated statistical confidence. `confidence` is kept as an alias for
-    // compatibility (baselines, desktop app, existing consumers).
-    confidence,
-    releaseScore: confidence,
+    // Exploratory web QA deliberately has no scalar. Its verdict derives from deterministic
+    // checks on exercised pages; budget-capped control probes remain visible but advisory.
+    // Native keeps the legacy heuristic score until it has an equivalent tier split.
+    confidence: platform === "web" ? null : riskScore,
+    releaseScore: platform === "web" ? null : riskScore,
+    scoreUnavailableReason: platform === "web"
+      ? "Exploratory web runs report deterministic findings, advisory sampled probes, and coverage instead of a scalar release score."
+      : null,
     headline,
     inconclusive,
     checkedFor,
@@ -286,6 +313,20 @@ export function buildQaReport(markersFilePath, { platform = "ios" } = {}) {
     screensExplored,
     actionsPerformed,
     findingCounts: { critical: crit, high, medium: med, low, total: findings.length },
+    verdictFindingCounts: {
+      critical: verdictCrit,
+      high: verdictHigh,
+      medium: verdictMed,
+      low: verdictLow,
+      total: verdictFindings.length,
+    },
+    sampledFindingCounts: {
+      critical: sampledFindings.filter((f) => f.severity === "critical").length,
+      high: sampledFindings.filter((f) => f.severity === "high").length,
+      medium: sampledFindings.filter((f) => f.severity === "medium").length,
+      low: sampledFindings.filter((f) => f.severity === "low").length,
+      total: sampledFindings.length,
+    },
     findings,
     screens: Array.from(screens),
     screenElementCounts,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aarwitz/tapp",
-  "version": "0.16.4",
+  "version": "0.16.5",
   "mcpName": "io.github.aarwitz/tapp",
   "description": "Release contracts, autonomous QA, and evidence-backed CI gates for iOS, Android, and web.",
   "license": "MIT",