@mmerterden/multi-agent-pipeline 20.27.2 → 20.28.0

package/pipeline/skills/shared/external/test-result-triage/SKILL.md

@@ -0,0 +1,160 @@
+---
+name: test-result-triage
+description: "Judge each test result on changed code as product-bug, stale-test, valid or ambiguous (the default), and list behaviour no test covers. stale-test needs a diff file:line, product-bug needs evidence. Use when tests ran on a new commit, PR or fix. Not for writing or editing tests."
+metadata:
+  source: multi-agent-pipeline
+  version: 1.0.0
+---
+
+# Test result triage
+
+The product moved to a new commit and the tests ran against it. For every test
+that ran, decide what happens next. The decision is a plan for a human (or a
+setting) to approve; this skill does not edit tests or product code.
+
+Requires `@mmerterden/multi-agent-toolkit-mcp` >= v3.25.0 for
+`test_results_classify`. On an older server, or with no server, derive the
+input statuses by hand as section 2 describes and say in the report that the
+run used the fallback.
+
+## 1. The verdicts
+
+Each test that ran gets one of four verdicts:
+
+| Verdict | Meaning | What follows |
+|---|---|---|
+| `product-bug` | The test is right: behaviour broke without intent. | A finding is filed. The test is not changed. |
+| `stale-test` | Behaviour changed on purpose, or a handle moved; the test expects the old state. | The test is revised. |
+| `valid` | The test still measures the right thing on the new code. | Nothing. |
+| `ambiguous` | The evidence does not decide it. | A human decides. |
+
+The fifth outcome, `new-scenario`, belongs to the change rather than to a test:
+behaviour the change adds that no test exercises. It goes in `newScenarios`
+(section 5) and on to `scenario-gap-analysis`, never as a verdict on a test
+that ran.
+
+**The default is `ambiguous`.** Marking a product bug as `stale-test` loosens
+the test and buries the bug, which is the most expensive mistake this step can
+make. Calling a red test `product-bug` without evidence sends a team chasing a
+bug that is not there. Every verdict other than `ambiguous` needs positive
+evidence from section 4; a red test that the change merely fails to explain
+stays `ambiguous`.
+
+## 2. Inputs
+
+- **Classified results.** When the multi-agent-toolkit MCP server provides
+  `test_results_classify`, use its output as the input. It compares the head
+  run with a base run (the unchanged commit) and a confirmation run, and gives
+  each test a status with a reason:
+
+  | Status | How triage treats it |
+  |---|---|
+  | `passed` | Rule 4. Without a change analysis, read the diff for the area the test covers before calling it `valid`. |
+  | `regression` | Green on base, red twice on head. Rules 1 to 3, in order. |
+  | `preexisting` | Red on base too, so this change did not cause it. Never `stale-test` on this diff's evidence. `ambiguous` with the reason "red before the change", listed in its own section; `product-bug` (preexisting) only with the product `file:line` that causes it. |
+  | `unstable` | Flipped on confirmation or has a recent flake history. `ambiguous`; never green, never a product bug. |
+  | `not_measured` | Base, setup or environment did not measure it. `ambiguous` with the tool's reason. |
+
+  Without the tool, derive the same distinctions by hand: a setup failure, a
+  missing device or a server that did not start is not measured, not red.
+- **The change**: the diff from base to head, commit and PR titles, release
+  notes.
+- **The change analysis**, if one ran: which areas it expected to change and
+  how (behaviour change, new feature, refactor with no behaviour change).
+- **Run evidence**: failure message, failing step, trace, screenshot or DOM.
+
+## 3. Decision rules
+
+Apply them in this order; the first that matches decides.
+
+1. **Red because an element or locator was not found** -> check the handle
+   before anything else. Search the new tree for the element (`git grep` the
+   identifier, the text, the component).
+   - The element exists but carries no handle: `ambiguous` with the reason
+     "handle missing", and a request for `handle-request`. Not a stale test.
+   - The diff renames or moves the handle on purpose: `stale-test`, with the
+     diff `file:line`.
+   - The element is gone and the removal is intentional (diff `file:line`
+     plus a title or note saying so): `stale-test`.
+   - The element is gone and nothing says it was intended: `ambiguous`,
+     unless rule 3 finds positive evidence of a bug.
+2. **Red, and the diff changes that behaviour on purpose** (a commit title, a
+   new flow, new copy) -> `stale-test`. Cite the diff `file:line` and the
+   intent signal.
+3. **Red, with positive evidence of a bug** -> `product-bug`. Positive
+   evidence is one of:
+   - the test checks a contract (data loss, an error code, an empty body,
+     request and response parity) and the diff does not change that contract
+     on purpose;
+   - a product `file:line` that produces the wrong result;
+   - the change analysis states that behaviour in this area did not change,
+     yet the test that passed on base now fails.
+   Name the behaviour that broke. A red test with none of these stays
+   `ambiguous`.
+4. **Green** -> do not pass it only because it is green. When the change
+   analysis or the diff shows the behaviour this test covers changed, check
+   whether the test asserts the old behaviour or the new one: old, passing by
+   accident, `stale-test`; new, `valid`. When neither touches the area,
+   `valid`.
+5. **Failed for an environmental reason** (timeout, server down, network,
+   device) -> `ambiguous` with the reason. Never a product bug.
+6. **Anything else** -> `ambiguous`, naming the missing evidence.
+
+A test written in this same chain that stays red after revision has often
+caught a real bug. Calling it `product-bug` still needs the product code
+`file:line` that causes it.
+
+Separately from the per-test verdicts, list behaviour the change adds that no
+test exercised as `newScenarios`.
+
+## 4. Evidence per verdict
+
+| Verdict | Required |
+|---|---|
+| `stale-test` | Diff `file:line` that changes the behaviour or the handle, and the intent signal (title, note, spec). |
+| `product-bug` | Expected versus observed from the run, the behaviour that broke, and one piece of positive evidence from rule 3. |
+| `valid` | The green run, plus for rule 4 the reason the test asserts the new behaviour or why the area is untouched. |
+| `ambiguous` | Which piece of evidence is missing. |
+| `newScenarios[]` entry | The new behaviour and the diff `file:line` that adds it. |
+
+Many tests failing on the same handle or the same fixture form a cluster: prove
+one representative, decide the cluster together, and say in the reason that it
+was decided as a cluster.
+
+## 5. Output
+
+Every test that ran gets a verdict; a red test left without one reaches a human
+as "ambiguous: no verdict given", so leaving one out only delays it.
+
+```json
+{
+  "verdicts": [
+    {
+      "testId": "checkout.spec.ts > applies coupon",
+      "verdict": "stale-test",
+      "reason": "coupon field moved into a sheet on purpose",
+      "evidence": "src/checkout/CouponSheet.tsx:18, PR title 'move coupon entry to a sheet'",
+      "inputStatus": "regression"
+    }
+  ],
+  "newScenarios": [{ "behaviour": "...", "diff": "file:line" }]
+}
+```
+
+Write a report next to it: verdict and reason per test, a summary table by
+verdict, the `preexisting` section, and what could not be measured. On a long
+run, work screen by screen and save after each one, so a timeout loses nothing
+already decided.
+
+## Boundaries
+
+- No test edits, no product code edits.
+- No red test becomes `stale-test` while there is doubt.
+- A `product-bug` verdict is a report, never an automatic change to the test.
+
+## Related
+
+- `scenario-gap-analysis` for `newScenarios` entries.
+- `handle-request` for elements that exist but carry no handle.
+- `e2e-testing` for the passed / failed / blocked verdicts of a single run.
+- `fix-review-verdicts` when the red test led to a fix that now needs review.

package/pipeline/skills/skills-index.md

@@ -3,7 +3,7 @@
 > Auto-generated by `pipeline/scripts/build-skills-index.mjs`  -  do not hand-edit.
 > Regenerate with `node pipeline/scripts/build-skills-index.mjs`.
 
-**229 skills** across 2 groups.
+**234 skills** across 2 groups.
 
 | Group | Name | Platform | Description |
 |-------|------|----------|-------------|
@@ -24,7 +24,7 @@
 | external | `app-intents` |  -  | App Intents for Siri, Shortcuts, Spotlight, widgets, Control Center and Apple Intelligence: AppIntent actions, AppEntity and EntityQuery, Ap |
 | external | `app-store-changelog` |  -  | Generate user-facing App Store release notes from git history since the last tag. Use when release notes have to be written from the commits |
 | external | `app-store-optimization` |  -  | App Store Optimization (ASO) for ranking and conversion: keyword research; app name, subtitle and 100-byte keyword field; description and pr |
-| external | `app-store-review` |  -  | App Store submission readiness and rejection prevention: Review Guidelines 2.1, 2.3, 2.5.1, 3.1.1, 4.2 and 5.1.2, PrivacyInfo.xcprivacy mani |
+| external | `app-store-review` |  -  | App Store submission readiness and rejection prevention: Review Guidelines 1.2, 2.1, 2.3, 2.3.14, 2.5.1, 2.5.19, 3.1.1, 4.2 and 5.1.2, uploa |
 | core | `apple-archive-compliance` |  -  | Apple App Store Review compliance  -  wraps the multi-agent-toolkit-mcp `ios_app_store_audit` tool (18-rule deep scan) with ITMS error code  |
 | external | `apple-on-device-ai` |  -  | On-device AI on Apple hardware: Foundation Models (SystemLanguageModel, LanguageModelSession, @Generable and @Guide structured output, tool  |
 | external | `architecture` |  -  | Stack-neutral architecture decisions: quality drivers, options with a trade-off matrix, reversibility, ADRs, fitness checks. Use when making |
@@ -61,9 +61,11 @@
 | external | `evidence-registry` |  -  | Check a package's real version, deprecation status and release notes on npm, PyPI, Maven Central or Swift Package Index. Use when a spec or  |
 | external | `fastapi-pro` |  -  | FastAPI with Pydantic v2: lifespan, settings, Annotated dependencies, async SQLAlchemy 2.0 sessions, error handlers, background work, httpx  |
 | external | `firebase` |  -  | You're a developer who has shipped dozens of Firebase projects. You've seen the easy path lead to security breaches, runaway costs, and impo |
+| external | `fix-review-verdicts` |  -  | Independent fix review with three verdicts, test-gaming, scope and root-cause, each backed by file:line findings; the result is computed fro |
 | external | `github-actions-templates` |  -  | Authoring GitHub Actions workflows: triggers, matrices, caching, concurrency, least-privilege permissions, SHA pinning, OIDC, reusable workf |
 | core | `google-play-compliance` |  -  | Google Play Store publication compliance  -  bundletool + aapt2 + apksigner orchestration + 21-rule policy catalog with Play Console error c |
 | external | `gradle-kotlin-dsl` |  -  | Configure Android builds with Gradle Kotlin DSL, version catalogs (libs.versions.toml), convention plugins (build-logic module), common conf |
+| external | `handle-request` |  -  | Add the test handles a scenario needs and answer each request with ADDED, ALREADY or COULD NOT ADD plus file:line or reason. Identity values |
 | external | `healthkit` |  -  | HealthKit on iOS, iPadOS, watchOS and visionOS: HKHealthStore authorization, sample, statistics and chart-ready collection queries, saving H |
 | external | `hig-app-icons` |  -  | iOS app icons: Icon Composer layered icons, default, dark, clear and tinted appearances, asset catalogs, alternate icons and the archive che |
 | external | `hig-components-content` |  -  | Apple HIG for content views mapped to SwiftUI: lists, tables, grids, charts, images, text, web views, empty states. Use when choosing or rev |
@@ -160,6 +162,7 @@
 | core | `multi-agent-test-accessibility` |  -  | Accessibility audit on a booted simulator / emulator: VoiceOver labels, sub-44pt tap targets, contrast, traits. Alias pinning the accessibil |
 | core | `multi-agent-test-dark-mode` |  -  | Dark mode UI test on a booted simulator / emulator: walk every screen light then dark, report contrast + colour bugs. Alias pinning the dark |
 | core | `multi-agent-test-dynamic-type` |  -  | Dynamic Type layout test on a booted simulator / emulator: re-walk every screen at XL through accessibility-extra-large, report truncation a |
+| core | `multi-agent-test-gen` |  -  | Write the tests an app lacks, stage by stage, each gated on its artefact: inventory, read-only gap analysis, handles, tests, runs, per-test  |
 | core | `multi-agent-test-screenshots` |  -  | App Store screenshot set from a booted simulator / emulator in a given locale, taken as an argument and defaulting to tr. Alias pinning the  |
 | core | `multi-agent-testflight-validation` |  -  | iOS-pinned alias of store-ready: same three gates on an iOS build, one implementation. Validates a TestFlight / App Store package, never upl |
 | core | `multi-agent-uninstall` |  -  | Uninstall the pipeline from Claude Code + Copilot CLI. Keychain access tokens are always left untouched; --all-data also clears pipeline set |
@@ -182,6 +185,7 @@
 | external | `rest-api-design` |  -  | REST API design: HTTP methods, status codes, pagination, versioning, rate limiting, caching headers, OpenAPI documentation. Use when a REST  |
 | external | `retrofit-networking` |  -  | Build networking layers on Android with Retrofit, OkHttp, and Kotlin Coroutines. Covers suspend functions in API interfaces, OkHttp intercep |
 | external | `room-database` |  -  | Implement Room database persistence on Android with @Entity, @Dao, @Database, Flow-based reactive queries, TypeConverters for complex types, |
+| external | `scenario-gap-analysis` |  -  | Set an inventory against existing tests, sort each behaviour into COVERED, WRONGLY COVERED or MISSING, and write gaps as measurable, priorit |
 | external | `search-first` |  -  | Research-before-coding workflow. Before writing a custom implementation, check what already exists (repo utilities, installed libraries, pac |
 | external | `security-review` |  -  | Run a defensive, static security review of a diff or a repo: build a run-scoped threat model, find vulnerabilities joined to OWASP + CWE wit |
 | external | `shareplay-activities` |  -  | GroupActivities and SharePlay on iOS, macOS, tvOS and visionOS: GroupActivity, GroupSession lifecycle, messengers, playback sync, file trans |
@@ -220,6 +224,7 @@
 | external | `swiftui-view-refactor` |  -  | Refactor SwiftUI views into smaller components with stable, explicit data flow. Use when a SwiftUI view has grown too large and needs splitt |
 | external | `swiftui-webkit` |  -  | Web content in SwiftUI with WebKit for SwiftUI: WebView, WebPage, observable page state, navigation policies, link interception, JavaScript  |
 | external | `tailwind-css` |  -  | Tailwind CSS v4: utility-first styling, responsive design, dark mode, custom themes, component patterns, and cn() utility. Use when styling  |
+| external | `test-result-triage` |  -  | Judge each test result on changed code as product-bug, stale-test, valid or ambiguous (the default), and list behaviour no test covers. stal |
 | external | `testing-backend` |  -  | Backend testing: pytest patterns (Python), Jest patterns (Node.js), integration testing, database fixtures/factories, API contract testing.  |
 | external | `tipkit` |  -  | TipKit in-app tips (iOS 17+): Tip protocol, TipView, popoverTip, rules, events, TipGroup, styles, CloudKit sync. Use when building or review |
 | external | `typescript-patterns` |  -  | Modern TypeScript patterns: strict mode, utility types, generics, conditional types, type guards, discriminated unions, Zod validation. Use  |

package/install/_telemetry.mjs

@@ -1,61 +0,0 @@
-/**
- * Anonymous install telemetry (opt-in only).
- *
- * Set `MULTI_AGENT_TELEMETRY=1` to enable. Nothing is sent otherwise.
- * Fire-and-forget; never throws and never delays the install.
- *
- * The payload is genuinely anonymous: package name, version, install
- * method, flags, platform/arch, and Node version. No usernames, emails,
- * hostnames, or other machine/user identifiers are collected.
- *
- * @module install/_telemetry
- */
-
-import { readFileSync } from "fs";
-import { platform, arch } from "os";
-import { join } from "path";
-
-/**
- * @param {{ packageRoot: string, flags: string[] }} ctx
- * @returns {Promise<void>}
- */
-export async function sendInstallTelemetry(ctx) {
-  if (process.env.MULTI_AGENT_TELEMETRY !== "1") return;
-
-  let method = "source";
-  const npmExecPath = process.env.npm_execpath ?? "";
-  if (npmExecPath.includes("npx-cli")) method = "npx";
-  else if (process.env.npm_config_global === "true") method = "global";
-  else if (process.env.npm_lifecycle_event === "install") method = "npm-install";
-
-  let version = "unknown";
-  try {
-    const pkg = JSON.parse(readFileSync(join(ctx.packageRoot, "package.json"), "utf-8"));
-    version = pkg.version ?? version;
-  } catch {
-    /* ignore */
-  }
-
-  const payload = {
-    package: "@mmerterden/multi-agent-pipeline",
-    version,
-    method,
-    flags: ctx.flags.filter((f) => f.startsWith("--")),
-    platform: `${platform()} ${arch()}`,
-    nodeVersion: process.version,
-  };
-
-  try {
-    const ctrl = new AbortController();
-    const timer = setTimeout(() => ctrl.abort(), 2000);
-    await fetch("https://mmerterden.vercel.app/api/install-notify", {
-      method: "POST",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify(payload),
-      signal: ctrl.signal,
-    });
-    clearTimeout(timer);
-  } catch {
-    // Silent fail - never break the install if telemetry is unreachable
-  }
-}