@mmerterden/multi-agent-pipeline 20.27.1 → 20.28.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +38 -0
- package/README.md +6 -5
- package/README.tr.md +6 -5
- package/docs/adr/0008-installer-modularization-and-secret-leak-defense.md +0 -1
- package/docs/architecture.md +4 -4
- package/docs/ecosystem.md +5 -5
- package/docs/facts.json +14 -12
- package/docs/features.md +2 -0
- package/install/_dev-only-files.mjs +1 -0
- package/install/catalog-history.json +1 -1
- package/install/index.mjs +0 -2
- package/manifest.json +74 -51
- package/package.json +4 -1
- package/pipeline/agents/code-reviewer.md +1 -0
- package/pipeline/agents/test-gen-analyst.md +98 -0
- package/pipeline/commands/multi-agent/sync/SKILL.md +3 -3
- package/pipeline/commands/multi-agent/test-gen/SKILL.md +124 -0
- package/pipeline/multi-agent-refs/cross-cli-contract.md +11 -5
- package/pipeline/multi-agent-refs/features/doctor.md +13 -0
- package/pipeline/multi-agent-refs/features/test-classification.md +76 -0
- package/pipeline/multi-agent-refs/features/usage-reporting.md +17 -0
- package/pipeline/multi-agent-refs/help/en.md +1 -0
- package/pipeline/multi-agent-refs/help/tr.md +1 -0
- package/pipeline/multi-agent-refs/phases/operations.md +2 -2
- package/pipeline/multi-agent-refs/phases/phase-2-dev.md +6 -7
- package/pipeline/multi-agent-refs/phases/phase-3-review.md +3 -3
- package/pipeline/multi-agent-refs/phases/phase-5-report.md +6 -5
- package/pipeline/multi-agent-refs/store-ready/gate-3-policy.md +6 -2
- package/pipeline/multi-agent-refs/test-gen/stages.md +227 -0
- package/pipeline/preferences-template.json +3 -0
- package/pipeline/rules/app-store-guidelines.md +29 -41
- package/pipeline/schemas/prefs.schema.json +14 -0
- package/pipeline/schemas/test-gen-discovery.schema.json +73 -0
- package/pipeline/schemas/test-gen-gaps.schema.json +122 -0
- package/pipeline/schemas/test-gen-inventory.schema.json +59 -0
- package/pipeline/schemas/test-gen-results.schema.json +44 -0
- package/pipeline/scripts/_handle-patterns.mjs +166 -0
- package/pipeline/scripts/_http-post.mjs +141 -0
- package/pipeline/scripts/_stack-routing.mjs +4 -0
- package/pipeline/scripts/_test-gaming.mjs +227 -0
- package/pipeline/scripts/_unified-diff.mjs +104 -0
- package/pipeline/scripts/_worktree-snapshot.mjs +113 -0
- package/pipeline/scripts/doctor.mjs +51 -0
- package/pipeline/scripts/evidence-gate.mjs +74 -3
- package/pipeline/scripts/handle-diff-audit.mjs +188 -0
- package/pipeline/scripts/routine-registry.mjs +5 -1
- package/pipeline/scripts/run-ui-tests.sh +27 -3
- package/pipeline/scripts/test-classify.mjs +296 -0
- package/pipeline/scripts/test-gen-gate.mjs +409 -0
- package/pipeline/scripts/test-gen-state.mjs +313 -0
- package/pipeline/scripts/test-integrity-gate.mjs +76 -11
- package/pipeline/scripts/uninstall.mjs +1 -0
- package/pipeline/scripts/usage-register-beacon.mjs +239 -0
- package/pipeline/scripts/usage-register.mjs +128 -33
- package/pipeline/scripts/usage-report.mjs +43 -33
- package/pipeline/skills/.skill-manifest.json +25 -5
- package/pipeline/skills/.skills-index.json +57 -2
- package/pipeline/skills/shared/README.md +11 -6
- package/pipeline/skills/shared/core/apple-archive-compliance/SKILL.md +5 -4
- package/pipeline/skills/shared/core/apple-archive-compliance/references/rules.yml +8 -8
- package/pipeline/skills/shared/core/multi-agent-sync/SKILL.md +3 -3
- package/pipeline/skills/shared/core/multi-agent-test-gen/SKILL.md +120 -0
- package/pipeline/skills/shared/external/app-store-review/SKILL.md +69 -8
- package/pipeline/skills/shared/external/app-store-review/evals/evals.json +12 -0
- package/pipeline/skills/shared/external/app-store-review/references/review-checklists.md +10 -1
- package/pipeline/skills/shared/external/fix-review-verdicts/SKILL.md +133 -0
- package/pipeline/skills/shared/external/handle-request/SKILL.md +97 -0
- package/pipeline/skills/shared/external/scenario-gap-analysis/SKILL.md +129 -0
- package/pipeline/skills/shared/external/scenario-gap-analysis/references/scenario-schema.md +60 -0
- package/pipeline/skills/shared/external/test-result-triage/SKILL.md +160 -0
- package/pipeline/skills/skills-index.md +7 -2
- package/install/_telemetry.mjs +0 -61
|
@@ -2,24 +2,24 @@
|
|
|
2
2
|
|
|
3
3
|
Single source of truth for skills delivered to both Claude Code (`~/.claude/skills/`) and Copilot CLI (`~/.copilot/skills/`) by the installer.
|
|
4
4
|
|
|
5
|
-
**Total:**
|
|
5
|
+
**Total:** 234 skills (69 core + 165 external). Auto-generated by `scripts/gen-skills-index.mjs` - do not edit by hand.
|
|
6
6
|
|
|
7
7
|
## Directory layout
|
|
8
8
|
|
|
9
|
-
- **`core/`** -
|
|
10
|
-
- **`external/`** -
|
|
9
|
+
- **`core/`** - 69 `multi-agent*` orchestration skills that are pipeline-critical. Edits here are core-code changes.
|
|
10
|
+
- **`external/`** - 165 iOS / Android / generic skills that ship into the stack plugins. Each names its origin in `metadata.source`: `multi-agent-pipeline` when written here, or the upstream and license a `NOTICE-*.md` in this directory covers.
|
|
11
11
|
- Install destinations (ADR-0009): Claude Code gets NO local copy of `external/` - it loads those skills from the `multi-agent-plugins` marketplace, namespaced (`ai-<stack>-toolkit:<name>`); only the two compliance catalogs from `core/` land in `~/.claude/skills/`. Copilot CLI and Codex CLI receive a flat copy filtered to the enabled stacks. `external/` remains the single authoring source that `build-stack-plugins.mjs` publishes from.
|
|
12
12
|
|
|
13
13
|
Source layout is logical grouping only - skill discovery at runtime is unchanged.
|
|
14
14
|
|
|
15
15
|
## Categories
|
|
16
16
|
|
|
17
|
-
- [Pipeline Orchestration](#pipeline-orchestration) -
|
|
17
|
+
- [Pipeline Orchestration](#pipeline-orchestration) - 69
|
|
18
18
|
- [iOS / Apple Ecosystem](#ios-apple-ecosystem) - 94
|
|
19
19
|
- [Android / Kotlin](#android-kotlin) - 16
|
|
20
20
|
- [Web](#web) - 10
|
|
21
21
|
- [Backend / API](#backend-api) - 11
|
|
22
|
-
- [Cross-cutting](#cross-cutting) -
|
|
22
|
+
- [Cross-cutting](#cross-cutting) - 34
|
|
23
23
|
|
|
24
24
|
## Pipeline Orchestration
|
|
25
25
|
|
|
@@ -89,6 +89,7 @@ Source layout is logical grouping only - skill discovery at runtime is unchang
|
|
|
89
89
|
| [`multi-agent-test-accessibility`](./core/multi-agent-test-accessibility/) | `core` | Accessibility audit on a booted simulator / emulator: VoiceOver labels, sub-44pt tap targets, contrast, traits. Alias pinning the accessibil |
|
|
90
90
|
| [`multi-agent-test-dark-mode`](./core/multi-agent-test-dark-mode/) | `core` | Dark mode UI test on a booted simulator / emulator: walk every screen light then dark, report contrast + colour bugs. Alias pinning the dark |
|
|
91
91
|
| [`multi-agent-test-dynamic-type`](./core/multi-agent-test-dynamic-type/) | `core` | Dynamic Type layout test on a booted simulator / emulator: re-walk every screen at XL through accessibility-extra-large, report truncation a |
|
|
92
|
+
| [`multi-agent-test-gen`](./core/multi-agent-test-gen/) | `core` | Write the tests an app lacks, stage by stage, each gated on its artefact: inventory, read-only gap analysis, handles, tests, runs, per-test |
|
|
92
93
|
| [`multi-agent-test-screenshots`](./core/multi-agent-test-screenshots/) | `core` | App Store screenshot set from a booted simulator / emulator in a given locale, taken as an argument and defaulting to tr. Alias pinning the |
|
|
93
94
|
| [`multi-agent-testflight-validation`](./core/multi-agent-testflight-validation/) | `core` | iOS-pinned alias of store-ready: same three gates on an iOS build, one implementation. Validates a TestFlight / App Store package, never upl |
|
|
94
95
|
| [`multi-agent-uninstall`](./core/multi-agent-uninstall/) | `core` | Uninstall the pipeline from Claude Code + Copilot CLI. Keychain access tokens are always left untouched; --all-data also clears pipeline set |
|
|
@@ -104,7 +105,7 @@ Source layout is logical grouping only - skill discovery at runtime is unchang
|
|
|
104
105
|
| [`app-intents`](./external/app-intents/) | `external` | App Intents for Siri, Shortcuts, Spotlight, widgets, Control Center and Apple Intelligence: AppIntent actions, AppEntity and EntityQuery, Ap |
|
|
105
106
|
| [`app-store-changelog`](./external/app-store-changelog/) | `external` | Generate user-facing App Store release notes from git history since the last tag. Use when release notes have to be written from the commits |
|
|
106
107
|
| [`app-store-optimization`](./external/app-store-optimization/) | `external` | App Store Optimization (ASO) for ranking and conversion: keyword research; app name, subtitle and 100-byte keyword field; description and pr |
|
|
107
|
-
| [`app-store-review`](./external/app-store-review/) | `external` | App Store submission readiness and rejection prevention: Review Guidelines 2.1, 2.3, 2.5.1, 3.1.1, 4.2 and 5.1.2,
|
|
108
|
+
| [`app-store-review`](./external/app-store-review/) | `external` | App Store submission readiness and rejection prevention: Review Guidelines 1.2, 2.1, 2.3, 2.3.14, 2.5.1, 2.5.19, 3.1.1, 4.2 and 5.1.2, uploa |
|
|
108
109
|
| [`apple-on-device-ai`](./external/apple-on-device-ai/) | `external` | On-device AI on Apple hardware: Foundation Models (SystemLanguageModel, LanguageModelSession, @Generable and @Guide structured output, tool |
|
|
109
110
|
| [`authentication`](./external/authentication/) | `external` | iOS sign-in with AuthenticationServices and LocalAuthentication: Sign in with Apple, passkey (WebAuthn) sign-up and login via ASAuthorizatio |
|
|
110
111
|
| [`background-processing`](./external/background-processing/) | `external` | iOS background work with BGTaskScheduler: BGAppRefreshTask short fetches, BGProcessingTask long maintenance, BGContinuedProcessingTask (iOS |
|
|
@@ -264,15 +265,19 @@ Source layout is logical grouping only - skill discovery at runtime is unchang
|
|
|
264
265
|
| [`e2e-testing`](./external/e2e-testing/) | `external` | \| |
|
|
265
266
|
| [`evidence-github`](./external/evidence-github/) | `external` | Search GitHub for facts an analysis can cite - a known issue in a dependency, the PR that fixed it, what a release actually changed, prior a |
|
|
266
267
|
| [`evidence-registry`](./external/evidence-registry/) | `external` | Check a package's real version, deprecation status and release notes on npm, PyPI, Maven Central or Swift Package Index. Use when a spec or |
|
|
268
|
+
| [`fix-review-verdicts`](./external/fix-review-verdicts/) | `external` | Independent fix review with three verdicts, test-gaming, scope and root-cause, each backed by file:line findings; the result is computed fro |
|
|
267
269
|
| [`github-actions-templates`](./external/github-actions-templates/) | `external` | Authoring GitHub Actions workflows: triggers, matrices, caching, concurrency, least-privilege permissions, SHA pinning, OIDC, reusable workf |
|
|
270
|
+
| [`handle-request`](./external/handle-request/) | `external` | Add the test handles a scenario needs and answer each request with ADDED, ALREADY or COULD NOT ADD plus file:line or reason. Identity values |
|
|
268
271
|
| [`humanizer`](./external/humanizer/) | `external` | \| |
|
|
269
272
|
| [`localization-reuse-map`](./external/localization-reuse-map/) | `external` | Localization reuse map for one redesigned screen on iOS, Android and web: new keys set against each platform's legacy key, legacy values and |
|
|
270
273
|
| [`pdfkit`](./external/pdfkit/) | `external` | PDFKit on iOS: PDFView display and SwiftUI wrapping, creating and editing PDFDocument and PDFPage, highlight, note, signature and form annot |
|
|
274
|
+
| [`scenario-gap-analysis`](./external/scenario-gap-analysis/) | `external` | Set an inventory against existing tests, sort each behaviour into COVERED, WRONGLY COVERED or MISSING, and write gaps as measurable, priorit |
|
|
271
275
|
| [`search-first`](./external/search-first/) | `external` | \| |
|
|
272
276
|
| [`security-review`](./external/security-review/) | `external` | Run a defensive, static security review of a diff or a repo: build a run-scoped threat model, find vulnerabilities joined to OWASP + CWE wit |
|
|
273
277
|
| [`signal-community`](./external/signal-community/) | `external` | Check whether other people hit the same problem, via Stack Overflow and Hacker News (keyless) plus opt-in Reddit and X. Use when a bug repor |
|
|
274
278
|
| [`skill-creator`](./external/skill-creator/) | `external` | House guide for authoring plugin skills: creating a new skill, trimming or splitting a bloated one, writing or sharpening a description, and |
|
|
275
279
|
| [`spm-build-analysis`](./external/spm-build-analysis/) | `external` | Analyze Swift Package Manager dependencies, package plugins, module variants, and CI-oriented build overhead that slow Xcode builds. Use whe |
|
|
280
|
+
| [`test-result-triage`](./external/test-result-triage/) | `external` | Judge each test result on changed code as product-bug, stale-test, valid or ambiguous (the default), and list behaviour no test covers. stal |
|
|
276
281
|
| [`valven-rules`](./external/valven-rules/) | `external` | Delivery-flow rules from metrics Valven reports: PR size, one concern per PR, tests with source, issue keys, no WIP commits, churn, PR body, |
|
|
277
282
|
| [`xcode-build-benchmark`](./external/xcode-build-benchmark/) | `external` | Benchmark Xcode clean and incremental builds with repeatable inputs, timing summaries, and timestamped `.build-benchmark/` artifacts. Use wh |
|
|
278
283
|
| [`xcode-build-fixer`](./external/xcode-build-fixer/) | `external` | Apply approved Xcode build optimization changes following best practices, then re-benchmark to verify improvement. Use when a developer has |
|
|
@@ -116,10 +116,10 @@ Apple App Store Review Guidelines + ITMS upload error codes that the underlying
|
|
|
116
116
|
| # | ruleID | Apple ref | What it checks | Default severity |
|
|
117
117
|
|---|---|---|---|---|
|
|
118
118
|
| 1 | `privacy-manifest` | Guideline 5.1.1 | `PrivacyInfo.xcprivacy` exists + well-formed + declares all required `NSPrivacyCollectedDataTypes` | error |
|
|
119
|
-
| 2 | `required-reason-api` | 5.1.1 + ITMS-
|
|
120
|
-
| 3 | `info-plist` |
|
|
119
|
+
| 2 | `required-reason-api` | 5.1.1 + ITMS-91053 | Binary-scanned API usage (UserDefaults, file timestamps, disk space, boot time, active keyboards) cross-referenced with privacy manifest reasons; reason codes must be on Apple's published list | error |
|
|
120
|
+
| 3 | `info-plist` | 2.1 / 2.3.7 / 5.1.1 | Purpose strings (`NS*UsageDescription`) for every declared permission, ATS configured, CFBundleDisplayName ≤ 30 chars, `MinimumOSVersion` ≥ 13 for iOS and iPadOS builds (upload requirement since 2026-09-09) | error/warning |
|
|
121
121
|
| 4 | `code-signing` | ITMS-90035 / 90034 / 90046 / 90717 | Debug entitlements absent in release, correct APS environment (`aps-environment=production`), signature present | error |
|
|
122
|
-
| 5 | `embedded-sdk` |
|
|
122
|
+
| 5 | `embedded-sdk` | ITMS-91061 / 91065 / 90174 | Third-party SDKs on [Apple's commonly used SDK list](https://developer.apple.com/support/third-party-SDK-requirements/) signed + have PrivacyInfo | error |
|
|
123
123
|
| 6 | `entitlement` | - | Entitlement ↔ Info.plist mappings (background modes, associated domains, app groups) consistent | warning |
|
|
124
124
|
| 7 | `asset-validation` | - | App icon set present + all required sizes, launch storyboard configured | error |
|
|
125
125
|
| 8 | `binary-size` | - | `.app` bundle size + estimated post-thinning download size (warns if > 200MB) | info |
|
|
@@ -132,7 +132,7 @@ Apple App Store Review Guidelines + ITMS upload error codes that the underlying
|
|
|
132
132
|
| 15 | `production-hygiene` | - | `print()` / `NSLog()` / `debugPrint()` symbol presence + mock/test JSON files shipped in bundle | warning |
|
|
133
133
|
| 16 | `duplicate-resource` | - | Same asset under multiple paths (bloats bundle) | info |
|
|
134
134
|
| 17 | `dead-reference` | - | Files referenced in Xcode proj but absent from archive | warning |
|
|
135
|
-
| 18 | `sdk-floor` | ITMS-90725 | `DTSDKName` / `DTPlatformVersion` major ≥ 26 and `DTXcode` ≥ 2600 (iOS 26 / Xcode 26 floor, in force since 2026-04-28) | error |
|
|
135
|
+
| 18 | `sdk-floor` | ITMS-90725 | `DTSDKName` / `DTPlatformVersion` major ≥ 26 and `DTXcode` ≥ 2600 (iOS 26 / Xcode 26 floor, in force since 2026-04-28); a build below the announced iOS 27 / Xcode 27 floor (uploads from April 2027) gets an info entry | error |
|
|
136
136
|
|
|
137
137
|
## Severity mapping for consumers
|
|
138
138
|
|
|
@@ -317,3 +317,4 @@ The appended section is structured:
|
|
|
317
317
|
- **v6.0.0** - `language: en` frontmatter tag; 4-consumer Examples section expanded.
|
|
318
318
|
- **v8.4.0** - Migrated off the standalone ArchiveGuard Swift binary. Backend is now the `ios_app_store_audit` tool inside `@mmerterden/multi-agent-toolkit-mcp` ≥ v2.4 (full Node port of the 17 rules). All four consumer surfaces unchanged; rule IDs and JSON output shape preserved for backward compatibility. Rule catalog now points to `multi-agent-toolkit-mcp/tools/ios-app-store-audit/rules/`. ArchiveGuard repo deprecated.
|
|
319
319
|
- **Unreleased** - Catalog grows to 18 rules: adds `sdk-floor` (ITMS-90725, iOS 26 / Xcode 26 build floor in force since 2026-04-28), which nothing previously checked. `embedded-sdk`'s missing-framework-privacy-manifest finding moves from a `5.1.1` WARNING to `ITMS-91061` ERROR, an enforced rejection since 2025-02-12 - archives that used to pass with a warning now fail. Requires `@mmerterden/multi-agent-toolkit-mcp` ≥ v3.0.0.
|
|
320
|
+
- **v20.27.2** - October 2026 guidelines: `info-plist` errors on an iOS or iPadOS `MinimumOSVersion` below 13 (App Store Connect upload requirement since 2026-09-09); `sdk-floor` names the announced iOS 27 / Xcode 27 floor as info; `required-reason-api` cites ITMS-91053 and validates against Apple's published reason-code list. Needs `@mmerterden/multi-agent-toolkit-mcp` ≥ v3.23.2; older servers keep the previous severities.
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
version: 1.0.0
|
|
2
|
-
updated: 2026-
|
|
2
|
+
updated: 2026-10-20
|
|
3
3
|
owner: iOS platform
|
|
4
4
|
description: >
|
|
5
5
|
App Store review + ITMS upload rules, as a machine-readable registry. IDs are the
|
|
@@ -47,18 +47,18 @@ rules:
|
|
|
47
47
|
rationale: app-store-compliance
|
|
48
48
|
- id: required-reason-api
|
|
49
49
|
title: >
|
|
50
|
-
Binary-scanned API usage (UserDefaults,
|
|
50
|
+
Binary-scanned API usage (UserDefaults, file timestamps, disk space, boot time, active keyboards) cross-referenced with privacy manifest reasons; reason codes must be on Apple's published list
|
|
51
51
|
severity: blocking
|
|
52
52
|
enforcement: scan
|
|
53
|
-
policy_ref: 5.1.1 + ITMS-
|
|
53
|
+
policy_ref: 5.1.1 + ITMS-91053
|
|
54
54
|
rationale: app-store-compliance
|
|
55
55
|
- id: info-plist
|
|
56
56
|
title: >
|
|
57
|
-
Purpose strings (`NS*UsageDescription`) for every declared permission, ATS configured, CFBundleDisplayName ≤ 30 chars
|
|
57
|
+
Purpose strings (`NS*UsageDescription`) for every declared permission, ATS configured, CFBundleDisplayName ≤ 30 chars, `MinimumOSVersion` ≥ 13 for iOS and iPadOS builds (upload requirement since 2026-09-09)
|
|
58
58
|
severity: blocking
|
|
59
59
|
severity_varies: error/warning
|
|
60
60
|
enforcement: scan
|
|
61
|
-
policy_ref:
|
|
61
|
+
policy_ref: 2.1 + 2.3.7 + 5.1.1
|
|
62
62
|
rationale: app-store-compliance
|
|
63
63
|
- id: code-signing
|
|
64
64
|
title: >
|
|
@@ -69,10 +69,10 @@ rules:
|
|
|
69
69
|
rationale: app-store-compliance
|
|
70
70
|
- id: embedded-sdk
|
|
71
71
|
title: >
|
|
72
|
-
Third-party SDKs on [Apple's
|
|
72
|
+
Third-party SDKs on [Apple's commonly used SDK list](https://developer.apple.com/support/third-party-SDK-requirements/) signed + have PrivacyInfo
|
|
73
73
|
severity: blocking
|
|
74
74
|
enforcement: scan
|
|
75
|
-
policy_ref:
|
|
75
|
+
policy_ref: ITMS-91061 + ITMS-91065 + ITMS-90174
|
|
76
76
|
rationale: app-store-compliance
|
|
77
77
|
- id: entitlement
|
|
78
78
|
title: >
|
|
@@ -160,7 +160,7 @@ rules:
|
|
|
160
160
|
rationale: app-store-compliance
|
|
161
161
|
- id: sdk-floor
|
|
162
162
|
title: >
|
|
163
|
-
`DTSDKName` / `DTPlatformVersion` major ≥ 26 and `DTXcode` ≥ 2600 (iOS 26 / Xcode 26 floor, in force since 2026-04-28)
|
|
163
|
+
`DTSDKName` / `DTPlatformVersion` major ≥ 26 and `DTXcode` ≥ 2600 (iOS 26 / Xcode 26 floor, in force since 2026-04-28); a build below the announced iOS 27 / Xcode 27 floor (uploads from April 2027) gets an info entry
|
|
164
164
|
severity: blocking
|
|
165
165
|
enforcement: scan
|
|
166
166
|
policy_ref: ITMS-90725
|
|
@@ -33,7 +33,7 @@ Run all steps automatically:
|
|
|
33
33
|
```
|
|
34
34
|
Step 0: DOCTOR node $HOME/.claude/scripts/doctor.mjs - exit 2 or 4 STOPS the sync
|
|
35
35
|
Step 1: DETECT Compare timestamps, find stale targets
|
|
36
|
-
Step 2: COPILOT Claude Code -> Copilot CLI (instructions +
|
|
36
|
+
Step 2: COPILOT Claude Code -> Copilot CLI (instructions + 66 sub-command skills)
|
|
37
37
|
Step 2b: CODEX Claude Code -> Codex CLI (1 router skill + sub-command specs as refs + agent TOML)
|
|
38
38
|
Step 3: REPO Claude Code -> pipeline repo (genericized, personal data scrub)
|
|
39
39
|
Step 3d: DEV-TOOLKIT Companion MCP server -> detect movement, ship gates, commit + publish
|
|
@@ -229,7 +229,7 @@ When invoked with the `release` argument:
|
|
|
229
229
|
|-------------|-------------|
|
|
230
230
|
| `~/.claude/commands/multi-agent/{cmd}/SKILL.md` | `~/.copilot/skills/multi-agent-{cmd}/SKILL.md` |
|
|
231
231
|
|
|
232
|
-
**
|
|
232
|
+
**66 commands are synced** (canonical inventory - must match `cross-cli-contract.md` section 1; drift = contract violation):
|
|
233
233
|
|
|
234
234
|
```
|
|
235
235
|
analysis, analysis-jira, analysis-resolve, autopilot, autopilot-off,
|
|
@@ -241,7 +241,7 @@ prune-prompts, purge, refactor, research, resume, review,
|
|
|
241
241
|
review-analysis, review-issue, review-jira, route-off, route-on,
|
|
242
242
|
route-status, routines, save, scaffold, scan, scenario-audit, search,
|
|
243
243
|
security-review, serve, setup, stack, status, steer, store-ready, sync, test, test-accessibility,
|
|
244
|
-
test-dark-mode, test-dynamic-type, test-screenshots, testflight-validation,
|
|
244
|
+
test-dark-mode, test-dynamic-type, test-gen, test-screenshots, testflight-validation,
|
|
245
245
|
uninstall, update
|
|
246
246
|
```
|
|
247
247
|
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: multi-agent-test-gen
|
|
3
|
+
description: "Write the tests an app lacks, stage by stage, each gated on its artefact: inventory, read-only gap analysis, handles, tests, runs, per-test classification. Use when screens, flows or a change need tests."
|
|
4
|
+
user-invocable: true
|
|
5
|
+
argument-hint: "[scope] [--platform ios|android|web] [--repo <path>] - scope: a screen, route or flow, or empty for what changed since the last test-gen run"
|
|
6
|
+
metadata:
|
|
7
|
+
language: en
|
|
8
|
+
not-for: test, bug-bash, scenario-audit, manual-test
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# multi-agent-test-gen
|
|
12
|
+
|
|
13
|
+
**Input**: $ARGUMENTS
|
|
14
|
+
|
|
15
|
+
Writes the tests an app is missing, in the repo's own frameworks, and reports
|
|
16
|
+
what they found. Seven stages run as subagents, each leaving one artefact that a
|
|
17
|
+
script gates before the next stage starts: inventory, discovery, gaps, handles,
|
|
18
|
+
write, run, classify. The two analysis stages run as a read-only agent type, so
|
|
19
|
+
nothing they do can change the repository.
|
|
20
|
+
|
|
21
|
+
Stage contract, read before Step 1:
|
|
22
|
+
|
|
23
|
+
```
|
|
24
|
+
$HOME/.claude/multi-agent-refs/test-gen/stages.md
|
|
25
|
+
$HOME/.claude/multi-agent-refs/features/test-classification.md
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Scripts: `test-gen-gate.mjs` (artefact, handle and write gates),
|
|
29
|
+
`handle-diff-audit.mjs` (handles only), `test-classify.mjs`,
|
|
30
|
+
`test-gen-state.mjs` (change base, revision cap), `evidence-gate.mjs`, all under
|
|
31
|
+
`$HOME/.claude/scripts/`. Schemas: `test-gen-inventory`, `test-gen-gaps`,
|
|
32
|
+
`test-gen-results` under `$HOME/.claude/schemas/`.
|
|
33
|
+
|
|
34
|
+
## Gotchas
|
|
35
|
+
|
|
36
|
+
- An analysis stage that wants to "just fix" something is out of its lane. Discovery and gaps go to `test-gen-analyst`, whose grant is an allowlist of read-only tools.
|
|
37
|
+
- The handle stage adds handles and nothing else. A wrapped view, a renamed element or a changed condition fails `handle-diff-audit.mjs`; revert the hunk, do not argue with the audit.
|
|
38
|
+
- `HANDLES: none-needed` is a claim that the handles exist. It needs the `file:line` of each.
|
|
39
|
+
- A test that passed once is not green. Unstable is a fail, and a red test with no base to compare is `not_measured`, not a regression.
|
|
40
|
+
- Test data is named in `dataNeeds[]`, never filled in by the run. A credential is a `secretRef`.
|
|
41
|
+
- A product bug is reported, never fixed here. In autopilot it parks the run.
|
|
42
|
+
|
|
43
|
+
## Step 1 - Prepare
|
|
44
|
+
|
|
45
|
+
1. Repo: `--repo`, else the current directory; confirm it in one line.
|
|
46
|
+
Platform: `--platform`, else detect as `multi-agent-test` does. Run id
|
|
47
|
+
`tg-<timestamp>`; `RUN="$HOME/.claude/logs/test-gen/<repo>/<timestamp>"`,
|
|
48
|
+
created with `mkdir -p "$RUN"`.
|
|
49
|
+
2. Toolkit: `test_results_classify`, `coverage_gap` and `ui_locator_codegen`
|
|
50
|
+
need `@mmerterden/multi-agent-toolkit-mcp` >= v3.25.0. Check the tool list
|
|
51
|
+
once. Each missing tool has a fallback in `stages.md` (classify from the
|
|
52
|
+
repeats, the analyst matches coverage itself with every match `inferred`,
|
|
53
|
+
locators written by hand from the handles); say which fallbacks are in use,
|
|
54
|
+
once.
|
|
55
|
+
3. Change base. `TG_STATE="$HOME/.claude/knowledge/<project>/test-gen-state.json"`:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
node "$HOME/.claude/scripts/test-gen-state.mjs" base-scan --state "$TG_STATE" --repo "$REPO" \
|
|
59
|
+
--graph "$HOME/.claude/knowledge/<project>/code-graph.json" > "$RUN/base-scan.json"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Exit 0: the changed files and their dependents narrow an empty scope. Exit 1
|
|
63
|
+
(`measured: false`, first run or base gone): nothing is narrowed, and the
|
|
64
|
+
reason goes in the report.
|
|
65
|
+
|
|
66
|
+
## Step 2 - Inventory and priority
|
|
67
|
+
|
|
68
|
+
Stage 1 of `stages.md`, then the priority pick: ask which items to cover
|
|
69
|
+
(one question to the user; autopilot: the top three, recorded as `autopilot-top3`). Gate:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
node "$HOME/.claude/scripts/test-gen-gate.mjs" check --stage inventory --file "$RUN/inventory.json"
|
|
73
|
+
node "$HOME/.claude/scripts/evidence-gate.mjs" --claim artefact --status passed \
|
|
74
|
+
--evidence "$RUN/inventory.json" --stage-start "$STAGE_START"
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Step 3 - Discovery and gaps (read-only)
|
|
78
|
+
|
|
79
|
+
Stages 2 and 3, both with `subagent_type: "test-gen-analyst"`. Write each answer
|
|
80
|
+
with the Write tool, then gate each file with `evidence-gate.mjs --claim
|
|
81
|
+
artefact` against its own stage start, and each with `test-gen-gate.mjs check`
|
|
82
|
+
(`--stage discovery`, `--stage gaps`). Show the gaps (bucket, priority,
|
|
83
|
+
expectation) and the `dataNeeds[]`; ask for the data or the credential names
|
|
84
|
+
(autopilot: write only the scenarios whose needs are met).
|
|
85
|
+
|
|
86
|
+
## Step 4 - Handles
|
|
87
|
+
|
|
88
|
+
Stage 4. Snapshot first (`test-gen-gate.mjs baseline`), then the subagent, then
|
|
89
|
+
both gates against that snapshot: `test-gen-gate.mjs handles` and
|
|
90
|
+
`handle-diff-audit.mjs --baseline` exit 0. The gates read only the stage's own
|
|
91
|
+
change; the person's uncommitted work is never the stage's to revert. Skip the
|
|
92
|
+
stage when no gap has `handlesNeeded`, and say so.
|
|
93
|
+
|
|
94
|
+
## Step 5 - Write, run, classify
|
|
95
|
+
|
|
96
|
+
Stages 5 to 7: a new snapshot, the tests and `written.json`, the
|
|
97
|
+
`test-gen-gate.mjs written --baseline` gate (test files only, inside the repo
|
|
98
|
+
and changed by the stage, no test gaming, a discovered runner, more than zero
|
|
99
|
+
tests run), the repeated runs, then
|
|
100
|
+
`results.json` through `test-classify.mjs` and `test-gen-gate.mjs results`.
|
|
101
|
+
|
|
102
|
+
## Step 6 - Triage
|
|
103
|
+
|
|
104
|
+
Stage 8, one verdict per non-pass result, each through `test-gen-state.mjs
|
|
105
|
+
revise`. Exit 3 parks the run: stop, keep every artefact, report why, and record
|
|
106
|
+
no change base, because the run did not finish.
|
|
107
|
+
|
|
108
|
+
## Step 7 - Report and base
|
|
109
|
+
|
|
110
|
+
Print, in this order: the tests written (file, scenario, expectation), the
|
|
111
|
+
results table from `results.json` with every `fail` and `not_measured` reason,
|
|
112
|
+
the triage verdicts, the handles added (`handles.md`), the scenarios blocked by
|
|
113
|
+
data, and `gaps.json`'s `unmeasured[]`. State which parts are mechanical (the
|
|
114
|
+
gates, the classification, the handle audit) and which are judgement (the
|
|
115
|
+
inventory, the gaps, the triage). Then record the change base
|
|
116
|
+
(`test-gen-state.mjs base-record`).
|
|
117
|
+
|
|
118
|
+
Offer, one question: open the product bugs through `multi-agent` with each
|
|
119
|
+
failing test as the acceptance check, or stop. **No commit, no push, no issue
|
|
120
|
+
from this command.**
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: app-store-review
|
|
3
|
-
description: "App Store submission readiness and rejection prevention: Review Guidelines 2.1, 2.3, 2.5.1, 3.1.1, 4.2 and 5.1.2, PrivacyInfo.xcprivacy manifests and required-reason codes, IAP and StoreKit rules, ATT consent, EU DMA distribution, entitlements and usage strings, metadata limits, screenshots, TestFlight, phased release, Resolution Center appeals, HIG review items, widget and Live Activity review rules. Use when preparing a submission, fixing a rejection, auditing a privacy manifest, implementing the ATT prompt or checking IAP compliance. Not for keyword research, ranking, conversion copy, screenshot order or A/B tests (app-store-optimization)."
|
|
3
|
+
description: "App Store submission readiness and rejection prevention: Review Guidelines 1.2, 2.1, 2.3, 2.3.14, 2.5.1, 2.5.19, 3.1.1, 4.2 and 5.1.2, upload floors, age rating and medical device declarations, PrivacyInfo.xcprivacy manifests and required-reason codes, IAP and StoreKit rules, ATT consent, EU DMA distribution, entitlements and usage strings, metadata limits, screenshots, TestFlight, phased release, Resolution Center appeals, HIG review items, widget and Live Activity review rules. Use when preparing a submission, fixing a rejection, auditing a privacy manifest, implementing the ATT prompt or checking IAP compliance. Not for keyword research, ranking, conversion copy, screenshot order or A/B tests (app-store-optimization)."
|
|
4
4
|
metadata:
|
|
5
5
|
source: multi-agent-pipeline
|
|
6
6
|
---
|
|
@@ -42,8 +42,10 @@ Limits to state on every metadata answer:
|
|
|
42
42
|
A full audit reports two lists: things that stop upload or review, and cleanup.
|
|
43
43
|
Treat these as blockers:
|
|
44
44
|
|
|
45
|
-
1. The build
|
|
46
|
-
or later
|
|
45
|
+
1. The build was not produced with Xcode 26 or later against the platform's 26
|
|
46
|
+
SDK or later (required since April 28, 2026; the 27 SDK is required from
|
|
47
|
+
April 2027), or its deployment target is below iOS 13 (required since
|
|
48
|
+
September 9, 2026).
|
|
47
49
|
2. The privacy manifest, the nutrition label, the privacy policy, the ATT state,
|
|
48
50
|
what bundled SDKs transmit and what the app does on the network (audited at
|
|
49
51
|
runtime) do not tell the same story.
|
|
@@ -54,11 +56,27 @@ Treat these as blockers:
|
|
|
54
56
|
missing for an app that runs on iPad.
|
|
55
57
|
5. Features behind a login, or features a reviewer would not find alone, ship
|
|
56
58
|
with no demo credentials, no demo mode and no App Review notes.
|
|
59
|
+
6. The App Store Connect declarations are missing: the age rating questionnaire
|
|
60
|
+
including its social media questions, or, for a Health & Fitness or Medical
|
|
61
|
+
app, the regulated medical device status.
|
|
57
62
|
|
|
58
63
|
Everything else (copy polish, an optional screenshot, a HIG nit) is cleanup.
|
|
59
64
|
|
|
60
65
|
## Guidelines that cause most rejections
|
|
61
66
|
|
|
67
|
+
### 1.2 User-generated and AI-generated content
|
|
68
|
+
|
|
69
|
+
Applies to social apps, user-generated content platforms and apps that create
|
|
70
|
+
content with AI. Required: filtering of objectionable material, reporting with
|
|
71
|
+
timely responses, blocking abusive users, and published contact information.
|
|
72
|
+
Apps used primarily for random or anonymous chat, pornography, objectifying
|
|
73
|
+
real people or bullying do not belong on the App Store, whatever moderation
|
|
74
|
+
they add. The guideline names nonconsensual intimate imagery among the risks,
|
|
75
|
+
and Apple's announcement of 16.10.2026 asks apps to prevent and remove it,
|
|
76
|
+
AI-generated imagery included. A developer who adds no safeguards, or does not
|
|
77
|
+
remediate after Apple's notice, may be considered in violation of the
|
|
78
|
+
Developer Code of Conduct.
|
|
79
|
+
|
|
62
80
|
### 2.1 App Completeness
|
|
63
81
|
|
|
64
82
|
Rejected: placeholder text, lorem ipsum or visible test records; broken links and
|
|
@@ -79,6 +97,13 @@ run every flow from start to finish including its empty and error states.
|
|
|
79
97
|
belong. The primary
|
|
80
98
|
category must match the main function.
|
|
81
99
|
|
|
100
|
+
### 2.3.14 Creative assets
|
|
101
|
+
|
|
102
|
+
Creative assets appear in the product page header and in search results. They
|
|
103
|
+
must relate to the app's content, function or brand, follow the metadata rules
|
|
104
|
+
and suit all audiences: no references to other apps meant to mislead, no
|
|
105
|
+
unverifiable claims, no specific or discounted prices.
|
|
106
|
+
|
|
82
107
|
### 4.2 Minimum Functionality
|
|
83
108
|
|
|
84
109
|
- A shell around `WKWebView` is rejected unless it adds meaningful native
|
|
@@ -92,12 +117,38 @@ run every flow from start to finish including its empty and error states.
|
|
|
92
117
|
|
|
93
118
|
- Public APIs only. Private API use is rejected outright.
|
|
94
119
|
- From April 28, 2026, uploads need Xcode 26+ and the matching 26+ platform SDK.
|
|
95
|
-
|
|
96
|
-
|
|
120
|
+
From April 2027 the floor is the 27 SDK.
|
|
121
|
+
- App Store Connect accepts iOS and iPadOS uploads that target iOS 13 or later.
|
|
122
|
+
Above that floor the deployment target is a product choice.
|
|
97
123
|
- The app may not download or run code that adds or changes features after
|
|
98
124
|
review. The exception is interpreted code that Apple's guidelines and
|
|
99
125
|
agreements explicitly allow.
|
|
100
126
|
|
|
127
|
+
### 2.5.19 Hidden, dormant or undocumented features
|
|
128
|
+
|
|
129
|
+
Every feature is visible to users and to App Review. A feature switched on by
|
|
130
|
+
remote config after approval, a dormant code path, or functionality hidden in
|
|
131
|
+
the binary through obfuscation leads to removal and can end the developer
|
|
132
|
+
account. Feature flags are fine when the review notes name what they gate and
|
|
133
|
+
the reviewer can reach it.
|
|
134
|
+
|
|
135
|
+
## App Store Connect declarations
|
|
136
|
+
|
|
137
|
+
These live in App Store Connect, not in the repo, so an audit reports them as
|
|
138
|
+
manual checks:
|
|
139
|
+
|
|
140
|
+
- Age rating questionnaire, including the social media capability questions
|
|
141
|
+
(required since September 2026). Apps with social features show a Social
|
|
142
|
+
Media descriptor; turning those features off for users under 13 exempts them
|
|
143
|
+
from the Social Media Time Allowance.
|
|
144
|
+
- Regulated medical device status for apps whose primary or secondary category
|
|
145
|
+
is Health & Fitness or Medical, or that frequently reference medical or
|
|
146
|
+
treatment information (EEA, UK, US). Required for new apps; existing apps
|
|
147
|
+
cannot submit updates without it from early 2027.
|
|
148
|
+
- Regional age ratings where they apply (for example Korea GRAC, Australia,
|
|
149
|
+
Vietnam) and age assurance through the Declared Age Range API where the law
|
|
150
|
+
requires it.
|
|
151
|
+
|
|
101
152
|
## Privacy manifest in brief
|
|
102
153
|
|
|
103
154
|
`PrivacyInfo.xcprivacy` is needed whenever code in the app, in an executable, in
|
|
@@ -146,7 +197,9 @@ Rules around the purchase:
|
|
|
146
197
|
happens, that it renews automatically, and how to cancel, all before purchase.
|
|
147
198
|
- Strip any route to an outside checkout for digital goods, whether a link, a
|
|
148
199
|
button or a nudge in copy, unless the storefront's current rule or a granted
|
|
149
|
-
entitlement permits it.
|
|
200
|
+
entitlement permits it. A StoreKit External Purchase Link entitlement or,
|
|
201
|
+
where permitted, a Music Streaming Services entitlement allows a link to a
|
|
202
|
+
permitted destination; the US storefront needs no entitlement. These rules differ by region and have changed recently; read the
|
|
150
203
|
current 3.1.1(a) and 3.1.3 text for the target storefront.
|
|
151
204
|
- Reader apps may link out to account sign-up with the matching entitlement, but
|
|
152
205
|
may not offer a way around IAP for in-app purchases.
|
|
@@ -327,8 +380,16 @@ implementation.
|
|
|
327
380
|
- [ ] IAP for digital content; terms shown before purchase; restore works; no
|
|
328
381
|
external purchase CTAs unless allowed
|
|
329
382
|
- [ ] Standard navigation; the UI holds up in Dark Mode and at large Dynamic Type
|
|
330
|
-
- [ ] Toolchain is Xcode 26+ with a 26+ SDK (required from April 28, 2026
|
|
331
|
-
private APIs; every
|
|
383
|
+
- [ ] Toolchain is Xcode 26+ with a 26+ SDK (required from April 28, 2026; 27 SDK
|
|
384
|
+
from April 2027); deployment target iOS 13+; no private APIs; every
|
|
385
|
+
entitlement justified
|
|
386
|
+
- [ ] No feature gated by remote config or hidden code that the review notes
|
|
387
|
+
leave out (2.5.19)
|
|
388
|
+
- [ ] User or AI-generated content has filtering, reporting, blocking, contact
|
|
389
|
+
and a way to report and remove NCII (1.2); the app is not primarily random
|
|
390
|
+
or anonymous chat
|
|
391
|
+
- [ ] Age rating (social media questions) and, for health apps, the medical
|
|
392
|
+
device declaration are filled in App Store Connect
|
|
332
393
|
- [ ] ATT prompt only if the app tracks
|
|
333
394
|
|
|
334
395
|
Full lists: [review checklists](references/review-checklists.md).
|
|
@@ -39,6 +39,18 @@
|
|
|
39
39
|
"Names the 6.9-inch iPhone and 13-inch iPad screenshot sets",
|
|
40
40
|
"Does not produce a full keyword plan or rewrite the marketing copy"
|
|
41
41
|
]
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
"id": 4,
|
|
45
|
+
"prompt": "Our photo app lets users generate stylized portraits with an on-device model and share them to a public feed. A new filter is switched on by remote config after approval. The deployment target is iOS 12. What does App Review object to?",
|
|
46
|
+
"expected_output": "A blocker-first answer covering the upload floor, 1.2 for AI-generated and shared content, 2.5.19 for the remote-config feature, and the age rating social media questions.",
|
|
47
|
+
"files": [],
|
|
48
|
+
"expectations": [
|
|
49
|
+
"States that App Store Connect rejects uploads targeting below iOS 13",
|
|
50
|
+
"Applies guideline 1.2 to the AI-generated, shared content: filtering, reporting, blocking, published contact, and prevention and removal of nonconsensual intimate imagery",
|
|
51
|
+
"Cites 2.5.19: a feature switched on after approval must be described in the review notes and reachable by the reviewer, or it is a hidden feature",
|
|
52
|
+
"Says the age rating questionnaire's social media questions must be answered because the app has a public feed"
|
|
53
|
+
]
|
|
42
54
|
}
|
|
43
55
|
]
|
|
44
56
|
}
|
|
@@ -125,9 +125,18 @@ Design:
|
|
|
125
125
|
|
|
126
126
|
Technical:
|
|
127
127
|
|
|
128
|
-
- [ ]
|
|
128
|
+
- [ ] Built with Xcode 26+ and SDK 26+ (required since April 28, 2026); SDK 27+ from April 2027.
|
|
129
|
+
- [ ] Deployment target iOS 13 or later (required for upload since September 9, 2026).
|
|
129
130
|
- [ ] No private APIs.
|
|
130
131
|
- [ ] No code downloaded and executed to change features.
|
|
132
|
+
- [ ] No hidden, dormant or undocumented features; every remote-config flag is named in the review notes (2.5.19).
|
|
131
133
|
- [ ] Every entitlement justified, with usage descriptions.
|
|
132
134
|
- [ ] Every background mode justified and in use.
|
|
133
135
|
- [ ] Deployment target chosen on purpose and tested on the oldest supported OS.
|
|
136
|
+
|
|
137
|
+
App Store Connect:
|
|
138
|
+
|
|
139
|
+
- [ ] Age rating questionnaire answered, social media capability questions included.
|
|
140
|
+
- [ ] Health & Fitness or Medical apps: regulated medical device status declared.
|
|
141
|
+
- [ ] Creative assets relate to the app and carry no prices or unverifiable claims (2.3.14).
|
|
142
|
+
- [ ] User or AI-generated content: filtering, reporting, blocking, published contact, a way to report and remove NCII (1.2); not primarily random or anonymous chat.
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: fix-review-verdicts
|
|
3
|
+
description: "Independent fix review with three verdicts, test-gaming, scope and root-cause, each backed by file:line findings; the result is computed from findings, never from a claim. Use when reviewing a fix made to turn tests green. Not a full review checklist."
|
|
4
|
+
metadata:
|
|
5
|
+
source: multi-agent-pipeline
|
|
6
|
+
version: 1.0.0
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Fix review verdicts
|
|
10
|
+
|
|
11
|
+
Tests can be made green without fixing anything: hide the element only under
|
|
12
|
+
test, branch on an automation flag, loosen a check, swallow the error. The
|
|
13
|
+
tests pass and report nothing. Only someone reading the diff catches it, and
|
|
14
|
+
that reader should not be the one who wrote the fix.
|
|
15
|
+
|
|
16
|
+
## 1. Inputs
|
|
17
|
+
|
|
18
|
+
The reviewer does not see the author's conversation. It reads:
|
|
19
|
+
|
|
20
|
+
- the approved root-cause statement: what broke, where, and the agreed fix;
|
|
21
|
+
- the diff;
|
|
22
|
+
- the evidence runs: did the expected tests turn green, did the guard tests
|
|
23
|
+
stay green;
|
|
24
|
+
- machine check results (lint, types, unit tests) from the project's own
|
|
25
|
+
approved commands;
|
|
26
|
+
- the author's report, as a claim to verify, not as evidence.
|
|
27
|
+
|
|
28
|
+
The reviewer changes no code. Its only output is the review.
|
|
29
|
+
|
|
30
|
+
## 2. The three required verdicts
|
|
31
|
+
|
|
32
|
+
Each one gets an explicit verdict and a reason.
|
|
33
|
+
|
|
34
|
+
**test-gaming: `clean` | `violation`.** A violation is any of:
|
|
35
|
+
|
|
36
|
+
- an environment branch in product code: `navigator.webdriver`,
|
|
37
|
+
`process.env.CI`, `window.Cypress`, a user-agent or URL-parameter check,
|
|
38
|
+
`ProcessInfo` arguments or environment, an `isRunningTests` flag, a
|
|
39
|
+
test-only build flag;
|
|
40
|
+
- detecting automation, or showing or hiding an element only under test;
|
|
41
|
+
- hard-coding the text or value the test looks for;
|
|
42
|
+
- a swallowed error: an empty `catch`, a discarded `try?`, an ignored
|
|
43
|
+
`runCatching`;
|
|
44
|
+
- a loosened product validation, an inflated timeout, a skipped or disabled
|
|
45
|
+
test;
|
|
46
|
+
- an existing assertion, fixture or snapshot changed so that it accepts what
|
|
47
|
+
it used to reject (a weaker matcher, a widened tolerance, a re-recorded
|
|
48
|
+
snapshot, a fixture edited to dodge the bug) in a fix whose cause is in
|
|
49
|
+
product code.
|
|
50
|
+
|
|
51
|
+
Not a violation: a new regression test that encodes the bug, fails on the
|
|
52
|
+
unfixed code and passes on the fix. That is the bug-fix protocol (a failing
|
|
53
|
+
test first), and it is expected. Check it actually fails without the fix; a
|
|
54
|
+
new test that passes on the unfixed base proves nothing and is a suggestion
|
|
55
|
+
finding.
|
|
56
|
+
|
|
57
|
+
**scope: `clean` | `violation`.** The diff touches only the approved cause.
|
|
58
|
+
Unrelated files, reformatting, opportunistic refactors, lockfile or dependency
|
|
59
|
+
changes and configuration changes are violations.
|
|
60
|
+
|
|
61
|
+
**root-cause: `root` | `symptom`.** Does the fix remove the cause or suppress
|
|
62
|
+
the result? Fixing contrast by hiding the element, overflow by clipping it, an
|
|
63
|
+
error by deleting its message, a race by retrying until it passes, or bad
|
|
64
|
+
input by clamping it are symptoms.
|
|
65
|
+
|
|
66
|
+
## 3. Findings
|
|
67
|
+
|
|
68
|
+
```json
|
|
69
|
+
{ "severity": "blocking", "area": "test-gaming", "file": "src/cart/Total.tsx", "line": 42, "note": "returns the expected total when navigator.webdriver is set" }
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
- `severity`: `blocking` or `suggestion`.
|
|
73
|
+
- `area`: `test-gaming`, `scope`, `symptom`, `correctness`, `security`,
|
|
74
|
+
`performance`, `maintainability`, `accessibility`.
|
|
75
|
+
- Each verdict maps to one finding area: test-gaming `violation` to area
|
|
76
|
+
`test-gaming`, scope `violation` to area `scope`, root-cause `symptom` to
|
|
77
|
+
area `symptom`. Every such verdict comes with at least one blocking finding
|
|
78
|
+
in its area. A correctness or security defect is blocking.
|
|
79
|
+
- A suspicion without a line to show is a suggestion with its reason. A line
|
|
80
|
+
that shows test-gaming, scope creep or a symptom fix is blocking, without
|
|
81
|
+
hedging.
|
|
82
|
+
|
|
83
|
+
## 4. Computing the result
|
|
84
|
+
|
|
85
|
+
The result is derived, never declared. An author's "all green", a reviewer's
|
|
86
|
+
"looks good" and a summary field saying passed are ignored. It has three
|
|
87
|
+
states: `passed`, `blocked`, `not_measured`.
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
VERDICT_AREA = {test-gaming: violation -> test-gaming,
|
|
91
|
+
scope: violation -> scope,
|
|
92
|
+
root-cause: symptom -> symptom}
|
|
93
|
+
|
|
94
|
+
if the review was never submitted:
|
|
95
|
+
result = not_measured
|
|
96
|
+
elif any of the three verdicts is missing or outside its allowed values:
|
|
97
|
+
result = not_measured # an invalid review measures nothing
|
|
98
|
+
else:
|
|
99
|
+
for each (verdict, value) -> area in VERDICT_AREA that matches:
|
|
100
|
+
if no blocking finding exists in that area:
|
|
101
|
+
add a blocking finding in that area, built from the verdict's reason
|
|
102
|
+
for each machine check:
|
|
103
|
+
failed on the fix and not on the base -> blocking finding (correctness)
|
|
104
|
+
changed the working tree -> blocking finding (checks read, never write)
|
|
105
|
+
failed on the base as well -> preexisting: reported, not blocking
|
|
106
|
+
had no approved command (undefined) -> unmeasured check
|
|
107
|
+
could not run (tool missing, timeout) -> unmeasured check
|
|
108
|
+
if any finding is blocking:
|
|
109
|
+
result = blocked
|
|
110
|
+
elif any unmeasured check:
|
|
111
|
+
result = not_measured
|
|
112
|
+
else:
|
|
113
|
+
result = passed
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`blocked` wins over `not_measured`: a known defect is reported even when part
|
|
117
|
+
of the review could not run. `not_measured` never counts as `passed`; the fix
|
|
118
|
+
does not ship on it, and the report names what was not measured and why.
|
|
119
|
+
|
|
120
|
+
## 5. Output
|
|
121
|
+
|
|
122
|
+
The structured review (three verdicts with reasons, findings, a two to four
|
|
123
|
+
sentence summary) and a report: each verdict with its reason, the findings
|
|
124
|
+
with `file:line`, and a "could not measure" section.
|
|
125
|
+
|
|
126
|
+
## Related
|
|
127
|
+
|
|
128
|
+
- The stack's review rubric for the general pass: `ai-android-toolkit`
|
|
129
|
+
`reference/review-checklist`, `ai-ios-toolkit:ios-coding-standard`, and the
|
|
130
|
+
built-in `/code-review`.
|
|
131
|
+
- `security-review` when the diff touches auth, input handling or secrets.
|
|
132
|
+
- `test-result-triage`, which decides whether a red test was a product bug
|
|
133
|
+
before any fix is written.
|