@ivorycanvas/qamap 0.3.1 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -0
- package/README.md +20 -16
- package/dist/cli.js +33 -6
- package/dist/cli.js.map +1 -1
- package/dist/domain-language.d.ts +1 -1
- package/dist/domain-language.js +35 -6
- package/dist/domain-language.js.map +1 -1
- package/dist/e2e.d.ts +1 -0
- package/dist/e2e.js +234 -46
- package/dist/e2e.js.map +1 -1
- package/dist/import-graph.d.ts +17 -0
- package/dist/import-graph.js +308 -0
- package/dist/import-graph.js.map +1 -0
- package/dist/manifest.js +1 -1
- package/dist/manifest.js.map +1 -1
- package/dist/test-plan.d.ts +7 -0
- package/dist/test-plan.js +47 -0
- package/dist/test-plan.js.map +1 -1
- package/dist/verify.js +19 -1
- package/dist/verify.js.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/docs/adoption.md +2 -2
- package/docs/assets/qamap-30s-demo.gif +0 -0
- package/docs/benchmarking.md +31 -0
- package/docs/configuration.md +12 -12
- package/docs/e2e-output-examples.md +40 -40
- package/docs/eval.md +1 -1
- package/docs/github-action.md +2 -2
- package/docs/manifest.md +13 -13
- package/docs/release-validation.md +21 -5
- package/docs/roadmap.md +0 -1
- package/docs/verify.md +1 -1
- package/package.json +3 -2
package/docs/adoption.md
CHANGED
|
@@ -76,8 +76,8 @@ Start advisory, then tighten the gate once the findings are understood.
|
|
|
76
76
|
When scanning a package inside a larger workspace, pass the workspace root so QAMap can separate package-local risks from repository guardrails:
|
|
77
77
|
|
|
78
78
|
```sh
|
|
79
|
-
qamap doctor services/
|
|
80
|
-
qamap scan services/
|
|
79
|
+
qamap doctor services/listing --workspace-root . --format markdown
|
|
80
|
+
qamap scan services/listing --workspace-root . --json
|
|
81
81
|
```
|
|
82
82
|
|
|
83
83
|
With `--workspace-root`, QAMap reads package-local files such as `package.json`, `.env.*`, and MCP config from the package path. It reads repo-level guardrails such as `AGENTS.md`, `.github/workflows`, `LICENSE`, `SECURITY.md`, and `CONTRIBUTING.md` from the workspace root.
|
|
Binary file
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Benchmarking QAMap
|
|
2
|
+
|
|
3
|
+
`scripts/bench.mjs` scores QAMap's plan/qa output against a fixed set of repositories with pinned commits, so heuristic changes can be judged by numbers instead of feel. The runner is read-only against every target.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
1. Copy `bench.config.example.json` to `bench.config.local.json` (gitignored).
|
|
8
|
+
2. Point each target at a local repository checkout, pin `base`/`head` commit SHAs, and optionally declare expectations:
|
|
9
|
+
- `expect.runner` — the runner a human would pick for this repo (`playwright`, `maestro`, `manual`).
|
|
10
|
+
- `expect.mustReachFiles` — files a human QA would name for this diff window; recall against generated flows is reported.
|
|
11
|
+
- `expect.mustNameFlows` — substrings that should appear in at least one flow title.
|
|
12
|
+
|
|
13
|
+
## Run
|
|
14
|
+
|
|
15
|
+
```sh
|
|
16
|
+
pnpm bench # table only
|
|
17
|
+
node scripts/bench.mjs --save # also writes bench-results/<timestamp>.json
|
|
18
|
+
node scripts/bench.mjs --baseline bench-results/<file>.json # delta vs a saved run
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Metrics
|
|
22
|
+
|
|
23
|
+
- `runner` mismatch lines — wrong tool recommendation for a known repo shape.
|
|
24
|
+
- `viaImport` — flows produced through the reverse import graph (shared-file changes reaching surfaces).
|
|
25
|
+
- `diffAnchor` — flows whose selector set includes at least one selector introduced by the diff itself; higher means drafts act on what the change added instead of pre-existing UI.
|
|
26
|
+
- `blank` — steps with blank action slots (must stay 0).
|
|
27
|
+
- `generic` — flows with content-free names (`... primary journey`, `... smoke flow`); lower is better.
|
|
28
|
+
- `reach` — recall of `mustReachFiles`; missing files are listed per target.
|
|
29
|
+
- `agentBytes` — size of the `--format agent` payload.
|
|
30
|
+
|
|
31
|
+
Re-run with `--baseline` before merging heuristic changes; a PR that moves these numbers should say so in its body.
|
package/docs/configuration.md
CHANGED
|
@@ -87,7 +87,7 @@ git switch main
|
|
|
87
87
|
git pull
|
|
88
88
|
qamap manifest context .
|
|
89
89
|
qamap manifest init .
|
|
90
|
-
qamap manifest init services/
|
|
90
|
+
qamap manifest init services/listing --workspace-root .
|
|
91
91
|
qamap manifest validate .
|
|
92
92
|
qamap manifest explain . --base origin/main --head HEAD
|
|
93
93
|
```
|
|
@@ -107,10 +107,10 @@ $schema: https://raw.githubusercontent.com/IvoryCanvas/qamap/main/schema/qamap-m
|
|
|
107
107
|
version: 1
|
|
108
108
|
|
|
109
109
|
domains:
|
|
110
|
-
- id:
|
|
111
|
-
name:
|
|
110
|
+
- id: bundle
|
|
111
|
+
name: Bundle
|
|
112
112
|
paths:
|
|
113
|
-
- src/pages/
|
|
113
|
+
- src/pages/bundle/**
|
|
114
114
|
criticality: medium
|
|
115
115
|
source:
|
|
116
116
|
kind: inferred
|
|
@@ -119,25 +119,25 @@ domains:
|
|
|
119
119
|
- pages
|
|
120
120
|
|
|
121
121
|
flows:
|
|
122
|
-
- id:
|
|
123
|
-
domain:
|
|
124
|
-
name:
|
|
122
|
+
- id: bundle-submission-complete
|
|
123
|
+
domain: bundle
|
|
124
|
+
name: Bundle Submission Complete
|
|
125
125
|
entry:
|
|
126
|
-
route: /
|
|
126
|
+
route: /bundle/official/submissionComplete
|
|
127
127
|
source: inferred
|
|
128
128
|
runner: playwright
|
|
129
129
|
anchors:
|
|
130
130
|
- kind: route
|
|
131
|
-
path: src/pages/
|
|
132
|
-
route: /
|
|
131
|
+
path: src/pages/bundle/official/submissionComplete.tsx
|
|
132
|
+
route: /bundle/official/submissionComplete
|
|
133
133
|
source: inferred
|
|
134
134
|
confidence: high
|
|
135
135
|
checks:
|
|
136
136
|
- id: happy-path
|
|
137
|
-
title:
|
|
137
|
+
title: Bundle Submission Complete happy path works
|
|
138
138
|
type: success
|
|
139
139
|
- id: api-failure-fixture
|
|
140
|
-
title:
|
|
140
|
+
title: Bundle Submission Complete handles failed, empty, or unauthorized responses
|
|
141
141
|
type: failure
|
|
142
142
|
source:
|
|
143
143
|
kind: inferred
|
|
@@ -59,19 +59,19 @@ When a repository has `.qamap/manifest.yaml`, QAMap should explain why a recomme
|
|
|
59
59
|
```txt
|
|
60
60
|
Manifest recommendations: 3
|
|
61
61
|
|
|
62
|
-
|
|
62
|
+
Bundle Submission Complete `bundle-submission-complete`
|
|
63
63
|
- Kind: flow
|
|
64
64
|
- Confidence: high
|
|
65
|
-
- Why this was recommended: Changed files match anchors for the
|
|
65
|
+
- Why this was recommended: Changed files match anchors for the Bundle Submission Complete flow.
|
|
66
66
|
- Evidence sources: product-qa
|
|
67
|
-
- Manifest evidence: .qamap/manifest.yaml > flows.
|
|
68
|
-
- If this is wrong: update .qamap/manifest.yaml > flows.
|
|
67
|
+
- Manifest evidence: .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
68
|
+
- If this is wrong: update .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
69
69
|
- Next actions:
|
|
70
|
-
- Draft or review E2E coverage for the
|
|
71
|
-
- Cover the declared checks: Submit
|
|
70
|
+
- Draft or review E2E coverage for the Bundle Submission Complete flow.
|
|
71
|
+
- Cover the declared checks: Submit media link successfully; Show validation error for invalid media link.
|
|
72
72
|
- Repair hints:
|
|
73
|
-
- If these files do not belong to this flow, update .qamap/manifest.yaml > flows.
|
|
74
|
-
- If the recommended assertions feel vague, rewrite .qamap/manifest.yaml > flows.
|
|
73
|
+
- If these files do not belong to this flow, update .qamap/manifest.yaml > flows.bundle-submission-complete.anchors.
|
|
74
|
+
- If the recommended assertions feel vague, rewrite .qamap/manifest.yaml > flows.bundle-submission-complete.checks in team language.
|
|
75
75
|
```
|
|
76
76
|
|
|
77
77
|
This is the feedback loop: static analysis proposes a baseline, humans correct durable manifest entries, and future E2E recommendations become more specific without spending another LLM prompt on the same explanation.
|
|
@@ -93,26 +93,26 @@ Changed files: 1
|
|
|
93
93
|
Matches: 3
|
|
94
94
|
|
|
95
95
|
Matches:
|
|
96
|
-
-
|
|
97
|
-
Why: Changed files match anchors for the
|
|
98
|
-
Evidence: .qamap/manifest.yaml > flows.
|
|
99
|
-
If wrong: update .qamap/manifest.yaml > flows.
|
|
100
|
-
Checks: Submit
|
|
96
|
+
- Bundle Submission Complete (flow, high)
|
|
97
|
+
Why: Changed files match anchors for the Bundle Submission Complete flow.
|
|
98
|
+
Evidence: .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
99
|
+
If wrong: update .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
100
|
+
Checks: Submit media link successfully; Show validation error for invalid media link
|
|
101
101
|
```
|
|
102
102
|
|
|
103
103
|
When that flow includes an entry route and checks, `qamap e2e draft` promotes it ahead of heuristic drafts:
|
|
104
104
|
|
|
105
105
|
```ts
|
|
106
106
|
// Verification manifest evidence:
|
|
107
|
-
// - Flow:
|
|
108
|
-
// - Entry route: /
|
|
107
|
+
// - Flow: Bundle Submission Complete (bundle-submission-complete)
|
|
108
|
+
// - Entry route: /bundle/official/submissionComplete
|
|
109
109
|
// - Required checks:
|
|
110
|
-
// - [ ] Submit
|
|
111
|
-
// - [ ] Show validation error for invalid
|
|
110
|
+
// - [ ] Submit media link successfully
|
|
111
|
+
// - [ ] Show validation error for invalid media link
|
|
112
112
|
|
|
113
|
-
test("
|
|
114
|
-
await test.step("Open route /
|
|
115
|
-
await page.goto("/
|
|
113
|
+
test("Bundle Submission Complete", async ({ page }) => {
|
|
114
|
+
await test.step("Open route /bundle/official/submissionComplete.", async () => {
|
|
115
|
+
await page.goto("/bundle/official/submissionComplete");
|
|
116
116
|
});
|
|
117
117
|
});
|
|
118
118
|
```
|
|
@@ -245,42 +245,42 @@ appId: ${APP_ID}
|
|
|
245
245
|
When the changed file name exposes a narrower user action, the generated flow should use that action instead of stopping at a broad domain journey:
|
|
246
246
|
|
|
247
247
|
```txt
|
|
248
|
-
Changed file: src/features/
|
|
249
|
-
Domain term:
|
|
250
|
-
Generated scenario:
|
|
251
|
-
Generated draft path: .maestro/
|
|
248
|
+
Changed file: src/features/listing/components/MediaLinkSubmitModal.tsx
|
|
249
|
+
Domain term: Listing
|
|
250
|
+
Generated scenario: Listing Media Link Submit
|
|
251
|
+
Generated draft path: .maestro/listing-media-link-submit.yaml
|
|
252
252
|
```
|
|
253
253
|
|
|
254
254
|
The draft should then carry the same product-action wording into the runnable skeleton:
|
|
255
255
|
|
|
256
256
|
```yaml
|
|
257
|
-
# Flow:
|
|
258
|
-
# Domain scenario:
|
|
259
|
-
# Intent: Verify the changed "
|
|
257
|
+
# Flow: Listing Media Link Submit
|
|
258
|
+
# Domain scenario: Listing Media Link Submit
|
|
259
|
+
# Intent: Verify the changed "Media Link Submit" behavior inside Listing instead of stopping at a generic primary journey.
|
|
260
260
|
appId: ${APP_ID}
|
|
261
261
|
---
|
|
262
262
|
- launchApp
|
|
263
|
-
- tapOn: { id: "
|
|
263
|
+
- tapOn: { id: "listing-media-link-submit" }
|
|
264
264
|
```
|
|
265
265
|
|
|
266
266
|
## API / Service Contract
|
|
267
267
|
|
|
268
|
-
For backend changes such as `src/v1/
|
|
268
|
+
For backend changes such as `src/v1/listing/utils.ts`, QAMap should not invent a browser journey. It should infer the domain word and stay contract-focused:
|
|
269
269
|
|
|
270
270
|
```txt
|
|
271
271
|
Project: API / service
|
|
272
272
|
Recommended runner: Manual
|
|
273
273
|
|
|
274
|
-
Flow:
|
|
274
|
+
Flow: Listing API contract smoke checklist
|
|
275
275
|
Actor: API consumer or upstream service
|
|
276
|
-
Trigger: Call the endpoint, handler, or service path affected by src/v1/
|
|
276
|
+
Trigger: Call the endpoint, handler, or service path affected by src/v1/listing/utils.ts.
|
|
277
277
|
Success signal: the changed contract returns the expected status, response shape, auth behavior, and failure handling
|
|
278
278
|
```
|
|
279
279
|
|
|
280
280
|
The manual draft should stay actionable:
|
|
281
281
|
|
|
282
282
|
```md
|
|
283
|
-
#
|
|
283
|
+
# Listing API contract
|
|
284
284
|
|
|
285
285
|
## Steps
|
|
286
286
|
|
|
@@ -367,7 +367,7 @@ If reusable repo-local evidence already exists, the PR QA output should point at
|
|
|
367
367
|
|
|
368
368
|
```txt
|
|
369
369
|
Missing evidence before trusting this PR
|
|
370
|
-
- [recommended] fixture: Confirm fixture coverage - Reuse or extend existing fixture/mock evidence for this flow: src/services/
|
|
370
|
+
- [recommended] fixture: Confirm fixture coverage - Reuse or extend existing fixture/mock evidence for this flow: src/services/demoSeedService.ts, src/services/metricsMockService.ts.
|
|
371
371
|
```
|
|
372
372
|
|
|
373
373
|
```ts
|
|
@@ -423,24 +423,24 @@ Expected root-level behavior:
|
|
|
423
423
|
Changed App/Package Targets
|
|
424
424
|
|
|
425
425
|
| Target | Package | Project | Runner | Scoped Command |
|
|
426
|
-
| services/
|
|
426
|
+
| services/listing | listing | Web | Playwright | qamap e2e plan services/listing --workspace-root . --base main --head HEAD |
|
|
427
427
|
| apps/mobile | @acme/mobile | Expo / React Native | Maestro | qamap e2e plan apps/mobile --workspace-root . --base main --head HEAD |
|
|
428
428
|
```
|
|
429
429
|
|
|
430
430
|
For package scans, QAMap should use workspace policy without leaking workspace-root paths into package-local drafts:
|
|
431
431
|
|
|
432
432
|
```sh
|
|
433
|
-
qamap e2e plan services/
|
|
433
|
+
qamap e2e plan services/listing --workspace-root . --base main --head HEAD
|
|
434
434
|
```
|
|
435
435
|
|
|
436
436
|
Expected behavior:
|
|
437
437
|
|
|
438
438
|
```txt
|
|
439
439
|
Workspace root: .
|
|
440
|
-
Package root: services/
|
|
441
|
-
Matched core flow:
|
|
442
|
-
Changed files: src/features/
|
|
443
|
-
Generated draft path: docs/e2e/
|
|
440
|
+
Package root: services/listing
|
|
441
|
+
Matched core flow: Listing submit
|
|
442
|
+
Changed files: src/features/listing/submit.ts
|
|
443
|
+
Generated draft path: docs/e2e/listing-submit.md
|
|
444
444
|
```
|
|
445
445
|
|
|
446
446
|
The draft should mention package-local files:
|
|
@@ -448,7 +448,7 @@ The draft should mention package-local files:
|
|
|
448
448
|
```md
|
|
449
449
|
Related Changed Files
|
|
450
450
|
|
|
451
|
-
- `src/features/
|
|
451
|
+
- `src/features/listing/submit.ts`
|
|
452
452
|
```
|
|
453
453
|
|
|
454
454
|
## What Good Output Feels Like
|
package/docs/eval.md
CHANGED
|
@@ -12,7 +12,7 @@ For a complete PR-facing report that also includes QAMap review findings and sug
|
|
|
12
12
|
qamap verify . --base origin/main --head HEAD --pr-body-file pr-body.md
|
|
13
13
|
qamap eval . --base origin/main --head HEAD --format markdown
|
|
14
14
|
qamap eval . --base origin/main --head HEAD --pr-body-file pr-body.md
|
|
15
|
-
qamap eval services/
|
|
15
|
+
qamap eval services/listing --workspace-root . --base origin/main --head HEAD --include-working-tree
|
|
16
16
|
```
|
|
17
17
|
|
|
18
18
|
The GitHub Action can append the same report to the PR comment. On pull request events it reads the PR body from `GITHUB_EVENT_PATH`.
|
package/docs/github-action.md
CHANGED
|
@@ -40,7 +40,7 @@ jobs:
|
|
|
40
40
|
```
|
|
41
41
|
|
|
42
42
|
Use `fetch-depth: 0` so QAMap can compare the pull request branch with the base ref.
|
|
43
|
-
For production workflows, pin to a version tag such as `IvoryCanvas/qamap@v0.3.
|
|
43
|
+
For production workflows, pin to a version tag such as `IvoryCanvas/qamap@v0.3.2` after that tag is published. Use `@main` only when intentionally testing unreleased behavior.
|
|
44
44
|
|
|
45
45
|
## Monorepo Package
|
|
46
46
|
|
|
@@ -48,7 +48,7 @@ For production workflows, pin to a version tag such as `IvoryCanvas/qamap@v0.3.1
|
|
|
48
48
|
- uses: IvoryCanvas/qamap@main
|
|
49
49
|
with:
|
|
50
50
|
mode: review
|
|
51
|
-
path: services/
|
|
51
|
+
path: services/listing
|
|
52
52
|
workspace-root: .
|
|
53
53
|
base: ${{ github.event.pull_request.base.sha }}
|
|
54
54
|
head: HEAD
|
package/docs/manifest.md
CHANGED
|
@@ -208,10 +208,10 @@ $schema: https://raw.githubusercontent.com/IvoryCanvas/qamap/main/schema/qamap-m
|
|
|
208
208
|
version: 1
|
|
209
209
|
|
|
210
210
|
domains:
|
|
211
|
-
- id:
|
|
212
|
-
name:
|
|
211
|
+
- id: bundle
|
|
212
|
+
name: Bundle
|
|
213
213
|
paths:
|
|
214
|
-
- src/pages/
|
|
214
|
+
- src/pages/bundle/**
|
|
215
215
|
criticality: medium
|
|
216
216
|
source:
|
|
217
217
|
kind: declared
|
|
@@ -220,28 +220,28 @@ domains:
|
|
|
220
220
|
- product-qa
|
|
221
221
|
|
|
222
222
|
flows:
|
|
223
|
-
- id:
|
|
224
|
-
domain:
|
|
225
|
-
name:
|
|
223
|
+
- id: bundle-submission-complete
|
|
224
|
+
domain: bundle
|
|
225
|
+
name: Bundle Submission Complete
|
|
226
226
|
entry:
|
|
227
|
-
route: /
|
|
227
|
+
route: /bundle/official/submissionComplete
|
|
228
228
|
source: declared
|
|
229
229
|
runner: playwright
|
|
230
230
|
anchors:
|
|
231
231
|
- kind: route
|
|
232
|
-
path: src/pages/
|
|
233
|
-
route: /
|
|
232
|
+
path: src/pages/bundle/official/submissionComplete.tsx
|
|
233
|
+
route: /bundle/official/submissionComplete
|
|
234
234
|
source: declared
|
|
235
235
|
confidence: high
|
|
236
236
|
checks:
|
|
237
237
|
- id: happy-path
|
|
238
|
-
title: Submit
|
|
238
|
+
title: Submit media link successfully
|
|
239
239
|
type: success
|
|
240
|
-
selector: "[data-testid=
|
|
240
|
+
selector: "[data-testid=media-link-submit]"
|
|
241
241
|
- id: invalid-input
|
|
242
|
-
title: Show validation error for invalid
|
|
242
|
+
title: Show validation error for invalid media link
|
|
243
243
|
type: failure
|
|
244
|
-
selector: "[data-testid=
|
|
244
|
+
selector: "[data-testid=media-link-error]"
|
|
245
245
|
source:
|
|
246
246
|
kind: declared
|
|
247
247
|
confidence: high
|
|
@@ -1,4 +1,20 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Release Validation
|
|
2
|
+
|
|
3
|
+
## 0.3.2 - 2026-07-04
|
|
4
|
+
|
|
5
|
+
Validated before publishing `0.3.2` (reverse import graph, diff-anchored steps and names, workspace-member project detection, Python service classification, start-here CLI guide):
|
|
6
|
+
|
|
7
|
+
| Gate | Result |
|
|
8
|
+
| --- | --- |
|
|
9
|
+
| `pnpm test` | 100/100 passing |
|
|
10
|
+
| `pnpm scan` (self-scan) | 0 findings |
|
|
11
|
+
| Coverage thresholds (lines/branches/functions >= 80) | Passing |
|
|
12
|
+
| `pnpm bench` against four pinned local benchmark targets (web monorepo, Python API server, Expo app, legacy Vue app) | Runner choice 4/4, labeled must-reach recall 9/9, blank actions 0, no metric regressions vs the previous baseline |
|
|
13
|
+
| README demo | Real recorded run; the generated starter spec passes against the demo app |
|
|
14
|
+
|
|
15
|
+
Historical validation notes for earlier releases follow below.
|
|
16
|
+
|
|
17
|
+
# 0.2.1 Patch Release Validation (historical)
|
|
2
18
|
|
|
3
19
|
QAMap should not publish a patch or minor version only because the CLI commands work in fixtures. The `0.2.1` release should prove that the manifest-backed QA skill flow is useful across representative repositories without requiring an LLM call.
|
|
4
20
|
|
|
@@ -148,7 +164,7 @@ The public demo must prove the product shape, not only command execution. Before
|
|
|
148
164
|
- Remaining gaps are specific, such as auth fixture, API mock, stable selector, runner config, or validation command.
|
|
149
165
|
- The demo makes clear when output is `--dry-run`, `review-only`, `near-runnable`, or `runnable-candidate`.
|
|
150
166
|
|
|
151
|
-
Avoid demos that only say broad phrases such as "fixture needed", "selector missing", or "
|
|
167
|
+
Avoid demos that only say broad phrases such as "fixture needed", "selector missing", or "Listing flow recommended" without showing why those gaps matter for the changed behavior.
|
|
152
168
|
|
|
153
169
|
## Current Fixture Evidence Matrix
|
|
154
170
|
|
|
@@ -160,8 +176,8 @@ The matrix below is public, fixture-backed evidence from the repository test sui
|
|
|
160
176
|
| Packaged PR QA skill template | `package metadata includes the portable PR QA skill template` | npm package metadata includes `skills`, and `skills/qamap-pr-qa/SKILL.md` contains the local PR QA workflow, `qamap qa` command, and manifest repair guidance. |
|
|
161
177
|
| Verification manifest loop | `manifest init creates a baseline verification manifest`; `manifest init keeps Expo app file domains specific`; `manifest init captures advisory instruction context`; `manifest bootstrap produces concrete PR E2E draft from repo QA memory`; `e2e draft can use an external verification manifest for read-only adoption preview`; `manifest matches explain e2e and verify recommendations`; `manifest validate reports missing and stale manifest policy` | Generated `.qamap/manifest.yaml` includes `$schema`, domains, flows, anchors, checks, runner, source, and confidence; context preview reports repo-local instruction sources, role summaries, validation commands, safety rules, and diagnostics; validator reports missing/stale/duplicate policy; explain output maps branch changes to manifest domains/flows/checks; E2E drafts prefer `verification-manifest` sources with manifest evidence, route entry, detected input/action selectors, required checks, and manifest repair paths; external manifests can be passed with `--manifest` for read-only adoption smoke tests. |
|
|
162
178
|
| Web app with Playwright routes | `generateE2ePlan matches committed core flow definitions`; `generateE2eDraft uses web selectors in Playwright specs`; `generateE2eDraft dry run previews files without writing drafts`; `generateE2eDraft asserts changed HTML success copy in Playwright specs`; `generateE2ePlan captures Playwright execution profile and self-check blockers`; `generateE2ePlan infers Playwright base URLs from dev scripts`; `generateE2eDraft supports Next app router route groups and concrete route hints`; `generateE2ePlan reads React Router object route paths`; `generateE2eDraft fills dynamic route params from concrete route hints`; `generateE2eDraft emits runnable Playwright role and input actions` | `Web` project profile, `playwright` runner, core-flow names such as `Checkout purchase`, route-aware Playwright drafts, dry-run preview status without filesystem writes, stable selector hints, changed HTML success copy assertions, execution profile, dev-script base URL hints, opt-in Playwright setup proposal, Next App Router route groups, React Router object paths, dynamic route params, draft self-check status, action items, and validation gaps. |
|
|
163
|
-
| Expo / React Native mobile app | `generateE2ePlan recommends mobile flows for Expo changes`; `generateE2ePlan detects Maestro app ids from app config files`; `generateE2eDraft scopes entrypoint hints to each domain scenario`; `generateE2eDraft names changed component actions before generic primary journeys` | `Expo / React Native` project profile, `maestro` runner, app id and launch command hints from `app.json` or `app.config.*`, Maestro YAML drafts, `testID`/`accessibilityLabel` selector hints, action-specific scenario names such as `
|
|
164
|
-
| API or backend service | `generateE2ePlan detects API service projects and suggests contract checklists`; `generateE2ePlan detects Django service apps from a workspace root`; `generateE2ePlan names versioned API service paths with domain language`; `generateE2ePlan uses matched core flow names for API service contracts` | `API / service` project profile, manual contract checklist, Django/FastAPI-style service signals when present, domain-aware titles such as `
|
|
179
|
+
| Expo / React Native mobile app | `generateE2ePlan recommends mobile flows for Expo changes`; `generateE2ePlan detects Maestro app ids from app config files`; `generateE2eDraft scopes entrypoint hints to each domain scenario`; `generateE2eDraft names changed component actions before generic primary journeys` | `Expo / React Native` project profile, `maestro` runner, app id and launch command hints from `app.json` or `app.config.*`, Maestro YAML drafts, `testID`/`accessibilityLabel` selector hints, action-specific scenario names such as `Listing Media Link Submit`, and mobile setup actions. |
|
|
180
|
+
| API or backend service | `generateE2ePlan detects API service projects and suggests contract checklists`; `generateE2ePlan detects Django service apps from a workspace root`; `generateE2ePlan names versioned API service paths with domain language`; `generateE2ePlan uses matched core flow names for API service contracts` | `API / service` project profile, manual contract checklist, Django/FastAPI-style service signals when present, domain-aware titles such as `Listing API contract`, API consumer actor, endpoint/handler/service-path trigger, service start/test command hints, and contract failure coverage. |
|
|
165
181
|
| CLI package | `generateE2ePlan detects CLI packages and suggests command verification checklists` | `CLI` project profile from `package.json` bin entries, manual command verification checklist, CLI user or maintainer actor language, command invocation trigger, stdout/stderr/generated-file/exit-code success signal, valid and invalid argument coverage, and no required API fixture action unless the changed command path explicitly exposes network or fixture evidence. |
|
|
166
182
|
| Design tokens and data catalogs | `generateE2ePlan detects design token packages and suggests artifact validation`; `generateE2ePlan detects data catalog repositories and suggests catalog verification` | `Design tokens` and `Data catalog` project profiles, manual artifact/catalog checklist, token or catalog actor language, schema/generated output/consumer fixture coverage, fixture readiness marked not needed for API mocks, and validation matrix rows that do not require browser/device selectors. |
|
|
167
183
|
| Monorepo root and package targeting | `generateE2ePlan surfaces package-scoped targets for monorepo root changes`; `generateE2ePlan matches workspace core flows for package scans`; `generateTestPlan scopes monorepo changes to the requested package` | Root plans list changed app/package targets with package names, project type, runner, and scoped commands; package scans keep package-local changed files, workspace-level `.qamap/flows.yml` matches, package-local generated drafts, and no leaked workspace path prefixes in package drafts. |
|
|
@@ -194,7 +210,7 @@ The same follow-up also showed the next quality gap. External manifest previews
|
|
|
194
210
|
|
|
195
211
|
| Target shape | Base/head mode | Result | Follow-up signal |
|
|
196
212
|
| --- | --- | --- | --- |
|
|
197
|
-
| Expo / React Native manifest baseline | `manifest init` wrote only to `/tmp/qamap-
|
|
213
|
+
| Expo / React Native manifest baseline | `manifest init` wrote only to `/tmp/qamap-journal-note-manifest.yaml` | Generated 9 domains, 8 flows, 8 anchors, and 16 checks without changing the target repo. Direct `app/*.tsx` screens now produce specific paths such as `app/SentimentChatPage.tsx`; `+not-found.tsx` is not promoted as a product domain. | Good 0.2.x signal for baseline quality. Remaining work is richer screen/route semantics and selector-specific checks. |
|
|
198
214
|
| Web monorepo package | Package scan with `--workspace-root`, feature branch compared with `main`, working tree included | Detected `web`, recommended Playwright, inferred a concrete route, produced changed-flow specs, and correctly blocked promotion because Playwright config, deterministic fixture/mock data, selector evidence, and validation evidence were missing. | Flow naming improved, but docs/design-only files can still create low-signal drafts that should be filtered or demoted. |
|
|
199
215
|
| Expo / React Native app | Feature branch compared with `develop`, working tree included | Detected `expo-react-native`, recommended Maestro, found an existing Jest suite, produced multiple near-runnable YAML drafts, and gave useful blockers for missing Maestro directory plus missing stable mobile selectors. | Strongest current real-repo result; remaining gap is selector and app-specific setup quality. |
|
|
200
216
|
| Nuxt / Vue web app with existing Playwright tests | `origin/develop` to `HEAD`, working tree included | Detected `web`, recommended Playwright, recognized existing test evidence, and blocked draft promotion because no Playwright config or runnable route/screen entrypoint was inferred. | Test-only or generated-test changes can still produce generic smoke draft names; QAMap should better distinguish changed tests from changed product behavior. |
|
package/docs/roadmap.md
CHANGED
|
@@ -45,7 +45,6 @@ Before treating the next public release as ready, the golden demo must satisfy t
|
|
|
45
45
|
- Improve `doctor` output with clearer scoring and remediation grouping.
|
|
46
46
|
- Improve `review` output for changed-line locations.
|
|
47
47
|
- Expand manifest support with richer anchors, symbol-level matching, and configurable taste rubrics.
|
|
48
|
-
- Match changed files to flows through a reverse import graph, not only declared anchor paths, so shared component and hook changes map to the flows that depend on them.
|
|
49
48
|
- Map changed symbols to manifest anchors after the path/route baseline is stable.
|
|
50
49
|
- Keep the `--format agent` output a stable, versioned contract that skills and MCP wrappers can rely on.
|
|
51
50
|
- Add language-specific domain patterns for backend services, CLIs, libraries, mobile apps, and infrastructure repositories.
|
package/docs/verify.md
CHANGED
|
@@ -24,7 +24,7 @@ For custom stacks or team-specific flows, add `validationCommands` to `qamap.con
|
|
|
24
24
|
```sh
|
|
25
25
|
qamap verify . --base origin/main --head HEAD --format markdown
|
|
26
26
|
qamap verify . --base origin/main --head HEAD --pr-body-file pr-body.md
|
|
27
|
-
qamap verify services/
|
|
27
|
+
qamap verify services/listing --workspace-root . --base origin/main --head HEAD --include-working-tree
|
|
28
28
|
```
|
|
29
29
|
|
|
30
30
|
Use `--fail-on high` or `--fail-on medium` to fail on QAMap review findings at or above a severity threshold. Readiness and manifest recommendations stay advisory until the team decides to make them required PR evidence.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ivorycanvas/qamap",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.2",
|
|
4
4
|
"description": "Local-first QA skill that turns PR diffs into affected flows, missing evidence, and E2E drafts with no cloud or LLM token.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"packageManager": "pnpm@10.32.1",
|
|
@@ -24,7 +24,8 @@
|
|
|
24
24
|
"coverage": "pnpm build && node --test --experimental-test-coverage --test-coverage-include=\"dist/**/*.js\" --test-coverage-lines=80 --test-coverage-branches=80 --test-coverage-functions=80 test/*.test.mjs",
|
|
25
25
|
"scan": "pnpm build && node dist/cli.js scan .",
|
|
26
26
|
"report": "pnpm build && node dist/cli.js report . --output QAMAP_REPORT.md",
|
|
27
|
-
"release:check": "pnpm test && pnpm scan && git diff --check && pnpm run coverage && pnpm pack --dry-run"
|
|
27
|
+
"release:check": "pnpm test && pnpm scan && git diff --check && pnpm run coverage && pnpm pack --dry-run",
|
|
28
|
+
"bench": "pnpm build && node scripts/bench.mjs"
|
|
28
29
|
},
|
|
29
30
|
"keywords": [
|
|
30
31
|
"ai",
|