@ivorycanvas/qamap 0.3.3 → 0.3.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +38 -0
- package/README.md +44 -578
- package/dist/agent-init.d.ts +16 -0
- package/dist/agent-init.js +110 -0
- package/dist/agent-init.js.map +1 -0
- package/dist/cli.js +17 -1
- package/dist/cli.js.map +1 -1
- package/dist/context.d.ts +1 -0
- package/dist/context.js +4 -0
- package/dist/context.js.map +1 -1
- package/dist/domain-language.js +48 -5
- package/dist/domain-language.js.map +1 -1
- package/dist/e2e.d.ts +4 -0
- package/dist/e2e.js +342 -73
- package/dist/e2e.js.map +1 -1
- package/dist/fixture-insight.d.ts +8 -0
- package/dist/fixture-insight.js +193 -0
- package/dist/fixture-insight.js.map +1 -0
- package/dist/fs.js +11 -0
- package/dist/fs.js.map +1 -1
- package/dist/index.d.ts +2 -1
- package/dist/index.js +2 -1
- package/dist/index.js.map +1 -1
- package/dist/manifest.d.ts +5 -0
- package/dist/manifest.js +478 -54
- package/dist/manifest.js.map +1 -1
- package/dist/qa.d.ts +4 -0
- package/dist/qa.js +125 -18
- package/dist/qa.js.map +1 -1
- package/dist/terminal.d.ts +2 -0
- package/dist/terminal.js +51 -0
- package/dist/terminal.js.map +1 -0
- package/dist/test-plan.js +56 -6
- package/dist/test-plan.js.map +1 -1
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/docs/adoption.md +62 -0
- package/docs/agent-format.md +52 -0
- package/docs/agent-skill.md +17 -1
- package/docs/benchmarking.md +60 -20
- package/docs/commands.md +242 -0
- package/docs/configuration.md +11 -0
- package/docs/e2e-output-examples.md +8 -4
- package/docs/guardrails.md +64 -0
- package/docs/manifest.md +3 -3
- package/docs/quickstart-demo.md +1 -1
- package/docs/release-validation.md +18 -4
- package/docs/releasing.md +13 -10
- package/docs/roadmap.md +9 -14
- package/package.json +4 -3
- package/schema/qamap-agent.schema.json +171 -0
- package/skills/qamap-pr-qa/SKILL.md +1 -0
package/README.md
CHANGED
|
@@ -1,245 +1,70 @@
|
|
|
1
1
|
# QAMap
|
|
2
2
|
|
|
3
3
|
[](https://github.com/IvoryCanvas/qamap/actions/workflows/ci.yml)
|
|
4
|
-
[](https://www.npmjs.com/package/@ivorycanvas/qamap)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
7
|
**A local-first QA skill for AI-generated PRs. QAMap turns a PR diff into affected flows, missing evidence, and E2E drafts. No cloud. No LLM token.**
|
|
8
8
|
|
|
9
|
-
QAMap
|
|
10
|
-
|
|
11
|
-
It is built for the moment when a reviewer asks: "This PR looks plausible, but which user flow could it break, and what should we test before merge?"
|
|
12
|
-
|
|
13
|
-
QAMap does not call an LLM API, upload source code, or require a service account. It runs in the repository you already have.
|
|
14
|
-
|
|
15
|
-
The core loop is intentionally simple:
|
|
9
|
+
QAMap reads git changes, project structure, runner signals, selectors, and optional repo QA memory, then answers the question every reviewer asks: *"This PR looks plausible — which user flow could it break, and what should we verify before merge?"* It focuses on the judgment step ("what deserves testing"), not on competing with LLMs at writing test code.
|
|
16
10
|
|
|
17
11
|
```txt
|
|
18
12
|
PR diff
|
|
19
13
|
-> qamap qa
|
|
14
|
+
-> affected flows + missing evidence + PR checklist + E2E starter draft
|
|
20
15
|
|
|
21
|
-
|
|
22
|
-
->
|
|
23
|
-
|
|
24
|
-
Optional team memory
|
|
25
|
-
-> .qamap/manifest.yaml
|
|
26
|
-
-> better future PR recommendations
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
## Install & Quick Start
|
|
30
|
-
|
|
31
|
-
Requires Node.js 20 or newer.
|
|
32
|
-
|
|
33
|
-
Run QAMap once without adding a dependency. Inside a repository whose default branch is `origin/main` (or `main`), the base is inferred automatically:
|
|
34
|
-
|
|
35
|
-
```sh
|
|
36
|
-
pnpm dlx @ivorycanvas/qamap qa
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
Pass `--base <ref> --head <ref>` for anything non-standard, and run bare `qamap` any time to see the start-here guide (`qamap help` prints the full reference).
|
|
40
|
-
|
|
41
|
-
That first command is intentionally manifest-free. It previews a PR comment/checklist that names the affected flow, recommended runner, draft file, missing fixture/selector/assertion evidence, and validation command.
|
|
42
|
-
|
|
43
|
-
Handing the result to a coding agent instead of a human? Use the compact agent format (see [For Coding Agents](#for-coding-agents)):
|
|
44
|
-
|
|
45
|
-
```sh
|
|
46
|
-
pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD --format agent
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
Install QAMap in a repository when you want a repeatable project command:
|
|
50
|
-
|
|
51
|
-
```sh
|
|
52
|
-
pnpm add -D @ivorycanvas/qamap
|
|
53
|
-
pnpm exec qamap qa . --base origin/main --head HEAD
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
Generate a Markdown artifact that an agent or reviewer can paste into a PR:
|
|
57
|
-
|
|
58
|
-
```sh
|
|
59
|
-
pnpm exec qamap qa . --base origin/main --head HEAD --output QAMAP_QA.md
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
When you are ready to create actual draft test files instead of a PR comment preview:
|
|
63
|
-
|
|
64
|
-
```sh
|
|
65
|
-
pnpm exec qamap e2e draft . --base origin/main --head HEAD --dry-run
|
|
66
|
-
pnpm exec qamap e2e draft . --base origin/main --head HEAD
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
Optional accuracy upgrade: create repo-local QA memory from the default branch and review it. Matching is anchor-path based today — a flow claims a PR only when changed files or routes hit its listed anchors — so add shared components to a flow's `anchors` when changes to them should map to that flow:
|
|
70
|
-
|
|
71
|
-
```sh
|
|
72
|
-
git switch main
|
|
73
|
-
pnpm exec qamap manifest context .
|
|
74
|
-
pnpm exec qamap manifest init .
|
|
75
|
-
git add .qamap/manifest.yaml
|
|
76
|
-
git commit -m "Add QAMap verification manifest"
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
Preview adoption without writing a manifest into the target repository:
|
|
80
|
-
|
|
81
|
-
```sh
|
|
82
|
-
pnpm exec qamap manifest init . --write /tmp/qamap-manifest.yaml
|
|
83
|
-
pnpm exec qamap qa . --manifest /tmp/qamap-manifest.yaml --base origin/main --head HEAD
|
|
16
|
+
Optional team memory (.qamap/manifest.yaml)
|
|
17
|
+
-> sharper recommendations on every future PR
|
|
84
18
|
```
|
|
85
19
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
```sh
|
|
89
|
-
pnpm exec qamap scan .
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
## 30-Second PR Demo
|
|
20
|
+
## 30-Second Demo
|
|
93
21
|
|
|
94
22
|

|
|
95
23
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
Try the same loop on your own branch:
|
|
99
|
-
|
|
100
|
-
```sh
|
|
101
|
-
pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
QAMap reads the changed files and project signals:
|
|
105
|
-
|
|
106
|
-
```txt
|
|
107
|
-
Input
|
|
108
|
-
- git diff: origin/main...HEAD
|
|
109
|
-
- project structure: package.json, routes, test config, selectors
|
|
110
|
-
- optional team context: .qamap/manifest.yaml, CONTEXT.md, ADRs, goals, QA runbooks
|
|
111
|
-
```
|
|
24
|
+
A real, unedited recording on a Next.js app with **zero committed tests**: `qamap qa` names the affected flow, `qamap e2e setup` writes the Playwright config and a starter spec, and `npm run test:e2e` finishes with `1 passed`. First-run assertions are smoke checks — the point is a runnable starting point, not finished coverage.
|
|
112
25
|
|
|
113
|
-
|
|
26
|
+
Every report now opens with the verdict (trimmed real output):
|
|
114
27
|
|
|
115
28
|
```txt
|
|
116
|
-
|
|
117
|
-
- PR comment/checklist draft for this branch
|
|
118
|
-
- changed domain language and candidate user flows
|
|
119
|
-
- recommended E2E runner or manual checklist
|
|
120
|
-
- draft Playwright, Maestro, CLI, API, or manual test files
|
|
121
|
-
- readiness status: runnable-candidate, near-runnable, or review-only
|
|
122
|
-
- blockers such as missing runner config, selectors, fixtures, or assertions
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
Trimmed real `qamap qa` output for a small Next.js notes-page change in a repository with no tests and no manifest (full output also lists validation gaps and a PR checklist):
|
|
126
|
-
|
|
127
|
-
```txt
|
|
128
|
-
# QAMap QA Draft
|
|
129
|
-
|
|
130
|
-
> Local-first PR QA skill output. No cloud. No LLM token. Manifest is optional, not required for first use.
|
|
29
|
+
## At a Glance
|
|
131
30
|
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
-
|
|
135
|
-
-
|
|
136
|
-
-
|
|
137
|
-
-
|
|
138
|
-
- Draft flows: 1
|
|
139
|
-
|
|
140
|
-
## First E2E Draft Bootstrap
|
|
141
|
-
|
|
142
|
-
QAMap did not find committed test files for this target. The next step is to
|
|
143
|
-
create the first runnable starter draft, not to stop at a checklist.
|
|
144
|
-
|
|
145
|
-
- Recommended first runner: Playwright
|
|
146
|
-
- Create command: `qamap e2e setup . --runner playwright`
|
|
147
|
-
- Install command: `npm install -D @playwright/test`
|
|
148
|
-
- Draft files QAMap can create:
|
|
149
|
-
- `playwright.config.ts`
|
|
150
|
-
- `tests/e2e/notes-primary-journey.spec.ts`
|
|
151
|
-
|
|
152
|
-
## PR Comment Draft
|
|
153
|
-
|
|
154
|
-
### Affected Flow
|
|
155
|
-
|
|
156
|
-
- Notes primary journey (domain-language)
|
|
157
|
-
- User journey: User -> Open route /notes. -> Protect Notes primary journey
|
|
158
|
-
by fill Write A Note with realistic data.
|
|
159
|
-
- Changed files: `app/notes/page.tsx`
|
|
160
|
-
- Why: Primary entrypoint inferred as route /notes [high] (app/notes/page.tsx).
|
|
31
|
+
- Affected behavior: Checkout Submit
|
|
32
|
+
- Verify before merge: ... this outcome is verified: visible text "Order confirmed" appears?
|
|
33
|
+
- Evidence found: changed file src/pages/checkout/index.tsx; route: /checkout (high); web-test-id: checkout-submit (...)
|
|
34
|
+
- Proposed draft: `tests/e2e/checkout-submit.spec.ts` (near runnable)
|
|
35
|
+
- Next command: `qamap e2e setup . --runner playwright`
|
|
36
|
+
- Missing before trust: Add deterministic fixture or mock data for /api/checkout (...)
|
|
161
37
|
```
|
|
162
38
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
```ts
|
|
166
|
-
await test.step("Open route /notes.", async () => {
|
|
167
|
-
await page.goto("/notes");
|
|
168
|
-
});
|
|
169
|
-
|
|
170
|
-
await test.step("Fill Write A Note with realistic data.", async () => {
|
|
171
|
-
await page.getByPlaceholder("Write a note").fill("QAMap sample value");
|
|
172
|
-
});
|
|
39
|
+
## Install & Quick Start
|
|
173
40
|
|
|
174
|
-
|
|
175
|
-
await page.getByTestId("add-note").click();
|
|
176
|
-
});
|
|
41
|
+
Requires Node.js 20 or newer. Inside a repository whose default branch is `origin/main` (or `main`), the base is inferred automatically:
|
|
177
42
|
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
43
|
+
```sh
|
|
44
|
+
pnpm dlx @ivorycanvas/qamap qa # what should this branch prove before merge?
|
|
45
|
+
pnpm dlx @ivorycanvas/qamap e2e setup . --runner playwright # no tests yet? create config + starter spec
|
|
46
|
+
pnpm dlx @ivorycanvas/qamap manifest init # optional: save reviewed team QA memory for sharper future runs
|
|
181
47
|
```
|
|
182
48
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
See [docs/quickstart-demo.md](docs/quickstart-demo.md) for a compact walkthrough, [docs/agent-skill.md](docs/agent-skill.md) for agent handoff usage, [docs/manifest.md](docs/manifest.md) for the verification manifest loop, and [docs/e2e-output-examples.md](docs/e2e-output-examples.md) for more output shapes.
|
|
49
|
+
Pass `--base <ref> --head <ref>` for anything non-standard. Run bare `qamap` for a start-here guide, `qamap help` for the full reference, and see the [command reference](docs/commands.md) for every command and output shape.
|
|
186
50
|
|
|
187
51
|
## For Coding Agents
|
|
188
52
|
|
|
189
|
-
Stop re-explaining the same QA context to your agent on every PR
|
|
53
|
+
Stop re-explaining the same QA context to your agent on every PR:
|
|
190
54
|
|
|
191
55
|
```sh
|
|
192
|
-
qamap qa
|
|
56
|
+
qamap qa --format agent
|
|
193
57
|
```
|
|
194
58
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
Three integration points ship with the package:
|
|
198
|
-
|
|
199
|
-
- `skills/qamap-pr-qa/SKILL.md` — a portable agent skill template: any local agent workflow that reads reusable instructions can run QAMap as its final QA pass before PR handoff.
|
|
200
|
-
- `qamap context . --write AGENTS.md` — generated agent instructions now include a Pre-PR QA section, so agents working in the repo learn to run `qamap qa` on their own.
|
|
201
|
-
- `--output` — write the qa report to a file an agent or reviewer can paste into the PR body.
|
|
202
|
-
|
|
203
|
-
Treat QAMap output as QA planning evidence, not proof that browser, device, or manual QA passed — the output says so itself.
|
|
204
|
-
|
|
205
|
-
## Why This Is Different
|
|
206
|
-
|
|
207
|
-
Recorders such as browser or mobile test studios are useful when you already know the flow to exercise. QAMap starts one step earlier: it asks what the PR changed, which repo-owned QA memory applies, and what test artifact should exist before merge.
|
|
208
|
-
|
|
209
|
-
A good QAMap result should answer:
|
|
210
|
-
|
|
211
|
-
- which product flow changed
|
|
212
|
-
- which manifest domain, flow, and checks caused the recommendation
|
|
213
|
-
- which draft test file was generated or previewed
|
|
214
|
-
- which success, failure, edge, contract, or visual cases the draft covers
|
|
215
|
-
- which selector, fixture, auth, runner, or validation gaps still block trusted regression evidence
|
|
216
|
-
- which manifest path to edit when the recommendation is wrong
|
|
217
|
-
|
|
218
|
-
That is the product bet: one human correction to the repo-local manifest should improve future PR recommendations without another LLM prompt.
|
|
219
|
-
|
|
220
|
-
You do not need a manifest to start. Without one, QAMap uses the PR diff, package signals, routes, selectors, runner config, and existing tests. Add a manifest only when the team wants durable QA language that improves future recommendations.
|
|
221
|
-
|
|
222
|
-
## What QAMap Is For
|
|
59
|
+
One minified JSON object (`schema: qamap.qa`, ~2 KB for a small PR) with affected flows, draft paths, required evidence, and the PR checklist — the decision content of the full report at a fraction of the context cost. The shape is a documented, versioned contract: [agent format contract](docs/agent-format.md). To make agents run this themselves, run `qamap init --agent` once: it adds a Pre-PR QA section to `AGENTS.md` and installs the packaged skill ([skills/qamap-pr-qa/SKILL.md](skills/qamap-pr-qa/SKILL.md)) into `.claude/skills/`. Details: [agent skill guide](docs/agent-skill.md).
|
|
223
60
|
|
|
224
|
-
|
|
61
|
+
## Why QAMap
|
|
225
62
|
|
|
226
|
-
-
|
|
227
|
-
-
|
|
228
|
-
-
|
|
229
|
-
- verification-focused: it tells reviewers what evidence is missing, not how to style code
|
|
230
|
-
- PR QA skill output: `qamap qa` turns a branch into a PR-ready affected-flow summary, suggested E2E/checklist draft, missing evidence list, and copyable checklist
|
|
231
|
-
- packaged agent skill: `skills/qamap-pr-qa/SKILL.md` gives coding agents a compact PR QA workflow for running QAMap before handoff
|
|
232
|
-
- domain-aware E2E drafting: it turns branch changes into flow language, draft specs, readiness summaries, and action items
|
|
233
|
-
- repo-local verification base: shared manifests can be committed, while generated run history stays ignored by default
|
|
234
|
-
- context-aware baseline generation: manifest init can use repo-local context, ADRs, goals, agent instructions, harness files, skills, and runbooks as advisory bootstrap signals
|
|
235
|
-
- harness/skill role hints: instruction-derived context is classified as agent skill, harness config, workflow lifecycle, verification rubric, safety policy, release policy, or test runner context
|
|
236
|
-
- ecosystem-aware: it suggests validation commands for JavaScript/TypeScript, Python, Go, Rust, Gradle, and Maven projects
|
|
237
|
-
- CI-friendly: text, JSON, Markdown, and SARIF output are supported
|
|
238
|
-
- explainable: every finding includes a concrete fix
|
|
63
|
+
- **Judgment first, generation second.** LLMs write test code well; deciding *what deserves testing* for a given diff is the missing layer. QAMap fills it statically, deterministically, for free.
|
|
64
|
+
- **The repo remembers.** Team QA knowledge lives in `.qamap/manifest.yaml`, reviewed once and reused on every PR — instead of re-prompting an agent each session.
|
|
65
|
+
- **Honest output.** Drafts state what blocks them from being trusted; changed endpoints are observed, never mocked away; generated specs never assert what cannot pass. Configuration, docs, generated artifacts, and changed tests stay in verification mode instead of fabricating a product-journey E2E.
|
|
239
66
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
For PR verification, QAMap treats the repository itself as the working base: committed manifests hold durable team language, ignored local history holds generated run observations, and the current branch diff supplies what changed now.
|
|
67
|
+
Positioning against recorders, LLM test generation, and impact-analysis tools: [where QAMap fits](docs/adoption.md#where-qamap-fits).
|
|
243
68
|
|
|
244
69
|
<details>
|
|
245
70
|
<summary>한국어 소개</summary>
|
|
@@ -258,381 +83,22 @@ pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD
|
|
|
258
83
|
|
|
259
84
|
</details>
|
|
260
85
|
|
|
261
|
-
##
|
|
262
|
-
|
|
263
|
-
```sh
|
|
264
|
-
pnpm exec qamap qa . --base origin/main --head HEAD
|
|
265
|
-
pnpm exec qamap qa . --base origin/main --head HEAD --format agent
|
|
266
|
-
pnpm exec qamap qa . --manifest /tmp/qamap-manifest.yaml --base origin/main --head HEAD --output QAMAP_QA.md
|
|
267
|
-
pnpm exec qamap scan .
|
|
268
|
-
pnpm exec qamap verify . --base origin/main --head HEAD --pr-body-file pr-body.md
|
|
269
|
-
pnpm exec qamap manifest context .
|
|
270
|
-
pnpm exec qamap manifest init .
|
|
271
|
-
pnpm exec qamap manifest validate .
|
|
272
|
-
pnpm exec qamap manifest explain . --base origin/main --head HEAD
|
|
273
|
-
pnpm exec qamap e2e draft . --base origin/main --head HEAD --dry-run
|
|
274
|
-
```
|
|
275
|
-
|
|
276
|
-
Use `pnpm dlx @ivorycanvas/qamap ...` for one-off runs without installing QAMap into the target repository.
|
|
277
|
-
|
|
278
|
-
## Repository Guardrails (Secondary Layer)
|
|
279
|
-
|
|
280
|
-
The QA draft loop above is the core product. QAMap also ships a static guardrails scanner as a secondary layer for repositories that let agents work broadly: it checks agent instructions, MCP configs, committed env files, risky package scripts, workflow permissions, and validation signals before agent work becomes review churn.
|
|
281
|
-
|
|
282
|
-
For a repository baseline before broad agent use, run:
|
|
283
|
-
|
|
284
|
-
```sh
|
|
285
|
-
qamap scan .
|
|
286
|
-
```
|
|
86
|
+
## Documentation
|
|
287
87
|
|
|
288
|
-
|
|
289
|
-
Findings: 6 (high: 3, medium: 2, low: 1, info: 0)
|
|
290
|
-
|
|
291
|
-
HIGH
|
|
292
|
-
- QM003 Suspicious agent instruction text (AGENTS.md)
|
|
293
|
-
Fix: Remove untrusted instruction text or move examples into clearly fenced documentation.
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
See the rule table below and [docs/rules.md](docs/rules.md) for what the scanner checks.
|
|
297
|
-
|
|
298
|
-
When developing QAMap from source:
|
|
299
|
-
|
|
300
|
-
```sh
|
|
301
|
-
git clone https://github.com/IvoryCanvas/qamap.git
|
|
302
|
-
cd qamap
|
|
303
|
-
pnpm install
|
|
304
|
-
pnpm build
|
|
305
|
-
node dist/cli.js scan /path/to/repo
|
|
306
|
-
```
|
|
307
|
-
|
|
308
|
-
QAMap is a local-first PR verification planner with a repository-level verification manifest loop, not a finished automatic QA bot. A good result is a clear answer to "what should this branch prove before merge?", plus manifest-backed E2E, fixture, selector, and validation work that a developer can turn into real regression coverage. Many first drafts will correctly report `review-only` or `near-runnable` until the project adds runner config, stable selectors, deterministic fixtures, or team-owned manifest entries.
|
|
309
|
-
|
|
310
|
-
## What QAMap Produces
|
|
311
|
-
|
|
312
|
-
On a changed branch, QAMap tries to produce reviewable verification artifacts instead of only saying "write more tests":
|
|
313
|
-
|
|
314
|
-
- a branch-aware verification plan that names the changed domain, actor, trigger, goal, success signal, and edge cases
|
|
315
|
-
- draft Playwright, Maestro, CLI command, or manual checklist files when the repository shape supports them
|
|
316
|
-
- a repo-level verification manifest loop where humans correct durable flows once and later PRs get sharper route/check/test draft suggestions
|
|
317
|
-
- a runner setup proposal that explains why Playwright or Maestro fits the changed surface and which files/commands would be created if the team accepts it
|
|
318
|
-
- readiness evidence that explains missing runner config, selectors, fixture data, assertions, validation commands, or flow manifests
|
|
319
|
-
- repo-local suggestions for `.qamap/domains.yml`, `.qamap/flows.yml`, and ignored `.qamap/runs/` history so teams can improve the next run without spending LLM tokens
|
|
320
|
-
|
|
321
|
-
That means QAMap is most valuable when it becomes the team's verification base: humans define the durable language and critical flows once, QAMap reuses that base on each PR, and generated observations stay local unless the team intentionally promotes them into shared policy.
|
|
322
|
-
|
|
323
|
-
## Commands
|
|
324
|
-
|
|
325
|
-
| Command | Purpose |
|
|
88
|
+
| Guide | What it covers |
|
|
326
89
|
| --- | --- |
|
|
327
|
-
|
|
|
328
|
-
|
|
|
329
|
-
|
|
|
330
|
-
|
|
|
331
|
-
|
|
|
332
|
-
|
|
|
333
|
-
|
|
|
334
|
-
|
|
|
335
|
-
|
|
|
336
|
-
|
|
|
337
|
-
|
|
|
338
|
-
| `qamap e2e plan . --base origin/main --head HEAD` | Suggest E2E runner, bootstrap steps, user flows, coverage targets, existing test evidence, and missing testability hooks for changed files. |
|
|
339
|
-
| `qamap e2e plan . --base origin/main --head HEAD --record-history` | Save a compact local run snapshot under `.qamap/runs/` while keeping JSON/Markdown output usable. |
|
|
340
|
-
| `qamap e2e setup . --runner playwright` | Explicitly apply the accepted runner setup and create the first changed-flow E2E draft without overwriting existing files. |
|
|
341
|
-
| `qamap e2e draft . --base origin/main --head HEAD --dry-run` | Preview generated Maestro, Playwright, or manual E2E drafts without writing files. |
|
|
342
|
-
| `qamap e2e draft . --base origin/main --head HEAD` | Write generated Maestro, Playwright, or manual E2E drafts with flow language, readiness summaries, and action items. |
|
|
343
|
-
| `qamap manifest init .` | Create a baseline `.qamap/manifest.yaml` with inferred domains, flows, anchors, checks, source, and confidence. |
|
|
344
|
-
| `qamap manifest validate .` | Check whether `.qamap/manifest.yaml` is present, parseable, anchored to real files, and ready to shape PR evidence. |
|
|
345
|
-
| `qamap manifest context .` | Preview repo-local context sources, role classifications, validation commands, safety rules, and manifest repair diagnostics. |
|
|
346
|
-
| `qamap manifest explain . --base origin/main --head HEAD` | Explain which manifest domains, flows, and checks match the current branch and which manifest path to edit if the match is wrong. |
|
|
347
|
-
| `qamap flows init .` | Create a starter `.qamap/flows.yml` for team-approved core flow definitions. |
|
|
348
|
-
| `qamap flows suggest . --base origin/main --head HEAD` | Generate suggested `.qamap/flows.yml` entries with commit-readiness guidance from changed files and E2E plan context. |
|
|
349
|
-
| `qamap domains init .` | Create a starter `.qamap/domains.yml` for shared product/domain language. |
|
|
350
|
-
| `qamap domains suggest . --base origin/main --head HEAD` | Generate suggested `.qamap/domains.yml` entries with commit-readiness guidance from changed files and inferred product language. |
|
|
351
|
-
| `qamap history init .` | Create local QAMap history directories and protect generated run history with `.gitignore`. |
|
|
352
|
-
| `qamap doctor services/listing --workspace-root .` | Scan a monorepo package while using root guardrails. |
|
|
353
|
-
| `qamap context . --write AGENTS.md` | Generate starter agent instructions for the repo. |
|
|
354
|
-
| `qamap init .` | Create a starter `qamap.config.json`. |
|
|
355
|
-
|
|
356
|
-
For monorepos, pass `--workspace-root` when scanning a package. Package-local checks still use the package directory, while repo-level guardrails such as `AGENTS.md`, `.github/workflows`, `LICENSE`, `SECURITY.md`, and `CONTRIBUTING.md` are read from the workspace root.
|
|
357
|
-
|
|
358
|
-
`qamap review` compares a branch against a base ref for PR-style workflows. It separates newly introduced findings from risky files that already had findings on the base branch but were modified again, which helps reviewers notice when a PR touches known-dangerous surfaces such as committed `.env` files, MCP configs, or release scripts.
|
|
359
|
-
|
|
360
|
-
`qamap verify` is the easiest PR-facing command. It combines `review`, `test-plan`, and `eval` into one report with review findings, readiness gates, suggested domain tests, suggested commands, and next actions.
|
|
361
|
-
|
|
362
|
-
`qamap test-plan` turns changed file paths into a review-ready domain test checklist. It also discovers common validation commands from `package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, Gradle files, and Maven `pom.xml`. Add `--include-working-tree` for local, uncommitted changes while iterating.
|
|
363
|
-
|
|
364
|
-
`qamap e2e plan` turns changed file paths into a first-pass E2E testing plan. It detects whether a project looks like Expo/React Native, web, API/service, or CLI package, recommends a runner such as Maestro or Playwright, starts backend services with an API contract checklist, starts executable packages with a CLI command verification checklist, suggests bootstrap steps for repos with little or no test history, suggests domain language for the changed behavior, suggests candidate user flows, adds coverage targets, compares those targets with existing test-suite evidence when tests are present, flags API-dependent flows that need mock or fixture responses, and points out missing stable selectors such as `testID` or `data-testid` before anyone starts writing tests from a blank file.
|
|
365
|
-
|
|
366
|
-
The plan also includes an execution profile: detected start command, test command, Playwright `baseURL`, mobile app id, runner config files, env fixture files, confidence, and blockers. This keeps generated E2E drafts honest about whether they are runnable candidates or still review-only scaffolds.
|
|
367
|
-
|
|
368
|
-
When a repository does not already have the selected E2E runner, the plan includes a runner setup proposal instead of silently changing the project. The proposal explains why the runner fits the changed surface, which package command installs the library, which config/script files would be created, and the explicit acceptance command such as `qamap e2e setup . --runner playwright`.
|
|
369
|
-
|
|
370
|
-
`qamap e2e setup` is the opt-in apply step. For Playwright it can create `playwright.config.ts`, `tests/e2e/`, a `test:e2e` script, and the first changed-flow Playwright spec. For Maestro it can create `.maestro/`, `.maestro/README.md`, a `test:e2e` script, and the first changed-flow YAML draft. Existing draft files are skipped unless `--force` is passed. It does not run package installation automatically; it prints the install command so teams can keep dependency policy under review.
|
|
371
|
-
|
|
372
|
-
When run at a monorepo root, the E2E plan also reports changed app/package targets. This helps a maintainer move from a broad workspace diff to scoped commands such as `qamap e2e plan services/listing --workspace-root . --base origin/main --head HEAD`, where package-specific runner detection and flow naming are usually sharper.
|
|
373
|
-
|
|
374
|
-
Each candidate flow also includes a flow language brief: actor, trigger, goal, success signal, reviewer question, and edge cases. The brief keeps generated tests tied to product behavior rather than only changed file names.
|
|
375
|
-
|
|
376
|
-
The bootstrap section answers what must happen before generated drafts can be treated as real regression coverage. For example, a testless web project can get required steps for Playwright setup, first draft generation, stable selector work, fixture/mock data, and missing validation evidence, plus recommended steps for `.qamap/manifest.yaml`, `.qamap/domains.yml`, `.qamap/flows.yml`, and local history recording.
|
|
377
|
-
|
|
378
|
-
Run `qamap manifest init .` to create a baseline verification manifest. QAMap infers domains, flows, route/component anchors, checks, runner hints, source, and confidence from the current checkout.
|
|
379
|
-
|
|
380
|
-
> **Important:** create the shared team baseline from the repository's default branch, after pulling the latest changes. QAMap does not silently switch branches or rewrite the repository state, so running `manifest init` from a feature branch creates a feature-branch snapshot, not the team's default QA map.
|
|
381
|
-
|
|
382
|
-
```sh
|
|
383
|
-
git switch main
|
|
384
|
-
git pull
|
|
385
|
-
qamap manifest context .
|
|
386
|
-
qamap manifest init . --write .qamap/manifest.yaml
|
|
387
|
-
```
|
|
388
|
-
|
|
389
|
-
`qamap manifest context .` is a read-only preview of the repo-local knowledge QAMap can see before writing the manifest. It reports context sources such as `CONTEXT.md`, ADRs, goals, runbooks, agent instructions, harness files, and skills, then shows role classifications, validation commands, safety rules, and diagnostics for stale or missing context.
|
|
390
|
-
|
|
391
|
-
After the baseline is committed, feature branches should usually run `manifest explain`, `e2e plan`, or `e2e draft` against the PR base such as `origin/main`. The manifest is not meant to be perfect on the first run. It is meant to start the feedback loop: QAMap recommends E2E work from the manifest, shows why a recommendation happened, and points to the manifest path to edit when the recommendation is wrong.
|
|
392
|
-
|
|
393
|
-
Generated manifests include a `$schema` reference to `schema/qamap-manifest.schema.json`, so teams can validate and edit `.qamap/manifest.yaml` with a documented contract. See [docs/manifest.md](docs/manifest.md) for the full field guide and adoption workflow.
|
|
394
|
-
|
|
395
|
-
Use `qamap manifest validate .` before treating the manifest as shared team policy. It reports missing manifests, invalid YAML/schema shape, duplicate ids, missing domain paths, stale anchor files, suspicious route hints, and low-confidence inferred entries that should be reviewed.
|
|
396
|
-
|
|
397
|
-
Use `qamap manifest explain . --base origin/main --head HEAD` when you want to understand one branch. It reads the git diff, lists the matched manifest domains/flows/checks, shows the declared entry route and required checks, and names the exact manifest path to update if the recommendation is wrong.
|
|
398
|
-
|
|
399
|
-
When `.qamap/manifest.yaml` exists, `qamap verify`, `qamap e2e plan`, and `qamap e2e draft` include a Manifest Recommendations section:
|
|
400
|
-
|
|
401
|
-
```txt
|
|
402
|
-
Why this was recommended:
|
|
403
|
-
- Changed files match anchors for the Bundle Submission Complete flow.
|
|
404
|
-
|
|
405
|
-
Manifest evidence:
|
|
406
|
-
- .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
407
|
-
|
|
408
|
-
Next actions:
|
|
409
|
-
- Draft or review E2E coverage for the Bundle Submission Complete flow.
|
|
410
|
-
- Cover the declared checks: Submit media link successfully; Show validation error for invalid media link.
|
|
411
|
-
|
|
412
|
-
If this is wrong:
|
|
413
|
-
- Update .qamap/manifest.yaml > flows.bundle-submission-complete.anchors
|
|
414
|
-
|
|
415
|
-
Repair hints:
|
|
416
|
-
- If these files do not belong to this flow, update .qamap/manifest.yaml > flows.bundle-submission-complete.anchors.
|
|
417
|
-
- If the recommended assertions feel vague, rewrite .qamap/manifest.yaml > flows.bundle-submission-complete.checks in team language.
|
|
418
|
-
```
|
|
419
|
-
|
|
420
|
-
When a matched manifest flow has an entry route and checks, `qamap e2e draft` promotes it ahead of heuristic candidates. The generated Playwright, Maestro, or manual draft carries the manifest evidence, uses the manifest route as an entrypoint when possible, and turns manifest checks into draft steps and required coverage notes. If a check includes concrete hints such as `[data-testid=coupon-input]`, `with WELCOME10`, or optional `selector`, `value`, and `steps` fields, QAMap uses those facts before falling back to fuzzy selector inference. This is the core cost-saving loop: humans fix durable QA context once, then future PRs start from a stronger draft instead of a blank test file.
|
|
421
|
-
|
|
422
|
-
The domain language section is intentionally less implementation-oriented than the raw file list. For example, changes under `src/features/in-app-purchase/` become terms such as `In App Purchase` and scenarios such as `In App Purchase primary journey`. When a changed component or service file names a concrete behavior, QAMap should prefer that behavior before the generic primary journey: `src/features/listing/components/MediaLinkSubmitModal.tsx` can become `Listing Media Link Submit`, and the generated draft file can become `.maestro/listing-media-link-submit.yaml`. When `.qamap/domains.yml` exists, declared product terms and routes receive higher confidence. When `.qamap/flows.yml` exists, team-approved flow names appear as preferred scenario names.
|
|
423
|
-
|
|
424
|
-
If `.qamap/domains.yml` exists, `qamap e2e plan` also matches changed files against shared product or domain language:
|
|
425
|
-
|
|
426
|
-
```yaml
|
|
427
|
-
domains:
|
|
428
|
-
- id: billing
|
|
429
|
-
name: Billing
|
|
430
|
-
aliases:
|
|
431
|
-
- checkout
|
|
432
|
-
- subscription
|
|
433
|
-
files:
|
|
434
|
-
- src/features/billing/**
|
|
435
|
-
routes:
|
|
436
|
-
- /billing
|
|
437
|
-
scenarios:
|
|
438
|
-
- title: Billing primary journey
|
|
439
|
-
checks:
|
|
440
|
-
- Start from the normal billing entry point.
|
|
441
|
-
- Complete the primary billing action with realistic data.
|
|
442
|
-
```
|
|
443
|
-
|
|
444
|
-
Run `qamap domains init .` to create a starter domain manifest. Run `qamap domains suggest . --base origin/main --head HEAD` when you want QAMap to draft manifest entries from the current branch and classify each candidate as `commit-candidate`, `needs-review`, or `low-signal`. Use domains for naming and route hints; use core flows when the team wants to define a durable verification journey.
|
|
445
|
-
|
|
446
|
-
If `.qamap/flows.yml` exists, `qamap e2e plan` also matches changed files against team-approved core flows. This lets maintainers encode the product or domain flows humans already care about:
|
|
447
|
-
|
|
448
|
-
```yaml
|
|
449
|
-
flows:
|
|
450
|
-
- id: checkout-purchase
|
|
451
|
-
name: Checkout purchase
|
|
452
|
-
priority: critical
|
|
453
|
-
domains:
|
|
454
|
-
- checkout
|
|
455
|
-
files:
|
|
456
|
-
- src/pages/checkout/**
|
|
457
|
-
- src/features/checkout/**
|
|
458
|
-
routes:
|
|
459
|
-
- /checkout
|
|
460
|
-
checks:
|
|
461
|
-
- Complete checkout with a valid payment method.
|
|
462
|
-
- Verify declined payment recovery.
|
|
463
|
-
```
|
|
464
|
-
|
|
465
|
-
Run `qamap flows init .` to create a starter manifest. Run `qamap flows suggest . --base origin/main --head HEAD` when you want QAMap to draft flow entries from changed files, inferred domain language, routes, and E2E checks, then classify which entries are close enough to review as shared policy. Unlike generated run history, `.qamap/flows.yml` is meant to be reviewed and committed when those flow definitions should become team policy.
|
|
466
|
-
|
|
467
|
-
Pass `--record-history` when you want QAMap to keep a compact local snapshot of an E2E plan under `.qamap/runs/`. QAMap automatically protects `.qamap/runs/`, `.qamap/cache/`, `.qamap/tmp/`, and `.qamap/*.local.json` with `.gitignore` so generated history stays local by default. Shared project policy, such as `qamap.config.json`, `.qamap/domains.yml`, and `.qamap/flows.yml`, remains commit-friendly.
|
|
468
|
-
|
|
469
|
-
`qamap e2e draft --dry-run` previews the same draft analysis without creating directories or files. Use it first when evaluating a new repository or PR. The output still includes planned file paths, self-checks, readiness status, action items, starter-code gaps, and execution blockers.
|
|
470
|
-
|
|
471
|
-
`qamap e2e draft` writes draft files from that plan. Expo and React Native projects get Maestro YAML flows under `.maestro/` by default, web projects get Playwright specs under `tests/e2e/`, API/service projects get contract checklist drafts, and CLI packages get command verification checklists until a project-specific runner is documented. For web apps, QAMap recognizes common Next.js, React Router, Vite, Vue/Nuxt, Svelte, Remix, Astro, and Angular signals. It can infer routes from Next Pages Router files, Next App Router files such as `src/app/(group)/products/[id]/page.tsx`, React Router `path` objects, links, and imperative navigation calls. Drafts infer stable selectors such as `testID`, `accessibilityLabel`, `data-testid`, `aria-label`, placeholder text, role-based buttons or links, and visible text where possible. They also carry fixture/mock readiness notes and inferred API endpoint hints. Client-only flows may get sample Playwright response scaffolds, but endpoints changed by the PR are observed rather than intercepted with synthetic responses so the generated draft does not hide the contract under test. When selectors or route params are incomplete, QAMap now prefers runnable starter code with safe smoke assertions and sample params over non-executable placeholder locators. Existing files are not overwritten unless `--force` is passed.
|
|
472
|
-
|
|
473
|
-
The draft result is meant to be useful as a PR artifact, not only as generated files. Markdown and JSON output include:
|
|
474
|
-
|
|
475
|
-
- `languageBrief`: actor, trigger, goal, success signal, reviewer question, and edge cases for each draft file
|
|
476
|
-
- `promotionStatus`: whether the draft is a `commit-candidate`, `needs-review`, or `low-signal`
|
|
477
|
-
- `runnableStatus`: whether the draft is a `runnable-candidate`, `near-runnable`, or `review-only`
|
|
478
|
-
- `selfCheck`: static runner checks for generated draft structure, unresolved placeholders, starter-code quality, and the execution profile
|
|
479
|
-
- `status`: whether the file was `preview`ed by `--dry-run`, `created`, or `skipped`
|
|
480
|
-
- `actionItems`: required and recommended follow-up work, grouped by assertion, fixture, selector, runner, validation, and manifest
|
|
481
|
-
- `actionSummary`: total required/recommended action counts, ready file count, and the most common action categories
|
|
482
|
-
- `readinessSummary`: an overall 0-100 score, readiness level, self-check counts, starter-code gaps, execution blocker counts, and top blockers
|
|
483
|
-
|
|
484
|
-
See [docs/e2e-output-examples.md](docs/e2e-output-examples.md) for compact examples of web, mobile, API/service, CLI, test-light, and monorepo output.
|
|
485
|
-
|
|
486
|
-
Generated Playwright drafts use the flow language as `test.step()` names so the file reads like the user journey it protects:
|
|
487
|
-
|
|
488
|
-
```ts
|
|
489
|
-
await test.step("Open route /checkout.", async () => {
|
|
490
|
-
await page.goto("/checkout");
|
|
491
|
-
});
|
|
492
|
-
|
|
493
|
-
await test.step("Complete checkout with a valid payment method.", async () => {
|
|
494
|
-
// Step intent: Complete checkout with a valid payment method.
|
|
495
|
-
await page.getByTestId("checkout-submit").click();
|
|
496
|
-
});
|
|
497
|
-
|
|
498
|
-
await test.step("Fill profile email.", async () => {
|
|
499
|
-
// Step intent: Fill profile email.
|
|
500
|
-
await page.getByPlaceholder("Profile email").fill("qamap@example.com");
|
|
501
|
-
});
|
|
502
|
-
```
|
|
503
|
-
|
|
504
|
-
`qamap history init` prepares that local storage explicitly without running an analysis. It creates `.qamap/runs/`, `.qamap/cache/`, and `.qamap/tmp/`, then adds the generated-history ignore patterns to `.gitignore` idempotently.
|
|
505
|
-
|
|
506
|
-
`qamap eval` scores whether a branch has enough validation evidence, changed-test coverage, intent capture, risk explanation, domain verification paths, and reviewable size. In GitHub Actions, QAMap can read the pull request body from the event payload and append the evaluation to the PR comment.
|
|
507
|
-
|
|
508
|
-
## What It Checks
|
|
509
|
-
|
|
510
|
-
The first release focuses on high-signal checks that are useful across many repositories.
|
|
511
|
-
|
|
512
|
-
| Rule | Severity | What it catches |
|
|
513
|
-
| --- | --- | --- |
|
|
514
|
-
| `QM001` | medium | Missing agent instruction files. |
|
|
515
|
-
| `QM002` | medium | Conflicting agent guidance. |
|
|
516
|
-
| `QM003` | high | Suspicious instruction text that can misdirect agents. |
|
|
517
|
-
| `QM004` | medium/high | Risky MCP command configuration. |
|
|
518
|
-
| `QM005` | high | Secret-like values embedded in MCP config. |
|
|
519
|
-
| `QM006` | medium | Missing or placeholder test scripts. |
|
|
520
|
-
| `QM007` | low | Missing GitHub Actions workflows. |
|
|
521
|
-
| `QM008` | high | Committed local environment files. |
|
|
522
|
-
| `QM009` | high | Package scripts that can publish, push, merge, or run unsafe shell pipelines. |
|
|
523
|
-
| `QM010` | medium | Broad workflow permissions or risky workflow triggers. |
|
|
524
|
-
| `QM011` | low | Missing community health files. |
|
|
525
|
-
| `QM012` | medium/high | Risky committed agent settings, hooks, or broad shell permissions. |
|
|
526
|
-
| `QM013` | low | API endpoints documented only in prose without a contract source. |
|
|
527
|
-
|
|
528
|
-
See [docs/rules.md](docs/rules.md) for the rule catalog.
|
|
529
|
-
See [docs/ecosystem.md](docs/ecosystem.md) for the agent ecosystem surfaces QAMap tracks.
|
|
530
|
-
See [docs/api-contracts.md](docs/api-contracts.md) for the API contract source-of-truth check.
|
|
531
|
-
See [docs/verify.md](docs/verify.md) for the combined PR verification report.
|
|
532
|
-
See [docs/eval.md](docs/eval.md) for the change readiness evaluation.
|
|
533
|
-
See [docs/releasing.md](docs/releasing.md) for the npm release runbook.
|
|
534
|
-
|
|
535
|
-
## Configuration
|
|
536
|
-
|
|
537
|
-
Use `qamap.config.json` or `.qamap.json` to tune repository policy.
|
|
538
|
-
|
|
539
|
-
```json
|
|
540
|
-
{
|
|
541
|
-
"$schema": "https://raw.githubusercontent.com/IvoryCanvas/qamap/main/schema/qamap.schema.json",
|
|
542
|
-
"failOn": "high",
|
|
543
|
-
"ignoreRules": ["QM011"],
|
|
544
|
-
"maxFiles": 2000,
|
|
545
|
-
"validationCommands": ["make test", "make lint"],
|
|
546
|
-
"severity": {
|
|
547
|
-
"QM007": "info"
|
|
548
|
-
}
|
|
549
|
-
}
|
|
550
|
-
```
|
|
551
|
-
|
|
552
|
-
See [docs/configuration.md](docs/configuration.md) for details.
|
|
553
|
-
|
|
554
|
-
## GitHub Actions
|
|
555
|
-
|
|
556
|
-
QAMap can run as a lightweight PR check with annotations, a step summary, and a sticky PR comment:
|
|
557
|
-
|
|
558
|
-
```yaml
|
|
559
|
-
name: QAMap
|
|
560
|
-
|
|
561
|
-
on:
|
|
562
|
-
pull_request:
|
|
563
|
-
|
|
564
|
-
permissions:
|
|
565
|
-
contents: read
|
|
566
|
-
pull-requests: write
|
|
567
|
-
|
|
568
|
-
jobs:
|
|
569
|
-
qamap:
|
|
570
|
-
runs-on: ubuntu-latest
|
|
571
|
-
steps:
|
|
572
|
-
- uses: actions/checkout@v7
|
|
573
|
-
with:
|
|
574
|
-
fetch-depth: 0
|
|
575
|
-
- uses: IvoryCanvas/qamap@main
|
|
576
|
-
with:
|
|
577
|
-
mode: review
|
|
578
|
-
base: ${{ github.event.pull_request.base.sha }}
|
|
579
|
-
head: HEAD
|
|
580
|
-
fail-on: high
|
|
581
|
-
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
582
|
-
```
|
|
583
|
-
|
|
584
|
-
The PR comment can include suggested domain tests and a verification readiness evaluation for changed files. For rollout guidance, see [docs/adoption.md](docs/adoption.md) and [docs/github-action.md](docs/github-action.md).
|
|
585
|
-
|
|
586
|
-
## Where QAMap Fits
|
|
587
|
-
|
|
588
|
-
On the QA side, QAMap starts one step earlier than test-writing tools — it decides what a PR must prove before anyone records, generates, or writes a test:
|
|
589
|
-
|
|
590
|
-
| Tool category | Typical focus | QAMap focus |
|
|
591
|
-
| --- | --- | --- |
|
|
592
|
-
| Test recorders and studios | Turning a known flow into a script by watching you run it. | Deciding which flow a PR affects and what evidence is missing, before recording starts. |
|
|
593
|
-
| LLM test generation | Spending model tokens to write test code from source. | Free, deterministic PR-to-QA mapping; drafts are starter scaffolds an agent or human finishes. |
|
|
594
|
-
| Re-prompting an agent per PR | Re-deriving repo QA context in every session. | Repo-owned QA memory (`.qamap/manifest.yaml`) plus a compact `--format agent` handoff. |
|
|
595
|
-
| Change-impact test selection | Choosing which existing unit/CI tests to run. | Naming the user-facing flow and E2E/checklist work that should exist at all. |
|
|
596
|
-
|
|
597
|
-
On the guardrails side, QAMap is not trying to replace the larger security ecosystem:
|
|
598
|
-
|
|
599
|
-
| Tool category | Typical focus | QAMap focus |
|
|
600
|
-
| --- | --- | --- |
|
|
601
|
-
| OpenSSF Scorecard | Broad open source security posture. | AI-agent readiness at the repository boundary. |
|
|
602
|
-
| Secret scanning | Exposed credentials in code or history. | Secret-like values plus unsafe agent, workflow, and script context. |
|
|
603
|
-
| MCP security scanners | Deep analysis of MCP servers, tools, prompts, and skills. | Static repo checks without executing untrusted MCP servers. |
|
|
604
|
-
| General linters | Code style, correctness, or framework rules. | Guardrails that affect AI-assisted development safety. |
|
|
605
|
-
|
|
606
|
-
## Roadmap
|
|
607
|
-
|
|
608
|
-
QAMap starts as a local CLI and should stay small enough that maintainers can understand every finding.
|
|
609
|
-
|
|
610
|
-
Near-term priorities:
|
|
611
|
-
|
|
612
|
-
- keep the [release validation](docs/release-validation.md) checklist current across representative manifest, web, mobile, API/service, CLI, monorepo, and test-light repositories
|
|
613
|
-
- keep the [release runbook](docs/releasing.md) aligned with the npm package and GitHub Action release process
|
|
614
|
-
- publish a versioned GitHub Action release tag after the first public package is ready
|
|
615
|
-
- improve branch-aware `review` changed-line locations
|
|
616
|
-
- improve generated domain test plans with framework-specific test skeletons
|
|
617
|
-
- refine generated Maestro and Playwright drafts with stronger app-specific selector discovery and self-check loops
|
|
618
|
-
- expand `eval` into repository-specific verification manifests and taste rubrics
|
|
619
|
-
- continue expanding agent surface detection across Codex, Claude Code, Cursor, Copilot, Gemini, and related tools
|
|
620
|
-
- generate rule documentation from scanner metadata
|
|
621
|
-
|
|
622
|
-
See [docs/roadmap.md](docs/roadmap.md) for more detail.
|
|
90
|
+
| [Command reference](docs/commands.md) | Every command, outputs, and the E2E draft pipeline in depth |
|
|
91
|
+
| [Quick start walkthrough](docs/quickstart-demo.md) | The 30-second demo, step by step |
|
|
92
|
+
| [Verification manifest](docs/manifest.md) | Repo-local QA memory: schema, init, explain, repair loop |
|
|
93
|
+
| [Adoption & rollout](docs/adoption.md) | First run to CI gate, plus positioning |
|
|
94
|
+
| [Agent skill guide](docs/agent-skill.md) | Using QAMap from coding-agent workflows |
|
|
95
|
+
| [Agent format contract](docs/agent-format.md) | The versioned `--format agent` JSON shape and stability policy |
|
|
96
|
+
| [Repository guardrails](docs/guardrails.md) | The optional static scanner and its rules |
|
|
97
|
+
| [Configuration](docs/configuration.md) | `qamap.config.json` policy options |
|
|
98
|
+
| [GitHub Action](docs/github-action.md) | PR annotations, summaries, and comments in CI |
|
|
99
|
+
| [Benchmarking](docs/benchmarking.md) | Scoring output quality against pinned repositories |
|
|
100
|
+
| [Roadmap](docs/roadmap.md) | Where this is going |
|
|
623
101
|
|
|
624
102
|
## Project Status
|
|
625
103
|
|
|
626
|
-
QAMap is early and pre-`1.0
|
|
627
|
-
|
|
628
|
-
## Contributing
|
|
629
|
-
|
|
630
|
-
Issues and pull requests are welcome. Maintainer permissions stay with IvoryCanvas members, and `main` is protected so external contributors cannot push or merge directly.
|
|
631
|
-
|
|
632
|
-
Good first contributions include new agent instruction file detectors, better SARIF locations, sample risky repository fixtures, and documentation improvements.
|
|
633
|
-
|
|
634
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md), [GOVERNANCE.md](GOVERNANCE.md), and [SECURITY.md](SECURITY.md).
|
|
635
|
-
|
|
636
|
-
## Philosophy
|
|
637
|
-
|
|
638
|
-
QAMap does not replace code review, tests, threat modeling, branch protection, or security review. It is a small preflight check that helps teams notice repo-level AI risks early enough to do something about them.
|
|
104
|
+
QAMap is early and pre-`1.0`; the public API may change. Issues and pull requests are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). QAMap does not replace review, tests, or security tooling; it removes the blank-page work that makes teams skip good verification.
|