@ivorycanvas/qamap 0.3.5 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/CHANGELOG.md +49 -0
  2. package/README.md +37 -23
  3. package/dist/agent-init.js +2 -2
  4. package/dist/agent-init.js.map +1 -1
  5. package/dist/behavior-intent.d.ts +6 -0
  6. package/dist/behavior-intent.js +183 -0
  7. package/dist/behavior-intent.js.map +1 -0
  8. package/dist/behavior-manifest.d.ts +6 -0
  9. package/dist/behavior-manifest.js +221 -0
  10. package/dist/behavior-manifest.js.map +1 -0
  11. package/dist/behavior.d.ts +151 -0
  12. package/dist/behavior.js +458 -0
  13. package/dist/behavior.js.map +1 -0
  14. package/dist/change-intent.d.ts +81 -0
  15. package/dist/change-intent.js +884 -0
  16. package/dist/change-intent.js.map +1 -0
  17. package/dist/cli.js +13 -8
  18. package/dist/cli.js.map +1 -1
  19. package/dist/e2e.d.ts +16 -1
  20. package/dist/e2e.js +290 -25
  21. package/dist/e2e.js.map +1 -1
  22. package/dist/index.d.ts +11 -3
  23. package/dist/index.js +5 -1
  24. package/dist/index.js.map +1 -1
  25. package/dist/qa.d.ts +1 -0
  26. package/dist/qa.js +293 -88
  27. package/dist/qa.js.map +1 -1
  28. package/dist/terminal.js +1 -1
  29. package/dist/terminal.js.map +1 -1
  30. package/dist/test-plan.d.ts +18 -0
  31. package/dist/test-plan.js +123 -13
  32. package/dist/test-plan.js.map +1 -1
  33. package/dist/version.d.ts +1 -1
  34. package/dist/version.js +1 -1
  35. package/docs/adoption.md +11 -10
  36. package/docs/agent-format.md +16 -10
  37. package/docs/agent-skill.md +8 -5
  38. package/docs/architecture.md +131 -0
  39. package/docs/benchmarking.md +23 -4
  40. package/docs/commands.md +13 -7
  41. package/docs/e2e-output-examples.md +29 -28
  42. package/docs/quickstart-demo.md +29 -17
  43. package/docs/release-validation.md +49 -6
  44. package/docs/releasing.md +26 -3
  45. package/docs/roadmap.md +20 -10
  46. package/package.json +4 -2
  47. package/schema/qamap-agent.schema.json +122 -3
  48. package/schema/qamap-behavior.schema.json +339 -0
  49. package/skills/qamap-pr-qa/SKILL.md +20 -14
@@ -1,6 +1,6 @@
1
1
  # Agent Format Contract
2
2
 
3
- `qamap qa --format agent` prints one line of JSON (~2–4KB) designed to be pasted into a coding agent's context instead of the full markdown report. This page is the contract for that output: what the fields mean, what an agent may rely on, and how the format is allowed to change.
3
+ `qamap qa --format agent` prints one compact line of JSON designed to be pasted into a coding agent's context instead of the full markdown report. The complete line is capped at 8KB. When the uncapped result would be larger, QAMap preserves the strongest evidence and reports total and omitted counts instead of silently overflowing the context budget. This page is the contract for that output: what the fields mean, what an agent may rely on, and how the format is allowed to change.
4
4
 
5
5
  ```sh
6
6
  qamap qa . --base origin/main --head HEAD --format agent
@@ -19,10 +19,11 @@ qamap qa . --base origin/main --head HEAD --format agent
19
19
  The intended loop for a coding agent:
20
20
 
21
21
  1. Run the command above and parse stdout as JSON.
22
- 2. If `readiness.level` is `blocked` or `needs-work`, treat generated drafts as review-only; `requiredBootstrap` lists what unlocks the next stage.
23
- 3. Use `flows[].changedFiles`, `flows[].evidence`, and `flows[].reviewQuestion` to understand why the flow was selected. Use `steps`, `selectors`, and `successSignal` to write or review tests; `runnable` says how much to trust the generated draft.
24
- 4. Surface `requiredEvidence` in the PR description, and paste `prChecklist` items into the PR body.
25
- 5. Run `commands` to validate.
22
+ 2. Read `intents` first. Confirm its confidence, lifecycle, and QA scenarios. Inspect `scenarios[].sources` before accepting a recommendation: diff sources identify the base/head file, line, symbol, hunk, and relation that caused the scenario to be proposed. `direct` is scenario-specific evidence, `supporting` completes the lifecycle, and `contextual` explains intent but cannot independently promote a scenario to critical. When `reviewRequired` is true, ask a human to confirm it.
23
+ 3. Treat `readiness` as automation readiness, not as a verdict on the PR. `requiredBootstrap` contains non-runner repository context only; runner adoption lives under the opt-in `automation` object.
24
+ 4. Use `flows[].changedFiles`, `flows[].evidence`, and `flows[].reviewQuestion` to understand why the flow was selected. Use `steps`, `selectors`, and `successSignal` to write or review tests; `runnable` says how much to trust the generated draft.
25
+ 5. Surface `requiredEvidence` in the PR description, and paste `prChecklist` items into the PR body.
26
+ 6. Run `commands` to validate.
26
27
 
27
28
  ## Fields
28
29
 
@@ -31,15 +32,20 @@ The intended loop for a coding agent:
31
32
  | `schema` | object | `{ name: "qamap.qa", version: 1 }` — check before parsing. |
32
33
  | `base`, `head` | string | Git refs the diff was computed from. |
33
34
  | `project` | string | Detected project type (for example `web`, `react-native`, `node`, `unknown`). |
34
- | `runner` | string | Recommended E2E runner: `maestro`, `playwright`, or `manual`. |
35
+ | `runner` | string | Automation output adapter selected after QA intent analysis: `maestro`, `playwright`, or `manual`. |
35
36
  | `manifest` | string \| null | Verification manifest path in use, or `null` when the run used repo signals and the PR diff only. |
36
37
  | `readiness` | object | `score` (0–100) and `level` (`ready` \| `near-runnable` \| `needs-work` \| `blocked`). Human reports render the same value as a four-stage journey; the machine value is stable. |
37
38
  | `testSuite` | object | `present` (boolean) and `files` (number of detected test files). |
38
- | `firstDraftCommand` | string? | One command that creates the first E2E draft. Present only when the repository has no test suite and the diff reaches a product journey. |
39
- | `flows` | array | Affected user flows, most relevant first (capped). Each has `title`, `source`, `draft`, optional `runnable`, `entry`, and `verificationMode`, plus `changedFiles`, `reviewQuestion`, `successSignal`, `steps`, `selectors`, and short `evidence` reasons. Test-only changes expose `existingEvidence`; configuration, docs, generated artifacts, and changed tests use `verificationMode`, keep `draft` only as a fallback artifact path, and omit `firstDraftCommand` rather than inventing a required product journey. |
39
+ | `intentCount`, `omittedIntentCount` | number | Total inferred intents and the count omitted from the compact payload. |
40
+ | `intents` | array | Evidence-backed change intents (capped). Each includes `title`, `confidence`, `reviewRequired`, backward-compatible string `evidence`, structured `sources`, ordered `lifecycle` phases, and runner-independent QA `scenarios`. Every compact scenario carries `confidence`, `reviewRequired`, structured `sources`, and assertions; `scenarioCount` and `omittedScenarioCount` disclose capping. Empty when commit and diff evidence cannot support a behavior intent. |
41
+ | `firstDraftCommand` | string? | Deprecated v1 compatibility field. New output omits it so runner setup is not promoted as the default QA action. |
42
+ | `automation` | object? | Explicitly optional adapter handoff: `optIn`, `adapter`, `setupStatus`, `draftCommand`, and optional `setupCommand`. Use it only after the QA scenario is accepted. |
43
+ | `flowCount`, `omittedFlowCount` | number | Total affected flows and the count omitted from the compact payload. |
44
+ | `flows` | array | Affected user flows, most relevant first (capped). Each has `title`, `source`, backward-compatible `draft`, optional `runnable`, `entry`, and `verificationMode`, plus `changedFiles`, `reviewQuestion`, `successSignal`, `steps`, `selectors`, and short `evidence` reasons. Test-only changes expose `existingEvidence`; configuration, docs, generated artifacts, and changed tests use `verificationMode`. |
45
+ | `compaction` | object | Present only when lower-priority detail was reduced to keep the complete line within 8KB. Carries `maxBytes` and the uncapped `originalBytes`. |
40
46
  | `requiredEvidence` | array | Required-priority QA evidence still missing, capped at 8: `flow`, `kind`, `title`. |
41
47
  | `recommendedEvidenceCount` | number | How many recommended-priority items were omitted; run without `--format agent` to see them. |
42
- | `requiredBootstrap` | array | Setup steps (capped at 3) that must happen before drafts count as regression coverage: `title`, `action`. |
48
+ | `requiredBootstrap` | array | Non-runner repository context steps (capped at 3): `title`, `action`. Runner setup is represented only under `automation`. |
43
49
  | `prChecklist` | array of string | Ready-to-paste PR checklist lines (capped). |
44
50
  | `commands` | array of string | Suggested next commands, most useful first (capped at 4). |
45
51
 
@@ -48,5 +54,5 @@ List fields are capped to keep the payload small; caps may grow within version 1
48
54
  ## Example
49
55
 
50
56
  ```json
51
- {"schema":{"name":"qamap.qa","version":1},"base":"main","head":"HEAD","project":"web","runner":"playwright","manifest":null,"readiness":{"score":51,"level":"needs-work"},"testSuite":{"present":false,"files":0},"firstDraftCommand":"qamap e2e setup . --runner playwright","flows":[{"title":"Checkout Submit","source":"domain-language","draft":"tests/e2e/checkout-submit.spec.ts","runnable":"near-runnable","entry":"route: /checkout (high)","changedFiles":["src/pages/checkout/index.tsx"],"reviewQuestion":"Can a reviewer confirm that Checkout Submit still works from this entrypoint and this outcome is verified: visible text Order confirmed appears?","successSignal":"visible text Order confirmed appears","steps":["Fill Email with realistic data.","Submit Submit using Checkout Submit."],"selectors":["web-test-id: checkout-submit"],"evidence":["Primary entrypoint inferred as route /checkout."]}],"requiredEvidence":[{"flow":"Checkout Submit","kind":"runner","title":"Configure Playwright execution"}],"recommendedEvidenceCount":2,"requiredBootstrap":[{"title":"Configure Playwright before making drafts required","action":"Run qamap e2e setup . --runner playwright."}],"prChecklist":["Review the generated draft path: tests/e2e/checkout-submit.spec.ts."],"commands":["npm run build"]}
57
+ {"schema":{"name":"qamap.qa","version":1},"base":"main","head":"HEAD","project":"web","runner":"playwright","manifest":null,"readiness":{"score":51,"level":"needs-work"},"testSuite":{"present":false,"files":0},"intentCount":1,"omittedIntentCount":0,"intents":[{"title":"Submit notification preferences","confidence":"high","reviewRequired":false,"evidence":["feat: submit notification preferences"],"sources":[{"kind":"diff","reason":"Invoke fetch.","file":"src/pages/preferences.tsx","symbol":"fetch","relation":"supporting","side":"head","startLine":7,"endLine":7,"hunk":"@@ -1,5 +1,19 @@"}],"scenarioCount":1,"omittedScenarioCount":0,"lifecycle":[{"phase":"trigger","label":"Submit notification preferences."},{"phase":"side-effect","label":"Invoke fetch."},{"phase":"observable-outcome","label":"Show the saved state."}],"scenarios":[{"priority":"critical","kind":"primary","title":"Submit notification preferences","confidence":"high","reviewRequired":false,"sources":[{"kind":"diff","reason":"Invoke fetch.","file":"src/pages/preferences.tsx","symbol":"fetch","relation":"supporting","side":"head","startLine":7,"endLine":7,"hunk":"@@ -1,5 +1,19 @@"}],"assertions":["Verify the saved state becomes observable."]}]}],"automation":{"optIn":true,"adapter":"playwright","setupStatus":"proposed","draftCommand":"qamap e2e draft . --base main --head HEAD","setupCommand":"qamap e2e setup . --runner playwright"},"flowCount":1,"omittedFlowCount":0,"flows":[{"title":"Submit notification preferences","source":"commit-and-diff-intent","draft":"tests/e2e/submit-notification-preferences.spec.ts","runnable":"near-runnable","entry":"route: /preferences (high)","changedFiles":["src/pages/preferences.tsx"],"reviewQuestion":"Does the changed preference lifecycle produce the saved state?","successSignal":"visible text Preferences saved appears","steps":["Submit preferences.","Invoke fetch.","Verify the saved state."],"selectors":["web-test-id: preferences-save"],"evidence":["Commit and diff evidence support this change intent."]}],"requiredEvidence":[],"recommendedEvidenceCount":1,"requiredBootstrap":[],"prChecklist":["Review the proposed QA scenario and its diff source."],"commands":["npm run build"]}
52
58
  ```
@@ -32,7 +32,7 @@ Run this before writing a PR body or asking for review. Agents should prefer the
32
32
  pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD --format agent
33
33
  ```
34
34
 
35
- The result carries `flows[]` (draft path, runnable status, entry route, steps, selectors), `requiredEvidence[]`, `requiredBootstrap[]`, `prChecklist[]`, and `commands[]` under `schema: qamap.qa`.
35
+ The result carries `intents[]` with scenario-level structured diff `sources`, `flows[]` (affected behavior, entry route, steps, selectors), `requiredEvidence[]`, optional `automation`, `prChecklist[]`, and `commands[]` under `schema: qamap.qa`.
36
36
 
37
37
  For a human-readable report, drop the flag; for installed projects write it to a file:
38
38
 
@@ -68,12 +68,15 @@ If your agent supports symlinked skills, point its skill directory at `skills/qa
68
68
 
69
69
  ## What The Agent Should Do With The Output
70
70
 
71
- Use the `PR Comment Draft` section as review context:
71
+ Use `Change Intent Evidence` and the `PR Comment Draft` as review context:
72
72
 
73
+ - commit-backed intent, confidence, and whether human review is required
74
+ - ordered behavior lifecycle
75
+ - primary, failure, boundary, and state-transition QA scenarios
76
+ - the strongest commit or `file:line` source for every proposed scenario
73
77
  - affected flow
74
- - recommended runner
75
- - draft E2E or checklist path
76
- - missing fixture, selector, assertion, runner, or validation evidence
78
+ - missing fixture, selector, or assertion evidence
79
+ - optional automation adapter selected only after QA design
77
80
  - PR checklist items
78
81
 
79
82
  If the command says a generated recommendation is wrong, do not keep re-prompting the agent with the same context. Update the repo-local manifest after human review:
@@ -0,0 +1,131 @@
1
+ # Architecture
2
+
3
+ QAMap is moving from deterministic PR-to-draft heuristics toward a local, change-aware QA engine. The core must remain useful without cloud services, source upload, or LLM calls.
4
+
5
+ The target pipeline is:
6
+
7
+ ```txt
8
+ commit range + base/head diff
9
+ -> change intent analysis
10
+ -> analyzer adapters
11
+ -> behavior graph
12
+ -> behavior lifecycle + impact selection
13
+ -> runner-independent QA scenarios
14
+ -> Playwright / Maestro / manual adapter
15
+ -> explicit local execution
16
+ -> normalized evidence and verdict
17
+ ```
18
+
19
+ The repository, not a model session, is the source of truth. Commit subjects and bodies provide author intent, source code supplies observable structure, and `.qamap/manifest.yaml` supplies reviewed product intent that commits and code alone cannot prove.
20
+
21
+ ## Change Intent
22
+
23
+ `src/change-intent.ts` reads behavior-bearing commits in the selected base/head range and joins related `feat`, `fix`, `hotfix`, `perf`, and supporting `refactor` commits through normalized domain terms. Added diff symbols provide independent evidence for triggers, conditions, state changes, side effects, and observable outcomes.
24
+
25
+ Each intent contains:
26
+
27
+ - the original commit evidence and source scope;
28
+ - an explicit confidence and `reviewRequired` flag;
29
+ - an ordered behavior lifecycle;
30
+ - runner-independent primary, failure, boundary, and state-transition QA scenarios.
31
+
32
+ One richly evidenced squash commit can reach high confidence. A title without connected diff evidence cannot. Working-tree-only inference is always low confidence and review-required. Release, docs, style, CI, and test-only commits do not become product intents.
33
+
34
+ The analysis is deterministic and local. It does not execute repository code, contact GitHub, upload source, or call an LLM.
35
+
36
+ ## Behavior Graph
37
+
38
+ `src/behavior.ts` defines the framework-neutral intermediate representation. A graph contains stable nodes, typed edges, confidence, evidence, and direct or propagated change impact.
39
+
40
+ Generated graphs identify the shipped contract through `schema/qamap-behavior.schema.json` and `schemaVersion: 1`. The schema URL and node, edge, surface, and evidence enums are exported from the package so adapters and consumers can reject unsupported shapes deterministically.
41
+
42
+ Initial node kinds cover:
43
+
44
+ - domains and flows;
45
+ - routes, screens, endpoints, commands, and artifact surfaces;
46
+ - actions, states, effects, contracts, and assertions;
47
+ - fixtures, locators, and source files.
48
+
49
+ Initial edges describe containment, entrypoints, ordering, expected outcomes, fixture use, locators, implementation sources, and diff impact.
50
+
51
+ Every inferred node must retain provenance. A node without a commit, source, diff, selector, fixture, test, manifest, or named inference reason should not influence a QA verdict.
52
+
53
+ Node and edge ids are content-derived and stable. Re-running analysis against the same repository state must produce the same graph identity even when report timestamps differ.
54
+
55
+ ## Compatibility Adapter
56
+
57
+ The first graph integration uses `qamap.inferred-flow-compat`. It translates the existing E2E flow observations into the new graph so the IR can be introduced without changing existing CLI recommendations.
58
+
59
+ This adapter is a migration bridge, not the final analysis architecture. Framework adapters should gradually emit graph fragments directly, after which draft generation will consume the graph instead of the graph consuming completed drafts.
60
+
61
+ `qamap.change-intent` is the first direct product adapter. It emits intent contracts, lifecycle actions/states/effects, scenario assertions, source links, and commit provenance before an automation runner is selected.
62
+
63
+ ## Analyzer Adapters
64
+
65
+ An analyzer adapter has two operations:
66
+
67
+ ```ts
68
+ interface BehaviorAnalyzerAdapter {
69
+ id: string;
70
+ version: string;
71
+ detect(context): Detection;
72
+ analyze(context): BehaviorGraphFragment;
73
+ }
74
+ ```
75
+
76
+ Detection must be evidence-based and may decline a repository. Analysis failures are isolated and reported as diagnostics so one optional adapter cannot erase useful output from the others.
77
+
78
+ Adapters should be layered:
79
+
80
+ 1. language adapters provide symbols, imports, calls, and schemas;
81
+ 2. framework adapters provide routes, screens, handlers, and lifecycle conventions;
82
+ 3. repository adapters provide manifests, tests, fixtures, and local policy;
83
+ 4. executor adapters compile selected scenarios for an existing runner.
84
+
85
+ Support is reported by capability rather than a single yes/no framework badge:
86
+
87
+ | Level | Contract |
88
+ | --- | --- |
89
+ | Deep | Behavior impact, deterministic scenarios, and local execution are supported. |
90
+ | Structural | Routes, contracts, existing tests, and validation commands are understood. |
91
+ | Generic | Diff and dependency evidence are available; QAMap does not invent a product journey. |
92
+
93
+ TypeScript-based web stacks are the first deep-analysis target because the existing import, route, selector, and fixture signals already provide a useful base. Vue, Nuxt, and SvelteKit should reuse the common web behavior model rather than fork the QA pipeline. Mobile, API, Python, Go, and JVM support should enter through the same adapter contract.
94
+
95
+ ## Manifest Boundary
96
+
97
+ The generated graph is local cache material and should not be committed in full. The verification manifest stores only durable, reviewed knowledge:
98
+
99
+ - important flows and criticality;
100
+ - expected and forbidden outcomes;
101
+ - auth, permission, fixture, and environment requirements;
102
+ - stable anchors and invariants;
103
+ - accepted corrections and suppressions.
104
+
105
+ This keeps first-run setup light while allowing one human correction to improve later PRs deterministically.
106
+
107
+ ## Execution Boundary
108
+
109
+ `qamap qa` remains static and read-only. It must not execute scanned project code.
110
+
111
+ Future execution belongs to an explicit `qamap verify` mode with a visible execution plan. The intended safeguards are:
112
+
113
+ - generated tests live in an operating-system temporary directory by default;
114
+ - target repository files are not modified unless a separate write command is requested;
115
+ - commands, environment variables, network access, and time limits are policy controlled;
116
+ - existing mocks and fixtures are preferred over fabricated data;
117
+ - missing setup becomes `not verifiable`, never a false pass;
118
+ - source code and evidence remain local and no LLM is called.
119
+
120
+ Playwright, Maestro, and other tools are executor implementations. They are not the product-level recommendation shown first to users.
121
+
122
+ ## Migration Order
123
+
124
+ 1. Derive change intent and runner-independent QA scenarios from commit and diff evidence.
125
+ 2. Move route, screen, endpoint, selector, fixture, and contract discovery into graph-producing adapters.
126
+ 3. Compare base and head graphs to select affected behavior rather than relying on file categories alone.
127
+ 4. Compile selected graph paths through Playwright, Maestro, or manual adapters.
128
+ 5. Add explicit, temporary execution and normalized evidence.
129
+ 6. Add manifest accept, reject, and repair commands so reviewed outcomes improve later analysis.
130
+
131
+ Version `0.4.0` establishes the commit-to-intent-to-scenario slice for synthetic web and mobile lifecycle changes. The next minor is reserved for a policy-controlled scenario execution and normalized evidence slice; compatible analyzer and adapter improvements remain patch releases during `0.x` development.
@@ -2,7 +2,7 @@
2
2
 
3
3
  QAMap's unit tests prove that the implementation behaves as coded. The benchmark contract checks a different question: does a representative PR receive a useful QA answer?
4
4
 
5
- `bench.config.json` is committed and runs in CI. Each target under `test/benchmarks/` contains a `base/` repository snapshot and a `head/` overlay. The runner materializes them as a temporary Git repository with a `main` baseline and one feature commit. QAMap reads those repositories but never installs dependencies or executes their code.
5
+ `bench.config.json` is committed and runs in CI. Each target under `test/benchmarks/` contains a `base/` repository snapshot and a `head/` overlay. The runner materializes them as a temporary Git repository with a `main` baseline and one feature commit. Intent fixtures set a synthetic `commitMessage` so commit-to-lifecycle behavior is part of the contract. QAMap reads those repositories but never installs dependencies or executes their code.
6
6
 
7
7
  ## Run the public contract
8
8
 
@@ -14,6 +14,9 @@ The command fails when any target violates its declared expectations. The initia
14
14
 
15
15
  - a web app with no tests;
16
16
  - a web app with Playwright and an existing mock handler;
17
+ - Vue and SvelteKit web changes with framework-native route files;
18
+ - a web preferences change that must become submit, persistence, request-failure, and re-entry QA instead of a generic journey;
19
+ - a mobile reminder change that must become scheduling, calendar, duplicate, resynchronization, and entry-routing QA;
17
20
  - an Expo app with Maestro;
18
21
  - an API service that should produce a contract checklist instead of a browser journey;
19
22
  - a design-token repository that should stay on artifact verification;
@@ -27,13 +30,26 @@ Each target can declare:
27
30
 
28
31
  | Field | Meaning |
29
32
  | --- | --- |
30
- | `runner` | Expected `playwright`, `maestro`, or `manual` recommendation. |
33
+ | `commitMessage` | Synthetic feature-commit message used when materializing a fixture. |
34
+ | `runner` | Expected `playwright`, `maestro`, or `manual` output adapter. Runner correctness alone is not a useful intent benchmark. |
31
35
  | `minFlows` | Minimum number of affected flows. |
36
+ | `minChangeIntents` | Minimum evidence-backed Change Intents. |
37
+ | `minHighConfidenceIntents` | Minimum intents supported strongly enough by commit and diff evidence to avoid mandatory review. |
38
+ | `minCommitBehaviorNodes` | Minimum Behavior Graph nodes carrying commit provenance. |
32
39
  | `minImportPropagatedFlows` | Minimum flows discovered through reverse imports. |
33
40
  | `minDiffAnchoredFlows` | Minimum flows using selector evidence introduced by the diff. |
34
41
  | `minManifestMatches` | Minimum domain, flow, and check matches from an external base manifest. |
35
42
  | `minManifestFlowMatches` | Minimum flow-level matches from the external base manifest. |
36
43
  | `minManifestBackedFlows` | Minimum QA flows that preserve manifest provenance. |
44
+ | `minManifestBehaviorNodes` | Minimum Behavior Graph nodes carrying verification-manifest evidence. |
45
+ | `mustHaveBehaviorKinds` | Behavior Graph node kinds that must be present, such as `flow`, `surface`, `source`, `assertion`, or `locator`. |
46
+ | `mustNameIntents` | Concrete terms that must appear in the inferred intent title. |
47
+ | `mustNotNameIntents` | Misleading terms that must not appear in inferred intent titles. |
48
+ | `mustIncludeLifecycle` | Trigger, condition, action, state, effect, or outcome terms that must survive in the ordered lifecycle. |
49
+ | `mustIncludeQaScenarios` | Failure, boundary, state-transition, or primary QA terms that must be proposed before runner compilation. |
50
+ | `mustFindIntentEvidence` | Commit or diff terms that must remain attached to intent provenance. |
51
+ | `mustTraceScenarioFiles` | Changed files that must appear in at least one scenario's exact direct/supporting base- or head-side diff source. |
52
+ | `maxUntracedCriticalScenarios` | Maximum critical scenarios without a direct/supporting diff source carrying a file and line number. Contextual commit evidence cannot satisfy this contract; lifecycle fixtures keep it at zero. |
37
53
  | `mustReachFiles` | Files that the selected flows must reach. |
38
54
  | `mustNameFlows` | Product terms that must appear in a user-facing flow title. |
39
55
  | `mustNotNameFlows` | Misleading flow-title terms that must not be emitted. |
@@ -41,12 +57,13 @@ Each target can declare:
41
57
  | `mustIncludeSteps` | Behavior terms that must appear in draft steps. |
42
58
  | `mustFindSelectors` | Stable selector evidence that must be recovered from the repository. |
43
59
  | `mustFindSuccessSignals` | Observable outcome text that must appear in the flow's success criteria. |
60
+ | `mustFindEntrypoints` | Route, screen, or command entrypoints that affected flows must recover. |
44
61
  | `mustFindEvidence` | Required evidence or fixture terms that must be reported. |
45
62
  | `mustNotFindEvidence` | Evidence terms that would be false positives for this change. |
46
63
  | `mustRecommendCommands` | Commands the setup or validation path must expose. |
47
64
  | `maxBlankActions` | Maximum malformed or empty draft steps; public fixtures keep this at zero. |
48
65
  | `maxGenericTitles` | Maximum titles ending in generic `primary journey` or `smoke flow` wording. |
49
- | `maxAgentBytes` | Maximum UTF-8 payload size for `qa --format agent`. |
66
+ | `maxAgentBytes` | Maximum UTF-8 payload size for `qa --format agent`. Production output also has a global 8KB ceiling and discloses omitted intent/flow counts. |
50
67
 
51
68
  Set `manifestBaseline: true` on a committed fixture to generate its manifest from the base snapshot into the benchmark temp directory, then pass that external manifest to analysis of the head commit. The fixture repository is never modified by this step. This protects the feedback loop itself: a baseline must affect the next PR, not merely serialize valid YAML.
52
69
 
@@ -62,7 +79,9 @@ node scripts/bench.mjs --baseline bench-results/<file>.json
62
79
 
63
80
  When both files exist, `pnpm bench` prefers the gitignored local config. CI always passes `--config bench.config.json --assert`, so private paths cannot affect the public quality gate.
64
81
 
65
- Saved results include flow titles, draft paths, recall gaps, readiness, agent payload size, and timing. Use a saved baseline to see heuristic movement, but treat the committed expectation contract as the merge gate.
82
+ Saved results include intent titles, lifecycle and scenario terms, scenario trace coverage, flow titles, draft paths, recall gaps, readiness, agent payload size, and timing. Use a saved baseline to see heuristic movement, but treat the committed expectation contract as the merge gate.
83
+
84
+ Every benchmark target also enforces the Behavior Graph base contract: graph schema version 1, at least one graph flow for every planned flow, at least one impacted node for a non-empty diff, and no edge whose endpoint is missing. The table reports `graph n/i` as total nodes versus impacted nodes. These checks keep the graph connected to real PR analysis while framework-specific adapters are introduced incrementally.
66
85
 
67
86
  ## Adding a regression
68
87
 
package/docs/commands.md CHANGED
@@ -21,7 +21,7 @@ Use `pnpm dlx @ivorycanvas/qamap ...` for one-off runs without installing QAMap
21
21
 
22
22
  ## Reading The Output
23
23
 
24
- Human-facing reports (`text` and `markdown` formats) open with an **At a Glance** section: affected behavior, the reviewer question to answer before merge, concrete repository evidence, the proposed draft path, the next command, and the one or two missing trust requirements. When printed to an interactive terminal the report is colorized (headings, statuses, priority tags, inline commands); files written with `--output`, pipes, CI logs, and the machine formats (`json`, `agent`, `sarif`) are always plain. The standard `NO_COLOR` and `FORCE_COLOR` environment variables are honored.
24
+ Human-facing reports (`text` and `markdown` formats) open with an **At a Glance** section: evidence-backed change intent, behavior lifecycle, affected behavior, the reviewer question to answer before merge, concrete repository evidence, the proposed draft path, and missing trust requirements. Runner information appears later as an automation adapter. When printed to an interactive terminal the report is colorized (headings, statuses, priority tags, inline commands); files written with `--output`, pipes, CI logs, and the machine formats (`json`, `agent`, `sarif`) are always plain. The standard `NO_COLOR` and `FORCE_COLOR` environment variables are honored.
25
25
 
26
26
  Draft readiness is reported as a **stage on a fixed four-step journey**, for example `Stage: setup needed (1 of 4) — readiness 0/100`. A fresh repository usually starts at stage 1 — that is the expected starting point, not a failure. Each stage maps to a stable `readiness.level` value in the `json` and `agent` formats, which keeps machine output unchanged:
27
27
 
@@ -37,7 +37,9 @@ Draft readiness is reported as a **stage on a fixed four-step journey**, for exa
37
37
 
38
38
  On a changed branch, QAMap tries to produce reviewable verification artifacts instead of only saying "write more tests":
39
39
 
40
- - a branch-aware verification plan that names the changed domain, actor, trigger, goal, success signal, and edge cases
40
+ - a commit-and-diff-backed change intent with confidence, review requirements, and source evidence
41
+ - an ordered trigger, condition, action, state-change, side-effect, and observable-outcome lifecycle
42
+ - runner-independent primary, failure, boundary, and state-transition QA scenarios
41
43
  - draft Playwright, Maestro, CLI command, or manual checklist files when the repository shape supports them
42
44
  - a repo-level verification manifest loop where humans correct durable flows once and later PRs get sharper route/check/test draft suggestions
43
45
  - a runner setup proposal that explains why Playwright or Maestro fits the changed surface and which files/commands would be created if the team accepts it
@@ -61,9 +63,9 @@ That means QAMap is most valuable when it becomes the team's verification base:
61
63
  | `qamap eval . --base origin/main --head HEAD --pr-body-file pr-body.md` | Score change readiness across intent, risk, tests, and review size. |
62
64
  | `qamap github-action . --mode review --base origin/main --head HEAD` | Generate GitHub Action annotations, step summary, and PR comment body. |
63
65
  | `qamap test-plan . --base origin/main --head HEAD --include-working-tree` | Suggest domain test scenarios for changed files. |
64
- | `qamap qa . --base origin/main --head HEAD` | One-command PR QA: affected flows, missing QA evidence, a PR checklist, and E2E starter drafts, opening with At a Glance. |
65
- | `qamap qa . --base origin/main --head HEAD --format agent` | The same decision content as one ~2-4KB JSON line for coding agents — a versioned contract documented in [docs/agent-format.md](agent-format.md). |
66
- | `qamap e2e plan . --base origin/main --head HEAD` | Suggest E2E runner, bootstrap steps, user flows, coverage targets, existing test evidence, and missing testability hooks for changed files. |
66
+ | `qamap qa . --base origin/main --head HEAD` | One-command PR QA: change intent, behavior lifecycle, QA scenarios, affected flows, missing evidence, and optional automation drafts. |
67
+ | `qamap qa . --base origin/main --head HEAD --format agent` | The same decision content as one compact JSON line for coding agents — a versioned contract documented in [docs/agent-format.md](agent-format.md). |
68
+ | `qamap e2e plan . --base origin/main --head HEAD` | Derive change intent and QA scenarios, then map them to coverage, test evidence, testability gaps, and an automation adapter. |
67
69
  | `qamap e2e plan . --base origin/main --head HEAD --record-history` | Save a compact local run snapshot under `.qamap/runs/` while keeping JSON/Markdown output usable. |
68
70
  | `qamap e2e setup . --runner playwright` | Explicitly apply the accepted runner setup and create the first changed-flow E2E draft without overwriting existing files. |
69
71
  | `qamap e2e draft . --base origin/main --head HEAD --dry-run` | Preview generated Maestro, Playwright, or manual E2E drafts without writing files. |
@@ -90,7 +92,9 @@ For monorepos, pass `--workspace-root` when scanning a package. Package-local ch
90
92
 
91
93
  `qamap test-plan` turns changed file paths into a review-ready domain test checklist. It also discovers common validation commands from `package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, Gradle files, and Maven `pom.xml`. Add `--include-working-tree` for local, uncommitted changes while iterating.
92
94
 
93
- `qamap e2e plan` turns changed file paths into a first-pass E2E testing plan. It detects whether a project looks like Expo/React Native, web, API/service, or CLI package, recommends a runner such as Maestro or Playwright, starts backend services with an API contract checklist, starts executable packages with a CLI command verification checklist, suggests bootstrap steps for repos with little or no test history, suggests domain language for the changed behavior, suggests candidate user flows, adds coverage targets, compares those targets with existing test-suite evidence when tests are present, flags API-dependent flows that need mock or fixture responses, and points out missing stable selectors such as `testID` or `data-testid` before anyone starts writing tests from a blank file.
95
+ `qamap e2e plan` first reads behavior-bearing commits in the selected base/head range. Related `feat`, `fix`, `hotfix`, `perf`, and supporting `refactor` commits are grouped into change intents, then connected to added diff symbols. The result is an evidence-backed lifecycle and a set of runner-independent primary, failure, boundary, and state-transition scenarios. Low-confidence or working-tree-only intent remains review-required instead of becoming trusted regression policy.
96
+
97
+ After that judgment, QAMap detects whether the target looks like Expo/React Native, web, API/service, CLI, or another repository shape and selects a Maestro, Playwright, or manual output adapter. The plan compares scenario targets with existing test evidence, identifies required mocks and fixtures, and reports missing stable selectors such as `testID` or `data-testid`. Runner detection is an implementation detail, not the first value shown to the user.
94
98
 
95
99
  The plan also includes an execution profile: detected start command, test command, Playwright `baseURL`, mobile app id, runner config files, env fixture files, confidence, and blockers. This keeps generated E2E drafts honest about whether they are runnable candidates or still review-only scaffolds.
96
100
 
@@ -100,7 +104,7 @@ When a repository does not already have the selected E2E runner, the plan includ
100
104
 
101
105
  When run at a monorepo root, the E2E plan also reports changed app/package targets. This helps a maintainer move from a broad workspace diff to scoped commands such as `qamap e2e plan services/listing --workspace-root . --base origin/main --head HEAD`, where package-specific runner detection and flow naming are usually sharper.
102
106
 
103
- Each candidate flow also includes a flow language brief: actor, trigger, goal, success signal, reviewer question, and edge cases. The brief keeps generated tests tied to product behavior rather than only changed file names.
107
+ Each candidate flow also includes a flow language brief: actor, trigger, goal, success signal, reviewer question, and edge cases. Intent-backed flows additionally retain the original commits, ordered lifecycle, QA scenarios, confidence, and review requirement so generated tests can be traced to evidence rather than only changed file names.
104
108
 
105
109
  The bootstrap section answers what must happen before generated drafts can be treated as real regression coverage. For example, a testless web project can get required steps for Playwright setup, first draft generation, stable selector work, fixture/mock data, and missing validation evidence, plus recommended steps for `.qamap/manifest.yaml`, `.qamap/domains.yml`, `.qamap/flows.yml`, and local history recording.
106
110
 
@@ -207,6 +211,8 @@ Pass `--record-history` when you want QAMap to keep a compact local snapshot of
207
211
 
208
212
  The draft result is meant to be useful as a PR artifact, not only as generated files. Markdown and JSON output include:
209
213
 
214
+ - `changeAnalysis`: commit-backed intents, confidence, review requirements, source evidence, lifecycle stages, and runner-independent QA scenarios
215
+ - intent-backed draft files retain `intentId`, `intentConfidence`, `lifecycle`, and `qaScenarios`; generated files include the same evidence as comments
210
216
  - `languageBrief`: actor, trigger, goal, success signal, reviewer question, and edge cases for each draft file
211
217
  - `promotionStatus`: whether the draft is a `commit-candidate`, `needs-review`, or `low-signal`
212
218
  - `runnableStatus`: whether the draft is a `runnable-candidate`, `near-runnable`, or `review-only`
@@ -15,20 +15,31 @@ The output should be specific enough to paste into a PR comment:
15
15
  ```txt
16
16
  # QAMap QA Draft
17
17
 
18
+ At a Glance
19
+ - Change intent: Submit checkout and persist the confirmed order [high]
20
+ - Behavior lifecycle: trigger: Submit checkout -> side-effect: Create order -> observable-outcome: Show order confirmation
21
+ - Affected behavior: Submit checkout and persist the confirmed order
22
+
23
+ Change Intent Evidence
24
+ - Commit: feat: submit checkout and persist the confirmed order
25
+ - Critical scenario: Submit checkout and persist the confirmed order
26
+ - Assert: order confirmation is visible and persisted
27
+ - Recommended scenario: Failure, timeout, and retry handling
28
+
18
29
  Summary
19
30
  - Project: Web
20
- - Recommended runner: Playwright
31
+ - Automation adapter: Playwright
21
32
  - Manifest: not found; using repo signals and PR diff only
22
33
  - Stage: almost runnable (3 of 4)
23
34
 
24
35
  PR Comment Draft
25
- - Affected flow: Checkout UI smoke flow
36
+ - Affected flow: Submit checkout and persist the confirmed order
26
37
  - User journey: Customer -> Open route /checkout -> Complete checkout with realistic form data
27
38
  - Success signal: confirmation state is visible after submit
28
39
  - Changed files: src/pages/checkout/index.tsx
29
40
 
30
41
  Suggested E2E / QA Draft
31
- - tests/e2e/checkout-ui-smoke-flow.spec.ts: near runnable
42
+ - tests/e2e/submit-checkout-and-persist-the-confirmed-order.spec.ts: near runnable
32
43
  - Open route /checkout.
33
44
  - Fill checkout email.
34
45
  - Submit checkout.
@@ -139,7 +150,7 @@ flows:
139
150
 
140
151
  ```txt
141
152
  Project: Web
142
- Recommended runner: Playwright
153
+ Automation adapter: Playwright
143
154
  Execution profile: high
144
155
  Start command: pnpm run dev
145
156
  Test command: pnpm run test:e2e
@@ -224,7 +235,7 @@ For an Expo or React Native change, QAMap should recommend Maestro and carry mob
224
235
 
225
236
  ```txt
226
237
  Project: Expo / React Native
227
- Recommended runner: Maestro
238
+ Automation adapter: Maestro
228
239
 
229
240
  Flow: Ink Drawing UI smoke flow
230
241
  Actor: User
@@ -269,7 +280,7 @@ For backend changes such as `src/v1/listing/utils.ts`, QAMap should not invent a
269
280
 
270
281
  ```txt
271
282
  Project: API / service
272
- Recommended runner: Manual
283
+ Automation adapter: Manual
273
284
 
274
285
  Flow: Listing API contract smoke checklist
275
286
  Actor: API consumer or upstream service
@@ -296,7 +307,7 @@ For an npm package that exposes `package.json` bin entries, QAMap should not inv
296
307
 
297
308
  ```txt
298
309
  Project: CLI
299
- Recommended runner: Manual
310
+ Automation adapter: Manual
300
311
 
301
312
  Flow: CLI command verification checklist
302
313
  Actor: CLI user or maintainer
@@ -319,30 +330,17 @@ The manual draft should name concrete command evidence:
319
330
 
320
331
  ## Test-Light Project
321
332
 
322
- When a project has little or no E2E setup, QAMap should move the user toward a concrete starter draft, not just a task list:
333
+ When a project has little or no E2E setup, `qamap qa` still returns the same runner-independent QA judgment. Runner setup remains an explicit team choice:
323
334
 
324
335
  ```txt
325
- ## First E2E Draft Bootstrap
326
-
327
- QAMap did not find committed test files for this target. The next step is to create the first runnable starter draft, not to stop at a checklist.
328
-
329
- - Recommended first runner: Playwright
330
- - Create command: `qamap e2e setup . --runner playwright`
331
- - Install command: `pnpm add -D @playwright/test`
332
- - Draft files QAMap can create:
333
- - `playwright.config.ts`
334
- - `tests/e2e/`
335
- - `tests/e2e/checkout-primary-journey.spec.ts`
336
- - Files to update: `package.json`
336
+ ## Optional Automation
337
337
 
338
- Bootstrap summary:
339
- 4 required bootstrap steps must be resolved before generated E2E drafts should be treated as regression coverage.
338
+ The QA judgment above does not require adopting this adapter.
340
339
 
341
- Required:
342
- - Configure Playwright before making drafts required
343
- - Create the first changed-flow E2E draft
344
- - Add deterministic fixture or mock responses
345
- - Add stable selectors for changed UI surfaces
340
+ - Adapter candidate: Playwright
341
+ - Draft target: `tests/e2e/checkout-primary-journey.spec.ts`
342
+ - Preview or create a draft: `qamap e2e draft . --base main --head HEAD`
343
+ - If the team accepts this adapter, inspect its setup proposal: `qamap e2e setup . --runner playwright`
346
344
  ```
347
345
 
348
346
  Draft action items should make the next developer action explicit:
@@ -460,9 +458,12 @@ Related Changed Files
460
458
  Good QAMap output should answer these questions quickly:
461
459
 
462
460
  - What behavior did this branch probably change?
461
+ - Which commits and diff symbols support that intent, and how confident is the inference?
462
+ - What trigger, condition, state change, side effect, and observable outcome form its lifecycle?
463
+ - Which primary, failure, boundary, and state-transition checks follow from that lifecycle?
463
464
  - What does the team call that behavior?
464
465
  - Who or what exercises it?
465
- - Which runner or checklist is the right first shape?
466
+ - Which existing automation adapter can compile the selected QA scenario?
466
467
  - Which setup, selector, fixture, or validation gap blocks this from becoming real regression coverage?
467
468
 
468
469
  If the output only says "make an E2E test" without these answers, it is not ready for the current release bar.
@@ -15,6 +15,7 @@ src/features/checkout/submitCheckout.ts
15
15
  The reviewer wants to know:
16
16
 
17
17
  - Which user flow is affected?
18
+ - Which diff hunk caused each QA scenario to be proposed?
18
19
  - Should this become a Playwright test, a manual checklist, or only a review note?
19
20
  - What blocks the generated draft from being trusted as regression coverage?
20
21
 
@@ -26,9 +27,9 @@ Run QAMap on the branch:
26
27
  pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD
27
28
  ```
28
29
 
29
- `qa` is important for first contact. It lets maintainers preview the PR comment draft, affected flow, draft path, readiness, and blockers without writing files.
30
+ `qa` is important for first contact. It lets maintainers review change intent, behavior lifecycle, scenario confidence, exact diff sources, and a PR checklist without writing files or selecting a test runner.
30
31
 
31
- When the team wants actual draft files:
32
+ After the team accepts a scenario and wants executable coverage:
32
33
 
33
34
  ```sh
34
35
  pnpm dlx @ivorycanvas/qamap e2e draft . --base origin/main --head HEAD --dry-run
@@ -145,11 +146,15 @@ Input
145
146
  ```txt
146
147
  Output
147
148
  - PR comment/checklist draft
149
+ - commit-backed change intent and confidence
150
+ - ordered behavior lifecycle
151
+ - primary, failure, boundary, and state-transition QA scenarios
152
+ - scenario-level confidence, review status, and exact commit or head-side `file:line` sources
148
153
  - changed domain language
149
154
  - candidate user flow
150
155
  - manifest evidence when a repo-local flow/check matches
151
- - recommended runner
152
- - draft file path
156
+ - optional automation adapter selected after QA design
157
+ - optional draft file path
153
158
  - flow language brief
154
159
  - runnable status
155
160
  - self-check status
@@ -157,33 +162,40 @@ Output
157
162
  - blockers that explain why a draft is not ready yet
158
163
  ```
159
164
 
160
- The result should not stop at "selector needed" or "fixture needed." A useful QAMap draft should say which flow changed, which test file would be generated, which checks it covers, why those checks were selected, and where to update the manifest if the recommendation is wrong.
165
+ The result should not stop at "selector needed" or "fixture needed." A useful QAMap report should say which behavior changed, which scenarios follow, which exact hunk supports each scenario, what remains uncertain, and where to update the manifest if the recommendation is wrong. Automation comes after that review.
161
166
 
162
167
  ## Markdown Preview
163
168
 
164
169
  ```txt
165
170
  # QAMap QA Draft
166
171
 
172
+ At a Glance
173
+ - Change intent: Submit notification preferences and show the saved state [high]
174
+ - Behavior lifecycle: trigger -> state-change -> side-effect -> observable-outcome
175
+
167
176
  Summary
168
177
  - Project: Web
169
- - Recommended runner: Playwright
170
178
  - Manifest: .qamap/manifest.yaml
171
- - Stage: almost runnable (3 of 4)
172
179
 
173
- PR Comment Draft
174
- - Affected flow: Checkout purchase
175
- - Suggested draft: tests/e2e/checkout-purchase.spec.ts
176
- - User journey: Customer -> Open route /checkout -> Complete checkout with realistic form data
177
- - Success signal: confirmation state is visible after submit
180
+ QA scenarios
181
+ - [critical] changed preference lifecycle [high]
182
+ - Source: src/pages/preferences.tsx:17, symbol onClick
183
+ - Assert: the saved state is visible
184
+ - [recommended] failure, timeout, and retry handling [medium; review required]
185
+ - Source: src/pages/preferences.tsx:7, symbol fetch
186
+ - Assert: retries do not duplicate requests or side effects
178
187
 
179
- Missing evidence before trusting this PR
180
- - [required] fixture: Add deterministic payment/customer fixture data.
181
- - [recommended] selector: Confirm stable selectors for changed checkout controls.
188
+ Evidence gaps in this QA proposal
189
+ - [required] fixture: Add deterministic response data for /api/preferences.
182
190
 
183
191
  PR checklist
184
- - [ ] Review tests/e2e/checkout-purchase.spec.ts.
185
- - [ ] Confirm success and failed-response assertions.
192
+ - [ ] Review each QA scenario and its diff source.
193
+ - [ ] Confirm success and failed-response assertions for /api/preferences.
186
194
  - [ ] Run pnpm run test:e2e.
195
+
196
+ Optional automation
197
+ - Adapter candidate: Playwright
198
+ - qamap e2e draft . --base origin/main --head HEAD
187
199
  ```
188
200
 
189
201
  ## Draft Shape