@ivorycanvas/qamap 0.3.0

package/docs/e2e-output-examples.md

@@ -0,0 +1,464 @@
+# E2E Output Examples
+
+These examples show the shape of QAMap output that should be good enough for the current public release. They are intentionally short snippets, not full generated files. The important property is that they are derived from repository structure, git changes, manifests, and test evidence without an LLM call.
+
+## PR QA Skill Preview
+
+First contact should work without a manifest:
+
+```sh
+pnpm dlx @ivorycanvas/qamap qa . --base origin/main --head HEAD
+```
+
+The output should be specific enough to paste into a PR comment:
+
+```txt
+# QAMap QA Draft
+
+Summary
+- Project: Web
+- Recommended runner: Playwright
+- Manifest: not found; using repo signals and PR diff only
+- Readiness: near-runnable
+
+PR Comment Draft
+- Affected flow: Checkout UI smoke flow
+- User journey: Customer -> Open route /checkout -> Complete checkout with realistic form data
+- Success signal: confirmation state is visible after submit
+- Changed files: src/pages/checkout/index.tsx
+
+Suggested E2E / QA Draft
+- tests/e2e/checkout-ui-smoke-flow.spec.ts: near runnable
+- Open route /checkout.
+- Fill checkout email.
+- Submit checkout.
+- Assert confirmation state is visible after submit.
+
+Missing evidence before trusting this PR
+- [required] fixture: Add deterministic fixture or mock data.
+- [recommended] selector: Confirm stable selectors.
+
+PR checklist
+- [ ] Review the generated draft path.
+- [ ] Answer the reviewer question for the affected flow.
+- [ ] Run local validation: pnpm run test:e2e
+```
+
+If this recommendation is useful but slightly wrong, the next step is not another long AI prompt. Generate and correct repo-local QA memory:
+
+```sh
+pnpm exec qamap manifest init .
+```
+
+Then future `qamap qa` runs can use `.qamap/manifest.yaml` to produce more precise flow names, checks, anchors, and repair paths.
+
+## Verification Manifest Feedback
+
+When a repository has `.qamap/manifest.yaml`, QAMap should explain why a recommendation happened and how a maintainer can correct it:
+
+```txt
+Manifest recommendations: 3
+
+Campaign Application Complete `campaign-application-complete`
+- Kind: flow
+- Confidence: high
+- Why this was recommended: Changed files match anchors for the Campaign Application Complete flow.
+- Evidence sources: product-qa
+- Manifest evidence: .qamap/manifest.yaml > flows.campaign-application-complete.anchors
+- If this is wrong: update .qamap/manifest.yaml > flows.campaign-application-complete.anchors
+- Next actions:
+  - Draft or review E2E coverage for the Campaign Application Complete flow.
+  - Cover the declared checks: Submit content URL successfully; Show validation error for invalid content URL.
+- Repair hints:
+  - If these files do not belong to this flow, update .qamap/manifest.yaml > flows.campaign-application-complete.anchors.
+  - If the recommended assertions feel vague, rewrite .qamap/manifest.yaml > flows.campaign-application-complete.checks in team language.
+```
+
+This is the feedback loop: static analysis proposes a baseline, humans correct durable manifest entries, and future E2E recommendations become more specific without spending another LLM prompt on the same explanation.
+
+`qamap manifest validate .` checks whether that repo-local knowledge is usable:
+
+```txt
+QAMap Manifest Validate
+Status: valid
+Manifest: .qamap/manifest.yaml
+Issues: 0 errors, 0 warnings, 0 info
+```
+
+`qamap manifest explain . --base origin/main --head HEAD` makes a single branch debuggable:
+
+```txt
+QAMap Manifest Explain
+Changed files: 1
+Matches: 3
+
+Matches:
+- Campaign Application Complete (flow, high)
+  Why: Changed files match anchors for the Campaign Application Complete flow.
+  Evidence: .qamap/manifest.yaml > flows.campaign-application-complete.anchors
+  If wrong: update .qamap/manifest.yaml > flows.campaign-application-complete.anchors
+  Checks: Submit content URL successfully; Show validation error for invalid content URL
+```
+
+When that flow includes an entry route and checks, `qamap e2e draft` promotes it ahead of heuristic drafts:
+
+```ts
+// Verification manifest evidence:
+// - Flow: Campaign Application Complete (campaign-application-complete)
+// - Entry route: /campaign/official/applicationComplete
+// - Required checks:
+//   - [ ] Submit content URL successfully
+//   - [ ] Show validation error for invalid content URL
+
+test("Campaign Application Complete", async ({ page }) => {
+  await test.step("Open route /campaign/official/applicationComplete.", async () => {
+    await page.goto("/campaign/official/applicationComplete");
+  });
+});
+```
+
+## Web Core Flow
+
+When a web app declares a core flow:
+
+```yaml
+flows:
+  - id: checkout-purchase
+    name: Checkout purchase
+    priority: critical
+    files:
+      - src/pages/checkout/**
+    routes:
+      - /checkout
+    checks:
+      - Complete checkout with a valid payment method.
+      - Verify declined payment recovery.
+```
+
+`qamap e2e plan` should prefer the team-approved name:
+
+```txt
+Project: Web
+Recommended runner: Playwright
+Execution profile: high
+Start command: pnpm run dev
+Test command: pnpm run test:e2e
+Base URL: http://127.0.0.1:4173
+Matched core flows: 1
+
+Flow: Checkout purchase UI smoke flow
+Actor: Customer
+Trigger: Open route /checkout.
+Success signal: Verify declined payment recovery
+```
+
+The Playwright draft should read like the product journey:
+
+```ts
+test("Checkout purchase", async ({ page }) => {
+  await test.step("Open route /checkout.", async () => {
+    await page.goto("/checkout");
+  });
+
+  await test.step("Complete checkout with a valid payment method.", async () => {
+    await page.getByTestId("checkout-submit").click();
+  });
+});
+```
+
+When QAMap can infer durable browser controls, the draft should prefer executable Playwright locators over placeholder text:
+
+```ts
+await test.step("Fill profile email.", async () => {
+  // Step intent: Fill profile email.
+  await page.getByPlaceholder("Profile email").fill("qamap@example.com");
+});
+
+await test.step("Save settings.", async () => {
+  // Step intent: Save settings.
+  await page.getByRole("button", { name: "Save settings" }).click();
+});
+```
+
+The draft result should also explain whether the generated file passed static runner checks:
+
+```txt
+Draft self-check: pass
+Command: pnpm run test:e2e
+Summary: Playwright draft passed static runner checks.
+```
+
+Before writing files, `--dry-run` should expose the same readiness data with preview status:
+
+```txt
+Mode: dry run (no files were written)
+Files: 0 created, 1 previewed, 0 skipped
+
+- preview: `tests/e2e/checkout-purchase.spec.ts` (Checkout purchase)
+  - source: core-flow
+  - runnable status: near-runnable
+  - self-check: pass
+```
+
+For framework-native routing, QAMap should preserve the route a user can actually open rather than framework-only folder syntax. A Next App Router file such as `src/app/(shop)/products/[productId]/page.tsx` should become `/products/:productId`, and a concrete link such as `/products/demo-product` can seed the generated route params:
+
+```ts
+const routeParams = {
+  productId: "demo-product",
+};
+
+await page.goto(`/products/${routeParams.productId}`);
+```
+
+React Router config should work similarly when route objects expose path values:
+
+```ts
+createBrowserRouter([
+  { path: "/reports/:reportId", element: <ReportPage /> },
+]);
+```
+
+## Expo / React Native Flow
+
+For an Expo or React Native change, QAMap should recommend Maestro and carry mobile selectors into the draft:
+
+```txt
+Project: Expo / React Native
+Recommended runner: Maestro
+
+Flow: Ink Drawing UI smoke flow
+Actor: User
+Trigger: Open the Ink Drawing screen.
+Selectors: testID=ink-save-button, testID=record-mode-ink
+```
+
+Example Maestro draft shape:
+
+```yaml
+appId: ${APP_ID}
+---
+- launchApp
+- tapOn: { id: "record-mode-ink" }
+- tapOn: { id: "ink-save-button" }
+```
+
+When the changed file name exposes a narrower user action, the generated flow should use that action instead of stopping at a broad domain journey:
+
+```txt
+Changed file: src/features/offer/components/ContentUrlSubmitModal.tsx
+Domain term: Offer
+Generated scenario: Offer Content URL Submit
+Generated draft path: .maestro/offer-content-url-submit.yaml
+```
+
+The draft should then carry the same product-action wording into the runnable skeleton:
+
+```yaml
+# Flow: Offer Content URL Submit
+# Domain scenario: Offer Content URL Submit
+# Intent: Verify the changed "Content URL Submit" behavior inside Offer instead of stopping at a generic primary journey.
+appId: ${APP_ID}
+---
+- launchApp
+- tapOn: { id: "offer-content-url-submit" }
+```
+
+## API / Service Contract
+
+For backend changes such as `src/v1/offer/utils.ts`, QAMap should not invent a browser journey. It should infer the domain word and stay contract-focused:
+
+```txt
+Project: API / service
+Recommended runner: Manual
+
+Flow: Offer API contract smoke checklist
+Actor: API consumer or upstream service
+Trigger: Call the endpoint, handler, or service path affected by src/v1/offer/utils.ts.
+Success signal: the changed contract returns the expected status, response shape, auth behavior, and failure handling
+```
+
+The manual draft should stay actionable:
+
+```md
+# Offer API contract
+
+## Steps
+
+- [ ] Call the changed endpoint, client, command, or handler with a valid request.
+- [ ] Verify the response shape, status, and parsed data match the public contract.
+- [ ] Verify invalid input, authorization failure, timeout, and server-error handling.
+- [ ] Check backward compatibility for existing callers.
+```
+
+## CLI Command Verification
+
+For an npm package that exposes `package.json` bin entries, QAMap should not invent a UI journey. It should stay focused on the command contract users run in terminals, scripts, and CI:
+
+```txt
+Project: CLI
+Recommended runner: Manual
+
+Flow: CLI command verification checklist
+Actor: CLI user or maintainer
+Trigger: Run the CLI command path affected by src/cli.ts.
+Success signal: the command returns the expected stdout, stderr, generated files, and exit code for valid and invalid inputs
+```
+
+The manual draft should name concrete command evidence:
+
+```md
+# CLI command verification checklist
+
+## Steps
+
+- [ ] Build or install the package in a clean local environment.
+- [ ] Run the changed command with a representative valid argument set.
+- [ ] Verify stdout, stderr, generated files, and exit code match the intended behavior.
+- [ ] Run one invalid, missing-argument, or unsupported-input path and verify the failure message and exit code.
+```
+
+## Test-Light Project
+
+When a project has little or no E2E setup, QAMap should move the user toward a concrete starter draft, not just a task list:
+
+```txt
+## First E2E Draft Bootstrap
+
+QAMap did not find committed test files for this target. The next step is to create the first runnable starter draft, not to stop at a checklist.
+
+- Recommended first runner: Playwright
+- Create command: `qamap e2e setup . --runner playwright`
+- Install command: `pnpm add -D @playwright/test`
+- Draft files QAMap can create:
+  - `playwright.config.ts`
+  - `tests/e2e/`
+  - `tests/e2e/checkout-primary-journey.spec.ts`
+- Files to update: `package.json`
+
+Bootstrap summary:
+4 required bootstrap steps must be resolved before generated E2E drafts should be treated as regression coverage.
+
+Required:
+- Configure Playwright before making drafts required
+- Create the first changed-flow E2E draft
+- Add deterministic fixture or mock responses
+- Add stable selectors for changed UI surfaces
+```
+
+Draft action items should make the next developer action explicit:
+
+```txt
+Action summary:
+- readiness score: 64/100 (needs-work)
+- runnable status: near-runnable
+- self-check: pass or warning based on generated starter code and execution profile
+- top blocker: No Playwright config file was detected.
+- required runner: Create `playwright.config.ts` with `testDir`, `use.baseURL`, and `webServer.command` when QAMap inferred the dev URL from scripts.
+- required fixture: Add deterministic fixture or mock data
+- required validation: Resolve missing validation evidence
+- recommended manifest: Promote durable product language
+```
+
+## API-Dependent Client Flow
+
+When a client-side change calls an API path but the branch does not include backend or fixture evidence, QAMap should name that as a readiness gap and still give the tester a concrete mock slot:
+
+If reusable repo-local evidence already exists, the PR QA output should point at it instead of only saying "add a fixture":
+
+```txt
+Missing evidence before trusting this PR
+- [recommended] fixture: Confirm fixture coverage - Reuse or extend existing fixture/mock evidence for this flow: src/services/devSeedService.ts, src/services/reportMockService.ts.
+```
+
+```ts
+const mockApiResponses = {
+  "**/api/orders/fixture-order-id": {
+    status: 200,
+    body: {
+      ok: true,
+      source: "qamap-draft",
+    },
+  },
+};
+
+// Replace sample responses with deterministic fixtures from the target domain before promoting this draft.
+for (const [urlPattern, response] of Object.entries(mockApiResponses)) {
+  await page.route(urlPattern, async (route) => {
+    await route.fulfill({
+      status: response.status,
+      contentType: "application/json",
+      body: JSON.stringify(response.body),
+    });
+  });
+}
+```
+
+This is useful for PRs where the UI can be built against mockdata before the server implementation is available. The generated file should still report fixture readiness as missing or partial until a reviewer replaces the sample response with domain-correct success, empty, unauthorized, timeout, or server-error fixtures.
+
+When the PR changes the endpoint implementation itself, QAMap should not hide that contract behind a synthetic response. In that case the Playwright draft records the endpoint as an observed API pattern instead of adding a response mock:
+
+```ts
+const changedApiEndpointPatterns = [
+  "**/api/checkout",
+];
+const observedChangedApiResponses: Array<{ url: string; status: number }> = [];
+page.on("response", (response) => {
+  if (changedApiEndpointPatterns.some((pattern) => response.url().includes(pattern.replace(/^\*\*/, "")))) {
+    observedChangedApiResponses.push({ url: response.url(), status: response.status() });
+  }
+});
+```
+
+## Monorepo Package
+
+When run at the workspace root, QAMap should identify changed package targets before asking the maintainer to choose a final runner:
+
+```sh
+qamap e2e plan . --base main --head HEAD
+```
+
+Expected root-level behavior:
+
+```txt
+Changed App/Package Targets
+
+| Target | Package | Project | Runner | Scoped Command |
+| services/offer | offer | Web | Playwright | qamap e2e plan services/offer --workspace-root . --base main --head HEAD |
+| apps/mobile | @acme/mobile | Expo / React Native | Maestro | qamap e2e plan apps/mobile --workspace-root . --base main --head HEAD |
+```
+
+For package scans, QAMap should use workspace policy without leaking workspace-root paths into package-local drafts:
+
+```sh
+qamap e2e plan services/offer --workspace-root . --base main --head HEAD
+```
+
+Expected behavior:
+
+```txt
+Workspace root: .
+Package root: services/offer
+Matched core flow: Offer submit
+Changed files: src/features/offer/submit.ts
+Generated draft path: docs/e2e/offer-submit.md
+```
+
+The draft should mention package-local files:
+
+```md
+Related Changed Files
+
+- `src/features/offer/submit.ts`
+```
+
+## What Good Output Feels Like
+
+Good QAMap output should answer these questions quickly:
+
+- What behavior did this branch probably change?
+- What does the team call that behavior?
+- Who or what exercises it?
+- Which runner or checklist is the right first shape?
+- Which setup, selector, fixture, or validation gap blocks this from becoming real regression coverage?
+
+If the output only says "make an E2E test" without these answers, it is not ready for the current release bar.

package/docs/ecosystem.md

@@ -0,0 +1,41 @@
+# Agent Ecosystem Coverage
+
+AI coding agents now read repository instructions, connect to MCP servers, run tool hooks, and prepare pull requests. QAMap tracks the repository files that shape those behaviors without executing project code.
+
+## Instruction Surfaces
+
+QAMap treats these files as agent instruction surfaces:
+
+| Ecosystem | Files |
+| --- | --- |
+| Shared / Codex-style | `AGENTS.md`, `AGENTS.override.md` |
+| Claude Code | `CLAUDE.md`, `.claude/CLAUDE.md`, `.claude/rules/*.md` |
+| Cursor | `.cursorrules`, `.cursor/rules/*.md`, `.cursor/rules/*.mdc` |
+| GitHub Copilot | `.github/copilot-instructions.md`, `.github/instructions/*.instructions.md` |
+| Gemini CLI | `GEMINI.md` |
+
+These files are scanned for missing guidance, conflicting guidance, and suspicious instruction text.
+
+## Tool And Settings Surfaces
+
+QAMap statically inspects committed MCP and agent settings files:
+
+| Surface | Files | Checks |
+| --- | --- | --- |
+| MCP config | `.mcp.json`, `mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`, `claude_desktop_config.json` | unreadable JSON, risky commands, committed secret-like env values |
+| Agent settings | `.claude/settings.json`, `.claude/settings.local.json`, `.gemini/settings.json` | MCP servers, risky hooks, broad shell permissions |
+
+The scanner does not run MCP servers, hooks, package scripts, or project code.
+
+## Source Signals
+
+These references guide the current coverage:
+
+- [OpenAI Codex AGENTS.md](https://developers.openai.com/codex/cloud/agents-md/) and the community [AGENTS.md](https://agents.md/) convention.
+- [Claude Code memory](https://docs.anthropic.com/en/docs/claude-code/memory), [settings](https://docs.anthropic.com/en/docs/claude-code/settings), and [hooks](https://docs.anthropic.com/en/docs/claude-code/hooks).
+- [GitHub Copilot custom instructions](https://docs.github.com/copilot/customizing-copilot/adding-custom-instructions-for-github-copilot).
+- [Cursor rules](https://cursor.com/docs/rules) and [Cursor MCP](https://cursor.com/docs/mcp).
+- [Gemini CLI configuration](https://google-gemini.github.io/gemini-cli/docs/get-started/configuration.html) and [Gemini MCP servers](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html).
+- [Model Context Protocol security best practices](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices).
+
+QAMap should keep this list practical rather than exhaustive: add a surface when it is common enough that maintainers would reasonably expect a repo preflight check to notice it.

package/docs/eval.md

@@ -0,0 +1,52 @@
+# QAMap Eval
+
+`qamap eval` scores whether a branch is ready for human review in an AI-assisted workflow.
+
+It does not try to prove that the code is correct. Instead, it checks whether reviewers have enough evidence to trust, challenge, and verify the change without absorbing unnecessary cognitive load.
+
+For a complete PR-facing report that also includes QAMap review findings and suggested domain tests, use `qamap verify`.
+
+## Usage
+
+```sh
+qamap verify . --base origin/main --head HEAD --pr-body-file pr-body.md
+qamap eval . --base origin/main --head HEAD --format markdown
+qamap eval . --base origin/main --head HEAD --pr-body-file pr-body.md
+qamap eval services/offer --workspace-root . --base origin/main --head HEAD --include-working-tree
+```
+
+The GitHub Action can append the same report to the PR comment. On pull request events it reads the PR body from `GITHUB_EVENT_PATH`.
+
+## Gates
+
+Each gate is scored from `0` to `2`.
+
+| Gate | What it checks |
+| --- | --- |
+| Validation commands | The project exposes a real test command and supporting commands such as typecheck, lint, build, e2e, tox, Ruff, mypy, `go vet`, clippy, or Maven verify. |
+| Changed test coverage | Source changes are paired with changed test files, or at least a runnable test command exists. |
+| Intent capture | The PR body, decision docs, or PR template capture the problem, rationale, context, alternatives, or tradeoffs. |
+| Risk explanation | Risky surfaces such as config, workflows, API contracts, auth, billing, migrations, and env files include risk or rollback context. |
+| Domain test plan | Changed files can be mapped to focused domain verification scenarios. |
+| Review size | The branch is small enough to review without unnecessary verification tax. |
+
+`validationCommands` in `qamap.config.json` can supply commands for custom stacks or monorepos when standard project files are not enough.
+
+## Ratings
+
+| Rating | Meaning |
+| --- | --- |
+| `strong` | The branch has clear verification evidence and should be easy to review. |
+| `ready` | The branch is probably reviewable, with some follow-up still useful. |
+| `needs-work` | Reviewers should ask for more tests, context, risk notes, or a smaller diff. |
+| `high-risk` | The branch is expensive or risky to verify; add evidence before relying on it. |
+
+## Why This Exists
+
+AI-assisted code often looks plausible before it is truly understood. That creates verification tax, cognitive debt, and intent debt:
+
+- verification tax: humans must spend extra time proving the generated result is safe
+- cognitive debt: reviewers receive more code than they can meaningfully understand
+- intent debt: the reason for a change disappears into a prompt or chat session
+
+`qamap eval` turns those risks into a small, static, explainable checklist that can run locally or in CI.

package/docs/github-action.md

@@ -0,0 +1,89 @@
+# GitHub Action
+
+QAMap ships a composite GitHub Action that can run `scan` or branch-aware `review` inside pull request workflows.
+
+The action writes:
+
+- a Markdown report file
+- a sticky PR comment body
+- an optional suggested domain test plan
+- an optional verification readiness evaluation
+- GitHub workflow annotations
+- a GitHub step summary
+
+## Pull Request Review
+
+```yaml
+name: QAMap
+
+on:
+  pull_request:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  qamap:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
+      - uses: IvoryCanvas/qamap@main
+        with:
+          mode: review
+          base: ${{ github.event.pull_request.base.sha }}
+          head: HEAD
+          fail-on: high
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+Use `fetch-depth: 0` so QAMap can compare the pull request branch with the base ref.
+For production workflows, pin to a version tag such as `IvoryCanvas/qamap@v0.3.0` after that tag is published. Use `@main` only when intentionally testing unreleased behavior.
+
+## Monorepo Package
+
+```yaml
+- uses: IvoryCanvas/qamap@main
+  with:
+    mode: review
+    path: services/offer
+    workspace-root: .
+    base: ${{ github.event.pull_request.base.sha }}
+    head: HEAD
+    fail-on: high
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+## Inputs
+
+| Input | Default | Description |
+| --- | --- | --- |
+| `mode` | `auto` | `auto`, `scan`, or `review`. `auto` uses review mode on pull request events. |
+| `path` | `.` | Repository path to scan. |
+| `workspace-root` | | Optional monorepo workspace root. |
+| `base` | | Base ref or SHA for review mode. |
+| `head` | `HEAD` | Head ref or SHA for review mode. |
+| `config` | | Optional QAMap config path. |
+| `max-files` | | Optional maximum file count. |
+| `fail-on` | `high` | Severity threshold that fails the action. |
+| `report-file` | `qamap-report.md` | Markdown report output path. |
+| `comment-file` | `qamap-pr-comment.md` | PR comment body output path. |
+| `test-plan` | `true` | Append suggested domain tests for changed files. |
+| `test-plan-file` | `qamap-test-plan.md` | Markdown test plan output path. |
+| `eval` | `true` | Append a verification readiness evaluation. |
+| `eval-file` | `qamap-eval.md` | Markdown evaluation output path. |
+| `annotations` | `true` | Emit GitHub workflow annotations. |
+| `step-summary` | `true` | Append the report to the GitHub step summary. |
+| `comment` | `true` | Create or update a sticky pull request comment. |
+| `github-token` | | Token for PR comments. Required when `comment` is `true`. |
+
+## Notes
+
+- Set `pull-requests: write` only when PR comments are enabled.
+- Set `comment: false` for advisory runs that should only use annotations and step summaries.
+- Set `test-plan: false` when the PR comment should contain only QAMap findings.
+- Set `eval: false` when the PR comment should not include readiness scoring.
+- Pass `github-token: ${{ secrets.GITHUB_TOKEN }}` when `comment` is enabled.
+- QAMap still does not execute scanned project code; the action only installs and builds QAMap itself before scanning the checked-out repository.