@ivorycanvas/qamap 0.3.3 → 0.3.4

package/docs/guardrails.md

@@ -0,0 +1,64 @@
+# Repository Guardrails
+
+The QA draft loop is QAMap's core product. This optional second layer statically scans a repository for the settings that make AI-assisted work risky.
+
+## Running the scanner
+
+The scanner checks agent instructions, MCP configs, committed env files, risky package scripts, workflow permissions, and validation signals before agent work becomes review churn.
+
+For a repository baseline before broad agent use, run:
+
+```sh
+qamap scan .
+```
+
+```txt
+Findings: 6 (high: 3, medium: 2, low: 1, info: 0)
+
+HIGH
+- QM003 Suspicious agent instruction text (AGENTS.md)
+  Fix: Remove untrusted instruction text or move examples into clearly fenced documentation.
+```
+
+See the rule table below and [docs/rules.md](rules.md) for what the scanner checks.
+
+When developing QAMap from source:
+
+```sh
+git clone https://github.com/IvoryCanvas/qamap.git
+cd qamap
+pnpm install
+pnpm build
+node dist/cli.js scan /path/to/repo
+```
+
+QAMap is a local-first PR verification planner with a repository-level verification manifest loop, not a finished automatic QA bot. A good result is a clear answer to "what should this branch prove before merge?", plus manifest-backed E2E, fixture, selector, and validation work that a developer can turn into real regression coverage. Many first drafts will correctly report `review-only` or `near-runnable` until the project adds runner config, stable selectors, deterministic fixtures, or team-owned manifest entries.
+
+
+## What It Checks
+
+The first release focuses on high-signal checks that are useful across many repositories.
+
+| Rule | Severity | What it catches |
+| --- | --- | --- |
+| `QM001` | medium | Missing agent instruction files. |
+| `QM002` | medium | Conflicting agent guidance. |
+| `QM003` | high | Suspicious instruction text that can misdirect agents. |
+| `QM004` | medium/high | Risky MCP command configuration. |
+| `QM005` | high | Secret-like values embedded in MCP config. |
+| `QM006` | medium | Missing or placeholder test scripts. |
+| `QM007` | low | Missing GitHub Actions workflows. |
+| `QM008` | high | Committed local environment files. |
+| `QM009` | high | Package scripts that can publish, push, merge, or run unsafe shell pipelines. |
+| `QM010` | medium | Broad workflow permissions or risky workflow triggers. |
+| `QM011` | low | Missing community health files. |
+| `QM012` | medium/high | Risky committed agent settings, hooks, or broad shell permissions. |
+| `QM013` | low | API endpoints documented only in prose without a contract source. |
+
+See [docs/rules.md](rules.md) for the rule catalog.
+See [docs/ecosystem.md](ecosystem.md) for the agent ecosystem surfaces QAMap tracks.
+See [docs/api-contracts.md](api-contracts.md) for the API contract source-of-truth check.
+See [docs/verify.md](verify.md) for the combined PR verification report.
+See [docs/eval.md](eval.md) for the change readiness evaluation.
+See [docs/releasing.md](releasing.md) for the npm release runbook.
+

package/docs/manifest.md

@@ -32,7 +32,7 @@ qamap e2e draft . --manifest /tmp/qamap-manifest.yaml --base origin/main --head
 
 `manifest init` reads the current checkout on disk. It does not silently switch to the default branch, because changing a developer's branch or working tree would be surprising and unsafe. If the team wants the manifest to represent the default product baseline, run it from the default branch after pulling the latest changes.
 
-`manifest context` is the read-only preview step. It shows which repo-local documents QAMap sees, how each source is classified, which validation commands and safety rules were extracted, and which manifest path should be reviewed if the context is missing or stale. Use it before `manifest init` when you want to understand the bootstrap input without writing `.qamap/manifest.yaml`.
+`manifest context` is the read-only preview step. It shows which repo-local documents QAMap sees, how each source is classified, which validation commands were collected (from project configuration such as `package.json` scripts and pytest setup, plus instruction docs), which safety rules were extracted, and which manifest path should be reviewed if the context is missing or stale. Use it before `manifest init` when you want to understand the bootstrap input without writing `.qamap/manifest.yaml`.
 
 During baseline generation, QAMap also looks for repo-local context documents that often contain verification knowledge not visible in source files:
 
@@ -196,7 +196,7 @@ The report includes:
 
 - role summary by source file, such as `verification-rubric`, `test-runner`, `agent-skill`, or `harness-config`
 - captured context sources with kind, confidence, roles, and signals
-- validation commands and safety rules extracted from local docs
+- validation commands (project configuration first, then local docs) and safety rules extracted from prose
 - diagnostics that point to the manifest path to edit when context is missing, stale, too broad, or not connected to checks
 
 This command is useful when an E2E recommendation feels too vague. Instead of asking an LLM to re-read the repository, inspect the report, correct the repo-local context or `.qamap/manifest.yaml`, then rerun `qamap e2e draft`.
@@ -268,7 +268,7 @@ flows:
 | `flows[].checks[].steps` | Optional concrete steps for this check. These are used before the title parser. |
 | `context.instructionFiles` | Advisory repo-local context sources used while bootstrapping the manifest. |
 | `context.instructionFiles[].roles` | Advisory role classification for a context source, such as `verification-rubric`, `workflow-lifecycle`, `agent-skill`, or `harness-config`. |
-| `context.validationCommands` | Validation commands inferred from context documents. |
+| `context.validationCommands` | Validation commands from project configuration (verification-shaped `package.json` scripts, detected pytest setup) and context documents, ground truth first. |
 | `context.safetyRules` | Workflow or safety rules inferred from context documents, with token-like values redacted. |
 | `source.kind` | `inferred` for QAMap-generated entries or `declared` after human review. |
 | `source.confidence` | `low`, `medium`, or `high` confidence in the entry. |

package/docs/quickstart-demo.md

@@ -168,7 +168,7 @@ Summary
 - Project: Web
 - Recommended runner: Playwright
 - Manifest: .qamap/manifest.yaml
-- Readiness: near-runnable
+- Stage: almost runnable (3 of 4)
 
 PR Comment Draft
 - Affected flow: Checkout purchase

package/docs/release-validation.md

@@ -1,5 +1,19 @@
 # Release Validation
 
+## 0.3.4 - 2026-07-10
+
+Validated before publishing `0.3.4` with a committed, CI-enforced recommendation contract instead of relying only on private smoke repositories:
+
+| Gate | Current result |
+| --- | --- |
+| `pnpm test` | 124/124 passing |
+| `pnpm bench:ci` | Eight public PR fixtures pass runner, flow naming, file reach, selector/evidence, command, generic-title, blank-action, and agent payload requirements |
+| Coverage | Lines 86.19%, branches 83.37%, functions 94.87% |
+| Reverse import fixture | Shared component change reaches and names the consuming checkout page |
+| API service fixture | Backend route changes produce an API contract, not a browser UI journey |
+| Agent contract | Real output validates against `schema/qamap-agent.schema.json` and stays below the 4KB fixture limit |
+| Verification-only regressions | Native config changes use existing build commands without fabricated journeys; changed Maestro files are returned as existing evidence without duplicate drafts or selector/fixture noise |
+
 ## 0.3.3 - 2026-07-05
 
 Validated before publishing `0.3.3` (at-a-glance qa verdict, diff-anchored action naming incl. button/link text and logic-only fallbacks, observed-response assertions with diff-derived status bounds, Vue bound-attribute/i18n selector fixes):
@@ -22,7 +36,7 @@ Validated before publishing `0.3.2` (reverse import graph, diff-anchored steps a
 | `pnpm test` | 100/100 passing |
 | `pnpm scan` (self-scan) | 0 findings |
 | Coverage thresholds (lines/branches/functions >= 80) | Passing |
-| `pnpm bench` against four pinned local benchmark targets (web monorepo, Python API server, Expo app, legacy Vue app) | Runner choice 4/4, labeled must-reach recall 9/9, blank actions 0, no metric regressions vs the previous baseline |
+| `pnpm bench` against four pinned local benchmark repositories covering common stack shapes (monorepo, API server, mobile, legacy web) | Runner choice 4/4, labeled must-reach recall 9/9, blank actions 0, no metric regressions vs the previous baseline |
 | README demo | Real recorded run; the generated starter spec passes against the demo app |
 
 Historical validation notes for earlier releases follow below.
@@ -33,7 +47,7 @@ QAMap should not publish a patch or minor version only because the CLI commands
 
 ## Release Bar
 
-QAMap is ready for `0.2.1` when the commands below produce useful, reviewable output for each representative repository type:
+QAMap is ready for the next public release when the commands below produce useful, reviewable output for each representative repository type:
 
 - manifest-free `qamap qa` output that works as a PR comment/checklist draft
 - packaged `skills/qamap-pr-qa/SKILL.md` template included in the npm tarball
@@ -71,7 +85,7 @@ Run these from a clean checkout of QAMap before any release candidate:
 pnpm run release:check
 ```
 
-`release:check` expands to the required local suite: `pnpm test`, `pnpm scan`, `git diff --check`, coverage thresholds, and `pnpm pack --dry-run`. If a release candidate fails, run the individual command directly to inspect the failure.
+`release:check` expands to the required local suite: `pnpm test`, `pnpm scan`, `pnpm bench:ci`, `git diff --check`, coverage thresholds, and `pnpm pack --dry-run`. If a release candidate fails, run the individual command directly to inspect the failure.
 
 Run the npm publish preview after the local release gate passes:
 
@@ -231,7 +245,7 @@ The same follow-up also showed the next quality gap. External manifest previews
 | Design token repository | `main` to `HEAD`, working tree included | Detected `design-tokens`, selected manual output, avoided browser/device selector requirements, and produced review-only artifact validation output. | Docs-only or style-only changes may still surface content/theme wording instead of token artifact language. |
 | Taxonomy / data catalog repository | `main` to `HEAD`, working tree included | Detected `data-catalog`, selected manual output, avoided API mock requirements, and produced review-only checklist output with no TODOs. | Catalog-specific changes are handled, but docs/config-only changes should be labeled more explicitly as low-signal. |
 
-Interpretation: the `0.2.1` release should be described as a planner that removes blank-page verification work by combining static analysis with repo-local manifest memory. The smoke results are useful, but they also show that many real repositories will start at `review-only` or `near-runnable` until teams add runner config, selectors, fixtures, validation evidence, and durable manifests.
+Interpretation: the next release should be described as a planner that removes blank-page verification work by combining static analysis with repo-local manifest memory. The smoke results are useful, but they also show that many real repositories will start at `review-only` or `near-runnable` until teams add runner config, selectors, fixtures, validation evidence, and durable manifests.
 
 ## Ongoing Validation Notes
 

package/docs/releasing.md

@@ -26,6 +26,7 @@ The gate must pass:
 
 - `pnpm test`
 - `pnpm scan`
+- `pnpm bench:ci`
 - `git diff --check`
 - coverage thresholds for lines, branches, and functions
 - `pnpm pack --dry-run`
@@ -45,6 +46,7 @@ The tarball must include runtime output, public documentation, schemas, and pack
 
 - `dist`
 - `docs`
+- `skills`
 - `schema`
 - `README.md`
 - `CHANGELOG.md`
@@ -65,9 +67,10 @@ npm publish --access public
 After publish, verify the public package can be executed without a source checkout:
 
 ```sh
-pnpm dlx @ivorycanvas/qamap@0.3.1 scan .
-pnpm dlx @ivorycanvas/qamap@0.3.1 manifest validate .
-pnpm dlx @ivorycanvas/qamap@0.3.1 e2e draft . --base origin/main --head HEAD --dry-run
+VERSION="$(node -p "require('./package.json').version")"
+pnpm dlx "@ivorycanvas/qamap@$VERSION" qa . --base origin/main --head HEAD
+pnpm dlx "@ivorycanvas/qamap@$VERSION" manifest validate .
+pnpm dlx "@ivorycanvas/qamap@$VERSION" e2e draft . --base origin/main --head HEAD --dry-run
 ```
 
 Use a fresh shell or temporary directory for the smoke check when possible.
@@ -77,8 +80,8 @@ Use a fresh shell or temporary directory for the smoke check when possible.
 After npm publish succeeds:
 
 ```sh
-git tag v0.3.1
-git push origin v0.3.1
+git tag "v$VERSION"
+git push origin "v$VERSION"
 ```
 
 Create a GitHub Release for the tag with:
@@ -93,13 +96,13 @@ Create a GitHub Release for the tag with:
 After the tag and GitHub Release are visible, run:
 
 ```sh
-pnpm dlx @ivorycanvas/qamap@0.3.1 --version
-pnpm dlx @ivorycanvas/qamap@0.3.1 scan .
-pnpm dlx @ivorycanvas/qamap@0.3.1 manifest explain . --base origin/main --head HEAD
-pnpm dlx @ivorycanvas/qamap@0.3.1 verify . --base origin/main --head HEAD
+pnpm dlx "@ivorycanvas/qamap@$VERSION" --version
+pnpm dlx "@ivorycanvas/qamap@$VERSION" qa . --base origin/main --head HEAD --format agent
+pnpm dlx "@ivorycanvas/qamap@$VERSION" manifest explain . --base origin/main --head HEAD
+pnpm dlx "@ivorycanvas/qamap@$VERSION" verify . --base origin/main --head HEAD
 ```
 
-Then update any public setup examples that should pin to `v0.3.1`.
+Then update any public setup examples that should pin to the new `v$VERSION` tag.
 
 ## Rollback Notes
 

package/docs/roadmap.md

@@ -28,28 +28,23 @@ Before treating the next public release as ready, the golden demo must satisfy t
 
 ## Now
 
-- Keep the scanner fast, static, and easy to understand.
-- Keep the [release validation checklist](release-validation.md), [manifest guide](manifest.md), and public [E2E output examples](e2e-output-examples.md) aligned with the current release bar.
-- Keep the [release runbook](releasing.md) ready for npm publishing and versioned GitHub Action follow-up.
-- Improve adoption docs and sample output so new maintainers understand the QA manifest plus PR-to-E2E draft loop quickly.
+- Treat the committed [benchmark contract](benchmarking.md) as the quality gate for recommendations, not only implementation correctness. Reduce real failures into public fixtures and require `pnpm bench:ci` on every PR.
+- Make `qa` the primary product surface. Its first screen and `--format agent` payload should agree on affected behavior, reviewer question, repository evidence, draft path, and missing trust requirements.
+- Improve changed-file impact mapping from shared symbols and components to consuming routes, screens, API contracts, and manifest flows.
+- Keep the [release validation checklist](release-validation.md), [manifest guide](manifest.md), public [E2E output examples](e2e-output-examples.md), and README examples aligned with captured output from the public fixtures.
 - Stabilize the manifest feedback loop with `.qamap/manifest.yaml`, `manifest init`, `manifest validate`, `manifest explain`, JSON Schema, and manifest-driven E2E draft shaping.
 - Keep `manifest context` useful as a pre-init sanity check for repo-local QA memory, harness docs, agent instructions, and runbooks.
 - Improve generated drafts until the golden demo feels like a real starting point, not a generic checklist.
-- Make `qa` the best first-run experience for AI-assisted PRs; keep `verify`, `e2e`, and `manifest` as the deeper layers behind it.
-- Keep `eval` explainable enough that maintainers trust the score and know what to fix.
-- Keep expanding representative validation targets beyond JavaScript so planning advice works for Python, Go, Rust, and JVM repositories.
+- Keep `verify`, `e2e`, and `manifest` as deeper layers behind `qa`; freeze new scanner, doctor, eval, domains, flows, and history features until the core QA contract is consistently useful.
 
 ## Next
 
-- Publish a versioned GitHub Action release tag after the first public package is ready.
-- Improve `doctor` output with clearer scoring and remediation grouping.
-- Improve `review` output for changed-line locations.
-- Expand manifest support with richer anchors, symbol-level matching, and configurable taste rubrics.
-- Map changed symbols to manifest anchors after the path/route baseline is stable.
+- Add symbol-level anchors for exported components, hooks, API clients, handlers, schemas, and queries after the public import-impact fixture stays stable.
+- Add a manifest correction command that proposes the exact flow/anchor patch and applies it only after human approval, avoiding routine hand-edits to YAML.
+- Add stronger deterministic draft adapters for Playwright and Maestro while keeping `manual` output for API, CLI, token, and catalog repositories.
+- Expand the public benchmark corpus with package-scoped monorepos, auth/session changes, dynamic routes, API failure fixtures, and non-JavaScript services.
 - Keep the `--format agent` output a stable, versioned contract that skills and MCP wrappers can rely on.
-- Add language-specific domain patterns for backend services, CLIs, libraries, mobile apps, and infrastructure repositories.
 - Continue expanding agent surface detection across popular coding-agent tools without making the public workflow depend on a single vendor.
-- Generate rule documentation from scanner metadata.
 
 ## Later
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ivorycanvas/qamap",
-  "version": "0.3.3",
+  "version": "0.3.4",
   "description": "Local-first QA skill that turns PR diffs into affected flows, missing evidence, and E2E drafts with no cloud or LLM token.",
   "type": "module",
   "packageManager": "pnpm@10.32.1",
@@ -24,8 +24,9 @@
     "coverage": "pnpm build && node --test --experimental-test-coverage --test-coverage-include=\"dist/**/*.js\" --test-coverage-lines=80 --test-coverage-branches=80 --test-coverage-functions=80 test/*.test.mjs",
     "scan": "pnpm build && node dist/cli.js scan .",
     "report": "pnpm build && node dist/cli.js report . --output QAMAP_REPORT.md",
-    "release:check": "pnpm test && pnpm scan && git diff --check && pnpm run coverage && pnpm pack --dry-run",
-    "bench": "pnpm build && node scripts/bench.mjs"
+    "release:check": "pnpm test && pnpm scan && pnpm bench:ci && git diff --check && pnpm run coverage && pnpm pack --dry-run",
+    "bench": "pnpm build && node scripts/bench.mjs",
+    "bench:ci": "pnpm build && node scripts/bench.mjs --config bench.config.json --assert"
   },
   "keywords": [
     "ai",

package/schema/qamap-agent.schema.json

@@ -0,0 +1,171 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://raw.githubusercontent.com/IvoryCanvas/qamap/main/schema/qamap-agent.schema.json",
+  "title": "QAMap agent QA summary (qamap.qa v1)",
+  "description": "The single-line JSON object printed by `qamap qa --format agent`. Within schema version 1, fields are only ever added — existing fields are never removed or retyped. A breaking change bumps schema.version to 2.",
+  "type": "object",
+  "additionalProperties": true,
+  "required": [
+    "schema",
+    "base",
+    "head",
+    "project",
+    "runner",
+    "manifest",
+    "readiness",
+    "testSuite",
+    "flows",
+    "requiredEvidence",
+    "recommendedEvidenceCount",
+    "requiredBootstrap",
+    "prChecklist",
+    "commands"
+  ],
+  "properties": {
+    "schema": {
+      "type": "object",
+      "required": ["name", "version"],
+      "properties": {
+        "name": { "const": "qamap.qa" },
+        "version": { "const": 1 }
+      }
+    },
+    "base": {
+      "type": "string",
+      "description": "Git ref the diff was computed against."
+    },
+    "head": {
+      "type": "string",
+      "description": "Git ref of the change under review."
+    },
+    "project": {
+      "type": "string",
+      "description": "Detected project type, for example web, react-native, node, python, unknown."
+    },
+    "runner": {
+      "type": "string",
+      "enum": ["maestro", "playwright", "manual"],
+      "description": "Recommended E2E runner for this repository."
+    },
+    "manifest": {
+      "type": ["string", "null"],
+      "description": "Path of the verification manifest in use, or null when the run used repo signals and the PR diff only."
+    },
+    "readiness": {
+      "type": "object",
+      "required": ["score", "level"],
+      "properties": {
+        "score": { "type": "number", "minimum": 0, "maximum": 100 },
+        "level": {
+          "type": "string",
+          "enum": ["ready", "near-runnable", "needs-work", "blocked"],
+          "description": "Stable machine value; human reports render this as a stage on a four-step journey."
+        }
+      }
+    },
+    "testSuite": {
+      "type": "object",
+      "required": ["present", "files"],
+      "properties": {
+        "present": { "type": "boolean" },
+        "files": { "type": "number", "minimum": 0 }
+      }
+    },
+    "firstDraftCommand": {
+      "type": "string",
+      "description": "One command that creates the first E2E draft. Only present when the repository has no test suite and the diff reaches a product journey."
+    },
+    "flows": {
+      "type": "array",
+      "description": "Affected user flows derived from the diff, most relevant first (capped).",
+      "items": {
+        "type": "object",
+        "required": ["title", "source", "draft", "steps", "selectors"],
+        "properties": {
+          "title": { "type": "string" },
+          "source": {
+            "type": "string",
+            "description": "Where the flow came from, for example verification-manifest, core-flow, domain-language, heuristic."
+          },
+          "draft": { "type": "string", "description": "Path of the fallback draft artifact for this flow. When verificationMode is present, run existing repository evidence first instead of generating it by default." },
+          "verificationMode": {
+            "type": "string",
+            "enum": ["existing-test-evidence", "configuration", "documentation", "generated-artifact"],
+            "description": "Why this flow should validate existing evidence instead of generating a new product-journey E2E draft."
+          },
+          "runnable": {
+            "type": "string",
+            "enum": ["runnable-candidate", "near-runnable", "review-only"],
+            "description": "How trustworthy the generated draft is as an executable test. Absent when not evaluated."
+          },
+          "entry": { "type": "string", "description": "Best entrypoint hint (route or screen). Absent when unknown." },
+          "changedFiles": {
+            "type": "array",
+            "items": { "type": "string" },
+            "description": "Changed or import-reached files that support this flow mapping (capped)."
+          },
+          "reviewQuestion": {
+            "type": "string",
+            "description": "The concrete question a reviewer should answer before merge."
+          },
+          "successSignal": {
+            "type": "string",
+            "description": "The observable result that would prove the flow still works."
+          },
+          "steps": { "type": "array", "items": { "type": "string" } },
+          "selectors": { "type": "array", "items": { "type": "string" } },
+          "existingEvidence": {
+            "type": "array",
+            "items": { "type": "string" },
+            "description": "Changed test files to execute instead of generating another draft. Present for test-evidence-only changes."
+          },
+          "evidence": {
+            "type": "array",
+            "items": { "type": "string" },
+            "description": "Short reasons and repository facts behind the mapping (capped)."
+          }
+        }
+      }
+    },
+    "requiredEvidence": {
+      "type": "array",
+      "description": "QA evidence the change still needs before merge (required priority only, capped at 8).",
+      "items": {
+        "type": "object",
+        "required": ["flow", "kind", "title"],
+        "properties": {
+          "flow": { "type": "string" },
+          "kind": { "type": "string" },
+          "title": { "type": "string" }
+        }
+      }
+    },
+    "recommendedEvidenceCount": {
+      "type": "number",
+      "minimum": 0,
+      "description": "Count of recommended-priority evidence items not listed here; run without --format agent to see them."
+    },
+    "requiredBootstrap": {
+      "type": "array",
+      "description": "Setup steps that must happen before drafts can be treated as regression coverage (capped at 3).",
+      "items": {
+        "type": "object",
+        "required": ["title", "action"],
+        "properties": {
+          "title": { "type": "string" },
+          "action": { "type": "string" }
+        }
+      }
+    },
+    "prChecklist": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Ready-to-paste PR checklist lines (capped)."
+    },
+    "commands": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Suggested commands to run next, most useful first (capped at 4)."
+    }
+  }
+}

package/skills/qamap-pr-qa/SKILL.md

@@ -25,6 +25,7 @@ Use QAMap as a final local QA pass before presenting a pull request for human re
    ```
 
    Drop `--format agent` when a human will read the output directly; the default markdown report is written for people.
+   The agent JSON is a versioned contract (`schema: qamap.qa` v1, additive-only): see docs/agent-format.md in the QAMap repository.
 
 3. If the repository is a monorepo and the changed files are clearly inside one package, run a scoped pass too: