@keber/qa-framework 1.6.1 → 1.7.0

package/CHANGELOG.md

@@ -33,6 +33,58 @@ has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/gen
 
 ---
 
+## [1.7.0] - 2026-05-06
+
+### Changed
+
+- **Agent instructions moved to `.github/instructions/qa-framework.instructions.md`** — the
+  framework no longer writes to `.github/copilot-instructions.md`. The new file is a VS Code
+  `*.instructions.md` file with `applyTo: '**'` frontmatter, which Copilot loads automatically.
+  This eliminates the conflict between framework-owned rules and project-owned custom instructions:
+  both files coexist independently and the framework file is freely upgradeable.
+
+### Added
+
+- **`upgrade` step 2b — automatic migration of old `copilot-instructions.md`**:
+  - If the file contains only QA Framework content (any previous version) → the file is deleted.
+  - If the file contains custom instructions and a QA Framework section → the QA section is stripped and the custom content is preserved.
+  - If the file contains no QA Framework content → the file is left untouched.
+  The detection marker is the heading `# QA Framework Instructions`, consistent across all versions.
+
+### Migration
+
+Run `npx qa-framework upgrade`. No manual steps required.
+See `MIGRATION-NOTES.md` — section "Upgrading from a version that used `.github/copilot-instructions.md`".
+
+---
+
+## [1.6.2] - 2026-05-05
+
+### Added
+
+- **`tests/helpers/debug/`** — diagnostic scripts directory created by `init` and `upgrade` inside
+  `07-automation/e2e/tests/`. Scripts here are excluded from test runs via `testIgnore`.
+- **`tests/seeds/`** — data seeding specs directory created by `init` and `upgrade` inside
+  `07-automation/e2e/tests/`. Excluded from test runs via `testIgnore`.
+- **`testIgnore`** in `playwright.config.ts` template — excludes `**/helpers/debug/**` and
+  `**/seeds/**` from Playwright test discovery.
+- **`upgrade` step 5e** — patches existing `playwright.config.ts` in `e2e/`: replaces
+  `testDir: '.'` with `'./tests'`, injects `testIgnore` if missing, creates `helpers/debug/`
+  and `seeds/` if absent.
+- **Dual-Track Pipeline** section in `docs/folder-structure-guide.md` and `qa/QA-STRUCTURE-GUIDE.md`
+  — documents the automation track and manual track as parallel paths sharing `01-specifications/`
+  as source of truth. Includes entry point table (3 scenarios) and track selection guidance.
+  Clarifies that `03-test-cases/` is the manual track artifact and `04-test-data/` is optional.
+
+### Fixed
+
+- `qa/QA-STRUCTURE-GUIDE.md` — removed stale reference to deleted `docs/generalization-decisions.md`
+  from the `06-defects/` section.
+- `README.md` — pipeline table updated with Track column; `03-test-cases/` and `04-test-data/`
+  annotated as manual-track / optional in the directory tree.
+
+---
+
 ## [1.6.0] - 2026-04-29
 
 ### Changed (breaking - automated by `upgrade`)
@@ -69,11 +121,10 @@ has been removed or parameterized. See `docs/comparison-matrix.md` and `docs/gen
 - `07-automation/load/` — placeholder for load and performance tests; created on `init` and `upgrade`.
 - **`scripts/upgrade.js` migration section 5** — detects old v1.5.x structure and relocates all
   affected files non-destructively (moves only when destination does not exist; warns on conflict).
-- **Stage 3.5 - Test Stabilization** (`agent-instructions/04b-test-stabilization.md`) — agent
-  instruction file for exhaustively reviewing and stabilizing generated Playwright tests before
-  ADO integration. Covers baseline collection, failure classification (9 categories), false-positive
-  validation, confidence scoring (>=90% exit threshold), `STABILIZATION-REPORT.md` schema, and
-  upstream artifact update rules.
+- **Stage 5b - Test Stabilization** (`skills/qa-test-stabilization/`) — skill for exhaustively
+  reviewing and stabilizing generated Playwright tests before ADO integration. Covers baseline
+  collection, failure classification (9 categories), false-positive validation, confidence scoring
+  (>=90% exit threshold), `STABILIZATION-REPORT.md` schema, and upstream artifact update rules.
 
 ### Planned
 

package/README.md

@@ -1,7 +1,7 @@
 # qa-framework
 
 > Reusable, installable, agent-oriented QA framework for spec-driven automated testing.
-> Current version: **v1.6.0**
+> Current version: **v1.7.0**
 
 ---
 
@@ -61,7 +61,7 @@ npm install @keber/qa-framework
 1. Create the `qa/` directory tree in your project root
 2. Copy all base templates into place
 3. Create `qa/qa-framework.config.json` for project-specific settings
-4. Generate `.github/copilot-instructions.md` with QA agent behavior rules (11 rules + ADO detection)
+4. Generate `.github/instructions/qa-framework.instructions.md` with QA agent behavior rules (11 rules + ADO detection)
 5. Copy the `skills/` set to `.github/skills/` so the agent can load them on demand
 6. Create `qa/AGENT-NEXT-STEPS.md` with follow-up steps
 
@@ -73,9 +73,13 @@ After `npm update @keber/qa-framework`, run:
 npx qa-framework upgrade
 ```
 
-This overwrites **only** framework-owned files (skills, `copilot-instructions.md`,
+This overwrites **only** framework-owned files (skills, `qa-framework.instructions.md`,
 `QA-STRUCTURE-GUIDE.md`) and never touches your specs, test plans, automation code, or memory.
 
+If upgrading from a version that wrote to `.github/copilot-instructions.md`, `upgrade`
+automatically removes the QA section from that file (or deletes it if it contained only
+framework content), and creates `.github/instructions/qa-framework.instructions.md` instead.
+
 ---
 
 ## Repository Structure (this package)
@@ -148,7 +152,7 @@ qa-framework/
 │
 └── scripts/                          <- CLI commands
     ├── cli.js                        <- Entry point (qa-framework <command>)
-    ├── init.js                       <- Scaffold qa/ tree + skills + copilot-instructions
+    ├── init.js                       <- Scaffold qa/ tree + skills + qa-framework.instructions.md
     ├── upgrade.js                    <- Refresh framework-owned files after npm update
     ├── generate.js                   <- Generate individual artifact from template
     └── validate.js                   <- Check structure compliance
@@ -170,8 +174,8 @@ qa/
 ├── 01-specifications/           <- Functional specs (used by generate command)
 ├── {module}/{submodule}/        <- 6-file spec sets created per module/submodule in config
 ├── 02-test-plans/               <- Test plans (automated + manual)
-├── 03-test-cases/               <- Explicit test case documents
-├── 04-test-data/                <- Test data, fixtures, factories
+├── 03-test-cases/               <- TC-*.md step-by-step docs (manual track; optional if fully automated)
+├── 04-test-data/                <- Shared test data factories/seeders (optional — per-module data lives in 01-specs)
 ├── 05-test-execution/           <- Execution reports and results
 ├── 06-defects/open|resolved/    <- Defect tracking
 ├── 07-automation/               <- Playwright automation code and config
@@ -194,10 +198,15 @@ Additionally, `init` writes to the project root:
 
 ```
 .github/
-├── copilot-instructions.md      <- Generated agent rules (11 rules + ADO detection)
-└── skills/                      <- Copied from package skills/ (one folder per pipeline stage)
+├── instructions/
+│   └── qa-framework.instructions.md  <- QA agent rules (framework-owned, applyTo: '**')
+└── skills/                           <- Copied from package skills/ (one folder per pipeline stage)
 ```
 
+> `qa-framework.instructions.md` is a VS Code `*.instructions.md` file, separate from any
+> custom `.github/copilot-instructions.md` the project may already have. The two files
+> coexist without conflict and are independently upgradeable.
+
 See [docs/folder-structure-guide.md](docs/folder-structure-guide.md) for a full explanation of every folder.
 
 ---
@@ -218,18 +227,18 @@ them via `read_file` without touching `node_modules`.
 
 ### QA Pipeline
 
-Skills map to the pipeline stages enforced by `.github/copilot-instructions.md`:
+Skills map to the pipeline stages enforced by `.github/instructions/qa-framework.instructions.md`:
 
-| Stage | Skill | Task | Prerequisite |
-|---|---|---|---|
-| 1 | `qa-module-analysis` | Analyze module, produce 6-file spec set | None |
-| 2 | `qa-spec-generation` | Generate/update 6-file spec set | `00-inventory.md` exists |
-| 3 | `qa-test-plan` | Generate test plan | `05-test-scenarios.md` exists |
-| 4 | `qa-test-cases` | Generate TC-{ID}.md files | Test plan exists |
-| 5 | `qa-automation` | Implement Playwright tests | Specs approved, no PENDING-CODE |
-| 5b | `qa-test-stabilization` | Stabilize failing tests | Failing tests exist |
-| 6 | `qa-maintenance` | Update artifacts after app changes | Delivered change |
-| - | `qa-ado-integration` | Sync with Azure DevOps Test Plans | ADO enabled in config |
+| Stage | Skill | Task | Track | Prerequisite |
+|---|---|---|---|---|
+| 1 | `qa-module-analysis` | Analyze module, produce 6-file spec set | Both | None |
+| 2 | `qa-spec-generation` | Generate/update 6-file spec set | Both | `00-inventory.md` exists |
+| 3 | `qa-test-plan` | Generate test plan | Both | `05-test-scenarios.md` exists |
+| 4 | `qa-test-cases` | Generate TC-{ID}.md files | Manual | Test plan exists |
+| 5 | `qa-automation` | Implement Playwright tests | Automation | Specs approved, no PENDING-CODE |
+| 5b | `qa-test-stabilization` | Stabilize failing tests | Automation | Failing tests exist |
+| 6 | `qa-maintenance` | Update artifacts after app changes | Both | Delivered change |
+| - | `qa-ado-integration` | Sync with Azure DevOps Test Plans | Both | ADO enabled in config |
 
 See [docs/skills-architecture.md](docs/skills-architecture.md) for the full design rationale.
 
@@ -254,7 +263,7 @@ Options:
   --version, -v               Show version
 ```
 
-**upgrade** overwrites only framework-owned files (skills, `copilot-instructions.md`,
+**upgrade** overwrites only framework-owned files (skills, `qa-framework.instructions.md`,
 `QA-STRUCTURE-GUIDE.md`) and never touches project-owned files. Run `--dry-run` to preview
 without writing.
 

package/docs/folder-structure-guide.md

@@ -31,6 +31,59 @@ Agent skills live in `.github/skills/` (not inside `qa/`). See the `.github/skil
 
 ---
 
+## Dual-Track Pipeline
+
+All pipeline paths share the same input: `01-specifications/`. From there, work splits into
+two tracks that can run in parallel or independently depending on team maturity.
+
+```
+01-specifications/  (source of truth — what the app does and what must be tested)
+        │
+        ├─── Automation track ──────────────────────────────────────────────────
+        │         │
+        │         ▼
+        │    02-test-plans/automated/      ← scope + priority decisions
+        │         │
+        │         ▼
+        │    07-automation/e2e/tests/      ← Playwright .spec.ts files
+        │         │
+        │         ▼
+        │    05-test-execution/automated/  ← execution reports
+        │         │
+        │         ▼
+        │    08-azure-integration/         ← ADO sync via playwright-azure-reporter
+        │
+        └─── Manual track ──────────────────────────────────────────────────────
+                  │
+                  ▼
+             02-test-plans/manual/         ← scope + manual test approach
+                  │
+                  ▼
+             03-test-cases/manual/         ← TC-*.md with steps (for human testers)
+                  │
+                  ▼
+             05-test-execution/manual/     ← manual run evidence
+                  │
+                  ▼
+             08-azure-integration/         ← ADO sync via ado-integration skill
+```
+
+**Entry points:**
+
+| Scenario | Entry point | Tracks active |
+|---|---|---|
+| New system, no prior QA | Module analysis → spec generation | Automation (primary), Manual (as needed) |
+| Sprint with ADO stories | Stories → spec generation → test plan | Both, driven by sprint scope |
+| Maintenance/continuous improvement | Changed module → maintenance skill | Automation (update existing tests) |
+
+**Track selection guidance:**
+- Use **automation track** when the goal is regression coverage and CI integration.
+- Use **manual track** when: (a) team has manual testers who need step-by-step documents; (b) auditability or traceability to external standards is required; (c) features are not yet automatable.
+- Both tracks use the same TC-IDs from `01-specifications/*/05-test-scenarios.md` — traceability is preserved regardless of which track executes a given TC.
+- `04-test-data/` is optional in both tracks. Use it when test data factories or seeders are shared across multiple modules. Per-submodule data belongs in `01-specifications/*/04-test-data.md`.
+
+---
+
 ## .github/skills/
 
 **Purpose**: 3-layer QA pipeline skills installed by the framework. Loaded by the agent on demand — only the relevant skill for the current task is loaded, not all skills at once.

package/docs/installation.md

@@ -51,7 +51,7 @@ qa/
 └── 08-azure-integration/README.md
 └── memory/INDEX.md
 
-.github/copilot-instructions.md     <- Generated agent rules
+.github/instructions/qa-framework.instructions.md  <- QA agent rules (framework-owned)
 .github/skills/                     <- 8 agent skill sets (SKILL.md + references/)
 ```
 
@@ -93,7 +93,7 @@ After `npm update @keber/qa-framework`:
 npx qa-framework upgrade
 ```
 
-This overwrites only framework-owned files (`.github/skills/`, `copilot-instructions.md`,
+This overwrites only framework-owned files (`.github/skills/`, `.github/instructions/qa-framework.instructions.md`,
 `QA-STRUCTURE-GUIDE.md`) and migrates the `07-automation/` structure if upgrading from
 v1.5.x. Your specs, tests, and memory are never touched.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@keber/qa-framework",
-  "version": "1.6.1",
+  "version": "1.7.0",
   "description": "Reusable spec-driven QA framework for IDE-agent-assisted automated testing. Installable as an npm package. Provides structure, templates, agent instructions, and optional integrations for Playwright and Azure DevOps.",
   "keywords": [
     "qa",

package/scripts/init.js

@@ -86,6 +86,8 @@ const topLevelFolders = [
   '06-defects/open',
   '06-defects/resolved',
   '07-automation/e2e/tests',
+  '07-automation/e2e/tests/helpers/debug',
+  '07-automation/e2e/tests/seeds',
   '08-azure-integration',
   'memory',
 ];
@@ -251,9 +253,9 @@ if (fs.existsSync(structureGuideSrc)) {
   writeIfMissing(structureGuideDest, fs.readFileSync(structureGuideSrc, 'utf8'));
 }
 
-// --- .github/copilot-instructions.md ---
+// --- .github/instructions/qa-framework.instructions.md ---
 const githubDir         = path.join(cwd, '.github');
-const copilotInstrPath  = path.join(githubDir, 'copilot-instructions.md');
+const copilotInstrPath  = path.join(githubDir, 'instructions', 'qa-framework.instructions.md');
 
 // Detect whether the ado-qa skill is available (installed or already copied)
 const adoQaInstalled = fs.existsSync(path.join(cwd, 'node_modules', '@keber', 'ado-qa'))
@@ -282,7 +284,10 @@ Refer to \`qa/00-guides/AGENT-INSTRUCTIONS-ADO-INTEGRATION.md\` for the full ADO
 > This will automatically copy the ADO skills to \`.github/skills/\` via postinstall.
 `;
 
-const copilotInstrContent = `# QA Framework Instructions
+const copilotInstrContent = `---
+applyTo: '**'
+---
+# QA Framework Instructions
 
 This project uses \`@keber/qa-framework\` v${config.frameworkVersion ?? '1.0.0'} for spec-driven automated testing.
 
@@ -335,7 +340,7 @@ const nextStepsContent = `# ✅ @keber/qa-framework installed successfully
 ## What was scaffolded
 
 - \`qa/\` folder structure with spec templates and agent instructions
-- \`.github/copilot-instructions.md\` with QA agent behavior rules
+- \`.github/instructions/qa-framework.instructions.md\` with QA agent behavior rules (framework-owned, safe to upgrade)
 
 ## Required next steps
 

package/scripts/upgrade.js

@@ -19,8 +19,8 @@
  *     - qa/07-automation/.env.example          -> qa/07-automation/e2e/
  *     - qa/07-automation/fixtures/             -> qa/07-automation/e2e/fixtures/
  *     - qa/07-automation/e2e/{module}/         -> qa/07-automation/e2e/tests/{module}/
- *     - Creates integration/ and load/ placeholders
- *
+ *     - Creates integration/ and load/ placeholders *     - Patches playwright.config.ts: testDir '.' -> './tests', adds testIgnore
+ *     - Creates tests/helpers/debug/ and tests/seeds/ if absent *
  *   NEVER touches (project-owned):
  *     - qa/01-specifications/       <- Your specs
  *     - qa/02-test-plans/           <- Your test plans
@@ -81,12 +81,48 @@ if (!fs.existsSync(skillsSrc)) {
 }
 
 // ---------------------------------------------------------------------------
-// 2. .github/copilot-instructions.md — overwrite (framework-owned sequencer)
+// 2. .github/instructions/qa-framework.instructions.md — overwrite (framework-owned)
 // ---------------------------------------------------------------------------
-const copilotInstrPath = path.join(githubDir, 'copilot-instructions.md');
+const copilotInstrPath = path.join(githubDir, 'instructions', 'qa-framework.instructions.md');
 const copilotContent = buildCopilotInstructions(config);
 forceWrite(copilotInstrPath, copilotContent);
 
+// ---------------------------------------------------------------------------
+// 2b. Migration: strip QA Framework section from old copilot-instructions.md
+//
+// Handles all cases:
+//   A. File has custom instructions + QA section  → keep custom, strip QA section
+//   B. File has ONLY QA section (any version)     → delete the file
+//   C. File does not mention QA Framework         → leave untouched
+//
+// The QA section always starts with "# QA Framework Instructions" across all
+// previous versions, so that heading is the reliable split point.
+// ---------------------------------------------------------------------------
+const oldCopilotPath = path.join(githubDir, 'copilot-instructions.md');
+if (fs.existsSync(oldCopilotPath)) {
+  const oldContent = fs.readFileSync(oldCopilotPath, 'utf8');
+  const qaMarker   = '# QA Framework Instructions';
+  const qaIdx      = oldContent.indexOf(qaMarker);
+
+  if (qaIdx !== -1) {
+    // Extract anything before the QA section, removing any trailing separator
+    const before = oldContent.slice(0, qaIdx).replace(/\s*\n---\s*$/, '').trim();
+
+    if (before.length === 0) {
+      // Case B: file contained only QA Framework content → delete it
+      if (!dryRun) fs.unlinkSync(oldCopilotPath);
+      updated.push(oldCopilotPath);
+      console.log(`  [deleted]  .github/copilot-instructions.md (contained only QA Framework rules — now in .github/instructions/qa-framework.instructions.md)`);
+    } else {
+      // Case A: file had custom content too → write back only the custom part
+      if (!dryRun) fs.writeFileSync(oldCopilotPath, before + '\n', 'utf8');
+      updated.push(oldCopilotPath);
+      console.log(`  [cleaned]  .github/copilot-instructions.md — removed QA Framework section, kept custom instructions`);
+    }
+  }
+  // Case C: no QA marker found → leave untouched (no log noise)
+}
+
 // ---------------------------------------------------------------------------
 // 3. qa/QA-STRUCTURE-GUIDE.md — overwrite (framework doc)
 // ---------------------------------------------------------------------------
@@ -187,6 +223,45 @@ if (!fs.existsSync(loadReadme)) {
   console.log(`  [created] ${path.relative(cwd, loadReadme)}`);
 }
 
+// 5e. Patch playwright.config.ts in e2e/:
+const playwrightConfigPath = path.join(e2eDir, 'playwright.config.ts');
+if (fs.existsSync(playwrightConfigPath)) {
+  let cfg = fs.readFileSync(playwrightConfigPath, 'utf8');
+  let cfgChanged = false;
+
+  // Fix testDir: '.' -> './tests'
+  if (/testDir\s*:\s*['"]\.['"]/.test(cfg)) {
+    cfg = cfg.replace(/testDir\s*:\s*['"]\.['"]/g, "testDir:  './tests'");
+    cfgChanged = true;
+    console.log(`  [patched] e2e/playwright.config.ts — testDir: '.' -> './tests'`);
+  }
+
+  // Inject testIgnore after testDir line if not present
+  if (!cfg.includes('testIgnore')) {
+    cfg = cfg.replace(
+      /(testDir\s*:.*\n)/,
+      "$1  testIgnore: ['**/helpers/debug/**', '**/seeds/**'],\n"
+    );
+    cfgChanged = true;
+    console.log(`  [patched] e2e/playwright.config.ts — added testIgnore`);
+  }
+
+  if (cfgChanged) {
+    if (!dryRun) fs.writeFileSync(playwrightConfigPath, cfg, 'utf8');
+    updated.push(playwrightConfigPath);
+  }
+}
+
+// Create tests/helpers/debug and tests/seeds if missing
+for (const subdir of ['tests/helpers/debug', 'tests/seeds']) {
+  const subdirPath = path.join(e2eDir, subdir);
+  if (!fs.existsSync(subdirPath)) {
+    if (!dryRun) fs.mkdirSync(subdirPath, { recursive: true });
+    updated.push(subdirPath);
+    console.log(`  [created] ${path.relative(cwd, subdirPath)}/`);
+  }
+}
+
 // ---------------------------------------------------------------------------
 // Summary
 // ---------------------------------------------------------------------------
@@ -239,7 +314,10 @@ function copyDirForce(srcDir, destDir) {
 
 function buildCopilotInstructions(cfg) {
   const version = cfg.frameworkVersion ?? '1.0.0';
-  return `# QA Framework Instructions
+  return `---
+applyTo: '**'
+---
+# QA Framework Instructions
 
 This project uses \`@keber/qa-framework\` v${version} for spec-driven automated testing.
 

package/templates/automation-scaffold/playwright.config.ts

@@ -39,6 +39,8 @@ for (const key of required) {
 export default defineConfig({
   // ------ Test discovery ------
   testDir:  './tests',
+  // Exclude non-suite directories from test runs
+  testIgnore: ['**/helpers/debug/**', '**/seeds/**'],
   // Use any subdir pattern your project standardizes on, e.g.:
   // testMatch: ['**/*.spec.ts'],