@keber/qa-framework 1.1.3 → 1.3.0-0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/agent-instructions/00-module-analysis.md +3 -0
- package/agent-instructions/01-spec-generation.md +3 -0
- package/agent-instructions/02-test-plan-generation.md +3 -0
- package/agent-instructions/03-test-case-generation.md +3 -0
- package/agent-instructions/04-automation-generation.md +33 -0
- package/agent-instructions/04b-test-stabilization.md +3 -0
- package/agent-instructions/05-ado-integration.md +3 -0
- package/agent-instructions/06-maintenance.md +10 -0
- package/docs/folder-structure-guide.md +76 -21
- package/docs/skills-architecture.md +136 -0
- package/package.json +1 -1
- package/qa-framework.config.json +1 -1
- package/scripts/cli.js +5 -0
- package/scripts/init.js +94 -35
- package/scripts/upgrade.js +187 -0
- package/templates/agent-next-steps.md +58 -0
- package/templates/qa-readme.md +86 -0
- package/templates/test-data-guidelines.md +130 -0
package/README.md
CHANGED
|
@@ -41,7 +41,7 @@ The postinstall script scaffolds the `qa/` structure automatically. No need to r
|
|
|
41
41
|
Use this prompt in your IDE agent chat:
|
|
42
42
|
|
|
43
43
|
```
|
|
44
|
-
Install @keber/qa-framework and set up the project
|
|
44
|
+
Install @keber/qa-framework, check it's readme file and set up the project for integration with azure devops
|
|
45
45
|
```
|
|
46
46
|
|
|
47
47
|
The install creates `qa/AGENT-NEXT-STEPS.md` with the exact steps the agent needs to complete the setup (optional integrations, module configuration, credentials, etc.).
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Module Analysis
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-module-analysis/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/00-module-analysis.md`
|
|
4
7
|
**Purpose**: Instructions for an AI agent to perform complete functional analysis of an application module and produce the 6-file specification set.
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Specification Generation
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-spec-generation/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/01-spec-generation.md`
|
|
4
7
|
**Purpose**: Detailed instructions for generating the 6-file submodule specification set from a completed module analysis or from existing source material (UI observation, requirements docs, developer input).
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Test Plan Generation
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-plan/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/02-test-plan-generation.md`
|
|
4
7
|
**Purpose**: Instructions for generating a test plan document for a module, after its spec set is complete.
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Test Case Generation
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-cases/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/03-test-case-generation.md`
|
|
4
7
|
**Purpose**: Instructions for writing detailed, standalone test case documents from approved spec scenarios.
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Automation Generation
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-automation/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/04-automation-generation.md`
|
|
4
7
|
**Purpose**: Instructions for writing Playwright E2E automation tests that trace back to approved test cases in `01-specifications/`.
|
|
5
8
|
|
|
@@ -308,3 +311,33 @@ After writing automation for a module, create `COVERAGE-MAPPING.md` in the test
|
|
|
308
311
|
| TC-OPER-CAT-003 | Duplicate rejection | ... | tests/operacion/... | ⚠️ Partially (UI only) |
|
|
309
312
|
| TC-OPER-ASI-001 | Access assistance list | ... | — | ⛔ Blocked (BLOCKED-PERMISSIONS) |
|
|
310
313
|
```
|
|
314
|
+
|
|
315
|
+
---
|
|
316
|
+
|
|
317
|
+
## Module Completion Checklist
|
|
318
|
+
|
|
319
|
+
Run this checklist when all P0 tests for a module are passing.
|
|
320
|
+
|
|
321
|
+
### Stability gate
|
|
322
|
+
- [ ] All P0 tests pass on **2 consecutive runs** with different `EXEC_IDX` values (≥ 60 seconds apart)
|
|
323
|
+
- [ ] No `waitForTimeout(N)` longer than 2000ms left in the suite
|
|
324
|
+
- [ ] All skipped tests have a `DEF-` or ADO WI reference in their skip message
|
|
325
|
+
|
|
326
|
+
### Artifacts
|
|
327
|
+
- [ ] `COVERAGE-MAPPING.md` created or updated in the test directory
|
|
328
|
+
- [ ] Execution report written to `qa/05-test-execution/automated/`
|
|
329
|
+
|
|
330
|
+
### Living Index update — `qa/README.md`
|
|
331
|
+
- [ ] Update the module's row in the status table (TCs automated, date, status → ✅)
|
|
332
|
+
- [ ] Add a row to the `## Sprint History` section with: sprint name, date, submódulos covered, TC count, link to execution report
|
|
333
|
+
|
|
334
|
+
### AGENT-NEXT-STEPS.md — trim
|
|
335
|
+
- [ ] Move the completed sprint checklist block to `qa/README.md` under `## Sprint History`
|
|
336
|
+
- [ ] Delete the completed sprint section from `AGENT-NEXT-STEPS.md`
|
|
337
|
+
- [ ] Verify `AGENT-NEXT-STEPS.md` now contains **only one active sprint** (the next one)
|
|
338
|
+
|
|
339
|
+
### Memory
|
|
340
|
+
- [ ] If new patterns or environment quirks were discovered: add them to `qa/memory/` with a descriptive filename
|
|
341
|
+
- [ ] Update `qa/memory/INDEX.md` to reflect any new or updated files
|
|
342
|
+
|
|
343
|
+
> **Rule**: `AGENT-NEXT-STEPS.md` is a task queue, not a log. Completed sprints live in `qa/README.md`. The file must never accumulate more than one active sprint.
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Test Stabilization
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-test-stabilization/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/04b-test-stabilization.md`
|
|
4
7
|
**Purpose**: Instructions for exhaustively reviewing and stabilizing generated Playwright tests to ensure they are logically correct, technically sound, and free of false positives and false negatives before ADO integration.
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Azure DevOps Integration
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-ado-integration/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/05-ado-integration.md`
|
|
4
7
|
**Purpose**: Instructions for an agent to create, configure, and maintain Azure DevOps Test Plans integration with Playwright automation results.
|
|
5
8
|
|
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# Agent Instructions: Maintenance and Updates
|
|
2
2
|
|
|
3
|
+
> ⚠️ **DEPRECATED**: This file is superseded by `.github/skills/qa-maintenance/SKILL.md`.
|
|
4
|
+
> Kept as reference during the transition period. Do not update this file.
|
|
5
|
+
|
|
3
6
|
**File**: `agent-instructions/06-maintenance.md`
|
|
4
7
|
**Purpose**: Instructions for updating QA artifacts after application code changes.
|
|
5
8
|
|
|
@@ -114,6 +117,13 @@ After every maintenance session, update the master index:
|
|
|
114
117
|
- Automation status (if changed)
|
|
115
118
|
- Last updated date for the module
|
|
116
119
|
|
|
120
|
+
If a **sprint was fully completed** as part of this maintenance pass (all TCs passing after the changes):
|
|
121
|
+
1. Move the completed sprint checklist from `AGENT-NEXT-STEPS.md` to the `## Sprint History` section of `qa/README.md`
|
|
122
|
+
2. Delete the completed sprint block from `AGENT-NEXT-STEPS.md`
|
|
123
|
+
3. Confirm `AGENT-NEXT-STEPS.md` contains **only one active sprint** — never let it accumulate history
|
|
124
|
+
|
|
125
|
+
Also update `qa/memory/INDEX.md` if any memory files were created or modified.
|
|
126
|
+
|
|
117
127
|
---
|
|
118
128
|
|
|
119
129
|
## Step 6: Write a Session Summary
|
|
@@ -11,10 +11,10 @@ This document explains the purpose, contents, and conventions for every folder i
|
|
|
11
11
|
```
|
|
12
12
|
qa/
|
|
13
13
|
├── README.md ← Living master index (project-maintained)
|
|
14
|
+
├── AGENT-NEXT-STEPS.md ← Active sprint queue for the agent (project-maintained)
|
|
14
15
|
├── QA-STRUCTURE-GUIDE.md ← Copy of this guide (installed by framework)
|
|
15
16
|
├── qa-framework.config.json ← Project configuration
|
|
16
17
|
│
|
|
17
|
-
├── 00-guides/ ← Process guides and agent instructions
|
|
18
18
|
├── 00-standards/ ← Naming conventions and artifact templates
|
|
19
19
|
├── 01-specifications/ ← Functional specifications per module
|
|
20
20
|
├── 02-test-plans/ ← Test plans (scope + priority decisions)
|
|
@@ -23,31 +23,34 @@ qa/
|
|
|
23
23
|
├── 05-test-execution/ ← Execution reports and results
|
|
24
24
|
├── 06-defects/ ← Defect tracking (optional)
|
|
25
25
|
├── 07-automation/ ← Automation code (Playwright, etc.)
|
|
26
|
-
|
|
26
|
+
├── 08-azure-integration/ ← Azure DevOps integration (optional)
|
|
27
|
+
└── memory/ ← Project QA learnings (project-maintained)
|
|
27
28
|
```
|
|
28
29
|
|
|
30
|
+
Agent skills live in `.github/skills/` (not inside `qa/`). See the `.github/skills/` section below.
|
|
31
|
+
|
|
29
32
|
---
|
|
30
33
|
|
|
31
|
-
##
|
|
34
|
+
## .github/skills/
|
|
32
35
|
|
|
33
|
-
**Purpose**:
|
|
36
|
+
**Purpose**: 3-layer QA pipeline skills installed by the framework. Loaded by the agent on demand — only the relevant skill for the current task is loaded, not all skills at once.
|
|
34
37
|
|
|
35
|
-
**Installed by framework**
|
|
38
|
+
**Installed by framework** via `npm install @keber/qa-framework`:
|
|
36
39
|
|
|
37
|
-
|
|
|
38
|
-
|
|
39
|
-
| `
|
|
40
|
-
| `
|
|
41
|
-
| `
|
|
42
|
-
| `
|
|
43
|
-
| `
|
|
44
|
-
| `
|
|
45
|
-
| `
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
- `
|
|
50
|
-
- `
|
|
40
|
+
| Skill folder | Stage | When the agent loads it |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| `qa-module-analysis/` | 1 | Analyzing a new module or refreshing specs |
|
|
43
|
+
| `qa-spec-generation/` | 2 | Writing the 6-file spec set |
|
|
44
|
+
| `qa-test-plan/` | 3 | Creating a test plan document |
|
|
45
|
+
| `qa-test-cases/` | 4 | Writing TC-*.md files |
|
|
46
|
+
| `qa-automation/` | 5 | Implementing Playwright specs |
|
|
47
|
+
| `qa-test-stabilization/` | 5b | Diagnosing and fixing failing tests |
|
|
48
|
+
| `qa-ado-integration/` | opt | Syncing with Azure DevOps Test Plans |
|
|
49
|
+
| `qa-maintenance/` | 6 | Updating specs and tests after app changes |
|
|
50
|
+
|
|
51
|
+
Each skill folder contains:
|
|
52
|
+
- `SKILL.md` — process outline (~400-600 tokens, always loaded when skill applies)
|
|
53
|
+
- `references/` — detailed templates and code patterns (loaded on demand)
|
|
51
54
|
|
|
52
55
|
---
|
|
53
56
|
|
|
@@ -267,17 +270,69 @@ See `docs/generalization-decisions.md §7` for the full evaluation of this folde
|
|
|
267
270
|
|
|
268
271
|
## `qa/README.md` — The Living Index
|
|
269
272
|
|
|
270
|
-
The `qa/README.md` is the
|
|
273
|
+
The `qa/README.md` is the primary human-readable project status document. It should be updated whenever:
|
|
271
274
|
- A new module is analyzed
|
|
272
275
|
- A new test execution is completed
|
|
273
276
|
- A blocker is identified or resolved
|
|
274
|
-
- A major milestone is
|
|
277
|
+
- A major milestone or sprint is completed
|
|
275
278
|
|
|
276
279
|
Minimum contents:
|
|
277
280
|
- Module status table (name, TCs documented, TCs automated, ADO plan link)
|
|
278
281
|
- Quick-start commands for common tasks
|
|
279
282
|
- Active blockers section
|
|
280
283
|
- Last execution results summary
|
|
284
|
+
- **`## Sprint History`** section — completed sprint checklists moved here from `AGENT-NEXT-STEPS.md`
|
|
285
|
+
|
|
286
|
+
Template: `templates/qa-readme.md`
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## `qa/AGENT-NEXT-STEPS.md` — Sprint Queue
|
|
291
|
+
|
|
292
|
+
The `AGENT-NEXT-STEPS.md` is the agent's task queue for the current sprint. It is read by the agent at the start of every conversation.
|
|
293
|
+
|
|
294
|
+
**Design principles (SRP + KISS)**:
|
|
295
|
+
- Contains **one active sprint only** — no history, no completed checklists
|
|
296
|
+
- Three sections maximum: module status table, active sprint checklist, context references
|
|
297
|
+
- Does not repeat standing instructions already in `copilot-instructions.md`
|
|
298
|
+
- Detailed environment notes and patterns belong in `qa/memory/`, referenced from here
|
|
299
|
+
|
|
300
|
+
When a sprint completes, the agent **moves** the completed checklist to `qa/README.md → ## Sprint History` and **deletes** that section from this file.
|
|
301
|
+
|
|
302
|
+
Template: `templates/agent-next-steps.md`
|
|
303
|
+
|
|
304
|
+
---
|
|
305
|
+
|
|
306
|
+
## `qa/memory/` — Project QA Learnings
|
|
307
|
+
|
|
308
|
+
Project-maintained folder for accumulated knowledge about the application under test: environment quirks, DOM patterns, Playwright gotchas, sprint-specific discoveries.
|
|
309
|
+
|
|
310
|
+
```
|
|
311
|
+
qa/memory/
|
|
312
|
+
├── INDEX.md ← Required entry point — index of all files
|
|
313
|
+
├── {technology}-patterns.md ← Framework/stack-specific patterns (e.g., playwright-blazor-wasm-patterns.md)
|
|
314
|
+
├── {sprint-name}-discovery.md ← DOM/UI findings for an in-progress sprint
|
|
315
|
+
└── {sprint-name}-lessons.md ← Retrospective patterns after a sprint completes
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
### `qa/memory/INDEX.md` — Required
|
|
319
|
+
|
|
320
|
+
The `INDEX.md` is the **gate** that the agent reads before loading any other memory file. Its purpose is to let the agent load only what is relevant to the current task, rather than loading all files.
|
|
321
|
+
|
|
322
|
+
```markdown
|
|
323
|
+
# Memory Index — {PROJECT_NAME}
|
|
324
|
+
|
|
325
|
+
| File | Topic | When to load |
|
|
326
|
+
|---|---|---|
|
|
327
|
+
| playwright-blazor-wasm-patterns.md | Blazor/Radzen DOM patterns | Any Playwright automation task |
|
|
328
|
+
| sprint4-discovery.md | DOM quirks for BusinessPartner module | Sprint 4 automation only |
|
|
329
|
+
| sprint3-lessons.md | Patterns from Sprint 3 | Reference when working on similar modules |
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
**Rules for `qa/memory/`**:
|
|
333
|
+
- Always update `INDEX.md` when adding or modifying a memory file
|
|
334
|
+
- Sprint-specific discovery files (`sprint-N-discovery.md`) are for the duration of that sprint; consolidate long-term patterns into a technology-specific file afterwards
|
|
335
|
+
- Never load all memory files unconditionally — always read `INDEX.md` first and select by relevance
|
|
281
336
|
|
|
282
337
|
---
|
|
283
338
|
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
# Skills Architecture
|
|
2
|
+
|
|
3
|
+
> How QA agent skills are organized in `@keber/qa-framework`.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Why Skills, Not Monolithic Instructions
|
|
8
|
+
|
|
9
|
+
The previous `agent-instructions/` model loaded one large file (~2,500 tokens each) per task. For an automation session covering 3 submodules, the agent would load the full automation file whether it needed the ADO config or not. With 8 instruction files, that was 20,000+ tokens of constant context pressure.
|
|
10
|
+
|
|
11
|
+
The 3-layer skill architecture reduces this to ~500 tokens for a typical task, loading detail only when needed.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## 3-Layer Model
|
|
16
|
+
|
|
17
|
+
| Layer | Location | Size | Purpose |
|
|
18
|
+
|---|---|---|---|
|
|
19
|
+
| 1 — Descriptor | YAML frontmatter in `SKILL.md` | ~50 tokens | Name + description; used for relevance detection |
|
|
20
|
+
| 2 — Process outline | `SKILL.md` body | ~400–600 tokens | Step-by-step process without code; always loaded when skill applies |
|
|
21
|
+
| 3 — Reference material | `references/*.md` | ~500–1500 tokens each | Detailed templates, code patterns, decision trees; loaded on demand |
|
|
22
|
+
|
|
23
|
+
The agent reads Layer 2 to understand what to do, then loads only the specific `references/` files it needs for the current step.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Pipeline Sequence
|
|
28
|
+
|
|
29
|
+
Skills map to the QA pipeline stages defined in `copilot-instructions.md`. The file acts as a **pipeline sequencer** — it defines which stage comes before which, and what the prerequisite is. Skills are the implementation of each stage.
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
Stage 1: qa-module-analysis (no prerequisite)
|
|
33
|
+
│
|
|
34
|
+
▼
|
|
35
|
+
Stage 2: qa-spec-generation (requires 00-inventory.md)
|
|
36
|
+
│
|
|
37
|
+
▼
|
|
38
|
+
Stage 3: qa-test-plan (requires 05-test-scenarios.md)
|
|
39
|
+
│
|
|
40
|
+
▼
|
|
41
|
+
Stage 4: qa-test-cases (requires test plan)
|
|
42
|
+
│
|
|
43
|
+
▼
|
|
44
|
+
Stage 5: qa-automation (requires approved specs, no PENDING-CODE)
|
|
45
|
+
│
|
|
46
|
+
├─ Stage 5b: qa-test-stabilization (if tests fail)
|
|
47
|
+
│
|
|
48
|
+
▼
|
|
49
|
+
Stage 6: qa-maintenance (after each application change)
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
Optional (any time after Stage 4):
|
|
53
|
+
qa-ado-integration (requires ADO enabled in config)
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## Skill Inventory
|
|
59
|
+
|
|
60
|
+
| Skill | SKILL.md | References |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| `qa-module-analysis` | Stage 1 process (4 phases) | `exploration-checklist.md`, `spec-file-formats.md` |
|
|
63
|
+
| `qa-spec-generation` | Stage 2 rules + anti-patterns | `spec-file-formats.md` (shared with qa-module-analysis) |
|
|
64
|
+
| `qa-test-plan` | Stage 3 — priority + feasibility rules | `priority-and-feasibility.md` (includes full template) |
|
|
65
|
+
| `qa-test-cases` | Stage 4 — TC document rules | `test-case-template.md` |
|
|
66
|
+
| `qa-automation` | Stage 5 — patterns + completion checklist | `patterns.md`, `config-checklist.md` |
|
|
67
|
+
| `qa-test-stabilization` | Stage 5b — 8 steps + confidence scoring | `classification-protocol.md` (includes report template) |
|
|
68
|
+
| `qa-ado-integration` | Optional ADO sync — 6 steps | `scripts-and-config.md` |
|
|
69
|
+
| `qa-maintenance` | Stage 6 — update rules | `update-rules.md` |
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## How Skills Are Installed
|
|
74
|
+
|
|
75
|
+
When a consumer project runs `npm install @keber/qa-framework`, the `postinstall` script (`scripts/init.js`) copies the entire `skills/` directory from the package to `.github/skills/` in the consumer project:
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
@keber/qa-framework/skills/
|
|
79
|
+
qa-module-analysis/
|
|
80
|
+
SKILL.md
|
|
81
|
+
references/
|
|
82
|
+
exploration-checklist.md
|
|
83
|
+
spec-file-formats.md
|
|
84
|
+
qa-spec-generation/
|
|
85
|
+
SKILL.md
|
|
86
|
+
...
|
|
87
|
+
|
|
88
|
+
→ copies to →
|
|
89
|
+
|
|
90
|
+
{consumer-project}/.github/skills/
|
|
91
|
+
qa-module-analysis/
|
|
92
|
+
SKILL.md
|
|
93
|
+
references/
|
|
94
|
+
exploration-checklist.md
|
|
95
|
+
spec-file-formats.md
|
|
96
|
+
...
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
The copy uses `writeIfMissing` — it will not overwrite files that already exist in the consumer project. This allows consumer projects to customize their `.github/skills/` without losing changes on re-install.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## How to Add a Project-Specific Skill
|
|
104
|
+
|
|
105
|
+
Consumer projects can extend the skills by adding skill folders directly to `.github/skills/`:
|
|
106
|
+
|
|
107
|
+
1. Create `.github/skills/{skill-name}/SKILL.md` with YAML frontmatter:
|
|
108
|
+
```yaml
|
|
109
|
+
---
|
|
110
|
+
name: my-custom-skill
|
|
111
|
+
description: >
|
|
112
|
+
Describe when this skill should be triggered.
|
|
113
|
+
---
|
|
114
|
+
```
|
|
115
|
+
2. Add the skill body following the same 4-section structure: prerequisite → steps → rules → outputs
|
|
116
|
+
3. Optionally add `references/*.md` for detailed templates
|
|
117
|
+
4. Reference the skill from `copilot-instructions.md` if it's part of your pipeline
|
|
118
|
+
|
|
119
|
+
Project-specific skills are not overwritten by `npm install` updates.
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## Mapping: Old vs New
|
|
124
|
+
|
|
125
|
+
| Former file | New skill | Token reduction |
|
|
126
|
+
|---|---|---|
|
|
127
|
+
| `agent-instructions/00-module-analysis.md` | `.github/skills/qa-module-analysis/SKILL.md` | ~2,500 → ~500 + on-demand references |
|
|
128
|
+
| `agent-instructions/01-spec-generation.md` | `.github/skills/qa-spec-generation/SKILL.md` | ~2,800 → ~400 + shared references |
|
|
129
|
+
| `agent-instructions/02-test-plan-generation.md` | `.github/skills/qa-test-plan/SKILL.md` | ~2,200 → ~400 + test-plan-template |
|
|
130
|
+
| `agent-instructions/03-test-case-generation.md` | `.github/skills/qa-test-cases/SKILL.md` | ~2,100 → ~400 + tc-template |
|
|
131
|
+
| `agent-instructions/04-automation-generation.md` | `.github/skills/qa-automation/SKILL.md` | ~3,500 → ~500 + patterns + config |
|
|
132
|
+
| `agent-instructions/04b-test-stabilization.md` | `.github/skills/qa-test-stabilization/SKILL.md` | ~3,800 → ~500 + classification |
|
|
133
|
+
| `agent-instructions/05-ado-integration.md` | `.github/skills/qa-ado-integration/SKILL.md` | ~2,400 → ~400 + scripts |
|
|
134
|
+
| `agent-instructions/06-maintenance.md` | `.github/skills/qa-maintenance/SKILL.md` | ~1,700 → ~400 + update-rules |
|
|
135
|
+
|
|
136
|
+
**Total context reduction**: ~21,000 tokens → ~3,500 tokens (Layer 2 only) + selective Layer 3 loading
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@keber/qa-framework",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.0-0",
|
|
4
4
|
"description": "Reusable spec-driven QA framework for IDE-agent-assisted automated testing. Installable as an npm package. Provides structure, templates, agent instructions, and optional integrations for Playwright and Azure DevOps.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"qa",
|
package/qa-framework.config.json
CHANGED
package/scripts/cli.js
CHANGED
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
*
|
|
9
9
|
* Commands:
|
|
10
10
|
* init Scaffold a qa/ folder structure from qa-framework.config.json
|
|
11
|
+
* upgrade Update framework-owned files after npm update (skills, copilot-instructions)
|
|
11
12
|
* generate Generate an artifact (spec template, test plan, etc.)
|
|
12
13
|
* validate Validate qa/ folder structure and conventions
|
|
13
14
|
*/
|
|
@@ -21,6 +22,7 @@ const [, , command, ...args] = process.argv;
|
|
|
21
22
|
|
|
22
23
|
const commands = {
|
|
23
24
|
init: 'init.js',
|
|
25
|
+
upgrade: 'upgrade.js',
|
|
24
26
|
generate: 'generate.js',
|
|
25
27
|
validate: 'validate.js',
|
|
26
28
|
};
|
|
@@ -54,6 +56,7 @@ Usage:
|
|
|
54
56
|
|
|
55
57
|
Commands:
|
|
56
58
|
init [--config <path>] Scaffold qa/ structure from config file
|
|
59
|
+
upgrade [--dry-run] Update framework-owned files after npm update
|
|
57
60
|
generate <artifact> Generate from template
|
|
58
61
|
artifact: spec | test-plan | test-case |
|
|
59
62
|
execution-report | defect-report |
|
|
@@ -67,6 +70,8 @@ Options:
|
|
|
67
70
|
Examples:
|
|
68
71
|
qa-framework init
|
|
69
72
|
qa-framework init --config ./my-project.config.json
|
|
73
|
+
qa-framework upgrade
|
|
74
|
+
qa-framework upgrade --dry-run
|
|
70
75
|
qa-framework generate spec --module suppliers --submodule create
|
|
71
76
|
qa-framework validate
|
|
72
77
|
qa-framework validate --strict
|
package/scripts/init.js
CHANGED
|
@@ -77,13 +77,17 @@ process.stderr.write(`[qa-framework/init] Using config source: ${configSource}\n
|
|
|
77
77
|
|
|
78
78
|
// --- Top-level folders ---
|
|
79
79
|
const topLevelFolders = [
|
|
80
|
-
'00-guides',
|
|
81
80
|
'00-standards',
|
|
82
|
-
'
|
|
81
|
+
'01-specifications',
|
|
82
|
+
'02-test-plans',
|
|
83
|
+
'03-test-cases',
|
|
84
|
+
'04-test-data',
|
|
85
|
+
'05-test-execution',
|
|
83
86
|
'06-defects/open',
|
|
84
87
|
'06-defects/resolved',
|
|
85
88
|
'07-automation/e2e',
|
|
86
89
|
'08-azure-integration',
|
|
90
|
+
'memory',
|
|
87
91
|
];
|
|
88
92
|
|
|
89
93
|
for (const folder of topLevelFolders) {
|
|
@@ -92,12 +96,47 @@ for (const folder of topLevelFolders) {
|
|
|
92
96
|
console.log(` [created] ${path.relative(cwd, fullPath)}/`);
|
|
93
97
|
}
|
|
94
98
|
|
|
99
|
+
// --- qa/README.md ---
|
|
100
|
+
const qaReadmeTemplate = path.resolve(__dirname, '..', 'templates', 'qa-readme.md');
|
|
101
|
+
if (fs.existsSync(qaReadmeTemplate)) {
|
|
102
|
+
let readmeContent = fs.readFileSync(qaReadmeTemplate, 'utf8');
|
|
103
|
+
readmeContent = readmeContent
|
|
104
|
+
.replace(/\{PROJECT_DISPLAY_NAME\}/g, config.project?.displayName ?? config.project?.name ?? 'Project')
|
|
105
|
+
.replace(/\{VERSION\}/g, config.frameworkVersion ?? '1.0.0')
|
|
106
|
+
.replace(/\{LANGUAGE\}/g, config.conventions?.language ?? 'en')
|
|
107
|
+
.replace(/\{QA_BASE_URL\}/g, config.project?.qaBaseUrl ?? '<QA_BASE_URL>');
|
|
108
|
+
writeIfMissing(path.join(qaRoot, 'README.md'), readmeContent);
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
// --- qa/memory/INDEX.md ---
|
|
112
|
+
const memoryDir = path.join(qaRoot, 'memory');
|
|
113
|
+
writeIfMissing(path.join(memoryDir, 'INDEX.md'),
|
|
114
|
+
`# Memory Index — ${config.project?.displayName ?? config.project?.name ?? 'Project'}
|
|
115
|
+
|
|
116
|
+
> The agent reads this file first before loading any memory file.
|
|
117
|
+
> Add a row here whenever you create or update a file in this directory.
|
|
118
|
+
|
|
119
|
+
| File | Topic | When to load |
|
|
120
|
+
|---|---|---|
|
|
121
|
+
|
|
122
|
+
_No memory files yet. Add files to this directory and register them above._
|
|
123
|
+
`);
|
|
124
|
+
|
|
95
125
|
// --- Standards placeholder ---
|
|
96
126
|
const stdDir = path.join(qaRoot, '00-standards');
|
|
97
127
|
writeIfMissing(path.join(stdDir, 'naming-conventions.md'), `# Naming Conventions\n\n> Fill in per your project. See keber/qa-framework docs/spec-driven-philosophy.md.\n`);
|
|
98
128
|
writeIfMissing(path.join(stdDir, 'bug-report-template.md'), fs.readFileSync(
|
|
99
129
|
path.resolve(__dirname, '..', 'templates', 'defect-report.md'), 'utf8'
|
|
100
130
|
));
|
|
131
|
+
writeIfMissing(path.join(stdDir, 'test-case-template.md'), fs.readFileSync(
|
|
132
|
+
path.resolve(__dirname, '..', 'templates', 'test-case.md'), 'utf8'
|
|
133
|
+
));
|
|
134
|
+
writeIfMissing(path.join(stdDir, 'execution-report-template.md'), fs.readFileSync(
|
|
135
|
+
path.resolve(__dirname, '..', 'templates', 'execution-report.md'), 'utf8'
|
|
136
|
+
));
|
|
137
|
+
writeIfMissing(path.join(stdDir, 'test-data-guidelines.md'), fs.readFileSync(
|
|
138
|
+
path.resolve(__dirname, '..', 'templates', 'test-data-guidelines.md'), 'utf8'
|
|
139
|
+
));
|
|
101
140
|
|
|
102
141
|
// --- Per-module folders ---
|
|
103
142
|
const modules = config.modules ?? [];
|
|
@@ -182,24 +221,18 @@ const adoDir = path.join(qaRoot, '08-azure-integration');
|
|
|
182
221
|
writeIfMissing(path.join(adoDir, 'README.md'), `# ADO Integration\n\nSee keber/qa-framework integrations/ado-powershell/ for setup instructions.\n`);
|
|
183
222
|
writeIfMissing(path.join(adoDir, 'module-registry.json'), JSON.stringify({ modules: [] }, null, 2));
|
|
184
223
|
|
|
185
|
-
// ---
|
|
186
|
-
const
|
|
187
|
-
const
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
};
|
|
198
|
-
for (const [src, dest] of Object.entries(agentFileMap)) {
|
|
199
|
-
const srcPath = path.join(agentInstrSrc, src);
|
|
200
|
-
const destPath = path.join(guidesDir, dest);
|
|
201
|
-
if (fs.existsSync(srcPath)) {
|
|
202
|
-
writeIfMissing(destPath, fs.readFileSync(srcPath, 'utf8'));
|
|
224
|
+
// --- Skills → .github/skills/ ---
|
|
225
|
+
const skillsSrc = path.resolve(__dirname, '..', 'skills');
|
|
226
|
+
const skillsDest = path.join(cwd, '.github', 'skills');
|
|
227
|
+
fs.mkdirSync(skillsDest, { recursive: true });
|
|
228
|
+
if (fs.existsSync(skillsSrc)) {
|
|
229
|
+
for (const skillName of fs.readdirSync(skillsSrc)) {
|
|
230
|
+
const skillSrcDir = path.join(skillsSrc, skillName);
|
|
231
|
+
if (!fs.statSync(skillSrcDir).isDirectory()) continue;
|
|
232
|
+
const skillDestDir = path.join(skillsDest, skillName);
|
|
233
|
+
fs.mkdirSync(skillDestDir, { recursive: true });
|
|
234
|
+
// Copy SKILL.md and references/ recursively
|
|
235
|
+
copyDirIfMissing(skillSrcDir, skillDestDir);
|
|
203
236
|
}
|
|
204
237
|
}
|
|
205
238
|
|
|
@@ -248,24 +281,37 @@ This project uses \`@keber/qa-framework\` v${config.frameworkVersion ?? '1.0.0'}
|
|
|
248
281
|
## Agent behavior rules
|
|
249
282
|
|
|
250
283
|
0. **On every conversation start:** check if \`qa/AGENT-NEXT-STEPS.md\` exists. If it does, read it and complete its steps before anything else.
|
|
251
|
-
1. Before performing any QA task,
|
|
284
|
+
1. Before performing any QA task, load the relevant skill from \`.github/skills/\`. For project context, read \`qa/memory/INDEX.md\` if it exists, then load only the memory files relevant to the current task — do not load all memory files unconditionally. Never execute a pipeline stage unless its prerequisite is satisfied.
|
|
252
285
|
2. Always save artifacts in the correct \`qa/\` subfolder - refer to \`qa/QA-STRUCTURE-GUIDE.md\`
|
|
253
286
|
3. Never hardcode credentials - always use env vars and \`<PLACEHOLDER>\` in documentation
|
|
254
287
|
4. Follow the naming conventions in \`qa/00-standards/naming-conventions.md\`
|
|
255
|
-
5.
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
|
288
|
+
5. Read previous test suites code in \`qa/07-automation/e2e/tests/\` if available before writing new automation to maintain consistency in style and approach
|
|
289
|
+
6. Project QA config is at \`qa/qa-framework.config.json\`
|
|
290
|
+
7. Save learnings in \`qa/memory/\` with proper naming and metadata. Always update \`qa/memory/INDEX.md\` after adding or updating any memory file.
|
|
291
|
+
8. **On module/sprint completion** (all TCs passing): (1) update the module status row in \`qa/README.md\`; (2) move the completed sprint checklist from \`AGENT-NEXT-STEPS.md\` to the \`## Sprint History\` section of \`qa/README.md\`; (3) trim \`AGENT-NEXT-STEPS.md\` so it contains only the next sprint — never let it accumulate more than one active sprint.
|
|
292
|
+
|
|
293
|
+
## Formatting rules for generated content
|
|
294
|
+
1. Generate the output in \`UTF-8\` encoding without BOM (\`UTF-8\`, no signature).
|
|
295
|
+
2. Ensure the raw output is \`UTF-8\` encoded with no \`EF BB BF\` bytes at the beginning.
|
|
296
|
+
|
|
297
|
+
## QA Pipeline
|
|
298
|
+
|
|
299
|
+
Stages must run in order. Never start a stage unless its prerequisite is met.
|
|
300
|
+
|
|
301
|
+
| Stage | Task | Skill | Prerequisite |
|
|
302
|
+
|---|---|---|---|
|
|
303
|
+
| 1 | Analyze module | \`.github/skills/qa-module-analysis/SKILL.md\` | None |
|
|
304
|
+
| 2 | Generate specifications | \`.github/skills/qa-spec-generation/SKILL.md\` | 00-inventory.md exists |
|
|
305
|
+
| 3 | Generate test plan | \`.github/skills/qa-test-plan/SKILL.md\` | 05-test-scenarios.md exists |
|
|
306
|
+
| 4 | Generate test cases | \`.github/skills/qa-test-cases/SKILL.md\` | Test plan exists |
|
|
307
|
+
| 5 | Generate automation | \`.github/skills/qa-automation/SKILL.md\` | Specs approved, no PENDING-CODE |
|
|
308
|
+
| 5b | Stabilize failing tests | \`.github/skills/qa-test-stabilization/SKILL.md\` | Failing tests exist |
|
|
309
|
+
| 6 | Maintenance | \`.github/skills/qa-maintenance/SKILL.md\` | Application change delivered |
|
|
310
|
+
| — | ADO integration | \`.github/skills/qa-ado-integration/SKILL.md\` | ADO enabled in config |
|
|
311
|
+
|
|
312
|
+
## Project KB
|
|
313
|
+
1. \`qa/README.md\` — The Living Index: module status, sprint history, blockers, quick-start commands.
|
|
314
|
+
2. \`qa/memory/INDEX.md\` — Index of all memory files; load this before selecting which files to read.
|
|
269
315
|
`;
|
|
270
316
|
writeIfMissing(copilotInstrPath, copilotInstrContent);
|
|
271
317
|
|
|
@@ -317,6 +363,19 @@ function writeIfMissing(filePath, content) {
|
|
|
317
363
|
}
|
|
318
364
|
}
|
|
319
365
|
|
|
366
|
+
function copyDirIfMissing(srcDir, destDir) {
|
|
367
|
+
for (const entry of fs.readdirSync(srcDir, { withFileTypes: true })) {
|
|
368
|
+
const srcPath = path.join(srcDir, entry.name);
|
|
369
|
+
const destPath = path.join(destDir, entry.name);
|
|
370
|
+
if (entry.isDirectory()) {
|
|
371
|
+
fs.mkdirSync(destPath, { recursive: true });
|
|
372
|
+
copyDirIfMissing(srcPath, destPath);
|
|
373
|
+
} else {
|
|
374
|
+
writeIfMissing(destPath, fs.readFileSync(srcPath, 'utf8'));
|
|
375
|
+
}
|
|
376
|
+
}
|
|
377
|
+
}
|
|
378
|
+
|
|
320
379
|
function specScaffold(modName, subName, modKey, subKey) {
|
|
321
380
|
return `import { test, expect } from '@playwright/test';
|
|
322
381
|
|
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* scripts/upgrade.js — Upgrade framework-owned files in an existing project
|
|
4
|
+
*
|
|
5
|
+
* Usage:
|
|
6
|
+
* qa-framework upgrade
|
|
7
|
+
* qa-framework upgrade --dry-run
|
|
8
|
+
*
|
|
9
|
+
* What this command does (safe by design):
|
|
10
|
+
* ALWAYS updates (framework-owned):
|
|
11
|
+
* - .github/skills/ ← New skills architecture; always current
|
|
12
|
+
* - .github/copilot-instructions.md ← Pipeline sequencer; framework-owned
|
|
13
|
+
* - qa/QA-STRUCTURE-GUIDE.md ← Folder reference; framework-owned
|
|
14
|
+
*
|
|
15
|
+
* NEVER touches (project-owned):
|
|
16
|
+
* - qa/01-specifications/ ← Your specs
|
|
17
|
+
* - qa/02-test-plans/ ← Your test plans
|
|
18
|
+
* - qa/03-test-cases/ ← Your TCs
|
|
19
|
+
* - qa/07-automation/ ← Your Playwright code
|
|
20
|
+
* - qa/memory/ ← Your learnings
|
|
21
|
+
* - qa/README.md ← Your living index
|
|
22
|
+
* - qa/AGENT-NEXT-STEPS.md ← Your sprint queue
|
|
23
|
+
*
|
|
24
|
+
* REPORTS but does not delete:
|
|
25
|
+
* - qa/00-guides/ ← Old location (can be deleted manually)
|
|
26
|
+
*/
|
|
27
|
+
|
|
28
|
+
'use strict';
|
|
29
|
+
|
|
30
|
+
const fs = require('fs');
|
|
31
|
+
const path = require('path');
|
|
32
|
+
|
|
33
|
+
const args = process.argv.slice(2);
|
|
34
|
+
const dryRun = args.includes('--dry-run');
|
|
35
|
+
|
|
36
|
+
const cwd = process.env.INIT_CWD || process.cwd();
|
|
37
|
+
|
|
38
|
+
// Resolve config (same logic as init.js)
|
|
39
|
+
const rootConfigPath = path.resolve(cwd, 'qa-framework.config.json');
|
|
40
|
+
const qaConfigPath = path.resolve(cwd, 'qa', 'qa-framework.config.json');
|
|
41
|
+
let config = {};
|
|
42
|
+
|
|
43
|
+
const configFlag = args.indexOf('--config');
|
|
44
|
+
if (configFlag !== -1 && args[configFlag + 1]) {
|
|
45
|
+
config = JSON.parse(fs.readFileSync(path.resolve(cwd, args[configFlag + 1]), 'utf8'));
|
|
46
|
+
} else if (fs.existsSync(rootConfigPath)) {
|
|
47
|
+
config = JSON.parse(fs.readFileSync(rootConfigPath, 'utf8'));
|
|
48
|
+
} else if (fs.existsSync(qaConfigPath)) {
|
|
49
|
+
config = JSON.parse(fs.readFileSync(qaConfigPath, 'utf8'));
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const qaRoot = path.join(cwd, 'qa');
|
|
53
|
+
const githubDir = path.join(cwd, '.github');
|
|
54
|
+
const skillsDest = path.join(githubDir, 'skills');
|
|
55
|
+
const skillsSrc = path.resolve(__dirname, '..', 'skills');
|
|
56
|
+
|
|
57
|
+
const updated = [];
|
|
58
|
+
const skipped = [];
|
|
59
|
+
const warnings = [];
|
|
60
|
+
|
|
61
|
+
console.log(`\n[qa-framework/upgrade] ${dryRun ? '(dry-run) ' : ''}Upgrading framework-owned files`);
|
|
62
|
+
console.log(`[qa-framework/upgrade] Project root: ${cwd}\n`);
|
|
63
|
+
|
|
64
|
+
// ---------------------------------------------------------------------------
|
|
65
|
+
// 1. .github/skills/ — always overwrite (framework-owned)
|
|
66
|
+
// ---------------------------------------------------------------------------
|
|
67
|
+
if (!fs.existsSync(skillsSrc)) {
|
|
68
|
+
warnings.push('skills/ source directory not found in package — skipping skill install');
|
|
69
|
+
} else {
|
|
70
|
+
fs.mkdirSync(skillsDest, { recursive: true });
|
|
71
|
+
copyDirForce(skillsSrc, skillsDest);
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
// ---------------------------------------------------------------------------
|
|
75
|
+
// 2. .github/copilot-instructions.md — overwrite (framework-owned sequencer)
|
|
76
|
+
// ---------------------------------------------------------------------------
|
|
77
|
+
const copilotInstrPath = path.join(githubDir, 'copilot-instructions.md');
|
|
78
|
+
const copilotContent = buildCopilotInstructions(config);
|
|
79
|
+
forceWrite(copilotInstrPath, copilotContent);
|
|
80
|
+
|
|
81
|
+
// ---------------------------------------------------------------------------
|
|
82
|
+
// 3. qa/QA-STRUCTURE-GUIDE.md — overwrite (framework doc)
|
|
83
|
+
// ---------------------------------------------------------------------------
|
|
84
|
+
const structureGuideSrc = path.resolve(__dirname, '..', 'docs', 'folder-structure-guide.md');
|
|
85
|
+
const structureGuideDest = path.join(qaRoot, 'QA-STRUCTURE-GUIDE.md');
|
|
86
|
+
if (fs.existsSync(structureGuideSrc) && fs.existsSync(qaRoot)) {
|
|
87
|
+
forceWrite(structureGuideDest, fs.readFileSync(structureGuideSrc, 'utf8'));
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
// ---------------------------------------------------------------------------
|
|
91
|
+
// 4. Migration: report qa/00-guides/ if still present
|
|
92
|
+
// ---------------------------------------------------------------------------
|
|
93
|
+
const oldGuidesDir = path.join(qaRoot, '00-guides');
|
|
94
|
+
if (fs.existsSync(oldGuidesDir)) {
|
|
95
|
+
warnings.push(
|
|
96
|
+
`qa/00-guides/ still exists (old location). It can be deleted:\n` +
|
|
97
|
+
` Remove-Item -Recurse -Force qa/00-guides`
|
|
98
|
+
);
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
// ---------------------------------------------------------------------------
|
|
102
|
+
// Summary
|
|
103
|
+
// ---------------------------------------------------------------------------
|
|
104
|
+
console.log('\n--- Results ---');
|
|
105
|
+
if (updated.length) {
|
|
106
|
+
console.log('\n Updated (framework-owned):');
|
|
107
|
+
for (const f of updated) console.log(` ✅ ${path.relative(cwd, f)}`);
|
|
108
|
+
}
|
|
109
|
+
if (skipped.length) {
|
|
110
|
+
console.log('\n Skipped (already up to date):');
|
|
111
|
+
for (const f of skipped) console.log(` — ${path.relative(cwd, f)}`);
|
|
112
|
+
}
|
|
113
|
+
if (warnings.length) {
|
|
114
|
+
console.log('\n Warnings:');
|
|
115
|
+
for (const w of warnings) console.log(` ⚠️ ${w}`);
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
console.log('\n[qa-framework/upgrade] Done.\n');
|
|
119
|
+
|
|
120
|
+
// -----------------------------------------------------------------------
|
|
121
|
+
// Helpers
|
|
122
|
+
// -----------------------------------------------------------------------
|
|
123
|
+
|
|
124
|
+
function forceWrite(filePath, content) {
|
|
125
|
+
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
|
126
|
+
const exists = fs.existsSync(filePath);
|
|
127
|
+
const changed = !exists || fs.readFileSync(filePath, 'utf8') !== content;
|
|
128
|
+
if (dryRun) {
|
|
129
|
+
if (changed) updated.push(filePath);
|
|
130
|
+
else skipped.push(filePath);
|
|
131
|
+
return;
|
|
132
|
+
}
|
|
133
|
+
fs.writeFileSync(filePath, content, 'utf8');
|
|
134
|
+
if (changed) updated.push(filePath);
|
|
135
|
+
else skipped.push(filePath);
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
function copyDirForce(srcDir, destDir) {
|
|
139
|
+
for (const entry of fs.readdirSync(srcDir, { withFileTypes: true })) {
|
|
140
|
+
const srcPath = path.join(srcDir, entry.name);
|
|
141
|
+
const destPath = path.join(destDir, entry.name);
|
|
142
|
+
if (entry.isDirectory()) {
|
|
143
|
+
fs.mkdirSync(destPath, { recursive: true });
|
|
144
|
+
copyDirForce(srcPath, destPath);
|
|
145
|
+
} else {
|
|
146
|
+
forceWrite(destPath, fs.readFileSync(srcPath, 'utf8'));
|
|
147
|
+
}
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
function buildCopilotInstructions(cfg) {
|
|
152
|
+
const version = cfg.frameworkVersion ?? '1.0.0';
|
|
153
|
+
return `# QA Framework Instructions
|
|
154
|
+
|
|
155
|
+
This project uses \`@keber/qa-framework\` v${version} for spec-driven automated testing.
|
|
156
|
+
|
|
157
|
+
## Agent behavior rules
|
|
158
|
+
|
|
159
|
+
0. **On every conversation start:** check if \`qa/AGENT-NEXT-STEPS.md\` exists. If it does, read it and complete its steps before anything else.
|
|
160
|
+
1. Before performing any QA task, load the relevant skill from \`.github/skills/\`. For project context, read \`qa/memory/INDEX.md\` if it exists, then load only the memory files relevant to the current task — do not load all memory files unconditionally. Never execute a pipeline stage unless its prerequisite is satisfied.
|
|
161
|
+
2. Always save artifacts in the correct \`qa/\` subfolder — refer to \`qa/QA-STRUCTURE-GUIDE.md\`
|
|
162
|
+
3. Never hardcode credentials — always use env vars and \`<PLACEHOLDER>\` in documentation
|
|
163
|
+
4. Follow the naming conventions in \`qa/00-standards/naming-conventions.md\`
|
|
164
|
+
5. Project QA config is at \`qa/qa-framework.config.json\`
|
|
165
|
+
6. Save learnings in \`qa/memory/\` with proper naming and metadata. Always update \`qa/memory/INDEX.md\` after adding or updating any memory file.
|
|
166
|
+
7. **On module/sprint completion** (all TCs passing): (1) update the module status row in \`qa/README.md\`; (2) move the completed sprint checklist from \`AGENT-NEXT-STEPS.md\` to the \`## Sprint History\` section of \`qa/README.md\`; (3) trim \`AGENT-NEXT-STEPS.md\` so it contains only the next sprint — never let it accumulate more than one active sprint.
|
|
167
|
+
|
|
168
|
+
## QA Pipeline
|
|
169
|
+
|
|
170
|
+
Stages must run in order. Never start a stage unless its prerequisite is met.
|
|
171
|
+
|
|
172
|
+
| Stage | Task | Skill | Prerequisite |
|
|
173
|
+
|---|---|---|---|
|
|
174
|
+
| 1 | Analyze module | \`.github/skills/qa-module-analysis/SKILL.md\` | None |
|
|
175
|
+
| 2 | Generate specifications | \`.github/skills/qa-spec-generation/SKILL.md\` | 00-inventory.md exists |
|
|
176
|
+
| 3 | Generate test plan | \`.github/skills/qa-test-plan/SKILL.md\` | 05-test-scenarios.md exists |
|
|
177
|
+
| 4 | Generate test cases | \`.github/skills/qa-test-cases/SKILL.md\` | Test plan exists |
|
|
178
|
+
| 5 | Generate automation | \`.github/skills/qa-automation/SKILL.md\` | Specs approved, no PENDING-CODE |
|
|
179
|
+
| 5b | Stabilize failing tests | \`.github/skills/qa-test-stabilization/SKILL.md\` | Failing tests exist |
|
|
180
|
+
| 6 | Maintenance | \`.github/skills/qa-maintenance/SKILL.md\` | Application change delivered |
|
|
181
|
+
| — | ADO integration | \`.github/skills/qa-ado-integration/SKILL.md\` | ADO enabled in config |
|
|
182
|
+
|
|
183
|
+
## Project KB
|
|
184
|
+
1. \`qa/README.md\` — The Living Index: module status, sprint history, blockers, quick-start commands.
|
|
185
|
+
2. \`qa/memory/INDEX.md\` — Index of all memory files; load this before selecting which files to read.
|
|
186
|
+
`;
|
|
187
|
+
}
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# AGENT-NEXT-STEPS — {PROJECT_NAME}
|
|
2
|
+
|
|
3
|
+
> **Sprint activo: Sprint {N} — {SPRINT_NAME}**
|
|
4
|
+
> Lee este archivo primero. Lee los archivos referenciados solo cuando la tarea lo requiera.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Estado del módulo
|
|
9
|
+
|
|
10
|
+
| Módulo / Submódulo | Specs | Plan | Auto | Estado |
|
|
11
|
+
|---|---|---|---|---|
|
|
12
|
+
| {MODULE_NAME} / {SUBMODULE_NAME} | ✅ | ✅ | ⬜ | Sprint {N} activo |
|
|
13
|
+
|
|
14
|
+
> Actualizar esta tabla cuando un sprint se complete. Historial de sprints completados → `qa/README.md ## Sprint History`.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Sprint {N} — Tareas pendientes
|
|
19
|
+
|
|
20
|
+
- [ ] {Tarea 1}
|
|
21
|
+
- [ ] {Tarea 2}
|
|
22
|
+
- [ ] {Tarea 3}
|
|
23
|
+
- [ ] Actualizar `qa/README.md` (fila del módulo + Sprint History)
|
|
24
|
+
- [ ] Trim este archivo: dejar solo el sprint siguiente
|
|
25
|
+
|
|
26
|
+
**Precondiciones**: {Dependencias de datos o módulos previos, si aplica}
|
|
27
|
+
|
|
28
|
+
**Notas específicas**: {Quirks del DOM, rutas, selectores problemáticos conocidos — o eliminar esta línea}
|
|
29
|
+
→ Si las notas son extensas, guardarlas en `qa/memory/{sprint-name}-discovery.md` y referenciar aquí.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Contexto obligatorio antes de codificar
|
|
34
|
+
|
|
35
|
+
- Patrones del proyecto: `qa/memory/INDEX.md` → cargar solo los archivos relevantes
|
|
36
|
+
- Referencia de estilo: `qa/07-automation/e2e/tests/{module}/` (leer suite existente más reciente)
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## Sprints siguientes (solo referencia — sin checklists)
|
|
41
|
+
|
|
42
|
+
- **Sprint {N+1}**: {Descripción breve — qué módulo/submódulo}
|
|
43
|
+
- **Sprint {N+2}**: {Descripción breve}
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
<!-- ═══════════════════════════════════════════════════════════════════
|
|
48
|
+
REGLAS DE MANTENIMIENTO DE ESTE ARCHIVO (para el agente)
|
|
49
|
+
——————————————————————————————————————————————————————————————————
|
|
50
|
+
1. MÁXIMO UN SPRINT ACTIVO. Cuando un sprint se completa:
|
|
51
|
+
a. Mover su checklist al README.md bajo ## Sprint History
|
|
52
|
+
b. Borrar esa sección de este archivo
|
|
53
|
+
c. Promover el siguiente sprint a "activo"
|
|
54
|
+
2. NO repetir instrucciones permanentes que ya estén en copilot-instructions.md.
|
|
55
|
+
3. Los detalles técnicos (DOM, patrones) van en qa/memory/, no aquí.
|
|
56
|
+
4. Los sprints "siguientes" son bullets de 1 línea, sin checklists.
|
|
57
|
+
Este archivo es una cola de trabajo. No es un log ni una bitácora.
|
|
58
|
+
══════════════════════════════════════════════════════════════════ -->
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# QA — {PROJECT_DISPLAY_NAME}
|
|
2
|
+
|
|
3
|
+
> Framework: `@keber/qa-framework` v{VERSION}
|
|
4
|
+
> Language: {LANGUAGE} | Base URL: `{QA_BASE_URL}`
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Module Status
|
|
9
|
+
|
|
10
|
+
| Module | Submodule | TCs Total | TCs Automated | Plan | ADO Suite | Status | Last Run |
|
|
11
|
+
|---|---|---|---|---|---|---|---|
|
|
12
|
+
| {MODULE_NAME} | {SUBMODULE_NAME} | — | — | ⬜ | — | 🔲 Not started | — |
|
|
13
|
+
|
|
14
|
+
**Legend**: ✅ Done · ⚠️ Partial · 🔲 Not started · ⛔ Blocked
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Quick-Start Commands
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
# Run all automated tests
|
|
22
|
+
cd qa/07-automation
|
|
23
|
+
npx playwright test
|
|
24
|
+
|
|
25
|
+
# Run a specific module
|
|
26
|
+
npx playwright test tests/{module-folder}/
|
|
27
|
+
|
|
28
|
+
# Run with UI (debug mode)
|
|
29
|
+
npx playwright test --ui
|
|
30
|
+
|
|
31
|
+
# Generate HTML report
|
|
32
|
+
npx playwright show-report
|
|
33
|
+
|
|
34
|
+
# Re-run only failed tests
|
|
35
|
+
npx playwright test --last-failed
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
> Set credentials in `qa/07-automation/.env` (copy from `.env.example`).
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
42
|
+
## Active Blockers
|
|
43
|
+
|
|
44
|
+
| ID | Description | Affects | Opened | Status |
|
|
45
|
+
|---|---|---|---|---|
|
|
46
|
+
| — | No active blockers | — | — | — |
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
50
|
+
## Last Execution
|
|
51
|
+
|
|
52
|
+
| Date | Suite | Pass | Skip | Fail | Duration | Report |
|
|
53
|
+
|---|---|---|---|---|---|---|
|
|
54
|
+
| — | — | — | — | — | — | — |
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## Sprint History
|
|
59
|
+
|
|
60
|
+
<!-- Completed sprint checklists are moved here from AGENT-NEXT-STEPS.md -->
|
|
61
|
+
<!-- Format per sprint: -->
|
|
62
|
+
<!--
|
|
63
|
+
### Sprint N — {SPRINT_NAME} ({YYYY-MM-DD})
|
|
64
|
+
|
|
65
|
+
**Submodules**: {list}
|
|
66
|
+
**TCs automated**: {N} P0 + {M} P1
|
|
67
|
+
**Execution report**: `qa/05-test-execution/automated/{date-slug}.md`
|
|
68
|
+
|
|
69
|
+
Checklist (completed):
|
|
70
|
+
- [x] {Task 1}
|
|
71
|
+
- [x] {Task 2}
|
|
72
|
+
-->
|
|
73
|
+
|
|
74
|
+
_No sprints completed yet._
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Resources
|
|
79
|
+
|
|
80
|
+
| Resource | Link |
|
|
81
|
+
|---|---|
|
|
82
|
+
| QA Structure Guide | `qa/QA-STRUCTURE-GUIDE.md` |
|
|
83
|
+
| Naming Conventions | `qa/00-standards/naming-conventions.md` |
|
|
84
|
+
| Memory Index | `qa/memory/INDEX.md` |
|
|
85
|
+
| Agent Next Steps | `qa/AGENT-NEXT-STEPS.md` |
|
|
86
|
+
| Automation Config | `qa/07-automation/playwright.config.ts` |
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# Test Data Guidelines
|
|
2
|
+
|
|
3
|
+
> Reference: `qa/04-test-data/`
|
|
4
|
+
> All test data definitions must follow these rules. No exceptions for credentials.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Rules
|
|
9
|
+
|
|
10
|
+
### 1. Never hardcode credentials
|
|
11
|
+
|
|
12
|
+
All user emails, passwords, API keys, and tokens must come from environment variables.
|
|
13
|
+
|
|
14
|
+
```typescript
|
|
15
|
+
// ✅ CORRECT
|
|
16
|
+
const email = process.env.QA_USER_EMAIL!;
|
|
17
|
+
const password = process.env.QA_USER_PASSWORD!;
|
|
18
|
+
|
|
19
|
+
// ❌ WRONG — never commit credentials
|
|
20
|
+
const email = 'admin@example.com';
|
|
21
|
+
const password = 'secret123';
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Document env var names in `qa/04-test-data/users.md`. Use `<PLACEHOLDER>` for values in all documentation.
|
|
25
|
+
|
|
26
|
+
### 2. Use EXEC_IDX for unique values per run
|
|
27
|
+
|
|
28
|
+
Never create records with static names — they collide between runs and can leave leftover data.
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
const EXEC_IDX = Math.floor(Date.now() / 60_000) % 100_000;
|
|
32
|
+
|
|
33
|
+
// Names / codes
|
|
34
|
+
const name = `Test-Entity-${EXEC_IDX}`;
|
|
35
|
+
|
|
36
|
+
// Future-safe dates (year 2120+ avoids real production data overlap)
|
|
37
|
+
const year = 2120 + (EXEC_IDX % 10);
|
|
38
|
+
const month = String((EXEC_IDX % 12) + 1).padStart(2, '0');
|
|
39
|
+
const day = String((EXEC_IDX % 28) + 1).padStart(2, '0');
|
|
40
|
+
const date = `${year}-${month}-${day}`;
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
### 3. Provision in `beforeAll`, not `beforeEach`
|
|
44
|
+
|
|
45
|
+
Create required records once per suite, not once per test. Each test that **consumes** a record needs its own provisioned record — never share a single record among multiple tests.
|
|
46
|
+
|
|
47
|
+
```typescript
|
|
48
|
+
test.beforeAll(async ({ browser }) => {
|
|
49
|
+
const page = await browser.newPage();
|
|
50
|
+
// If 3 tests each need 1 record, create 3:
|
|
51
|
+
for (let i = 0; i < 3; i++) {
|
|
52
|
+
await createEntity(page, `Record-${EXEC_IDX}-${i}`);
|
|
53
|
+
}
|
|
54
|
+
await page.close();
|
|
55
|
+
});
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### 4. Seed scripts go in `04-test-data/seeders/`
|
|
59
|
+
|
|
60
|
+
Complex setup that cannot be done via the UI goes in a seeder script, not inside a spec file.
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
qa/04-test-data/
|
|
64
|
+
├── users.md ← Role catalog + env var names (no real values)
|
|
65
|
+
├── fixtures/ ← Static JSON/Markdown for reference data
|
|
66
|
+
├── factories/ ← Dynamic data generation patterns
|
|
67
|
+
└── seeders/ ← DB/API seed scripts for complex prerequisites
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### 5. Add `.env` and `.auth/` to `.gitignore`
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
qa/07-automation/.env
|
|
74
|
+
qa/07-automation/.auth/
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
The `.env.example` file (committed) lists all required keys with `<PLACEHOLDER>` as the value.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## User Table Format (`04-test-data/users.md`)
|
|
82
|
+
|
|
83
|
+
```markdown
|
|
84
|
+
| Role | Env var (email) | Env var (password) | Permissions |
|
|
85
|
+
|---|---|---|---|
|
|
86
|
+
| Admin | QA_ADMIN_EMAIL | QA_ADMIN_PASSWORD | Full access |
|
|
87
|
+
| Read-only | QA_READONLY_EMAIL | QA_READONLY_PASSWORD | View only |
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Fixture Files (`04-test-data/fixtures/`)
|
|
93
|
+
|
|
94
|
+
Use Markdown tables or JSON for static reference data that tests depend on (catalogs, option lists, etc.).
|
|
95
|
+
|
|
96
|
+
```markdown
|
|
97
|
+
# fixtures/catalogs.md
|
|
98
|
+
## Labor Types
|
|
99
|
+
| Code | Name |
|
|
100
|
+
|---|---|
|
|
101
|
+
| LT-01 | Consulting |
|
|
102
|
+
| LT-02 | Development |
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Reference fixtures in specs via `import` (JSON) or inline expectations matching the fixture values.
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Factory Pattern (`04-test-data/factories/`)
|
|
110
|
+
|
|
111
|
+
Document how to generate complex data shapes needed by multiple tests:
|
|
112
|
+
|
|
113
|
+
```typescript
|
|
114
|
+
// factories/supplier.ts
|
|
115
|
+
export function buildSupplier(idx: number) {
|
|
116
|
+
return {
|
|
117
|
+
name: `Supplier-${idx}`,
|
|
118
|
+
rut: `${76_000_000 + idx}-0`,
|
|
119
|
+
email: `supplier-${idx}@test.example`,
|
|
120
|
+
};
|
|
121
|
+
}
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## On test teardown
|
|
127
|
+
|
|
128
|
+
Do **not** delete test data after each run unless the test explicitly requires a clean state. Leftover data from old EXEC_IDX values does not affect subsequent runs because EXEC_IDX generates distinct names.
|
|
129
|
+
|
|
130
|
+
Exception: if the system enforces uniqueness on a field that EXEC_IDX cannot make unique (e.g., a singleton config), document the teardown strategy in `04-test-data/seeders/`.
|