@ai-qa/workflow 2.0.11 → 2.0.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/agents/playwright-test-generator.agent.md +49 -0
- package/.github/agents/playwright-test-healer.agent.md +32 -3
- package/.github/agents/playwright-test-planner.agent.md +26 -0
- package/.github/copilot-instructions.md +44 -2
- package/.opencode/agents/qa-generator.md +16 -0
- package/.opencode/agents/qa-healer.md +18 -0
- package/.opencode/agents/qa-planner.md +17 -0
- package/.opencode/rules.md +66 -2
- package/.qa-context/auth.json +29 -0
- package/.qa-context/heal-history.json +40 -0
- package/.qa-context/pipeline.json +34 -0
- package/.qa-context/selectors.json +64 -0
- package/.qa-context/traceability.json +30 -0
- package/README.md +399 -196
- package/ai-qa-workflow.js +82 -104
- package/install.js +7 -12
- package/package.json +5 -6
- package/prompting_template.md +283 -0
- package/qa-dashboard/app.js +1 -0
- package/qa-dashboard/routes/review.js +114 -0
- package/qa-dashboard/views/layouts/main.ejs +1 -0
- package/qa-dashboard/views/review.ejs +201 -0
- package/router.md +109 -29
- package/scripts/auth-manager.js +186 -0
- package/scripts/context-manager.js +226 -0
- package/scripts/executor.js +18 -7
- package/scripts/generator.js +18 -124
- package/scripts/healer.js +78 -157
- package/scripts/planner.js +18 -136
- package/scripts/reporter.js +21 -1
- package/scripts/utils.js +2 -0
package/README.md
CHANGED
|
@@ -1,8 +1,41 @@
|
|
|
1
|
-
# AI QA
|
|
1
|
+
# AI QA Workflow Template
|
|
2
2
|
|
|
3
|
-
**
|
|
3
|
+
**Turn any AI agent into an autonomous QA engineer.**
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
This template gives an AI coding assistant (opencode, Copilot, Claude Code, Cursor, etc.) everything it needs to: explore your web app, plan tests, generate real Playwright code, execute them, debug failures, and visualize results.
|
|
6
|
+
|
|
7
|
+
The template handles the **mechanical** parts (execution, reporting, dashboard). The AI handles the **creative** parts (planning, test generation, debugging, healing).
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## How It Works
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
┌──────────────────────────────────────┐
|
|
16
|
+
│ AI AGENT (does thinking) │
|
|
17
|
+
│ │
|
|
18
|
+
│ Phase 1: Environment check │
|
|
19
|
+
│ Phase 2: Explore app + write plan │
|
|
20
|
+
│ Phase 3: Generate Playwright tests │
|
|
21
|
+
│ Phase 4: Execute tests │
|
|
22
|
+
│ Phase 5: Debug + heal failures │
|
|
23
|
+
│ Phase 6: Report + update context │
|
|
24
|
+
└───────┬────────────────────┬────────┘
|
|
25
|
+
│ │
|
|
26
|
+
┌─────────────┘ └─────────────┐
|
|
27
|
+
▼ ▼
|
|
28
|
+
┌─────────────────────────┐ ┌─────────────────────────┐
|
|
29
|
+
│ HUMAN SUPERVISION │ │ SCRIPTS (mechanics) │
|
|
30
|
+
│ │ │ │
|
|
31
|
+
│ At EVERY phase: │ │ npm run qa:execute │
|
|
32
|
+
│ AI proposes ──▶ you │ │ npm run qa:report │
|
|
33
|
+
│ approve ──▶ AI acts │ │ npm run qa:retry │
|
|
34
|
+
│ │ │ npm run qa:status │
|
|
35
|
+
│ You are always in │ │ npm run qa:list │
|
|
36
|
+
│ control. │ │ npm run dashboard │
|
|
37
|
+
└─────────────────────────┘ └─────────────────────────┘
|
|
38
|
+
```
|
|
6
39
|
|
|
7
40
|
---
|
|
8
41
|
|
|
@@ -12,13 +45,13 @@ One-command install into any web project. Works standalone or with any AI assist
|
|
|
12
45
|
npx @ai-qa/workflow init --yes
|
|
13
46
|
```
|
|
14
47
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
-
|
|
48
|
+
Installs the full template into your project:
|
|
49
|
+
- Playwright MCP + GitHub MCP configuration (`opencode.json`)
|
|
50
|
+
- AI agent definitions (`.github/agents/`)
|
|
51
|
+
- Workflow prompts (`prompts/`)
|
|
52
|
+
- CLI scripts for execution and reporting
|
|
53
|
+
- QA Dashboard (web UI)
|
|
54
|
+
- Directory structure (`user-story/`, `specs/`, `tests/`, `test-results/`, `docs/`, `.qa-context/`)
|
|
22
55
|
|
|
23
56
|
### Update an existing installation
|
|
24
57
|
|
|
@@ -26,68 +59,276 @@ That's it. This single command installs the full pipeline into your current proj
|
|
|
26
59
|
npx @ai-qa/workflow update --yes
|
|
27
60
|
```
|
|
28
61
|
|
|
29
|
-
Updates
|
|
62
|
+
Updates template files while preserving your user stories, test specs, config, and run results.
|
|
63
|
+
|
|
64
|
+
---
|
|
30
65
|
|
|
31
|
-
|
|
66
|
+
## What You Get
|
|
32
67
|
|
|
33
68
|
```
|
|
34
69
|
your-project/
|
|
35
|
-
├── ai-qa-workflow.js # CLI orchestrator (
|
|
36
|
-
├── .qa-workflow.json # Project config (auto-detected
|
|
37
|
-
├── scripts/
|
|
70
|
+
├── ai-qa-workflow.js # CLI orchestrator (8 commands)
|
|
71
|
+
├── .qa-workflow.json # Project config (auto-detected)
|
|
72
|
+
├── scripts/
|
|
73
|
+
│ ├── utils.js # Config loading, shared helpers
|
|
74
|
+
│ ├── executor.js # Runs Playwright tests
|
|
75
|
+
│ ├── retrier.js # Re-runs failed tests (longer timeout)
|
|
76
|
+
│ └── reporter.js # Markdown + JSON reports
|
|
38
77
|
├── opencode.json # MCP config (Playwright + GitHub)
|
|
39
78
|
├── .github/agents/ # AI agent definitions
|
|
40
|
-
├──
|
|
79
|
+
│ ├── playwright-test-planner.agent.md
|
|
80
|
+
│ ├── playwright-test-generator.agent.md
|
|
81
|
+
│ └── playwright-test-healer.agent.md
|
|
82
|
+
├── prompts/
|
|
83
|
+
│ ├── QAe2eprompt.md # Full 9-step AI workflow
|
|
84
|
+
│ ├── general_prompt.md # Quick-start prompt
|
|
85
|
+
├── prompting_template.md # Conversation guide — what to say to the AI at each phase
|
|
86
|
+
├── router.md # AI entry point — routes to correct agent
|
|
41
87
|
├── qa-dashboard/ # Web UI (port 4000)
|
|
42
|
-
├──
|
|
43
|
-
├──
|
|
44
|
-
├──
|
|
45
|
-
|
|
46
|
-
|
|
88
|
+
├── .qa-context/ # Persistent AI memory (pipeline state, selectors, healing history, traceability)
|
|
89
|
+
├── .auth/ # Auth credentials + Playwright storage state (gitignored)
|
|
90
|
+
├── user-story/ # Your .md stories
|
|
91
|
+
├── specs/ # AI-generated test plans
|
|
92
|
+
├── tests/ # AI-generated Playwright specs
|
|
93
|
+
├── test-results/ # Run results, reports, screenshots
|
|
94
|
+
└── docs/ # Application context (AI knowledge base)
|
|
47
95
|
```
|
|
48
96
|
|
|
49
|
-
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## Quick Start
|
|
50
100
|
|
|
51
101
|
```bash
|
|
52
|
-
# 1.
|
|
102
|
+
# 1. Install the template
|
|
103
|
+
npx @ai-qa/workflow init --yes
|
|
104
|
+
|
|
105
|
+
# 2. Initialize project config
|
|
53
106
|
npm run qa:init
|
|
54
107
|
|
|
55
|
-
#
|
|
108
|
+
# 3. Write a user story in user-story/my-feature.md (see format below)
|
|
109
|
+
|
|
110
|
+
# 4. Open the project in your AI editor (opencode, Copilot, Cursor)
|
|
111
|
+
# The AI agent auto-detects its role and runs an environment check.
|
|
112
|
+
# It will report what's ready and what's missing.
|
|
113
|
+
|
|
114
|
+
# 5. Send this one prompt to start:
|
|
115
|
+
# "Read router.md and follow the QA workflow for my-feature.md"
|
|
116
|
+
|
|
117
|
+
# The AI will:
|
|
118
|
+
# - Read router.md → understand its mission
|
|
119
|
+
# - Read playwright-test-planner.agent.md
|
|
120
|
+
# - Run environment check (if not already done)
|
|
121
|
+
# - Explore your app with Playwright MCP
|
|
122
|
+
# - Save a test plan to specs/
|
|
123
|
+
# - STOP and ask for your approval
|
|
124
|
+
# - Once approved → generate tests → STOP again
|
|
125
|
+
# - Tell you when ready to execute
|
|
56
126
|
|
|
57
|
-
#
|
|
58
|
-
npm run qa:
|
|
127
|
+
# 6. Execute tests
|
|
128
|
+
npm run qa:execute
|
|
129
|
+
|
|
130
|
+
# 7. If tests fail, tell your AI agent:
|
|
131
|
+
# "Debug and fix the failing tests"
|
|
59
132
|
|
|
60
|
-
#
|
|
133
|
+
# 8. Launch dashboard
|
|
61
134
|
npm run dashboard
|
|
62
135
|
```
|
|
63
136
|
|
|
137
|
+
> **💡 Full conversation guide:** See `prompting_template.md` for a complete script of what to say to the AI at every phase — from installation to final report.
|
|
138
|
+
|
|
64
139
|
---
|
|
65
140
|
|
|
66
|
-
##
|
|
141
|
+
## AI Agent Auto-Bootstrap
|
|
67
142
|
|
|
68
|
-
|
|
143
|
+
When you open this project in an AI-powered editor, the agent **automatically** understands its purpose:
|
|
69
144
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
3. Visually explores your application in real-time
|
|
75
|
-
4. Generates a flawless test plan (`specs/`) and executable code (`tests/`) using **real** semantic selectors. No more hallucinated CSS classes!
|
|
145
|
+
1. **Reads `.opencode/rules.md`** (or `.github/copilot-instructions.md`) → discovers it's an **AI QA Engineer**
|
|
146
|
+
2. **Runs an environment check** → reports what's ready ✅ and what's missing ❌
|
|
147
|
+
3. **Reads `router.md`** → learns the workflow and supervision rules
|
|
148
|
+
4. **Stops and waits** for you to provide a user story
|
|
76
149
|
|
|
77
|
-
|
|
78
|
-
Once the test scripts are generated by the Agent, you move to execution:
|
|
79
|
-
1. Open the Web Dashboard (`npm run dashboard`)
|
|
80
|
-
2. Click **Execute** or trigger it via your CI/CD pipeline
|
|
81
|
-
3. The framework handles Playwright execution, auto-healing (fixing minor flakiness), and report generation
|
|
82
|
-
4. View deep analytics, execution history, and Allure reports directly in the Dashboard UI.
|
|
150
|
+
No manual instructions needed. The agent knows its role on first contact.
|
|
83
151
|
|
|
84
152
|
---
|
|
85
153
|
|
|
86
|
-
##
|
|
154
|
+
## Your First Prompt
|
|
155
|
+
|
|
156
|
+
After installation and running `npm run qa:init`, open the project in your AI editor.
|
|
157
|
+
|
|
158
|
+
The very first thing you should say to the AI agent:
|
|
159
|
+
|
|
160
|
+
> **"Read router.md and follow the QA workflow for my-story.md"**
|
|
161
|
+
|
|
162
|
+
(Replace `my-story.md` with the name of your user story file in `user-story/`.)
|
|
163
|
+
|
|
164
|
+
> **📖 Need more prompts?** See `prompting_template.md` for the full conversation script — approval responses, healing prompts, report prompts, and an example session.
|
|
165
|
+
|
|
166
|
+
### What happens when you send this prompt:
|
|
167
|
+
|
|
168
|
+
| Step | AI does this | You do this |
|
|
169
|
+
|------|-------------|-------------|
|
|
170
|
+
| 1 | Runs environment check (if first time) | Read the status report |
|
|
171
|
+
| 2 | Reads `router.md` → `playwright-test-planner.agent.md` | — |
|
|
172
|
+
| 3 | Explores your app with Playwright MCP | — |
|
|
173
|
+
| 4 | Writes test plan to `specs/` | **Review and approve** |
|
|
174
|
+
| 5 | Reads `playwright-test-generator.agent.md` | — |
|
|
175
|
+
| 6 | Generates test files to `tests/` | **Review and approve** |
|
|
176
|
+
| 7 | Tells you tests are ready to run | Run `npm run qa:execute` |
|
|
177
|
+
| 8 | If tests fail, debugs and proposes fix | **Review and approve fix** |
|
|
178
|
+
|
|
179
|
+
**You are the supervisor.** The AI never moves to the next phase without your approval.
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
## For Humans: Understanding the Workflow
|
|
184
|
+
|
|
185
|
+
The template follows a **9-step AI workflow** defined in `prompts/QAe2eprompt.md`. Here's what happens at each step:
|
|
186
|
+
|
|
187
|
+
### Step 1 — Context Discovery
|
|
188
|
+
The AI reads `docs/application-context.md` and explores your project to understand the tech stack, authentication, and environment.
|
|
189
|
+
|
|
190
|
+
### Step 2 — Test Strategy & Plan
|
|
191
|
+
The AI reads `playwright-test-planner.agent.md` and uses Playwright MCP to explore your app visually. It maps user flows, identifies critical paths, and designs comprehensive test scenarios. The plan is saved to `specs/`.
|
|
192
|
+
|
|
193
|
+
> **⛔ Approval gate:** The AI **stops** after saving the plan and presents it to you. You review, give feedback, and say "approved" or "continue" before the AI proceeds to test generation.
|
|
194
|
+
|
|
195
|
+
### Step 3 — Manual Exploratory Testing
|
|
196
|
+
The AI executes each scenario manually using Playwright MCP, capturing real selectors, observing behavior, and noting issues.
|
|
197
|
+
|
|
198
|
+
### Step 4 — Test Code Generation
|
|
199
|
+
The AI reads `playwright-test-generator.agent.md` and writes real Playwright `.spec.ts` files using selectors it discovered during exploration. Generated files go in `tests/`.
|
|
200
|
+
|
|
201
|
+
> **⛔ Approval gate:** The AI **stops** after generating test files and presents them to you. You review the code, check selectors and logic, then say "approved" or "execute" before the AI proceeds.
|
|
202
|
+
|
|
203
|
+
### Step 5 — Execution
|
|
204
|
+
Run the tests mechanically:
|
|
205
|
+
|
|
206
|
+
```bash
|
|
207
|
+
npm run qa:execute [test-name]
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Or run all tests:
|
|
211
|
+
```bash
|
|
212
|
+
npm run qa:execute
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
The executor runs Playwright with the configured options and saves results to `test-results/`.
|
|
216
|
+
|
|
217
|
+
### Step 6 — Debug & Heal (AI)
|
|
218
|
+
If tests fail, the AI reads `playwright-test-healer.agent.md` and:
|
|
219
|
+
1. Runs the failing test with `test_run`
|
|
220
|
+
2. Debugs with `test_debug` — examines the actual UI state
|
|
221
|
+
3. Classifies the failure and proposes a fix (1-3 line change)
|
|
222
|
+
|
|
223
|
+
> **⛔ Approval gate:** The AI **stops** and presents its diagnosis to you. It shows what's broken, why, and what it wants to change. You approve the fix before the AI edits any file.
|
|
224
|
+
|
|
225
|
+
4. Once approved, applies fix and re-runs once
|
|
226
|
+
5. If still failing, marks as `test.fixme()` and reports as defect
|
|
227
|
+
|
|
228
|
+
For a quick mechanical re-run (no AI diagnosis):
|
|
229
|
+
|
|
230
|
+
```bash
|
|
231
|
+
npm run qa:retry [run-id]
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
This re-runs failed tests with a longer timeout. If they still fail, the AI investigates.
|
|
235
|
+
|
|
236
|
+
### Step 7 — Bug Classification (AI)
|
|
237
|
+
The AI classifies every defect by severity, priority, type, root cause, and reproducibility. Results are saved to `test-results/defects-log.md`.
|
|
238
|
+
|
|
239
|
+
### Step 8 — Report
|
|
240
|
+
```bash
|
|
241
|
+
npm run qa:report [run-id]
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Generates a markdown report from execution results.
|
|
245
|
+
|
|
246
|
+
### Step 9 — Knowledge Retention
|
|
247
|
+
The AI updates `docs/application-context.md` with:
|
|
248
|
+
- Stable selectors discovered
|
|
249
|
+
- Known flaky areas
|
|
250
|
+
- Healing strategies that worked
|
|
251
|
+
- Environment observations
|
|
252
|
+
|
|
253
|
+
---
|
|
254
|
+
|
|
255
|
+
## Installation
|
|
256
|
+
|
|
257
|
+
### Requirements
|
|
258
|
+
|
|
259
|
+
| Component | Needed for | Install |
|
|
260
|
+
|-----------|-----------|---------|
|
|
261
|
+
| **Node.js 18+** | Running the pipeline | — |
|
|
262
|
+
| **Playwright** | Test execution | `npm install @playwright/test` |
|
|
263
|
+
| **Chromium** | Running tests | `npx playwright install chromium` |
|
|
264
|
+
| **Playwright MCP** | AI browser automation | `npm install -D @playwright/mcp` |
|
|
265
|
+
| **Applitools MCP** | Visual testing (screenshot comparison) | `npm install -D @applitools/mcp` + `APPLITOOLS_API_KEY` |
|
|
266
|
+
| **GitHub MCP** | AI creating PRs/issues | `npm install -D @modelcontextprotocol/server-github` + `GITHUB_TOKEN` |
|
|
267
|
+
|
|
268
|
+
### Install into a project
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
npx @ai-qa/workflow init --yes
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Or from local template:
|
|
275
|
+
```bash
|
|
276
|
+
node install.js ../my-project --yes
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
### Update
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
npx @ai-qa/workflow update --yes
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
---
|
|
286
|
+
|
|
287
|
+
## Visual Testing (Applitools)
|
|
288
|
+
|
|
289
|
+
The template supports **Applitools MCP** for automated visual testing.
|
|
290
|
+
|
|
291
|
+
If `APPLITOOLS_API_KEY` is configured in your environment, the AI agent automatically adds visual checkpoints to critical pages during test generation. It captures screenshots of pages like login, dashboard, and checkout, and compares them against baselines to detect visual regressions.
|
|
292
|
+
|
|
293
|
+
### Setup
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
# 1. Install Applitools MCP
|
|
297
|
+
npm install -D @applitools/mcp
|
|
298
|
+
|
|
299
|
+
# 2. Set your API key (get it from https://applitools.com)
|
|
300
|
+
# Option A: Export in terminal
|
|
301
|
+
export APPLITOOLS_API_KEY=votre_clé_ici
|
|
302
|
+
|
|
303
|
+
# Option B: Add to .env file
|
|
304
|
+
echo "APPLITOOLS_API_KEY=votre_clé_ici" >> .env
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
The AI will detect the key during its environment check and use Applitools automatically. If the key is not set, visual testing is skipped entirely — no errors, no blocks.
|
|
308
|
+
|
|
309
|
+
---
|
|
310
|
+
|
|
311
|
+
## Commands
|
|
312
|
+
|
|
313
|
+
| Command | Description | Who does it |
|
|
314
|
+
|---------|-------------|-------------|
|
|
315
|
+
| `npm run qa:init` | Create directories + auto-detect config | You (once) |
|
|
316
|
+
| `npm run qa:execute [test]` | Run Playwright tests | You or CI |
|
|
317
|
+
| `npm run qa:retry [run-id]` | Re-run failed tests (longer timeout) | You or CI |
|
|
318
|
+
| `npm run qa:report [run-id]` | Generate markdown report | You or CI |
|
|
319
|
+
| `npm run qa:status` | Show pipeline state | You |
|
|
320
|
+
| `npm run qa:list` | List stories, plans, specs | You |
|
|
321
|
+
| `node ai-qa-workflow.js context <phase> <story>` | Mark a pipeline phase complete for a story | AI agent or you |
|
|
322
|
+
| `npm run dashboard` | Launch web UI at :4000 | You |
|
|
87
323
|
|
|
88
|
-
The
|
|
324
|
+
> The commands `qa:plan`, `qa:generate`, and `qa:run` do not exist in this template.
|
|
325
|
+
> Planning, test generation, and healing are done by the AI agent, not by scripts.
|
|
89
326
|
|
|
90
|
-
|
|
327
|
+
---
|
|
328
|
+
|
|
329
|
+
## Zero-Config Design
|
|
330
|
+
|
|
331
|
+
On `npm run qa:init`, the template auto-detects your project configuration:
|
|
91
332
|
|
|
92
333
|
| Scan source | What it detects |
|
|
93
334
|
|-------------|----------------|
|
|
@@ -98,15 +339,12 @@ On `npm run qa:init`, it scans your project and generates `.qa-workflow.json`:
|
|
|
98
339
|
| `playwright.config.*` | Browser type (edge, webkit) |
|
|
99
340
|
| Directory structure | Framework (Angular, Next.js, Python) |
|
|
100
341
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
### The Config File (`.qa-workflow.json`)
|
|
342
|
+
Generated `.qa-workflow.json`:
|
|
104
343
|
|
|
105
344
|
```json
|
|
106
345
|
{
|
|
107
346
|
"project": {
|
|
108
347
|
"name": "my-app",
|
|
109
|
-
"description": "My web application",
|
|
110
348
|
"url": "http://localhost:5173",
|
|
111
349
|
"environment": "Development"
|
|
112
350
|
},
|
|
@@ -127,100 +365,121 @@ If a file is missing, the template falls back to sensible defaults. No manual se
|
|
|
127
365
|
}
|
|
128
366
|
```
|
|
129
367
|
|
|
130
|
-
Edit this file to override auto-detected values.
|
|
368
|
+
Edit this file to override auto-detected values.
|
|
131
369
|
|
|
132
|
-
|
|
370
|
+
---
|
|
133
371
|
|
|
134
|
-
|
|
372
|
+
## Application Context (`docs/application-context.md`)
|
|
373
|
+
|
|
374
|
+
This file serves as the **AI's knowledge base**. On `init` it's auto-generated. The AI populates it during exploration and testing with:
|
|
135
375
|
|
|
136
376
|
- **Stable selectors** — CSS/XPath selectors discovered during exploration
|
|
137
377
|
- **Known flaky areas** — elements or flows that frequently break
|
|
138
378
|
- **Auth details** — users, tokens, environments
|
|
139
379
|
- **Tech stack notes** — framework specifics the AI should know
|
|
140
380
|
|
|
141
|
-
For best results, edit this file with your project's specifics before
|
|
381
|
+
For best results, edit this file with your project's specifics before the AI starts.
|
|
142
382
|
|
|
143
383
|
---
|
|
144
384
|
|
|
145
|
-
##
|
|
385
|
+
## Persistent AI Memory (`.qa-context/`)
|
|
146
386
|
|
|
147
|
-
|
|
387
|
+
The `.qa-context/` directory stores **structured memory** that persists between AI sessions and pipeline phases:
|
|
148
388
|
|
|
149
|
-
|
|
|
150
|
-
|
|
151
|
-
|
|
|
152
|
-
|
|
|
153
|
-
|
|
|
154
|
-
|
|
|
155
|
-
| **GitHub MCP** | AI creating PRs/issues from results | `npm install -D @modelcontextprotocol/server-github` + `GITHUB_TOKEN` env |
|
|
389
|
+
| File | Content | Used by |
|
|
390
|
+
|------|---------|---------|
|
|
391
|
+
| `pipeline.json` | Current pipeline state: which phases are complete, current story, last run | All agents (read on start, write on complete) |
|
|
392
|
+
| `selectors.json` | All discovered selectors with reliability scores, healing history, recommended alternatives | Generator (write best selectors), Healer (read alternatives) |
|
|
393
|
+
| `heal-history.json` | Every healing attempt: what was tried, which strategy, whether it succeeded | Healer (avoid repeating failed attempts), Planner (avoid flaky pages) |
|
|
394
|
+
| `traceability.json` | Full mapping: story → plan → spec → runs → healing | Reporter (link report to story), Dashboard (display traceability) |
|
|
156
395
|
|
|
157
|
-
|
|
396
|
+
Each agent reads `.qa-context/` **before starting** and updates it **after completing**. This means:
|
|
397
|
+
- The Generator reuses stable selectors discovered by the Planner
|
|
398
|
+
- The Healer knows which selectors were tried and failed before
|
|
399
|
+
- The Reporter links every run back to its original story
|
|
400
|
+
- The Dashboard can display the full audit trail
|
|
158
401
|
|
|
159
|
-
|
|
160
|
-
# From npm (recommended — works anywhere)
|
|
161
|
-
npx @ai-qa/workflow init --yes
|
|
402
|
+
The AI automatically consults these files — no manual setup needed.
|
|
162
403
|
|
|
163
|
-
|
|
164
|
-
node install.js ../my-project --yes
|
|
165
|
-
```
|
|
404
|
+
---
|
|
166
405
|
|
|
167
|
-
|
|
406
|
+
## Authentication Management (`.auth/`)
|
|
168
407
|
|
|
169
|
-
|
|
408
|
+
The `.auth/` directory stores everything needed for automated login during tests:
|
|
170
409
|
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
410
|
+
| File | Content | Git |
|
|
411
|
+
|------|---------|:---:|
|
|
412
|
+
| `credentials.json` | Username + password for the app | ❌ ignored |
|
|
413
|
+
| `storage-state.json` | Playwright session (cookies, localStorage, tokens) | ❌ ignored |
|
|
414
|
+
| `.gitignore` | Ensures nothing in `.auth/` is committed | ✅ |
|
|
174
415
|
|
|
175
|
-
|
|
176
|
-
node install.js ../my-project --update --yes
|
|
177
|
-
```
|
|
416
|
+
### How it works
|
|
178
417
|
|
|
179
|
-
|
|
418
|
+
1. **`npm run qa:init`** creates `.auth/` with `.gitignore` — ready to use
|
|
419
|
+
2. **AI discovers login** during Phase 1 (Plan) — navigates to `/login`, detects form fields, saves structure to `.qa-context/auth.json`
|
|
420
|
+
3. **You provide credentials** once — the AI saves them to `.auth/credentials.json`
|
|
421
|
+
4. **Session is persisted** — after first login, Playwright's storage state is saved to `.auth/storage-state.json`
|
|
422
|
+
5. **Tests reuse the session** — generated `auth.setup.ts` loads storage state before each run
|
|
423
|
+
6. **If credentials change**, the AI detects auth failures via `auth-manager.js` and prompts you to update `.auth/credentials.json`
|
|
180
424
|
|
|
181
|
-
###
|
|
425
|
+
### First-time setup
|
|
182
426
|
|
|
183
427
|
```bash
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
npm run qa:execute # Run Playwright tests
|
|
189
|
-
npm run qa:heal # Self-heal failed tests
|
|
190
|
-
npm run qa:report # Generate final report
|
|
191
|
-
npm run qa:status # Check pipeline state
|
|
192
|
-
npm run qa:list # List stories, plans, specs
|
|
193
|
-
npm run dashboard # Launch web dashboard at :4000
|
|
428
|
+
# 1. After qa:init, open the project in your AI editor
|
|
429
|
+
# 2. The AI will ask for credentials during Phase 1 (Plan)
|
|
430
|
+
# 3. Provide them once — they're saved to .auth/credentials.json
|
|
431
|
+
# 4. All future runs reuse the stored session
|
|
194
432
|
```
|
|
195
433
|
|
|
196
|
-
|
|
434
|
+
### Manual setup
|
|
197
435
|
|
|
198
|
-
|
|
436
|
+
```json
|
|
437
|
+
{
|
|
438
|
+
"username": "test@example.com",
|
|
439
|
+
"password": "your-password",
|
|
440
|
+
"url": "http://localhost:3000/login"
|
|
441
|
+
}
|
|
442
|
+
```
|
|
199
443
|
|
|
200
|
-
|
|
444
|
+
Save this as `.auth/credentials.json`. The AI detects it automatically on next run.
|
|
201
445
|
|
|
202
|
-
|
|
446
|
+
---
|
|
203
447
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
448
|
+
## How the AI Agents Work
|
|
449
|
+
|
|
450
|
+
The template defines three specialized AI agents in `.github/agents/`:
|
|
451
|
+
|
|
452
|
+
### Planner Agent
|
|
453
|
+
Triggers when the AI needs to create a test plan. The AI:
|
|
454
|
+
1. Opens your app with Playwright MCP
|
|
455
|
+
2. Explores navigation, flows, UI components
|
|
456
|
+
3. Maps critical paths and edge cases
|
|
457
|
+
4. Saves a structured test plan to `specs/`
|
|
458
|
+
5. **⛔ Stops and presents the plan to you for approval**
|
|
459
|
+
|
|
460
|
+
### Generator Agent
|
|
461
|
+
Triggers when the AI needs to write test code. The AI:
|
|
462
|
+
1. Reads the test plan
|
|
463
|
+
2. Uses Playwright to manually execute each step
|
|
464
|
+
3. Captures real selectors (data-testid, aria-label, role)
|
|
465
|
+
4. Adds visual checkpoints via Applitools on critical pages (if `APPLITOOLS_API_KEY` is set)
|
|
466
|
+
5. Writes complete, executable Playwright tests to `tests/`
|
|
467
|
+
6. **⛔ Stops and presents the code to you for approval**
|
|
468
|
+
|
|
469
|
+
### Healer Agent
|
|
470
|
+
Triggers when tests fail. The AI:
|
|
471
|
+
1. Runs only the failing tests
|
|
472
|
+
2. Debugs with `test_debug` — sees the actual UI state
|
|
473
|
+
3. Classifies the failure (selector, timing, or bug)
|
|
474
|
+
4. **⛔ Stops and presents diagnosis + proposed fix for your approval**
|
|
475
|
+
5. Once approved, applies a targeted 1-3 line fix
|
|
476
|
+
6. Re-runs once — if still failing, marks as `test.fixme()`
|
|
218
477
|
|
|
219
|
-
|
|
478
|
+
---
|
|
220
479
|
|
|
221
|
-
|
|
480
|
+
## Writing Good User Stories
|
|
222
481
|
|
|
223
|
-
|
|
482
|
+
The AI reads your user stories to understand what to test. Write them in `user-story/`:
|
|
224
483
|
|
|
225
484
|
```markdown
|
|
226
485
|
# User Login
|
|
@@ -243,122 +502,66 @@ As a registered user, I want to log in with my credentials so I can access the d
|
|
|
243
502
|
4. User name displayed in header
|
|
244
503
|
```
|
|
245
504
|
|
|
246
|
-
Well-structured stories produce better test plans
|
|
247
|
-
|
|
248
|
-
### 3. Review generated test plans
|
|
249
|
-
|
|
250
|
-
After `npm run qa:plan`, check `specs/*-test-plan.md`. The AI generates scenarios from acceptance criteria. Review and edit before generating test code:
|
|
251
|
-
|
|
252
|
-
- Add edge cases the AI missed
|
|
253
|
-
- Add negative test scenarios
|
|
254
|
-
- Fix any incorrect assumptions about the app
|
|
255
|
-
|
|
256
|
-
### 4. Use the run mode that fits your workflow
|
|
505
|
+
Well-structured stories produce better test plans and better automated tests.
|
|
257
506
|
|
|
258
|
-
|
|
259
|
-
|------|---------|----------|
|
|
260
|
-
| **Full auto** | `npm run qa:run <story>` | Quick smoke tests, CI pipelines |
|
|
261
|
-
| **Step by step** | plan → generate → execute → heal → report | Reviewing each stage, manual edits |
|
|
262
|
-
| **Manual test only** | `npm run qa:execute <test>` | When tests are already written |
|
|
263
|
-
| **Heal only** | `npm run qa:heal [runId]` | After manual test edits |
|
|
264
|
-
|
|
265
|
-
### 5. Understand self-healing
|
|
266
|
-
|
|
267
|
-
The healer makes up to **2 attempts** per failed test:
|
|
268
|
-
|
|
269
|
-
1. **Standard re-run** — flakes due to timing or network
|
|
270
|
-
2. **Longer timeout (60s)** — slow responses
|
|
271
|
-
3. **Stop** — marks as defect, classifies the failure
|
|
272
|
-
|
|
273
|
-
Failure classification:
|
|
274
|
-
| Type | Label | Auto-healed? |
|
|
275
|
-
|------|-------|-------------|
|
|
276
|
-
| Selector not found | `selector` | ✅ Usually |
|
|
277
|
-
| Timeout | `timing` | ✅ Usually |
|
|
278
|
-
| Target closed | `environment` | ❌ Bug |
|
|
279
|
-
| Network error | `environment` | ❌ Bug |
|
|
280
|
-
| Test syntax error | `test-syntax` | ❌ Manual fix |
|
|
507
|
+
---
|
|
281
508
|
|
|
282
|
-
|
|
509
|
+
## Dashboard
|
|
283
510
|
|
|
284
511
|
```bash
|
|
285
512
|
npm run dashboard
|
|
286
513
|
# → http://localhost:4000
|
|
287
514
|
```
|
|
288
515
|
|
|
289
|
-
The dashboard
|
|
516
|
+
The dashboard provides:
|
|
517
|
+
- Multi-project management
|
|
518
|
+
- Execution history with pass/fail/healed stats
|
|
519
|
+
- Analytics charts (pass rate, duration, healing vs defects)
|
|
520
|
+
- Allure report integration
|
|
521
|
+
- Export to HTML / plain text
|
|
290
522
|
|
|
291
523
|
---
|
|
292
524
|
|
|
293
|
-
## Self-Healing
|
|
525
|
+
## Self-Healing Protocol (AI-Driven)
|
|
294
526
|
|
|
295
|
-
|
|
296
|
-
|
|
527
|
+
This is **not** an automatic script. The AI agent follows this protocol:
|
|
528
|
+
|
|
529
|
+
| Attempt | What happens | If still fails |
|
|
530
|
+
|---------|-------------|----------------|
|
|
297
531
|
| 1 | Standard re-run | Classify failure |
|
|
298
532
|
| 2 | Longer timeout (60s) | Mark as defect |
|
|
299
533
|
| STOP | — | `test.fixme()` + classify |
|
|
300
534
|
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
## Project Structure
|
|
304
|
-
|
|
305
|
-
```
|
|
306
|
-
your-project/
|
|
307
|
-
├── ai-qa-workflow.js # CLI orchestrator (9 commands)
|
|
308
|
-
├── .qa-workflow.json # Project config (auto-detected, editable)
|
|
309
|
-
├── scripts/
|
|
310
|
-
│ ├── utils.js # Config loading, auto-detection, shared helpers
|
|
311
|
-
│ ├── planner.js # user-story → test-plan template
|
|
312
|
-
│ ├── generator.js # test-plan → test spec skeleton
|
|
313
|
-
│ ├── executor.js # runs Playwright tests
|
|
314
|
-
│ ├── healer.js # 2 attempts max, classifies defects
|
|
315
|
-
│ └── reporter.js # markdown + JSON reports
|
|
316
|
-
├── opencode.json # MCP config (Playwright + GitHub)
|
|
317
|
-
├── .github/agents/ # AI agent definitions
|
|
318
|
-
├── prompts/ # AI workflow instructions
|
|
319
|
-
├── qa-dashboard/ # Web UI (port 4000)
|
|
320
|
-
├── user-story/ # Your .md stories
|
|
321
|
-
├── specs/ # Generated test plans
|
|
322
|
-
├── tests/ # Generated Playwright specs
|
|
323
|
-
├── test-results/ # Run results, reports, screenshots
|
|
324
|
-
└── docs/ # Application context (AI knowledge base)
|
|
325
|
-
```
|
|
535
|
+
The mechanical `npm run qa:retry` just re-runs. True healing (fixing selectors, adjusting waits, fixing assertions) is done by the AI agent using Playwright MCP.
|
|
326
536
|
|
|
327
537
|
---
|
|
328
538
|
|
|
329
|
-
##
|
|
330
|
-
|
|
331
|
-
The pipeline supports **Allure** for rich, interactive test reports.
|
|
332
|
-
|
|
333
|
-
### Setup
|
|
334
|
-
|
|
335
|
-
```bash
|
|
336
|
-
# 1. Install Allure dependencies
|
|
337
|
-
npm install -D allure-playwright
|
|
539
|
+
## Human Supervision (You Are in Control)
|
|
338
540
|
|
|
339
|
-
|
|
340
|
-
# reporter: [["list"], ["allure-playwright"]],
|
|
541
|
+
The AI agent **never acts without your approval**. Every major phase follows this cycle:
|
|
341
542
|
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
# 4. Generate the HTML report
|
|
346
|
-
npm run qa:report:allure
|
|
543
|
+
```
|
|
544
|
+
AI proposes ──▶ You review ──▶ You approve ──▶ AI executes ──▶ You verify
|
|
347
545
|
```
|
|
348
546
|
|
|
349
|
-
|
|
547
|
+
| Phase | AI does | You approve |
|
|
548
|
+
|-------|---------|-------------|
|
|
549
|
+
| Plan | Explore app, write test plan | Review plan content |
|
|
550
|
+
| Generate | Write Playwright test code | Review selectors and logic |
|
|
551
|
+
| Heal | Diagnose failure, propose fix | Review the fix before it's applied |
|
|
350
552
|
|
|
351
|
-
|
|
553
|
+
The only exception: `npm run qa:execute` runs tests mechanically — you run this yourself or via CI.
|
|
352
554
|
|
|
353
|
-
|
|
354
|
-
-
|
|
355
|
-
-
|
|
555
|
+
The AI never:
|
|
556
|
+
- Generates tests without showing you the plan first
|
|
557
|
+
- Edits test code without showing you the diagnosis first
|
|
558
|
+
- Deploys, commits, or pushes without asking
|
|
356
559
|
|
|
357
|
-
|
|
560
|
+
**You are the supervisor. The AI is your engineer.**
|
|
358
561
|
|
|
359
562
|
---
|
|
360
563
|
|
|
361
|
-
## Token Efficiency
|
|
564
|
+
## Token Efficiency
|
|
362
565
|
|
|
363
566
|
1. Body text assertions over complex selectors
|
|
364
567
|
2. Screenshots off by default (only on failure)
|