@ai-qa/workflow 2.0.10 → 2.0.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,8 +1,41 @@
1
- # AI QA Pipeline
1
+ # AI QA Workflow Template
2
2
 
3
- **User Story Test Plan Test Generation → Execution → Self-Healing → Dashboard**
3
+ **Turn any AI agent into an autonomous QA engineer.**
4
4
 
5
- One-command install into any web project. Works standalone or with any AI assistant (opencode, GitHub Copilot, Claude Code).
5
+ This template gives an AI coding assistant (opencode, Copilot, Claude Code, Cursor, etc.) everything it needs to: explore your web app, plan tests, generate real Playwright code, execute them, debug failures, and visualize results.
6
+
7
+ The template handles the **mechanical** parts (execution, reporting, dashboard). The AI handles the **creative** parts (planning, test generation, debugging, healing).
8
+
9
+ ---
10
+
11
+ ## How It Works
12
+
13
+ ```
14
+
15
+ ┌──────────────────────────────────────┐
16
+ │ AI AGENT (does thinking) │
17
+ │ │
18
+ │ Phase 1: Environment check │
19
+ │ Phase 2: Explore app + write plan │
20
+ │ Phase 3: Generate Playwright tests │
21
+ │ Phase 4: Execute tests │
22
+ │ Phase 5: Debug + heal failures │
23
+ │ Phase 6: Report + update context │
24
+ └───────┬────────────────────┬────────┘
25
+ │ │
26
+ ┌─────────────┘ └─────────────┐
27
+ ▼ ▼
28
+ ┌─────────────────────────┐ ┌─────────────────────────┐
29
+ │ HUMAN SUPERVISION │ │ SCRIPTS (mechanics) │
30
+ │ │ │ │
31
+ │ At EVERY phase: │ │ npm run qa:execute │
32
+ │ AI proposes ──▶ you │ │ npm run qa:report │
33
+ │ approve ──▶ AI acts │ │ npm run qa:retry │
34
+ │ │ │ npm run qa:status │
35
+ │ You are always in │ │ npm run qa:list │
36
+ │ control. │ │ npm run dashboard │
37
+ └─────────────────────────┘ └─────────────────────────┘
38
+ ```
6
39
 
7
40
  ---
8
41
 
@@ -12,13 +45,13 @@ One-command install into any web project. Works standalone or with any AI assist
12
45
  npx @ai-qa/workflow init --yes
13
46
  ```
14
47
 
15
- That's it. This single command installs the full pipeline into your current project:
16
-
17
- - Copies pipeline scripts (planner, generator, executor, healer, reporter)
18
- - Installs Playwright + dashboard dependencies
19
- - Adds npm scripts (`qa:*` commands)
20
- - Creates directories (`user-story/`, `specs/`, `tests/`, `test-results/`)
21
- - Registers your project in the dashboard
48
+ Installs the full template into your project:
49
+ - Playwright MCP + GitHub MCP configuration (`opencode.json`)
50
+ - AI agent definitions (`.github/agents/`)
51
+ - Workflow prompts (`prompts/`)
52
+ - CLI scripts for execution and reporting
53
+ - QA Dashboard (web UI)
54
+ - Directory structure (`user-story/`, `specs/`, `tests/`, `test-results/`, `docs/`, `.qa-context/`)
22
55
 
23
56
  ### Update an existing installation
24
57
 
@@ -26,68 +59,276 @@ That's it. This single command installs the full pipeline into your current proj
26
59
  npx @ai-qa/workflow update --yes
27
60
  ```
28
61
 
29
- Updates pipeline files while **preserving** your configuration (`.qa-workflow.json`, `opencode.json`), user stories, test specs, and run results.
62
+ Updates template files while preserving your user stories, test specs, config, and run results.
63
+
64
+ ---
30
65
 
31
- ### What you get
66
+ ## What You Get
32
67
 
33
68
  ```
34
69
  your-project/
35
- ├── ai-qa-workflow.js # CLI orchestrator (10 commands)
36
- ├── .qa-workflow.json # Project config (auto-detected, editable)
37
- ├── scripts/ # 6 automation modules
70
+ ├── ai-qa-workflow.js # CLI orchestrator (8 commands)
71
+ ├── .qa-workflow.json # Project config (auto-detected)
72
+ ├── scripts/
73
+ │ ├── utils.js # Config loading, shared helpers
74
+ │ ├── executor.js # Runs Playwright tests
75
+ │ ├── retrier.js # Re-runs failed tests (longer timeout)
76
+ │ └── reporter.js # Markdown + JSON reports
38
77
  ├── opencode.json # MCP config (Playwright + GitHub)
39
78
  ├── .github/agents/ # AI agent definitions
40
- ├── prompts/ # AI workflow instructions
79
+ ├── playwright-test-planner.agent.md
80
+ │ ├── playwright-test-generator.agent.md
81
+ │ └── playwright-test-healer.agent.md
82
+ ├── prompts/
83
+ │ ├── QAe2eprompt.md # Full 9-step AI workflow
84
+ │ ├── general_prompt.md # Quick-start prompt
85
+ ├── prompting_template.md # Conversation guide — what to say to the AI at each phase
86
+ ├── router.md # AI entry point — routes to correct agent
41
87
  ├── qa-dashboard/ # Web UI (port 4000)
42
- ├── user-story/ # Your .md stories
43
- ├── specs/ # Generated test plans
44
- ├── tests/ # Generated Playwright specs
45
- └── test-results/
46
- # Run results, reports, screenshots
88
+ ├── .qa-context/ # Persistent AI memory (pipeline state, selectors, healing history, traceability)
89
+ ├── .auth/ # Auth credentials + Playwright storage state (gitignored)
90
+ ├── user-story/ # Your .md stories
91
+ ├── specs/ # AI-generated test plans
92
+ ├── tests/ # AI-generated Playwright specs
93
+ ├── test-results/ # Run results, reports, screenshots
94
+ └── docs/ # Application context (AI knowledge base)
47
95
  ```
48
96
 
49
- ### Quick start after install
97
+ ---
98
+
99
+ ## Quick Start
50
100
 
51
101
  ```bash
52
- # 1. Initialize (auto-detects project config)
102
+ # 1. Install the template
103
+ npx @ai-qa/workflow init --yes
104
+
105
+ # 2. Initialize project config
53
106
  npm run qa:init
54
107
 
55
- # 2. Write a user story in user-story/ or let the AI create one
108
+ # 3. Write a user story in user-story/my-feature.md (see format below)
109
+
110
+ # 4. Open the project in your AI editor (opencode, Copilot, Cursor)
111
+ # The AI agent auto-detects its role and runs an environment check.
112
+ # It will report what's ready and what's missing.
113
+
114
+ # 5. Send this one prompt to start:
115
+ # "Read router.md and follow the QA workflow for my-feature.md"
116
+
117
+ # The AI will:
118
+ # - Read router.md → understand its mission
119
+ # - Read playwright-test-planner.agent.md
120
+ # - Run environment check (if not already done)
121
+ # - Explore your app with Playwright MCP
122
+ # - Save a test plan to specs/
123
+ # - STOP and ask for your approval
124
+ # - Once approved → generate tests → STOP again
125
+ # - Tell you when ready to execute
56
126
 
57
- # 3. Run full pipeline
58
- npm run qa:run my-story.md
127
+ # 6. Execute tests
128
+ npm run qa:execute
129
+
130
+ # 7. If tests fail, tell your AI agent:
131
+ # "Debug and fix the failing tests"
59
132
 
60
- # 4. Start the dashboard
133
+ # 8. Launch dashboard
61
134
  npm run dashboard
62
135
  ```
63
136
 
137
+ > **💡 Full conversation guide:** See `prompting_template.md` for a complete script of what to say to the AI at every phase — from installation to final report.
138
+
64
139
  ---
65
140
 
66
- ## How It Works
141
+ ## AI Agent Auto-Bootstrap
67
142
 
68
- The framework operates in two distinct, powerful phases:
143
+ When you open this project in an AI-powered editor, the agent **automatically** understands its purpose:
69
144
 
70
- ### Phase 1: Creation (Agent-Driven)
71
- You interact with an AI Agent in your IDE (Cursor, OpenCode, Copilot, Antigravity). You simply provide a URL or a Markdown User Story. The agent automatically:
72
- 1. Loads its intelligence from `agents/router.md`
73
- 2. Uses Playwright MCP to open a hidden browser
74
- 3. Visually explores your application in real-time
75
- 4. Generates a flawless test plan (`specs/`) and executable code (`tests/`) using **real** semantic selectors. No more hallucinated CSS classes!
145
+ 1. **Reads `.opencode/rules.md`** (or `.github/copilot-instructions.md`) → discovers it's an **AI QA Engineer**
146
+ 2. **Runs an environment check** reports what's ready and what's missing
147
+ 3. **Reads `router.md`** → learns the workflow and supervision rules
148
+ 4. **Stops and waits** for you to provide a user story
76
149
 
77
- ### Phase 2: Execution & Tracking (Dashboard-Driven)
78
- Once the test scripts are generated by the Agent, you move to execution:
79
- 1. Open the Web Dashboard (`npm run dashboard`)
80
- 2. Click **Execute** or trigger it via your CI/CD pipeline
81
- 3. The framework handles Playwright execution, auto-healing (fixing minor flakiness), and report generation
82
- 4. View deep analytics, execution history, and Allure reports directly in the Dashboard UI.
150
+ No manual instructions needed. The agent knows its role on first contact.
83
151
 
84
152
  ---
85
153
 
86
- ## Zero-Config Design
154
+ ## Your First Prompt
155
+
156
+ After installation and running `npm run qa:init`, open the project in your AI editor.
157
+
158
+ The very first thing you should say to the AI agent:
159
+
160
+ > **"Read router.md and follow the QA workflow for my-story.md"**
161
+
162
+ (Replace `my-story.md` with the name of your user story file in `user-story/`.)
163
+
164
+ > **📖 Need more prompts?** See `prompting_template.md` for the full conversation script — approval responses, healing prompts, report prompts, and an example session.
165
+
166
+ ### What happens when you send this prompt:
167
+
168
+ | Step | AI does this | You do this |
169
+ |------|-------------|-------------|
170
+ | 1 | Runs environment check (if first time) | Read the status report |
171
+ | 2 | Reads `router.md` → `playwright-test-planner.agent.md` | — |
172
+ | 3 | Explores your app with Playwright MCP | — |
173
+ | 4 | Writes test plan to `specs/` | **Review and approve** |
174
+ | 5 | Reads `playwright-test-generator.agent.md` | — |
175
+ | 6 | Generates test files to `tests/` | **Review and approve** |
176
+ | 7 | Tells you tests are ready to run | Run `npm run qa:execute` |
177
+ | 8 | If tests fail, debugs and proposes fix | **Review and approve fix** |
178
+
179
+ **You are the supervisor.** The AI never moves to the next phase without your approval.
180
+
181
+ ---
182
+
183
+ ## For Humans: Understanding the Workflow
184
+
185
+ The template follows a **9-step AI workflow** defined in `prompts/QAe2eprompt.md`. Here's what happens at each step:
186
+
187
+ ### Step 1 — Context Discovery
188
+ The AI reads `docs/application-context.md` and explores your project to understand the tech stack, authentication, and environment.
189
+
190
+ ### Step 2 — Test Strategy & Plan
191
+ The AI reads `playwright-test-planner.agent.md` and uses Playwright MCP to explore your app visually. It maps user flows, identifies critical paths, and designs comprehensive test scenarios. The plan is saved to `specs/`.
192
+
193
+ > **⛔ Approval gate:** The AI **stops** after saving the plan and presents it to you. You review, give feedback, and say "approved" or "continue" before the AI proceeds to test generation.
194
+
195
+ ### Step 3 — Manual Exploratory Testing
196
+ The AI executes each scenario manually using Playwright MCP, capturing real selectors, observing behavior, and noting issues.
197
+
198
+ ### Step 4 — Test Code Generation
199
+ The AI reads `playwright-test-generator.agent.md` and writes real Playwright `.spec.ts` files using selectors it discovered during exploration. Generated files go in `tests/`.
200
+
201
+ > **⛔ Approval gate:** The AI **stops** after generating test files and presents them to you. You review the code, check selectors and logic, then say "approved" or "execute" before the AI proceeds.
202
+
203
+ ### Step 5 — Execution
204
+ Run the tests mechanically:
205
+
206
+ ```bash
207
+ npm run qa:execute [test-name]
208
+ ```
209
+
210
+ Or run all tests:
211
+ ```bash
212
+ npm run qa:execute
213
+ ```
214
+
215
+ The executor runs Playwright with the configured options and saves results to `test-results/`.
216
+
217
+ ### Step 6 — Debug & Heal (AI)
218
+ If tests fail, the AI reads `playwright-test-healer.agent.md` and:
219
+ 1. Runs the failing test with `test_run`
220
+ 2. Debugs with `test_debug` — examines the actual UI state
221
+ 3. Classifies the failure and proposes a fix (1-3 line change)
222
+
223
+ > **⛔ Approval gate:** The AI **stops** and presents its diagnosis to you. It shows what's broken, why, and what it wants to change. You approve the fix before the AI edits any file.
224
+
225
+ 4. Once approved, applies fix and re-runs once
226
+ 5. If still failing, marks as `test.fixme()` and reports as defect
227
+
228
+ For a quick mechanical re-run (no AI diagnosis):
229
+
230
+ ```bash
231
+ npm run qa:retry [run-id]
232
+ ```
233
+
234
+ This re-runs failed tests with a longer timeout. If they still fail, the AI investigates.
235
+
236
+ ### Step 7 — Bug Classification (AI)
237
+ The AI classifies every defect by severity, priority, type, root cause, and reproducibility. Results are saved to `test-results/defects-log.md`.
238
+
239
+ ### Step 8 — Report
240
+ ```bash
241
+ npm run qa:report [run-id]
242
+ ```
243
+
244
+ Generates a markdown report from execution results.
245
+
246
+ ### Step 9 — Knowledge Retention
247
+ The AI updates `docs/application-context.md` with:
248
+ - Stable selectors discovered
249
+ - Known flaky areas
250
+ - Healing strategies that worked
251
+ - Environment observations
252
+
253
+ ---
254
+
255
+ ## Installation
256
+
257
+ ### Requirements
258
+
259
+ | Component | Needed for | Install |
260
+ |-----------|-----------|---------|
261
+ | **Node.js 18+** | Running the pipeline | — |
262
+ | **Playwright** | Test execution | `npm install @playwright/test` |
263
+ | **Chromium** | Running tests | `npx playwright install chromium` |
264
+ | **Playwright MCP** | AI browser automation | `npm install -D @playwright/mcp` |
265
+ | **Applitools MCP** | Visual testing (screenshot comparison) | `npm install -D @applitools/mcp` + `APPLITOOLS_API_KEY` |
266
+ | **GitHub MCP** | AI creating PRs/issues | `npm install -D @modelcontextprotocol/server-github` + `GITHUB_TOKEN` |
267
+
268
+ ### Install into a project
269
+
270
+ ```bash
271
+ npx @ai-qa/workflow init --yes
272
+ ```
273
+
274
+ Or from local template:
275
+ ```bash
276
+ node install.js ../my-project --yes
277
+ ```
278
+
279
+ ### Update
280
+
281
+ ```bash
282
+ npx @ai-qa/workflow update --yes
283
+ ```
284
+
285
+ ---
286
+
287
+ ## Visual Testing (Applitools)
288
+
289
+ The template supports **Applitools MCP** for automated visual testing.
290
+
291
+ If `APPLITOOLS_API_KEY` is configured in your environment, the AI agent automatically adds visual checkpoints to critical pages during test generation. It captures screenshots of pages like login, dashboard, and checkout, and compares them against baselines to detect visual regressions.
292
+
293
+ ### Setup
294
+
295
+ ```bash
296
+ # 1. Install Applitools MCP
297
+ npm install -D @applitools/mcp
298
+
299
+ # 2. Set your API key (get it from https://applitools.com)
300
+ # Option A: Export in terminal
301
+ export APPLITOOLS_API_KEY=votre_clé_ici
302
+
303
+ # Option B: Add to .env file
304
+ echo "APPLITOOLS_API_KEY=votre_clé_ici" >> .env
305
+ ```
306
+
307
+ The AI will detect the key during its environment check and use Applitools automatically. If the key is not set, visual testing is skipped entirely — no errors, no blocks.
308
+
309
+ ---
310
+
311
+ ## Commands
312
+
313
+ | Command | Description | Who does it |
314
+ |---------|-------------|-------------|
315
+ | `npm run qa:init` | Create directories + auto-detect config | You (once) |
316
+ | `npm run qa:execute [test]` | Run Playwright tests | You or CI |
317
+ | `npm run qa:retry [run-id]` | Re-run failed tests (longer timeout) | You or CI |
318
+ | `npm run qa:report [run-id]` | Generate markdown report | You or CI |
319
+ | `npm run qa:status` | Show pipeline state | You |
320
+ | `npm run qa:list` | List stories, plans, specs | You |
321
+ | `node ai-qa-workflow.js context <phase> <story>` | Mark a pipeline phase complete for a story | AI agent or you |
322
+ | `npm run dashboard` | Launch web UI at :4000 | You |
87
323
 
88
- The template works out of the box by **auto-detecting** your project configuration.
324
+ > The commands `qa:plan`, `qa:generate`, and `qa:run` do not exist in this template.
325
+ > Planning, test generation, and healing are done by the AI agent, not by scripts.
89
326
 
90
- On `npm run qa:init`, it scans your project and generates `.qa-workflow.json`:
327
+ ---
328
+
329
+ ## Zero-Config Design
330
+
331
+ On `npm run qa:init`, the template auto-detects your project configuration:
91
332
 
92
333
  | Scan source | What it detects |
93
334
  |-------------|----------------|
@@ -98,15 +339,12 @@ On `npm run qa:init`, it scans your project and generates `.qa-workflow.json`:
98
339
  | `playwright.config.*` | Browser type (edge, webkit) |
99
340
  | Directory structure | Framework (Angular, Next.js, Python) |
100
341
 
101
- If a file is missing, the template falls back to sensible defaults. No manual setup required.
102
-
103
- ### The Config File (`.qa-workflow.json`)
342
+ Generated `.qa-workflow.json`:
104
343
 
105
344
  ```json
106
345
  {
107
346
  "project": {
108
347
  "name": "my-app",
109
- "description": "My web application",
110
348
  "url": "http://localhost:5173",
111
349
  "environment": "Development"
112
350
  },
@@ -127,100 +365,121 @@ If a file is missing, the template falls back to sensible defaults. No manual se
127
365
  }
128
366
  ```
129
367
 
130
- Edit this file to override auto-detected values. The template merges your settings with defaults — you only need to set what differs.
368
+ Edit this file to override auto-detected values.
131
369
 
132
- ### Application Context (`docs/application-context.md`)
370
+ ---
133
371
 
134
- This file serves as a **knowledge base for AI agents** during test planning and healing. On `init` it's created from config values. As you use the pipeline, AI agents populate it with:
372
+ ## Application Context (`docs/application-context.md`)
373
+
374
+ This file serves as the **AI's knowledge base**. On `init` it's auto-generated. The AI populates it during exploration and testing with:
135
375
 
136
376
  - **Stable selectors** — CSS/XPath selectors discovered during exploration
137
377
  - **Known flaky areas** — elements or flows that frequently break
138
378
  - **Auth details** — users, tokens, environments
139
379
  - **Tech stack notes** — framework specifics the AI should know
140
380
 
141
- For best results, edit this file with your project's specifics before running the pipeline.
381
+ For best results, edit this file with your project's specifics before the AI starts.
142
382
 
143
383
  ---
144
384
 
145
- ## Installation
385
+ ## Persistent AI Memory (`.qa-context/`)
146
386
 
147
- ### Requirements
387
+ The `.qa-context/` directory stores **structured memory** that persists between AI sessions and pipeline phases:
148
388
 
149
- | Component | Needed for | Install |
150
- |-----------|-----------|---------|
151
- | **Node.js 18+** | Running the pipeline | |
152
- | **Playwright** | Test execution | `npm install @playwright/test` |
153
- | **Chromium** | Running tests | `npx playwright install chromium` |
154
- | **Playwright MCP** | AI browser automation (explore, click, type) | `npm install -D @playwright/mcp` |
155
- | **GitHub MCP** | AI creating PRs/issues from results | `npm install -D @modelcontextprotocol/server-github` + `GITHUB_TOKEN` env |
389
+ | File | Content | Used by |
390
+ |------|---------|---------|
391
+ | `pipeline.json` | Current pipeline state: which phases are complete, current story, last run | All agents (read on start, write on complete) |
392
+ | `selectors.json` | All discovered selectors with reliability scores, healing history, recommended alternatives | Generator (write best selectors), Healer (read alternatives) |
393
+ | `heal-history.json` | Every healing attempt: what was tried, which strategy, whether it succeeded | Healer (avoid repeating failed attempts), Planner (avoid flaky pages) |
394
+ | `traceability.json` | Full mapping: story plan spec → runs → healing | Reporter (link report to story), Dashboard (display traceability) |
156
395
 
157
- ### Install into a project
396
+ Each agent reads `.qa-context/` **before starting** and updates it **after completing**. This means:
397
+ - The Generator reuses stable selectors discovered by the Planner
398
+ - The Healer knows which selectors were tried and failed before
399
+ - The Reporter links every run back to its original story
400
+ - The Dashboard can display the full audit trail
158
401
 
159
- ```bash
160
- # From npm (recommended — works anywhere)
161
- npx @ai-qa/workflow init --yes
402
+ The AI automatically consults these files — no manual setup needed.
162
403
 
163
- # From local template
164
- node install.js ../my-project --yes
165
- ```
404
+ ---
166
405
 
167
- The installer copies all pipeline files, creates directories, adds npm scripts, installs dashboard dependencies, and registers your project in the dashboard.
406
+ ## Authentication Management (`.auth/`)
168
407
 
169
- ### Update an installed project
408
+ The `.auth/` directory stores everything needed for automated login during tests:
170
409
 
171
- ```bash
172
- # From npm
173
- npx @ai-qa/workflow update --yes
410
+ | File | Content | Git |
411
+ |------|---------|:---:|
412
+ | `credentials.json` | Username + password for the app | ❌ ignored |
413
+ | `storage-state.json` | Playwright session (cookies, localStorage, tokens) | ❌ ignored |
414
+ | `.gitignore` | Ensures nothing in `.auth/` is committed | ✅ |
174
415
 
175
- # From local template
176
- node install.js ../my-project --update --yes
177
- ```
416
+ ### How it works
178
417
 
179
- Updates pipeline files while preserving: `.qa-workflow.json`, `opencode.json`, `user-story/`, `specs/`, `tests/`, `test-results/`, `qa-dashboard/data/`, `qa-dashboard/node_modules/`.
418
+ 1. **`npm run qa:init`** creates `.auth/` with `.gitignore` ready to use
419
+ 2. **AI discovers login** during Phase 1 (Plan) — navigates to `/login`, detects form fields, saves structure to `.qa-context/auth.json`
420
+ 3. **You provide credentials** once — the AI saves them to `.auth/credentials.json`
421
+ 4. **Session is persisted** — after first login, Playwright's storage state is saved to `.auth/storage-state.json`
422
+ 5. **Tests reuse the session** — generated `auth.setup.ts` loads storage state before each run
423
+ 6. **If credentials change**, the AI detects auth failures via `auth-manager.js` and prompts you to update `.auth/credentials.json`
180
424
 
181
- ### Commands added to your project
425
+ ### First-time setup
182
426
 
183
427
  ```bash
184
- npm run qa:init # Initialize + auto-detect config
185
- npm run qa:run <story> # Full pipeline: plan generate execute heal report
186
- npm run qa:plan <story> # Generate test plan only
187
- npm run qa:generate # Generate test code from plan
188
- npm run qa:execute # Run Playwright tests
189
- npm run qa:heal # Self-heal failed tests
190
- npm run qa:report # Generate final report
191
- npm run qa:status # Check pipeline state
192
- npm run qa:list # List stories, plans, specs
193
- npm run dashboard # Launch web dashboard at :4000
428
+ # 1. After qa:init, open the project in your AI editor
429
+ # 2. The AI will ask for credentials during Phase 1 (Plan)
430
+ # 3. Provide them once they're saved to .auth/credentials.json
431
+ # 4. All future runs reuse the stored session
194
432
  ```
195
433
 
196
- ---
434
+ ### Manual setup
197
435
 
198
- ## How to Get the Most Out of It
436
+ ```json
437
+ {
438
+ "username": "test@example.com",
439
+ "password": "your-password",
440
+ "url": "http://localhost:3000/login"
441
+ }
442
+ ```
199
443
 
200
- ### 1. Provide good context upfront
444
+ Save this as `.auth/credentials.json`. The AI detects it automatically on next run.
201
445
 
202
- Before running the pipeline, populate `docs/application-context.md`:
446
+ ---
203
447
 
204
- ```markdown
205
- ## Project
206
- - Name: My App
207
- - URL: https://staging.myapp.com
208
- - Environment: Staging
209
-
210
- ## Authentication
211
- - Login flow: email + password via /api/auth/login
212
- - Test user: test@example.com / TestPass123
213
-
214
- ## Stable Selectors
215
- - Login button: button[data-testid="login-btn"]
216
- - Error toast: .toast-error
217
- ```
448
+ ## How the AI Agents Work
449
+
450
+ The template defines three specialized AI agents in `.github/agents/`:
451
+
452
+ ### Planner Agent
453
+ Triggers when the AI needs to create a test plan. The AI:
454
+ 1. Opens your app with Playwright MCP
455
+ 2. Explores navigation, flows, UI components
456
+ 3. Maps critical paths and edge cases
457
+ 4. Saves a structured test plan to `specs/`
458
+ 5. **⛔ Stops and presents the plan to you for approval**
459
+
460
+ ### Generator Agent
461
+ Triggers when the AI needs to write test code. The AI:
462
+ 1. Reads the test plan
463
+ 2. Uses Playwright to manually execute each step
464
+ 3. Captures real selectors (data-testid, aria-label, role)
465
+ 4. Adds visual checkpoints via Applitools on critical pages (if `APPLITOOLS_API_KEY` is set)
466
+ 5. Writes complete, executable Playwright tests to `tests/`
467
+ 6. **⛔ Stops and presents the code to you for approval**
468
+
469
+ ### Healer Agent
470
+ Triggers when tests fail. The AI:
471
+ 1. Runs only the failing tests
472
+ 2. Debugs with `test_debug` — sees the actual UI state
473
+ 3. Classifies the failure (selector, timing, or bug)
474
+ 4. **⛔ Stops and presents diagnosis + proposed fix for your approval**
475
+ 5. Once approved, applies a targeted 1-3 line fix
476
+ 6. Re-runs once — if still failing, marks as `test.fixme()`
218
477
 
219
- The AI reads this file during test planning and healing. The more accurate this is, the better the generated tests.
478
+ ---
220
479
 
221
- ### 2. Write structured user stories
480
+ ## Writing Good User Stories
222
481
 
223
- Your `user-story/*.md` files should follow this format:
482
+ The AI reads your user stories to understand what to test. Write them in `user-story/`:
224
483
 
225
484
  ```markdown
226
485
  # User Login
@@ -243,122 +502,66 @@ As a registered user, I want to log in with my credentials so I can access the d
243
502
  4. User name displayed in header
244
503
  ```
245
504
 
246
- Well-structured stories produce better test plans. The planner extracts acceptance criteria, preconditions, and feature name automatically.
247
-
248
- ### 3. Review generated test plans
249
-
250
- After `npm run qa:plan`, check `specs/*-test-plan.md`. The AI generates scenarios from acceptance criteria. Review and edit before generating test code:
251
-
252
- - Add edge cases the AI missed
253
- - Add negative test scenarios
254
- - Fix any incorrect assumptions about the app
255
-
256
- ### 4. Use the run mode that fits your workflow
505
+ Well-structured stories produce better test plans and better automated tests.
257
506
 
258
- | Mode | Command | Best for |
259
- |------|---------|----------|
260
- | **Full auto** | `npm run qa:run <story>` | Quick smoke tests, CI pipelines |
261
- | **Step by step** | plan → generate → execute → heal → report | Reviewing each stage, manual edits |
262
- | **Manual test only** | `npm run qa:execute <test>` | When tests are already written |
263
- | **Heal only** | `npm run qa:heal [runId]` | After manual test edits |
264
-
265
- ### 5. Understand self-healing
266
-
267
- The healer makes up to **2 attempts** per failed test:
268
-
269
- 1. **Standard re-run** — flakes due to timing or network
270
- 2. **Longer timeout (60s)** — slow responses
271
- 3. **Stop** — marks as defect, classifies the failure
272
-
273
- Failure classification:
274
- | Type | Label | Auto-healed? |
275
- |------|-------|-------------|
276
- | Selector not found | `selector` | ✅ Usually |
277
- | Timeout | `timing` | ✅ Usually |
278
- | Target closed | `environment` | ❌ Bug |
279
- | Network error | `environment` | ❌ Bug |
280
- | Test syntax error | `test-syntax` | ❌ Manual fix |
507
+ ---
281
508
 
282
- ### 6. Check the dashboard
509
+ ## Dashboard
283
510
 
284
511
  ```bash
285
512
  npm run dashboard
286
513
  # → http://localhost:4000
287
514
  ```
288
515
 
289
- The dashboard shows multi-project management, execution history with pass/fail/healed stats, analytics charts (pass rate, duration, healing vs defects), and test data generation tools.
516
+ The dashboard provides:
517
+ - Multi-project management
518
+ - Execution history with pass/fail/healed stats
519
+ - Analytics charts (pass rate, duration, healing vs defects)
520
+ - Allure report integration
521
+ - Export to HTML / plain text
290
522
 
291
523
  ---
292
524
 
293
- ## Self-Healing Rules
525
+ ## Self-Healing Protocol (AI-Driven)
294
526
 
295
- | Attempt | Strategy | If still fails |
296
- |---------|----------|----------------|
527
+ This is **not** an automatic script. The AI agent follows this protocol:
528
+
529
+ | Attempt | What happens | If still fails |
530
+ |---------|-------------|----------------|
297
531
  | 1 | Standard re-run | Classify failure |
298
532
  | 2 | Longer timeout (60s) | Mark as defect |
299
533
  | STOP | — | `test.fixme()` + classify |
300
534
 
301
- ---
302
-
303
- ## Project Structure
304
-
305
- ```
306
- your-project/
307
- ├── ai-qa-workflow.js # CLI orchestrator (9 commands)
308
- ├── .qa-workflow.json # Project config (auto-detected, editable)
309
- ├── scripts/
310
- │ ├── utils.js # Config loading, auto-detection, shared helpers
311
- │ ├── planner.js # user-story → test-plan template
312
- │ ├── generator.js # test-plan → test spec skeleton
313
- │ ├── executor.js # runs Playwright tests
314
- │ ├── healer.js # 2 attempts max, classifies defects
315
- │ └── reporter.js # markdown + JSON reports
316
- ├── opencode.json # MCP config (Playwright + GitHub)
317
- ├── .github/agents/ # AI agent definitions
318
- ├── prompts/ # AI workflow instructions
319
- ├── qa-dashboard/ # Web UI (port 4000)
320
- ├── user-story/ # Your .md stories
321
- ├── specs/ # Generated test plans
322
- ├── tests/ # Generated Playwright specs
323
- ├── test-results/ # Run results, reports, screenshots
324
- └── docs/ # Application context (AI knowledge base)
325
- ```
535
+ The mechanical `npm run qa:retry` just re-runs. True healing (fixing selectors, adjusting waits, fixing assertions) is done by the AI agent using Playwright MCP.
326
536
 
327
537
  ---
328
538
 
329
- ## Allure Reports
330
-
331
- The pipeline supports **Allure** for rich, interactive test reports.
332
-
333
- ### Setup
334
-
335
- ```bash
336
- # 1. Install Allure dependencies
337
- npm install -D allure-playwright
539
+ ## Human Supervision (You Are in Control)
338
540
 
339
- # 2. Add to Playwright config (playwright.config.ts or .js)
340
- # reporter: [["list"], ["allure-playwright"]],
541
+ The AI agent **never acts without your approval**. Every major phase follows this cycle:
341
542
 
342
- # 3. Run tests (generates allure-results/)
343
- npm run qa:execute
344
-
345
- # 4. Generate the HTML report
346
- npm run qa:report:allure
543
+ ```
544
+ AI proposes ──▶ You review ──▶ You approve ──▶ AI executes ──▶ You verify
347
545
  ```
348
546
 
349
- ### View in Dashboard
547
+ | Phase | AI does | You approve |
548
+ |-------|---------|-------------|
549
+ | Plan | Explore app, write test plan | Review plan content |
550
+ | Generate | Write Playwright test code | Review selectors and logic |
551
+ | Heal | Diagnose failure, propose fix | Review the fix before it's applied |
350
552
 
351
- Once generated, the Allure report is available from the dashboard:
553
+ The only exception: `npm run qa:execute` runs tests mechanically — you run this yourself or via CI.
352
554
 
353
- - **Runs list** → "Allure Report" button on each run
354
- - **Run detail** "Open Allure Report ↗" in the Report Actions card
355
- - Direct URL: `/allure-report?project=<project-id>`
555
+ The AI never:
556
+ - Generates tests without showing you the plan first
557
+ - Edits test code without showing you the diagnosis first
558
+ - Deploys, commits, or pushes without asking
356
559
 
357
- The dashboard auto-detects `allure-report/index.html` and shows the buttons only when the report exists.
560
+ **You are the supervisor. The AI is your engineer.**
358
561
 
359
562
  ---
360
563
 
361
- ## Token Efficiency (for AI)
564
+ ## Token Efficiency
362
565
 
363
566
  1. Body text assertions over complex selectors
364
567
  2. Screenshots off by default (only on failure)