qaa-agent 1.9.2 → 1.9.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/CHANGELOG.md +230 -179
  2. package/CLAUDE.md +720 -557
  3. package/README.md +384 -357
  4. package/VERSION +1 -1
  5. package/agents/qa-pipeline-orchestrator.md +1739 -1425
  6. package/agents/qaa-analyzer.md +0 -1
  7. package/agents/qaa-bug-detective.md +792 -655
  8. package/agents/qaa-codebase-mapper.md +54 -1
  9. package/agents/qaa-discovery.md +431 -384
  10. package/agents/qaa-e2e-runner.md +717 -577
  11. package/agents/qaa-executor.md +966 -830
  12. package/agents/qaa-planner.md +14 -1
  13. package/agents/qaa-project-researcher.md +539 -400
  14. package/agents/qaa-scanner.md +86 -1
  15. package/agents/qaa-testid-injector.md +0 -1
  16. package/agents/qaa-validator.md +86 -1
  17. package/bin/install.cjs +374 -252
  18. package/bin/lib/context7-cache.cjs +299 -0
  19. package/bin/lib/intent-detector.cjs +490 -0
  20. package/commands/qa-audit.md +269 -126
  21. package/commands/qa-create-test-ado.md +439 -404
  22. package/commands/qa-create-test.md +672 -365
  23. package/commands/qa-fix.md +691 -513
  24. package/commands/qa-map.md +319 -139
  25. package/commands/qa-pr.md +63 -0
  26. package/commands/qa-research.md +212 -157
  27. package/commands/qa-start.md +62 -6
  28. package/commands/qa-test-report.md +254 -219
  29. package/commands/qa-testid.md +31 -0
  30. package/frameworks/cypress.json +54 -0
  31. package/frameworks/jest.json +59 -0
  32. package/frameworks/playwright.json +58 -0
  33. package/frameworks/pytest.json +59 -0
  34. package/frameworks/robot-framework.json +54 -0
  35. package/frameworks/selenium.json +65 -0
  36. package/frameworks/vitest.json +57 -0
  37. package/package.json +41 -38
  38. package/workflows/qa-analyze.md +100 -4
  39. package/workflows/qa-from-ticket.md +50 -2
  40. package/workflows/qa-gap.md +50 -2
  41. package/workflows/qa-start.md +2048 -1405
  42. package/workflows/qa-testid.md +50 -2
  43. package/workflows/qa-validate.md +50 -2
@@ -1,404 +1,439 @@
1
- # QA Create Test — Azure DevOps
2
-
3
- Retrieve an Azure DevOps work item, analyze its content, and generate well-structured Test Cases directly in Azure DevOps using the ADO MCP tools. Each test case is tagged for test plan membership (Smoke, Regression, Critical) and linked back to the source work item for full traceability. Integrates with the QAA pipeline: reads codebase map, locator registry, and user preferences for context-aware test case generation.
4
-
5
- ## Usage
6
-
7
- ```
8
- /qa-create-test-ado <work-item-id> [--area-path=<path>] [--iteration-path=<path>] [--skip-map] [--skip-dedup] [--app-url <url>]
9
- ```
10
-
11
- ### Arguments
12
-
13
- | Parameter | Purpose | Default |
14
- |-----------|---------|---------|
15
- | `<work-item-id>` | Azure DevOps work item ID to generate test cases from | Required |
16
- | `--area-path=<path>` | Override area path for all created test artifacts | Source work item's area path |
17
- | `--iteration-path=<path>` | Override iteration path for all created test artifacts | Source work item's iteration path |
18
- | `--skip-map` | Skip codebase map check and proceed without project context | false |
19
- | `--skip-dedup` | Skip deduplication check against existing linked test cases | false |
20
- | `--app-url <url>` | URL of running application for locator extraction via Playwright MCP | auto-detect |
21
-
22
- ## What It Produces
23
-
24
- - Test Cases created directly in Azure DevOps (via `testplan_create_test_case`)
25
- - Test Cases linked to source work item via *Tested By* relationship
26
- - Tags applied: `Smoke`, `Regression`, `Critical`, `AutomationCandidate`, `NeedsReview`
27
- - `ai-tasks/ticket-{id}/test-cases.md` structured report
28
- - Report attached to work item (if `ADO_MCP_AUTH_TOKEN` is set) or written to `Custom.QATestCasesReport` field (fallback)
29
-
30
- ---
31
-
32
- ## Process
33
-
34
- ### Phase 1: Read Pipeline Context
35
-
36
- Before retrieving the work item, read QAA pipeline artifacts for context-aware generation.
37
-
38
- 1. **Read `CLAUDE.md`** — POM rules, locator tiers, assertion rules, naming conventions, quality gates, test spec rules.
39
-
40
- 2. **Read user preferences** — `~/.claude/qaa/MY_PREFERENCES.md` (if exists). User overrides win over defaults.
41
-
42
- 3. **Check for codebase map** (`.qa-output/codebase/`):
43
- - Look for: `CODE_PATTERNS.md`, `API_CONTRACTS.md`, `TEST_SURFACE.md`, `TESTABILITY.md`, `RISK_MAP.md`, `CRITICAL_PATHS.md`
44
- - If at least 2 exist: read them all for project context (naming conventions, API shapes, testable surfaces, risk areas).
45
- - If NONE exist and `--skip-map` not passed: warn the user that test cases will lack project context, suggest running `/qa-map` first. Continue anyway (ADO test cases are higher-level than code-level tests).
46
-
47
- 4. **Check locator registry** — `.qa-output/locators/LOCATOR_REGISTRY.md` (if exists):
48
- - If locators exist for pages related to the work item's feature: reference them in test step expected results (e.g., "Verify element `[data-testid='login-submit-btn']` is visible").
49
- - If `--app-url` provided and locators missing: use Playwright MCP to extract locators from the live app before designing test steps:
50
- ```
51
- mcp__playwright__browser_navigate({ url: "{app_url}/{feature_path}" })
52
- mcp__playwright__browser_snapshot()
53
- ```
54
- - Write extracted locators to `.qa-output/locators/{feature}.locators.md` and update the registry.
55
-
56
- ---
57
-
58
- ### Phase 2: Retrieve the Work Item
59
-
60
- Use `wit_get_work_item` with `expand: "relations"` to fetch the full work item:
61
-
62
- - Capture: **title**, **type** (`Bug`, `User Story`, `Ticket`), **state**, **assigned-to**, **area path**, **iteration path**
63
- - Capture all relevant content fields based on type (see Phase 3)
64
- - Note the project for all subsequent calls
65
-
66
- **Also retrieve comments** using `wit_list_work_item_comments`:
67
-
68
- - Read all comments in chronological order
69
- - Look for: acceptance criteria added in comments, QA notes, scope clarifications, tester feedback, or any conditions of satisfaction mentioned informally
70
- - These often contain implied test cases not captured in the formal fields
71
-
72
- **Also check attachments** from the relations list (entries where `rel` equals `AttachedFile`):
73
-
74
- - Filter to `.csv` and `.txt` files (case-insensitive) by inspecting `attributes.name`
75
- - If found, download via:
76
- ```bash
77
- curl -s --user ":{AZURE_DEVOPS_PAT}" "{attachment-url}"
78
- ```
79
- - Read content for test data, expected values, error logs, or sample datasets that define expected behavior
80
-
81
- ---
82
-
83
- ### Phase 2b: Deduplication Check Query Existing Test Cases
84
-
85
- Before generating any new test cases, check whether the source work item already has linked test cases to prevent duplicates.
86
-
87
- 1. Inspect the relations returned in Phase 2 — filter for link type `"Microsoft.VSTS.Common.TestedBy-Forward"` (i.e., *Tested By* links).
88
- 2. For each linked test case ID found, call `wit_get_work_item` to retrieve its **title** and **state**.
89
- 3. Build an **existing TC registry** — a list of `{ id, title, state }` for all currently linked test cases.
90
- 4. In Phase 5, before calling `testplan_create_test_case` for each planned TC, compare its title (normalized: lowercase, trimmed) against every title in the registry.
91
- - **If match found** and existing TC is in state `Design`, `Ready`, or `Closed`: skip creation, log `"Skipped — duplicate of TC #{id}"`.
92
- - **If match found** but existing TC is in state `Removed`: create the new TC anyway (the old one was intentionally discarded).
93
- - **If no match**: proceed with creation.
94
- 5. Include a **Dedup Summary** section in the output report.
95
-
96
- Skip this check with `--skip-dedup`.
97
-
98
- ---
99
-
100
- ### Phase 3: Identify Work Item Type and Extract Test Source Content
101
-
102
- Apply the correct extraction strategy based on work item type:
103
-
104
- #### If type is `Bug` or `Ticket`:
105
-
106
- Primary source — **Repro Steps** (`Microsoft.VSTS.TCM.ReproSteps`):
107
- - Each distinct action sequence is a candidate test case
108
- - The repro steps define the *negative path* (what triggers the bug)
109
- - Derive the *positive/fix-verification path* by inverting the expected outcome
110
- - Also read: **System Info** (`Microsoft.VSTS.TCM.SystemInfo`), **Description**, **QA Notes** (`CIIScrum.QANotes`)
111
- - Check `Custom.Whatisexpectedtohappen` and `Custom.Whatisactuallyhappening` to anchor pass/fail assertions
112
-
113
- Secondary sources:
114
- - Comments for tester observations or specific scenarios to cover
115
- - Attachments for error data or sample inputs
116
-
117
- #### If type is `User Story`:
118
-
119
- Primary source — **Acceptance Criteria** (`Microsoft.VSTS.Common.AcceptanceCriteria`):
120
- - Each acceptance criterion (Given/When/Then or checklist) maps to one or more test cases
121
- - Also read: **Description** for context and implied behaviors
122
-
123
- Secondary sources:
124
- - Comments for clarifications, edge cases raised in refinement, or stakeholder scenarios
125
- - Attachments for wireframes described in text, sample data, or business rules documents
126
-
127
- #### If type is unrecognized or fields are empty:
128
-
129
- Fall back to **Description** as the primary source. Extract any stated behaviors, expected outcomes, or constraints. Note the fallback in the output.
130
-
131
- **Cross-reference with codebase map** (if available):
132
- - Match mentioned components/features against `TEST_SURFACE.md` entry points
133
- - Check `RISK_MAP.md` for risk level of affected areas
134
- - Use `API_CONTRACTS.md` for exact endpoint shapes if the work item mentions API behavior
135
- - Use `CODE_PATTERNS.md` to align test step language with project conventions
136
-
137
- ---
138
-
139
- ### Phase 4: Analyze and Design Test Cases
140
-
141
- Before creating anything in Azure DevOps, plan out all test cases:
142
-
143
- **For each distinct scenario identified, determine:**
144
-
145
- 1. **Test Case Title** concise action-oriented name (e.g., "Verify guest pass entry counter resets at midnight")
146
- 2. **Steps** — formatted as `{step action} | {expected result}` per step, using `|` as the delimiter
147
- 3. **Priority** — 1 (Critical), 2 (High), 3 (Medium), 4 (Low)
148
- 4. **Tags** — one or more of: `Smoke`, `Regression`, `Critical`, `AutomationCandidate`, `NeedsReview`
149
- 5. **Preconditions** required setup before executing the test
150
- 6. **Confidence** `Specified` or `Draft`
151
-
152
- **Minimum test case coverage per work item type:**
153
-
154
- | Scenario Type | Bug/Ticket | User Story |
155
- |---------------|-----------|------------|
156
- | Happy path (fix verified / AC met) | Required | Required per AC item |
157
- | Negative / error path | Required (original repro) | Where AC implies failure states |
158
- | Boundary / edge cases | If data-driven | If AC contains limits or conditions |
159
- | Boundary value triplets (n-1, n, n+1) | If limits detected | If AC contains limits/ranges |
160
- | Regression guard (related area) | Required | Required |
161
-
162
- #### Boundary Value Detection
163
-
164
- Scan all source content for **boundary keyword triggers**:
165
-
166
- > `max`, `min`, `limit`, `threshold`, `cap`, `ceiling`, `floor`, `range`, `between`, `up to`, `at most`, `at least`, `no more than`, `no fewer than`, `maximum`, `minimum`, `exactly`, `exceeds`, `boundary`
167
-
168
- When a trigger is found alongside a numeric value **N**:
169
-
170
- 1. **Generate three test cases** (the boundary triplet):
171
- - **N - 1** — just below the boundary
172
- - **N** — exactly at the boundary
173
- - **N + 1** — just above the boundary
174
- 2. Title them clearly: e.g., `"Verify entry limit at 99 (below threshold)"`, `"...at 100 (at threshold)"`, `"...at 101 (above threshold)"`.
175
- 3. Tag all three with `Regression`.
176
- 4. If the boundary is on a critical-path field (per `CRITICAL_PATHS.md` or keyword detection), also tag `Critical`.
177
-
178
- If the source mentions a range, generate boundary triplets for **both** ends.
179
-
180
- #### Tagging Rules
181
-
182
- | Tag | Assign when... |
183
- |-----|---------------|
184
- | `Smoke` | Verifies core, user-facing functionality that must work for the app to be usable at all. Limit to the most essential 1-2 cases per work item. |
185
- | `Regression` | Guards against the specific bug or behavior being re-introduced. Every fix-verification test for a Bug/Ticket should be tagged. For User Stories, tag tests covering AC that touches shared or high-traffic code paths. |
186
- | `Critical` | Covers functionality whose failure would directly impact revenue, security, data integrity, or legal compliance. **Also apply when critical keywords are detected** (see Keyword-Based Critical Tagging below). Apply conservatively. |
187
- | `AutomationCandidate` | Test has: (a) deterministic steps with no subjective judgment, (b) assertions based on concrete data/state, (c) no manual-only prerequisites. Advisory only — QA confirms. |
188
-
189
- **Do not assign Smoke to every test case.** Smoke tests are a small, fast-running set.
190
-
191
- #### Keyword-Based Critical Tagging
192
-
193
- Automatically tag as `Critical` when any of the following keywords appear in the source content:
194
-
195
- > `auth`, `authentication`, `login`, `password`, `OAuth`, `SSO`, `payment`, `billing`, `charge`, `invoice`, `PII`, `personal data`, `SSN`, `date of birth`, `security`, `encryption`, `token`, `certificate`, `data integrity`, `transaction`, `rollback`, `compliance`, `HIPAA`, `GDPR`, `SOC`, `audit`, `permission`, `role-based`, `access control`
196
-
197
- Cross-reference with `RISK_MAP.md` (if available) for additional risk-based tagging.
198
-
199
- #### Confidence Scoring
200
-
201
- | Confidence | Criteria | Behavior |
202
- |------------|----------|----------|
203
- | **Specified** | Source content explicitly describes the scenario, expected outcome, and data. | Create the TC normally. |
204
- | **Draft** | Scenario is implied or partially described — inferred from context or sparse source. | Prefix TC title with `[DRAFT]`. Add `NeedsReview` tag. Add final step: `"Review — this test case was auto-generated from sparse source material and requires QA validation before execution." | "QA has reviewed and confirmed or updated the steps."` |
205
-
206
- **Threshold**: If more than 50% of the source content fields are empty or contain fewer than 20 words, default all inferred TCs to Draft.
207
-
208
- #### Preconditions Block
209
-
210
- Every test case documents preconditions:
211
-
212
- | Field | Description | Example |
213
- |-------|-------------|--------|
214
- | **Required Role(s)** | User role(s) or permission level(s) needed | `Admin`, `Property Manager`, `Resident` |
215
- | **Application State** | System/feature state that must be true before step 1 | `User is logged in`, `Feature flag X is enabled` |
216
- | **Test Data** | Specific data that must exist or be created | `Resident account with active lease` |
217
- | **Environment** | Environment-specific requirements | `Staging`, `API key configured` |
218
-
219
- Prepend preconditions to the TC description field in Azure DevOps:
220
-
221
- ```
222
- **Preconditions**
223
- - Role(s): {roles}
224
- - State: {state}
225
- - Test Data: {data}
226
- - Environment: {env}
227
- ```
228
-
229
- If locator registry data is available, include relevant locator references in test steps for E2E-related scenarios.
230
-
231
- ---
232
-
233
- ### Phase 5: Create Test Cases in Azure DevOps
234
-
235
- **Dedup gate**: Before creating each TC, check against the registry from Phase 2b.
236
-
237
- For each planned test case, call `testplan_create_test_case` with:
238
-
239
- - `project`: the work item's project
240
- - `title`: the test case title — prefixed with `[DRAFT]` if confidence is Draft
241
- - `steps`: formatted as `1. {action}|{expected result}\n2. {action}|{expected result}` use `|` as delimiter. **Never pass XML or pre-formatted `<steps>` markup** the tool generates XML from plain-text format.
242
- - `priority`: numeric priority (1-4)
243
- - `iterationPath`: use `--iteration-path` override if provided, otherwise source work item's iteration path
244
- - `areaPath`: use `--area-path` override if provided, otherwise source work item's area path
245
-
246
- **After creating each test case:**
247
-
248
- 1. Call `wit_add_artifact_link` or `wit_work_items_link` to link the new TC to the source work item using link type `"tested by"`:
249
- ```
250
- source work item --[Tested By]--> test case
251
- ```
252
-
253
- 2. Call `wit_update_work_item` on the new TC to set `System.Tags` to semicolon-separated tags (e.g., `"Regression; Critical; AutomationCandidate"`).
254
- - Draft TCs always include `NeedsReview`.
255
-
256
- Create all test cases sequentially — capture each new TC ID before proceeding.
257
-
258
- ---
259
-
260
- ### Phase 6: Synthesize the Output Report
261
-
262
- Save the report to `ai-tasks/ticket-$ARGUMENTS/test-cases.md`.
263
-
264
- **Required document structure:**
265
-
266
- ```markdown
267
- # Test Cases: {work-item-id} — {Work Item Title}
268
-
269
- **Generated**: {current date}
270
- **Work Item**: [{work-item-id}]({azure-devops-url}) {type} | {state}
271
- **Assigned To**: {assigned-to}
272
- **Area Path**: {area path}
273
- **Iteration**: {iteration path}
274
- **Test Source**: {Repro Steps / Acceptance Criteria / Description (fallback)}
275
- **Pipeline Context**: Codebase map: {yes/no}, Locator registry: {yes/no}, Preferences: {yes/no}
276
-
277
- ---
278
-
279
- ## Source Analysis
280
-
281
- ### Work Item Summary
282
- {2-3 sentences describing the work item and what behavior needed to be tested.}
283
-
284
- ### Key Scenarios Identified
285
- {Bulleted list of distinct testable scenarios extracted before designing test cases.}
286
-
287
- ### Source Content Notes
288
- {Observations about quality/completeness of source material. Were repro steps/AC clear? Did comments add scenarios?}
289
-
290
- ### Codebase Context Used
291
- {If codebase map was available: list which documents were read and what context they provided. If not available: note that test cases were generated without codebase context.}
292
-
293
- ---
294
-
295
- ## Test Cases Created
296
-
297
- ### TC-{azure-devops-id}: {title}
298
-
299
- **Confidence**: `Specified` or `[DRAFT] — NeedsReview`
300
- **Tags**: `{Smoke}` · `{Regression}` · `{Critical}` · `{AutomationCandidate}` · `{NeedsReview}` *(show only tags that apply)*
301
- **Priority**: {1 – Critical / 2 – High / 3 – Medium / 4 – Low}
302
- **Linked To**: Work Item #{work-item-id} via *Tested By*
303
- **Azure DevOps ID**: {test-case-id}
304
-
305
- **Preconditions:**
306
- - **Role(s)**: {required roles or N/A}
307
- - **State**: {required application state or N/A}
308
- - **Test Data**: {required data or N/A}
309
- - **Environment**: {environment requirements or N/A}
310
-
311
- **Test Steps:**
312
-
313
- | # | Action | Expected Result |
314
- |---|--------|-----------------|
315
- | 1 | {action} | {expected result} |
316
- | 2 | {action} | {expected result} |
317
-
318
- {Repeat for each test case.}
319
-
320
- ---
321
-
322
- ## Tag Summary
323
-
324
- | Tag | Count | Test Case IDs |
325
- |-----|-------|---------------|
326
- | Smoke | {n} | {comma-separated IDs} |
327
- | Regression | {n} | {comma-separated IDs} |
328
- | Critical | {n} | {comma-separated IDs} |
329
- | AutomationCandidate | {n} | {comma-separated IDs} |
330
- | NeedsReview | {n} | {comma-separated IDs} |
331
-
332
- ---
333
-
334
- ## Dedup Summary
335
-
336
- | Planned Title | Skipped Reason | Existing TC |
337
- |---------------|---------------|-------------|
338
- | {title} | Duplicate of TC #{id} | #{id} — {state} |
339
-
340
- {If no duplicates: "No duplicates detected — all test cases were created."}
341
-
342
- ---
343
-
344
- ## Traceability
345
-
346
- All test cases linked to work item **#{work-item-id}** via *Tested By*.
347
-
348
- **Path Overrides Applied**: {If --area-path or --iteration-path provided, state them. Otherwise: "None — used source work item paths."}
349
- **Confidence Breakdown**: {n} Specified, {n} Draft (NeedsReview)
350
- **Boundary Triplets Generated**: {n} (from {n} detected boundaries)
351
- ```
352
-
353
- ---
354
-
355
- ### Phase 7: Attach Report to Source Work Item
356
-
357
- **If `ADO_MCP_AUTH_TOKEN` is set:**
358
-
359
- Upload `test-cases.md` as an attachment:
360
-
361
- ```bash
362
- # Step 1: Upload file
363
- ATTACHMENT_URL=$(curl -s \
364
- --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
365
- --header "Content-Type: application/octet-stream" \
366
- --request POST \
367
- --data-binary "@ai-tasks/ticket-$ARGUMENTS/test-cases.md" \
368
- "https://dev.azure.com/{org}/{project}/_apis/wit/attachments?fileName=test-cases.md&api-version=7.1" \
369
- | python3 -c "import sys,json; print(json.load(sys.stdin)['url'])")
370
-
371
- # Step 2: Link attachment to work item
372
- curl -s \
373
- --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
374
- --header "Content-Type: application/json-patch+json" \
375
- --request PATCH \
376
- --data "[{\"op\":\"add\",\"path\":\"/relations/-\",\"value\":{\"rel\":\"AttachedFile\",\"url\":\"${ATTACHMENT_URL}\",\"attributes\":{\"comment\":\"Generated test cases report\"}}}]" \
377
- "https://dev.azure.com/{org}/{project}/_apis/wit/workItems/$ARGUMENTS?api-version=7.1"
378
- ```
379
-
380
- **If `ADO_MCP_AUTH_TOKEN` is NOT set (fallback):**
381
-
382
- Write the full report as HTML to the work item's `Custom.QATestCasesReport` field via `wit_update_work_item`. Include all sections converted to HTML.
383
-
384
- Note in the final report which method was used.
385
-
386
- ---
387
-
388
- ## Final Report to User
389
-
390
- After completing all phases, provide:
391
-
392
- 1. Brief inline summary (2-3 sentences) of scenarios covered
393
- 2. Full path to generated file: `ai-tasks/ticket-{id}/test-cases.md`
394
- 3. Table of every created TC: ID, title, tags, confidence
395
- 4. Counts by tag: Smoke, Regression, Critical, AutomationCandidate, NeedsReview
396
- 5. Dedup summary: how many planned TCs were skipped
397
- 6. Confidence summary: Specified vs Draft counts
398
- 7. Boundary summary: how many boundary triplets generated
399
- 8. Pipeline context: which codebase map documents and locator registry data were used
400
- 9. Gaps or assumptions made
401
- 10. Path override confirmation (if used)
402
- 11. Report delivery confirmation (attached as file or written to custom field)
403
-
404
- $ARGUMENTS
1
+ # QA Create Test — Azure DevOps
2
+
3
+ Retrieve an Azure DevOps work item, analyze its content, and generate well-structured Test Cases directly in Azure DevOps using the ADO MCP tools. Each test case is tagged for test plan membership (Smoke, Regression, Critical) and linked back to the source work item for full traceability. Integrates with the QAA pipeline: reads codebase map, locator registry, and user preferences for context-aware test case generation.
4
+
5
+ ## ⚠ MANDATORY: How to Execute This Command
6
+
7
+ Execute these steps IN ORDER. Every step is mandatory. Do NOT improvise,
8
+ reorder, skip, or invent steps. This command talks to Azure DevOps via the ADO
9
+ MCP tools; it spawns no sub-agents.
10
+
11
+ 1. Resolve the work item ID from `$ARGUMENTS` — it is REQUIRED. If missing, ask
12
+ the user before doing anything else.
13
+ 2. PHASE 1 Read pipeline context: `CLAUDE.md`, `~/.claude/qaa/MY_PREFERENCES.md`
14
+ (if it exists), the codebase map (`.qa-output/codebase/`), and the locator
15
+ registry. If no codebase map and `--skip-map` was not passed, warn and continue.
16
+ 3. PHASE 2 Retrieve the work item (`wit_get_work_item` with relations), its
17
+ comments, and its attachments.
18
+ 4. PHASE 2b Deduplication check: build the registry of already-linked test
19
+ cases (unless `--skip-dedup`). This gate is mandatory before creating anything.
20
+ 5. PHASE 3-4 Identify the work item type, extract test-source content, and
21
+ design ALL test cases (titles, steps, priority, tags, preconditions,
22
+ confidence) BEFORE creating anything in ADO.
23
+ 6. PHASE 5 — Create the test cases in ADO: apply the dedup gate before each one,
24
+ link every new TC to the source work item via *Tested By*, and set its tags.
25
+ Pass steps as plain text with `|` delimiters NEVER XML / `<steps>` markup.
26
+ 7. PHASE 6-7 Synthesize the report to `ai-tasks/ticket-{id}/test-cases.md`,
27
+ attach it to the work item (or write to `Custom.QATestCasesReport` as a
28
+ fallback), then give the final summary to the user.
29
+
30
+ ### DO NOT
31
+ - Create any test case before the deduplication check (unless `--skip-dedup`)
32
+ - Skip linking each new TC via *Tested By*, or skip setting its tags
33
+ - Pass pre-formatted XML / `<steps>` to `testplan_create_test_case` (plain text + `|`)
34
+ - Overwrite unrelated work-item fields
35
+ - Invent steps not in this command, or reorder the phases
36
+
37
+ If you find yourself creating test cases before dedup, skipping the *Tested By*
38
+ links, or passing XML steps STOP and restart from step 1.
39
+
40
+ ## Usage
41
+
42
+ ```
43
+ /qa-create-test-ado <work-item-id> [--area-path=<path>] [--iteration-path=<path>] [--skip-map] [--skip-dedup] [--app-url <url>]
44
+ ```
45
+
46
+ ### Arguments
47
+
48
+ | Parameter | Purpose | Default |
49
+ |-----------|---------|---------|
50
+ | `<work-item-id>` | Azure DevOps work item ID to generate test cases from | Required |
51
+ | `--area-path=<path>` | Override area path for all created test artifacts | Source work item's area path |
52
+ | `--iteration-path=<path>` | Override iteration path for all created test artifacts | Source work item's iteration path |
53
+ | `--skip-map` | Skip codebase map check and proceed without project context | false |
54
+ | `--skip-dedup` | Skip deduplication check against existing linked test cases | false |
55
+ | `--app-url <url>` | URL of running application for locator extraction via Playwright MCP | auto-detect |
56
+
57
+ ## What It Produces
58
+
59
+ - Test Cases created directly in Azure DevOps (via `testplan_create_test_case`)
60
+ - Test Cases linked to source work item via *Tested By* relationship
61
+ - Tags applied: `Smoke`, `Regression`, `Critical`, `AutomationCandidate`, `NeedsReview`
62
+ - `ai-tasks/ticket-{id}/test-cases.md` structured report
63
+ - Report attached to work item (if `ADO_MCP_AUTH_TOKEN` is set) or written to `Custom.QATestCasesReport` field (fallback)
64
+
65
+ ---
66
+
67
+ ## Process
68
+
69
+ ### Phase 1: Read Pipeline Context
70
+
71
+ Before retrieving the work item, read QAA pipeline artifacts for context-aware generation.
72
+
73
+ 1. **Read `CLAUDE.md`** — POM rules, locator tiers, assertion rules, naming conventions, quality gates, test spec rules.
74
+
75
+ 2. **Read user preferences** — `~/.claude/qaa/MY_PREFERENCES.md` (if exists). User overrides win over defaults.
76
+
77
+ 3. **Check for codebase map** (`.qa-output/codebase/`):
78
+ - Look for: `CODE_PATTERNS.md`, `API_CONTRACTS.md`, `TEST_SURFACE.md`, `TESTABILITY.md`, `RISK_MAP.md`, `CRITICAL_PATHS.md`
79
+ - If at least 2 exist: read them all for project context (naming conventions, API shapes, testable surfaces, risk areas).
80
+ - If NONE exist and `--skip-map` not passed: warn the user that test cases will lack project context, suggest running `/qa-map` first. Continue anyway (ADO test cases are higher-level than code-level tests).
81
+
82
+ 4. **Check locator registry** — `.qa-output/locators/LOCATOR_REGISTRY.md` (if exists):
83
+ - If locators exist for pages related to the work item's feature: reference them in test step expected results (e.g., "Verify element `[data-testid='login-submit-btn']` is visible").
84
+ - If `--app-url` provided and locators missing: use Playwright MCP to extract locators from the live app before designing test steps:
85
+ ```
86
+ mcp__playwright__browser_navigate({ url: "{app_url}/{feature_path}" })
87
+ mcp__playwright__browser_snapshot()
88
+ ```
89
+ - Write extracted locators to `.qa-output/locators/{feature}.locators.md` and update the registry.
90
+
91
+ ---
92
+
93
+ ### Phase 2: Retrieve the Work Item
94
+
95
+ Use `wit_get_work_item` with `expand: "relations"` to fetch the full work item:
96
+
97
+ - Capture: **title**, **type** (`Bug`, `User Story`, `Ticket`), **state**, **assigned-to**, **area path**, **iteration path**
98
+ - Capture all relevant content fields based on type (see Phase 3)
99
+ - Note the project for all subsequent calls
100
+
101
+ **Also retrieve comments** using `wit_list_work_item_comments`:
102
+
103
+ - Read all comments in chronological order
104
+ - Look for: acceptance criteria added in comments, QA notes, scope clarifications, tester feedback, or any conditions of satisfaction mentioned informally
105
+ - These often contain implied test cases not captured in the formal fields
106
+
107
+ **Also check attachments** from the relations list (entries where `rel` equals `AttachedFile`):
108
+
109
+ - Filter to `.csv` and `.txt` files (case-insensitive) by inspecting `attributes.name`
110
+ - If found, download via:
111
+ ```bash
112
+ curl -s --user ":{AZURE_DEVOPS_PAT}" "{attachment-url}"
113
+ ```
114
+ - Read content for test data, expected values, error logs, or sample datasets that define expected behavior
115
+
116
+ ---
117
+
118
+ ### Phase 2b: Deduplication Check — Query Existing Test Cases
119
+
120
+ Before generating any new test cases, check whether the source work item already has linked test cases to prevent duplicates.
121
+
122
+ 1. Inspect the relations returned in Phase 2 — filter for link type `"Microsoft.VSTS.Common.TestedBy-Forward"` (i.e., *Tested By* links).
123
+ 2. For each linked test case ID found, call `wit_get_work_item` to retrieve its **title** and **state**.
124
+ 3. Build an **existing TC registry** a list of `{ id, title, state }` for all currently linked test cases.
125
+ 4. In Phase 5, before calling `testplan_create_test_case` for each planned TC, compare its title (normalized: lowercase, trimmed) against every title in the registry.
126
+ - **If match found** and existing TC is in state `Design`, `Ready`, or `Closed`: skip creation, log `"Skipped — duplicate of TC #{id}"`.
127
+ - **If match found** but existing TC is in state `Removed`: create the new TC anyway (the old one was intentionally discarded).
128
+ - **If no match**: proceed with creation.
129
+ 5. Include a **Dedup Summary** section in the output report.
130
+
131
+ Skip this check with `--skip-dedup`.
132
+
133
+ ---
134
+
135
+ ### Phase 3: Identify Work Item Type and Extract Test Source Content
136
+
137
+ Apply the correct extraction strategy based on work item type:
138
+
139
+ #### If type is `Bug` or `Ticket`:
140
+
141
+ Primary source **Repro Steps** (`Microsoft.VSTS.TCM.ReproSteps`):
142
+ - Each distinct action sequence is a candidate test case
143
+ - The repro steps define the *negative path* (what triggers the bug)
144
+ - Derive the *positive/fix-verification path* by inverting the expected outcome
145
+ - Also read: **System Info** (`Microsoft.VSTS.TCM.SystemInfo`), **Description**, **QA Notes** (`CIIScrum.QANotes`)
146
+ - Check `Custom.Whatisexpectedtohappen` and `Custom.Whatisactuallyhappening` to anchor pass/fail assertions
147
+
148
+ Secondary sources:
149
+ - Comments for tester observations or specific scenarios to cover
150
+ - Attachments for error data or sample inputs
151
+
152
+ #### If type is `User Story`:
153
+
154
+ Primary source **Acceptance Criteria** (`Microsoft.VSTS.Common.AcceptanceCriteria`):
155
+ - Each acceptance criterion (Given/When/Then or checklist) maps to one or more test cases
156
+ - Also read: **Description** for context and implied behaviors
157
+
158
+ Secondary sources:
159
+ - Comments for clarifications, edge cases raised in refinement, or stakeholder scenarios
160
+ - Attachments for wireframes described in text, sample data, or business rules documents
161
+
162
+ #### If type is unrecognized or fields are empty:
163
+
164
+ Fall back to **Description** as the primary source. Extract any stated behaviors, expected outcomes, or constraints. Note the fallback in the output.
165
+
166
+ **Cross-reference with codebase map** (if available):
167
+ - Match mentioned components/features against `TEST_SURFACE.md` entry points
168
+ - Check `RISK_MAP.md` for risk level of affected areas
169
+ - Use `API_CONTRACTS.md` for exact endpoint shapes if the work item mentions API behavior
170
+ - Use `CODE_PATTERNS.md` to align test step language with project conventions
171
+
172
+ ---
173
+
174
+ ### Phase 4: Analyze and Design Test Cases
175
+
176
+ Before creating anything in Azure DevOps, plan out all test cases:
177
+
178
+ **For each distinct scenario identified, determine:**
179
+
180
+ 1. **Test Case Title** — concise action-oriented name (e.g., "Verify guest pass entry counter resets at midnight")
181
+ 2. **Steps** — formatted as `{step action} | {expected result}` per step, using `|` as the delimiter
182
+ 3. **Priority** 1 (Critical), 2 (High), 3 (Medium), 4 (Low)
183
+ 4. **Tags** — one or more of: `Smoke`, `Regression`, `Critical`, `AutomationCandidate`, `NeedsReview`
184
+ 5. **Preconditions** required setup before executing the test
185
+ 6. **Confidence** `Specified` or `Draft`
186
+
187
+ **Minimum test case coverage per work item type:**
188
+
189
+ | Scenario Type | Bug/Ticket | User Story |
190
+ |---------------|-----------|------------|
191
+ | Happy path (fix verified / AC met) | Required | Required per AC item |
192
+ | Negative / error path | Required (original repro) | Where AC implies failure states |
193
+ | Boundary / edge cases | If data-driven | If AC contains limits or conditions |
194
+ | Boundary value triplets (n-1, n, n+1) | If limits detected | If AC contains limits/ranges |
195
+ | Regression guard (related area) | Required | Required |
196
+
197
+ #### Boundary Value Detection
198
+
199
+ Scan all source content for **boundary keyword triggers**:
200
+
201
+ > `max`, `min`, `limit`, `threshold`, `cap`, `ceiling`, `floor`, `range`, `between`, `up to`, `at most`, `at least`, `no more than`, `no fewer than`, `maximum`, `minimum`, `exactly`, `exceeds`, `boundary`
202
+
203
+ When a trigger is found alongside a numeric value **N**:
204
+
205
+ 1. **Generate three test cases** (the boundary triplet):
206
+ - **N - 1** just below the boundary
207
+ - **N** — exactly at the boundary
208
+ - **N + 1** — just above the boundary
209
+ 2. Title them clearly: e.g., `"Verify entry limit at 99 (below threshold)"`, `"...at 100 (at threshold)"`, `"...at 101 (above threshold)"`.
210
+ 3. Tag all three with `Regression`.
211
+ 4. If the boundary is on a critical-path field (per `CRITICAL_PATHS.md` or keyword detection), also tag `Critical`.
212
+
213
+ If the source mentions a range, generate boundary triplets for **both** ends.
214
+
215
+ #### Tagging Rules
216
+
217
+ | Tag | Assign when... |
218
+ |-----|---------------|
219
+ | `Smoke` | Verifies core, user-facing functionality that must work for the app to be usable at all. Limit to the most essential 1-2 cases per work item. |
220
+ | `Regression` | Guards against the specific bug or behavior being re-introduced. Every fix-verification test for a Bug/Ticket should be tagged. For User Stories, tag tests covering AC that touches shared or high-traffic code paths. |
221
+ | `Critical` | Covers functionality whose failure would directly impact revenue, security, data integrity, or legal compliance. **Also apply when critical keywords are detected** (see Keyword-Based Critical Tagging below). Apply conservatively. |
222
+ | `AutomationCandidate` | Test has: (a) deterministic steps with no subjective judgment, (b) assertions based on concrete data/state, (c) no manual-only prerequisites. Advisory only — QA confirms. |
223
+
224
+ **Do not assign Smoke to every test case.** Smoke tests are a small, fast-running set.
225
+
226
+ #### Keyword-Based Critical Tagging
227
+
228
+ Automatically tag as `Critical` when any of the following keywords appear in the source content:
229
+
230
+ > `auth`, `authentication`, `login`, `password`, `OAuth`, `SSO`, `payment`, `billing`, `charge`, `invoice`, `PII`, `personal data`, `SSN`, `date of birth`, `security`, `encryption`, `token`, `certificate`, `data integrity`, `transaction`, `rollback`, `compliance`, `HIPAA`, `GDPR`, `SOC`, `audit`, `permission`, `role-based`, `access control`
231
+
232
+ Cross-reference with `RISK_MAP.md` (if available) for additional risk-based tagging.
233
+
234
+ #### Confidence Scoring
235
+
236
+ | Confidence | Criteria | Behavior |
237
+ |------------|----------|----------|
238
+ | **Specified** | Source content explicitly describes the scenario, expected outcome, and data. | Create the TC normally. |
239
+ | **Draft** | Scenario is implied or partially described — inferred from context or sparse source. | Prefix TC title with `[DRAFT]`. Add `NeedsReview` tag. Add final step: `"Review — this test case was auto-generated from sparse source material and requires QA validation before execution." | "QA has reviewed and confirmed or updated the steps."` |
240
+
241
+ **Threshold**: If more than 50% of the source content fields are empty or contain fewer than 20 words, default all inferred TCs to Draft.
242
+
243
+ #### Preconditions Block
244
+
245
+ Every test case documents preconditions:
246
+
247
+ | Field | Description | Example |
248
+ |-------|-------------|--------|
249
+ | **Required Role(s)** | User role(s) or permission level(s) needed | `Admin`, `Property Manager`, `Resident` |
250
+ | **Application State** | System/feature state that must be true before step 1 | `User is logged in`, `Feature flag X is enabled` |
251
+ | **Test Data** | Specific data that must exist or be created | `Resident account with active lease` |
252
+ | **Environment** | Environment-specific requirements | `Staging`, `API key configured` |
253
+
254
+ Prepend preconditions to the TC description field in Azure DevOps:
255
+
256
+ ```
257
+ **Preconditions**
258
+ - Role(s): {roles}
259
+ - State: {state}
260
+ - Test Data: {data}
261
+ - Environment: {env}
262
+ ```
263
+
264
+ If locator registry data is available, include relevant locator references in test steps for E2E-related scenarios.
265
+
266
+ ---
267
+
268
+ ### Phase 5: Create Test Cases in Azure DevOps
269
+
270
+ **Dedup gate**: Before creating each TC, check against the registry from Phase 2b.
271
+
272
+ For each planned test case, call `testplan_create_test_case` with:
273
+
274
+ - `project`: the work item's project
275
+ - `title`: the test case title prefixed with `[DRAFT]` if confidence is Draft
276
+ - `steps`: formatted as `1. {action}|{expected result}\n2. {action}|{expected result}` — use `|` as delimiter. **Never pass XML or pre-formatted `<steps>` markup** — the tool generates XML from plain-text format.
277
+ - `priority`: numeric priority (1-4)
278
+ - `iterationPath`: use `--iteration-path` override if provided, otherwise source work item's iteration path
279
+ - `areaPath`: use `--area-path` override if provided, otherwise source work item's area path
280
+
281
+ **After creating each test case:**
282
+
283
+ 1. Call `wit_add_artifact_link` or `wit_work_items_link` to link the new TC to the source work item using link type `"tested by"`:
284
+ ```
285
+ source work item --[Tested By]--> test case
286
+ ```
287
+
288
+ 2. Call `wit_update_work_item` on the new TC to set `System.Tags` to semicolon-separated tags (e.g., `"Regression; Critical; AutomationCandidate"`).
289
+ - Draft TCs always include `NeedsReview`.
290
+
291
+ Create all test cases sequentially capture each new TC ID before proceeding.
292
+
293
+ ---
294
+
295
+ ### Phase 6: Synthesize the Output Report
296
+
297
+ Save the report to `ai-tasks/ticket-$ARGUMENTS/test-cases.md`.
298
+
299
+ **Required document structure:**
300
+
301
+ ```markdown
302
+ # Test Cases: {work-item-id} {Work Item Title}
303
+
304
+ **Generated**: {current date}
305
+ **Work Item**: [{work-item-id}]({azure-devops-url}) — {type} | {state}
306
+ **Assigned To**: {assigned-to}
307
+ **Area Path**: {area path}
308
+ **Iteration**: {iteration path}
309
+ **Test Source**: {Repro Steps / Acceptance Criteria / Description (fallback)}
310
+ **Pipeline Context**: Codebase map: {yes/no}, Locator registry: {yes/no}, Preferences: {yes/no}
311
+
312
+ ---
313
+
314
+ ## Source Analysis
315
+
316
+ ### Work Item Summary
317
+ {2-3 sentences describing the work item and what behavior needed to be tested.}
318
+
319
+ ### Key Scenarios Identified
320
+ {Bulleted list of distinct testable scenarios extracted before designing test cases.}
321
+
322
+ ### Source Content Notes
323
+ {Observations about quality/completeness of source material. Were repro steps/AC clear? Did comments add scenarios?}
324
+
325
+ ### Codebase Context Used
326
+ {If codebase map was available: list which documents were read and what context they provided. If not available: note that test cases were generated without codebase context.}
327
+
328
+ ---
329
+
330
+ ## Test Cases Created
331
+
332
+ ### TC-{azure-devops-id}: {title}
333
+
334
+ **Confidence**: `Specified` or `[DRAFT] — NeedsReview`
335
+ **Tags**: `{Smoke}` · `{Regression}` · `{Critical}` · `{AutomationCandidate}` · `{NeedsReview}` *(show only tags that apply)*
336
+ **Priority**: {1 Critical / 2 High / 3 – Medium / 4 – Low}
337
+ **Linked To**: Work Item #{work-item-id} via *Tested By*
338
+ **Azure DevOps ID**: {test-case-id}
339
+
340
+ **Preconditions:**
341
+ - **Role(s)**: {required roles or N/A}
342
+ - **State**: {required application state or N/A}
343
+ - **Test Data**: {required data or N/A}
344
+ - **Environment**: {environment requirements or N/A}
345
+
346
+ **Test Steps:**
347
+
348
+ | # | Action | Expected Result |
349
+ |---|--------|-----------------|
350
+ | 1 | {action} | {expected result} |
351
+ | 2 | {action} | {expected result} |
352
+
353
+ {Repeat for each test case.}
354
+
355
+ ---
356
+
357
+ ## Tag Summary
358
+
359
+ | Tag | Count | Test Case IDs |
360
+ |-----|-------|---------------|
361
+ | Smoke | {n} | {comma-separated IDs} |
362
+ | Regression | {n} | {comma-separated IDs} |
363
+ | Critical | {n} | {comma-separated IDs} |
364
+ | AutomationCandidate | {n} | {comma-separated IDs} |
365
+ | NeedsReview | {n} | {comma-separated IDs} |
366
+
367
+ ---
368
+
369
+ ## Dedup Summary
370
+
371
+ | Planned Title | Skipped Reason | Existing TC |
372
+ |---------------|---------------|-------------|
373
+ | {title} | Duplicate of TC #{id} | #{id} — {state} |
374
+
375
+ {If no duplicates: "No duplicates detected — all test cases were created."}
376
+
377
+ ---
378
+
379
+ ## Traceability
380
+
381
+ All test cases linked to work item **#{work-item-id}** via *Tested By*.
382
+
383
+ **Path Overrides Applied**: {If --area-path or --iteration-path provided, state them. Otherwise: "None — used source work item paths."}
384
+ **Confidence Breakdown**: {n} Specified, {n} Draft (NeedsReview)
385
+ **Boundary Triplets Generated**: {n} (from {n} detected boundaries)
386
+ ```
387
+
388
+ ---
389
+
390
+ ### Phase 7: Attach Report to Source Work Item
391
+
392
+ **If `ADO_MCP_AUTH_TOKEN` is set:**
393
+
394
+ Upload `test-cases.md` as an attachment:
395
+
396
+ ```bash
397
+ # Step 1: Upload file
398
+ ATTACHMENT_URL=$(curl -s \
399
+ --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
400
+ --header "Content-Type: application/octet-stream" \
401
+ --request POST \
402
+ --data-binary "@ai-tasks/ticket-$ARGUMENTS/test-cases.md" \
403
+ "https://dev.azure.com/{org}/{project}/_apis/wit/attachments?fileName=test-cases.md&api-version=7.1" \
404
+ | python3 -c "import sys,json; print(json.load(sys.stdin)['url'])")
405
+
406
+ # Step 2: Link attachment to work item
407
+ curl -s \
408
+ --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
409
+ --header "Content-Type: application/json-patch+json" \
410
+ --request PATCH \
411
+ --data "[{\"op\":\"add\",\"path\":\"/relations/-\",\"value\":{\"rel\":\"AttachedFile\",\"url\":\"${ATTACHMENT_URL}\",\"attributes\":{\"comment\":\"Generated test cases report\"}}}]" \
412
+ "https://dev.azure.com/{org}/{project}/_apis/wit/workItems/$ARGUMENTS?api-version=7.1"
413
+ ```
414
+
415
+ **If `ADO_MCP_AUTH_TOKEN` is NOT set (fallback):**
416
+
417
+ Write the full report as HTML to the work item's `Custom.QATestCasesReport` field via `wit_update_work_item`. Include all sections converted to HTML.
418
+
419
+ Note in the final report which method was used.
420
+
421
+ ---
422
+
423
+ ## Final Report to User
424
+
425
+ After completing all phases, provide:
426
+
427
+ 1. Brief inline summary (2-3 sentences) of scenarios covered
428
+ 2. Full path to generated file: `ai-tasks/ticket-{id}/test-cases.md`
429
+ 3. Table of every created TC: ID, title, tags, confidence
430
+ 4. Counts by tag: Smoke, Regression, Critical, AutomationCandidate, NeedsReview
431
+ 5. Dedup summary: how many planned TCs were skipped
432
+ 6. Confidence summary: Specified vs Draft counts
433
+ 7. Boundary summary: how many boundary triplets generated
434
+ 8. Pipeline context: which codebase map documents and locator registry data were used
435
+ 9. Gaps or assumptions made
436
+ 10. Path override confirmation (if used)
437
+ 11. Report delivery confirmation (attached as file or written to custom field)
438
+
439
+ $ARGUMENTS