qaa-agent 1.9.0 → 1.9.1

package/CHANGELOG.md

@@ -3,6 +3,15 @@
 
 All notable changes to QAA (QA Automation Agent) are documented here.
 
+## [1.9.1] - 2026-04-27
+
+### Added
+- **`/qa-test-report` command** — generates a per-ticket QA execution summary and appends it to the Azure DevOps work item's `Custom.QATestCasesReport` field.
+  - Resolves the work item and its linked test cases (TestedBy-Forward relations)
+  - Pulls test case execution status via REST `/_apis/test/points` (using `ADO_MCP_AUTH_TOKEN` env var as PAT, Basic auth — same pattern as `/qa-create-test --ado`), or falls back to a manual prompt when no run result exists or the token is not set
+  - Renders a markdown report in chat (for review) and an HTML report appended to the ADO field — preserving prior content with a blank-line separator (no local file is written)
+  - Smoke-tested end-to-end (manual mode, all passed) — field write and ADO render verified
+
 ## [1.9.0] - 2026-04-24
 
 ### Added

package/commands/qa-test-report.md

@@ -0,0 +1,219 @@
+# QA Test Report
+
+Generate a per-ticket QA execution summary and append it to the Azure DevOps work item's `Custom.QATestCasesReport` field. For each test case linked to the work item, pull the execution status from ADO Test Runs — or prompt the user when ADO has no run result — then render a bulleted "Tested Scenarios" list (with a separate "Failures" section when anything failed).
+
+## Usage
+
+```
+/qa-test-report <work-item-id>
+```
+
+### Arguments
+
+| Parameter | Purpose | Default |
+|-----------|---------|---------|
+| `<work-item-id>` | Azure DevOps work item ID (Bug, Feature, User Story, Ticket) whose linked test cases you want reported | Required — prompt the user if missing |
+
+## What It Produces
+
+- Markdown report printed to chat (for user review)
+- HTML report **appended** to the work item's `Custom.QATestCasesReport` field, separated from prior content by a blank line
+- No local file is written
+
+---
+
+## Process
+
+### Phase 1 — Resolve the Work Item
+
+1. If `$ARGUMENTS` is empty or missing a work item ID, ask the user: *"Which Azure DevOps work item ID should I build the QA Test Cases report for?"* Wait for the answer before proceeding.
+2. Call `wit_get_work_item` with `expand: "relations"` for the resolved ID.
+   - Capture: **title**, **type** (`Bug`, `Feature`, `User Story`, `Ticket`), **state**, **area path**, **iteration path**.
+   - For context used in Phase 4, capture type-appropriate content fields:
+     - `Bug` / `Ticket`: **Repro Steps** (`Microsoft.VSTS.TCM.ReproSteps`), **System Info** (`Microsoft.VSTS.TCM.SystemInfo`), **Description**, **QA Notes** (`CIIScrum.QANotes`), **What is expected to happen** (`Custom.Whatisexpectedtohappen`), **What is actually happening** (`Custom.Whatisactuallyhappening`).
+     - `User Story`: **Acceptance Criteria** (`Microsoft.VSTS.Common.AcceptanceCriteria`), **Description**.
+     - `Feature`: **Description**, **Acceptance Criteria** if present.
+   - Note the project for all subsequent ADO calls.
+3. Also call `wit_list_work_item_comments` — comments often contain fail reasons or tester observations referenced in Phase 3 fallbacks.
+
+---
+
+### Phase 2 — Resolve the Test Case List
+
+Source-resolution order:
+
+1. **Check for a local file** at `ai-tasks/ticket-{id}/test-cases.md` (produced by `/qa-create-test --ado`). If it exists, parse the TC IDs it contains — this is a hint list only.
+2. **Always query ADO** — inspect the relations returned in Phase 1, filter for link type `"Microsoft.VSTS.Common.TestedBy-Forward"` (*Tested By*), and build the authoritative list of linked TC IDs.
+3. **If both exist and disagree**: trust **ADO**. Log the discrepancy in chat as an FYI (e.g., `"TC#4521 was in local file but not linked in ADO — skipped"` or `"TC#4530 is linked in ADO but missing from local file — included"`) but proceed with the ADO list.
+4. **If no TCs are found in ADO** (and the local file has none either): print `"No test cases linked to work item #{id} — nothing to report."` and stop.
+
+For every TC ID in the final list, call `wit_get_work_item` with `expand: "relations"` to capture: **title**, **state**, **steps** (`Microsoft.VSTS.TCM.Steps`), **priority**, **tags**, and any linked runs.
+
+---
+
+### Phase 3 — Resolve Execution Status Per Test Case
+
+For each TC, determine a status of **Passed**, **Failed**, or **Blocked** using this order.
+
+**Step 1 — Fetch Test Point outcomes via ADO REST (formal path).**
+
+The ADO MCP surface does not expose Test Point outcomes, but the REST API does. Use the same auth pattern as `/qa-create-test --ado`: the `ADO_MCP_AUTH_TOKEN` env var as a PAT, sent via Basic auth.
+
+1. Derive **org** and **project** from the work item context captured in Phase 1 (not hardcoded). The work-item URL / project name on the `wit_get_work_item` response gives you both.
+2. If `ADO_MCP_AUTH_TOKEN` is **not set**, skip directly to Step 2 (and note in the final report: *"Auto-status skipped — ADO_MCP_AUTH_TOKEN not set."*).
+3. Otherwise, call the Test Points REST endpoint once per run with the list of linked TC IDs:
+
+   ```bash
+   curl -s \
+     --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
+     --header "Content-Type: application/json" \
+     --request POST \
+     --data '{"pointsFilter":{"testcaseIds":[<id1>,<id2>,...]}}' \
+     "https://dev.azure.com/{org}/{project}/_apis/test/points?api-version=7.1"
+   ```
+
+4. Parse the response. For each returned point you get: `testCase.id`, `testPlan.id`, `suite.id`, `outcome`, `lastTestRun.id`, `lastResultDetails.dateCompleted`, `lastResultDetails.runBy`.
+5. **If a TC has multiple points** (e.g., it lives in several suites or runs on multiple configurations), pick the point with the **most recent `lastResultDetails.dateCompleted`**.
+6. Map `outcome` (case-insensitive) to our statuses:
+   - `passed` → **Passed**
+   - `failed` → **Failed**
+   - `blocked` → **Blocked**
+   - `notExecuted` / `none` / `notApplicable` / `paused` / `inProgress` / `warning` / `error` → treat as missing, fall through to Step 2 for this TC
+7. **If the call fails** (non-2xx, network error, token rejected): log the failure briefly in chat, skip to Step 2 for all TCs, and note in the final report that auto-status was unavailable.
+
+**Step 2 — Ask the user** (per TC that didn't resolve in Step 1). Show the TC ID and its synthesized scenario description (from Phase 4) for context:
+
+```
+TC #{id} — {scenario description}
+Status? [Passed / Failed / Blocked]
+```
+
+Accept case-insensitive input. Re-prompt if the answer is not one of the three.
+
+**Step 3 — If the resolved status is Failed**, capture a reason:
+
+1. Check the Test Run data first — when Step 1 returned an outcome, also fetch the Test Result for `lastTestRun.id` to read `errorMessage`/`comment`:
+   ```bash
+   curl -s \
+     --header "Authorization: Basic $(echo -n :${ADO_MCP_AUTH_TOKEN} | base64)" \
+     "https://dev.azure.com/{org}/{project}/_apis/test/Runs/{runId}/results?api-version=7.1"
+   ```
+   If the result has a non-empty `errorMessage` or `comment`, use that as the reason.
+2. Also check the TC's own comments (`wit_list_work_item_comments` for the TC ID) and the parent work item's comments for any line mentioning this TC.
+3. If no reason is found, ask the user:
+   ```
+   TC #{id} failed — what was the reason?
+   ```
+   Accept a free-text one-liner.
+
+Keep the per-TC answers (status, reason, and auto/manual source) in memory; do not write them anywhere intermediate.
+
+---
+
+### Phase 4 — Synthesize Scenario Descriptions
+
+For each TC, produce a **readable one-line scenario description** for the bullet list. Do **not** copy the raw TC title when it is terse or cryptic — TC titles are often shorthand.
+
+Build the description from:
+- TC **title** (starting point)
+- TC **steps** (each step's action + expected result — gives you what was actually verified)
+- Parent work item **description / repro steps / acceptance criteria** (gives you the user-facing phrasing of *what should be true after the fix*)
+
+Write the description in the voice of the tested outcome, not the test action. Examples:
+
+| Raw TC title | Good scenario description |
+|---|---|
+| `Verify PH row display logic` | `$0 Previous Balance row no longer appears in Payment History` |
+| `Check login 401 scenario` | `Invalid credentials return a 401 with an error banner` |
+| `Regression: limit 100` | `Entry limit enforced at 100 — the 101st entry is rejected` |
+
+Keep bullets short — one line, past-tense or assertive present-tense, user-visible wording.
+
+---
+
+### Phase 5 — Build the Report
+
+**Markdown version (for chat output):**
+
+```markdown
+Tested Scenarios
+
+- ✅ {scenario}
+- ✅ {scenario}
+- ❌ {scenario}
+- ⚠️ {scenario}
+
+Failures
+
+- ❌ {scenario}: {reason}
+```
+
+Rules:
+- Status emoji: **Passed → ✅**, **Failed → ❌**, **Blocked → ⚠️**
+- Include the `Failures` section **only if at least one TC failed**. One bullet per failure with the captured reason appended after `:`.
+- Do **not** include a "Result" section.
+- Do **not** include TC IDs, priority, tags, or counts in the bullets.
+- Keep order stable — preserve the order TCs appeared in the ADO linked list.
+
+**HTML version (for the ADO field):**
+
+Render with the minimum markup needed for ADO's rich-text field to display bullets and line breaks:
+
+```html
+<b>Tested Scenarios</b><br>
+<ul>
+  <li>✅ {scenario}</li>
+  <li>✅ {scenario}</li>
+  <li>❌ {scenario}</li>
+  <li>⚠️ {scenario}</li>
+</ul>
+<b>Failures</b><br>
+<ul>
+  <li>❌ {scenario}: {reason}</li>
+</ul>
+```
+
+Rules:
+- Escape scenario text for HTML (`&` → `&amp;`, `<` → `&lt;`, `>` → `&gt;`).
+- Emit the `<b>Failures</b>…` block **only if at least one TC failed**.
+- No inline styles, no classes, no wrapping `<div>`.
+
+---
+
+### Phase 6 — Append to the ADO Field
+
+1. Re-read the current `Custom.QATestCasesReport` value for the work item (it may contain prior runs you must preserve).
+2. Build the new field value:
+   - If the existing value is empty or whitespace: new value = the HTML report from Phase 5.
+   - Otherwise: new value = existing content + `<br><br>` + the HTML report from Phase 5.
+3. Call `wit_update_work_item` setting `Custom.QATestCasesReport` to the new value. Pass only this field — do not include any others in the update.
+
+Do **not** overwrite other fields. Do **not** create a local file.
+
+---
+
+### Phase 7 — Present to User
+
+Print the **markdown** version of the report to the user in chat. After the report, print a one-line confirmation:
+
+```
+Appended to work item #{id} → Custom.QATestCasesReport.
+```
+
+If any scenarios required a manual status answer or a manual failure reason during Phase 3, append a short note listing how many:
+
+```
+{n} status(es) entered manually · {m} failure reason(s) entered manually.
+```
+
+If the TC list had any local-vs-ADO discrepancies (Phase 2), append a short note listing them.
+
+---
+
+## Notes
+
+- This skill does **not** create test cases. Use `/qa-create-test --ado <id>` for that.
+- This skill is read-mostly on the ADO side — it only writes to `Custom.QATestCasesReport`. It does not change TC state, does not create Test Runs, and does not modify links.
+- Scenario descriptions are synthesized, not verbatim TC titles — review the chat output before trusting the field content.
+
+$ARGUMENTS

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "qaa-agent",
-  "version": "1.9.0",
+  "version": "1.9.1",
   "description": "QA Automation Agent for Claude Code — multi-agent pipeline that analyzes repos, generates tests, validates, and creates PRs",
   "bin": {
     "qaa-agent": "./bin/install.cjs"