@chris1807/claude-kit 2.3.6 → 2.3.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -7
- package/bin/cli.js +10 -1
- package/package.json +1 -1
- package/templates/agents/global/qa.md +135 -15
- package/templates/commands/autopilot.md +188 -0
- package/templates/commands/qa.md +100 -26
- package/templates/commands/repro.md +285 -0
- package/templates/hooks/uat-reminder.sh +1 -0
- package/templates/infrastructure/CLAUDE-WORKFLOW.md +86 -23
package/templates/commands/qa.md
CHANGED
|
@@ -1,10 +1,12 @@
|
|
|
1
|
-
Regression-test a work item in a real browser against a running environment. Usage: `/qa <work-item-id> [environment] [--role=<role>] [--screens=/a,/b]`
|
|
1
|
+
Regression-test a work item in a real browser against a running environment. Usage: `/qa <work-item-id> [environment] [--role=<role>] [--as=<persona-or-user>] [--as-role=<role>] [--screens=/a,/b]`
|
|
2
2
|
|
|
3
3
|
Parse `$ARGUMENTS` to extract the work item ID. Accept formats like `AB#1234`, `#1234`, or just `1234`. If no ID is provided, report the error and stop.
|
|
4
4
|
|
|
5
5
|
- **Environment**: optional — `local`, `dev`, `test`, or `staging`. Defaults to `dev`.
|
|
6
6
|
- **`--role=<role>`**: optional — which test account to sign in as. Defaults to the row marked `Default` in the Environment URLs table, else the first row.
|
|
7
|
-
- **`--
|
|
7
|
+
- **`--as=<persona-or-user>`**: optional — after signing in, **impersonate** this user, using the app's own impersonation feature. A name from the project's persona table picks a QA test user. Anything else — an email, a username, or a name — is searched for literally, as an **ad-hoc user**. Requires an `**Impersonation:**` block in the project's `## Environment URLs`.
|
|
8
|
+
- **`--as-role=<role>`**: optional — the role to impersonate a multi-role ad-hoc user as. A persona's role comes from its row.
|
|
9
|
+
- **`--screens=/a,/b`**: optional — skip the affected-screen scoping in Step 6 and test exactly these routes.
|
|
8
10
|
|
|
9
11
|
**`prod` and `production` are refused.** An unattended regression pass signs in and clicks through live member data. Say so and stop.
|
|
10
12
|
|
|
@@ -37,7 +39,7 @@ Read the **Environment URLs** table from the current repo's `CLAUDE.md`:
|
|
|
37
39
|
| test | Test | https://test.myapp.example.com | https://test.api.myapp.example.com |
|
|
38
40
|
```
|
|
39
41
|
|
|
40
|
-
Capture the target row's App URL and API URL, the `**Sign-in:**` descriptor, the test-account table, and — for the differential re-check in Step 7 — the **App URL of the row immediately below the target**. That row serves the pre-change build, which is what lets a failure be attributed rather than guessed at.
|
|
42
|
+
Capture the target row's App URL and API URL, the `**Sign-in:**` descriptor, the test-account table, the `**Impersonation:**` block and its persona table if present, and — for the differential re-check in Step 7 — the **App URL of the row immediately below the target**. That row serves the pre-change build, which is what lets a failure be attributed rather than guessed at.
|
|
41
43
|
|
|
42
44
|
If the section is missing, or the target environment has no row, report:
|
|
43
45
|
|
|
@@ -57,7 +59,8 @@ everything below that heading is replaced on the next kit update:
|
|
|
57
59
|
| dev | Dev | https://… | {inferred, or …} |
|
|
58
60
|
| test | Test | https://… | {inferred, or …} |
|
|
59
61
|
|
|
60
|
-
**Sign-in:** form at `/login`. Signed-in landing: `/dashboard`.
|
|
62
|
+
**Sign-in:** `form` at `/login`. Signed-in landing: `/dashboard`.
|
|
63
|
+
(or `microsoft` — Entra ID redirect; sign-in button "…". Signed-in marker: …)
|
|
61
64
|
|
|
62
65
|
**Test accounts** — values live in environment variables; never write a password here.
|
|
63
66
|
|
|
@@ -91,40 +94,84 @@ so I can't open a browser. /qa cannot run.
|
|
|
91
94
|
browser binaries may be missing — npx playwright install chromium}
|
|
92
95
|
```
|
|
93
96
|
|
|
94
|
-
## Step 4: Gate —
|
|
97
|
+
## Step 4: Gate — Sign-in Method, Credentials, Impersonation
|
|
95
98
|
|
|
96
|
-
|
|
99
|
+
Read the **sign-in method** from the `**Sign-in:**` line, which starts with `` `form` `` or `` `microsoft` ``. An older descriptor may name neither. Treat it as `microsoft` only if it says sign-in **goes through** Microsoft, Entra ID, Azure AD, or MSAL. A line that says "no SSO, no MFA" is `form` — match the meaning, not the keywords. If the line is ambiguous, ask. Either way, say in the report that the descriptor should name its method.
|
|
100
|
+
|
|
101
|
+
Resolve the role: `--role=<x>`, else the row marked `Default`, else the first row. With `--as`, 4c overrides this.
|
|
102
|
+
|
|
103
|
+
### 4a. Method `form`
|
|
104
|
+
|
|
105
|
+
Credentials are **per developer**. Each developer exports their own account's credentials on their own machine, under the variable names the table declares. The same person's account often has a different password on each environment, because test usually has its own database. So each variable resolves **per environment**: `{VAR}_{ENV}` first, then `{VAR}`. `{ENV}` is `LOCAL`, `DEV`, `TEST`, or `STAGING`. For example, `COMPASS_QA_ADMIN_PASSWORD_TEST` wins over `COMPASS_QA_ADMIN_PASSWORD` on test.
|
|
106
|
+
|
|
107
|
+
Check **presence only**. Never read a credential value in this step — the agent does that, inside its own transcript:
|
|
97
108
|
|
|
98
109
|
```bash
|
|
99
110
|
for v in {PROJECT}_QA_ADMIN_USERNAME {PROJECT}_QA_ADMIN_PASSWORD; do
|
|
100
|
-
|
|
111
|
+
e="${v}_{ENV}"
|
|
112
|
+
if [ -n "$(printenv "$e")" ]; then echo "$e SET"
|
|
113
|
+
elif [ -n "$(printenv "$v")" ]; then echo "$v SET"
|
|
114
|
+
else echo "$v MISSING"; fi
|
|
101
115
|
done
|
|
102
116
|
```
|
|
103
117
|
|
|
104
|
-
|
|
118
|
+
Use `printenv`, not `${!v}`. Indirect expansion is bash-only and fails with `bad substitution` in zsh, the default macOS shell. The value is captured, never printed.
|
|
119
|
+
|
|
120
|
+
Pass the agent the **resolved** names: whichever of each pair was `SET`.
|
|
121
|
+
|
|
122
|
+
**Both set** → **unattended**: the agent signs in itself.
|
|
123
|
+
|
|
124
|
+
**Either missing** → this is not an error. It means the developer hasn't stored credentials, so offer the **hand-off** instead:
|
|
105
125
|
|
|
106
126
|
```
|
|
107
127
|
AB#{id}: {title}
|
|
108
128
|
|
|
109
|
-
|
|
110
|
-
|
|
129
|
+
No stored credentials for the {role} role on {environment}
|
|
130
|
+
({PROJECT}_QA_ADMIN_USERNAME / {PROJECT}_QA_ADMIN_PASSWORD, or their _{ENV} versions).
|
|
111
131
|
|
|
112
|
-
|
|
132
|
+
sign in → you sign in yourself, in the browser window Playwright opens, once the
|
|
133
|
+
plan is approved. Your password never passes through Claude.
|
|
134
|
+
stop → set them first, for unattended runs:
|
|
135
|
+
export {PROJECT}_QA_ADMIN_USERNAME="you@example.com"
|
|
136
|
+
export {PROJECT}_QA_ADMIN_PASSWORD="…"
|
|
137
|
+
export {PROJECT}_QA_ADMIN_PASSWORD_TEST="…" # if test differs
|
|
138
|
+
(add them to ~/.zshrc, then restart Claude Code)
|
|
113
139
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
Use a throwaway QA account with least privilege, never a personal login. Never put
|
|
118
|
-
the password in CLAUDE.md, .env, or any file in the repo.
|
|
140
|
+
Your own account is fine, as long as it holds the {role} role on {environment}. Never put
|
|
141
|
+
a password in CLAUDE.md, .env, or any file in the repo.
|
|
119
142
|
```
|
|
120
143
|
|
|
121
|
-
|
|
144
|
+
**What a hand-off costs.** The session belongs to the person who signed in, in that one browser tab. The agent can't sign in anywhere else on its own. That rules out three things:
|
|
122
145
|
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
146
|
+
- a differential re-check on the baseline — record `not checked — no stored credentials for {baseline}`;
|
|
147
|
+
- a second role for checklist row 14 — record `NOT TESTABLE — one interactive session per run`;
|
|
148
|
+
- returning to the admin account after impersonation ends — later screens run as the target user.
|
|
149
|
+
|
|
150
|
+
Say all of this in the plan, so the developer can choose to set the variables instead.
|
|
151
|
+
|
|
152
|
+
**Why offer it at all?** When the agent fills a login form, the password is a tool argument. It is stored in the session transcript on this machine and sent to the model, like every other tool call. For a throwaway QA account that is acceptable. For a developer's own **System Administrator** password it may not be. The hand-off is how a developer tests as themselves without that.
|
|
153
|
+
|
|
154
|
+
### 4b. Method `microsoft`
|
|
155
|
+
|
|
156
|
+
Two ways in. Pick by the role's row in the test-account table:
|
|
157
|
+
|
|
158
|
+
- **The row names credential variables, and the `Sign-in:` line says the account is MFA-exempt** → **unattended**. Run the same per-environment presence check as 4a. The agent signs in on the Microsoft page itself.
|
|
159
|
+
- **Otherwise** → **hand-off**. The person running `/qa` signs in once in the browser window Playwright opened. MFA is fine, and no credential enters any transcript. Nothing is checked here; the hand-off runs in Step 7a, after the plan is approved, so nobody signs in for a run they then cancel.
|
|
160
|
+
|
|
161
|
+
Under `microsoft`, a hand-off costs less than under `form`. The session persists in the Playwright profile, and Microsoft often signs a second host in silently, so the baseline re-check usually still works. Role gating (checklist row 14) still needs a second sign-in, so it stays `NOT TESTABLE — one interactive session per run` unless `--as` supplies a second persona.
|
|
162
|
+
|
|
163
|
+
### 4c. Impersonation — only when `--as` is given
|
|
164
|
+
|
|
165
|
+
1. **The project must describe it.** No `**Impersonation:**` block in `## Environment URLs`:
|
|
166
|
+
```
|
|
167
|
+
This project's CLAUDE.md doesn't describe an impersonation feature, so --as can't be used.
|
|
168
|
+
If the app has one, add an **Impersonation:** block to ## Environment URLs — the Claude Kit
|
|
169
|
+
Workflow section of CLAUDE.md shows the format.
|
|
170
|
+
```
|
|
171
|
+
2. **The environment must offer it.** The block lists where impersonation exists. If the target environment isn't listed, stop: `Impersonation isn't available on {environment} — the project lists {envs}. Run on one of those, or drop --as.`
|
|
172
|
+
3. **Resolve the target.** If `--as` matches a persona name, case-insensitively, it is a **persona**: take the search term and role from the row. Anything else is an **ad-hoc user**: the value itself is the search term, and `--as-role` gives the role if one is needed.
|
|
173
|
+
4. **The sign-in account is the one the block names as able to impersonate**, in the role it requires. That replaces `--role`; if both were given, say `--role` was ignored.
|
|
174
|
+
5. **An ad-hoc user means a read-only run.** The app records everything done while impersonating as that user. So the agent submits nothing, and checklist rows 8–10 and 12, plus every criterion that needs a write, become `NOT TESTABLE — would write as {target}`. Say this in the plan. If the story *is* a write path, suggest a persona instead.
|
|
128
175
|
|
|
129
176
|
## Step 5: Gate — Is the Code Actually There?
|
|
130
177
|
|
|
@@ -217,7 +264,8 @@ Then present the plan and **wait**:
|
|
|
217
264
|
|
|
218
265
|
Work item: {title} ({type}, {state})
|
|
219
266
|
Target: {environment} — {app-url}
|
|
220
|
-
Role: {role} ({USERNAME_VAR})
|
|
267
|
+
Role: {role} ({USERNAME_VAR} | you'll sign in yourself in the browser window | Microsoft, unattended from {USERNAME_VAR})
|
|
268
|
+
Impersonating: {persona} ({role}) | {search term} — ad-hoc user, READ-ONLY run | none
|
|
221
269
|
Build: {n} of {n} commits on `{branch}`, {pipeline} run {id} green
|
|
222
270
|
Baseline for attributing failures: {baseline-env} — {baseline-url}
|
|
223
271
|
|
|
@@ -238,7 +286,32 @@ Open the browser and run it? (yes / edit the screen list / cancel)
|
|
|
238
286
|
|
|
239
287
|
## Step 7: Run It, Then Attribute Every Failure
|
|
240
288
|
|
|
241
|
-
|
|
289
|
+
### 7a. Sign-in hand-off — whenever Step 4 chose a hand-off, for either method
|
|
290
|
+
|
|
291
|
+
The agent can't ask anyone to sign in, so this step runs here, in the main loop, in the same browser the agent will use. Use only `browser_navigate`, `browser_find`, `browser_click`, and `browser_wait_for`. **Never type into a login form or a Microsoft page**, and never take a full snapshot of the landing page.
|
|
292
|
+
|
|
293
|
+
First read the `playwright` entry's `args` in `.mcp.json`. With `--headless` there is no window to sign in through, so stop: `Signing in yourself needs a visible browser, but Playwright MCP is configured --headless. Remove the flag and restart the session.` With `--isolated` the hand-off still works, but a Microsoft session is gone at the end of every run. Say so once.
|
|
294
|
+
|
|
295
|
+
1. `browser_navigate` to the App URL, then `browser_find` the signed-in marker from the `Sign-in:` line. **Found** → the browser still holds a session. Say `Reusing the session already in the Playwright browser. If that isn't a {role} account, say "switch" and I'll sign out first.`, then go to 7b.
|
|
296
|
+
2. **Not found** → open the sign-in page. For `form`, that is the login route. For `microsoft`, click the app's sign-in button named in the descriptor and wait for `login.microsoftonline.com`. Then ask:
|
|
297
|
+
```
|
|
298
|
+
Sign in to {environment} in the Chromium window Playwright opened, as an account
|
|
299
|
+
with the {role} role — your own is fine.{ If it asks which role, choose {role}.}
|
|
300
|
+
I don't see or type your credentials{, and MFA is fine}.
|
|
301
|
+
|
|
302
|
+
{microsoft: The session stays in the Playwright browser profile, so later /qa and
|
|
303
|
+
/repro runs skip this step until it expires.}
|
|
304
|
+
{form: The session lives in that browser tab — leave the window open until I'm done.}
|
|
305
|
+
|
|
306
|
+
Reply "done" when you can see the app.
|
|
307
|
+
```
|
|
308
|
+
3. On `done`, `browser_find` the marker again. Still missing → say which page the browser is on and ask once more. A second miss → stop. Microsoft error `AADSTS50011` means `{app-url}` isn't a registered redirect URI on the app registration. That is a configuration problem, not a test failure; say so and stop.
|
|
309
|
+
|
|
310
|
+
### 7b. Delegate
|
|
311
|
+
|
|
312
|
+
Delegate to the **qa** agent with a brief containing: task `regression`; the work item id, title, type and state; the target App and API URLs; the baseline App URL; the sign-in method and descriptor, with `session handed over` or `unattended`; the **names** of the credential variables (never the values — say "they are set, read them yourself, never echo them"); the impersonation block and target, marked persona or ad-hoc, if `--as` was given; the ordered screen list with each screen's changed files; and the criteria. Above 3 screens, invoke it once per batch so a long sweep can't exhaust its context mid-run.
|
|
313
|
+
|
|
314
|
+
If the agent returns `BLOCKED — session ended`, run 7a once more and re-invoke it with only the screens it didn't reach. A second session loss → the report is `INCOMPLETE`. After a handed-over session, close the browser (`browser_close`) yourself once the last batch is done.
|
|
242
315
|
|
|
243
316
|
When the report comes back, sort every failure using three filters, in order:
|
|
244
317
|
|
|
@@ -254,7 +327,8 @@ Only filter 1's output can drive a `FAIL` verdict. A run that finds nothing but
|
|
|
254
327
|
QA — AB#{id}: {title}
|
|
255
328
|
{type} · {state} · {environment} — {app-url}
|
|
256
329
|
Build: {pipeline} run {id} · succeeded · commit {sha7}
|
|
257
|
-
Signed in as: {role} (from
|
|
330
|
+
Signed in as: {role} ({from USERNAME_VAR | signed in by you | Microsoft, from USERNAME_VAR}) — value not shown
|
|
331
|
+
Impersonating: {persona | search term} as {role} | none
|
|
258
332
|
Screens tested: {n} · {route list}
|
|
259
333
|
|
|
260
334
|
## Acceptance Criteria
|
|
@@ -303,7 +377,7 @@ The body, as HTML (Azure DevOps renders HTML in comments):
|
|
|
303
377
|
<b>Automated QA — {verdict}</b> · {environment} · {date}<br>
|
|
304
378
|
Build: {pipeline} run {id} ({result}) · commit {sha7}<br>
|
|
305
379
|
Screens tested: {route list}<br>
|
|
306
|
-
Signed in as: {role} test account<br><br>
|
|
380
|
+
Signed in as: {role} test account{, impersonating {persona | "an ad-hoc user (read-only run)"}}<br><br>
|
|
307
381
|
|
|
308
382
|
<b>Acceptance criteria — {x} of {y} pass</b>
|
|
309
383
|
<ul>
|
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
Reproduce a bug in a real browser — before a fix, to confirm it and pin it down, or after one, to prove it's gone. Usage: `/repro <work-item-id> [environment] [--verify | --reproduce] [--role=<role>] [--as=<persona-or-user>] [--as-role=<role>] [--attempts=<n>]`
|
|
2
|
+
|
|
3
|
+
Parse `$ARGUMENTS` to extract the work item ID. Accept formats like `AB#1234`, `#1234`, or just `1234`. If no ID is provided, report the error and stop.
|
|
4
|
+
|
|
5
|
+
- **Environment**: optional — `local`, `dev`, `test`, or `staging`. Defaults to `dev`.
|
|
6
|
+
- **`--verify` / `--reproduce`**: optional — force the mode. Without either, Step 3 picks it from where the fix is.
|
|
7
|
+
- **`--role`, `--as`, `--as-role`**: exactly as in `/qa`. `--as` is how you reproduce a bug **as the user who reported it**, on an app that supports impersonation.
|
|
8
|
+
- **`--attempts=<n>`**: optional — how many times to run the steps, 1–5. Defaults to 2, or 5 when an earlier `/repro` called the bug intermittent.
|
|
9
|
+
|
|
10
|
+
**`prod` and `production` are refused**, for the same reason as `/qa`: an unattended run signs in and clicks through live member data. Say so and stop.
|
|
11
|
+
|
|
12
|
+
`/repro` and `/qa` share one browser agent (**qa**) and one set of gates. `/qa` asks *"does this story work, and did it break anything?"* `/repro` asks a narrower question: *"does this bug happen?"* It asks it twice — once before the fix, so the developer starts from a confirmed reproduction, and once after, so "fixed" means something.
|
|
13
|
+
|
|
14
|
+
## Step 1: Fetch the Bug
|
|
15
|
+
|
|
16
|
+
Read the work item from Azure DevOps using the project from the current repo's CLAUDE.md configuration. Expand with `relations` so PR links are included. Not found → report the error and stop.
|
|
17
|
+
|
|
18
|
+
**Type gate.** Only a **Bug** or **Hot Fix** has repro steps. For any other type, stop:
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
AB#{id} is a {type}, which has no repro steps. /repro follows a bug's steps.
|
|
22
|
+
To test a story's acceptance criteria, run /qa AB#{id} {environment}.
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Capture: id, title, type, state, created by, linked Pull Request artifacts, and these fields:
|
|
26
|
+
|
|
27
|
+
- `Microsoft.VSTS.TCM.ReproSteps` — the steps, usually followed by *actual* and *expected* results.
|
|
28
|
+
- `Microsoft.VSTS.TCM.SystemInfo` — browser, environment, and account notes, when filled in.
|
|
29
|
+
- `System.Description` — the fallback.
|
|
30
|
+
|
|
31
|
+
Apply `/qa` Step 1's empty-field rules. `""`, whitespace, `<div></div>`, `<p><br></p>` and the `💡 Tip:` placeholder are all empty. Repro Steps empty → use the Description, and note `Repro steps read from Description`. Both empty → stop:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
AB#{id} has no repro steps, so there's nothing to follow.
|
|
35
|
+
|
|
36
|
+
Add them to the work item's Repro Steps and re-run — or give them to me here and I'll
|
|
37
|
+
use them for this run only (they won't be written back to the work item).
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
**Read the comments.** Look for two things:
|
|
41
|
+
|
|
42
|
+
1. **Earlier `/repro` results.** They start with `Automated repro —`. Record each one's mode, verdict, environment, commit, and date. A prior `REPRODUCED` can serve as the control in Step 3, and a prior `INTERMITTENT` raises the default attempts to 5.
|
|
43
|
+
2. **Detail the steps leave out.** Which user hit it, which record, which environment, which browser. A named user feeds the `--as` suggestion in Step 4.
|
|
44
|
+
|
|
45
|
+
Download and view every embedded `<img>` in the steps, description, and comments with WebFetch. A screenshot of the bug is often the clearest statement of its signature.
|
|
46
|
+
|
|
47
|
+
## Step 2: Environment, Browser, Sign-in
|
|
48
|
+
|
|
49
|
+
Read `.claude/commands/qa.md` and run its **Step 2** (Resolve the Environment), **Step 3** (Is a Browser Available?) and **Step 4** (Sign-in Method, Credentials, Impersonation) exactly as written. In their messages, say `/repro` where they say `/qa`. Every stop condition in those steps stops `/repro` too. One refinement applies: under an ad-hoc `--as`, `/qa` makes the whole run read-only, but `/repro` lets the user approve individual writing steps in Step 4.
|
|
50
|
+
|
|
51
|
+
## Step 3: Pick the Mode
|
|
52
|
+
|
|
53
|
+
Find the fix's commits the way `/qa` Step 5a does: linked PRs → `lastMergeCommit`; `git log --all --grep="AB#<id>\b" --grep="#<id>\b" -i --format=%H`; and branches matching `*AB#<id>-*`. Dedupe by SHA. Then check each commit against the target with `git merge-base --is-ancestor <sha> origin/<target-branch>`, or against `HEAD` for `local`.
|
|
54
|
+
|
|
55
|
+
| Fix commits | On the target | Mode |
|
|
56
|
+
|---|---|---|
|
|
57
|
+
| none | — | **reproduce** — no fix exists yet |
|
|
58
|
+
| some | not all | **reproduce** — the fix exists but isn't fully on {environment}, so this run confirms the bug on the old build |
|
|
59
|
+
| some | all | **verify-fix** |
|
|
60
|
+
|
|
61
|
+
For `local`, also count it as verify-fix when the current branch matches `*AB#<id>-*` and the working tree has uncommitted changes — that is a fix in progress. Remind the user that the running dev server must be serving this working tree.
|
|
62
|
+
|
|
63
|
+
**Overrides.** `--reproduce` always wins. `--verify` on a target that lacks some fix commits is refused: verifying a fix that isn't deployed only tests the old build.
|
|
64
|
+
|
|
65
|
+
```
|
|
66
|
+
The fix for AB#{id} isn't fully on {environment} — {x} of {y} commits present.
|
|
67
|
+
/repro --verify here would test the build without it.
|
|
68
|
+
|
|
69
|
+
Run /repro AB#{id} {environment} to confirm the bug on this build, or verify where the
|
|
70
|
+
fix is: {environments whose branch has every commit, or "nowhere yet"}.
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
**Verify-fix on `dev` / `test` / `staging`** → also run `/qa` **Step 5b**. The pipeline that deployed the fix must be green. Being merged is not being deployed. **`local`, either mode** → run `/qa` Step 5's `local` probe. Reproduce mode on a deployed environment needs no pipeline gate, because it tests whatever is there.
|
|
74
|
+
|
|
75
|
+
**Pick the control** — verify-fix only. The control is what tells *fixed* apart from *these steps never triggered it*. Take the first one of these that applies:
|
|
76
|
+
|
|
77
|
+
1. A prior `/repro` comment recorded `REPRODUCED` with the same steps, on a commit without the fix → it is the control. No control run is needed.
|
|
78
|
+
2. Otherwise, walk down the `## Environment URLs` rows below the target, and take the first one whose branch lacks at least one fix commit. That environment still serves the bug. Under `--as`, it must also offer impersonation; skip any row that doesn't.
|
|
79
|
+
3. Neither → no control. Step 6 can then only say `FIXED — UNCONFIRMED`, and the plan says so up front.
|
|
80
|
+
|
|
81
|
+
## Step 4: Turn the Report Into a Script, Then Get Approval
|
|
82
|
+
|
|
83
|
+
Rewrite the repro steps as a numbered script the agent can follow literally: a route, a control's visible label, the value to enter. Keep the report's own wording where it is already concrete. Resolve "go to the bills page" to the route. When a step names something vaguely, like *"a bill"* or *"any provider"*, keep it vague and let the agent pick a suitable record. Never invent a record id.
|
|
84
|
+
|
|
85
|
+
Then define the two things the agent checks for. Without them a run proves nothing:
|
|
86
|
+
|
|
87
|
+
- **Bug signature** — the *actual result*, made observable: a status code, a message, a wrong value, a console error, a control stuck in the wrong state.
|
|
88
|
+
- **Expected behaviour** — the *expected result*: what a correct build shows at the same point.
|
|
89
|
+
|
|
90
|
+
If the report doesn't say what the bug looks like, ask one question and wait: `What do you see when the bug happens? I need something observable — an error message, a wrong value, a failed request.`
|
|
91
|
+
|
|
92
|
+
Mark every step that writes:
|
|
93
|
+
|
|
94
|
+
- `← writes: {what}` — submits, saves, approves, or sends. It runs under the normal write rules: prefixed QA data, and nothing that emails a real person.
|
|
95
|
+
- `← manual only — destructive` — it breaks one of the agent's write-safety rules: payment, settlement, bulk action, tax document, export-all, or a delete. The agent records it as `STEP FAILED`. If the bug sits behind such a step, say plainly that `/repro` can't reach it.
|
|
96
|
+
- `← manual only — protected input` — it requires typing an SSN, TIN, bank account number, or another protected value. The agent never types one.
|
|
97
|
+
- **Under an ad-hoc `--as`,** every writing step needs explicit approval here. Everything done while impersonating is recorded as that user. Approved steps go to the agent as **approved mutating steps**; the rest become `STEP FAILED — would write as {target}`.
|
|
98
|
+
|
|
99
|
+
**Suggest impersonation when it fits.** Step 1 may have found a named user who hit the bug. If the project's `**Impersonation:**` block covers this environment and `--as` wasn't given, add `Tip: the report says this happens to {user} — re-run with --as="{user}" to reproduce it as them.` If the bug is about what a *role* sees, suggest the matching persona.
|
|
100
|
+
|
|
101
|
+
Present the plan and **wait**:
|
|
102
|
+
|
|
103
|
+
```
|
|
104
|
+
## Repro plan — AB#{id} on {environment}
|
|
105
|
+
|
|
106
|
+
Bug: {title} ({type}, {state})
|
|
107
|
+
Mode: {reproduce — no fix exists yet
|
|
108
|
+
| reproduce — fix is on {branch} but not on {environment}
|
|
109
|
+
| verify-fix — {n} of {n} fix commits on `{branch}`, {pipeline} run {id} green}
|
|
110
|
+
Target: {environment} — {app-url}
|
|
111
|
+
Sign-in: {role} ({method})
|
|
112
|
+
Impersonating: {persona} ({role}) | {search term} — ad-hoc user, read-only except approved steps | none
|
|
113
|
+
Control: {baseline-env} — {url}, which doesn't have the fix
|
|
114
|
+
| prior /repro on {date} reproduced it on {env} at {sha7}
|
|
115
|
+
| none — {reason}; a pass can only be FIXED — UNCONFIRMED
|
|
116
|
+
Attempts: {n}
|
|
117
|
+
|
|
118
|
+
Steps:
|
|
119
|
+
1. Go to /bills
|
|
120
|
+
2. Open a bill in "Pending" status
|
|
121
|
+
3. Click "Convert to Check"
|
|
122
|
+
4. Click "Confirm" in the dialog ← writes: converts the bill
|
|
123
|
+
5. Open the Payments tab
|
|
124
|
+
|
|
125
|
+
Bug signature: POST /api/bills/{ref}/convert → 500, and the error toast "Something went wrong"
|
|
126
|
+
Expected behaviour: the bill shows status "Check Issued" and a payment row appears
|
|
127
|
+
|
|
128
|
+
{Tip: …}
|
|
129
|
+
|
|
130
|
+
Run it? (yes / edit the steps / edit the signature / cancel)
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
## Step 5: Run It
|
|
134
|
+
|
|
135
|
+
If `/qa` Step 4 chose a hand-off, for either method, run `/qa` **Step 7a** (the sign-in hand-off) first.
|
|
136
|
+
|
|
137
|
+
Delegate to the **qa** agent with a brief containing:
|
|
138
|
+
|
|
139
|
+
- the task (`reproduce` or `verify-fix`), plus the bug's id, title, type and state;
|
|
140
|
+
- the target App and API URLs;
|
|
141
|
+
- the control's App URL, if the control is an environment, with "run a control" — otherwise `none`;
|
|
142
|
+
- the sign-in method and descriptor, with `session handed over` or `unattended`;
|
|
143
|
+
- the **names** of the credential variables, never the values;
|
|
144
|
+
- the impersonation block and target, marked persona or ad-hoc, if `--as` was given;
|
|
145
|
+
- the approved script, the bug signature, the expected behaviour, the approved mutating steps, and the number of attempts.
|
|
146
|
+
|
|
147
|
+
If the agent returns `BLOCKED — session ended`, run `/qa` Step 7a once more and re-invoke it for the attempts it didn't reach. After a handed-over session, close the browser yourself at the end.
|
|
148
|
+
|
|
149
|
+
**Under a `form` hand-off there is no control run on another environment.** The agent can't sign in there. Step 3's control then falls back to a prior `/repro` comment or none, and the plan must say so.
|
|
150
|
+
|
|
151
|
+
## Step 6: Verdict
|
|
152
|
+
|
|
153
|
+
Assign the verdict from the agent's attempts and the control. Count a `STEP FAILED` at or before the signature step as **not run**, never as *not seen*.
|
|
154
|
+
|
|
155
|
+
### Reproduce mode
|
|
156
|
+
|
|
157
|
+
| Attempts | Verdict |
|
|
158
|
+
|---|---|
|
|
159
|
+
| Signature seen in every attempt | `REPRODUCED` |
|
|
160
|
+
| Seen in some attempts | `REPRODUCED — INTERMITTENT ({k} of {n})` |
|
|
161
|
+
| Seen only under a variation | `REPRODUCED — under V{n}: {what changed}`. The report's steps are incomplete; recommend updating them |
|
|
162
|
+
| Not seen, expected behaviour observed | `NOT REPRODUCED` |
|
|
163
|
+
| Not seen, something else went wrong | `NOT REPRODUCED — DIFFERENT SYMPTOM: {what}` |
|
|
164
|
+
| A step couldn't be performed | `BLOCKED — {reason}` |
|
|
165
|
+
|
|
166
|
+
### Verify-fix mode
|
|
167
|
+
|
|
168
|
+
| Target | Control | Verdict |
|
|
169
|
+
|---|---|---|
|
|
170
|
+
| Not seen in any attempt, expected behaviour observed, screen check clean | seen, or a prior `REPRODUCED` | `FIXED` |
|
|
171
|
+
| Same | not seen, or no control | `FIXED — UNCONFIRMED` — the steps didn't trigger the bug on the old build either, so this run can't tell fixed from never-triggered |
|
|
172
|
+
| Not seen, expected behaviour observed, screen check fails | any | `FIXED — SCREEN REGRESSION: {check}`. Attribute it the way `/qa` Step 7 does: does the failing surface touch the fix's changed files? |
|
|
173
|
+
| Not seen, but expected behaviour **not** observed | any | `NOT FIXED — DIFFERENT SYMPTOM: {what happened instead}` |
|
|
174
|
+
| Seen in any attempt | any | `STILL REPRODUCES ({k} of {n})` |
|
|
175
|
+
| A step couldn't be performed | any | `BLOCKED — {reason}` |
|
|
176
|
+
|
|
177
|
+
A verify-fix with fewer than 3 attempts on a bug a prior run called intermittent can only be `FIXED — UNCONFIRMED`. Two clean runs of a 1-in-3 bug is just luck.
|
|
178
|
+
|
|
179
|
+
## Step 7: Where to Look — whenever the signature was seen
|
|
180
|
+
|
|
181
|
+
The developer's next question is *where*. Answer it from the evidence, reading only:
|
|
182
|
+
|
|
183
|
+
1. **Failing request → handler.** For each failing `METHOD path → status` the agent recorded, grep the API's controllers for the route attributes (`[Route]`, `[HttpPost("…")]`) that match. Report the action by `file:line`.
|
|
184
|
+
2. **Screen → component.** Map the route where the signature appeared to its component through the client router, as `/qa` Step 6.3 does. Then look one hop down to the components that make the failing call.
|
|
185
|
+
3. **Console error → source.** If the first console line names a function or a component, grep for it.
|
|
186
|
+
|
|
187
|
+
Report at most 5 locations, each with the evidence that led there. **Never claim a root cause** — these are places to start reading.
|
|
188
|
+
|
|
189
|
+
In **verify-fix** mode, when the bug still reproduces, run the same mapping. Then compare the locations with the fix commits' changed files. `the fix changed BillService.cs, but the failing call is handled in BillsController.cs:212` is the most useful sentence this command can produce.
|
|
190
|
+
|
|
191
|
+
## Step 8: Report
|
|
192
|
+
|
|
193
|
+
```
|
|
194
|
+
Repro — AB#{id}: {title}
|
|
195
|
+
{type} · {state} · {environment} — {app-url}
|
|
196
|
+
Mode: {reproduce | verify-fix} · Build: {pipeline run {id}, commit {sha7} | local working tree}
|
|
197
|
+
Signed in as: {role} ({method}) — value not shown
|
|
198
|
+
Impersonating: {persona | search term} as {role} | none
|
|
199
|
+
|
|
200
|
+
Verdict: {verdict}
|
|
201
|
+
|
|
202
|
+
## Attempts
|
|
203
|
+
| Attempt | Variation | Result | At step | Evidence |
|
|
204
|
+
|---------|-----------|--------|---------|----------|
|
|
205
|
+
| 1 | — | SIGNATURE SEEN | 4 | network `POST /api/bills/…/convert → 500` |
|
|
206
|
+
| 2 | — | SIGNATURE SEEN | 4 | network `POST /api/bills/…/convert → 500` |
|
|
207
|
+
|
|
208
|
+
## Control — {baseline-env | prior run {date}}
|
|
209
|
+
{SIGNATURE SEEN at step n | NOT SEEN | not run — {reason}}
|
|
210
|
+
|
|
211
|
+
## Screen check — {route} ← verify-fix only
|
|
212
|
+
{rows 1–5 from the agent}
|
|
213
|
+
|
|
214
|
+
## Where to look ← when the bug reproduced
|
|
215
|
+
1. `{file}:{line}` — {why: handles POST /api/bills/{ref}/convert, which returned 500}
|
|
216
|
+
2. …
|
|
217
|
+
|
|
218
|
+
## Not testable
|
|
219
|
+
- {step or check} — {why}
|
|
220
|
+
|
|
221
|
+
## Test data created on {environment}
|
|
222
|
+
- {entity} "QA AB#{id} — …" (id {x}) — left in place for review
|
|
223
|
+
|
|
224
|
+
Screenshot: {local path} — local only, not attached to the work item
|
|
225
|
+
|
|
226
|
+
Next: {REPRODUCED → `/implement AB#{id}`
|
|
227
|
+
| NOT REPRODUCED → ask the reporter for the record, the user, and the environment
|
|
228
|
+
where it happened; or re-run with --as or in another environment
|
|
229
|
+
| FIXED → `Ready for Testing` / `Ready for Staging` is yours to set
|
|
230
|
+
| FIXED — UNCONFIRMED → reproduce it on the old build first: /repro AB#{id} {baseline-env} --reproduce
|
|
231
|
+
| STILL REPRODUCES / NOT FIXED → `/rework AB#{id}`, with the "Where to look" list above}
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## Step 9: Comment on the Work Item
|
|
235
|
+
|
|
236
|
+
Draft the comment, preview it, and **wait**, exactly as in `/qa` Step 9. Before drafting, check the existing comments for a `/repro` result with the **same mode and commit**. If one exists: `A /repro {mode} comment already exists for commit {sha7} ({date}). Post an updated one anyway? (yes / skip)`.
|
|
237
|
+
|
|
238
|
+
This comment matters more than `/qa`'s, because the next `/repro` reads it. A `REPRODUCED` here becomes the control that lets a later verify-fix say `FIXED` rather than `FIXED — UNCONFIRMED`. So **always include the script and the signature** — they are what make two runs comparable.
|
|
239
|
+
|
|
240
|
+
```html
|
|
241
|
+
<b>Automated repro — {verdict}</b> · {mode} · {environment} · {date}<br>
|
|
242
|
+
Build: {pipeline} run {id} · commit {sha7}{ | local working tree}<br>
|
|
243
|
+
Signed in as: {role} test account{, impersonating {persona | "the reporting user"}}<br><br>
|
|
244
|
+
|
|
245
|
+
<b>Steps run</b>
|
|
246
|
+
<ol>
|
|
247
|
+
<li>{step}</li>
|
|
248
|
+
</ol>
|
|
249
|
+
<b>Bug signature:</b> {signature}<br>
|
|
250
|
+
<b>Expected:</b> {expected behaviour}<br><br>
|
|
251
|
+
|
|
252
|
+
<b>Result — signature seen in {k} of {n} attempts</b>
|
|
253
|
+
<ul>
|
|
254
|
+
<li>Attempt 1: {result} at step {n} — {evidence}</li>
|
|
255
|
+
</ul>
|
|
256
|
+
{<b>Control</b> — {baseline-env}: {result}<br>}
|
|
257
|
+
{<b>Where to look</b><ul><li><code>{file}:{line}</code> — {why}</li></ul>}
|
|
258
|
+
{<b>Test data created</b><ul><li>{entity} "QA AB#{id} — …"</li></ul>}
|
|
259
|
+
|
|
260
|
+
<hr>
|
|
261
|
+
<i>Posted by <code>/repro</code>. Work item state unchanged. No screenshots and no field
|
|
262
|
+
values are attached, per the sensitive-data policy.</i>
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
The comment never names an ad-hoc impersonation target. It says *"the reporting user"*, because the search term is often a real person's email.
|
|
266
|
+
|
|
267
|
+
Then:
|
|
268
|
+
|
|
269
|
+
```
|
|
270
|
+
Repro drafted — nothing has been posted to AB#{id} yet.
|
|
271
|
+
|
|
272
|
+
## Work item comment
|
|
273
|
+
{the full body as it will appear}
|
|
274
|
+
|
|
275
|
+
Reply with one of:
|
|
276
|
+
- "approve" → post the comment exactly as shown
|
|
277
|
+
- "skip" → post nothing (the report above is yours to keep)
|
|
278
|
+
- "edit <section>: <new text>" → revise, then I'll re-preview before posting
|
|
279
|
+
|
|
280
|
+
Wait for my response. Never post to the work item until I reply "approve".
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
On `approve`, post exactly one comment via `mcp__azure-devops__wit_work_item_comment_write`.
|
|
284
|
+
|
|
285
|
+
That's it. `/repro` never changes the work item's state — a `FIXED` doesn't move the bug forward, and a `NOT REPRODUCED` doesn't close it or tag it. Those are human calls. It also never merges, pushes, creates a PR, triggers a pipeline, edits code, attaches a screenshot to the work item, or transcribes a field value from a screen. It reads, it clicks, it reports.
|
|
@@ -8,3 +8,4 @@
|
|
|
8
8
|
echo "Reminder: If you just completed a feature implementation, run UAT before marking it done."
|
|
9
9
|
echo "Ask Claude to generate a UAT checklist from the work item's acceptance criteria."
|
|
10
10
|
echo "Once it is merged and deployed, /qa AB#<id> runs an automated browser regression of the affected screens."
|
|
11
|
+
echo "For a bug fix, /repro AB#<id> re-runs the repro steps and confirms the bug is gone."
|