@chris1807/claude-kit 2.1.56 → 2.1.58
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -12
- package/bin/cli.js +99 -9
- package/package.json +1 -1
- package/templates/agents/global/qa.md +136 -0
- package/templates/commands/cherry-pick.md +1 -1
- package/templates/commands/deploy-release.md +1 -1
- package/templates/commands/promote.md +1 -1
- package/templates/commands/qa.md +353 -0
- package/templates/commands/{status.md → track.md} +7 -7
- package/templates/commands/where.md +1 -1
- package/templates/hooks/sensitive-data-output-blocker.sh +56 -18
- package/templates/hooks/uat-reminder.sh +1 -0
- package/templates/infrastructure/CLAUDE-WORKFLOW.md +68 -6
- package/templates/infrastructure/settings.json +9 -0
package/README.md
CHANGED
|
@@ -17,7 +17,7 @@ The slash commands assume Azure DevOps as the system of record:
|
|
|
17
17
|
- **Releases & deployments** — tracked as Azure DevOps iterations (`Release #N`) and tags; CD runs on Azure Pipelines
|
|
18
18
|
- **Wiki, test plans, advanced security alerts** — all surfaced through the same Azure DevOps MCP server
|
|
19
19
|
|
|
20
|
-
> **Not using Azure DevOps?** The Claude Code primitives (agents, hooks, memory) are still useful, but the slash commands and the deployment workflow won't apply out of the box — you'd need to rewrite the `/implement`, `/review`, `/deploy`, `/create-release`, `/deploy-release`, `/cherry-pick`, `/promote`, `/rollback`, `/
|
|
20
|
+
> **Not using Azure DevOps?** The Claude Code primitives (agents, hooks, memory) are still useful, but the slash commands and the deployment workflow won't apply out of the box — you'd need to rewrite the `/implement`, `/review`, `/deploy`, `/create-release`, `/deploy-release`, `/cherry-pick`, `/promote`, `/rollback`, `/track`, `/rework`, and `/resolve-feedback` commands against GitHub / GitLab / Jira / etc.
|
|
21
21
|
|
|
22
22
|
Every session Claude learns from your feedback and gets better at helping you specifically. The infrastructure is modular — install only what your project needs.
|
|
23
23
|
|
|
@@ -28,7 +28,7 @@ Every session Claude learns from your feedback and gets better at helping you sp
|
|
|
28
28
|
| **Global Agents** | 10 | `~/.claude/agents/` (your machine, all projects) | backend, frontend, legacy (Lucee/CFML), mockup, reviewer, test-runner, build-validator, lint-checker, azure-ops, security-auditor |
|
|
29
29
|
| **Project Agents** | 2 | `.claude/agents/` (in the project) | deployer, db-admin |
|
|
30
30
|
| **Hooks** | 9 | `.claude/hooks/` (in the project) | Secret blocker, sensitive data blocker (Bash + MCP + output), protected files, auto-format, test suggestions, UAT reminder, self-improve |
|
|
31
|
-
| **Slash Commands** | 24 | `.claude/commands/` (in the project) | `/implement`, `/review`, `/deep-review`, `/resolve-feedback`, `/fix-review`, `/deploy`, `/create-release`, `/deploy-release`, `/add-to-release`, `/cherry-pick`, `/promote`, `/rollback`, `/
|
|
31
|
+
| **Slash Commands** | 24 | `.claude/commands/` (in the project) | `/implement`, `/review`, `/deep-review`, `/resolve-feedback`, `/fix-review`, `/deploy`, `/create-release`, `/deploy-release`, `/add-to-release`, `/cherry-pick`, `/promote`, `/rollback`, `/track`, `/plan-backlog`, `/plan-sprint`, `/quote-backlog`, `/cleanup-branches`, `/close-orphan-tasks`, `/quote`, `/explain`, `/create-work-item`, `/edit-work-item` |
|
|
32
32
|
| **MCP Servers** | Up to 6 | `.mcp.json` (in the project) | **Azure DevOps** (work items, repos, pipelines, wiki), Playwright, MongoDB/SQL/Postgres, Teams, Stripe, Azure CLI |
|
|
33
33
|
| **Workflow Template** | 1 | Appended to `CLAUDE.md` | Documents the full development process |
|
|
34
34
|
| **Settings** | 1 | `.claude/settings.json` (in the project) | Registers all hooks and MCP servers |
|
|
@@ -73,7 +73,7 @@ You'll be asked:
|
|
|
73
73
|
2. **Components** — checkboxes to pick which parts to install:
|
|
74
74
|
- ☑ Project Agents (deployer, db-admin)
|
|
75
75
|
- ☑ Hooks (secret blocker, auto-format, etc.)
|
|
76
|
-
- ☑ Slash Commands (/implement, /review, /resolve-feedback, /deploy, /create-release, /deploy-release, /add-to-release, /cherry-pick, /promote, /rollback, /
|
|
76
|
+
- ☑ Slash Commands (/implement, /review, /resolve-feedback, /deploy, /create-release, /deploy-release, /add-to-release, /cherry-pick, /promote, /rollback, /track, /cleanup-branches, /quote, /explain)
|
|
77
77
|
- ☑ MCP Servers
|
|
78
78
|
- ☑ Settings
|
|
79
79
|
- ☑ CLAUDE.md Workflow
|
|
@@ -181,6 +181,7 @@ The installer automatically **skips global agents** that are already installed (
|
|
|
181
181
|
├── mockup.md # HTML mockup designer
|
|
182
182
|
├── reviewer.md # Code reviewer (read-only)
|
|
183
183
|
├── test-runner.md # Test executor — xUnit, Vitest, Playwright (read-only)
|
|
184
|
+
├── qa.md # Browser QA on a deployed environment (read-only)
|
|
184
185
|
├── build-validator.md # Build checker (read-only)
|
|
185
186
|
├── lint-checker.md # ESLint + dotnet format checker
|
|
186
187
|
├── azure-ops.md # Azure infrastructure management
|
|
@@ -213,7 +214,7 @@ your-project/ ← Project-specific
|
|
|
213
214
|
│ │ ├── promote.md # /promote main dev
|
|
214
215
|
│ │ ├── rollback.md # /rollback AB#1234 prod
|
|
215
216
|
│ │ ├── add-to-release.md # /add-to-release 24 AB#4599
|
|
216
|
-
│ │ ├──
|
|
217
|
+
│ │ ├── track.md # /track release 24
|
|
217
218
|
│ │ ├── cleanup-branches.md # /cleanup-branches
|
|
218
219
|
│ │ ├── close-orphan-tasks.md # /close-orphan-tasks
|
|
219
220
|
│ │ ├── quote.md # /quote AB#1234
|
|
@@ -260,7 +261,7 @@ Kit operations that benefit from ultracode when it's on:
|
|
|
260
261
|
| Multi-file or cross-layer review | one agent per file/dimension, then adversarial verify before reporting |
|
|
261
262
|
| Repo-wide sweeps (rename, dependency bump, pattern migration) | one agent per site, worktree-isolated |
|
|
262
263
|
|
|
263
|
-
Short, single-query operations (`/
|
|
264
|
+
Short, single-query operations (`/track`, `/explain`, `/close-orphan-tasks`) don't need ultracode — they're already one pass and gain nothing from fan-out. Reach for it when the work-list is large and the per-item work is independent.
|
|
264
265
|
|
|
265
266
|
### Implement a Work Item
|
|
266
267
|
|
|
@@ -372,11 +373,11 @@ Adds work items to an existing release — assigns them to the iteration and tag
|
|
|
372
373
|
### Check Status
|
|
373
374
|
|
|
374
375
|
```
|
|
375
|
-
/
|
|
376
|
-
/
|
|
377
|
-
/
|
|
378
|
-
/
|
|
379
|
-
/
|
|
376
|
+
/track release 24
|
|
377
|
+
/track pipeline
|
|
378
|
+
/track AB#4521
|
|
379
|
+
/track staging
|
|
380
|
+
/track
|
|
380
381
|
```
|
|
381
382
|
|
|
382
383
|
Shows the status of a release, pipeline, work item, environment, or a high-level overview of everything.
|
|
@@ -440,7 +441,7 @@ Every environment has a work item state, and the promotion commands keep them in
|
|
|
440
441
|
| `test → staging` | `Ready for Staging` (QA sign-off) | `Staging` |
|
|
441
442
|
| `staging → prod` | `Ready to Deploy` (stakeholder sign-off) | `Deployed` |
|
|
442
443
|
|
|
443
|
-
`/deploy-release`, `/cherry-pick`, and `/promote` check the gate before creating the PR, ask who verifies each product group while the PR is in review, then watch the PR and the environment's CD pipeline — on a green pipeline they set the state and the assignee together. A failed pipeline changes nothing. `/
|
|
444
|
+
`/deploy-release`, `/cherry-pick`, and `/promote` check the gate before creating the PR, ask who verifies each product group while the PR is in review, then watch the PR and the environment's CD pipeline — on a green pipeline they set the state and the assignee together. A failed pipeline changes nothing. `/track` flags anything whose state lags the branch it's on.
|
|
444
445
|
|
|
445
446
|
### Branch Naming
|
|
446
447
|
|
|
@@ -678,7 +679,8 @@ Claude reviews for:
|
|
|
678
679
|
| `/cherry-pick` | `/cherry-pick AB#1234 AB#1235 prod` | Gate-check → cherry-pick specific work items to environment via PR → ask who verifies → advance + assign on a green pipeline |
|
|
679
680
|
| `/promote` | `/promote main dev` | PR to promote all code to the next branch in the chain (`main → dev → test → staging → prod`) → ask who verifies → advance + assign on a green pipeline |
|
|
680
681
|
| `/rollback` | `/rollback AB#1234 prod` | Revert specific commits on an environment via PR |
|
|
681
|
-
| `/
|
|
682
|
+
| `/track` | `/track release 24` | Check status of a release, pipeline, work item, or environment; flags work items whose state lags their environment |
|
|
683
|
+
| `/qa` | `/qa AB#1234 [env]` | Verify the item is fully deployed with a green pipeline → open the app in a real browser → sign in as a test account → full regression of the screens the story touched → pass/fail comment on the work item (never a state change). Environments: `local`, `dev`, `test`, `staging` |
|
|
682
684
|
| `/plan-backlog` | `/plan-backlog [project]` | Sweep backlog for Dev Ready stories with points and no tasks → propose one child task with hours per story |
|
|
683
685
|
| `/plan-sprint` | `/plan-sprint [project]` | Sweep the current sprint for stories/bugs with no child tasks → propose one child task with hours per item |
|
|
684
686
|
| `/quote-backlog` | `/quote-backlog [project]` | Sweep backlog for unpointed items ready to estimate — stories in `Design Approved`, bugs in `New` (bugs have no design states) → review completeness, check for duplicates, suggest rewrites, propose points + creator comments (10 at a time, approval-gated). Also audits the drop-out queue — tagged bugs and stories bounced to Design Review — for items whose creator answered but which nobody returned to the sweep |
|
|
@@ -731,6 +733,12 @@ export STRIPE_SECRET_KEY="sk_test_..."
|
|
|
731
733
|
# Azure CLI (no env var needed — just log in)
|
|
732
734
|
az login
|
|
733
735
|
|
|
736
|
+
# QA test accounts (required for /qa — browser testing on a deployed environment).
|
|
737
|
+
# One pair per role. The variable NAMES go in your project's `## Environment URLs`
|
|
738
|
+
# section; the VALUES live only here. Use throwaway least-privilege QA accounts.
|
|
739
|
+
export COMPASS_QA_ADMIN_USERNAME="qa-admin@example.com"
|
|
740
|
+
export COMPASS_QA_ADMIN_PASSWORD="..."
|
|
741
|
+
|
|
734
742
|
# Microsoft Teams (no env vars needed — uses Microsoft Graph device-code auth;
|
|
735
743
|
# first invocation prints a code + URL to sign in)
|
|
736
744
|
```
|
package/bin/cli.js
CHANGED
|
@@ -161,12 +161,84 @@ function mergeHooks(existing, template) {
|
|
|
161
161
|
// don't linger. manager/uat-generator/api-tester were superseded by the
|
|
162
162
|
// Workflow tool and inline command steps; devops-tracker by direct Azure
|
|
163
163
|
// DevOps MCP access in the main loop.
|
|
164
|
+
// The workflow section written into a target CLAUDE.md is delimited by this marker so
|
|
165
|
+
// a re-install can replace it without touching anything the project added afterwards.
|
|
166
|
+
// Installs from before the marker existed are recovered heuristically — see
|
|
167
|
+
// splitWorkflowSection.
|
|
168
|
+
const WORKFLOW_END_MARKER = '<!-- claude-kit:workflow:end -->';
|
|
169
|
+
|
|
170
|
+
const WORKFLOW_HEADING_RE = /\n## (?:Claude Kit Workflow|Care Solutions AI Workflow)/;
|
|
171
|
+
|
|
172
|
+
/**
|
|
173
|
+
* Split a target CLAUDE.md around the kit's workflow section.
|
|
174
|
+
*
|
|
175
|
+
* Returns { head, preserved } where `head` is everything before the workflow
|
|
176
|
+
* heading and `preserved` is project content that followed the kit's block —
|
|
177
|
+
* or null when there is no workflow section at all.
|
|
178
|
+
*
|
|
179
|
+
* The old implementation replaced /heading[\s\S]*$/, i.e. everything to end of
|
|
180
|
+
* file, which silently deleted any `## Pipeline Configuration` or
|
|
181
|
+
* `## Environment URLs` a project had placed below the heading. With the marker
|
|
182
|
+
* present the boundary is exact. Without it (a pre-marker install) we fall back
|
|
183
|
+
* to a heading scan: the template's only top-level heading outside a fenced code
|
|
184
|
+
* block is the workflow heading itself, so any other non-fenced `## ` heading in
|
|
185
|
+
* the tail is project content and is kept. Fenced headings are skipped because
|
|
186
|
+
* the template quotes `## Pipeline Configuration` inside ```markdown examples.
|
|
187
|
+
*/
|
|
188
|
+
function splitWorkflowSection(existing) {
|
|
189
|
+
const match = existing.match(WORKFLOW_HEADING_RE);
|
|
190
|
+
if (!match) return null;
|
|
191
|
+
|
|
192
|
+
const head = existing.slice(0, match.index).trimEnd();
|
|
193
|
+
const tail = existing.slice(match.index);
|
|
194
|
+
|
|
195
|
+
const markerIdx = tail.indexOf(WORKFLOW_END_MARKER);
|
|
196
|
+
if (markerIdx !== -1) {
|
|
197
|
+
return {
|
|
198
|
+
head,
|
|
199
|
+
preserved: tail.slice(markerIdx + WORKFLOW_END_MARKER.length).trim(),
|
|
200
|
+
recovered: false,
|
|
201
|
+
};
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
const lines = tail.split('\n');
|
|
205
|
+
// `tail` begins with the newline preceding the heading, so the heading is not at
|
|
206
|
+
// index 0. Find it explicitly and scan from the line after it — starting at a
|
|
207
|
+
// fixed offset would treat the heading itself as project content.
|
|
208
|
+
const headingIdx = lines.findIndex((l) =>
|
|
209
|
+
/^## (?:Claude Kit Workflow|Care Solutions AI Workflow)/.test(l)
|
|
210
|
+
);
|
|
211
|
+
let inFence = false;
|
|
212
|
+
let cut = -1;
|
|
213
|
+
for (let i = headingIdx + 1; i < lines.length; i++) {
|
|
214
|
+
if (/^\s*```/.test(lines[i])) {
|
|
215
|
+
inFence = !inFence;
|
|
216
|
+
continue;
|
|
217
|
+
}
|
|
218
|
+
if (inFence) continue;
|
|
219
|
+
if (/^## /.test(lines[i])) {
|
|
220
|
+
cut = i;
|
|
221
|
+
break;
|
|
222
|
+
}
|
|
223
|
+
}
|
|
224
|
+
|
|
225
|
+
return {
|
|
226
|
+
head,
|
|
227
|
+
preserved: cut === -1 ? '' : lines.slice(cut).join('\n').trim(),
|
|
228
|
+
recovered: cut !== -1,
|
|
229
|
+
};
|
|
230
|
+
}
|
|
231
|
+
|
|
164
232
|
const RETIRED_GLOBAL_AGENTS = ['manager.md', 'uat-generator.md', 'api-tester.md'];
|
|
165
233
|
const RETIRED_PROJECT_AGENTS = ['devops-tracker.md'];
|
|
234
|
+
// status.md was renamed to track.md in 2.1.58 — Claude Code ships a built-in /status,
|
|
235
|
+
// which shadowed the kit's command. The old file has to be deleted from existing
|
|
236
|
+
// installs or the collision never goes away, since installs only ever add files.
|
|
237
|
+
const RETIRED_COMMANDS = ['status.md'];
|
|
166
238
|
|
|
167
|
-
async function
|
|
239
|
+
async function removeRetiredFiles(dir, retired) {
|
|
168
240
|
for (const file of retired) {
|
|
169
|
-
const target = join(
|
|
241
|
+
const target = join(dir, file);
|
|
170
242
|
if (await fs.pathExists(target)) {
|
|
171
243
|
await fs.remove(target);
|
|
172
244
|
console.log(chalk.gray(` - ${file} (retired, removed)`));
|
|
@@ -190,7 +262,7 @@ async function installGlobalAgents() {
|
|
|
190
262
|
);
|
|
191
263
|
}
|
|
192
264
|
}
|
|
193
|
-
await
|
|
265
|
+
await removeRetiredFiles(globalAgentsDir, RETIRED_GLOBAL_AGENTS);
|
|
194
266
|
}
|
|
195
267
|
|
|
196
268
|
// ============================================================================
|
|
@@ -240,7 +312,7 @@ async function main() {
|
|
|
240
312
|
choices: [
|
|
241
313
|
{ name: 'Project Agents (deployer, db-admin)', value: 'agents', checked: true },
|
|
242
314
|
{ name: 'Hooks (secret blocker, auto-format, test suggestions)', value: 'hooks', checked: true },
|
|
243
|
-
{ name: 'Slash Commands (implement, review, deploy, releases, cherry-pick, promote, rollback,
|
|
315
|
+
{ name: 'Slash Commands (implement, review, deploy, releases, cherry-pick, promote, rollback, track, cleanup, create-work-item, …)', value: 'commands', checked: true },
|
|
244
316
|
{ name: 'MCP Servers (Playwright, DB, Teams, Stripe, Azure, Azure DevOps)', value: 'mcp', checked: true },
|
|
245
317
|
{ name: 'Settings (hook registration)', value: 'settings', checked: true },
|
|
246
318
|
{ name: 'CLAUDE.md Workflow Section', value: 'workflow', checked: true },
|
|
@@ -265,7 +337,7 @@ async function main() {
|
|
|
265
337
|
);
|
|
266
338
|
}
|
|
267
339
|
}
|
|
268
|
-
await
|
|
340
|
+
await removeRetiredFiles(agentsDir, RETIRED_PROJECT_AGENTS);
|
|
269
341
|
}
|
|
270
342
|
|
|
271
343
|
// ── Hooks ─────────────────────────────────────────────────────────────
|
|
@@ -301,6 +373,7 @@ async function main() {
|
|
|
301
373
|
);
|
|
302
374
|
}
|
|
303
375
|
}
|
|
376
|
+
await removeRetiredFiles(cmdsDir, RETIRED_COMMANDS);
|
|
304
377
|
}
|
|
305
378
|
|
|
306
379
|
// ── MCP Servers (interactive selection) ───────────────────────────────
|
|
@@ -515,13 +588,16 @@ async function main() {
|
|
|
515
588
|
'utf8'
|
|
516
589
|
);
|
|
517
590
|
|
|
591
|
+
// Delimited so a re-install replaces only the kit's own section.
|
|
592
|
+
const workflowBlock = `${workflowContent.trimEnd()}\n\n${WORKFLOW_END_MARKER}\n`;
|
|
593
|
+
|
|
518
594
|
const sensitiveDataPolicy = `## SENSITIVE DATA — MANDATORY RULE
|
|
519
595
|
|
|
520
596
|
**NEVER query, display, read, grep, or expose sensitive PII fields from the database or codebase — even if the values are encrypted.** Blocked fields: TIN, SSN, EIN, TaxId, BankAccountNumber, RoutingNumber, and any \`Encrypted*\` variants. Always use explicit inclusion projections listing only non-sensitive fields. Direct users to the application UI for sensitive data access.
|
|
521
597
|
`;
|
|
522
598
|
|
|
523
599
|
if (!await fs.pathExists(claudeMdPath)) {
|
|
524
|
-
await fs.writeFile(claudeMdPath, `# ${basename(targetDir)}\n\n${sensitiveDataPolicy}\n${
|
|
600
|
+
await fs.writeFile(claudeMdPath, `# ${basename(targetDir)}\n\n${sensitiveDataPolicy}\n${workflowBlock}`);
|
|
525
601
|
console.log(chalk.green(' ✓ Created CLAUDE.md with sensitive data policy and workflow'));
|
|
526
602
|
} else {
|
|
527
603
|
let existing = await fs.readFile(claudeMdPath, 'utf8');
|
|
@@ -547,8 +623,22 @@ async function main() {
|
|
|
547
623
|
if (existing.includes('Claude Kit Workflow') || existing.includes('Care Solutions AI Workflow')) {
|
|
548
624
|
// Check whether the existing workflow section matches the current template.
|
|
549
625
|
// If it does, nothing to do. If not, replace it (auto in --all, prompt otherwise).
|
|
550
|
-
const
|
|
551
|
-
const
|
|
626
|
+
const split = splitWorkflowSection(existing);
|
|
627
|
+
const { head, preserved, recovered } = split;
|
|
628
|
+
const expected = preserved
|
|
629
|
+
? `${head}\n\n${workflowBlock}\n${preserved}\n`
|
|
630
|
+
: `${head}\n\n${workflowBlock}`;
|
|
631
|
+
if (recovered && preserved) {
|
|
632
|
+
const names = (preserved.match(/^## .*$/gm) || []).map((h) => h.replace(/^##\s*/, ''));
|
|
633
|
+
console.log(
|
|
634
|
+
chalk.yellow(
|
|
635
|
+
` ! Kept project content found below the workflow section: ${names.join(', ') || 'unlabelled section'}`
|
|
636
|
+
)
|
|
637
|
+
);
|
|
638
|
+
console.log(
|
|
639
|
+
chalk.gray(' It now sits after the end marker, so future updates will leave it alone.')
|
|
640
|
+
);
|
|
641
|
+
}
|
|
552
642
|
if (existing.trim() === expected.trim()) {
|
|
553
643
|
console.log(chalk.gray(' = Workflow section up to date'));
|
|
554
644
|
} else if (installAll) {
|
|
@@ -569,7 +659,7 @@ async function main() {
|
|
|
569
659
|
}
|
|
570
660
|
}
|
|
571
661
|
} else {
|
|
572
|
-
await fs.appendFile(claudeMdPath, `\n\n${
|
|
662
|
+
await fs.appendFile(claudeMdPath, `\n\n${workflowBlock}`);
|
|
573
663
|
console.log(chalk.green(' ✓ Workflow appended to CLAUDE.md'));
|
|
574
664
|
}
|
|
575
665
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@chris1807/claude-kit",
|
|
3
|
-
"version": "2.1.
|
|
3
|
+
"version": "2.1.58",
|
|
4
4
|
"description": "Claude Code starter kit for Azure DevOps teams — agents, hooks, MCP servers, slash commands, and end-to-end work item → PR → release → deploy workflow automation",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: qa
|
|
3
|
+
description: Regression-tests a work item in a real browser via Playwright MCP. Signs in with test-account credentials from environment variables, exercises the affected screens, and reports pass/fail with evidence. Read-only against the codebase and against Azure DevOps.
|
|
4
|
+
tools: Read, Glob, Grep, Bash, mcp__playwright__browser_navigate, mcp__playwright__browser_navigate_back, mcp__playwright__browser_snapshot, mcp__playwright__browser_find, mcp__playwright__browser_click, mcp__playwright__browser_type, mcp__playwright__browser_fill_form, mcp__playwright__browser_select_option, mcp__playwright__browser_press_key, mcp__playwright__browser_hover, mcp__playwright__browser_wait_for, mcp__playwright__browser_resize, mcp__playwright__browser_handle_dialog, mcp__playwright__browser_console_messages, mcp__playwright__browser_network_requests, mcp__playwright__browser_take_screenshot, mcp__playwright__browser_tabs, mcp__playwright__browser_close
|
|
5
|
+
disallowedTools: Write, Edit
|
|
6
|
+
model: sonnet
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# QA Agent
|
|
10
|
+
|
|
11
|
+
You test a work item in a real browser against a running environment, then report what you found. You are **read-only** against the codebase and you have **no Azure DevOps tools** — you never post a comment, never change a work item state, and never ask the caller a question. You return one report; the main loop does the writing.
|
|
12
|
+
|
|
13
|
+
## Tools you deliberately do not have
|
|
14
|
+
|
|
15
|
+
Do not work around these. If a check genuinely needs one, report the check as `NOT TESTABLE` and say which capability was missing.
|
|
16
|
+
|
|
17
|
+
| Missing | Why it was withheld |
|
|
18
|
+
|---|---|
|
|
19
|
+
| `browser_evaluate`, `browser_run_code_unsafe` | Arbitrary page JS could read any DOM value and defeat every rule below. The omission is the enforcement. |
|
|
20
|
+
| `browser_network_request` (singular) | It returns full request and response bodies — the likeliest route for a TIN or SSN into a transcript. Use `browser_network_requests` (plural), which gives method, URL, and status. |
|
|
21
|
+
| `browser_file_upload`, `browser_drag`, `browser_drop` | Not needed by the checklist below. |
|
|
22
|
+
| Any `mcp__azure-devops__*` tool | You must be structurally unable to advance a work item. |
|
|
23
|
+
| `Write`, `Edit` | You test the app; you never change the code. |
|
|
24
|
+
|
|
25
|
+
## What you are given
|
|
26
|
+
|
|
27
|
+
The caller passes a brief containing: the work item id, title, type and state; the target environment name and its App URL (and API URL if known); a **baseline** environment URL for differential re-checks; the sign-in descriptor; the **names** of the environment variables holding the test account credentials; the ordered list of screens with each screen's changed files; and the acceptance criteria.
|
|
28
|
+
|
|
29
|
+
You are never given a password. You read it yourself.
|
|
30
|
+
|
|
31
|
+
## Signing in
|
|
32
|
+
|
|
33
|
+
1. Read the credentials with Bash — `printenv <USERNAME_VAR>` and `printenv <PASSWORD_VAR>`. If either is empty, stop immediately and report `BLOCKED — <VAR> is not set`.
|
|
34
|
+
2. `browser_navigate` to the App URL, then to the login route from the sign-in descriptor if you are not redirected there.
|
|
35
|
+
3. Fill both fields in **one** `browser_fill_form` call, then submit. One call, not two `browser_type` calls — the credential should appear once.
|
|
36
|
+
4. Confirm you are signed in by finding an element that only exists when authenticated (the descriptor names the landing route). If you land back on the login form, stop and report `BLOCKED — sign-in failed`. Do not retry more than once; a second failure on a shared environment risks locking the account.
|
|
37
|
+
5. If the browser is redirected to an external identity provider (`login.microsoftonline.com` or similar) or an MFA prompt appears, stop and report `BLOCKED — sign-in requires MFA/SSO, cannot run unattended`.
|
|
38
|
+
|
|
39
|
+
**Never** echo, log, repeat, screenshot, or include either credential value in your report. Never write one to a file.
|
|
40
|
+
|
|
41
|
+
## Regression Method
|
|
42
|
+
|
|
43
|
+
Work the screens in the order given. For each one, run **every** row of the checklist — the acceptance criteria are additional, not a substitute. Record a result per row before moving on.
|
|
44
|
+
|
|
45
|
+
| # | Test | How | Expected |
|
|
46
|
+
|---|------|-----|----------|
|
|
47
|
+
| 1 | Route loads | `browser_navigate` to the route | Renders. No error boundary, no blank region |
|
|
48
|
+
| 2 | Deep link | Navigate straight to the URL rather than clicking through | Route rehydrates; no bounce to login, no 404 |
|
|
49
|
+
| 3 | Console clean | `browser_console_messages` | No uncaught errors. React key/prop warnings are noted, not failed |
|
|
50
|
+
| 4 | Network clean | `browser_network_requests` | No 4xx/5xx to the API. Record `METHOD path → status` only |
|
|
51
|
+
| 5 | Data or empty state | `browser_snapshot` | Rows render, **or** an explicit empty-state message. A blank region is a **fail** |
|
|
52
|
+
| 6 | Filters, search, sort | Exercise each control once | Result set changes, no error, control reflects its state |
|
|
53
|
+
| 7 | Pagination | Next, then previous | Content changes; the indicator tracks |
|
|
54
|
+
| 8 | Form — invalid | Submit each form empty | **Inline** validation; nothing submitted |
|
|
55
|
+
| 9 | Form — valid | Submit valid data | Success indication; the list or detail updates |
|
|
56
|
+
| 10 | Form — cancel | Fill, then cancel | No mutation; returns to the prior view |
|
|
57
|
+
| 11 | Dialogs | Open and close each modal and row action | Opens, closes, focus returns. A native `window.confirm`/`alert`/`prompt` is a **fail** |
|
|
58
|
+
| 12 | Back button | Browser back after a mutation | Lands somewhere coherent; no stale or duplicated data |
|
|
59
|
+
| 13 | Responsive | `browser_resize` to 1440, then 768 | No overlap, no clipped controls, no horizontal scroll |
|
|
60
|
+
| 14 | Role gating | If a second role's credentials are named in the brief, sign in as it and revisit | Restricted controls are **absent**, not present-and-erroring |
|
|
61
|
+
|
|
62
|
+
Then one explicit test per acceptance criterion: `PASS`, `FAIL`, or `NOT TESTABLE (reason)`.
|
|
63
|
+
|
|
64
|
+
**Retry once before recording a failure**, using `browser_wait_for` on the element you expect rather than a fixed delay. A cold-start 502 on a Dev App Service is the most common false failure. If it passes on the retry, record `FLAKY`, not `PASS`.
|
|
65
|
+
|
|
66
|
+
**Differential re-check.** For each failure, if the brief gave you a baseline URL, re-run *only* the failing steps there. Fails on the baseline too → the issue predates this story. Passes on the baseline → this story introduced it. Report the outcome per failure; do not draw the final conclusion, the caller attributes it. Cap at 5 re-checks and say if you hit the cap.
|
|
67
|
+
|
|
68
|
+
## Write Safety
|
|
69
|
+
|
|
70
|
+
You are operating on a shared environment with real integrations. A wrong click here sends real money or real email.
|
|
71
|
+
|
|
72
|
+
- Prefer creating a new record over editing an existing one.
|
|
73
|
+
- Prefix everything you create with `QA AB#<id>` so a human can find it.
|
|
74
|
+
- **Never delete a pre-existing record.**
|
|
75
|
+
- **Never** fire a bulk or destructive action, a payment submission or settlement, a batch approval, a 1099 or tax-document generation, an export-all, or anything that emails or notifies a real person.
|
|
76
|
+
- If a screen can only be exercised by a destructive action, record `NOT TESTABLE — destructive` and move on.
|
|
77
|
+
- List everything you created in the report.
|
|
78
|
+
|
|
79
|
+
## Sensitive Data
|
|
80
|
+
|
|
81
|
+
The project's `CLAUDE.md` forbids exposing TIN, SSN, EIN, TaxId, BankAccountNumber, RoutingNumber, and any `Encrypted*` variant — including encrypted values. On a real screen these are rendered, so:
|
|
82
|
+
|
|
83
|
+
- **Never transcribe a field value** from a snapshot. Report only *present*, *populated*, *empty*, *masked*, or *validation shown*.
|
|
84
|
+
- Prefer `browser_find` over a full `browser_snapshot` when you only need to check one element on a screen that displays protected fields.
|
|
85
|
+
- **Never screenshot a screen showing a protected value.** Screenshots are failure evidence only, saved outside the repo, and referenced by local path — they are never attached to a work item.
|
|
86
|
+
- If a screen exists to display protected data, test its structure, validation, and navigation, and record the content checks as `NOT TESTABLE — sensitive screen`.
|
|
87
|
+
- If a tool result comes back blocked by the sensitive-data hook, do not retry it. Record `BLOCKED — protected fields on screen, manual QA required` and move to the next screen.
|
|
88
|
+
|
|
89
|
+
## Report Format
|
|
90
|
+
|
|
91
|
+
```markdown
|
|
92
|
+
## QA — AB#{id}: {title}
|
|
93
|
+
{type} · {state} · {environment} — {app-url}
|
|
94
|
+
Signed in as: {role} (from {USERNAME_VAR}) — value not shown
|
|
95
|
+
Screens tested: {n} · {route list}
|
|
96
|
+
|
|
97
|
+
### Acceptance Criteria
|
|
98
|
+
| # | Criterion | Result | Evidence |
|
|
99
|
+
|---|-----------|--------|----------|
|
|
100
|
+
| 1 | {criterion} | PASS | {route} — {what you observed} |
|
|
101
|
+
|
|
102
|
+
### Screen Regression — {route}
|
|
103
|
+
| # | Test | Steps | Expected Result | Result |
|
|
104
|
+
|---|------|-------|-----------------|--------|
|
|
105
|
+
| 1 | Route loads | Navigate to {route} | Renders, no error boundary | PASS |
|
|
106
|
+
|
|
107
|
+
### Failures
|
|
108
|
+
#### F1 — {short title}
|
|
109
|
+
- Screen: {route}
|
|
110
|
+
- Steps: 1. … 2. … 3. …
|
|
111
|
+
- Expected: {what should happen}
|
|
112
|
+
- Actual: {what happened}
|
|
113
|
+
- Evidence: console `{first line}` | network `POST /api/x → 500`
|
|
114
|
+
- Differential: {passes on baseline | fails on baseline too | not checked — {reason}}
|
|
115
|
+
- Touches this story's changed files: {yes — <file> | no}
|
|
116
|
+
- Screenshot: {local path}
|
|
117
|
+
|
|
118
|
+
### Not testable
|
|
119
|
+
- {check} — {why}
|
|
120
|
+
|
|
121
|
+
### Test data created
|
|
122
|
+
- {entity} "QA AB#{id} — …" (id {x})
|
|
123
|
+
|
|
124
|
+
**Sweep status:** COMPLETE | INCOMPLETE — {what was not reached}
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Rules
|
|
128
|
+
|
|
129
|
+
- Never modify any file. Never post anything anywhere. Never ask the caller a question — report and stop.
|
|
130
|
+
- Never change a work item state. You have no tool that can, and that is deliberate.
|
|
131
|
+
- Never echo a credential value, and never include one in the report.
|
|
132
|
+
- Never transcribe a protected field's value. See Sensitive Data.
|
|
133
|
+
- A control you could not find is a **FAIL**, never a PASS. "I didn't see it" is not "it works."
|
|
134
|
+
- If you run out of room and cannot finish the screens, report `INCOMPLETE` and name what you did not reach. A truncated sweep must never read as a pass.
|
|
135
|
+
- Report what you observed, not what you expected the code to do. You are testing the deployed build, not the source.
|
|
136
|
+
- Close the browser (`browser_close`) when you are done.
|
|
@@ -233,4 +233,4 @@ Cherry-pick is deployed to {environment} (pipeline {build-number}, succeeded).
|
|
|
233
233
|
Next: {the human step — "QA tests on Test and sets Ready for Staging" / "stakeholders verify on Staging and set Ready to Deploy" / "verify in production, then Close"}.
|
|
234
234
|
```
|
|
235
235
|
|
|
236
|
-
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/
|
|
236
|
+
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/track` will flag the lag later.
|
|
@@ -292,7 +292,7 @@ Release #{N} is on {environment} (pipeline {build-number}, succeeded).
|
|
|
292
292
|
Next: {the human step — "QA tests on Test and sets Ready for Staging" / "stakeholders verify on Staging and set Ready to Deploy" / "verify in production, then Close"}.
|
|
293
293
|
```
|
|
294
294
|
|
|
295
|
-
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/
|
|
295
|
+
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/track release {N}` will flag the lag later.
|
|
296
296
|
|
|
297
297
|
## Step 11: Notify Team (if Teams MCP is configured)
|
|
298
298
|
|
|
@@ -227,4 +227,4 @@ Report both fields, so a wrong assignment is easy to put back:
|
|
|
227
227
|
Next: {the human step — "QA tests on Test and sets Ready for Staging" / "stakeholders verify on Staging and set Ready to Deploy" / "verify in production, then Close"}.
|
|
228
228
|
```
|
|
229
229
|
|
|
230
|
-
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/
|
|
230
|
+
If the user said `stop watching`, or an update fails, leave the rest alone, say exactly which items were and were not updated, and note that `/track` will flag the lag later.
|
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
Regression-test a work item in a real browser against a running environment. Usage: `/qa <work-item-id> [environment] [--role=<role>] [--screens=/a,/b]`
|
|
2
|
+
|
|
3
|
+
Parse `$ARGUMENTS` to extract the work item ID. Accept formats like `AB#1234`, `#1234`, or just `1234`. If no ID is provided, report the error and stop.
|
|
4
|
+
|
|
5
|
+
- **Environment**: optional — `local`, `dev`, `test`, or `staging`. Defaults to `dev`.
|
|
6
|
+
- **`--role=<role>`**: optional — which test account to sign in as. Defaults to the row marked `Default` in the Environment URLs table, else the first row.
|
|
7
|
+
- **`--screens=/a,/b`**: optional — skip the affected-screen scoping in Step 5 and test exactly these routes.
|
|
8
|
+
|
|
9
|
+
**`prod` and `production` are refused.** An unattended regression pass signs in and clicks through live member data. Say so and stop.
|
|
10
|
+
|
|
11
|
+
## Step 1: Fetch the Work Item
|
|
12
|
+
|
|
13
|
+
Read the work item from Azure DevOps using the project from the current repo's CLAUDE.md configuration. Expand with `relations` so PR links are included.
|
|
14
|
+
|
|
15
|
+
If the work item is not found, report the error and stop.
|
|
16
|
+
|
|
17
|
+
Capture: id, title, type, state, and any linked Pull Request artifacts. Then extract the requirements **by type** — the field differs and getting this wrong silently produces an empty test plan:
|
|
18
|
+
|
|
19
|
+
- **User Story / Feature** → `Microsoft.VSTS.Common.AcceptanceCriteria`.
|
|
20
|
+
- **Bug / Hot Fix** → `Microsoft.VSTS.TCM.ReproSteps`. Neither type has an acceptance criteria field.
|
|
21
|
+
|
|
22
|
+
Treat all of these as **empty**: `""`, whitespace, `<div></div>`, `<p><br></p>`, and the process-template placeholder `💡 Tip: Add "@serena rewrite" to Description for AI suggestions Define acceptance criteria: - [ ] - [ ] - [ ]`. That last one is the trap — it is non-empty, so a length check says the opposite of the truth.
|
|
23
|
+
|
|
24
|
+
If the type's field is empty, fall back to `System.Description` and note `AC read from Description — misfiled` in the report. If both are empty, **continue anyway** and report `Acceptance criteria: none — regression sweep only`. The screen sweep is the point of `/qa`; it does not need acceptance criteria.
|
|
25
|
+
|
|
26
|
+
Download and view every embedded `<img>` in the description or criteria with WebFetch — mockups define the expected layout.
|
|
27
|
+
|
|
28
|
+
## Step 2: Resolve the Environment
|
|
29
|
+
|
|
30
|
+
Read the **Environment URLs** table from the current repo's `CLAUDE.md`:
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
| Branch | Environment | App URL | API URL |
|
|
34
|
+
|--------|------------|---------|---------|
|
|
35
|
+
| (none) | local | http://localhost:5173 | https://localhost:7001 |
|
|
36
|
+
| dev | Dev | https://dev.myapp.example.com | https://dev.api.myapp.example.com |
|
|
37
|
+
| test | Test | https://test.myapp.example.com | https://test.api.myapp.example.com |
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Capture the target row's App URL and API URL, the `**Sign-in:**` descriptor, the test-account table, and — for the differential re-check in Step 7 — the **App URL of the row immediately below the target**. That row serves the pre-change build, which is what lets a failure be attributed rather than guessed at.
|
|
41
|
+
|
|
42
|
+
If the section is missing, or the target environment has no row, report:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
AB#{id}: {title}
|
|
46
|
+
|
|
47
|
+
This project has no `## Environment URLs` section in CLAUDE.md, so I don't know where
|
|
48
|
+
the {environment} app is hosted. /qa cannot run.
|
|
49
|
+
|
|
50
|
+
Add this to CLAUDE.md — **above the `## Claude Kit Workflow` heading**, because
|
|
51
|
+
everything below that heading is replaced on the next kit update:
|
|
52
|
+
|
|
53
|
+
## Environment URLs
|
|
54
|
+
|
|
55
|
+
| Branch | Environment | App URL | API URL |
|
|
56
|
+
|--------|------------|---------|---------|
|
|
57
|
+
| dev | Dev | https://… | {inferred, or …} |
|
|
58
|
+
| test | Test | https://… | {inferred, or …} |
|
|
59
|
+
|
|
60
|
+
**Sign-in:** form at `/login`. Signed-in landing: `/dashboard`.
|
|
61
|
+
|
|
62
|
+
**Test accounts** — values live in environment variables; never write a password here.
|
|
63
|
+
|
|
64
|
+
| Role | Username variable | Password variable | Default |
|
|
65
|
+
|------|-------------------|-------------------|---------|
|
|
66
|
+
| Admin | `{PROJECT}_QA_ADMIN_USERNAME` | `{PROJECT}_QA_ADMIN_PASSWORD` | yes |
|
|
67
|
+
|
|
68
|
+
Want me to draft it with the API URLs filled in from your .env files? (yes / no)
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
The API URLs can be inferred from the client's per-environment env files (`Client/.env.development`, `.env.testing`, …), which already carry the API base URL. Grep for the single base-URL key and echo **only that value** — never print the contents of those files.
|
|
72
|
+
|
|
73
|
+
For `dev`, `test`, and `staging` also read the **Pipeline Configuration** table; the two tables are joined on the branch column, and a mismatch is a configuration error worth reporting. If Pipeline Configuration is missing, report that the project has no pipeline configuration in `CLAUDE.md` and stop.
|
|
74
|
+
|
|
75
|
+
## Step 3: Gate — Is a Browser Available?
|
|
76
|
+
|
|
77
|
+
Two checks, both required:
|
|
78
|
+
|
|
79
|
+
1. `.mcp.json` contains a `playwright` server. It is opt-in at install time, so a project that unchecked it has no browser.
|
|
80
|
+
2. `mcp__playwright__browser_navigate` is present in this session's tools. The server can be configured and still have failed to start.
|
|
81
|
+
|
|
82
|
+
```
|
|
83
|
+
AB#{id}: {title}
|
|
84
|
+
|
|
85
|
+
Playwright MCP isn't {configured for this project | connected in this session},
|
|
86
|
+
so I can't open a browser. /qa cannot run.
|
|
87
|
+
|
|
88
|
+
{To add it: npx @chris1807/claude-kit init . --all --db=<your-db> and check
|
|
89
|
+
"Playwright (browser testing)", then restart the session.}
|
|
90
|
+
{It's in .mcp.json but didn't connect. Restart the session; if it still fails, the
|
|
91
|
+
browser binaries may be missing — npx playwright install chromium}
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
## Step 4: Gate — Credentials and Sign-in Method
|
|
95
|
+
|
|
96
|
+
Resolve the role: `--role=<x>`, else the row marked `Default`, else the first row. Then check **presence only**. Never read a credential value in this step — the agent does that, inside its own transcript:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
for v in {PROJECT}_QA_ADMIN_USERNAME {PROJECT}_QA_ADMIN_PASSWORD; do
|
|
100
|
+
[ -n "${!v}" ] && echo "$v SET" || echo "$v MISSING"
|
|
101
|
+
done
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Any `MISSING`:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
AB#{id}: {title}
|
|
108
|
+
|
|
109
|
+
Missing test-account credentials for the {role} role:
|
|
110
|
+
- {PROJECT}_QA_ADMIN_PASSWORD
|
|
111
|
+
|
|
112
|
+
/qa signs in unattended, so these must be exported before it can run:
|
|
113
|
+
|
|
114
|
+
export {PROJECT}_QA_ADMIN_USERNAME="qa-admin@example.com"
|
|
115
|
+
export {PROJECT}_QA_ADMIN_PASSWORD="…"
|
|
116
|
+
|
|
117
|
+
Use a throwaway QA account with least privilege, never a personal login. Never put
|
|
118
|
+
the password in CLAUDE.md, .env, or any file in the repo.
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
If the `**Sign-in:**` descriptor says the app uses SSO or Entra ID with MFA, stop here:
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
{environment} sign-in goes through {provider} with MFA, which can't be completed
|
|
125
|
+
unattended. /qa can't test this project until a cloud-only, MFA-exempt QA account
|
|
126
|
+
exists — or run the checks by hand against {app-url}.
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
## Step 5: Gate — Is the Code Actually There?
|
|
130
|
+
|
|
131
|
+
### For `dev`, `test`, `staging`
|
|
132
|
+
|
|
133
|
+
Being merged is not being deployed. Two separate facts, both required.
|
|
134
|
+
|
|
135
|
+
**5a. Every commit is on the branch.** Same method as `/where`:
|
|
136
|
+
|
|
137
|
+
1. `git fetch origin <target-branch>` and `git fetch origin <baseline-branch>`.
|
|
138
|
+
2. Find the item's commits three ways and dedupe by SHA: linked PRs → `lastMergeCommit`; `git log --all --grep="AB#<id>\b" --grep="#<id>\b" -i --format=%H`; branch names matching `*AB#<id>-*`.
|
|
139
|
+
3. For each commit: `git merge-base --is-ancestor <commit> origin/<target-branch>`.
|
|
140
|
+
|
|
141
|
+
**`/qa` requires all of them.** A partial match is the cherry-pick case `/where` reports as `1 of 2`, and testing half a story produces failures that are really missing code — a FAIL comment that blames a developer for a deployment mistake:
|
|
142
|
+
|
|
143
|
+
```
|
|
144
|
+
AB#{id}: {title}
|
|
145
|
+
|
|
146
|
+
Not fully deployed to {environment} — 1 of 2 commits present.
|
|
147
|
+
|
|
148
|
+
Present on origin/{branch}:
|
|
149
|
+
a1b2c3d AB#{id}: add export button
|
|
150
|
+
Missing:
|
|
151
|
+
e4f5g6h AB#{id}: fix CSV encoding
|
|
152
|
+
|
|
153
|
+
This is a partial cherry-pick. Get the rest onto {branch} (`/promote` or
|
|
154
|
+
`/cherry-pick AB#{id} {environment}`), then re-run. `/where AB#{id}` shows the
|
|
155
|
+
full picture.
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
No commits at all → `No commits found referencing this work item — nothing to test.` and stop.
|
|
159
|
+
|
|
160
|
+
**5b. The pipeline that deployed them is green.** Same method as `/promote`:
|
|
161
|
+
|
|
162
|
+
1. Take the merge time of the item's newest commit on the target branch.
|
|
163
|
+
2. Look up the branch's pipeline ID(s) in Pipeline Configuration. Poll `pipelines_build` for runs on `refs/heads/<target-branch>` **queued at or after that time**, matching the merge commit where the run exposes it, and **capture each run's id**. A run that completed before the merge is not this deployment — never take "the latest run on the branch".
|
|
164
|
+
3. Every pipeline on the branch (API **and** client) must be `succeeded`.
|
|
165
|
+
4. **Never block on a foreground `sleep`.** If a run is still going, watch its **run id** with `Monitor` or Bash `run_in_background`, and match failures as well as success.
|
|
166
|
+
|
|
167
|
+
| Result | Action |
|
|
168
|
+
|---|---|
|
|
169
|
+
| all `succeeded` | Proceed. Record `{pipeline} run {id}` for the report. |
|
|
170
|
+
| `inProgress` | `The {environment} deployment is still running ({pipeline} run {id}). I'm watching it and will start testing when it's green. Say \`stop\` to cancel.` |
|
|
171
|
+
| `partiallySucceeded` | `Not green — {stage} failed in run {id}. Test anyway? (proceed / hold)` and record it in the report. |
|
|
172
|
+
| `failed` / `canceled` | Stop. `The {environment} deployment failed — run {id}, stage {stage}: {log link}. The app may not contain AB#{id} at all.` |
|
|
173
|
+
| no run after the merge | Stop. `The commits are on {branch} but no CD run was queued after the merge. Nothing deployed AB#{id}.` |
|
|
174
|
+
|
|
175
|
+
### For `local`
|
|
176
|
+
|
|
177
|
+
No commit or pipeline gate — you are testing the working tree. Instead, probe the local App URL. If nothing answers:
|
|
178
|
+
|
|
179
|
+
```
|
|
180
|
+
Nothing is listening on {local-url}.
|
|
181
|
+
|
|
182
|
+
/qa doesn't manage dev servers — start the app first:
|
|
183
|
+
{the project's start commands, e.g. `dotnet run --project src/API` and `npm run dev`}
|
|
184
|
+
|
|
185
|
+
Then re-run /qa AB#{id} local.
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Also check whether the working tree actually contains the item's commits (`git log --grep="AB#<id>"` on `HEAD`). If not, warn but continue — the user may be mid-implementation, which is a legitimate reason to run `/qa local`.
|
|
189
|
+
|
|
190
|
+
## Step 6: Scope the Affected Screens, Then Get Approval
|
|
191
|
+
|
|
192
|
+
Skip to the approval block if `--screens=` was given.
|
|
193
|
+
|
|
194
|
+
1. **Changed files** — union of `git show --name-only --format= <sha>` over the item's commits, cross-checked against the linked PR's iteration changes. If they disagree, prefer the union and note it.
|
|
195
|
+
2. **Partition** into frontend (`*.tsx`, `*.ts`, `*.css`, `*.scss` under the client project found via `Glob("**/package.json")`), backend (`*.cs`), and other.
|
|
196
|
+
3. **Find the router** — `Glob("**/src/routes/**", "**/src/{App,Routes,router}.{tsx,ts}")`, then grep for `<Route`, `createBrowserRouter`, `path:`, `lazy(` to build a `{path → component file}` map.
|
|
197
|
+
4. **Reverse-import walk, max 4 hops.** For each changed frontend file, grep the client `src/` for importers of its basename or path, and walk upward until you reach a file that is a router element. The union of reached routes is the candidate screen list. Keep the hop chain — it is the evidence for *why* a screen is in scope.
|
|
198
|
+
5. **Backend endpoints → screens.** For each changed `*.cs` controller, read its route attributes (`[HttpGet("…")]`, `[Route("…")]`) to get endpoint paths, grep the client for each path to find the calling component, and feed those components back into step 4. This is what catches a backend-only story that still has a UI surface.
|
|
199
|
+
6. **Name matching** — match nouns from the title and criteria against route paths and nav labels. Union in.
|
|
200
|
+
7. **Confidence:**
|
|
201
|
+
- 1–6 routes reached by a clean import chain → proceed, and list them.
|
|
202
|
+
- More than 6 → present them all, test the 6 with the most changed files, say plainly that you are capping, and mention `--screens=`.
|
|
203
|
+
- **0 routes → stop and ask.** Never silently test nothing:
|
|
204
|
+
```
|
|
205
|
+
I couldn't map the changed files to any screen. Changed: {n} frontend, {m} backend files.
|
|
206
|
+
{reason: no router found | only shared utilities changed | backend-only, no client caller found}
|
|
207
|
+
|
|
208
|
+
Which screens should I test? /qa AB#{id} --screens=/payments,/invoices
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Known blind spots, worth saying out loud in the report when they apply: computed `import()`, barrel-file re-exports that flatten the graph, and theme, design-token, or i18n changes that touch every screen. For a genuinely global change, say so and cap at the 4 highest-traffic routes rather than claiming to be exhaustive.
|
|
212
|
+
|
|
213
|
+
Then present the plan and **wait**:
|
|
214
|
+
|
|
215
|
+
```
|
|
216
|
+
## QA plan — AB#{id} on {environment}
|
|
217
|
+
|
|
218
|
+
Work item: {title} ({type}, {state})
|
|
219
|
+
Target: {environment} — {app-url}
|
|
220
|
+
Role: {role} ({USERNAME_VAR})
|
|
221
|
+
Build: {n} of {n} commits on `{branch}`, {pipeline} run {id} green
|
|
222
|
+
Baseline for attributing failures: {baseline-env} — {baseline-url}
|
|
223
|
+
|
|
224
|
+
Screens the story touched:
|
|
225
|
+
| Screen | Route | Why |
|
|
226
|
+
|--------|-------|-----|
|
|
227
|
+
| Payment History | /payments/history | PaymentHistory.tsx changed |
|
|
228
|
+
| Export dialog | /payments/history | new component |
|
|
229
|
+
| Payment detail | /payments/:id | calls the changed endpoint |
|
|
230
|
+
|
|
231
|
+
Full regression of these {n} screens — every control, not just the new behaviour —
|
|
232
|
+
plus {m} acceptance criteria, console errors, and failed network calls.
|
|
233
|
+
|
|
234
|
+
This will create test data on {environment}, prefixed `QA AB#{id}`.
|
|
235
|
+
|
|
236
|
+
Open the browser and run it? (yes / edit the screen list / cancel)
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
## Step 7: Run It, Then Attribute Every Failure
|
|
240
|
+
|
|
241
|
+
Delegate to the **qa** agent with a brief containing: the work item id, title, type and state; the target App and API URLs; the baseline App URL; the sign-in descriptor; the **names** of the credential variables (never the values — say "they are set, read them yourself, never echo them"); the ordered screen list with each screen's changed files; and the criteria. Above 3 screens, invoke it once per batch so a long sweep can't exhaust its context mid-run.
|
|
242
|
+
|
|
243
|
+
When the report comes back, sort every failure using three filters, in order:
|
|
244
|
+
|
|
245
|
+
1. **Blame filter** — if the failing surface is not in the story's changed-file set and no changed file reaches it in the import walk, it is **pre-existing**. State the reasoning.
|
|
246
|
+
2. **Differential result** — the agent re-ran failing steps against the baseline environment, which serves the pre-change build. Fails on both → **pre-existing**; passes on the baseline → **regression introduced by this story**, which is the strongest evidence a FAIL can carry. Caveat that seed-data and config drift between environments can confound this.
|
|
247
|
+
3. **Flake filter** — the agent retried once. A pass on retry is `FLAKY`, not a failure.
|
|
248
|
+
|
|
249
|
+
Only filter 1's output can drive a `FAIL` verdict. A run that finds nothing but pre-existing issues is `PASS WITH FINDINGS` — do not let old bugs turn into this story's failure.
|
|
250
|
+
|
|
251
|
+
## Step 8: Report
|
|
252
|
+
|
|
253
|
+
```
|
|
254
|
+
QA — AB#{id}: {title}
|
|
255
|
+
{type} · {state} · {environment} — {app-url}
|
|
256
|
+
Build: {pipeline} run {id} · succeeded · commit {sha7}
|
|
257
|
+
Signed in as: {role} (from {USERNAME_VAR}) — value not shown
|
|
258
|
+
Screens tested: {n} · {route list}
|
|
259
|
+
|
|
260
|
+
## Acceptance Criteria
|
|
261
|
+
| # | Criterion | Result | Evidence |
|
|
262
|
+
|---|-----------|--------|----------|
|
|
263
|
+
| 1 | {criterion} | PASS | {route} — {observation} |
|
|
264
|
+
| 2 | {criterion} | FAIL | see R1 |
|
|
265
|
+
| 3 | {criterion} | NOT TESTABLE | {reason} |
|
|
266
|
+
|
|
267
|
+
## Screen Regression — {route}
|
|
268
|
+
| # | Test | Steps | Expected Result | Result |
|
|
269
|
+
|---|------|-------|-----------------|--------|
|
|
270
|
+
| 1 | Route loads | Navigate to {route} | Renders, no error boundary | PASS |
|
|
271
|
+
|
|
272
|
+
## Regressions (attributable to AB#{id}) — {n}
|
|
273
|
+
### R1 — {short title}
|
|
274
|
+
- Screen: {route}
|
|
275
|
+
- Steps: 1. … 2. … 3. …
|
|
276
|
+
- Expected: {x}. Actual: {y}
|
|
277
|
+
- Evidence: console `{line}` | network `POST /api/x → 500`
|
|
278
|
+
- Differential: passes on {baseline-env} — introduced by this story
|
|
279
|
+
- Screenshot: {local path} — local only, not attached to the work item
|
|
280
|
+
|
|
281
|
+
## Pre-existing issues (found, not caused by AB#{id}) — {n}
|
|
282
|
+
- {route} — {issue}. Also fails on {baseline-env}. Recommend a separate Bug.
|
|
283
|
+
|
|
284
|
+
## Not testable — {n}
|
|
285
|
+
- {check} — {why}
|
|
286
|
+
|
|
287
|
+
## Test data created on {environment}
|
|
288
|
+
- {entity} "QA AB#{id} — …" (id {x}) — left in place for review
|
|
289
|
+
|
|
290
|
+
Verdict: PASS | PASS WITH FINDINGS | FAIL | INCOMPLETE
|
|
291
|
+
Work item state unchanged — `Ready for Staging` is yours to set.
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
## Step 9: Comment on the Work Item
|
|
295
|
+
|
|
296
|
+
Draft the comment, preview it, and **wait**. Before drafting, check the item's existing comments for a prior `/qa` result on the same commit SHA — three runs should not leave three near-identical comments. If one exists: `A /qa comment already exists for commit {sha7} ({date}). Post an updated one anyway? (yes / skip)`.
|
|
297
|
+
|
|
298
|
+
> **Why this comment is not the redundant kind.** `/rework` deliberately posts no summary comment, because the commits and the diff are already the record of what changed. A QA result duplicates nothing: it is the outcome of *executing* the software at a specific build in a specific environment, which exists in no diff, no PR, and no pipeline log. It is precisely the evidence the person who later sets `Ready for Staging` needs. Don't delete this step by applying the `/rework` rule uniformly.
|
|
299
|
+
|
|
300
|
+
The body, as HTML (Azure DevOps renders HTML in comments):
|
|
301
|
+
|
|
302
|
+
```html
|
|
303
|
+
<b>Automated QA — {verdict}</b> · {environment} · {date}<br>
|
|
304
|
+
Build: {pipeline} run {id} ({result}) · commit {sha7}<br>
|
|
305
|
+
Screens tested: {route list}<br>
|
|
306
|
+
Signed in as: {role} test account<br><br>
|
|
307
|
+
|
|
308
|
+
<b>Acceptance criteria — {x} of {y} pass</b>
|
|
309
|
+
<ul>
|
|
310
|
+
<li>✅ {criterion}</li>
|
|
311
|
+
<li>❌ {criterion} — {one-line reason}</li>
|
|
312
|
+
<li>➖ {criterion} — not testable: {reason}</li>
|
|
313
|
+
</ul>
|
|
314
|
+
|
|
315
|
+
<b>Regressions introduced by this story ({n})</b>
|
|
316
|
+
<ol>
|
|
317
|
+
<li><b>{title}</b> — {route}. Steps: {1-2-3}. Expected {x}, got {y}.
|
|
318
|
+
Passes on {baseline-env}, so it came in with this story.</li>
|
|
319
|
+
</ol>
|
|
320
|
+
|
|
321
|
+
<b>Pre-existing issues found ({n})</b> — not caused by this story; also fail on {baseline-env}.
|
|
322
|
+
<ul><li>{route} — {issue}. Separate Bug recommended.</li></ul>
|
|
323
|
+
|
|
324
|
+
<b>Test data created on {environment}</b>
|
|
325
|
+
<ul><li>{entity} "QA AB#{id} — …"</li></ul>
|
|
326
|
+
|
|
327
|
+
<hr>
|
|
328
|
+
<i>Posted by <code>/qa</code>. Work item state unchanged — <b>Ready for Staging</b> remains
|
|
329
|
+
a human QA gate. No screenshots and no field values are attached, per the sensitive-data
|
|
330
|
+
policy.</i>
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
A `PASS` collapses to the header, the criteria list, the test-data list, and the footer.
|
|
334
|
+
|
|
335
|
+
Then:
|
|
336
|
+
|
|
337
|
+
```
|
|
338
|
+
QA drafted — nothing has been posted to AB#{id} yet.
|
|
339
|
+
|
|
340
|
+
## Work item comment
|
|
341
|
+
{the full body as it will appear}
|
|
342
|
+
|
|
343
|
+
Reply with one of:
|
|
344
|
+
- "approve" → post the comment exactly as shown
|
|
345
|
+
- "skip" → post nothing (the report above is yours to keep)
|
|
346
|
+
- "edit <section>: <new text>" → revise, then I'll re-preview before posting
|
|
347
|
+
|
|
348
|
+
Wait for my response. Never post to the work item until I reply "approve".
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
On `approve`, post exactly one comment via `mcp__azure-devops__wit_work_item_comment_write`.
|
|
352
|
+
|
|
353
|
+
That's it. `/qa` never changes the work item's state — not `Ready for Testing`, not `Ready for Staging`, not anything. `Ready for Staging` is the human QA gate and no command in this kit sets it. `/qa` also never merges, pushes, creates a PR, triggers a pipeline, edits code, attaches a screenshot to the work item, or transcribes a field value from a screen. It reads, it clicks, it reports.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
Check the status of a release, pipeline, environment, or work item. Usage: `/
|
|
1
|
+
Check the status of a release, pipeline, environment, or work item. Usage: `/track <target>`
|
|
2
2
|
|
|
3
3
|
Parse `$ARGUMENTS` to determine what to check:
|
|
4
4
|
- **Release**: "release 24" or "r24" → show release status
|
|
@@ -23,7 +23,7 @@ Work item states track environments. Use this to spot **state lag** — an item
|
|
|
23
23
|
|
|
24
24
|
`Ready for Staging` and `Ready to Deploy` are human sign-offs and are never "lag". Only User Stories, Bugs, and Hot Fixes are checked — Features and Tasks have no environment state.
|
|
25
25
|
|
|
26
|
-
## If Release (e.g., `/
|
|
26
|
+
## If Release (e.g., `/track release 24`):
|
|
27
27
|
|
|
28
28
|
Query all work items tagged `release-{N}` or in the `Release #{N}` iteration. For each, find which environment branches contain its commits (`repo_search_commits` with `includeWorkItems: true`, or `git merge-base --is-ancestor`).
|
|
29
29
|
|
|
@@ -51,7 +51,7 @@ Catch these up? (yes/no)
|
|
|
51
51
|
|
|
52
52
|
If the user says yes, set `System.State` on each lagging item to the expected state via `wit_update_work_item` — never backward, never on a Feature or Task — and report `Was → Now`.
|
|
53
53
|
|
|
54
|
-
## If Pipeline (e.g., `/
|
|
54
|
+
## If Pipeline (e.g., `/track pipeline` or `/track build`):
|
|
55
55
|
|
|
56
56
|
Query recent builds for each pipeline in the Pipeline Configuration table.
|
|
57
57
|
|
|
@@ -67,7 +67,7 @@ Query recent builds for each pipeline in the Pipeline Configuration table.
|
|
|
67
67
|
|
|
68
68
|
A run on `main` would be unexpected — `main` has no pipeline. If one appears, say so.
|
|
69
69
|
|
|
70
|
-
## If Work Item (e.g., `/
|
|
70
|
+
## If Work Item (e.g., `/track AB#4521`):
|
|
71
71
|
|
|
72
72
|
Read the work item from Azure DevOps and show its full status, including which environment branches contain its commits.
|
|
73
73
|
|
|
@@ -102,7 +102,7 @@ State `Staging` matches the furthest environment. ✓
|
|
|
102
102
|
|
|
103
103
|
If the state lags the furthest environment, flag it and offer to catch it up as in the release view.
|
|
104
104
|
|
|
105
|
-
## If Environment (e.g., `/
|
|
105
|
+
## If Environment (e.g., `/track staging`):
|
|
106
106
|
|
|
107
107
|
Map the name to its branch through the Pipeline Configuration table. Show the last deployment and what's currently there.
|
|
108
108
|
|
|
@@ -123,9 +123,9 @@ Map the name to its branch through the Pipeline Configuration table. Show the la
|
|
|
123
123
|
{count} commits on `test` are not on `staging` — run `/promote test staging` or `/deploy-release <N> staging`.
|
|
124
124
|
```
|
|
125
125
|
|
|
126
|
-
`/
|
|
126
|
+
`/track main` is allowed but reports "compare branch — nothing deploys from main" and lists what's on `main` that hasn't been promoted to `dev`.
|
|
127
127
|
|
|
128
|
-
## If No Argument (e.g., `/
|
|
128
|
+
## If No Argument (e.g., `/track`):
|
|
129
129
|
|
|
130
130
|
Show a high-level overview.
|
|
131
131
|
|
|
@@ -88,6 +88,6 @@ Open PRs:
|
|
|
88
88
|
- PR #<n> → <target branch> (<status>)
|
|
89
89
|
```
|
|
90
90
|
|
|
91
|
-
Finally, compare the work item's state with its furthest environment (`dev` → `Ready for Testing`, `test` → `Testing`, `staging` → `Staging`, `prod` → `Deployed`). If the state lags — e.g. commits on `staging` but the item is still `Testing` — add one line: `⚠ State lags environment: Testing, expected at least Staging — /
|
|
91
|
+
Finally, compare the work item's state with its furthest environment (`dev` → `Ready for Testing`, `test` → `Testing`, `staging` → `Staging`, `prod` → `Deployed`). If the state lags — e.g. commits on `staging` but the item is still `Testing` — add one line: `⚠ State lags environment: Testing, expected at least Staging — /track AB#<id> can catch it up.`
|
|
92
92
|
|
|
93
93
|
That's it — read-only. Do not modify branches, push, create PRs, or change states.
|
|
@@ -1,12 +1,29 @@
|
|
|
1
1
|
#!/bin/bash
|
|
2
|
-
# Sensitive Data Output Blocker Hook (PostToolUse - Bash, Read, Grep, MCP DB tools)
|
|
3
|
-
# Scans tool output for sensitive PII
|
|
4
|
-
#
|
|
2
|
+
# Sensitive Data Output Blocker Hook (PostToolUse - Bash, Read, Grep, Playwright, MCP DB tools)
|
|
3
|
+
# Scans tool output for sensitive PII that may have been returned by broad database
|
|
4
|
+
# queries, file reads of seed/dump data, grep results, or a rendered browser page.
|
|
5
5
|
# Exit code 2 = BLOCK (prevents the output from being used).
|
|
6
|
+
#
|
|
7
|
+
# Two pattern sets, chosen by tool:
|
|
8
|
+
# * Database / file / grep tools -> match sensitive field NAMES. A query or a seed
|
|
9
|
+
# file that names TIN or SSN at all is already returning data it shouldn't.
|
|
10
|
+
# * Playwright (mcp__playwright__*) -> match a field name only when followed by
|
|
11
|
+
# something VALUE-shaped. A rendered screen legitimately shows "SSN:" as a form
|
|
12
|
+
# label, and blocking on the label alone would make every such screen untestable
|
|
13
|
+
# and send /qa into a retry loop.
|
|
6
14
|
|
|
7
15
|
# Read the tool result from stdin
|
|
8
16
|
INPUT=$(cat)
|
|
9
17
|
|
|
18
|
+
# Extract the tool name so the pattern set can be chosen below
|
|
19
|
+
TOOL=$(echo "$INPUT" | python3 -c "
|
|
20
|
+
import sys, json
|
|
21
|
+
try:
|
|
22
|
+
print(json.load(sys.stdin).get('tool_name', ''))
|
|
23
|
+
except:
|
|
24
|
+
pass
|
|
25
|
+
" 2>/dev/null)
|
|
26
|
+
|
|
10
27
|
# Extract output from any tool type (Bash stdout, Read content, Grep results, MCP results)
|
|
11
28
|
OUTPUT=$(echo "$INPUT" | python3 -c "
|
|
12
29
|
import sys, json
|
|
@@ -38,24 +55,45 @@ if [ -z "$OUTPUT" ]; then
|
|
|
38
55
|
exit 0
|
|
39
56
|
fi
|
|
40
57
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
#
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
58
|
+
FIELDS='TIN|Tin|TaxId|TaxIdentificationNumber|EIN|SSN|SocialSecurityNumber|EncryptedTin|EncryptedSSN|EncryptedTaxId|BankAccountNumber|AccountNumber|RoutingNumber'
|
|
59
|
+
|
|
60
|
+
if [[ "$TOOL" == mcp__playwright__* ]]; then
|
|
61
|
+
# A rendered page: block only when a value actually accompanies the field name.
|
|
62
|
+
SENSITIVE_PATTERNS=(
|
|
63
|
+
# JSON body surfaced through the browser: "SSN": "123-45-6789"
|
|
64
|
+
"\"($FIELDS)\"\s*:\s*\"?[0-9]"
|
|
65
|
+
# Rendered value: SSN: 123-45-6789 / TIN = 12-3456789 / Account #: ****1234.
|
|
66
|
+
# Requires at least one real digit, so a fully masked "***-**-****" is allowed
|
|
67
|
+
# through -- that carries no data, and /qa needs to verify masking works.
|
|
68
|
+
"($FIELDS)\s*#?\s*[:=]\s*[\"']?[*Xx-]{0,8}[0-9][0-9*Xx-]{2,}"
|
|
69
|
+
# A bare SSN or EIN anywhere on the page, label or not
|
|
70
|
+
'\b[0-9]{3}-[0-9]{2}-[0-9]{4}\b'
|
|
71
|
+
'\b[0-9]{2}-[0-9]{7}\b'
|
|
72
|
+
)
|
|
73
|
+
BLOCK_MSG="BLOCKED: Browser output contains a sensitive value (SSN, TIN, bank account, etc.)"
|
|
74
|
+
BLOCK_HELP="Do not retry this snapshot. Record the screen as 'BLOCKED — protected fields on screen,
|
|
75
|
+
manual QA required', test only its structure/validation/navigation, and move on.
|
|
76
|
+
Never transcribe a protected field's value into a report or a work item comment."
|
|
77
|
+
else
|
|
78
|
+
# Database / file / grep output: the field NAME appearing at all means PII was returned.
|
|
79
|
+
SENSITIVE_PATTERNS=(
|
|
80
|
+
# JSON/MongoDB style: "TIN": or 'TIN':
|
|
81
|
+
"\"($FIELDS)\"\s*:"
|
|
82
|
+
# C# property style: .TIN = or .TaxId =
|
|
83
|
+
"\.($FIELDS)\s*="
|
|
84
|
+
# YAML/config style: TIN: (start of line or after whitespace)
|
|
85
|
+
"(^|\s)($FIELDS):\s"
|
|
86
|
+
)
|
|
87
|
+
BLOCK_MSG="BLOCKED: Output contains sensitive PII fields (TIN, SSN, bank account, etc.)"
|
|
88
|
+
BLOCK_HELP="For database queries: use an explicit inclusion projection listing only non-sensitive fields.
|
|
89
|
+
For code searches: avoid reading seed data, test fixtures, or dump files containing PII.
|
|
90
|
+
If you need to work with this data, use the application UI instead."
|
|
91
|
+
fi
|
|
51
92
|
|
|
52
93
|
for PATTERN in "${SENSITIVE_PATTERNS[@]}"; do
|
|
53
94
|
if echo "$OUTPUT" | grep -qE "$PATTERN"; then
|
|
54
|
-
echo "
|
|
55
|
-
echo "
|
|
56
|
-
echo "For database queries: use an explicit inclusion projection listing only non-sensitive fields."
|
|
57
|
-
echo "For code searches: avoid reading seed data, test fixtures, or dump files containing PII."
|
|
58
|
-
echo "If you need to work with this data, use the application UI instead."
|
|
95
|
+
echo "$BLOCK_MSG"
|
|
96
|
+
echo "$BLOCK_HELP"
|
|
59
97
|
exit 2
|
|
60
98
|
fi
|
|
61
99
|
done
|
|
@@ -7,3 +7,4 @@
|
|
|
7
7
|
|
|
8
8
|
echo "Reminder: If you just completed a feature implementation, run UAT before marking it done."
|
|
9
9
|
echo "Ask Claude to generate a UAT checklist from the work item's acceptance criteria."
|
|
10
|
+
echo "Once it is merged and deployed, /qa AB#<id> runs an automated browser regression of the affected screens."
|
|
@@ -18,6 +18,7 @@ You (give task)
|
|
|
18
18
|
├── deployer → commits, pushes, triggers CD pipeline
|
|
19
19
|
├── db-admin → queries/fixes MongoDB data
|
|
20
20
|
├── test-runner → runs xUnit, Vitest, Playwright tests
|
|
21
|
+
├── qa → drives a real browser on a deployed environment, regression-tests the affected screens
|
|
21
22
|
├── build-validator → confirms builds pass
|
|
22
23
|
├── lint-checker → runs ESLint and dotnet format
|
|
23
24
|
├── security-auditor → scans for secrets, vulnerabilities
|
|
@@ -46,7 +47,7 @@ Ultracode is **opt-in**. It is on only when a system-reminder confirms it, when
|
|
|
46
47
|
| Multi-file or cross-layer review | one agent per file/dimension, then adversarial verify before reporting |
|
|
47
48
|
| Repo-wide sweeps (rename, dependency bump, pattern migration) | one agent per site, worktree-isolated |
|
|
48
49
|
|
|
49
|
-
Short, single-query operations (`/
|
|
50
|
+
Short, single-query operations (`/track`, `/explain`, `/close-orphan-tasks`) do **not** need ultracode — they are already one pass and gain nothing from fan-out. Reach for `Workflow` when the work-list is large and the per-item work is independent.
|
|
50
51
|
|
|
51
52
|
**`/rework` always uses ultracode** — it does not wait to be asked. It fans out feedback gathering, codebase exploration, per-acceptance-criterion coverage checks, and the find → adversarially-verify review, while keeping every approval gate and write in the main loop.
|
|
52
53
|
|
|
@@ -79,7 +80,7 @@ These run automatically — no action needed:
|
|
|
79
80
|
| Server | What It Does |
|
|
80
81
|
|--------|-------------|
|
|
81
82
|
| **Azure DevOps** *(core — drives every slash command)* | Work items, repos, pull requests, pipelines, wiki, test plans, advanced security |
|
|
82
|
-
| **Playwright** | Browser testing (navigate, click, fill, screenshot) |
|
|
83
|
+
| **Playwright** | Browser testing (navigate, click, fill, screenshot) — **required by `/qa`** |
|
|
83
84
|
| **MongoDB** | Direct database queries and updates |
|
|
84
85
|
| **Microsoft Teams** | Send/read team messages and notifications |
|
|
85
86
|
| **Stripe** | Payment management (when configured) |
|
|
@@ -170,12 +171,14 @@ Every environment in the chain has a work item state, so the board shows where e
|
|
|
170
171
|
| `staging` | Staging | `Ready for Staging` | `Staging` |
|
|
171
172
|
| `prod` | Production | `Ready to Deploy` | `Deployed` |
|
|
172
173
|
|
|
174
|
+
`/qa` is the exception that proves the rule: it tests a work item in a browser and posts a pass/fail comment, but it **never writes a state** — not even `Ready for Testing`.
|
|
175
|
+
|
|
173
176
|
Human-only transitions — no slash command ever makes these: `Testing → Ready for Staging` (QA sign-off), `Staging → Ready to Deploy` (stakeholder / UAT sign-off), `Deployed → Closed` (verified in production).
|
|
174
177
|
|
|
175
178
|
How the commands use this table:
|
|
176
179
|
|
|
177
180
|
- **Gate check before the PR.** `/deploy-release`, `/cherry-pick`, and `/promote` compare each carried work item's state against the gate for the target environment. Anything behind the gate (e.g. still `Testing` when deploying to `staging`) is flagged and the user decides whether to include it. Items *ahead* of the gate (e.g. already `Deployed` on a cherry-pick to `prod`) are reported and left alone.
|
|
178
|
-
- **Advance on a green pipeline, automatically.** `/promote`, `/cherry-pick`, and `/deploy-release` do not stop at the PR. They poll it until it is `completed`, then poll the target branch's CD pipeline until every run finishes. Only on `succeeded` do they write the new state — for every carried User Story, Bug, and Hot Fix, never a Feature or Task, and never backward. A failed or canceled run changes **nothing** and is reported with the log link. `/
|
|
181
|
+
- **Advance on a green pipeline, automatically.** `/promote`, `/cherry-pick`, and `/deploy-release` do not stop at the PR. They poll it until it is `completed`, then poll the target branch's CD pipeline until every run finishes. Only on `succeeded` do they write the new state — for every carried User Story, Bug, and Hot Fix, never a Feature or Task, and never backward. A failed or canceled run changes **nothing** and is reported with the log link. `/track` reports items whose state lags the branch their commits are on and offers to catch them up.
|
|
179
182
|
- **Assign the verifier at the same time.** The state change hands the item to a person, so the same three commands ask **before the merge** who each group of items should go to, then apply the assignee alongside the state once the pipeline is green. See **Verifier Assignment** below.
|
|
180
183
|
- **Never assume the state list.** Confirm with `mcp__azure-devops__wit_work_item` (`action: get_type`) on an unfamiliar project before writing a state name — a bad name is accepted silently by a query and rejected only at write time.
|
|
181
184
|
|
|
@@ -304,8 +307,9 @@ All deployment and release operations are available as slash commands:
|
|
|
304
307
|
| `/cherry-pick` | `/cherry-pick AB#1234 AB#1235 prod` | Gate-check → cherry-pick specific work items → PR → ask who verifies → advance + assign on a green pipeline |
|
|
305
308
|
| `/promote` | `/promote main dev` | Promote all code between environments (`main → dev → test → staging → prod`) → ask who verifies → advance + assign on a green pipeline |
|
|
306
309
|
| `/rollback` | `/rollback AB#1234 prod` | Revert commits on an environment |
|
|
307
|
-
| `/
|
|
310
|
+
| `/track` | `/track release 24` | Check release, pipeline, environment, or work item status; flags work items whose state lags their environment |
|
|
308
311
|
| `/where` | `/where AB#1234` | Show which environment branches contain a work item's commits |
|
|
312
|
+
| `/qa` | `/qa AB#1234 [env]` | Verify the item is fully deployed with a green pipeline → open the app in a real browser → sign in as a test account → full regression of the screens the story touched → pass/fail comment (never a state change) |
|
|
309
313
|
| `/plan-backlog` | `/plan-backlog [project]` | Sweep backlog for Dev Ready stories with points and no tasks → propose child tasks with hours |
|
|
310
314
|
| `/plan-sprint` | `/plan-sprint [project]` | Sweep the current sprint for stories/bugs with no child tasks → propose one child task with hours per item |
|
|
311
315
|
| `/quote-backlog` | `/quote-backlog [project]` | Sweep backlog for unpointed items ready to estimate — stories in `Design Approved`, **bugs in `New`** (bugs have no design states) → review completeness, check for duplicates, suggest rewrites, propose points + creator comments (10 at a time, approval-gated). Pointed items move to Dev Ready; stories that can't be quoted move back to **Design Review**, bugs that can't be quoted get a **`needs-info`** tag, so the next sweep skips them. Every run also audits the drop-out queue (tagged bugs **and** stories bounced to Design Review) and reports any whose creator answered but which nobody returned to the sweep — neither mechanism expires, so those are otherwise invisible forever |
|
|
@@ -319,7 +323,7 @@ The CD pipeline is only triggered manually when pushing directly to an environme
|
|
|
319
323
|
|
|
320
324
|
### Pipeline Configuration
|
|
321
325
|
|
|
322
|
-
Each project must define its compare branch, its environment chain, and its pipeline IDs so the slash commands (`/deploy`, `/promote`, `/deploy-release`, `/cherry-pick`, `/where`, `/
|
|
326
|
+
Each project must define its compare branch, its environment chain, and its pipeline IDs so the slash commands (`/deploy`, `/promote`, `/deploy-release`, `/cherry-pick`, `/where`, `/track`) know which pipelines to trigger and what the promotion order is.
|
|
323
327
|
|
|
324
328
|
Add this section to your project's `CLAUDE.md`:
|
|
325
329
|
|
|
@@ -341,7 +345,7 @@ Add this section to your project's `CLAUDE.md`:
|
|
|
341
345
|
- `/deploy` triggers the pipeline(s) listed for the current branch. If the current branch is not in this table, or its pipeline cell is `—`, no pipeline is triggered.
|
|
342
346
|
- `/promote` uses this table to determine the next environment (the next row down).
|
|
343
347
|
- `/deploy-release` and `/cherry-pick` create PRs targeting environment branches listed here, and use the **Work Item States ↔ Environments** table to gate-check and advance work items.
|
|
344
|
-
- `/where` and `/
|
|
348
|
+
- `/where` and `/track` check every row; the compare branch reports as "merged to main", not as a deployment.
|
|
345
349
|
|
|
346
350
|
**Examples from actual projects:**
|
|
347
351
|
|
|
@@ -375,6 +379,53 @@ Glasswing and Monarch (not yet migrated — 1 environment):
|
|
|
375
379
|
| develop | Dev | CD - Development (28) |
|
|
376
380
|
```
|
|
377
381
|
|
|
382
|
+
### Environment URLs
|
|
383
|
+
|
|
384
|
+
`/qa` opens the app in a real browser, so it needs to know where each environment is hosted and which test account to sign in as. Without this section `/qa` reports and stops rather than guessing a hostname.
|
|
385
|
+
|
|
386
|
+
Add this to your project's `CLAUDE.md` — **above the `## Claude Kit Workflow` heading.** Everything from that heading to the end of the file is replaced on each kit update, so configuration placed below it is lost.
|
|
387
|
+
|
|
388
|
+
```markdown
|
|
389
|
+
## Environment URLs
|
|
390
|
+
|
|
391
|
+
| Branch | Environment | App URL | API URL |
|
|
392
|
+
|--------|------------|---------|---------|
|
|
393
|
+
| (none) | local | http://localhost:5173 | https://localhost:7001 |
|
|
394
|
+
| dev | Dev | https://dev.myapp.example.com | https://dev.api.myapp.example.com |
|
|
395
|
+
| test | Test | https://test.myapp.example.com | https://test.api.myapp.example.com |
|
|
396
|
+
| staging | Staging | https://staging.myapp.example.com | https://staging.api.myapp.example.com |
|
|
397
|
+
|
|
398
|
+
**Sign-in:** form at `/login` — app-issued JWT, no SSO. Signed-in landing: `/dashboard`.
|
|
399
|
+
|
|
400
|
+
**Test accounts** — values live in environment variables; never write a password here.
|
|
401
|
+
|
|
402
|
+
| Role | Username variable | Password variable | Default |
|
|
403
|
+
|------|-------------------|-------------------|---------|
|
|
404
|
+
| Admin | `MYAPP_QA_ADMIN_USERNAME` | `MYAPP_QA_ADMIN_PASSWORD` | yes |
|
|
405
|
+
| Provider | `MYAPP_QA_PROVIDER_USERNAME` | `MYAPP_QA_PROVIDER_PASSWORD` | |
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
**Rules for slash commands:**
|
|
409
|
+
- **Branch names must match the `## Pipeline Configuration` table** row for row. The two tables are joined on the branch column — `/qa` reads the URL here and the pipeline ID there. A mismatch is a configuration error worth reporting.
|
|
410
|
+
- **There is no `prod` row, deliberately.** `/qa` refuses production: an unattended regression pass signs in and clicks through live member data.
|
|
411
|
+
- **Include the rows below the one you test.** The row immediately below the target serves the pre-change build, which is how `/qa` tells a regression this story caused from a bug that was already there.
|
|
412
|
+
- **App URL is required; API URL is optional** — it is used to attribute a failing network call to the backend rather than the screen.
|
|
413
|
+
- **`Sign-in:`** is optional prose. Include it when the login route is not `/login`, when the landing page is not obvious, or — importantly — when sign-in goes through **SSO / Entra ID with MFA**. Say so, and `/qa` reports that it cannot sign in unattended instead of hanging on a browser prompt.
|
|
414
|
+
- **Test accounts** are required by `/qa` and nothing else. The `Default` column marks the account used when `--role` is not given; if no row is marked, the first row wins. A second role lets `/qa` verify that restricted controls are correctly *absent* — a class of bug a manual checklist rarely catches.
|
|
415
|
+
- **Never put a credential in this table.** Only variable names. The values belong in your shell profile — see Environment Setup below.
|
|
416
|
+
|
|
417
|
+
**Example from an actual project:**
|
|
418
|
+
|
|
419
|
+
COMPASS:
|
|
420
|
+
```markdown
|
|
421
|
+
| Branch | Environment | App URL | API URL |
|
|
422
|
+
|--------|------------|---------|---------|
|
|
423
|
+
| (none) | local | http://localhost:5173 | https://localhost:7169 |
|
|
424
|
+
| dev | Dev | https://dev.compass.caresolutions.com | https://dev.api.compass.caresolutions.com |
|
|
425
|
+
| test | Test | https://test.compass.caresolutions.com | https://test.api.compass.caresolutions.com |
|
|
426
|
+
| staging | Staging | https://staging.compass.caresolutions.com | https://staging.api.compass.caresolutions.com |
|
|
427
|
+
```
|
|
428
|
+
|
|
378
429
|
### Environment Setup
|
|
379
430
|
|
|
380
431
|
Each team member needs to set their own environment variables (never commit these):
|
|
@@ -391,4 +442,15 @@ az login
|
|
|
391
442
|
|
|
392
443
|
# Microsoft Teams MCP — no env vars needed. First invocation prints a
|
|
393
444
|
# device code + URL to sign in via Microsoft Graph (OAuth device flow).
|
|
445
|
+
|
|
446
|
+
# QA test accounts (required for /qa — browser testing on a deployed environment).
|
|
447
|
+
# One pair per role. The variable NAMES are declared in the project's
|
|
448
|
+
# `## Environment URLs` section; the VALUES live only here, never in any file.
|
|
449
|
+
# Use throwaway QA accounts with least privilege, never a personal login.
|
|
450
|
+
export COMPASS_QA_ADMIN_USERNAME="qa-admin@example.com"
|
|
451
|
+
export COMPASS_QA_ADMIN_PASSWORD="..."
|
|
452
|
+
export COMPASS_QA_PROVIDER_USERNAME="qa-provider@example.com"
|
|
453
|
+
export COMPASS_QA_PROVIDER_PASSWORD="..."
|
|
394
454
|
```
|
|
455
|
+
|
|
456
|
+
`/qa` reads these itself **inside a subagent** and fills the login form directly. It never prints a value, never writes one to a file, and never includes one in a report or a work item comment. If your app's sign-in requires MFA or SSO, unattended login is not possible — say so in the `Sign-in:` line of `## Environment URLs` and `/qa` will report that instead of hanging on a browser prompt.
|
|
@@ -74,6 +74,15 @@
|
|
|
74
74
|
"command": "bash .claude/hooks/sensitive-data-output-blocker.sh"
|
|
75
75
|
}
|
|
76
76
|
]
|
|
77
|
+
},
|
|
78
|
+
{
|
|
79
|
+
"matcher": "mcp__playwright",
|
|
80
|
+
"hooks": [
|
|
81
|
+
{
|
|
82
|
+
"type": "command",
|
|
83
|
+
"command": "bash .claude/hooks/sensitive-data-output-blocker.sh"
|
|
84
|
+
}
|
|
85
|
+
]
|
|
77
86
|
}
|
|
78
87
|
],
|
|
79
88
|
"Stop": [
|