baldart 3.40.0 → 3.41.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +1 -1
- package/VERSION +1 -1
- package/framework/.claude/agents/REGISTRY.md +8 -6
- package/framework/.claude/agents/qa-sentinel.md +50 -22
- package/framework/.claude/commands/new.md +7 -5
- package/framework/.claude/commands/qa.md +7 -2
- package/framework/.claude/skills/new/SKILL.md +12 -8
- package/framework/agents/index.md +2 -1
- package/framework/agents/testing.md +26 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,23 @@ All notable changes to BALDART will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.41.0] - 2026-06-01
|
|
9
|
+
|
|
10
|
+
**`qa-sentinel` is now profile-driven: `balanced` cards run SCOPED tests (only the tests related to the touched files), and the FULL suite is reserved for `deep` cards (or a `balanced` change whose diff reveals undeclared risk).** This is a **behavioral change to QA depth** — the default for an ordinary feature/refactor changes from "run the whole suite" to "run the related tests" — not an additive feature. **No new `baldart.config.yml` keys** — it reuses the existing `review_profile` field; scoped/smoke test selection is auto-probed at runtime from the project's test runner.
|
|
11
|
+
|
|
12
|
+
> **Why.** The `review_profile` (skip/light/balanced/deep) is computed deterministically upstream by `prd-card-writer` (Rule C) and passed to `qa-sentinel` by `/new` — but the agent **ignored it**. Its prompt literally said `Run FULL VALIDATION MODE` while the line below passed `QA profile: balanced`, and the agent re-decided QUICK-vs-FULL from a crude `>5 files` heuristic. Net effect: a 6-file UI-only `balanced` card triggered the entire regression suite. The fix makes the profile **authoritative**: it selects the tier; file *count* no longer escalates (the framework already treats file count as advisory-only — `prd-card-writer.md`). The only upgrade path is **risk drift** — the diff itself touching auth/permission/payment/schema/migration/API-contract — which catches a card the author under-classified. An adversarial review of the first design rejected three over-reaches kept out of this release: putting the executable tier table in `agents/testing.md` (the agent never reads it), a ghost `SMOKE` tier mapped to `light` (which never invokes the agent), and a `>15 files` escalation (re-introducing the very full-too-often problem).
|
|
13
|
+
|
|
14
|
+
### Changed — `qa-sentinel` honors `review_profile` (scoped-by-default)
|
|
15
|
+
|
|
16
|
+
- **[framework/.claude/agents/qa-sentinel.md](framework/.claude/agents/qa-sentinel.md)** — Step 0 + `Operating Modes`: tier is now selected from the `QA profile` in the prompt, not file count. MODE 1 is **SCOPED VALIDATION** (profile `balanced`): a stack-aware test-impact cascade (`jest --findRelatedTests` → `vitest related` → `pytest <paths>` → `--testPathPattern`, monorepo-workspace-aware) that runs only related tests and **never** falls through to the full suite. MODE 2 is **FULL VALIDATION** (profile `deep`, or `balanced` escalated by risk drift). Confidence rule added: a SCOPED run that finds no related tests for a *logic* change records `Tests: SKIP — no related tests` and caps confidence at ≤70% instead of reporting PASS 95%. The H2 headings are unchanged (preserves consumer agent overlays); only the section bodies were rewritten.
|
|
17
|
+
- **[framework/.claude/skills/new/SKILL.md](framework/.claude/skills/new/SKILL.md)** — Phase 3.5 qa-sentinel prompt no longer says "FULL VALIDATION MODE"; it passes the tier contract (`balanced → SCOPED`, `deep → FULL`, risk-drift-only escalation) and derives the E2E depth from the profile (`deep` full+blocking, `balanced` `--grep` advisory). Duplicate step "22" renumbered. Team-mode **D.4** documents that a mixed group with any `deep` card runs FULL (max-profile intentionally overrides the per-card SCOPED tier).
|
|
18
|
+
- **[framework/.claude/commands/qa.md](framework/.claude/commands/qa.md)** — `/qa` now passes `QA profile` to qa-sentinel (so the agent never defaults to absent), and the LIGHT path no longer falls back to the full suite when the test pattern is unclear.
|
|
19
|
+
|
|
20
|
+
### Changed — documentation aligned
|
|
21
|
+
|
|
22
|
+
- **[framework/agents/testing.md](framework/agents/testing.md)** — new stack-neutral **Scope-aware Test Selection** section documenting the principle (balanced→scoped / deep→full / risk-drift→escalate), explicitly pointing at `qa-sentinel` as the executable SSOT and fencing TIA as heuristic (no coverage maps / AST / ML). Runner-specific commands deliberately kept out (Codex / non-Node consumers read this module).
|
|
23
|
+
- **[framework/agents/index.md](framework/agents/index.md)** + **[README.md](README.md)** — routing pointer and the qa-sentinel one-liner updated (`Quick/Full/Deep profiles` → `profile-driven: scoped-by-default, full on deep`).
|
|
24
|
+
|
|
8
25
|
## [3.40.0] - 2026-06-01
|
|
9
26
|
|
|
10
27
|
**Doc fixes surfaced by `/new`'s doc-review are now applied by the `doc-reviewer` itself, not delegated to `coder`.** The `doc` fix-domain owner changes from `coder` to `doc-reviewer` (write mode) across every `/new` site that previously spawned a coder for documentation: sequential Phase 3, team-mode D.4a, and the post-batch Final review. **No new `baldart.config.yml` keys** — this is a fix-routing change internal to `/new`.
|
package/README.md
CHANGED
|
@@ -99,7 +99,7 @@ No additional activation steps needed — once installed, Claude Code (and Codex
|
|
|
99
99
|
9. **plan-auditor**: MANDATORY after planning — 4-persona plan review + High-Risk Path triggers + Specialist Auto-Spawn
|
|
100
100
|
10. **senior-researcher**: Evidence-based research with AI-readable retrieval-optimized reports
|
|
101
101
|
11. **api-perf-cost-auditor**: API performance and cost analysis with Findings Schema emission
|
|
102
|
-
12. **qa-sentinel**: SDET-grade gate runner (
|
|
102
|
+
12. **qa-sentinel**: SDET-grade gate runner (profile-driven: scoped-by-default, full on deep) — lint, tsc, test, build, audit
|
|
103
103
|
|
|
104
104
|
**Design & UX**
|
|
105
105
|
13. **ui-expert**: UI/UX design and review
|
package/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
3.
|
|
1
|
+
3.41.0
|
|
@@ -97,21 +97,23 @@ Weekly skill/agent improvement? --> skill-improver
|
|
|
97
97
|
|
|
98
98
|
### Profile selection
|
|
99
99
|
|
|
100
|
-
| Profile |
|
|
100
|
+
| Profile | Tier | Use when |
|
|
101
101
|
|---------|------|----------|
|
|
102
|
-
| `light` |
|
|
103
|
-
| `balanced` |
|
|
104
|
-
| `deep` | FULL
|
|
102
|
+
| `light` | SCOPED (related tests) | Small bugfix, no core modules, style/docs only |
|
|
103
|
+
| `balanced` | SCOPED (related tests) | Normal feature card, medium scope (default) |
|
|
104
|
+
| `deep` | FULL (whole suite) + Playwright | Auth/payments/DB/permissions, broad blast radius, Feature Complete cards |
|
|
105
105
|
| `skip` | — | Docs-only or cosmetic cards with zero logic changes |
|
|
106
106
|
|
|
107
|
+
Tier is selected by the **profile**, not file count. A `balanced` change is escalated to FULL only when the diff reveals undeclared risk (auth/permission/payment/schema/migration/API contract); file count never escalates.
|
|
108
|
+
|
|
107
109
|
### Invocation pattern
|
|
108
110
|
|
|
109
111
|
```
|
|
110
112
|
subagent_type: "qa-sentinel"
|
|
111
113
|
prompt: |
|
|
112
|
-
Run
|
|
114
|
+
Run QA on card <CARD-ID> at the tier dictated by the profile (do NOT default to FULL).
|
|
113
115
|
Worktree: <path> Branch: <branch> Changed files: <list>
|
|
114
|
-
|
|
116
|
+
QA profile: [light | balanced | deep] (light/balanced → SCOPED, deep → FULL)
|
|
115
117
|
Run gates ONLY. No AC verification, no code analysis, no recommendations.
|
|
116
118
|
Write gate results + verdict to: /qa/<CARD-ID>.md (under 40 lines)
|
|
117
119
|
```
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: qa-sentinel
|
|
3
|
-
description: "Use this agent when code changes need quality assurance validation before merging, after implementing a feature or bugfix, when running pre-commit or pre-merge checks, when a backlog card is marked Feature Complete, or when you need structured QA reports with risk assessments. Also use proactively after significant code changes to core modules like authentication, payments, DB schema, or API contracts.\\n\\n<example>\\nContext: The user has just implemented a new feature touching the authentication middleware and multiple API routes.\\nuser: \"I've finished implementing the merchant session refresh logic across 8 files. Can you check it's ready?\"\\nassistant: \"This touches authentication and multiple API routes — I'll invoke QA-Sentinel to run a full validation.\"\\n<commentary>\\nThe change set affects core auth logic (>threshold files, critical path), so use the Task tool to launch qa-sentinel in FULL VALIDATION MODE.\\n</commentary>\\nassistant: \"Let me launch the QA-Sentinel agent to validate this change set.\"\\n</example>\\n\\n<example>\\nContext: Developer refactored a utility function used only in non-critical UI components.\\nuser: \"Just cleaned up the formatCurrency helper and updated the 3 components that use it.\"\\nassistant: \"Small refactor on a non-core module — I'll have QA-Sentinel do a quick validation pass.\"\\n<commentary>\\nThis is a small, non-core change, so use the Task tool to launch qa-sentinel in
|
|
3
|
+
description: "Use this agent when code changes need quality assurance validation before merging, after implementing a feature or bugfix, when running pre-commit or pre-merge checks, when a backlog card is marked Feature Complete, or when you need structured QA reports with risk assessments. Also use proactively after significant code changes to core modules like authentication, payments, DB schema, or API contracts.\\n\\n<example>\\nContext: The user has just implemented a new feature touching the authentication middleware and multiple API routes.\\nuser: \"I've finished implementing the merchant session refresh logic across 8 files. Can you check it's ready?\"\\nassistant: \"This touches authentication and multiple API routes — I'll invoke QA-Sentinel to run a full validation.\"\\n<commentary>\\nThe change set affects core auth logic (>threshold files, critical path), so use the Task tool to launch qa-sentinel in FULL VALIDATION MODE.\\n</commentary>\\nassistant: \"Let me launch the QA-Sentinel agent to validate this change set.\"\\n</example>\\n\\n<example>\\nContext: Developer refactored a utility function used only in non-critical UI components.\\nuser: \"Just cleaned up the formatCurrency helper and updated the 3 components that use it.\"\\nassistant: \"Small refactor on a non-core module — I'll have QA-Sentinel do a quick validation pass.\"\\n<commentary>\\nThis is a small, non-core change (profile balanced), so use the Task tool to launch qa-sentinel in SCOPED VALIDATION.\\n</commentary>\\nassistant: \"Launching QA-Sentinel for a quick validation of the refactor.\"\\n</example>\\n\\n<example>\\nContext: A backlog card has just been marked as Feature Complete by the implementing agent.\\nuser: \"FEAT-0096 is done, all acceptance criteria checked.\"\\nassistant: \"Feature Complete status reached — I'll invoke QA-Sentinel to run release-level validation before we open the PR.\"\\n<commentary>\\nFeature Complete cards require FULL VALIDATION MODE. Use the Task tool to launch qa-sentinel.\\n</commentary>\\nassistant: \"Invoking QA-Sentinel for release-level confidence on FEAT-0096.\"\\n</example>\\n\\n<example>\\nContext: A bug was fixed and the developer wants to ensure it won't regress.\\nuser: \"Fixed the table control race condition in FEAT-0177.\"\\nassistant: \"Bug fix complete — I'll run QA-Sentinel to validate the fix and ensure a regression test is in place.\"\\n<commentary>\\nEvery bug fix should be validated and covered by a regression test. Use the Task tool to launch qa-sentinel.\\n</commentary>\\nassistant: \"Launching QA-Sentinel to validate the fix and check regression coverage.\"\\n</example>"
|
|
4
4
|
model: sonnet
|
|
5
5
|
color: pink
|
|
6
6
|
memory: project
|
|
@@ -35,7 +35,11 @@ Default pre-commit gates (MUST pass before any commit verdict):
|
|
|
35
35
|
|
|
36
36
|
Before running any checks:
|
|
37
37
|
1. Run `git diff --name-only HEAD` (or against base branch) to get the actual change set.
|
|
38
|
-
2.
|
|
38
|
+
2. Read the authoritative `QA profile` from your invocation prompt (`balanced` | `deep`) and map it
|
|
39
|
+
to a test tier via the table in **Operating Modes**. The profile — NOT file count — selects the
|
|
40
|
+
tier. The only allowed adjustment is `balanced → FULL` when the diff itself reveals risk the card
|
|
41
|
+
did not declare (see the risk-drift rule below). If the prompt carries no profile, default to
|
|
42
|
+
SCOPED and note it — never invent your own classification.
|
|
39
43
|
|
|
40
44
|
Never fabricate test results. Never assume what changed — always read the diff.
|
|
41
45
|
Do NOT read the changed files to "understand scope" — that's the code-reviewer's job. Just run the gates.
|
|
@@ -73,7 +77,12 @@ Long-running commands (especially `npm run build`) produce thousands of lines th
|
|
|
73
77
|
# Build — only capture exit code and last 20 lines
|
|
74
78
|
npm run build 2>&1 | tail -20; echo "EXIT:$?"
|
|
75
79
|
|
|
76
|
-
# Tests —
|
|
80
|
+
# Tests (SCOPED — default for balanced) — related tests only, summary
|
|
81
|
+
npx jest --findRelatedTests <changed files> 2>&1 | tail -30; echo "EXIT:$?"
|
|
82
|
+
npm test -- --testPathPattern="<touched-module-regex>" 2>&1 | tail -30; echo "EXIT:$?"
|
|
83
|
+
pytest <related test paths> 2>&1 | tail -30; echo "EXIT:$?"
|
|
84
|
+
|
|
85
|
+
# Tests (FULL — only for deep / risk-drift) — whole suite, summary
|
|
77
86
|
npm test 2>&1 | tail -30; echo "EXIT:$?"
|
|
78
87
|
|
|
79
88
|
# ESLint — full output is OK (usually short), but cap at 50 lines
|
|
@@ -89,38 +98,54 @@ If a gate FAILS, you may re-run WITHOUT `tail` to get the full error — but onl
|
|
|
89
98
|
|
|
90
99
|
## Operating Modes
|
|
91
100
|
|
|
92
|
-
|
|
101
|
+
The `QA profile` from your prompt selects the tier. The `skip` profile never reaches you (handled
|
|
102
|
+
upstream). Two execution tiers — **SCOPED** (default for `light`/`balanced`) and **FULL** (`deep`):
|
|
103
|
+
|
|
104
|
+
| QA profile | Tier | Tests | Audit | E2E | Target |
|
|
105
|
+
|------------|------|-------|-------|-----|--------|
|
|
106
|
+
| `light` | SCOPED | related tests only (TIA cascade below); skip if upstream gates already ran | skip | none | <3 min |
|
|
107
|
+
| `balanced` | SCOPED | related tests only (TIA cascade below) | skip unless new deps | advisory | <5 min |
|
|
108
|
+
| `deep` | FULL | whole suite | `npm audit --audit-level=high` | blocking | <15 min |
|
|
109
|
+
|
|
110
|
+
> Some orchestrator entry points run the upstream gates themselves and skip invoking you for `light`;
|
|
111
|
+
> others invoke you with `light` — in that case run SCOPED. Either way `light` never means FULL.
|
|
93
112
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
113
|
+
**Risk-drift escalation (the ONLY upgrade path):** run `balanced` as FULL when the diff itself
|
|
114
|
+
reveals risk the card did not declare — paths touching `auth`/`permission`/`payment`/`billing`,
|
|
115
|
+
DB schema / migration / index changes, or an API contract change. File **count** never escalates
|
|
116
|
+
(it is advisory only). When you escalate, log: `escalated balanced→FULL (risk drift: <reason>)`.
|
|
117
|
+
Never de-escalate `deep` to SCOPED.
|
|
118
|
+
|
|
119
|
+
### MODE 1: SCOPED VALIDATION (profile `balanced`)
|
|
100
120
|
|
|
101
121
|
Execution:
|
|
102
122
|
1. `npx eslint --max-warnings=0 <changed files>`
|
|
103
123
|
2. `npx tsc --noEmit`
|
|
104
|
-
3.
|
|
124
|
+
3. **Scoped tests** — run only the tests related to the changed files, via the first cascade rung
|
|
125
|
+
that applies to the repo (graceful degradation, never fall through to the full suite):
|
|
126
|
+
- Jest (`jest.config*` / `jest` in package.json): `npx jest --findRelatedTests <changed files>`
|
|
127
|
+
- Vitest (`vitest.config*` / `vitest`): `npx vitest related --run <changed files>`
|
|
128
|
+
- Pytest (`pytest.ini` / `pyproject` with pytest): `pytest <test paths matching touched modules>`
|
|
129
|
+
- Generic fallback: `npm test -- --testPathPattern="<regex of touched module dirs>"`
|
|
130
|
+
- Monorepo: resolve the workspace from the changed-file paths before invoking the runner.
|
|
131
|
+
- If no runner is determinable, or the runner reports "no tests found" for a **logic** change:
|
|
132
|
+
do NOT FAIL and do NOT run the full suite — record `Tests: SKIP — no related tests found` and
|
|
133
|
+
apply the confidence cap (see QA Report Format). For a docs/config-only change, a test SKIP is
|
|
134
|
+
legitimate at full confidence.
|
|
105
135
|
4. `npm run build`
|
|
106
|
-
5.
|
|
136
|
+
5. Smoke check if the project exposes one (`test:smoke` script or `tests/smoke/` — opt-in convention,
|
|
137
|
+
absent is fine).
|
|
107
138
|
|
|
108
|
-
Goal: Fast confidence without blocking flow. Target: <
|
|
139
|
+
Goal: Fast confidence on the touched surface without blocking flow. Target: <5 minutes.
|
|
109
140
|
|
|
110
141
|
---
|
|
111
142
|
|
|
112
|
-
### MODE 2: FULL VALIDATION
|
|
113
|
-
|
|
114
|
-
Triggers (any one is sufficient):
|
|
115
|
-
- >5 files changed
|
|
116
|
-
- Core logic touched (auth, payments, Firestore schema, API contracts)
|
|
117
|
-
- Backlog card marked "Feature Complete"
|
|
118
|
-
- Bug fix, DB queries/transactions/indexes changed, new external dependency, API route changes, permission logic
|
|
143
|
+
### MODE 2: FULL VALIDATION (profile `deep`, or `balanced` escalated by risk drift)
|
|
119
144
|
|
|
120
145
|
Execution:
|
|
121
146
|
1. `npx eslint --max-warnings=0 <changed .ts/.tsx files>`
|
|
122
147
|
2. `npx tsc --noEmit`
|
|
123
|
-
3. `npm run test` (full
|
|
148
|
+
3. `npm run test` (whole suite — or the project's full-suite equivalent, e.g. `pytest`)
|
|
124
149
|
4. `npm run build`
|
|
125
150
|
5. `npm audit --audit-level=high` (security baseline — flag HIGH/CRITICAL only)
|
|
126
151
|
6. `npx markdownlint-cli2 <changed .md files>` (if any)
|
|
@@ -169,7 +194,7 @@ Keep reports **under 40 lines**. No prose, no analysis, no recommendations. Just
|
|
|
169
194
|
```
|
|
170
195
|
## QA REPORT — [FEAT-XXXX] — [DATE]
|
|
171
196
|
|
|
172
|
-
|
|
197
|
+
Tier: SCOPED / FULL | Profile: balanced / deep | Files changed: N
|
|
173
198
|
|
|
174
199
|
| Gate | Status | Notes |
|
|
175
200
|
|------|--------|-------|
|
|
@@ -193,6 +218,9 @@ Rules:
|
|
|
193
218
|
- Pre-existing test failures: exclude from count, note as "N pre-existing excluded"
|
|
194
219
|
- Security audit: only report NEW vulnerabilities vs baseline (24 vulns as of 2026-03-06). If no new packages added, write "no delta"
|
|
195
220
|
- If all gates PASS → verdict is PASS, confidence 95-100%
|
|
221
|
+
- **Scoped coverage gap**: if a SCOPED run found no related tests for a **logic** change, write
|
|
222
|
+
`Tests: SKIP — no related tests found` and cap confidence at **≤70%** with a one-line coverage-gap
|
|
223
|
+
note. Do not report PASS at 95% when no test actually ran on a logic change.
|
|
196
224
|
- If any gate FAIL → verdict is FAIL, list only the failing gates
|
|
197
225
|
|
|
198
226
|
---
|
|
@@ -146,7 +146,7 @@ For each card, execute these phases in order:
|
|
|
146
146
|
21. **If profile is LIGHT, BALANCED, or DEEP**: invoke the **`qa-sentinel`** agent (subagent_type: `qa-sentinel`) via Task tool with the following context:
|
|
147
147
|
|
|
148
148
|
```
|
|
149
|
-
Run
|
|
149
|
+
Run QA on card <CARD-ID> at the tier dictated by the QA profile (do NOT default to FULL).
|
|
150
150
|
|
|
151
151
|
Context:
|
|
152
152
|
- Worktree path: <worktree-path>
|
|
@@ -156,10 +156,12 @@ For each card, execute these phases in order:
|
|
|
156
156
|
- Risk level: [Low | Medium | High]
|
|
157
157
|
- QA profile selected: [light | balanced | deep]
|
|
158
158
|
|
|
159
|
-
Profile →
|
|
160
|
-
- light
|
|
161
|
-
- balanced →
|
|
162
|
-
- deep
|
|
159
|
+
Profile → tier mapping:
|
|
160
|
+
- light → SCOPED (related tests only)
|
|
161
|
+
- balanced → SCOPED (related tests only; risk drift in the diff may escalate to FULL)
|
|
162
|
+
- deep → FULL (whole suite + audit + e2e smoke suite if configured)
|
|
163
|
+
Only path-based risk drift (auth/permission/payment/schema/migration/API contract) escalates;
|
|
164
|
+
file count never does.
|
|
163
165
|
|
|
164
166
|
After running all gates, write the complete QA Report to:
|
|
165
167
|
/qa/<CARD-ID>.md
|
|
@@ -105,7 +105,7 @@ Run directly in this session (no sub-agents):
|
|
|
105
105
|
1. Extract changed source files from the diff.
|
|
106
106
|
2. Lint check on changed files — if no changed files, skip with note.
|
|
107
107
|
3. Type check (e.g., `npx tsc --noEmit`)
|
|
108
|
-
4. Scoped tests —
|
|
108
|
+
4. Scoped tests — run only tests related to the changed files (related-tests / testPathPattern on touched modules). If no related tests are found, record `Tests: SKIP — no related tests` and cap confidence; do NOT fall through to the full suite.
|
|
109
109
|
5. Build check.
|
|
110
110
|
|
|
111
111
|
For each step: report PASS / FAIL. On FAIL, show the exact error output.
|
|
@@ -127,7 +127,7 @@ No findings file required. Present final verdict immediately.
|
|
|
127
127
|
Invoke `qa-sentinel` via Task tool:
|
|
128
128
|
|
|
129
129
|
```
|
|
130
|
-
Run
|
|
130
|
+
Run QA on the current changeset at the tier dictated by the QA profile below (do NOT default to FULL).
|
|
131
131
|
|
|
132
132
|
Context:
|
|
133
133
|
- Branch: <branch>
|
|
@@ -135,6 +135,11 @@ Context:
|
|
|
135
135
|
- Changed files: <list from git diff>
|
|
136
136
|
- Risk level: <Low/Medium/High>
|
|
137
137
|
- Card: <CARD-ID or "none">
|
|
138
|
+
- QA profile: <the profile this /qa was invoked with: balanced | deep>
|
|
139
|
+
|
|
140
|
+
Tier contract: balanced → SCOPED (related tests on touched modules); deep → FULL suite. Only
|
|
141
|
+
path-based risk drift (auth/permission/payment/schema/migration/API contract in the diff) may
|
|
142
|
+
escalate balanced→FULL.
|
|
138
143
|
|
|
139
144
|
After completing validation gates (lint, tsc, tests, build, audit):
|
|
140
145
|
1. Start the dev server if appropriate for the change, exercise key flows, and check for server/browser console errors and unexpected 4xx/5xx responses. Focus on flows directly touched by the diff.
|
|
@@ -1199,7 +1199,7 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1199
1199
|
22. **If profile is BALANCED or DEEP**: invoke the **`qa-sentinel`** agent (subagent_type: `qa-sentinel`) via Task tool with the following context:
|
|
1200
1200
|
|
|
1201
1201
|
```
|
|
1202
|
-
Run
|
|
1202
|
+
Run QA on card <CARD-ID> at the tier dictated by the QA profile below (do NOT default to FULL).
|
|
1203
1203
|
|
|
1204
1204
|
Context:
|
|
1205
1205
|
- Worktree path: <worktree-path>
|
|
@@ -1207,9 +1207,13 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1207
1207
|
- Changed files: <list from implementation phase>
|
|
1208
1208
|
- QA profile: [balanced | deep]
|
|
1209
1209
|
|
|
1210
|
-
|
|
1211
|
-
|
|
1212
|
-
|
|
1210
|
+
Tier contract: balanced → SCOPED (related tests on touched modules); deep → FULL suite.
|
|
1211
|
+
Only path-based risk drift (auth/permission/payment/schema/migration/API contract in the diff)
|
|
1212
|
+
may escalate balanced→FULL. File count never escalates.
|
|
1213
|
+
|
|
1214
|
+
E2E: deep → `npx playwright test --reporter=list` (BLOCKING);
|
|
1215
|
+
balanced → `npx playwright test --grep "<changed area>" --reporter=list` (ADVISORY).
|
|
1216
|
+
Also run gates (lint, tsc, build, markdownlint) as sanity check.
|
|
1213
1217
|
Do NOT verify acceptance criteria (Phase 2.5 already did this).
|
|
1214
1218
|
Do NOT analyze code for bugs/patterns (deferred to /codexreview post-batch).
|
|
1215
1219
|
Do NOT write recommendations or follow-up actions.
|
|
@@ -1219,14 +1223,14 @@ skill's Phase 1 falls back to deriving Gherkin scenarios from
|
|
|
1219
1223
|
Return verdict: PASS or FAIL.
|
|
1220
1224
|
```
|
|
1221
1225
|
|
|
1222
|
-
|
|
1223
|
-
|
|
1226
|
+
23. **Read qa-sentinel's output.** Verify the findings file was written to `/qa/<CARD-ID>.md`.
|
|
1227
|
+
24. **If QA verdict is FAIL**:
|
|
1224
1228
|
- Spawn the **coder** agent to fix all BLOCKER findings (pass it the findings file path + list of blockers). Do NOT ask the user.
|
|
1225
1229
|
- After coder fixes, re-invoke `qa-sentinel` in the same mode to re-validate. Repeat up to **2 times**.
|
|
1226
1230
|
- If still FAIL after 2 retries: log in `## Issues & Flags` and **ask the user** whether to proceed or stop.
|
|
1227
1231
|
- The commit in Phase 4 MUST NOT happen until QA verdict is PASS (or user explicitly overrides).
|
|
1228
1232
|
- **Telemetry** — after each coder spawn, append one row per blocker to `## Fix Application Log`: `3.5 | qa-blocker | est_lines=<n> | decision=coder | applied_by=coder | retry=<0|1|2>`. After PASS without any spawn (initial verdict PASS), append one row: `3.5 | qa-none | est_lines=0 | decision=skipped | applied_by=- | reason=qa-pass`.
|
|
1229
|
-
|
|
1233
|
+
25. **Update tracker**: phase = "3.5-qa DONE", log: profile used, tier run (SCOPED/FULL) + any risk-drift escalation, verdict (PASS/FAIL/SKIP), confidence %, findings count (blockers/majors/minors), findings file path.
|
|
1230
1234
|
|
|
1231
1235
|
### Phase 3.7 — Pre-Merge Codex Review Gate (MANDATORY — UNCONDITIONAL)
|
|
1232
1236
|
|
|
@@ -2237,7 +2241,7 @@ After ALL agents in the group complete successfully:
|
|
|
2237
2241
|
|
|
2238
2242
|
3c. **D.3c — Phase 2.6 E2E-Review (per-card)** — For EACH card in the group, honor the existing Gate table (skip when `features.has_e2e_review: false`, backend-only diff per the diff predicate documented in Phase 2.6, or card type `docs`/`chore`/`config`). For cards that pass the gate, invoke `/e2e-review` in programmatic mode with that card's payload. BLOCKING per-card: if `/e2e-review` returns `"blocked"` or `"error"`, surface to the user via the same `AskUserQuestion` documented in Phase 2.6 — do NOT proceed to D.4 with an unresolved card. Skips are logged with the documented gate reason, never with `"time budget"` or similar.
|
|
2239
2243
|
|
|
2240
|
-
4. **D.4 — QA gate (group)** — Read each card's `review_profile` field (fallback: compute via the QA Profile Selector table for legacy cards). Select the HIGHEST profile across all cards in the group (e.g., if one card is BALANCED and another is DEEP, use DEEP)
|
|
2244
|
+
4. **D.4 — QA gate (group)** — Read each card's `review_profile` field (fallback: compute via the QA Profile Selector table for legacy cards). Select the HIGHEST profile across all cards in the group (e.g., if one card is BALANCED and another is DEEP, use DEEP) and pass it as the `QA profile` to **qa-sentinel** (invoked once for the group) using the same prompt contract as Phase 3.5 step 22 (`do NOT default to FULL`; balanced → SCOPED, deep → FULL). Note: a mixed group containing any DEEP card runs the group at FULL — the max-profile intentionally overrides the SCOPED tier the BALANCED cards would get individually, because the combined diff must be validated together.
|
|
2241
2245
|
|
|
2242
2246
|
4a. **D.4a — Per-card doc gate (consumes D.2, since v3.35.0; doc-reviewer-applied since v3.40.0)** — Do NOT re-run the doc AUDIT (D.2 already produced per-card-attributed findings). If any card in the group has doc findings, invoke the **doc-reviewer once in write mode** over the group, passing the per-card-attributed findings from D.2, to APPLY all doc fixes in a single pass (it is now alone — code-reviewer is done — so the D.2 parallel-safety constraint no longer applies). Do **NOT** spawn a fix-coder for doc findings: `doc` is owned by `doc-reviewer` (see "Domain-Override Domains"); a code-oriented coder lacks the doc-invariant contract. The only exception is a doc-drift→bug finding rooted in CODE — that follows the D.4b code fix path. (Previously D.4a spawned a fix-coder per card; the audit was already collapsed to a single attributed D.2 pass. D.4b's per-card `/codexreview` also skips its doc-reviewer via the lean contract, so the doc AUDIT runs once per group while the doc FIXES run once per group via doc-reviewer.) **Telemetry** — append one row per applied doc finding to `## Fix Application Log`: `D.4a | doc | est_lines=<n> | decision=doc-reviewer | applied_by=doc-reviewer | card=<ID> | finding=<1-line>`.
|
|
2243
2247
|
|
|
@@ -22,7 +22,8 @@ Route agents to the right module with minimal reading.
|
|
|
22
22
|
- If touching design-review workflows or UI guidelines -> read `agents/design-review.md` and project-specific UI guidelines.
|
|
23
23
|
- If touching architecture, auth, or tech stack -> read `agents/architecture.md`.
|
|
24
24
|
- If touching workflow/process/commits/backlog -> read `agents/workflows.md`.
|
|
25
|
-
- If touching testing or QA issues -> read `agents/testing.md
|
|
25
|
+
- If touching testing or QA issues -> read `agents/testing.md` (also documents the scope-aware,
|
|
26
|
+
profile-driven test-selection strategy consumed by `qa-sentinel`).
|
|
26
27
|
- If touching GitHub issues or issue workflow -> read `agents/github-issue-subagent.md`.
|
|
27
28
|
- If touching terminology or naming -> read `agents/coding-standards.md`.
|
|
28
29
|
- If touching env vars or scripts -> read `agents/runbook.md` and `agents/env-reference.md`.
|
|
@@ -9,6 +9,32 @@ Define required manual testing and QA issue workflow, including browser automati
|
|
|
9
9
|
**In**: Manual testing steps, QA issue creation and handling, browser automation.
|
|
10
10
|
**Out**: Performance testing strategy (see agents/performance.md).
|
|
11
11
|
|
|
12
|
+
## Scope-aware Test Selection (Automated QA)
|
|
13
|
+
|
|
14
|
+
The automated QA gate (`qa-sentinel`) does **not** run the full test suite on every change. It runs
|
|
15
|
+
the depth dictated by the card's `review_profile` (the deterministic SSOT computed at authoring
|
|
16
|
+
time), so test effort matches change risk:
|
|
17
|
+
|
|
18
|
+
- **balanced → SCOPED**: run only the tests *related to the changed files* (test-impact selection),
|
|
19
|
+
not the whole suite. This is the default for ordinary features and refactors.
|
|
20
|
+
- **deep → FULL**: run the whole suite + security audit + blocking E2E. Reserved for high-risk
|
|
21
|
+
changes (data model, security, broad blast radius).
|
|
22
|
+
- **skip / light**: the gate is not invoked — these are handled upstream (docs/chore, or gates
|
|
23
|
+
already run during implementation).
|
|
24
|
+
- **Risk-drift escalation**: a `balanced` change is promoted to FULL only when the diff itself
|
|
25
|
+
reveals undeclared risk — paths touching auth, permissions, payment/billing, DB schema/migration,
|
|
26
|
+
or an API contract. Change *count* never escalates (it is advisory only).
|
|
27
|
+
|
|
28
|
+
Test selection is **heuristic**, not instrumented: it relies on file paths, naming, and the test
|
|
29
|
+
runner's native related-tests capability — no coverage maps, AST analysis, or ML. When no related
|
|
30
|
+
tests can be resolved for a logic change, the gate records a coverage-gap and lowers its confidence
|
|
31
|
+
rather than silently passing or falling back to the full suite. The concrete per-runner commands
|
|
32
|
+
live in the `qa-sentinel` agent (which is stack-aware); this module states the principle only.
|
|
33
|
+
|
|
34
|
+
> The executable tier table + runner cascade is owned by `qa-sentinel` (`.claude/agents/qa-sentinel.md`),
|
|
35
|
+
> kept stack-specific there. Optional smoke convention: a `test:smoke` script or `tests/smoke/`
|
|
36
|
+
> directory, if present, is used as a minimal sanity set — its absence is fine, not a misconfiguration.
|
|
37
|
+
|
|
12
38
|
## Do
|
|
13
39
|
|
|
14
40
|
- Run project build command before marking a card done.
|