bmad-method-test-architecture-enterprise 1.2.0 → 1.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/how-to/workflows/run-test-review.md +40 -44
- package/docs/how-to/workflows/run-trace.md +14 -6
- package/docs/reference/commands.md +6 -6
- package/package.json +1 -1
- package/release_notes.md +5 -5
- package/src/module-help.csv +9 -9
- package/src/workflows/testarch/test-review/checklist.md +3 -0
- package/src/workflows/testarch/test-review/instructions.md +2 -0
- package/src/workflows/testarch/test-review/steps-c/step-01-load-context.md +2 -0
- package/src/workflows/testarch/test-review/steps-c/step-03-quality-evaluation.md +25 -36
- package/src/workflows/testarch/test-review/steps-c/step-03f-aggregate-scores.md +19 -16
- package/src/workflows/testarch/test-review/steps-c/step-04-generate-report.md +1 -0
- package/src/workflows/testarch/test-review/test-review-template.md +4 -15
- package/src/workflows/testarch/trace/checklist.md +7 -2
- package/src/workflows/testarch/trace/steps-c/step-02-discover-tests.md +19 -1
- package/src/workflows/testarch/trace/steps-c/step-03-map-criteria.md +7 -0
- package/src/workflows/testarch/trace/steps-c/step-04-analyze-gaps.md +82 -21
- package/src/workflows/testarch/trace/steps-c/step-05-gate-decision.md +36 -16
- package/src/workflows/testarch/trace/trace-template.md +25 -0
- package/src/workflows/testarch/test-review/steps-c/step-03d-subprocess-coverage.md +0 -111
|
@@ -7,14 +7,17 @@ description: Audit test quality using TEA's comprehensive knowledge base and get
|
|
|
7
7
|
|
|
8
8
|
Use TEA's `test-review` workflow to audit test quality with objective scoring and actionable feedback. TEA reviews tests against its knowledge base of best practices.
|
|
9
9
|
|
|
10
|
+
Coverage scoring is intentionally excluded from `test-review`. Use `trace` for requirements coverage analysis and coverage gate decisions.
|
|
11
|
+
|
|
10
12
|
## When to Use This
|
|
11
13
|
|
|
12
14
|
- Want to validate test quality objectively
|
|
13
|
-
- Need quality metrics for
|
|
15
|
+
- Need quality metrics for test quality gates
|
|
14
16
|
- Preparing for production deployment
|
|
15
17
|
- Reviewing team-written tests
|
|
16
18
|
- Auditing AI-generated tests
|
|
17
19
|
- Onboarding new team members (show good patterns)
|
|
20
|
+
- Validating migration quality before coverage expansion
|
|
18
21
|
|
|
19
22
|
## Prerequisites
|
|
20
23
|
|
|
@@ -311,42 +314,34 @@ test('should show validation error for expired card', async ({ page }) => {});
|
|
|
311
314
|
|
|
312
315
|
## Quality Scores by Category
|
|
313
316
|
|
|
314
|
-
| Category
|
|
315
|
-
|
|
|
316
|
-
| **Determinism**
|
|
317
|
-
| **Isolation**
|
|
318
|
-
| **
|
|
319
|
-
| **
|
|
320
|
-
| **Performance** | 3/10 | 8/10 | ❌ Critical |
|
|
317
|
+
| Category | Score | Target | Status |
|
|
318
|
+
| ------------------- | ----- | ------ | -------------------- |
|
|
319
|
+
| **Determinism** | 72 | 80 | ⚠️ Needs Improvement |
|
|
320
|
+
| **Isolation** | 88 | 80 | ✅ Good |
|
|
321
|
+
| **Maintainability** | 70 | 80 | ⚠️ Needs Improvement |
|
|
322
|
+
| **Performance** | 60 | 80 | ❌ Critical |
|
|
321
323
|
|
|
322
324
|
### Scoring Breakdown
|
|
323
325
|
|
|
324
|
-
**Determinism (
|
|
325
|
-
|
|
326
|
-
- No hard waits: 0/10 ❌ (found 3 instances)
|
|
327
|
-
- No conditionals: 8/10 ⚠️ (found 2 instances)
|
|
328
|
-
- No try-catch flow control: 10/10 ✅
|
|
329
|
-
- Network-first patterns: 8/15 ⚠️ (some tests missing)
|
|
330
|
-
|
|
331
|
-
**Isolation (25 points max):**
|
|
326
|
+
**Determinism (30% weight):**
|
|
332
327
|
|
|
333
|
-
-
|
|
334
|
-
-
|
|
335
|
-
- Parallel-safe: 0/0 ✅ (not tested)
|
|
328
|
+
- Hard waits and race conditions penalized
|
|
329
|
+
- Unstable control flow patterns penalized
|
|
336
330
|
|
|
337
|
-
**
|
|
331
|
+
**Isolation (30% weight):**
|
|
338
332
|
|
|
339
|
-
-
|
|
340
|
-
-
|
|
333
|
+
- Shared state and order dependency penalized
|
|
334
|
+
- Cleanup and parallel-safety rewarded
|
|
341
335
|
|
|
342
|
-
**
|
|
336
|
+
**Maintainability (25% weight):**
|
|
343
337
|
|
|
344
|
-
-
|
|
345
|
-
-
|
|
338
|
+
- Overly large files and copy-paste patterns penalized
|
|
339
|
+
- Naming clarity and structure rewarded
|
|
346
340
|
|
|
347
|
-
**Performance (
|
|
341
|
+
**Performance (15% weight):**
|
|
348
342
|
|
|
349
|
-
-
|
|
343
|
+
- Serial bottlenecks and inefficient setup penalized
|
|
344
|
+
- Parallel-friendly structure rewarded
|
|
350
345
|
|
|
351
346
|
## Files Reviewed
|
|
352
347
|
|
|
@@ -402,31 +397,30 @@ TEA reviewed against these patterns:
|
|
|
402
397
|
|
|
403
398
|
### Scoring Criteria
|
|
404
399
|
|
|
405
|
-
**Determinism (
|
|
400
|
+
**Determinism (30%):**
|
|
406
401
|
- Tests produce same result every run
|
|
407
402
|
- No random failures (flakiness)
|
|
408
403
|
- No environment-dependent behavior
|
|
409
404
|
|
|
410
|
-
**Isolation (
|
|
405
|
+
**Isolation (30%):**
|
|
411
406
|
- Tests don't depend on each other
|
|
412
407
|
- Can run in any order
|
|
413
408
|
- Clean up after themselves
|
|
414
409
|
|
|
415
|
-
**
|
|
416
|
-
- Verify actual behavior
|
|
417
|
-
- Specific and meaningful
|
|
418
|
-
- Not abstracted away in helpers
|
|
419
|
-
|
|
420
|
-
**Structure (10 points):**
|
|
410
|
+
**Maintainability (25%):**
|
|
421
411
|
- Readable and maintainable
|
|
422
412
|
- Appropriate size
|
|
423
413
|
- Clear naming
|
|
424
414
|
|
|
425
|
-
**Performance (
|
|
415
|
+
**Performance (15%):**
|
|
426
416
|
- Fast execution
|
|
427
417
|
- Efficient selectors
|
|
428
418
|
- No unnecessary waits
|
|
429
419
|
|
|
420
|
+
**Coverage:**
|
|
421
|
+
- Not scored in `test-review`
|
|
422
|
+
- Use `trace` for coverage percentage, requirement mapping, and gate decisions
|
|
423
|
+
|
|
430
424
|
## What You Get
|
|
431
425
|
|
|
432
426
|
### Quality Report
|
|
@@ -457,9 +451,9 @@ TEA reviewed against these patterns:
|
|
|
457
451
|
Make test review part of release checklist:
|
|
458
452
|
|
|
459
453
|
```markdown
|
|
460
|
-
##
|
|
454
|
+
## Quality Checklist (Test-Review)
|
|
461
455
|
- [ ] All tests passing
|
|
462
|
-
- [ ] Test
|
|
456
|
+
- [ ] Test-review quality score > 80
|
|
463
457
|
- [ ] Critical issues resolved
|
|
464
458
|
- [ ] Performance within budget
|
|
465
459
|
````
|
|
@@ -483,21 +477,23 @@ Use scores as quality gates:
|
|
|
483
477
|
# .github/workflows/test.yml
|
|
484
478
|
- name: Review test quality
|
|
485
479
|
run: |
|
|
486
|
-
# Run test
|
|
487
|
-
# Parse score from report
|
|
480
|
+
# Run test-review quality gate
|
|
481
|
+
# Parse quality score from report
|
|
488
482
|
if [ $SCORE -lt 80 ]; then
|
|
489
|
-
echo "Test quality below threshold"
|
|
483
|
+
echo "Test-review quality gate below threshold"
|
|
490
484
|
exit 1
|
|
491
485
|
fi
|
|
492
486
|
```
|
|
493
487
|
|
|
488
|
+
Coverage gate checks are handled by `trace`, not `test-review`.
|
|
489
|
+
|
|
494
490
|
### Review Regularly
|
|
495
491
|
|
|
496
492
|
Schedule periodic reviews:
|
|
497
493
|
|
|
498
494
|
- **Per story:** Optional (spot check new tests)
|
|
499
495
|
- **Per epic:** Recommended (ensure consistency)
|
|
500
|
-
- **Per release:** Recommended for quality gates (
|
|
496
|
+
- **Per release:** Recommended for test quality gates (coverage gates remain in `trace`)
|
|
501
497
|
- **Quarterly:** Audit entire suite
|
|
502
498
|
|
|
503
499
|
### Focus Reviews
|
|
@@ -619,8 +615,8 @@ Don't try to fix everything at once.
|
|
|
619
615
|
## Related Guides
|
|
620
616
|
|
|
621
617
|
- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate tests to review
|
|
622
|
-
- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand
|
|
623
|
-
- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Coverage
|
|
618
|
+
- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand and improve tests before review
|
|
619
|
+
- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Coverage analysis and gate decisions
|
|
624
620
|
|
|
625
621
|
## Understanding the Concepts
|
|
626
622
|
|
|
@@ -489,7 +489,7 @@ TEA makes evidence-based gate decision and writes to separate file.
|
|
|
489
489
|
|
|
490
490
|
| Metric | Threshold | Actual | Status |
|
|
491
491
|
| ------------------ | --------- | ------ | ------ |
|
|
492
|
-
| P0/P1 Coverage |
|
|
492
|
+
| P0/P1 Coverage | P0=100%, P1>=90% | 100% / 100% | ✅ |
|
|
493
493
|
| Test Quality Score | >80 | 84 | ✅ |
|
|
494
494
|
| NFR Status | PASS | PASS | ✅ |
|
|
495
495
|
|
|
@@ -609,7 +609,7 @@ TEA uses deterministic rules when decision_mode = "deterministic":
|
|
|
609
609
|
|
|
610
610
|
**Evidence:**
|
|
611
611
|
|
|
612
|
-
- P0 coverage: 60% (below
|
|
612
|
+
- P0 coverage: 60% (below required 100%)
|
|
613
613
|
- Critical security vulnerability (CVE-2024-12345)
|
|
614
614
|
- Test quality: 55/100
|
|
615
615
|
|
|
@@ -787,8 +787,16 @@ Use traceability in CI:
|
|
|
787
787
|
run: |
|
|
788
788
|
# Run trace Phase 1
|
|
789
789
|
# Parse coverage percentages
|
|
790
|
-
if [ $P0_COVERAGE -lt
|
|
791
|
-
echo "P0 coverage below
|
|
790
|
+
if [ $P0_COVERAGE -lt 100 ]; then
|
|
791
|
+
echo "P0 coverage below required 100%"
|
|
792
|
+
exit 1
|
|
793
|
+
fi
|
|
794
|
+
if [ $P1_COVERAGE -lt 80 ]; then
|
|
795
|
+
echo "P1 coverage below minimum 80%"
|
|
796
|
+
exit 1
|
|
797
|
+
fi
|
|
798
|
+
if [ $OVERALL_COVERAGE -lt 80 ]; then
|
|
799
|
+
echo "Overall coverage below minimum 80%"
|
|
792
800
|
exit 1
|
|
793
801
|
fi
|
|
794
802
|
```
|
|
@@ -916,7 +924,7 @@ Result: PARTIAL coverage (3/4 criteria)
|
|
|
916
924
|
|
|
917
925
|
**Use CONCERNS** ⚠️ if:
|
|
918
926
|
|
|
919
|
-
- P1 coverage
|
|
927
|
+
- P1 coverage 80-89% (below PASS target, above minimum)
|
|
920
928
|
- Minor quality issues (score 70-79)
|
|
921
929
|
- NFRs have mitigation plans
|
|
922
930
|
- Team agrees risk is acceptable
|
|
@@ -924,7 +932,7 @@ Result: PARTIAL coverage (3/4 criteria)
|
|
|
924
932
|
**Use FAIL** ❌ if:
|
|
925
933
|
|
|
926
934
|
- P0 coverage <100% (critical path gaps)
|
|
927
|
-
- P1 coverage <
|
|
935
|
+
- P1 coverage <80%
|
|
928
936
|
- Critical security/performance issues
|
|
929
937
|
- No mitigation possible
|
|
930
938
|
|
|
@@ -232,15 +232,15 @@ Quick reference for all 9 TEA (Test Engineering Architect) workflows. For detail
|
|
|
232
232
|
- `test-review.md` with quality score (0-100)
|
|
233
233
|
- Critical issues with fixes
|
|
234
234
|
- Recommendations
|
|
235
|
-
- Category scores (Determinism, Isolation,
|
|
235
|
+
- Category scores (Determinism, Isolation, Maintainability, Performance)
|
|
236
|
+
- Coverage guidance is informational only; coverage scoring and gates are handled by `trace`
|
|
236
237
|
|
|
237
238
|
**Scoring Categories:**
|
|
238
239
|
|
|
239
|
-
- Determinism:
|
|
240
|
-
- Isolation:
|
|
241
|
-
-
|
|
242
|
-
-
|
|
243
|
-
- Performance: 10 points
|
|
240
|
+
- Determinism: 30%
|
|
241
|
+
- Isolation: 30%
|
|
242
|
+
- Maintainability: 25%
|
|
243
|
+
- Performance: 15%
|
|
244
244
|
|
|
245
245
|
**How-To Guide:** [Run Test Review](/docs/how-to/workflows/run-test-review.md)
|
|
246
246
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.2.
|
|
4
|
+
"version": "1.2.2",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
package/release_notes.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
## 🚀 What's New in v1.2.
|
|
1
|
+
## 🚀 What's New in v1.2.2
|
|
2
2
|
|
|
3
|
-
###
|
|
4
|
-
-
|
|
3
|
+
### 🐛 Bug Fixes
|
|
4
|
+
- fix: issue 29
|
|
5
5
|
|
|
6
6
|
### 📦 Other Changes
|
|
7
|
-
- Merge pull request #
|
|
7
|
+
- Merge pull request #31 from bmad-code-org/fix/issue-29
|
|
8
8
|
|
|
9
9
|
|
|
10
10
|
## 📦 Installation
|
|
@@ -14,4 +14,4 @@ npx bmad-method install
|
|
|
14
14
|
# Select "Test Architect" from module menu
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
-
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.1
|
|
17
|
+
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.2.1...v1.2.2
|
package/src/module-help.csv
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
module,phase,name,code,sequence,workflow-file,command,required,agent,options,description,output-location,outputs,
|
|
2
|
-
tea,0-learning,Teach Me Testing,TMT,10,_bmad/tea/workflows/testarch/teach-me-testing/workflow.md,
|
|
3
|
-
tea,3-solutioning,Test Design,TD,10,_bmad/tea/workflows/testarch/test-design/workflow.yaml,
|
|
4
|
-
tea,3-solutioning,Test Framework,TF,20,_bmad/tea/workflows/testarch/framework/workflow.yaml,
|
|
5
|
-
tea,3-solutioning,CI Setup,CI,30,_bmad/tea/workflows/testarch/ci/workflow.yaml,
|
|
6
|
-
tea,4-implementation,ATDD,AT,10,_bmad/tea/workflows/testarch/atdd/workflow.yaml,
|
|
7
|
-
tea,4-implementation,Test Automation,TA,20,_bmad/tea/workflows/testarch/automate/workflow.yaml,
|
|
8
|
-
tea,4-implementation,Test Review,RV,30,_bmad/tea/workflows/testarch/test-review/workflow.yaml,
|
|
9
|
-
tea,4-implementation,NFR Assessment,NR,40,_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml,
|
|
10
|
-
tea,4-implementation,Traceability,TR,50,_bmad/tea/workflows/testarch/trace/workflow.yaml,
|
|
2
|
+
tea,0-learning,Teach Me Testing,TMT,10,_bmad/tea/workflows/testarch/teach-me-testing/workflow.md,bmad-tea-teach-me-testing,false,tea,Create Mode,"Teach testing fundamentals through 7 sessions (TEA Academy)",test_artifacts,"progress file|session notes|certificate",
|
|
3
|
+
tea,3-solutioning,Test Design,TD,10,_bmad/tea/workflows/testarch/test-design/workflow.yaml,bmad-tea-testarch-test-design,false,tea,Create Mode,"Risk-based test planning",test_artifacts,"test design document",
|
|
4
|
+
tea,3-solutioning,Test Framework,TF,20,_bmad/tea/workflows/testarch/framework/workflow.yaml,bmad-tea-testarch-framework,false,tea,Create Mode,"Initialize production-ready test framework",test_artifacts,"framework scaffold",
|
|
5
|
+
tea,3-solutioning,CI Setup,CI,30,_bmad/tea/workflows/testarch/ci/workflow.yaml,bmad-tea-testarch-ci,false,tea,Create Mode,"Configure CI/CD quality pipeline",test_artifacts,"ci config",
|
|
6
|
+
tea,4-implementation,ATDD,AT,10,_bmad/tea/workflows/testarch/atdd/workflow.yaml,bmad-tea-testarch-atdd,false,tea,Create Mode,"Generate failing tests (TDD red phase)",test_artifacts,"atdd tests",
|
|
7
|
+
tea,4-implementation,Test Automation,TA,20,_bmad/tea/workflows/testarch/automate/workflow.yaml,bmad-tea-testarch-automate,false,tea,Create Mode,"Expand test coverage",test_artifacts,"test suite",
|
|
8
|
+
tea,4-implementation,Test Review,RV,30,_bmad/tea/workflows/testarch/test-review/workflow.yaml,bmad-tea-testarch-test-review,false,tea,Validate Mode,"Quality audit (0-100 scoring)",test_artifacts,"review report",
|
|
9
|
+
tea,4-implementation,NFR Assessment,NR,40,_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml,bmad-tea-testarch-nfr,false,tea,Create Mode,"Non-functional requirements",test_artifacts,"nfr report",
|
|
10
|
+
tea,4-implementation,Traceability,TR,50,_bmad/tea/workflows/testarch/trace/workflow.yaml,bmad-tea-testarch-trace,false,tea,Create Mode,"Coverage traceability and gate",test_artifacts,"traceability matrix|gate decision",
|
|
@@ -7,6 +7,7 @@ Use this checklist to validate that the test quality review workflow completed s
|
|
|
7
7
|
## Prerequisites
|
|
8
8
|
|
|
9
9
|
Note: `test-review` is optional and only audits existing tests; it does not generate tests.
|
|
10
|
+
Coverage analysis is out of scope for this workflow. Use `trace` for coverage metrics and coverage gate decisions.
|
|
10
11
|
|
|
11
12
|
### Test File Discovery
|
|
12
13
|
|
|
@@ -72,6 +73,8 @@ Note: `test-review` is optional and only audits existing tests; it does not gene
|
|
|
72
73
|
|
|
73
74
|
### Step 3: Quality Criteria Validation
|
|
74
75
|
|
|
76
|
+
Coverage criteria are intentionally excluded from this checklist.
|
|
77
|
+
|
|
75
78
|
**For Each Enabled Criterion:**
|
|
76
79
|
|
|
77
80
|
#### BDD Format (if `check_given_when_then: true`)
|
|
@@ -9,6 +9,8 @@
|
|
|
9
9
|
|
|
10
10
|
Review test quality using TEA knowledge base and produce a 0–100 quality score with actionable findings.
|
|
11
11
|
|
|
12
|
+
Coverage assessment is intentionally out of scope for this workflow. Use `trace` for requirements coverage and coverage gate decisions.
|
|
13
|
+
|
|
12
14
|
---
|
|
13
15
|
|
|
14
16
|
## WORKFLOW ARCHITECTURE
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: 'step-03-quality-evaluation'
|
|
3
|
-
description: 'Orchestrate parallel quality dimension checks (
|
|
3
|
+
description: 'Orchestrate parallel quality dimension checks (4 subprocesses)'
|
|
4
4
|
nextStepFile: './step-03f-aggregate-scores.md'
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -8,14 +8,21 @@ nextStepFile: './step-03f-aggregate-scores.md'
|
|
|
8
8
|
|
|
9
9
|
## STEP GOAL
|
|
10
10
|
|
|
11
|
-
Launch
|
|
11
|
+
Launch 4 parallel subprocesses to evaluate test quality dimensions:
|
|
12
|
+
|
|
13
|
+
- Determinism
|
|
14
|
+
- Isolation
|
|
15
|
+
- Maintainability
|
|
16
|
+
- Performance
|
|
17
|
+
|
|
18
|
+
Coverage is intentionally excluded from this workflow and handled by `trace`.
|
|
12
19
|
|
|
13
20
|
## MANDATORY EXECUTION RULES
|
|
14
21
|
|
|
15
22
|
- 📖 Read the entire step file before acting
|
|
16
23
|
- ✅ Speak in `{communication_language}`
|
|
17
|
-
- ✅ Launch
|
|
18
|
-
- ✅ Wait for
|
|
24
|
+
- ✅ Launch four subprocesses in PARALLEL
|
|
25
|
+
- ✅ Wait for all subprocesses to complete
|
|
19
26
|
- ❌ Do NOT evaluate quality sequentially (use subprocesses)
|
|
20
27
|
- ❌ Do NOT proceed until all subprocesses finish
|
|
21
28
|
|
|
@@ -57,33 +64,27 @@ const subprocessContext = {
|
|
|
57
64
|
|
|
58
65
|
---
|
|
59
66
|
|
|
60
|
-
### 2. Launch
|
|
67
|
+
### 2. Launch 4 Parallel Quality Subprocesses
|
|
61
68
|
|
|
62
|
-
**Subprocess A: Determinism
|
|
69
|
+
**Subprocess A: Determinism**
|
|
63
70
|
|
|
64
71
|
- File: `./step-03a-subprocess-determinism.md`
|
|
65
72
|
- Output: `/tmp/tea-test-review-determinism-${timestamp}.json`
|
|
66
73
|
- Status: Running in parallel... ⟳
|
|
67
74
|
|
|
68
|
-
**Subprocess B: Isolation
|
|
75
|
+
**Subprocess B: Isolation**
|
|
69
76
|
|
|
70
77
|
- File: `./step-03b-subprocess-isolation.md`
|
|
71
78
|
- Output: `/tmp/tea-test-review-isolation-${timestamp}.json`
|
|
72
79
|
- Status: Running in parallel... ⟳
|
|
73
80
|
|
|
74
|
-
**Subprocess C: Maintainability
|
|
81
|
+
**Subprocess C: Maintainability**
|
|
75
82
|
|
|
76
83
|
- File: `./step-03c-subprocess-maintainability.md`
|
|
77
84
|
- Output: `/tmp/tea-test-review-maintainability-${timestamp}.json`
|
|
78
85
|
- Status: Running in parallel... ⟳
|
|
79
86
|
|
|
80
|
-
**Subprocess D:
|
|
81
|
-
|
|
82
|
-
- File: `./step-03d-subprocess-coverage.md`
|
|
83
|
-
- Output: `/tmp/tea-test-review-coverage-${timestamp}.json`
|
|
84
|
-
- Status: Running in parallel... ⟳
|
|
85
|
-
|
|
86
|
-
**Subprocess E: Performance Check**
|
|
87
|
+
**Subprocess D: Performance**
|
|
87
88
|
|
|
88
89
|
- File: `./step-03e-subprocess-performance.md`
|
|
89
90
|
- Output: `/tmp/tea-test-review-performance-${timestamp}.json`
|
|
@@ -94,22 +95,8 @@ const subprocessContext = {
|
|
|
94
95
|
### 3. Wait for All Subprocesses
|
|
95
96
|
|
|
96
97
|
```
|
|
97
|
-
⏳ Waiting for
|
|
98
|
-
|
|
99
|
-
├── Subprocess B (Isolation): Running... ⟳
|
|
100
|
-
├── Subprocess C (Maintainability): Running... ⟳
|
|
101
|
-
├── Subprocess D (Coverage): Running... ⟳
|
|
102
|
-
└── Subprocess E (Performance): Running... ⟳
|
|
103
|
-
|
|
104
|
-
[... time passes ...]
|
|
105
|
-
|
|
106
|
-
├── Subprocess A (Determinism): Complete ✅
|
|
107
|
-
├── Subprocess B (Isolation): Complete ✅
|
|
108
|
-
├── Subprocess C (Maintainability): Complete ✅
|
|
109
|
-
├── Subprocess D (Coverage): Complete ✅
|
|
110
|
-
└── Subprocess E (Performance): Complete ✅
|
|
111
|
-
|
|
112
|
-
✅ All 5 quality subprocesses completed successfully!
|
|
98
|
+
⏳ Waiting for 4 quality subprocesses to complete...
|
|
99
|
+
✅ All 4 quality subprocesses completed successfully!
|
|
113
100
|
```
|
|
114
101
|
|
|
115
102
|
---
|
|
@@ -117,7 +104,7 @@ const subprocessContext = {
|
|
|
117
104
|
### 4. Verify All Outputs Exist
|
|
118
105
|
|
|
119
106
|
```javascript
|
|
120
|
-
const outputs = ['determinism', 'isolation', 'maintainability', '
|
|
107
|
+
const outputs = ['determinism', 'isolation', 'maintainability', 'performance'].map(
|
|
121
108
|
(dim) => `/tmp/tea-test-review-${dim}-${timestamp}.json`,
|
|
122
109
|
);
|
|
123
110
|
|
|
@@ -134,7 +121,7 @@ outputs.forEach((output) => {
|
|
|
134
121
|
|
|
135
122
|
```
|
|
136
123
|
🚀 Performance Report:
|
|
137
|
-
- Execution Mode: PARALLEL (
|
|
124
|
+
- Execution Mode: PARALLEL (4 subprocesses)
|
|
138
125
|
- Total Elapsed: ~max(all subprocesses) minutes
|
|
139
126
|
- Sequential Would Take: ~sum(all subprocesses) minutes
|
|
140
127
|
- Performance Gain: ~60-70% faster!
|
|
@@ -144,11 +131,13 @@ outputs.forEach((output) => {
|
|
|
144
131
|
|
|
145
132
|
### 6. Proceed to Aggregation
|
|
146
133
|
|
|
134
|
+
Pass the same `timestamp` value to Step 3F (do not regenerate it). Step 3F must read the exact temp files written in this step.
|
|
135
|
+
|
|
147
136
|
Load next step: `{nextStepFile}`
|
|
148
137
|
|
|
149
138
|
The aggregation step (3F) will:
|
|
150
139
|
|
|
151
|
-
- Read all
|
|
140
|
+
- Read all 4 subprocess outputs
|
|
152
141
|
- Calculate weighted overall score (0-100)
|
|
153
142
|
- Aggregate violations by severity
|
|
154
143
|
- Generate review report with top suggestions
|
|
@@ -159,7 +148,7 @@ The aggregation step (3F) will:
|
|
|
159
148
|
|
|
160
149
|
Proceed to Step 3F when:
|
|
161
150
|
|
|
162
|
-
- ✅ All
|
|
151
|
+
- ✅ All 4 subprocesses completed successfully
|
|
163
152
|
- ✅ All output files exist and are valid JSON
|
|
164
153
|
- ✅ Performance metrics displayed
|
|
165
154
|
|
|
@@ -171,7 +160,7 @@ Proceed to Step 3F when:
|
|
|
171
160
|
|
|
172
161
|
### ✅ SUCCESS:
|
|
173
162
|
|
|
174
|
-
- All
|
|
163
|
+
- All 4 subprocesses launched and completed
|
|
175
164
|
- Output files generated and valid
|
|
176
165
|
- Parallel execution achieved ~60% performance gain
|
|
177
166
|
|
|
@@ -9,7 +9,7 @@ outputFile: '{test_artifacts}/test-review.md'
|
|
|
9
9
|
|
|
10
10
|
## STEP GOAL
|
|
11
11
|
|
|
12
|
-
Read outputs from
|
|
12
|
+
Read outputs from 4 quality subprocesses, calculate weighted overall score (0-100), and aggregate violations for report generation.
|
|
13
13
|
|
|
14
14
|
---
|
|
15
15
|
|
|
@@ -17,7 +17,7 @@ Read outputs from 5 parallel quality subprocesses, calculate weighted overall sc
|
|
|
17
17
|
|
|
18
18
|
- 📖 Read the entire step file before acting
|
|
19
19
|
- ✅ Speak in `{communication_language}`
|
|
20
|
-
- ✅ Read all
|
|
20
|
+
- ✅ Read all 4 subprocess outputs
|
|
21
21
|
- ✅ Calculate weighted overall score
|
|
22
22
|
- ✅ Aggregate violations by severity
|
|
23
23
|
- ❌ Do NOT re-evaluate quality (use subprocess outputs)
|
|
@@ -37,11 +37,16 @@ Read outputs from 5 parallel quality subprocesses, calculate weighted overall sc
|
|
|
37
37
|
### 1. Read All Subprocess Outputs
|
|
38
38
|
|
|
39
39
|
```javascript
|
|
40
|
-
|
|
40
|
+
// Use the SAME timestamp generated in Step 3 (do not regenerate).
|
|
41
|
+
const timestamp = subprocessContext?.timestamp;
|
|
42
|
+
if (!timestamp) {
|
|
43
|
+
throw new Error('Missing timestamp from Step 3 context. Pass Step 3 timestamp into Step 3F.');
|
|
44
|
+
}
|
|
45
|
+
const dimensions = ['determinism', 'isolation', 'maintainability', 'performance'];
|
|
41
46
|
const results = {};
|
|
42
47
|
|
|
43
48
|
dimensions.forEach((dim) => {
|
|
44
|
-
const outputPath = `/tmp/tea-test-review-${dim}
|
|
49
|
+
const outputPath = `/tmp/tea-test-review-${dim}-${timestamp}.json`;
|
|
45
50
|
results[dim] = JSON.parse(fs.readFileSync(outputPath, 'utf8'));
|
|
46
51
|
});
|
|
47
52
|
```
|
|
@@ -63,11 +68,10 @@ if (!allSucceeded) {
|
|
|
63
68
|
|
|
64
69
|
```javascript
|
|
65
70
|
const weights = {
|
|
66
|
-
determinism: 0.
|
|
67
|
-
isolation: 0.
|
|
68
|
-
maintainability: 0.
|
|
69
|
-
|
|
70
|
-
performance: 0.15, // 15% - Important but less critical than correctness
|
|
71
|
+
determinism: 0.3, // 30% - Reliability and flake prevention
|
|
72
|
+
isolation: 0.3, // 30% - Parallel safety and independence
|
|
73
|
+
maintainability: 0.25, // 25% - Readability and long-term health
|
|
74
|
+
performance: 0.15, // 15% - Speed and execution efficiency
|
|
71
75
|
};
|
|
72
76
|
```
|
|
73
77
|
|
|
@@ -157,7 +161,6 @@ const reviewSummary = {
|
|
|
157
161
|
determinism: results.determinism.score,
|
|
158
162
|
isolation: results.isolation.score,
|
|
159
163
|
maintainability: results.maintainability.score,
|
|
160
|
-
coverage: results.coverage.score,
|
|
161
164
|
performance: results.performance.score,
|
|
162
165
|
},
|
|
163
166
|
|
|
@@ -165,7 +168,6 @@ const reviewSummary = {
|
|
|
165
168
|
determinism: results.determinism.grade,
|
|
166
169
|
isolation: results.isolation.grade,
|
|
167
170
|
maintainability: results.maintainability.grade,
|
|
168
|
-
coverage: results.coverage.grade,
|
|
169
171
|
performance: results.performance.grade,
|
|
170
172
|
},
|
|
171
173
|
|
|
@@ -177,12 +179,12 @@ const reviewSummary = {
|
|
|
177
179
|
|
|
178
180
|
top_10_recommendations: prioritizedRecommendations,
|
|
179
181
|
|
|
180
|
-
subprocess_execution: 'PARALLEL (
|
|
182
|
+
subprocess_execution: 'PARALLEL (4 quality dimensions)',
|
|
181
183
|
performance_gain: '~60% faster than sequential',
|
|
182
184
|
};
|
|
183
185
|
|
|
184
186
|
// Save for Step 4 (report generation)
|
|
185
|
-
fs.writeFileSync(
|
|
187
|
+
fs.writeFileSync(`/tmp/tea-test-review-summary-${timestamp}.json`, JSON.stringify(reviewSummary, null, 2), 'utf8');
|
|
186
188
|
```
|
|
187
189
|
|
|
188
190
|
---
|
|
@@ -198,9 +200,10 @@ fs.writeFileSync('/tmp/tea-test-review-summary-{{timestamp}}.json', JSON.stringi
|
|
|
198
200
|
- Determinism: {determinism_score}/100 ({determinism_grade})
|
|
199
201
|
- Isolation: {isolation_score}/100 ({isolation_grade})
|
|
200
202
|
- Maintainability: {maintainability_score}/100 ({maintainability_grade})
|
|
201
|
-
- Coverage: {coverage_score}/100 ({coverage_grade})
|
|
202
203
|
- Performance: {performance_score}/100 ({performance_grade})
|
|
203
204
|
|
|
205
|
+
ℹ️ Coverage is excluded from `test-review` scoring. Use `trace` for coverage analysis and gates.
|
|
206
|
+
|
|
204
207
|
⚠️ Violations Found:
|
|
205
208
|
- HIGH: {high_count} violations
|
|
206
209
|
- MEDIUM: {medium_count} violations
|
|
@@ -260,7 +263,7 @@ Load next step: `{nextStepFile}`
|
|
|
260
263
|
|
|
261
264
|
### ✅ SUCCESS:
|
|
262
265
|
|
|
263
|
-
- All
|
|
266
|
+
- All 4 subprocess outputs read and parsed
|
|
264
267
|
- Overall score calculated with proper weights
|
|
265
268
|
- Violations aggregated correctly
|
|
266
269
|
- Summary complete and saved
|
|
@@ -271,4 +274,4 @@ Load next step: `{nextStepFile}`
|
|
|
271
274
|
- Score calculation incorrect
|
|
272
275
|
- Summary missing or incomplete
|
|
273
276
|
|
|
274
|
-
**Master Rule:**
|
|
277
|
+
**Master Rule:** Aggregate determinism, isolation, maintainability, and performance only.
|
|
@@ -42,6 +42,7 @@ Use `test-review-template.md` to produce `{outputFile}` including:
|
|
|
42
42
|
- Critical findings with fixes
|
|
43
43
|
- Warnings and recommendations
|
|
44
44
|
- Context references (story/test-design if available)
|
|
45
|
+
- Coverage boundary note: `test-review` does not score coverage. Direct coverage findings to `trace`.
|
|
45
46
|
|
|
46
47
|
---
|
|
47
48
|
|
|
@@ -14,6 +14,7 @@ lastSaved: ''
|
|
|
14
14
|
---
|
|
15
15
|
|
|
16
16
|
Note: This review audits existing tests; it does not generate tests.
|
|
17
|
+
Coverage mapping and coverage gates are out of scope here. Use `trace` for coverage decisions.
|
|
17
18
|
|
|
18
19
|
## Executive Summary
|
|
19
20
|
|
|
@@ -216,7 +217,7 @@ Grade: {grade}
|
|
|
216
217
|
- **Fixtures Used**: {fixture_count} ({fixture_names})
|
|
217
218
|
- **Data Factories Used**: {factory_count} ({factory_names})
|
|
218
219
|
|
|
219
|
-
### Test
|
|
220
|
+
### Test Scope
|
|
220
221
|
|
|
221
222
|
- **Test IDs**: {test_id_list}
|
|
222
223
|
- **Priority Distribution**:
|
|
@@ -241,7 +242,6 @@ Grade: {grade}
|
|
|
241
242
|
{If story file found:}
|
|
242
243
|
|
|
243
244
|
- **Story File**: [{story_filename}]({story_path})
|
|
244
|
-
- **Acceptance Criteria Mapped**: {ac_mapped}/{ac_total} ({ac_coverage}%)
|
|
245
245
|
|
|
246
246
|
{If test-design found:}
|
|
247
247
|
|
|
@@ -249,18 +249,6 @@ Grade: {grade}
|
|
|
249
249
|
- **Risk Assessment**: {risk_level}
|
|
250
250
|
- **Priority Framework**: P0-P3 applied
|
|
251
251
|
|
|
252
|
-
### Acceptance Criteria Validation
|
|
253
|
-
|
|
254
|
-
{If story file available, map tests to ACs:}
|
|
255
|
-
|
|
256
|
-
| Acceptance Criterion | Test ID | Status | Notes |
|
|
257
|
-
| -------------------- | --------- | -------------------------- | ------- |
|
|
258
|
-
| {AC_1} | {test_id} | {✅ Covered \| ❌ Missing} | {notes} |
|
|
259
|
-
| {AC_2} | {test_id} | {✅ Covered \| ❌ Missing} | {notes} |
|
|
260
|
-
| {AC_3} | {test_id} | {✅ Covered \| ❌ Missing} | {notes} |
|
|
261
|
-
|
|
262
|
-
**Coverage**: {covered_count}/{total_count} criteria covered ({coverage_percentage}%)
|
|
263
|
-
|
|
264
252
|
---
|
|
265
253
|
|
|
266
254
|
## Knowledge Base References
|
|
@@ -276,7 +264,8 @@ This review consulted the following knowledge base fragments:
|
|
|
276
264
|
- **[selective-testing.md](../../../testarch/knowledge/selective-testing.md)** - Duplicate coverage detection
|
|
277
265
|
- **[ci-burn-in.md](../../../testarch/knowledge/ci-burn-in.md)** - Flakiness detection patterns (10-iteration loop)
|
|
278
266
|
- **[test-priorities.md](../../../testarch/knowledge/test-priorities.md)** - P0/P1/P2/P3 classification framework
|
|
279
|
-
|
|
267
|
+
|
|
268
|
+
For coverage mapping, consult `trace` workflow outputs.
|
|
280
269
|
|
|
281
270
|
See [tea-index.csv](../../../testarch/tea-index.csv) for complete knowledge base.
|
|
282
271
|
|
|
@@ -95,6 +95,10 @@ This checklist covers **two sequential phases**:
|
|
|
95
95
|
- [ ] Criteria with PARTIAL status
|
|
96
96
|
- [ ] Criteria with UNIT-ONLY status
|
|
97
97
|
- [ ] Criteria with INTEGRATION-ONLY status
|
|
98
|
+
- [ ] Coverage heuristics gaps identified:
|
|
99
|
+
- [ ] Endpoints referenced in requirements but not covered by API tests
|
|
100
|
+
- [ ] Auth/authz criteria missing denied/invalid path tests
|
|
101
|
+
- [ ] Criteria with happy-path-only coverage (missing error scenarios)
|
|
98
102
|
- [ ] Gaps prioritized by risk level using test-priorities framework:
|
|
99
103
|
- [ ] **CRITICAL** - P0 criteria without FULL coverage (BLOCKER)
|
|
100
104
|
- [ ] **HIGH** - P1 criteria without FULL coverage (PR blocker)
|
|
@@ -306,9 +310,10 @@ Knowledge fragments referenced:
|
|
|
306
310
|
**P1 Criteria Evaluation:**
|
|
307
311
|
|
|
308
312
|
- [ ] P1 test pass rate evaluated (threshold: min_p1_pass_rate)
|
|
309
|
-
- [ ] P1 acceptance criteria coverage evaluated (
|
|
313
|
+
- [ ] P1 acceptance criteria coverage evaluated (PASS >=90%, CONCERNS 80-89%, FAIL <80%)
|
|
310
314
|
- [ ] Overall test pass rate evaluated (threshold: min_overall_pass_rate)
|
|
311
|
-
- [ ]
|
|
315
|
+
- [ ] Overall requirements coverage evaluated (threshold: >=80%)
|
|
316
|
+
- [ ] Code coverage considered if available (informational unless explicitly required by policy)
|
|
312
317
|
- [ ] P1 decision recorded: PASS or CONCERNS
|
|
313
318
|
|
|
314
319
|
**P2/P3 Criteria Evaluation:**
|
|
@@ -58,7 +58,25 @@ Record test IDs, describe blocks, and priority markers if present.
|
|
|
58
58
|
|
|
59
59
|
---
|
|
60
60
|
|
|
61
|
-
|
|
61
|
+
## 3. Build Coverage Heuristics Inventory
|
|
62
|
+
|
|
63
|
+
Capture explicit coverage signals so Phase 1 can detect common blind spots:
|
|
64
|
+
|
|
65
|
+
- API endpoint coverage
|
|
66
|
+
- Inventory endpoints referenced by requirements/specs and endpoints exercised by API tests
|
|
67
|
+
- Mark endpoints with no direct tests
|
|
68
|
+
- Authentication/authorization coverage
|
|
69
|
+
- Detect tests for login/session/token flows and permission-denied paths
|
|
70
|
+
- Mark auth/authz requirements with missing negative-path tests
|
|
71
|
+
- Error-path coverage
|
|
72
|
+
- Detect validation, timeout, network-failure, and server-error scenarios
|
|
73
|
+
- Mark criteria with happy-path-only tests
|
|
74
|
+
|
|
75
|
+
Record these findings in step output as `coverage_heuristics` for Step 3/4.
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
### 4. Save Progress
|
|
62
80
|
|
|
63
81
|
**Save this step's accumulated work to `{outputFile}`.**
|
|
64
82
|
|
|
@@ -42,6 +42,10 @@ For each acceptance criterion:
|
|
|
42
42
|
- Map to matching tests
|
|
43
43
|
- Mark coverage status: FULL / PARTIAL / NONE / UNIT-ONLY / INTEGRATION-ONLY
|
|
44
44
|
- Record test level and priority
|
|
45
|
+
- Record heuristic signals:
|
|
46
|
+
- Endpoint coverage present/missing (for API-impacting criteria)
|
|
47
|
+
- Auth/authz coverage present/missing (positive and negative paths)
|
|
48
|
+
- Error-path coverage present/missing (validation, timeout, network/server failures)
|
|
45
49
|
|
|
46
50
|
---
|
|
47
51
|
|
|
@@ -51,6 +55,9 @@ Ensure:
|
|
|
51
55
|
|
|
52
56
|
- P0/P1 criteria have coverage
|
|
53
57
|
- No duplicate coverage across levels without justification
|
|
58
|
+
- Criteria are not happy-path-only when requirements imply error handling
|
|
59
|
+
- API criteria are not marked FULL if endpoint-level checks are missing
|
|
60
|
+
- Auth/authz criteria include at least one denied/invalid-path test where applicable
|
|
54
61
|
|
|
55
62
|
---
|
|
56
63
|
|
|
@@ -10,7 +10,7 @@ tempOutputFile: '/tmp/tea-trace-coverage-matrix-{{timestamp}}.json'
|
|
|
10
10
|
|
|
11
11
|
## STEP GOAL
|
|
12
12
|
|
|
13
|
-
**Phase 1 Final Step:** Analyze coverage gaps, generate recommendations, and output complete coverage matrix to temp file for Phase 2 (gate decision).
|
|
13
|
+
**Phase 1 Final Step:** Analyze coverage gaps (including endpoint/auth/error-path blind spots), generate recommendations, and output complete coverage matrix to temp file for Phase 2 (gate decision).
|
|
14
14
|
|
|
15
15
|
---
|
|
16
16
|
|
|
@@ -60,7 +60,27 @@ const lowGaps = uncoveredRequirements.filter((req) => req.priority === 'P3');
|
|
|
60
60
|
|
|
61
61
|
---
|
|
62
62
|
|
|
63
|
-
### 2.
|
|
63
|
+
### 2. Coverage Heuristics Checks
|
|
64
|
+
|
|
65
|
+
Use the heuristics inventory from Step 2 and mapped criteria from Step 3 to flag common coverage blind spots:
|
|
66
|
+
|
|
67
|
+
```javascript
|
|
68
|
+
const endpointCoverageGaps = coverageHeuristics?.endpoints_without_tests || [];
|
|
69
|
+
const authCoverageGaps = coverageHeuristics?.auth_missing_negative_paths || [];
|
|
70
|
+
const errorPathGaps = coverageHeuristics?.criteria_happy_path_only || [];
|
|
71
|
+
|
|
72
|
+
const heuristicGapCounts = {
|
|
73
|
+
endpoints_without_tests: endpointCoverageGaps.length,
|
|
74
|
+
auth_missing_negative_paths: authCoverageGaps.length,
|
|
75
|
+
happy_path_only_criteria: errorPathGaps.length,
|
|
76
|
+
};
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Heuristics are advisory but must influence gap severity and recommendations, especially for P0/P1 criteria.
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
### 3. Generate Recommendations
|
|
64
84
|
|
|
65
85
|
**Based on gap analysis:**
|
|
66
86
|
|
|
@@ -94,6 +114,30 @@ if (partialCoverage.length > 0) {
|
|
|
94
114
|
});
|
|
95
115
|
}
|
|
96
116
|
|
|
117
|
+
if (endpointCoverageGaps.length > 0) {
|
|
118
|
+
recommendations.push({
|
|
119
|
+
priority: 'HIGH',
|
|
120
|
+
action: `Add API tests for ${endpointCoverageGaps.length} uncovered endpoint(s)`,
|
|
121
|
+
requirements: endpointCoverageGaps.map((r) => r.id || r.endpoint || 'unknown'),
|
|
122
|
+
});
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
if (authCoverageGaps.length > 0) {
|
|
126
|
+
recommendations.push({
|
|
127
|
+
priority: 'HIGH',
|
|
128
|
+
action: `Add negative-path auth/authz tests for ${authCoverageGaps.length} requirement(s)`,
|
|
129
|
+
requirements: authCoverageGaps.map((r) => r.id || 'unknown'),
|
|
130
|
+
});
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
if (errorPathGaps.length > 0) {
|
|
134
|
+
recommendations.push({
|
|
135
|
+
priority: 'MEDIUM',
|
|
136
|
+
action: `Add error/edge scenario tests for ${errorPathGaps.length} happy-path-only criterion/criteria`,
|
|
137
|
+
requirements: errorPathGaps.map((r) => r.id || 'unknown'),
|
|
138
|
+
});
|
|
139
|
+
}
|
|
140
|
+
|
|
97
141
|
// Quality issues
|
|
98
142
|
recommendations.push({
|
|
99
143
|
priority: 'LOW',
|
|
@@ -104,24 +148,35 @@ recommendations.push({
|
|
|
104
148
|
|
|
105
149
|
---
|
|
106
150
|
|
|
107
|
-
###
|
|
151
|
+
### 4. Calculate Coverage Statistics
|
|
108
152
|
|
|
109
153
|
```javascript
|
|
110
154
|
const totalRequirements = traceabilityMatrix.length;
|
|
111
155
|
const coveredRequirements = traceabilityMatrix.filter((r) => r.coverage === 'FULL' || r.coverage === 'PARTIAL').length;
|
|
112
156
|
const fullyCovered = traceabilityMatrix.filter((r) => r.coverage === 'FULL').length;
|
|
113
157
|
|
|
114
|
-
const
|
|
158
|
+
const safePct = (covered, total) => (total > 0 ? Math.round((covered / total) * 100) : 100);
|
|
159
|
+
const coveragePercentage = safePct(fullyCovered, totalRequirements);
|
|
115
160
|
|
|
116
161
|
// Priority-specific coverage
|
|
117
162
|
const p0Total = traceabilityMatrix.filter((r) => r.priority === 'P0').length;
|
|
118
163
|
const p0Covered = traceabilityMatrix.filter((r) => r.priority === 'P0' && r.coverage === 'FULL').length;
|
|
119
|
-
const
|
|
164
|
+
const p1Total = traceabilityMatrix.filter((r) => r.priority === 'P1').length;
|
|
165
|
+
const p1Covered = traceabilityMatrix.filter((r) => r.priority === 'P1' && r.coverage === 'FULL').length;
|
|
166
|
+
const p2Total = traceabilityMatrix.filter((r) => r.priority === 'P2').length;
|
|
167
|
+
const p2Covered = traceabilityMatrix.filter((r) => r.priority === 'P2' && r.coverage === 'FULL').length;
|
|
168
|
+
const p3Total = traceabilityMatrix.filter((r) => r.priority === 'P3').length;
|
|
169
|
+
const p3Covered = traceabilityMatrix.filter((r) => r.priority === 'P3' && r.coverage === 'FULL').length;
|
|
170
|
+
|
|
171
|
+
const p0CoveragePercentage = safePct(p0Covered, p0Total);
|
|
172
|
+
const p1CoveragePercentage = safePct(p1Covered, p1Total);
|
|
173
|
+
const p2CoveragePercentage = safePct(p2Covered, p2Total);
|
|
174
|
+
const p3CoveragePercentage = safePct(p3Covered, p3Total);
|
|
120
175
|
```
|
|
121
176
|
|
|
122
177
|
---
|
|
123
178
|
|
|
124
|
-
###
|
|
179
|
+
### 5. Generate Complete Coverage Matrix
|
|
125
180
|
|
|
126
181
|
**Compile all Phase 1 outputs:**
|
|
127
182
|
|
|
@@ -141,15 +196,9 @@ const coverageMatrix = {
|
|
|
141
196
|
|
|
142
197
|
priority_breakdown: {
|
|
143
198
|
P0: { total: p0Total, covered: p0Covered, percentage: p0CoveragePercentage },
|
|
144
|
-
P1: {
|
|
145
|
-
|
|
146
|
-
},
|
|
147
|
-
P2: {
|
|
148
|
-
/* calculate */
|
|
149
|
-
},
|
|
150
|
-
P3: {
|
|
151
|
-
/* calculate */
|
|
152
|
-
},
|
|
199
|
+
P1: { total: p1Total, covered: p1Covered, percentage: p1CoveragePercentage },
|
|
200
|
+
P2: { total: p2Total, covered: p2Covered, percentage: p2CoveragePercentage },
|
|
201
|
+
P3: { total: p3Total, covered: p3Covered, percentage: p3CoveragePercentage },
|
|
153
202
|
},
|
|
154
203
|
},
|
|
155
204
|
|
|
@@ -162,13 +211,20 @@ const coverageMatrix = {
|
|
|
162
211
|
unit_only_items: unitOnlyCoverage,
|
|
163
212
|
},
|
|
164
213
|
|
|
214
|
+
coverage_heuristics: {
|
|
215
|
+
endpoint_gaps: endpointCoverageGaps,
|
|
216
|
+
auth_negative_path_gaps: authCoverageGaps,
|
|
217
|
+
happy_path_only_gaps: errorPathGaps,
|
|
218
|
+
counts: heuristicGapCounts,
|
|
219
|
+
},
|
|
220
|
+
|
|
165
221
|
recommendations: recommendations,
|
|
166
222
|
};
|
|
167
223
|
```
|
|
168
224
|
|
|
169
225
|
---
|
|
170
226
|
|
|
171
|
-
###
|
|
227
|
+
### 6. Output Coverage Matrix to Temp File
|
|
172
228
|
|
|
173
229
|
**Write to temp file for Phase 2:**
|
|
174
230
|
|
|
@@ -181,7 +237,7 @@ console.log(`✅ Phase 1 Complete: Coverage matrix saved to ${outputPath}`);
|
|
|
181
237
|
|
|
182
238
|
---
|
|
183
239
|
|
|
184
|
-
###
|
|
240
|
+
### 7. Display Phase 1 Summary
|
|
185
241
|
|
|
186
242
|
```
|
|
187
243
|
✅ Phase 1 Complete: Coverage Matrix Generated
|
|
@@ -194,9 +250,9 @@ console.log(`✅ Phase 1 Complete: Coverage matrix saved to ${outputPath}`);
|
|
|
194
250
|
|
|
195
251
|
🎯 Priority Coverage:
|
|
196
252
|
- P0: {p0Covered}/{p0Total} ({p0CoveragePercentage}%)
|
|
197
|
-
- P1: {
|
|
198
|
-
- P2: {
|
|
199
|
-
- P3: {
|
|
253
|
+
- P1: {p1Covered}/{p1Total} ({p1CoveragePercentage}%)
|
|
254
|
+
- P2: {p2Covered}/{p2Total} ({p2CoveragePercentage}%)
|
|
255
|
+
- P3: {p3Covered}/{p3Total} ({p3CoveragePercentage}%)
|
|
200
256
|
|
|
201
257
|
⚠️ Gaps Identified:
|
|
202
258
|
- Critical (P0): {criticalGaps.length}
|
|
@@ -204,6 +260,11 @@ console.log(`✅ Phase 1 Complete: Coverage matrix saved to ${outputPath}`);
|
|
|
204
260
|
- Medium (P2): {mediumGaps.length}
|
|
205
261
|
- Low (P3): {lowGaps.length}
|
|
206
262
|
|
|
263
|
+
🔍 Coverage Heuristics:
|
|
264
|
+
- Endpoints without tests: {endpointCoverageGaps.length}
|
|
265
|
+
- Auth negative-path gaps: {authCoverageGaps.length}
|
|
266
|
+
- Happy-path-only criteria: {errorPathGaps.length}
|
|
267
|
+
|
|
207
268
|
📝 Recommendations: {recommendations.length}
|
|
208
269
|
|
|
209
270
|
🔄 Phase 2: Gate decision (next step)
|
|
@@ -225,7 +286,7 @@ console.log(`✅ Phase 1 Complete: Coverage matrix saved to ${outputPath}`);
|
|
|
225
286
|
|
|
226
287
|
---
|
|
227
288
|
|
|
228
|
-
###
|
|
289
|
+
### 8. Save Progress
|
|
229
290
|
|
|
230
291
|
**Save this step's accumulated work to `{outputFile}`.**
|
|
231
292
|
|
|
@@ -64,6 +64,9 @@ if (coverageMatrix.phase !== 'PHASE_1_COMPLETE') {
|
|
|
64
64
|
```javascript
|
|
65
65
|
const stats = coverageMatrix.coverage_statistics;
|
|
66
66
|
const p0Coverage = stats.priority_breakdown.P0.percentage;
|
|
67
|
+
const p1Coverage = stats.priority_breakdown.P1.percentage;
|
|
68
|
+
const hasP1Requirements = (stats.priority_breakdown.P1.total || 0) > 0;
|
|
69
|
+
const effectiveP1Coverage = hasP1Requirements ? p1Coverage : 100;
|
|
67
70
|
const overallCoverage = stats.overall_coverage_percentage;
|
|
68
71
|
const criticalGaps = coverageMatrix.gap_analysis.critical_gaps.length;
|
|
69
72
|
|
|
@@ -75,23 +78,34 @@ if (p0Coverage < 100) {
|
|
|
75
78
|
gateDecision = 'FAIL';
|
|
76
79
|
rationale = `P0 coverage is ${p0Coverage}% (required: 100%). ${criticalGaps} critical requirements uncovered.`;
|
|
77
80
|
}
|
|
78
|
-
// Rule 2: Overall coverage
|
|
79
|
-
else if (overallCoverage
|
|
81
|
+
// Rule 2: Overall coverage must be >= 80%
|
|
82
|
+
else if (overallCoverage < 80) {
|
|
83
|
+
gateDecision = 'FAIL';
|
|
84
|
+
rationale = `Overall coverage is ${overallCoverage}% (minimum: 80%). Significant gaps exist.`;
|
|
85
|
+
}
|
|
86
|
+
// Rule 3: P1 coverage < 80% → FAIL
|
|
87
|
+
else if (effectiveP1Coverage < 80) {
|
|
88
|
+
gateDecision = 'FAIL';
|
|
89
|
+
rationale = hasP1Requirements
|
|
90
|
+
? `P1 coverage is ${effectiveP1Coverage}% (minimum: 80%). High-priority gaps must be addressed.`
|
|
91
|
+
: `P1 requirements are not present; continuing with remaining gate criteria.`;
|
|
92
|
+
}
|
|
93
|
+
// Rule 4: P1 coverage >= 90% and overall >= 80% with P0 at 100% → PASS
|
|
94
|
+
else if (effectiveP1Coverage >= 90) {
|
|
80
95
|
gateDecision = 'PASS';
|
|
81
|
-
rationale =
|
|
96
|
+
rationale = hasP1Requirements
|
|
97
|
+
? `P0 coverage is 100%, P1 coverage is ${effectiveP1Coverage}% (target: 90%), and overall coverage is ${overallCoverage}% (minimum: 80%).`
|
|
98
|
+
: `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%). No P1 requirements detected.`;
|
|
82
99
|
}
|
|
83
|
-
// Rule
|
|
84
|
-
else if (
|
|
100
|
+
// Rule 5: P1 coverage 80-89% with P0 at 100% and overall >= 80% → CONCERNS
|
|
101
|
+
else if (effectiveP1Coverage >= 80) {
|
|
85
102
|
gateDecision = 'CONCERNS';
|
|
86
|
-
rationale =
|
|
87
|
-
}
|
|
88
|
-
|
|
89
|
-
else {
|
|
90
|
-
gateDecision = 'FAIL';
|
|
91
|
-
rationale = `Overall coverage is ${overallCoverage}% (minimum: 75%). Significant gaps exist.`;
|
|
103
|
+
rationale = hasP1Requirements
|
|
104
|
+
? `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%), but P1 coverage is ${effectiveP1Coverage}% (target: 90%).`
|
|
105
|
+
: `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%), but additional non-P1 gaps need mitigation.`;
|
|
92
106
|
}
|
|
93
107
|
|
|
94
|
-
// Rule
|
|
108
|
+
// Rule 6: Manual waiver option
|
|
95
109
|
const manualWaiver = false; // Can be set via config or user input
|
|
96
110
|
if (manualWaiver) {
|
|
97
111
|
gateDecision = 'WAIVED';
|
|
@@ -116,9 +130,14 @@ const gateReport = {
|
|
|
116
130
|
p0_coverage_actual: `${p0Coverage}%`,
|
|
117
131
|
p0_status: p0Coverage === 100 ? 'MET' : 'NOT MET',
|
|
118
132
|
|
|
119
|
-
|
|
133
|
+
p1_coverage_target_pass: '90%',
|
|
134
|
+
p1_coverage_minimum: '80%',
|
|
135
|
+
p1_coverage_actual: `${effectiveP1Coverage}%`,
|
|
136
|
+
p1_status: effectiveP1Coverage >= 90 ? 'MET' : effectiveP1Coverage >= 80 ? 'PARTIAL' : 'NOT MET',
|
|
137
|
+
|
|
138
|
+
overall_coverage_minimum: '80%',
|
|
120
139
|
overall_coverage_actual: `${overallCoverage}%`,
|
|
121
|
-
overall_status: overallCoverage >=
|
|
140
|
+
overall_status: overallCoverage >= 80 ? 'MET' : 'NOT MET',
|
|
122
141
|
},
|
|
123
142
|
|
|
124
143
|
uncovered_requirements: coverageMatrix.gap_analysis.critical_gaps.concat(coverageMatrix.gap_analysis.high_gaps),
|
|
@@ -174,7 +193,8 @@ fs.writeFileSync('{outputFile}', reportContent, 'utf8');
|
|
|
174
193
|
|
|
175
194
|
📊 Coverage Analysis:
|
|
176
195
|
- P0 Coverage: {p0Coverage}% (Required: 100%) → {p0_status}
|
|
177
|
-
-
|
|
196
|
+
- P1 Coverage: {effectiveP1Coverage}% (PASS target: 90%, minimum: 80%) → {p1_status}
|
|
197
|
+
- Overall Coverage: {overallCoverage}% (Minimum: 80%) → {overall_status}
|
|
178
198
|
|
|
179
199
|
✅ Decision Rationale:
|
|
180
200
|
{rationale}
|
|
@@ -243,4 +263,4 @@ Then append the gate decision summary (from section 5 above) to the end of the e
|
|
|
243
263
|
- Gate decision logic incorrect
|
|
244
264
|
- Report missing or incomplete
|
|
245
265
|
|
|
246
|
-
**Master Rule:** Gate decision MUST be deterministic based on clear criteria (P0 100%,
|
|
266
|
+
**Master Rule:** Gate decision MUST be deterministic based on clear criteria (P0 100%, P1 90/80, overall >=80).
|
|
@@ -136,6 +136,31 @@ Note: This workflow does not generate tests. If gaps exist, run `*atdd` or `*aut
|
|
|
136
136
|
|
|
137
137
|
---
|
|
138
138
|
|
|
139
|
+
### Coverage Heuristics Findings
|
|
140
|
+
|
|
141
|
+
#### Endpoint Coverage Gaps
|
|
142
|
+
|
|
143
|
+
- Endpoints without direct API tests: {endpoint_gap_count}
|
|
144
|
+
- Examples:
|
|
145
|
+
- {endpoint_gap_1}
|
|
146
|
+
- {endpoint_gap_2}
|
|
147
|
+
|
|
148
|
+
#### Auth/Authz Negative-Path Gaps
|
|
149
|
+
|
|
150
|
+
- Criteria missing denied/invalid-path tests: {auth_negative_gap_count}
|
|
151
|
+
- Examples:
|
|
152
|
+
- {auth_gap_1}
|
|
153
|
+
- {auth_gap_2}
|
|
154
|
+
|
|
155
|
+
#### Happy-Path-Only Criteria
|
|
156
|
+
|
|
157
|
+
- Criteria missing error/edge scenarios: {happy_path_only_gap_count}
|
|
158
|
+
- Examples:
|
|
159
|
+
- {happy_path_gap_1}
|
|
160
|
+
- {happy_path_gap_2}
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
139
164
|
### Quality Assessment
|
|
140
165
|
|
|
141
166
|
#### Tests with Issues
|
|
@@ -1,111 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: 'step-03d-subprocess-coverage'
|
|
3
|
-
description: 'Subprocess: Check test coverage (completeness, edge cases)'
|
|
4
|
-
subprocess: true
|
|
5
|
-
outputFile: '/tmp/tea-test-review-coverage-{{timestamp}}.json'
|
|
6
|
-
---
|
|
7
|
-
|
|
8
|
-
# Subprocess 3D: Coverage Quality Check
|
|
9
|
-
|
|
10
|
-
## SUBPROCESS CONTEXT
|
|
11
|
-
|
|
12
|
-
This is an **isolated subprocess** running in parallel with other quality dimension checks.
|
|
13
|
-
|
|
14
|
-
**Your task:** Analyze test files for COVERAGE violations only.
|
|
15
|
-
|
|
16
|
-
---
|
|
17
|
-
|
|
18
|
-
## MANDATORY EXECUTION RULES
|
|
19
|
-
|
|
20
|
-
- ✅ Check COVERAGE only (not other quality dimensions)
|
|
21
|
-
- ✅ Output structured JSON to temp file
|
|
22
|
-
- ❌ Do NOT check determinism, isolation, maintainability, or performance
|
|
23
|
-
|
|
24
|
-
---
|
|
25
|
-
|
|
26
|
-
## SUBPROCESS TASK
|
|
27
|
-
|
|
28
|
-
### 1. Identify Coverage Violations
|
|
29
|
-
|
|
30
|
-
**HIGH SEVERITY Violations**:
|
|
31
|
-
|
|
32
|
-
- Critical user paths not tested (P0 functionality missing)
|
|
33
|
-
- API endpoints without tests
|
|
34
|
-
- Error handling not tested (no negative test cases)
|
|
35
|
-
- Missing authentication/authorization tests
|
|
36
|
-
|
|
37
|
-
**MEDIUM SEVERITY Violations**:
|
|
38
|
-
|
|
39
|
-
- Edge cases not covered (boundary values, null/empty inputs)
|
|
40
|
-
- Only happy path tested (no error scenarios)
|
|
41
|
-
- Missing integration tests (only unit or only E2E)
|
|
42
|
-
- Insufficient assertion coverage (tests don't verify important outcomes)
|
|
43
|
-
|
|
44
|
-
**LOW SEVERITY Violations**:
|
|
45
|
-
|
|
46
|
-
- Could benefit from additional test cases
|
|
47
|
-
- Minor edge cases not covered
|
|
48
|
-
- Documentation incomplete
|
|
49
|
-
|
|
50
|
-
### 2. Calculate Coverage Score
|
|
51
|
-
|
|
52
|
-
```javascript
|
|
53
|
-
const criticalGaps = violations.filter((v) => v.severity === 'HIGH').length;
|
|
54
|
-
const score = criticalGaps === 0 ? Math.max(0, 100 - violations.length * 5) : Math.max(0, 50 - criticalGaps * 10); // Heavy penalty for critical gaps
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
---
|
|
58
|
-
|
|
59
|
-
## OUTPUT FORMAT
|
|
60
|
-
|
|
61
|
-
```json
|
|
62
|
-
{
|
|
63
|
-
"dimension": "coverage",
|
|
64
|
-
"score": 70,
|
|
65
|
-
"max_score": 100,
|
|
66
|
-
"grade": "C",
|
|
67
|
-
"violations": [
|
|
68
|
-
{
|
|
69
|
-
"file": "tests/api/",
|
|
70
|
-
"severity": "HIGH",
|
|
71
|
-
"category": "missing-endpoint-tests",
|
|
72
|
-
"description": "API endpoint /api/users/delete not tested",
|
|
73
|
-
"suggestion": "Add tests for user deletion including error scenarios"
|
|
74
|
-
},
|
|
75
|
-
{
|
|
76
|
-
"file": "tests/e2e/checkout.spec.ts",
|
|
77
|
-
"line": 25,
|
|
78
|
-
"severity": "MEDIUM",
|
|
79
|
-
"category": "missing-error-case",
|
|
80
|
-
"description": "Only happy path tested - no error handling tests",
|
|
81
|
-
"suggestion": "Add tests for payment failure, network errors, validation failures"
|
|
82
|
-
}
|
|
83
|
-
],
|
|
84
|
-
"passed_checks": 8,
|
|
85
|
-
"failed_checks": 4,
|
|
86
|
-
"violation_summary": {
|
|
87
|
-
"HIGH": 1,
|
|
88
|
-
"MEDIUM": 2,
|
|
89
|
-
"LOW": 1
|
|
90
|
-
},
|
|
91
|
-
"coverage_gaps": {
|
|
92
|
-
"untested_endpoints": ["/api/users/delete", "/api/orders/cancel"],
|
|
93
|
-
"untested_user_paths": ["Password reset flow"],
|
|
94
|
-
"missing_error_scenarios": ["Payment failures", "Network timeouts"]
|
|
95
|
-
},
|
|
96
|
-
"recommendations": [
|
|
97
|
-
"Add tests for all CRUD operations (especially DELETE)",
|
|
98
|
-
"Test error scenarios for each user path",
|
|
99
|
-
"Add integration tests between API and E2E layers"
|
|
100
|
-
],
|
|
101
|
-
"summary": "Coverage has critical gaps - 4 violations (1 HIGH critical endpoint missing)"
|
|
102
|
-
}
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
---
|
|
106
|
-
|
|
107
|
-
## EXIT CONDITION
|
|
108
|
-
|
|
109
|
-
Subprocess completes when JSON output written to temp file.
|
|
110
|
-
|
|
111
|
-
**Subprocess terminates here.**
|